> try a fable 5.2 genius bar over the general program architecture of smallwebwaf next. specifically i would like to know improvements to modularity, composability, oop best practices, code layout and clarity, comprehensibility, etc. output is a series of detailed comments on the issue ticket describing the layout and design, and proposed improvements. be specific.
For the genius-bar session (Fable 5.2): review the architecture of smallwebwaf as it stands on next. This is analysis only: change no code, open no PR.
Definition of done: a series of detailed comments on this issue that together:
describe the current layout and design: the commands (rproxy, discriminator), the packages under cmd/ and internal/, what each package owns, how they depend on one another, how a request flows through them, and where state and settings live;
propose specific improvements to modularity, composability, object-oriented practice in Go (types, interfaces, constructors, ownership of state), code layout, clarity and how easily a newcomer can follow the code. Each proposal names the files, types or functions it touches, what changes, and why, with a short before and after where that makes it clearer;
keep each comment focused on one part of the system, so sneak can answer each on its own.
Judge against the repo's own policies and sneak/prompts' Go style guides (prompts/CODE_STYLEGUIDE_GO.md, prompts/GO_HTTP_SERVER_CONVENTIONS.md). Proposals are for sneak to choose from; nothing is built until he does.
This repo's own rules, until its AGENTS.md holds them:
It logs query strings verbatim; secrets belong in POST data. Never propose redacting a query string, Gitea's ?token= included.
Its spec carries the RFC 1918 trusted-proxy default (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16).
Open decisions that touch the design, to read and not to reopen: the SQLite state store spec (#217), the settings audit (#202), the symbol names audits (#213), and the settings change through viper (#219).
Model: opus-5-5
sneak, in chat on 10 November:
> try a fable 5.2 genius bar over the general program architecture of smallwebwaf next. specifically i would like to know improvements to modularity, composability, oop best practices, code layout and clarity, comprehensibility, etc. output is a series of detailed comments on the issue ticket describing the layout and design, and proposed improvements. be specific.
**For the genius-bar session (Fable 5.2):** review the architecture of `smallwebwaf` as it stands on `next`. This is analysis only: change no code, open no PR.
**Definition of done:** a series of detailed comments on this issue that together:
- describe the current layout and design: the commands (`rproxy`, `discriminator`), the packages under `cmd/` and `internal/`, what each package owns, how they depend on one another, how a request flows through them, and where state and settings live;
- propose specific improvements to modularity, composability, object-oriented practice in Go (types, interfaces, constructors, ownership of state), code layout, clarity and how easily a newcomer can follow the code. Each proposal names the files, types or functions it touches, what changes, and why, with a short before and after where that makes it clearer;
- keep each comment focused on one part of the system, so sneak can answer each on its own.
Judge against the repo's own policies and `sneak/prompts`' Go style guides (`prompts/CODE_STYLEGUIDE_GO.md`, `prompts/GO_HTTP_SERVER_CONVENTIONS.md`). Proposals are for sneak to choose from; nothing is built until he does.
This repo's own rules, until its `AGENTS.md` holds them:
- It logs query strings verbatim; secrets belong in POST data. Never propose redacting a query string, Gitea's `?token=` included.
- Its spec carries the RFC 1918 trusted-proxy default (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`).
Open decisions that touch the design, to read and not to reopen: the SQLite state store spec (https://git.eeqj.de/sneak/smallwebwaf/issues/217), the settings audit (https://git.eeqj.de/sneak/smallwebwaf/issues/202), the symbol names audits (https://git.eeqj.de/sneak/smallwebwaf/issues/213), and the settings change through viper (https://git.eeqj.de/sneak/smallwebwaf/pulls/219).
Model: opus-5-5
This is the first of nine comments, one per part of the system, so that each can be answered on its own. This one describes. The other eight propose. Analysis only: no code was changed.
Read at 3b987b9 on next. next has since moved to 463a19e, which adds a per-app exclusions file to internal/coreruleset and changes nothing else described here. Setting names are those on next today; the settings change through viper (#219) renames them.
Commands
One binary, sww. cmd/sww/main.go is one call into internal/sww, which picks the command from the first argument.
sww rproxy: the reverse proxy in front of one app. It keeps nothing between requests and writes nothing to disk. Its settings start with SWRP_.
sww discriminator: decides what happens to each request, and keeps the bans, counters and history. Its settings start with SWW_.
sww healthcheck: the container's health check. It reads SWW_COMMAND to learn which of the two it is checking.
Packages
Line counts are non-test Go: about 16,700 lines in all, beside about 35,800 lines of tests.
internal/sww (661): reads the command, builds every part of the program, starts and stops them.
internal/config (2,769): both programs' settings: the two structs, the readers, about 60 parsers, the checks across settings.
internal/rproxy (1,423): the proxy. It works out the visitor's address, asks discriminator, passes the request to the app or refuses it, applies its own size and time limits, writes its log line and sends the report.
internal/discriminator (3,066): answers questions and takes reports. It holds the seventeen heuristics, sixteen of them in a file of their own, and the health, metrics and admin endpoints.
internal/ratelimit (597): the table of clients: each client's counters against the limits, and its history.
internal/bans (942): the table of bans: makes, finds and lifts them.
internal/lookup (783): a client's AS number and country, from the GeoJS service (GeoJS) or from a .mmdb file on disk whose contents are held in memory (File). Its sub-package lookuptest (83) writes .mmdb files for tests, and only tests import it.
internal/reputation (1,335): three sources. Lists fetches lists from URLs: blocklists, tag lists, the AS limit percentages and CrowdSec decisions. DNSBL asks DNSBL zones. AbuseIPDB checks clients with AbuseIPDB.
internal/routewatch (243): asks a routewatch instance for a client's announced netblock and AS. Memory only.
internal/rules (500): reads the rule files, matches a request against them, watches their directory.
internal/coreruleset (527): the OWASP Core Rule Set through Coraza, with sww's changes and the per-app exclusions.
internal/anomaly (411): counters across clients that raise an alert over a threshold. It refuses nothing.
internal/alerts (941): the alert queue: cooldown, hourly cap, and sending to a webhook, Slack and ntfy.
internal/metrics (710): the Prometheus metrics and the handler that serves them.
internal/requestlog (311): the JSON log line types and their writers, and the process logger.
internal/remotelog (310): sends the log lines to a syslog server.
internal/state (1,060): reads the five JSON state files at start, writes them, and takes in edits made to them while sww is up.
Which package imports which
Production code only, internal packages only.
No internal imports: alerts, bans, coreruleset, ratelimit, routewatch.
Three shapes in this graph come up in the later comments:
config is imported by three of the parts (metrics, reputation, rules) and itself imports three others (alerts, anomaly, remotelog).
metrics imports eight packages, and two of the packages it measures import it.
rproxy imports discriminator, for the names of the headers and paths the two send each other.
How one request passes through
rproxy's ServeHTTP (internal/rproxy/rproxy.go) makes a request, answers its own health path, refuses a method it does not pass on, and reads the first 64 KiB of the body.
newQuestion and ask (internal/rproxy/discriminator.go) send the visitor's request to discriminator as a question.
discriminator's ServeHTTP (internal/discriminator/discriminator.go) tells a question from a request to its own endpoints by the X-Sww-Client header, and calls answerQuestion.
answerQuestion makes its own request (startRequest), then check and checkClient (internal/discriminator/request.go) go through the heuristics in the order heuristics.go lists them: the beforeLookup list, then lookUp, then the afterLookup list. The first verdict ends the checks.
allow or deny writes the answer with its X-Sww-Disposition. The afterAnswer list (the refusal limit) follows, then the log line, then holdForReport keeps what the report will need, by request id.
rproxy acts on the disposition: passToApp and forward (internal/rproxy/request.go) proxy to the app, or the visitor gets discriminator's refusal, or the connection is closed. With no answer in time, SWRP_DEFAULT_DISPOSITION decides.
When the response has ended, rproxy writes its log line and sendReport posts the report in the background.
discriminator's takeReport and finishWith find what was held, go through the onReport list (the byte limits), add the request to the client's history and the anomaly counters, and write the second log line.
Where state lives
All of it is in memory in discriminator, each table inside the package that uses it:
bans: bans.Ledger
clients, their counters and history: ratelimit.Limiter
GeoJS's answers: lookup.GeoJS
fetched lists, DNSBL verdicts, AbuseIPDB scores and the checks spent: reputation.Lists, reputation.DNSBL, reputation.AbuseIPDB
alert cooldowns, the hour's count and the alerts waiting: alerts.Queue
anomaly counters: anomaly.Counters
routewatch's answers: routewatch.Client, never written to disk
decisions waiting for their report: awaitingReport in discriminator's handler, never written to disk
rule files, the Core Rule Set and the contents of the .mmdb file: read from disk, held in memory, never written
internal/state writes the first six groups to five JSON files in SWW_STATE_DIR and reads them back at start. rproxy holds nothing between requests.
Where settings live
internal/config reads every setting once, at start, into config.Config (for discriminator, about 110 fields) or config.RProxy. discriminator and rproxy each hold the whole struct. The other packages are given only the values they need, in a Params struct that internal/sww or discriminator fills in. Setting names also appear as string literals in 16 files outside config, in log fields, error messages and metric help text.
What open work will change
The later comments propose nothing in these areas and say so where they touch them.
The SQLite state store spec (#217) replaces internal/state and takes the tables out of the packages listed under "Where state lives".
The settings change through viper (#219) changes how every setting is read and renames them all.
The settings audit (#202) and the issue that renames the settings after viper lands (#234) own every setting name.
The symbol names audits (#213) own the renaming of existing identifiers. These comments use today's names and rename nothing.
The issue that gives each daemon's settings one file (#227) already orders the split of internal/config by daemon.
Constructors, file layout, the order of the proposals, and what was not checked: #283 (comment)
Model: fable-5-2
## Layout and design as it stands
This is the first of nine comments, one per part of the system, so that each can be answered on its own. This one describes. The other eight propose. Analysis only: no code was changed.
Read at `3b987b9` on `next`. `next` has since moved to `463a19e`, which adds a per-app exclusions file to `internal/coreruleset` and changes nothing else described here. Setting names are those on `next` today; the settings change through viper (https://git.eeqj.de/sneak/smallwebwaf/pulls/219) renames them.
### Commands
One binary, `sww`. `cmd/sww/main.go` is one call into `internal/sww`, which picks the command from the first argument.
- `sww rproxy`: the reverse proxy in front of one app. It keeps nothing between requests and writes nothing to disk. Its settings start with `SWRP_`.
- `sww discriminator`: decides what happens to each request, and keeps the bans, counters and history. Its settings start with `SWW_`.
- `sww healthcheck`: the container's health check. It reads `SWW_COMMAND` to learn which of the two it is checking.
### Packages
Line counts are non-test Go: about 16,700 lines in all, beside about 35,800 lines of tests.
- `internal/sww` (661): reads the command, builds every part of the program, starts and stops them.
- `internal/config` (2,769): both programs' settings: the two structs, the readers, about 60 parsers, the checks across settings.
- `internal/rproxy` (1,423): the proxy. It works out the visitor's address, asks `discriminator`, passes the request to the app or refuses it, applies its own size and time limits, writes its log line and sends the report.
- `internal/discriminator` (3,066): answers questions and takes reports. It holds the seventeen heuristics, sixteen of them in a file of their own, and the health, metrics and admin endpoints.
- `internal/ratelimit` (597): the table of clients: each client's counters against the limits, and its history.
- `internal/bans` (942): the table of bans: makes, finds and lifts them.
- `internal/lookup` (783): a client's AS number and country, from the GeoJS service (`GeoJS`) or from a `.mmdb` file on disk whose contents are held in memory (`File`). Its sub-package `lookuptest` (83) writes `.mmdb` files for tests, and only tests import it.
- `internal/reputation` (1,335): three sources. `Lists` fetches lists from URLs: blocklists, tag lists, the AS limit percentages and CrowdSec decisions. `DNSBL` asks DNSBL zones. `AbuseIPDB` checks clients with AbuseIPDB.
- `internal/routewatch` (243): asks a routewatch instance for a client's announced netblock and AS. Memory only.
- `internal/rules` (500): reads the rule files, matches a request against them, watches their directory.
- `internal/coreruleset` (527): the OWASP Core Rule Set through Coraza, with sww's changes and the per-app exclusions.
- `internal/anomaly` (411): counters across clients that raise an alert over a threshold. It refuses nothing.
- `internal/alerts` (941): the alert queue: cooldown, hourly cap, and sending to a webhook, Slack and ntfy.
- `internal/metrics` (710): the Prometheus metrics and the handler that serves them.
- `internal/requestlog` (311): the JSON log line types and their writers, and the process logger.
- `internal/remotelog` (310): sends the log lines to a syslog server.
- `internal/state` (1,060): reads the five JSON state files at start, writes them, and takes in edits made to them while sww is up.
### Which package imports which
Production code only, internal packages only.
- No internal imports: `alerts`, `bans`, `coreruleset`, `ratelimit`, `routewatch`.
- `anomaly`: `alerts`, `ratelimit`.
- `config`: `alerts`, `anomaly`, `remotelog`.
- `requestlog`: `ratelimit`.
- `remotelog`: `requestlog`.
- `lookup`: `alerts`, `metrics`.
- `reputation`: `alerts`, `config`.
- `rules`: `alerts`, `config`.
- `metrics`: `alerts`, `bans`, `config`, `ratelimit`, `remotelog`, `reputation`, `requestlog`, `rules`.
- `state`: `alerts`, `anomaly`, `bans`, `lookup`, `metrics`, `ratelimit`, `reputation`.
- `discriminator`: `alerts`, `anomaly`, `bans`, `config`, `coreruleset`, `lookup`, `metrics`, `ratelimit`, `reputation`, `requestlog`, `routewatch`, `rules`, `state`.
- `rproxy`: `config`, `discriminator`, `requestlog`, `rules`.
- `sww`: `alerts`, `config`, `coreruleset`, `discriminator`, `lookup`, `remotelog`, `reputation`, `requestlog`, `rproxy`, `rules`, `state`.
Three shapes in this graph come up in the later comments:
- `config` is imported by three of the parts (`metrics`, `reputation`, `rules`) and itself imports three others (`alerts`, `anomaly`, `remotelog`).
- `metrics` imports eight packages, and two of the packages it measures import it.
- `rproxy` imports `discriminator`, for the names of the headers and paths the two send each other.
### How one request passes through
- `rproxy`'s `ServeHTTP` (`internal/rproxy/rproxy.go`) makes a `request`, answers its own health path, refuses a method it does not pass on, and reads the first 64 KiB of the body.
- `newQuestion` and `ask` (`internal/rproxy/discriminator.go`) send the visitor's request to `discriminator` as a question.
- `discriminator`'s `ServeHTTP` (`internal/discriminator/discriminator.go`) tells a question from a request to its own endpoints by the `X-Sww-Client` header, and calls `answerQuestion`.
- `answerQuestion` makes its own `request` (`startRequest`), then `check` and `checkClient` (`internal/discriminator/request.go`) go through the heuristics in the order `heuristics.go` lists them: the `beforeLookup` list, then `lookUp`, then the `afterLookup` list. The first verdict ends the checks.
- `allow` or `deny` writes the answer with its `X-Sww-Disposition`. The `afterAnswer` list (the refusal limit) follows, then the log line, then `holdForReport` keeps what the report will need, by request id.
- `rproxy` acts on the disposition: `passToApp` and `forward` (`internal/rproxy/request.go`) proxy to the app, or the visitor gets `discriminator`'s refusal, or the connection is closed. With no answer in time, `SWRP_DEFAULT_DISPOSITION` decides.
- When the response has ended, `rproxy` writes its log line and `sendReport` posts the report in the background.
- `discriminator`'s `takeReport` and `finishWith` find what was held, go through the `onReport` list (the byte limits), add the request to the client's history and the anomaly counters, and write the second log line.
### Where state lives
All of it is in memory in `discriminator`, each table inside the package that uses it:
- bans: `bans.Ledger`
- clients, their counters and history: `ratelimit.Limiter`
- GeoJS's answers: `lookup.GeoJS`
- fetched lists, DNSBL verdicts, AbuseIPDB scores and the checks spent: `reputation.Lists`, `reputation.DNSBL`, `reputation.AbuseIPDB`
- alert cooldowns, the hour's count and the alerts waiting: `alerts.Queue`
- anomaly counters: `anomaly.Counters`
- routewatch's answers: `routewatch.Client`, never written to disk
- decisions waiting for their report: `awaitingReport` in `discriminator`'s `handler`, never written to disk
- rule files, the Core Rule Set and the contents of the `.mmdb` file: read from disk, held in memory, never written
`internal/state` writes the first six groups to five JSON files in `SWW_STATE_DIR` and reads them back at start. `rproxy` holds nothing between requests.
### Where settings live
`internal/config` reads every setting once, at start, into `config.Config` (for `discriminator`, about 110 fields) or `config.RProxy`. `discriminator` and `rproxy` each hold the whole struct. The other packages are given only the values they need, in a `Params` struct that `internal/sww` or `discriminator` fills in. Setting names also appear as string literals in 16 files outside `config`, in log fields, error messages and metric help text.
### What open work will change
The later comments propose nothing in these areas and say so where they touch them.
- The SQLite state store spec (https://git.eeqj.de/sneak/smallwebwaf/issues/217) replaces `internal/state` and takes the tables out of the packages listed under "Where state lives".
- The settings change through viper (https://git.eeqj.de/sneak/smallwebwaf/pulls/219) changes how every setting is read and renames them all.
- The settings audit (https://git.eeqj.de/sneak/smallwebwaf/issues/202) and the issue that renames the settings after viper lands (https://git.eeqj.de/sneak/smallwebwaf/issues/234) own every setting name.
- The symbol names audits (https://git.eeqj.de/sneak/smallwebwaf/issues/213) own the renaming of existing identifiers. These comments use today's names and rename nothing.
- The issue that gives each daemon's settings one file (https://git.eeqj.de/sneak/smallwebwaf/issues/227) already orders the split of `internal/config` by daemon.
### The other comments
- Construction, start and stop: https://git.eeqj.de/sneak/smallwebwaf/issues/283#issuecomment-173831
- The decision pipeline, the heuristics: https://git.eeqj.de/sneak/smallwebwaf/issues/283#issuecomment-173833
- The HTTP surface: https://git.eeqj.de/sneak/smallwebwaf/issues/283#issuecomment-173836
- What `rproxy` and `discriminator` send each other: https://git.eeqj.de/sneak/smallwebwaf/issues/283#issuecomment-173839
- `internal/config`, and values with a fixed set of choices: https://git.eeqj.de/sneak/smallwebwaf/issues/283#issuecomment-173841
- The smaller packages, `metrics`, the watchers, `coreruleset` and what is repeated: https://git.eeqj.de/sneak/smallwebwaf/issues/283#issuecomment-173847
- Inside `rproxy`: https://git.eeqj.de/sneak/smallwebwaf/issues/283#issuecomment-173849
- Constructors, file layout, the order of the proposals, and what was not checked: https://git.eeqj.de/sneak/smallwebwaf/issues/283#issuecomment-173850
Model: fable-5-2
Summary: the program's parts are built in two packages and reached through public fields, and background work is started and waited for one named line at a time. Six proposals follow, then the question of fx, which would replace most of this code.
The parts are built in two places
What is there:
internal/sww builds five parts: the remote log sender, the alert queue, the rule files, the Core Rule Set and the lookup file.
newHandler and newReputation in internal/discriminator/proxy.go build nine more: the metrics, the limiter, the ledger, the anomaly counters, Lists, DNSBL, AbuseIPDB, the routewatch client and GeoJS.
discriminator.Server then exports ten parts as public fields, so that sww can reach back in. loadStateFiles hands eight of them to state.Load. newServer and RunDiscriminator call server.Metrics.Add… three times. serve starts server.Lists.Run and server.LookupFile.Watch.
Why it matters:
No one place shows what the program is made of, or in what order it is put together.
discriminator.Params takes two URLs (GeoJSURL, AbuseIPDBURL) in place of the parts they are for. A caller cannot hand in a part, and the package's tests read the public fields to see what New built.
The metrics are connected in three files (proxy.go lines 179, 180, 212, 215; sww.go lines 169, 230, 233), in an order that only the comments on metrics.AddRules, AddReputation and AddAbuseIPDB state.
The decision package also carries the code that turns 110 settings into every part's Params.
Proposal: discriminator.Params takes every part already built. internal/sww builds them all in one function, in order. Server exports no parts. A test builds the parts it wants to inspect and keeps them.
Before (internal/discriminator/proxy.go, shortened):
typeParamsstruct{Config*config.ConfigLedger*bans.LedgerLimiter*ratelimit.LimiterGeoJS*lookup.GeoJSLists*reputation.ListsDNSBL*reputation.DNSBLAbuseIPDB*reputation.AbuseIPDBRouteWatch*routewatch.Client// nil while offAnomalies*anomaly.CountersMetrics*metrics.MetricsLookupFile*lookup.File// nil unless SWW_LOOKUP_SOURCE is fileCoreRuleSet*coreruleset.CoreRuleSetRules*rules.FilesAlerts*alerts.Queue// ...}
Catch: one link points backwards. lookup.Params.Answered is handler.addLookup today, which needs the limiter, the ledger and the ban netblock size. With GeoJS built outside, that callback becomes a function discriminator exports, or is set once both exist.
The SQLite state store spec (#217) already goes this way: it adds Params.Store and removes the Server fields that only fed state.Load. This proposal finishes that move for the rest.
Confidence: high. The change is mechanical, and it touches every test helper that builds a server.
Background work is started and waited for by name
What is there:
serve starts six background loops through inBackground, and later waits for six channels, one named line each: <-written, <-watched, <-rulesWatched, <-lookupFileWatched, <-listsFetched, <-alertsSent.
The remote log sender is started apart, by startSending, with a stop function of its own.
rproxy has a third pattern: handler.reports (a sync.WaitGroup) and Server.WaitForReports.
Four parts start a goroutine from inside a request that nothing waits for: lookup/lookup.go:344, reputation/dnsbl.go:166, reputation/abuseipdb.go:157 and routewatch/routewatch.go:142. Each is go x.ask(context.WithoutCancel(ctx), …).
Why it matters:
A seventh loop needs a line to start it, a line to wait for it, and a correct place in both lists.
At an orderly stop, those four goroutines can still be asking when the state is written, and what they learn then is not written. serve's own comment records a second loss: a request cut off at the shutdown timeout can be missing from clients.json.
The pending store spec says nothing uses the store or its tables after Store.Close. A goroutine nobody waits for can.
Proposal: every type that starts work in the background has one method that returns when that work has ended, as rproxy's WaitForReports does. sww starts the loops and waits for them through errgroup (golang.org/x/sync is a decided default and already an indirect dependency), in place of inBackground and the six channels. Acceptable is: an orderly stop loses nothing a part had started.
Before (internal/sww/sww.go):
written:=inBackground(func(){files.Run(writing)})watched:=inBackground(func(){files.Watch(writing)})rulesWatched:=inBackground(func(){ruleFiles.Watch(writing)})// three more, then later:<-written<-watched<-rulesWatched// three more
After:
varbackgrounderrgroup.Groupbackground.Go(func()error{files.Run(writing);returnnil})background.Go(func()error{ruleFiles.Watch(writing);returnnil})// one line per loop, then later:_=background.Wait()
Confidence: high on the finding. Medium on how much is lost in practice: for AbuseIPDB the spent budget is counted before the goroutine starts, so only a late score is lost.
serve and serveRProxy are one sequence written twice
What is there: both serve on a goroutine, wait for the context or a failure, log "stopping", call Shutdown with shutdownTimeout, call Close when that times out, wait for Serve to return, and check for http.ErrServerClosed. Then each does its own last step (files.WriteAll(), server.WaitForReports()) and logs "stopped".
Proposal: one function in internal/sww that takes the *http.Server, the listener and the process logger, and returns an error. Each command calls it, then does its own last step.
Confidence: high.
Start-up errors are labelled away from where they happen
What is there: RunDiscriminator has six blocks of this form, at lines 114, 149, 156, 163, 174 and 181:
server,err:=newServer(cfg,stdout,processLog,now,ruleFiles,alertQueue,crs)iferr!=nil{processLog.Error("cannot use the lookup database","error",err.Error())return1}
Why it matters: the label is the caller's guess at what failed. Any error newServer returns is logged as "cannot use the lookup database", which is correct only while opening the .mmdb file is the one thing in newServer that can fail.
Proposal: RunDiscriminator becomes a function that returns an error, plus a short wrapper that logs it once and returns the status. Each step wraps its own error with what it was doing: fmt.Errorf("read the rule files: %w", err).
Catch: the messages change. Tests and documents that quote them change too.
Confidence: high.
discriminator.NewCoreRuleSet is construction code in the decision package
Where: internal/discriminator/proxy.go:230.
What is there: it takes the whole *config.Config, picks the Core Rule Set's settings out of it, and calls coreruleset.New. It returns nil, nil twice when the Core Rule Set is off, each under //nolint:nilnil. It wraps one error with a setting's name. Its callers are sww and one test.
Proposal: it moves to internal/sww, beside newAlertQueue and newRemoteLogSender, which do the same job for their parts. With every part handed in, discriminator has no use for it.
Confidence: high.
Commands are dispatched by hand
What is there: sww.Main switches on os.Args[1] and prints a usage text kept by hand. HealthCheck rejects arguments by hand.
GO_PACKAGE_DEFAULTS.md names github.com/spf13/cobra as the decided default for CLI commands.
Proposal: three cobra commands in internal/sww. cmd/sww/main.go stays one call.
Catches: a usage error keeps exit status 2. Cobra's completion command and its printing of usage after a failed start are turned off, so stdout and stderr stay as they are.
This is the smallest gain here: three commands and no flags. It is listed because the default is decided, not because the switch is hard to follow.
Confidence: high that it follows the default; low that it changes how easily the code reads.
fx
What the guides say:
CODE_STYLEGUIDE_GO.md asks for go.uber.org/fx in any project with more than two or three modules.
GO_HTTP_SERVER_CONVENTIONS.md lists it as mandatory.
GO_PACKAGE_DEFAULTS.md calls it the standard app skeleton.
This repo connects 17 internal packages by hand. The findings above are the work fx does: build order written out, parts reached through public fields, start and stop kept in step by hand.
No ruling on record applies fx to this repo. The basis here is the guides alone.
Options:
fx with providers in internal/sww: the other packages keep their Params of plain values and import neither fx nor, where they do not today, config. internal/sww holds one small function per part that turns the settings into that part's Params and builds it, as newAlertQueue and newRemoteLogSender already do. fx calls those functions in dependency order. Each background loop is one lifecycle hook, registered where its part is built. This replaces newServer, loadStateFiles, the inBackground list and both serve functions.
fx in every package: the pattern in GO_HTTP_SERVER_CONVENTIONS.md to the letter. Each package's Params embeds fx.In and takes the whole config and the logger, and each New(lc fx.Lifecycle, params Params) registers its own start and stop. A part then no longer states which settings it uses, a test of one part needs fx's test lifecycle and a whole config, and config must first stop importing alerts, anomaly and remotelog.
no fx, one hand-written lifecycle: the six proposals above and nothing more. Fewest dependencies and nothing hidden, and it leaves the repo outside the guides.
Recommendation: fx with providers in internal/sww. It gives the guides' skeleton and keeps each part's Params as the full statement of what that part needs. That is how the five packages with no internal imports are built today.
A sketch of the recommended option. The function names are placeholders; new names are sneak's to approve.
// internal/sww: the discriminator commandapp:=fx.New(fx.Supply(params),fx.Provide(readSettings,newProcessLog,newRemoteLogSender,newAlertQueue,newRuleFiles,newCoreRuleSet,newLookupFile,newGeoJS,newLists,newDNSBL,newAbuseIPDB,newRouteWatch,newLimiter,newLedger,newAnomalyCounters,newMetrics,newServer,),fx.Invoke(listenAndServe),fx.WithLogger(/* fx's own events through slog */),)
Catches, for either fx option:
fx's own event log goes through slog, since stdout carries JSON lines only.
A start-up error still names the setting or the file at fault.
Parts that are off stay nil: the lookup file, the routewatch client, the remote log sender, the Core Rule Set.
RunDiscriminator(ctx, params) int stays as the entry the tests call, using fx's Start and Stop and not its signal handling.
fx stops in the reverse of start order. That is the order serve writes out by hand, so the last state write stays last.
The process logger writes to the remote log sender, and the sender reports its own failures to the process logger. Today the sender is given the logger at Run, not in Params. That stays.
fx resolves by type. func() time.Time and io.Writer need named types of their own, which are new names.
Reversal cost of the recommended option: low. fx is confined to internal/sww, and taking it out means writing the constructor calls back in one function.
Not checked: whether fx's start and stop behave inside testing/synctest, which the tests of internal/sww use.
Model: fable-5-2
## Construction, start and stop
Where: `internal/sww/sww.go`, `internal/sww/rproxy.go`, `internal/discriminator/proxy.go`.
Summary: the program's parts are built in two packages and reached through public fields, and background work is started and waited for one named line at a time. Six proposals follow, then the question of fx, which would replace most of this code.
### The parts are built in two places
- What is there:
- `internal/sww` builds five parts: the remote log sender, the alert queue, the rule files, the Core Rule Set and the lookup file.
- `newHandler` and `newReputation` in `internal/discriminator/proxy.go` build nine more: the metrics, the limiter, the ledger, the anomaly counters, `Lists`, `DNSBL`, `AbuseIPDB`, the routewatch client and `GeoJS`.
- `discriminator.Server` then exports ten parts as public fields, so that `sww` can reach back in. `loadStateFiles` hands eight of them to `state.Load`. `newServer` and `RunDiscriminator` call `server.Metrics.Add…` three times. `serve` starts `server.Lists.Run` and `server.LookupFile.Watch`.
- Why it matters:
- No one place shows what the program is made of, or in what order it is put together.
- `discriminator.Params` takes two URLs (`GeoJSURL`, `AbuseIPDBURL`) in place of the parts they are for. A caller cannot hand in a part, and the package's tests read the public fields to see what `New` built.
- The metrics are connected in three files (`proxy.go` lines 179, 180, 212, 215; `sww.go` lines 169, 230, 233), in an order that only the comments on `metrics.AddRules`, `AddReputation` and `AddAbuseIPDB` state.
- The decision package also carries the code that turns 110 settings into every part's `Params`.
- Proposal: `discriminator.Params` takes every part already built. `internal/sww` builds them all in one function, in order. `Server` exports no parts. A test builds the parts it wants to inspect and keeps them.
Before (`internal/discriminator/proxy.go`, shortened):
```go
type Params struct {
Config *config.Config
GeoJSURL string
AbuseIPDBURL string
LookupFile *lookup.File
CoreRuleSet *coreruleset.CoreRuleSet
Rules *rules.Files
Alerts *alerts.Queue
// ...
}
type Server struct {
*http.Server
Ledger *bans.Ledger
Limiter *ratelimit.Limiter
GeoJS *lookup.GeoJS
// seven more
}
```
After:
```go
type Params struct {
Config *config.Config
Ledger *bans.Ledger
Limiter *ratelimit.Limiter
GeoJS *lookup.GeoJS
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
RouteWatch *routewatch.Client // nil while off
Anomalies *anomaly.Counters
Metrics *metrics.Metrics
LookupFile *lookup.File // nil unless SWW_LOOKUP_SOURCE is file
CoreRuleSet *coreruleset.CoreRuleSet
Rules *rules.Files
Alerts *alerts.Queue
// ...
}
```
- Catch: one link points backwards. `lookup.Params.Answered` is `handler.addLookup` today, which needs the limiter, the ledger and the ban netblock size. With `GeoJS` built outside, that callback becomes a function `discriminator` exports, or is set once both exist.
- The SQLite state store spec (https://git.eeqj.de/sneak/smallwebwaf/issues/217) already goes this way: it adds `Params.Store` and removes the `Server` fields that only fed `state.Load`. This proposal finishes that move for the rest.
- Confidence: high. The change is mechanical, and it touches every test helper that builds a server.
### Background work is started and waited for by name
- What is there:
- `serve` starts six background loops through `inBackground`, and later waits for six channels, one named line each: `<-written`, `<-watched`, `<-rulesWatched`, `<-lookupFileWatched`, `<-listsFetched`, `<-alertsSent`.
- The remote log sender is started apart, by `startSending`, with a stop function of its own.
- `rproxy` has a third pattern: `handler.reports` (a `sync.WaitGroup`) and `Server.WaitForReports`.
- Four parts start a goroutine from inside a request that nothing waits for: `lookup/lookup.go:344`, `reputation/dnsbl.go:166`, `reputation/abuseipdb.go:157` and `routewatch/routewatch.go:142`. Each is `go x.ask(context.WithoutCancel(ctx), …)`.
- Why it matters:
- A seventh loop needs a line to start it, a line to wait for it, and a correct place in both lists.
- At an orderly stop, those four goroutines can still be asking when the state is written, and what they learn then is not written. `serve`'s own comment records a second loss: a request cut off at the shutdown timeout can be missing from `clients.json`.
- The pending store spec says nothing uses the store or its tables after `Store.Close`. A goroutine nobody waits for can.
- Proposal: every type that starts work in the background has one method that returns when that work has ended, as `rproxy`'s `WaitForReports` does. `sww` starts the loops and waits for them through `errgroup` (`golang.org/x/sync` is a decided default and already an indirect dependency), in place of `inBackground` and the six channels. Acceptable is: an orderly stop loses nothing a part had started.
Before (`internal/sww/sww.go`):
```go
written := inBackground(func() { files.Run(writing) })
watched := inBackground(func() { files.Watch(writing) })
rulesWatched := inBackground(func() { ruleFiles.Watch(writing) })
// three more, then later:
<-written
<-watched
<-rulesWatched
// three more
```
After:
```go
var background errgroup.Group
background.Go(func() error { files.Run(writing); return nil })
background.Go(func() error { ruleFiles.Watch(writing); return nil })
// one line per loop, then later:
_ = background.Wait()
```
- Confidence: high on the finding. Medium on how much is lost in practice: for AbuseIPDB the spent budget is counted before the goroutine starts, so only a late score is lost.
### `serve` and `serveRProxy` are one sequence written twice
- What is there: both serve on a goroutine, wait for the context or a failure, log "stopping", call `Shutdown` with `shutdownTimeout`, call `Close` when that times out, wait for `Serve` to return, and check for `http.ErrServerClosed`. Then each does its own last step (`files.WriteAll()`, `server.WaitForReports()`) and logs "stopped".
- Proposal: one function in `internal/sww` that takes the `*http.Server`, the listener and the process logger, and returns an error. Each command calls it, then does its own last step.
- Confidence: high.
### Start-up errors are labelled away from where they happen
- What is there: `RunDiscriminator` has six blocks of this form, at lines 114, 149, 156, 163, 174 and 181:
```go
server, err := newServer(cfg, stdout, processLog, now, ruleFiles, alertQueue, crs)
if err != nil {
processLog.Error("cannot use the lookup database", "error", err.Error())
return 1
}
```
- Why it matters: the label is the caller's guess at what failed. Any error `newServer` returns is logged as "cannot use the lookup database", which is correct only while opening the `.mmdb` file is the one thing in `newServer` that can fail.
- Proposal: `RunDiscriminator` becomes a function that returns an `error`, plus a short wrapper that logs it once and returns the status. Each step wraps its own error with what it was doing: `fmt.Errorf("read the rule files: %w", err)`.
- Catch: the messages change. Tests and documents that quote them change too.
- Confidence: high.
### `discriminator.NewCoreRuleSet` is construction code in the decision package
- Where: `internal/discriminator/proxy.go:230`.
- What is there: it takes the whole `*config.Config`, picks the Core Rule Set's settings out of it, and calls `coreruleset.New`. It returns `nil, nil` twice when the Core Rule Set is off, each under `//nolint:nilnil`. It wraps one error with a setting's name. Its callers are `sww` and one test.
- Proposal: it moves to `internal/sww`, beside `newAlertQueue` and `newRemoteLogSender`, which do the same job for their parts. With every part handed in, `discriminator` has no use for it.
- Confidence: high.
### Commands are dispatched by hand
- What is there: `sww.Main` switches on `os.Args[1]` and prints a usage text kept by hand. `HealthCheck` rejects arguments by hand.
- `GO_PACKAGE_DEFAULTS.md` names `github.com/spf13/cobra` as the decided default for CLI commands.
- Proposal: three `cobra` commands in `internal/sww`. `cmd/sww/main.go` stays one call.
- Catches: a usage error keeps exit status 2. Cobra's completion command and its printing of usage after a failed start are turned off, so stdout and stderr stay as they are.
- This is the smallest gain here: three commands and no flags. It is listed because the default is decided, not because the switch is hard to follow.
- Confidence: high that it follows the default; low that it changes how easily the code reads.
### fx
- What the guides say:
- `CODE_STYLEGUIDE_GO.md` asks for `go.uber.org/fx` in any project with more than two or three modules.
- `GO_HTTP_SERVER_CONVENTIONS.md` lists it as mandatory.
- `GO_PACKAGE_DEFAULTS.md` calls it the standard app skeleton.
- This repo connects 17 internal packages by hand. The findings above are the work fx does: build order written out, parts reached through public fields, start and stop kept in step by hand.
- No ruling on record applies fx to this repo. The basis here is the guides alone.
- Options:
- **fx with providers in `internal/sww`**: the other packages keep their `Params` of plain values and import neither fx nor, where they do not today, `config`. `internal/sww` holds one small function per part that turns the settings into that part's `Params` and builds it, as `newAlertQueue` and `newRemoteLogSender` already do. fx calls those functions in dependency order. Each background loop is one lifecycle hook, registered where its part is built. This replaces `newServer`, `loadStateFiles`, the `inBackground` list and both serve functions.
- **fx in every package**: the pattern in `GO_HTTP_SERVER_CONVENTIONS.md` to the letter. Each package's `Params` embeds `fx.In` and takes the whole config and the logger, and each `New(lc fx.Lifecycle, params Params)` registers its own start and stop. A part then no longer states which settings it uses, a test of one part needs fx's test lifecycle and a whole config, and `config` must first stop importing `alerts`, `anomaly` and `remotelog`.
- **no fx, one hand-written lifecycle**: the six proposals above and nothing more. Fewest dependencies and nothing hidden, and it leaves the repo outside the guides.
- Recommendation: **fx with providers in `internal/sww`**. It gives the guides' skeleton and keeps each part's `Params` as the full statement of what that part needs. That is how the five packages with no internal imports are built today.
A sketch of the recommended option. The function names are placeholders; new names are sneak's to approve.
```go
// internal/sww: the discriminator command
app := fx.New(
fx.Supply(params),
fx.Provide(
readSettings, newProcessLog, newRemoteLogSender, newAlertQueue,
newRuleFiles, newCoreRuleSet, newLookupFile, newGeoJS,
newLists, newDNSBL, newAbuseIPDB, newRouteWatch,
newLimiter, newLedger, newAnomalyCounters, newMetrics,
newServer,
),
fx.Invoke(listenAndServe),
fx.WithLogger(/* fx's own events through slog */),
)
```
- Catches, for either fx option:
- fx's own event log goes through `slog`, since stdout carries JSON lines only.
- A start-up error still names the setting or the file at fault.
- Parts that are off stay nil: the lookup file, the routewatch client, the remote log sender, the Core Rule Set.
- `RunDiscriminator(ctx, params) int` stays as the entry the tests call, using fx's `Start` and `Stop` and not its signal handling.
- fx stops in the reverse of start order. That is the order `serve` writes out by hand, so the last state write stays last.
- The process logger writes to the remote log sender, and the sender reports its own failures to the process logger. Today the sender is given the logger at `Run`, not in `Params`. That stays.
- fx resolves by type. `func() time.Time` and `io.Writer` need named types of their own, which are new names.
- Reversal cost of the recommended option: low. fx is confined to `internal/sww`, and taking it out means writing the constructor calls back in one function.
- Not checked: whether fx's start and stop behave inside `testing/synctest`, which the tests of `internal/sww` use.
Model: fable-5-2
Where: internal/discriminator: heuristics.go, the heuristic files, request.go, discriminator.go, bans.go, biased.go.
Summary: the heuristics are split into files, and not yet into units. A heuristic cannot be built, read or tested without the whole discriminator, and adding one still means changes outside its file. That is the aim of the issue that ordered modular heuristics (#195): one file, one line in one place, no change to the others. Six findings follow, largest first.
What stays as it is: one small interface, the first verdict ends the checks, the name and figures go to the log line, and the tests that drive a whole discriminator.
A heuristic declares nothing it needs
What is there:
All seventeen heuristics are empty structs: type denyNets struct{}.
check(ctx, rq *request, now) is handed the whole request. Through rq.h it reaches the whole handler: 17 collaborators and all of config.Config.
The four lists are built again for every request: beforeLookup(), afterLookup(), afterAnswer(), onReport().
Why it matters:
What a heuristic reads is found only by reading its body. The comment at the top of each file names its settings, and nothing holds that comment to the code.
None can be built with test values and called. Every heuristic's test starts a whole discriminator from a map of settings, sends requests over HTTP and reads the log line.
A heuristic with a new collaborator needs a new field on handler, set in newHandler: a change outside its file.
Proposal: each heuristic is a struct holding its own settings and collaborators. newHandler builds the lists once. check stops using rq.h.
typedenyNetsstruct{netblocks[]netip.Prefix// SWW_DENY_NETS}func(ddenyNets)check(_context.Context,rq*request,_time.Time)*verdict{for_,netblock:=ranged.netblocks{// ...// in newHandler, onceh.beforeLookup=[]heuristic{tagLists{lists:lists},routeWatchTags{routeWatch:routeWatch},denyNets{netblocks:cfg.DenyNets},activeBan{ledger:ledger,observe:cfg.Observe},}
A test then builds denyNets{netblocks: …}, gives it a request with a client address, and checks the verdict. The whole-discriminator tests stay.
The list in newHandler becomes the one place that shows which settings each heuristic uses.
Confidence: high.
Heuristics pass facts to each other through fields nobody lists
What is there, as writer, then reader:
rq.line.Tags: written by tagLists, routeWatchTags, lookUp, crowdSec, dnsbl, abuseIPDB, ruleFiles and coreRuleSet. Read by tagDeny and by limitPercentages in biased.go.
rq.blocklisted, rq.dnsblListed, rq.abuseIPDBHit: written by blocklists, dnsbl, abuseIPDB. Read by limitPercentages.
rq.counted, rq.rateLimitPercent, rq.bytesLimitPercent: written by rateLimits. Read by byteLimits and banForLimit.
rq.attack, rq.blockedByBlockRule, rq.blockedByCoreRuleSet: written by trapPaths, ruleFiles, coreRuleSet. Read by refusalLimit and addToHistory.
rq.line.ASN, rq.line.ASName, rq.line.Country: written by lookUp. Read by countries, limitPercentages, the three ban functions and the alerts. decision.lookupAnswer holds the same three values and is not what they read.
rq.line.WouldAction: written by check. Read by byteLimits and refusalLimit.
rq.line.Offence: written by banForLimit. Read by addToHistory.
Why it matters:
The order of the lists carries meaning that the lists do not show. tagDeny is listed twice because tags are added on both sides of it. rateLimits must come after the three reputation heuristics because it reads their flags.
The JSON log line doubles as working memory. Six of its fields are read back to make decisions, so a change to how a field is logged can change a decision.
A new heuristic that has something to tell a later one needs a new field on request: again a change outside its file.
Proposal:
decision holds what heuristics tell each other, as typed fields of its own: the tags, the lookup answer it already has, what would have been done, the offence. The comment on each field names what sets it.
The log line is filled from decision and never read back.
Confidence: high on the finding. Medium on the size of the change: about 25 reading sites.
A verdict does not say what the visitor gets
What is there:
verdict is {action string; figures map[string]any}. The action is a log word.
check in request.go works out the status afterwards, by a switch over that word: 403 for the two blocked actions, 431 for headers too large, SWW_BAN_RESPONSE for the rest.
refusalLimit repeats part of that switch, over rq.line.WouldAction.
In the afterAnswer and onReport lists the action is never read: refusalLimit and byteLimits return &verdict{figures: …}.
Why it matters: a new heuristic that answers with a status of its own edits the switch in check, outside its file. The status is decided by a string comparison far from the code that knows it.
Proposal: verdict carries the status, set by the heuristic. check then only applies observe mode. refusalLimit takes the status from the verdict that refused the request.
typeverdictstruct{actionstringstatusint// what the visitor is answered with; 0 closes the connectionfiguresmap[string]any}
Confidence: high.
Observe mode is decided in eleven places
What is there:
rq.h.config.Observe is tested at request.go:186, discriminator.go:151, coreruleset.go:85, and eight times in discriminator/bans.go (lines 34, 118, 142, 167, 178, 203, 216, 267).
bans.Ledger has each way of making a ban twice: BanForLimit and WouldBanForLimit, BanForAttack and WouldBanForAttack, BanForCrowdSec and WouldBanForCrowdSec. All six call one private ban(…, keep bool). Check and Find are a fourth pair.
banForLimit, banForAttack and banForCrowdSec in discriminator/bans.go repeat one skeleton: find the netblock; in observe mode stop unless an alert would be sent; fill the notes; in observe mode call the Would… method and alert; otherwise call the other, set BanExpires and alert.
Why it matters: a fourth cause of ban means a new pair of ledger methods and a fourth copy of the skeleton, with two observe branches each. A slip there makes a ban in observe mode.
Proposal:
bans.Ledger gets one method that makes a ban and one that reports what it would make. Both take the cause, the notes and, for CrowdSec, the end and the scenario, in one struct.
discriminator/bans.go gets one function holding the skeleton. It is the one place that tests observe mode for a ban. The three banFor… functions shrink to filling in what differs.
The symbol names audits (#213) propose new names for Check and Find on the same type. The two changes meet there and should be made together.
Confidence: high on the finding. Medium on the exact form: BanForAdmin and WouldBePermanent were not traced through.
The lists are not the whole order
What is there:
heuristics.go says the order is its lists. checkClient adds three things outside them: the SWW_ALLOW_NETS exemption before the first list, lookUp between the two lists, and reading the time again after the lookup.
answerQuestion holds one more step, the 404 for a path under /_sww/.
tagLists and routeWatchTags never give a verdict. They only add tags. lookUp does the same kind of work, and is the only one written into the code and not into a list.
Why it matters: SPEC's "What happens to a request" lists steps 3 to 16 in one sequence. The code has them in four functions, and there are two lists only because the lookup is not an entry.
Proposal: the lookup becomes an entry, as tagLists is, and beforeLookup() and afterLookup() become one list. heuristics.go then reads top to bottom as SPEC's steps do.
Catch: the time is read again after the lookup, so that a request that waited is not counted at the time it arrived. One list keeps that by reading the time for each entry.
Confidence: medium. This is a gain in reading only, and the two lists work.
The steps after the answer are ordered by defer
What is there:
answerQuestion registers five defers. Their reverse order is the order of the work, and two comments explain it ("The last deferred runs first").
An allowed request does runAfterAnswer, finish, holdForReport. A refused one does runAfterAnswer, countAnomalies, addToHistory, finish, holdForReport.
finishWith does the matching steps for a reported request, as plain statements: the onReport list, countAnomalies, addToHistory. It does them on a second request built by hand around a stand-in responseWriter that carries the reported status and bytes.
Why it matters: whether a broken refusal limit is in the client's history depends on this order. A reader gets it only by reversing the defers in their head, and the same steps are coded two ways.
Proposal: one function does the steps in order, as plain statements, and both paths call it. answerQuestion keeps one defer, for the log line and the hold for the report.
Before (discriminator.go):
// The last deferred runs first: runAfterAnswer before addToHistory, so that// a broken refusal limit is in the client's history.deferrq.addToHistory()deferrq.countAnomalies()deferrq.runAfterAnswer(r.Context())
Catch: today the steps also happen if a check panics. With plain statements only the one remaining defer does.
Confidence: medium.
One condition is written twice, and two more live elsewhere
What is there:
refusedWhateverTheAnswer (discriminator.go:150) lists again the conditions of four heuristics: a CrowdSec decision, the header limits, a blocklist with deny, a whole rate limit. lookUp uses it to skip the wait for GeoJS.
The Wait: value in newHandler lists which settings need the lookup's answer.
checkTags and requestCanGetTag in internal/config hold which setting can give a request which tag, and in what order tags are acted on.
Why it matters: a heuristic that changes its condition, needs the lookup, or adds a tag must also change one of these. The comment in heuristics.go on adding a heuristic names none of them.
Proposal, smallest first:
The comment in heuristics.go names the three places.
With heuristics as built values, a heuristic that can refuse without the lookup says so itself, through a second small interface it may implement. refusedWhateverTheAnswer then asks the list. The cost is one more interface.
Confidence: high on the finding. Medium on the second step: it pays only as the number of heuristics grows.
Model: fable-5-2
## The decision pipeline: the heuristics
Where: `internal/discriminator`: `heuristics.go`, the heuristic files, `request.go`, `discriminator.go`, `bans.go`, `biased.go`.
Summary: the heuristics are split into files, and not yet into units. A heuristic cannot be built, read or tested without the whole discriminator, and adding one still means changes outside its file. That is the aim of the issue that ordered modular heuristics (https://git.eeqj.de/sneak/smallwebwaf/issues/195): one file, one line in one place, no change to the others. Six findings follow, largest first.
What stays as it is: one small interface, the first verdict ends the checks, the name and figures go to the log line, and the tests that drive a whole discriminator.
### A heuristic declares nothing it needs
- What is there:
- All seventeen heuristics are empty structs: `type denyNets struct{}`.
- `check(ctx, rq *request, now)` is handed the whole request. Through `rq.h` it reaches the whole `handler`: 17 collaborators and all of `config.Config`.
- The four lists are built again for every request: `beforeLookup()`, `afterLookup()`, `afterAnswer()`, `onReport()`.
- Why it matters:
- What a heuristic reads is found only by reading its body. The comment at the top of each file names its settings, and nothing holds that comment to the code.
- None can be built with test values and called. Every heuristic's test starts a whole discriminator from a map of settings, sends requests over HTTP and reads the log line.
- A heuristic with a new collaborator needs a new field on `handler`, set in `newHandler`: a change outside its file.
- Proposal: each heuristic is a struct holding its own settings and collaborators. `newHandler` builds the lists once. `check` stops using `rq.h`.
Before (`denynets.go`, `heuristics.go`):
```go
type denyNets struct{}
func (denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict {
for _, netblock := range rq.h.config.DenyNets {
// ...
func beforeLookup() []heuristic {
return []heuristic{tagLists{}, routeWatchTags{}, denyNets{}, activeBan{}}
}
```
After:
```go
type denyNets struct {
netblocks []netip.Prefix // SWW_DENY_NETS
}
func (d denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict {
for _, netblock := range d.netblocks {
// ...
// in newHandler, once
h.beforeLookup = []heuristic{
tagLists{lists: lists},
routeWatchTags{routeWatch: routeWatch},
denyNets{netblocks: cfg.DenyNets},
activeBan{ledger: ledger, observe: cfg.Observe},
}
```
- A test then builds `denyNets{netblocks: …}`, gives it a request with a client address, and checks the verdict. The whole-discriminator tests stay.
- The list in `newHandler` becomes the one place that shows which settings each heuristic uses.
- Confidence: high.
### Heuristics pass facts to each other through fields nobody lists
- What is there, as writer, then reader:
- `rq.line.Tags`: written by `tagLists`, `routeWatchTags`, `lookUp`, `crowdSec`, `dnsbl`, `abuseIPDB`, `ruleFiles` and `coreRuleSet`. Read by `tagDeny` and by `limitPercentages` in `biased.go`.
- `rq.blocklisted`, `rq.dnsblListed`, `rq.abuseIPDBHit`: written by `blocklists`, `dnsbl`, `abuseIPDB`. Read by `limitPercentages`.
- `rq.counted`, `rq.rateLimitPercent`, `rq.bytesLimitPercent`: written by `rateLimits`. Read by `byteLimits` and `banForLimit`.
- `rq.attack`, `rq.blockedByBlockRule`, `rq.blockedByCoreRuleSet`: written by `trapPaths`, `ruleFiles`, `coreRuleSet`. Read by `refusalLimit` and `addToHistory`.
- `rq.line.ASN`, `rq.line.ASName`, `rq.line.Country`: written by `lookUp`. Read by `countries`, `limitPercentages`, the three ban functions and the alerts. `decision.lookupAnswer` holds the same three values and is not what they read.
- `rq.line.WouldAction`: written by `check`. Read by `byteLimits` and `refusalLimit`.
- `rq.line.Offence`: written by `banForLimit`. Read by `addToHistory`.
- Why it matters:
- The order of the lists carries meaning that the lists do not show. `tagDeny` is listed twice because tags are added on both sides of it. `rateLimits` must come after the three reputation heuristics because it reads their flags.
- The JSON log line doubles as working memory. Six of its fields are read back to make decisions, so a change to how a field is logged can change a decision.
- A new heuristic that has something to tell a later one needs a new field on `request`: again a change outside its file.
- Proposal:
- `decision` holds what heuristics tell each other, as typed fields of its own: the tags, the lookup answer it already has, what would have been done, the offence. The comment on each field names what sets it.
- The log line is filled from `decision` and never read back.
Before (`tagdeny.go`, `bytelimits.go`):
```go
for _, tag := range rq.line.Tags {
// ...
if !rq.counted || rq.line.WouldAction != "" {
```
After (field names are placeholders):
```go
for _, tag := range rq.tags {
// ...
if !rq.counted || rq.wouldRefuse {
```
- Confidence: high on the finding. Medium on the size of the change: about 25 reading sites.
### A verdict does not say what the visitor gets
- What is there:
- `verdict` is `{action string; figures map[string]any}`. The action is a log word.
- `check` in `request.go` works out the status afterwards, by a `switch` over that word: 403 for the two blocked actions, 431 for headers too large, `SWW_BAN_RESPONSE` for the rest.
- `refusalLimit` repeats part of that `switch`, over `rq.line.WouldAction`.
- In the `afterAnswer` and `onReport` lists the action is never read: `refusalLimit` and `byteLimits` return `&verdict{figures: …}`.
- Why it matters: a new heuristic that answers with a status of its own edits the `switch` in `check`, outside its file. The status is decided by a string comparison far from the code that knows it.
- Proposal: `verdict` carries the status, set by the heuristic. `check` then only applies observe mode. `refusalLimit` takes the status from the verdict that refused the request.
Before (`request.go`):
```go
case action == requestlog.ActionBlockedByBlockRule ||
action == requestlog.ActionBlockedByCoreRuleSet:
return &refusal{status: http.StatusForbidden, action: action}
case action == requestlog.ActionRequestHeaderTooLarge:
return &refusal{status: http.StatusRequestHeaderFieldsTooLarge, action: action}
default:
return rq.banResponse(action)
```
After:
```go
type verdict struct {
action string
status int // what the visitor is answered with; 0 closes the connection
figures map[string]any
}
```
- Confidence: high.
### Observe mode is decided in eleven places
- What is there:
- `rq.h.config.Observe` is tested at `request.go:186`, `discriminator.go:151`, `coreruleset.go:85`, and eight times in `discriminator/bans.go` (lines 34, 118, 142, 167, 178, 203, 216, 267).
- `bans.Ledger` has each way of making a ban twice: `BanForLimit` and `WouldBanForLimit`, `BanForAttack` and `WouldBanForAttack`, `BanForCrowdSec` and `WouldBanForCrowdSec`. All six call one private `ban(…, keep bool)`. `Check` and `Find` are a fourth pair.
- `banForLimit`, `banForAttack` and `banForCrowdSec` in `discriminator/bans.go` repeat one skeleton: find the netblock; in observe mode stop unless an alert would be sent; fill the notes; in observe mode call the `Would…` method and alert; otherwise call the other, set `BanExpires` and alert.
- Why it matters: a fourth cause of ban means a new pair of ledger methods and a fourth copy of the skeleton, with two observe branches each. A slip there makes a ban in observe mode.
- Proposal:
- `bans.Ledger` gets one method that makes a ban and one that reports what it would make. Both take the cause, the notes and, for CrowdSec, the end and the scenario, in one struct.
- `discriminator/bans.go` gets one function holding the skeleton. It is the one place that tests observe mode for a ban. The three `banFor…` functions shrink to filling in what differs.
Before (`internal/bans/bans.go`):
```go
func (l *Ledger) BanForLimit(netblock netip.Prefix, now time.Time, notes Notes) (Ban, bool)
func (l *Ledger) WouldBanForLimit(netblock netip.Prefix, now time.Time, notes Notes) (Ban, bool)
func (l *Ledger) BanForAttack(netblock netip.Prefix, now time.Time, notes Notes) (Ban, bool)
func (l *Ledger) WouldBanForAttack(netblock netip.Prefix, now time.Time, notes Notes) (Ban, bool)
func (l *Ledger) BanForCrowdSec(netblock netip.Prefix, now, expires time.Time, scenario string, notes Notes) (Ban, bool)
func (l *Ledger) WouldBanForCrowdSec(netblock netip.Prefix, now, expires time.Time, scenario string, notes Notes) (Ban, bool)
```
After (the struct's name is a placeholder):
```go
func (l *Ledger) Ban(netblock netip.Prefix, now time.Time, why Cause) (Ban, bool)
func (l *Ledger) WouldBan(netblock netip.Prefix, now time.Time, why Cause) (Ban, bool)
```
- The symbol names audits (https://git.eeqj.de/sneak/smallwebwaf/issues/213) propose new names for `Check` and `Find` on the same type. The two changes meet there and should be made together.
- Confidence: high on the finding. Medium on the exact form: `BanForAdmin` and `WouldBePermanent` were not traced through.
### The lists are not the whole order
- What is there:
- `heuristics.go` says the order is its lists. `checkClient` adds three things outside them: the `SWW_ALLOW_NETS` exemption before the first list, `lookUp` between the two lists, and reading the time again after the lookup.
- `answerQuestion` holds one more step, the 404 for a path under `/_sww/`.
- `tagLists` and `routeWatchTags` never give a verdict. They only add tags. `lookUp` does the same kind of work, and is the only one written into the code and not into a list.
- Why it matters: SPEC's "What happens to a request" lists steps 3 to 16 in one sequence. The code has them in four functions, and there are two lists only because the lookup is not an entry.
- Proposal: the lookup becomes an entry, as `tagLists` is, and `beforeLookup()` and `afterLookup()` become one list. `heuristics.go` then reads top to bottom as SPEC's steps do.
- Catch: the time is read again after the lookup, so that a request that waited is not counted at the time it arrived. One list keeps that by reading the time for each entry.
- Confidence: medium. This is a gain in reading only, and the two lists work.
### The steps after the answer are ordered by `defer`
- What is there:
- `answerQuestion` registers five `defer`s. Their reverse order is the order of the work, and two comments explain it ("The last deferred runs first").
- An allowed request does `runAfterAnswer`, `finish`, `holdForReport`. A refused one does `runAfterAnswer`, `countAnomalies`, `addToHistory`, `finish`, `holdForReport`.
- `finishWith` does the matching steps for a reported request, as plain statements: the `onReport` list, `countAnomalies`, `addToHistory`. It does them on a second `request` built by hand around a stand-in `responseWriter` that carries the reported status and bytes.
- Why it matters: whether a broken refusal limit is in the client's history depends on this order. A reader gets it only by reversing the `defer`s in their head, and the same steps are coded two ways.
- Proposal: one function does the steps in order, as plain statements, and both paths call it. `answerQuestion` keeps one `defer`, for the log line and the hold for the report.
Before (`discriminator.go`):
```go
// The last deferred runs first: runAfterAnswer before addToHistory, so that
// a broken refusal limit is in the client's history.
defer rq.addToHistory()
defer rq.countAnomalies()
defer rq.runAfterAnswer(r.Context())
```
After:
```go
rq.runAfterAnswer(ctx)
rq.countAnomalies()
rq.addToHistory()
```
- Catch: today the steps also happen if a check panics. With plain statements only the one remaining `defer` does.
- Confidence: medium.
### One condition is written twice, and two more live elsewhere
- What is there:
- `refusedWhateverTheAnswer` (`discriminator.go:150`) lists again the conditions of four heuristics: a CrowdSec decision, the header limits, a blocklist with `deny`, a whole rate limit. `lookUp` uses it to skip the wait for GeoJS.
- The `Wait:` value in `newHandler` lists which settings need the lookup's answer.
- `checkTags` and `requestCanGetTag` in `internal/config` hold which setting can give a request which tag, and in what order tags are acted on.
- Why it matters: a heuristic that changes its condition, needs the lookup, or adds a tag must also change one of these. The comment in `heuristics.go` on adding a heuristic names none of them.
- Proposal, smallest first:
- The comment in `heuristics.go` names the three places.
- With heuristics as built values, a heuristic that can refuse without the lookup says so itself, through a second small interface it may implement. `refusedWhateverTheAnswer` then asks the list. The cost is one more interface.
- Confidence: high on the finding. Medium on the second step: it pays only as the number of heuristics grows.
Model: fable-5-2
Summary: discriminator's own endpoints are matched by hand in three functions, with the token check written into one of them. Two proposals follow, then what is left alone and why.
discriminator's own endpoints are routed by hand
What is there:
ServeHTTP tells a question from anything else by the presence of X-Sww-Client. Anything else goes to answerOwnEndpoint.
answerOwnEndpoint matches POST /_sww/report and GET /_sww/healthz with two ifs, and hands the rest to answerAdmin.
answerAdmin calls endpoint(): a switch on method and path that returns two things, the token the endpoint needs and a closure that answers it.
Path parameters are cut out by hand. Two cases match with strings.HasPrefix, and pathAddress removes the prefix with strings.TrimPrefix.
Every endpoint is a method on request, the same type that carries a question's decision. startRequest takes an isQuestion bool to tell the two uses apart.
Why it matters:
The list of endpoints is spread over three functions. No one place shows what discriminator serves.
The token check lives inside answerAdmin. An endpoint added one function earlier, beside the health endpoint, gets no check and nothing says so.
CODE_STYLEGUIDE_GO.md asks for handlers returned as http.HandlerFunc from a function, not methods used as handlers. GO_HTTP_SERVER_CONVENTIONS.md asks for the routes in routes.go on a chi router, and for middleware of the form func(http.Handler) http.Handler.
The list will grow. The pending SQLite state store spec (#217) keeps a request log for a web analytics interface, and the live request viewer (#174) is held for later.
Proposal:
internal/discriminator/routes.go holds one function that lists every own endpoint on a chi router. go-chi/chi/v5 is the decided default in GO_PACKAGE_DEFAULTS.md.
One middleware checks a token. The metrics endpoint and the admin endpoints each sit in a group that uses it with their own token.
The log line for an own endpoint (startRequest, defer rq.finish()) becomes a second middleware around the router.
ServeHTTP keeps its first test. A question never reaches the router, because it carries the visitor's own method and path.
Before (admin.go):
func(rq*request)endpoint()(string,func()){cfg:=rq.h.configmethod,path:=rq.in.Method,rq.in.URL.Pathswitch{casemethod==http.MethodGet&&path==MetricsPath:returncfg.MetricsToken,func(){rq.h.metrics.ServeHTTP(rq.out,rq.in)}casemethod==http.MethodGet&&path==BansPath:returncfg.AdminToken,rq.listBanscasemethod==http.MethodDelete&&strings.HasPrefix(path,BansPath+"/"):returncfg.AdminToken,rq.liftBans// two moredefault:return"",nil}}
After (routes.go; the function names are placeholders, and new names are sneak's to approve):
A wrong or missing token answers 401 with WWW-Authenticate: Bearer, compared in constant time.
An unknown path answers 404. So does a known path with another method: chi answers 405 unless its handler for that case is set to answer 404.
Each request to an own endpoint writes one log line. The report keeps its own second line.
An IPv6 address in the path is read whole.
Confidence: high that this follows the guides and puts the endpoints in one place. Medium on the middleware for the log line: the report endpoint logs differently from the others and may stay outside it.
Both Server types embed *http.Server
What is there: discriminator.Server and rproxy.Server embed *http.Server. Every field and method of Go's server is then part of their own API: Handler, Addr, ListenAndServe, RegisterOnShutdown and the rest. internal/sww uses three: Serve, Shutdown, Close. The tests use a fourth: server.Handler.ServeHTTP.
GO_HTTP_SERVER_CONVENTIONS.md holds Go's server as a private field, httpServer *http.Server, and makes Server the handler.
Proposal: a private field. Server gets ServeHTTP and what internal/sww needs to start and stop it. With the one serve function proposed in the comment on construction, start and stop, that is one method per Server.
Confidence: high that it follows the conventions. The gain in reading is small: nothing misuses the embedded server today.
Left alone
rproxy's ServeHTTP is one linear sequence with nothing to route: make the request, answer the health path, refuse a method, read the body, ask, act. It reads well as it is. No router is proposed for it.
These differ from the two guides and are the owner's design as built. They are listed as context, and none is proposed for change:
Health is at /_sww/healthz and answers ok in plain text. The guides put it at /.well-known/healthcheck with a JSON body.
Metrics and admin take a bearer token. The conventions put metrics behind basic auth.
There is no globals package, no Sentry and no CORS. The log level is SWW_LOG_LEVEL, where the guides use a DEBUG variable.
Model: fable-5-2
## The HTTP surface
Where: `internal/discriminator/discriminator.go`, `admin.go`, `proxy.go`; `internal/rproxy/rproxy.go`.
Summary: `discriminator`'s own endpoints are matched by hand in three functions, with the token check written into one of them. Two proposals follow, then what is left alone and why.
### `discriminator`'s own endpoints are routed by hand
- What is there:
- `ServeHTTP` tells a question from anything else by the presence of `X-Sww-Client`. Anything else goes to `answerOwnEndpoint`.
- `answerOwnEndpoint` matches `POST /_sww/report` and `GET /_sww/healthz` with two `if`s, and hands the rest to `answerAdmin`.
- `answerAdmin` calls `endpoint()`: a `switch` on method and path that returns two things, the token the endpoint needs and a closure that answers it.
- Path parameters are cut out by hand. Two cases match with `strings.HasPrefix`, and `pathAddress` removes the prefix with `strings.TrimPrefix`.
- Every endpoint is a method on `request`, the same type that carries a question's decision. `startRequest` takes an `isQuestion bool` to tell the two uses apart.
- Why it matters:
- The list of endpoints is spread over three functions. No one place shows what `discriminator` serves.
- The token check lives inside `answerAdmin`. An endpoint added one function earlier, beside the health endpoint, gets no check and nothing says so.
- `CODE_STYLEGUIDE_GO.md` asks for handlers returned as `http.HandlerFunc` from a function, not methods used as handlers. `GO_HTTP_SERVER_CONVENTIONS.md` asks for the routes in `routes.go` on a `chi` router, and for middleware of the form `func(http.Handler) http.Handler`.
- The list will grow. The pending SQLite state store spec (https://git.eeqj.de/sneak/smallwebwaf/issues/217) keeps a request log for a web analytics interface, and the live request viewer (https://git.eeqj.de/sneak/smallwebwaf/issues/174) is held for later.
- Proposal:
- `internal/discriminator/routes.go` holds one function that lists every own endpoint on a `chi` router. `go-chi/chi/v5` is the decided default in `GO_PACKAGE_DEFAULTS.md`.
- One middleware checks a token. The metrics endpoint and the admin endpoints each sit in a group that uses it with their own token.
- The log line for an own endpoint (`startRequest`, `defer rq.finish()`) becomes a second middleware around the router.
- `ServeHTTP` keeps its first test. A question never reaches the router, because it carries the visitor's own method and path.
Before (`admin.go`):
```go
func (rq *request) endpoint() (string, func()) {
cfg := rq.h.config
method, path := rq.in.Method, rq.in.URL.Path
switch {
case method == http.MethodGet && path == MetricsPath:
return cfg.MetricsToken, func() { rq.h.metrics.ServeHTTP(rq.out, rq.in) }
case method == http.MethodGet && path == BansPath:
return cfg.AdminToken, rq.listBans
case method == http.MethodDelete && strings.HasPrefix(path, BansPath+"/"):
return cfg.AdminToken, rq.liftBans
// two more
default:
return "", nil
}
}
```
After (`routes.go`; the function names are placeholders, and new names are sneak's to approve):
```go
func (h *handler) routes() http.Handler {
r := chi.NewRouter()
r.Use(h.logOwnEndpoint)
r.Post(ReportPath, h.handleReport())
r.Get(HealthPath, h.handleHealth())
r.Group(func(r chi.Router) {
r.Use(requireToken(h.config.MetricsToken))
r.Get(MetricsPath, h.metrics.ServeHTTP)
})
r.Group(func(r chi.Router) {
r.Use(requireToken(h.config.AdminToken))
r.Get(BansPath, h.handleListBans())
r.Post(BansPath, h.handleAddBan())
r.Delete(BansPath+"/{address}", h.handleLiftBans())
r.Get(ClientsPath+"{address}", h.handleShowClient())
})
return r
}
```
- What must stay as it is today:
- An endpoint whose token is unset answers 404.
- A wrong or missing token answers 401 with `WWW-Authenticate: Bearer`, compared in constant time.
- An unknown path answers 404. So does a known path with another method: `chi` answers 405 unless its handler for that case is set to answer 404.
- Each request to an own endpoint writes one log line. The report keeps its own second line.
- An IPv6 address in the path is read whole.
- Confidence: high that this follows the guides and puts the endpoints in one place. Medium on the middleware for the log line: the report endpoint logs differently from the others and may stay outside it.
### Both `Server` types embed `*http.Server`
- What is there: `discriminator.Server` and `rproxy.Server` embed `*http.Server`. Every field and method of Go's server is then part of their own API: `Handler`, `Addr`, `ListenAndServe`, `RegisterOnShutdown` and the rest. `internal/sww` uses three: `Serve`, `Shutdown`, `Close`. The tests use a fourth: `server.Handler.ServeHTTP`.
- `GO_HTTP_SERVER_CONVENTIONS.md` holds Go's server as a private field, `httpServer *http.Server`, and makes `Server` the handler.
- Proposal: a private field. `Server` gets `ServeHTTP` and what `internal/sww` needs to start and stop it. With the one serve function proposed in the comment on construction, start and stop, that is one method per `Server`.
- Confidence: high that it follows the conventions. The gain in reading is small: nothing misuses the embedded server today.
### Left alone
- `rproxy`'s `ServeHTTP` is one linear sequence with nothing to route: make the request, answer the health path, refuse a method, read the body, ask, act. It reads well as it is. No router is proposed for it.
- These differ from the two guides and are the owner's design as built. They are listed as context, and none is proposed for change:
- Health is at `/_sww/healthz` and answers `ok` in plain text. The guides put it at `/.well-known/healthcheck` with a JSON body.
- Metrics and admin take a bearer token. The conventions put metrics behind basic auth.
- There is no `globals` package, no Sentry and no CORS. The log level is `SWW_LOG_LEVEL`, where the guides use a `DEBUG` variable.
Model: fable-5-2
Summary: SPEC defines three messages between the two programs: the question, the answer and the report. No package owns them. Each message is written in one package and read in the other, and rproxy imports the decision package to get their names. One proposal.
The messages have no package of their own
What is there:
The names live in internal/discriminator. rproxy imports it for eleven of them: ClientHeader, PeerHeader, ContentLengthHeader, DispositionHeader, DispositionAllow, DispositionDeny, DispositionClose, QuestionBodyMaxBytes, ReportPath, HealthPath and the type Report. internal/sww/healthcheck.go imports it for HealthPath alone.
Each message has its writer and its reader in different packages:
The question: newQuestion in rproxy writes it. readQuestion and asTheVisitorSentIt in discriminator read it.
The answer: allow, deny and setLookupHeaders in discriminator write it. noteAnswer in rproxy reads it.
The report: finish and postReport in rproxy write it. readReport in discriminator reads it.
Four names that both sides must agree on are declared twice, once in each package: requestIDHeader, asnHeader, countryHeader and ownEndpointPrefix.
Why it matters:
Importing discriminator makes rproxy depend, through it, on every package the decisions use: the Core Rule Set, the ban ledger, the lookups, the state files. rproxy needs only the message format, and its imports do not show that.
A change to a message is made in two packages, and only the tests that start both programs show whether the two halves still agree. Nothing pairs newQuestion with asTheVisitorSentIt directly, and those two must mirror each other header for header.
A duplicated header name that drifts compiles, and fails only when the program is up.
The format is still growing. The pending SQLite state store spec (#217) adds one more header that both sides must know.
Proposal: one small package under internal/ owns the three messages. It holds:
the header names, the three dispositions, QuestionBodyMaxBytes, ReportPath, HealthPath and the /_sww/ prefix;
Report;
for the question and for the answer, one function that writes it and one that reads it, moved from the two packages;
a test that passes each message through its writer and then its reader.
rproxy, discriminator and sww import it. rproxy stops importing discriminator.
MetricsPath, BansPath and ClientsPath stay in discriminator: rproxy never uses them.
Before (internal/rproxy/discriminator.go, then internal/discriminator/discriminator.go):
After, both in the new package (names are placeholders; new names are sneak's to approve):
// NewQuestion returns the question about the visitor's request in.funcNewQuestion(ctxcontext.Context,backend*url.URL,in*http.Request,qQuestion)(*http.Request,error)// ReadQuestion returns what a question says, and the visitor's request as// the visitor sent it.funcReadQuestion(r*http.Request)(Question,*http.Request,error)
This is not a package of type definitions, which CODE_STYLEGUIDE_GO.md forbids. It owns behaviour: how each message is put on the wire and taken off it.
Its name has to say which two programs it sits between, and is sneak's to choose.
Left as they are: peerAddress, isInside, scheme and the two responseWriter types also exist once in each package. Each is a few lines, the two scheme functions differ, and a shared package for them would be the util package the guide bans.
Confidence: high. The move changes no behaviour and no message.
Model: fable-5-2
## What `rproxy` and `discriminator` send each other
Where: `internal/rproxy/discriminator.go`, `internal/rproxy/client.go`, `internal/discriminator/discriminator.go`, `internal/discriminator/client.go`, `internal/discriminator/lookup.go`, `internal/sww/healthcheck.go`.
Summary: SPEC defines three messages between the two programs: the question, the answer and the report. No package owns them. Each message is written in one package and read in the other, and `rproxy` imports the decision package to get their names. One proposal.
### The messages have no package of their own
- What is there:
- The names live in `internal/discriminator`. `rproxy` imports it for eleven of them: `ClientHeader`, `PeerHeader`, `ContentLengthHeader`, `DispositionHeader`, `DispositionAllow`, `DispositionDeny`, `DispositionClose`, `QuestionBodyMaxBytes`, `ReportPath`, `HealthPath` and the type `Report`. `internal/sww/healthcheck.go` imports it for `HealthPath` alone.
- Each message has its writer and its reader in different packages:
- The question: `newQuestion` in `rproxy` writes it. `readQuestion` and `asTheVisitorSentIt` in `discriminator` read it.
- The answer: `allow`, `deny` and `setLookupHeaders` in `discriminator` write it. `noteAnswer` in `rproxy` reads it.
- The report: `finish` and `postReport` in `rproxy` write it. `readReport` in `discriminator` reads it.
- Four names that both sides must agree on are declared twice, once in each package: `requestIDHeader`, `asnHeader`, `countryHeader` and `ownEndpointPrefix`.
- Why it matters:
- Importing `discriminator` makes `rproxy` depend, through it, on every package the decisions use: the Core Rule Set, the ban ledger, the lookups, the state files. `rproxy` needs only the message format, and its imports do not show that.
- A change to a message is made in two packages, and only the tests that start both programs show whether the two halves still agree. Nothing pairs `newQuestion` with `asTheVisitorSentIt` directly, and those two must mirror each other header for header.
- A duplicated header name that drifts compiles, and fails only when the program is up.
- The format is still growing. The pending SQLite state store spec (https://git.eeqj.de/sneak/smallwebwaf/issues/217) adds one more header that both sides must know.
- Proposal: one small package under `internal/` owns the three messages. It holds:
- the header names, the three dispositions, `QuestionBodyMaxBytes`, `ReportPath`, `HealthPath` and the `/_sww/` prefix;
- `Report`;
- for the question and for the answer, one function that writes it and one that reads it, moved from the two packages;
- a test that passes each message through its writer and then its reader.
- `rproxy`, `discriminator` and `sww` import it. `rproxy` stops importing `discriminator`.
- `MetricsPath`, `BansPath` and `ClientsPath` stay in `discriminator`: `rproxy` never uses them.
Before (`internal/rproxy/discriminator.go`, then `internal/discriminator/discriminator.go`):
```go
question.Header = in.Header.Clone()
question.Header.Set(discriminator.ClientHeader, rq.client.String())
question.Header.Set(discriminator.PeerHeader, rq.peer.String())
question.Header.Del(discriminator.ContentLengthHeader)
// ...
```
```go
sent.Header.Del(ClientHeader)
sent.Header.Del(PeerHeader)
sent.Header.Del(ContentLengthHeader)
sent.Header.Del("Content-Length")
// ...
```
After, both in the new package (names are placeholders; new names are sneak's to approve):
```go
// NewQuestion returns the question about the visitor's request in.
func NewQuestion(ctx context.Context, backend *url.URL, in *http.Request, q Question) (*http.Request, error)
// ReadQuestion returns what a question says, and the visitor's request as
// the visitor sent it.
func ReadQuestion(r *http.Request) (Question, *http.Request, error)
```
- This is not a package of type definitions, which `CODE_STYLEGUIDE_GO.md` forbids. It owns behaviour: how each message is put on the wire and taken off it.
- Its name has to say which two programs it sits between, and is sneak's to choose.
- Left as they are: `peerAddress`, `isInside`, `scheme` and the two `responseWriter` types also exist once in each package. Each is a few lines, the two `scheme` functions differ, and a shared package for them would be the `util` package the guide bans.
- Confidence: high. The move changes no behaviour and no message.
Model: fable-5-2
internal/config, and values with a fixed set of choices
Where: internal/config/config.go, and the packages that use it.
Already ordered, and not touched here:
how settings are read: the settings change through viper (#219);
the split by daemon: the issue that gives each daemon's settings one file (#227);
setting names: the settings audit (#202) and the issue that renames the settings after viper lands (#234);
names of existing identifiers: the symbol names audits (#213).
Everything below applies after that pull request lands. Line numbers and setting names are next's today. That pull request's diff was not read, so sizes and line numbers will differ.
config sits on both sides of the parts
What is there:
config imports three of the parts for their types: alerts.Events, anomaly.Thresholds, anomaly.NamedNetblock, and the three remotelog.Scheme… constants.
Three parts import config for helpers: reputation (TagList, ParseNetblock, ParseASN, ParsePercent, MaskZoneKey), rules (IsHeaderName) and metrics (MaskZoneKey).
The other parts take plain values in a Params struct and do not import config.
Why it matters: a reader cannot state which way the dependency goes. A part that config imports can never use a config helper, and a part that imports config can never give config a type. The next helper or type lands wherever it compiles.
Options:
config above the parts: no part imports config. Only sww, discriminator and rproxy read it. TagList and MaskZoneKey move to reputation, which owns tag lists and DNSBL zones. IsHeaderName moves to rules, or both users call httpguts.ValidHeaderFieldName from golang.org/x/net, already an indirect dependency. ParseNetblock, ParseASN and ParsePercent move to reputation, the lower of their two users, and config imports them from there.
config imports nothing internal: Events, Thresholds, NamedNetblock and the scheme constants move into config, and every part that uses them imports config. Every part then depends on the settings package, and a part's types sit away from its behaviour, which CODE_STYLEGUIDE_GO.md advises against.
Recommendation: config above the parts. It is the pattern most parts follow already, and it keeps each part testable with plain values.
Confidence: high on the direction. Medium on the home of the three parsers: they read one value from text, for a setting or for a line of a fetched list, and belong to neither user more than the other. Medium that httpguts.ValidHeaderFieldName accepts exactly what IsHeaderName does: both read as the RFC 9110 token, and this was compared by reading only.
One setting's code is in six places
What is there: config.go is 2,577 lines, ordered by kind of code: the struct, the constants, one block of errors, FromEnvironment, about 60 reader methods, the checks across settings, about 60 parsers.
Example, SWW_DNSBL_ZONES:
the field DNSBLZones at line 139;
its errors errNotZone and errZoneTooLong at 455 and 458;
the line that reads it at 544;
the reader method zones at 986;
the check against the tags at 1214;
the parser parseZones at 2410.
Why it matters: following or changing one setting means six jumps in one file. This is the file a newcomer opens first to learn what a setting does.
Proposal: once each daemon has its file, order the code under the struct by setting, in the order of SPEC's settings sections. A setting's parser, its errors and its check then sit together. The struct and the function that reads every setting stay whole, so the list of settings stays in one place. Moves only.
Confidence: high on the finding. Medium on the proposal's size after viper, which replaces the reader methods.
Values with a fixed set of choices are bare strings
What is there:
Four settings with a fixed set of values are string fields: LookupSource, BytesCount, BlocklistAction, ReputationAction.
Their users compare with literals: cfg.LookupSource == "file" at sww/sww.go:204, discriminator/request.go:369 and discriminator/lookup.go:40; == "off" at discriminator/lookup.go:33; == "limit" at discriminator/biased.go:60 and :65.
The same values exist as private constants twice more: fileSource and off in config, deny in discriminator.
More than thirty exported constants in six files are untyped strings: ban causes (bans.CauseLimit), limit kinds (ratelimit.KindBytes), alert events (alerts.EventBan), rule actions (rules.ActionBlock), the log line's actions and offences (requestlog.Action…), the dispositions. The fields and parameters that carry them are string: ban(…, cause, reason string, …), verdict.action string.
Two places already do it the other way: routewatch.Missing is a named type, and config exports WAFModeOff, WAFModeDetect and WAFModeBlock.
Why it matters: a mistyped literal compiles and is false when the program is up. Any string is accepted where a cause is expected, and two neighbouring string parameters can be swapped unnoticed. A reader cannot find the allowed values from the type.
Proposal: each set becomes a named string type with its constants, in the package that owns it. Fields, parameters and comparisons use the type. The settings' types live in config.
The JSON in logs, state and alerts does not change: a named string type is written as its string.
exhaustive is already on in .golangci.yml. It then checks every switch over these types for a missing case.
Setting names are written out again where they are logged
What is there: 31 string literals in 16 files outside config hold a setting's name. They fall in three groups:
Data that leaves the program. discriminator/biased.go has seven and countries.go:32 one: the setting figure of the log line and the LimitPercentSetting of a ban's notes.
Error and log messages: rules/rules.go:274, state/state.go:188, lookup/file.go:234 and :239, sww/sww.go:181, sww/rproxy.go:41, remotelog/remotelog.go:246, discriminator/proxy.go:241, and five in alerts/alerts.go.
Metric help text in internal/metrics.
Why it matters: every setting is renamed by the viper change, and may be again by the issue that follows it. Each of these is found by text search only. One missed in the first group leaves log lines and ban notes that name a setting which does not exist.
Proposal, for the first group only: config exports a constant for the name of each setting that is written into a log line or a ban's notes. config's own reader and discriminator both use it.
The second group shrinks under the proposal on start-up errors in the comment on construction, start and stop: a part reports what is wrong with a path, and the caller that knows the setting adds its name.
The third group is prose and can stay.
Confidence: medium. The viper change may already keep the names in one place.
Model: fable-5-2
## `internal/config`, and values with a fixed set of choices
Where: `internal/config/config.go`, and the packages that use it.
Already ordered, and not touched here:
- how settings are read: the settings change through viper (https://git.eeqj.de/sneak/smallwebwaf/pulls/219);
- the split by daemon: the issue that gives each daemon's settings one file (https://git.eeqj.de/sneak/smallwebwaf/issues/227);
- setting names: the settings audit (https://git.eeqj.de/sneak/smallwebwaf/issues/202) and the issue that renames the settings after viper lands (https://git.eeqj.de/sneak/smallwebwaf/issues/234);
- names of existing identifiers: the symbol names audits (https://git.eeqj.de/sneak/smallwebwaf/issues/213).
Everything below applies after that pull request lands. Line numbers and setting names are `next`'s today. That pull request's diff was not read, so sizes and line numbers will differ.
### `config` sits on both sides of the parts
- What is there:
- `config` imports three of the parts for their types: `alerts.Events`, `anomaly.Thresholds`, `anomaly.NamedNetblock`, and the three `remotelog.Scheme…` constants.
- Three parts import `config` for helpers: `reputation` (`TagList`, `ParseNetblock`, `ParseASN`, `ParsePercent`, `MaskZoneKey`), `rules` (`IsHeaderName`) and `metrics` (`MaskZoneKey`).
- The other parts take plain values in a `Params` struct and do not import `config`.
- Why it matters: a reader cannot state which way the dependency goes. A part that `config` imports can never use a `config` helper, and a part that imports `config` can never give `config` a type. The next helper or type lands wherever it compiles.
- Options:
- **`config` above the parts**: no part imports `config`. Only `sww`, `discriminator` and `rproxy` read it. `TagList` and `MaskZoneKey` move to `reputation`, which owns tag lists and DNSBL zones. `IsHeaderName` moves to `rules`, or both users call `httpguts.ValidHeaderFieldName` from `golang.org/x/net`, already an indirect dependency. `ParseNetblock`, `ParseASN` and `ParsePercent` move to `reputation`, the lower of their two users, and `config` imports them from there.
- **`config` imports nothing internal**: `Events`, `Thresholds`, `NamedNetblock` and the scheme constants move into `config`, and every part that uses them imports `config`. Every part then depends on the settings package, and a part's types sit away from its behaviour, which `CODE_STYLEGUIDE_GO.md` advises against.
- Recommendation: **`config` above the parts**. It is the pattern most parts follow already, and it keeps each part testable with plain values.
- Confidence: high on the direction. Medium on the home of the three parsers: they read one value from text, for a setting or for a line of a fetched list, and belong to neither user more than the other. Medium that `httpguts.ValidHeaderFieldName` accepts exactly what `IsHeaderName` does: both read as the RFC 9110 token, and this was compared by reading only.
### One setting's code is in six places
- What is there: `config.go` is 2,577 lines, ordered by kind of code: the struct, the constants, one block of errors, `FromEnvironment`, about 60 reader methods, the checks across settings, about 60 parsers.
- Example, `SWW_DNSBL_ZONES`:
- the field `DNSBLZones` at line 139;
- its errors `errNotZone` and `errZoneTooLong` at 455 and 458;
- the line that reads it at 544;
- the reader method `zones` at 986;
- the check against the tags at 1214;
- the parser `parseZones` at 2410.
- Why it matters: following or changing one setting means six jumps in one file. This is the file a newcomer opens first to learn what a setting does.
- Proposal: once each daemon has its file, order the code under the struct by setting, in the order of SPEC's settings sections. A setting's parser, its errors and its check then sit together. The struct and the function that reads every setting stay whole, so the list of settings stays in one place. Moves only.
- Confidence: high on the finding. Medium on the proposal's size after viper, which replaces the reader methods.
### Values with a fixed set of choices are bare strings
- What is there:
- Four settings with a fixed set of values are `string` fields: `LookupSource`, `BytesCount`, `BlocklistAction`, `ReputationAction`.
- Their users compare with literals: `cfg.LookupSource == "file"` at `sww/sww.go:204`, `discriminator/request.go:369` and `discriminator/lookup.go:40`; `== "off"` at `discriminator/lookup.go:33`; `== "limit"` at `discriminator/biased.go:60` and `:65`.
- The same values exist as private constants twice more: `fileSource` and `off` in `config`, `deny` in `discriminator`.
- More than thirty exported constants in six files are untyped strings: ban causes (`bans.CauseLimit`), limit kinds (`ratelimit.KindBytes`), alert events (`alerts.EventBan`), rule actions (`rules.ActionBlock`), the log line's actions and offences (`requestlog.Action…`), the dispositions. The fields and parameters that carry them are `string`: `ban(…, cause, reason string, …)`, `verdict.action string`.
- Two places already do it the other way: `routewatch.Missing` is a named type, and `config` exports `WAFModeOff`, `WAFModeDetect` and `WAFModeBlock`.
- Why it matters: a mistyped literal compiles and is false when the program is up. Any string is accepted where a cause is expected, and two neighbouring `string` parameters can be swapped unnoticed. A reader cannot find the allowed values from the type.
- Proposal: each set becomes a named string type with its constants, in the package that owns it. Fields, parameters and comparisons use the type. The settings' types live in `config`.
- The JSON in logs, state and alerts does not change: a named string type is written as its string.
- `exhaustive` is already on in `.golangci.yml`. It then checks every `switch` over these types for a missing case.
Before (`discriminator/lookup.go`):
```go
if rq.h.config.LookupSource == "off" || !canBePlaced(rq.client) {
return
}
// ...
case rq.h.config.LookupSource == "file":
```
After (names are placeholders; new names are sneak's to approve):
```go
if rq.h.config.LookupSource == config.LookupSourceOff || !canBePlaced(rq.client) {
return
}
// ...
case rq.h.config.LookupSource == config.LookupSourceFile:
```
- Confidence: high.
### Setting names are written out again where they are logged
- What is there: 31 string literals in 16 files outside `config` hold a setting's name. They fall in three groups:
- Data that leaves the program. `discriminator/biased.go` has seven and `countries.go:32` one: the `setting` figure of the log line and the `LimitPercentSetting` of a ban's notes.
- Error and log messages: `rules/rules.go:274`, `state/state.go:188`, `lookup/file.go:234` and `:239`, `sww/sww.go:181`, `sww/rproxy.go:41`, `remotelog/remotelog.go:246`, `discriminator/proxy.go:241`, and five in `alerts/alerts.go`.
- Metric help text in `internal/metrics`.
- Why it matters: every setting is renamed by the viper change, and may be again by the issue that follows it. Each of these is found by text search only. One missed in the first group leaves log lines and ban notes that name a setting which does not exist.
- Proposal, for the first group only: `config` exports a constant for the name of each setting that is written into a log line or a ban's notes. `config`'s own reader and `discriminator` both use it.
- The second group shrinks under the proposal on start-up errors in the comment on construction, start and stop: a part reports what is wrong with a path, and the caller that knows the setting adds its name.
- The third group is prose and can stay.
- Confidence: medium. The viper change may already keep the names in one place.
Model: fable-5-2
Summary: metrics depends on everything it measures, and is connected by seven calls in a fixed order. Two packages hold the same loop that watches a directory. Adding an app to the Core Rule Set exclusions edits about seven places. Proposals for those three, then shorter notes.
metrics depends on what it measures
What is there:
metrics imports eight internal packages. Six of them it needs for the concrete types its seven Add… methods take: AddBansAndClients(*bans.Ledger, *ratelimit.Limiter, …), AddRules(*rules.Files), AddRemoteLog(*remotelog.Sender), AddReputation(*reputation.Lists, *reputation.DNSBL), AddAbuseIPDB(*reputation.AbuseIPDB), AddAlerts(*alerts.Queue), AddLookupFile(…).
The seven are called from two files, in an order the comments set: AddRules "once, before RuleMatched", AddAbuseIPDB "once, after AddReputation". Until its Add… call, a part's metric fields are nil.
lookup goes the other way. GeoJS is handed the whole *metrics.Metrics and increments three of its public counters. So metrics cannot name lookup.File, and AddLookupFile takes two closures instead. Its comment says so.
state also calls into metrics, for the state file counts. It goes away under the pending SQLite state store spec (#217).
metrics has no test file. Every other package with production code has one.
Why it matters:
A new part that has numbers adds an import to metrics, an Add… method, and a call at the correct place in another file. A call out of order is a nil pointer when the program is up.
Nothing can build metrics with stand-in parts, so nothing tests it alone.
Options:
interfaces in metrics: metrics declares, beside each group of metrics, the two or three methods it reads. All parts are handed in at once through metrics.Params, so there is no call order and no nil field. GeoJS exposes its three counts as methods, as every other part does, and lookup stops importing metrics. metrics drops the imports it holds only for a part's type. What remains is the types its event methods take, such as requestlog.Line and ratelimit.Request. The metric names stay in one file.
each part is a prometheus.Collector: each part describes its own metrics, and metrics only registers them. Every part then imports the Prometheus client, and the metric names spread over a dozen files.
Recommendation: interfaces in metrics.
Before (internal/metrics/metrics.go):
funcNew(topNint,instanceNamestring)*Metricsfunc(m*Metrics)AddBansAndClients(ledger*bans.Ledger,limiter*ratelimit.Limiter,nowfunc()time.Time,)// six more Add… methods
After (names are placeholders; new names are sneak's to approve):
// banCounts is what the metrics read from the ban ledger.typebanCountsinterface{Made(causestring)intCount(nowtime.Time)(active,permanentint)}typeParamsstruct{TopNintInstanceNamestringNowfunc()time.TimeBansbanCountsClientsclientCounts// one field per part; nil for a part that is off}funcNew(paramsParams)*Metrics
This needs the parts built in one place first, which the comment on construction, start and stop proposes.
CODE_STYLEGUIDE_GO.md asks for interfaces defined near the functions that use them. Production code has two interfaces today.
Confidence: high.
The loop that watches a directory exists three times
What is there: fsnotify.NewWatcher() is called in rules/rules.go, lookup/file.go and state/state.go. In rules and lookup, Watch and readAfterChanges are the same code line for line: watch the directory, start a timer at once, restart it at each change, read again when it fires, log watcher errors. They differ in the directory, in which file names count, and in the messages. Both declare const quietTime = 2 * time.Second.
Why it matters: the loop has three details that are easy to get wrong. It watches the directory and not the file, so that a rename is seen, as GO_PACKAGE_DEFAULTS.md records for fsnotify. It starts its wait at once, to catch an edit made before the watch began. It waits for quiet before reading. A fix to one copy has to be made in the others.
Proposal: one function that both call. It takes the directory, a test for which changes count, and what to do once the directory has gone quiet. readAgain stays in each package, since what is read differs.
After (names are placeholders):
// Watch calls changed once at the start and again each time dir has gone// quiet after a change that counts, until ctx is done.funcWatch(ctxcontext.Context,dirstring,countsfunc(fsnotify.Event)bool,changedfunc())error
It needs a home both packages can import. That is one new small package, whose name is sneak's to choose.
The copy in state is left alone. The store spec removes internal/state, and keeps fsnotify for the rule files and the lookup database.
Confidence: high on the finding. Medium on whether two copies of about 60 lines justify a package.
Adding an app's exclusions to coreruleset edits about seven places
Where: internal/coreruleset/coreruleset.go on next at 463a19e.
What is there:
The directives template has one %d and nine %s. Seven of the %s are the per-app exclusions: gitea, WordPress, Drupal, MediaWiki, Mastodon, Vaultwarden, Nextcloud.
Each app has a //go:embed variable. New passes them to fmt.Sprintf by position.
The comment on directives maps each position to a variable in words: "giteaExclusions for the third, wordpressExclusions for the fourth".
The comments number sww's changes to the Core Rule Set: "the eighth change" to "the fourteenth change", in the template and again on each variable.
Why it matters: an eighth app means a new .conf file, a new variable, a new %s, a new template comment, a new argument in the correct position, a longer mapping comment and the next ordinal. Two arguments in the wrong order compile.
Proposal: the per-app files are embedded as one directory with embed.FS, joined in name order, and take one %s. Adding an app is then adding its file. The comments name each change by what it does, not by its number.
Before:
//go:embed gitea.confvargiteaExclusionsstring//go:embed wordpress.confvarwordpressExclusionsstring// five moretext:=fmt.Sprintf(directives,params.ParanoiaLevel,body,exclusions,giteaExclusions,wordpressExclusions,drupalExclusions,mediawikiExclusions,mastodonExclusions,vaultwardenExclusions,nextcloudExclusions)
After (the names are placeholders):
//go:embed apps/*.confvarappExclusionsembed.FS// in New, with apps the files of appExclusions joined in name order:text:=fmt.Sprintf(directives,params.ParanoiaLevel,body,exclusions,apps)
Not checked: that the order of the per-app files among themselves has no effect. Name order differs from today's order.
Confidence: medium, for that reason.
Shorter notes
requestlog holds three things: the two log line types and their writers, the words a decision is logged with (Action…, Offence…), and NewProcessLogger, the process's own slog logger. The conventions give the process logger a package of its own. Moving it is a small gain: requestlog would then be only the request log.
One-symbol imports between parts, looked at and left:
anomaly uses ratelimit.Buckets and requestlog.Line uses ratelimit.Counts. Both reuse a real type. The store spec reshapes ratelimit, so nothing is proposed.
remotelog imports requestlog for FormatTime, one line. Left.
rproxy imports rules for rules.Path, the path as the visitor sent it. discriminator uses it three times and rproxy twice. It is a fact about a request, not about rule files, and any other home trades this import for another. Left.
Four types ask an outside service in the background and remember the answers: lookup.GeoJS, reputation.DNSBL, reputation.AbuseIPDB, routewatch.Client. Each has the same parts: a client, a mutex, a table of answers, a set of questions under way, a time to try again. No shared type is proposed: they differ in what they ask, when they wait and how they count failures, and a generic type would be harder to follow than four plain ones. One difference is worth removing: GeoJS copies nine of its params into fields of its own, where the other three keep params whole.
reputation.Lists keeps every list in one map by URL. One entries struct has a field for each shape: netblocks for a blocklist or a tag list, percentages for the AS file, decisions for CrowdSec. It tells which a list is by comparing its URL with the settings (reputation.go lines 387, 469, 515). The store spec gives each shape a table of its own, so this is an observation only. Confidence: medium.
alerts.Alert.Detail is a map[string]any read back by key. The store spec already replaces it with a typed alerts.Detail.
Model: fable-5-2
## The smaller packages: `metrics`, the watchers, `coreruleset` and what is repeated
Where: `internal/metrics`, `internal/lookup`, `internal/rules`, `internal/coreruleset`, `internal/requestlog`, `internal/reputation`.
Summary: `metrics` depends on everything it measures, and is connected by seven calls in a fixed order. Two packages hold the same loop that watches a directory. Adding an app to the Core Rule Set exclusions edits about seven places. Proposals for those three, then shorter notes.
### `metrics` depends on what it measures
- What is there:
- `metrics` imports eight internal packages. Six of them it needs for the concrete types its seven `Add…` methods take: `AddBansAndClients(*bans.Ledger, *ratelimit.Limiter, …)`, `AddRules(*rules.Files)`, `AddRemoteLog(*remotelog.Sender)`, `AddReputation(*reputation.Lists, *reputation.DNSBL)`, `AddAbuseIPDB(*reputation.AbuseIPDB)`, `AddAlerts(*alerts.Queue)`, `AddLookupFile(…)`.
- The seven are called from two files, in an order the comments set: `AddRules` "once, before RuleMatched", `AddAbuseIPDB` "once, after AddReputation". Until its `Add…` call, a part's metric fields are nil.
- `lookup` goes the other way. `GeoJS` is handed the whole `*metrics.Metrics` and increments three of its public counters. So `metrics` cannot name `lookup.File`, and `AddLookupFile` takes two closures instead. Its comment says so.
- `state` also calls into `metrics`, for the state file counts. It goes away under the pending SQLite state store spec (https://git.eeqj.de/sneak/smallwebwaf/issues/217).
- `metrics` has no test file. Every other package with production code has one.
- Why it matters:
- A new part that has numbers adds an import to `metrics`, an `Add…` method, and a call at the correct place in another file. A call out of order is a nil pointer when the program is up.
- Nothing can build `metrics` with stand-in parts, so nothing tests it alone.
- Options:
- **interfaces in `metrics`**: `metrics` declares, beside each group of metrics, the two or three methods it reads. All parts are handed in at once through `metrics.Params`, so there is no call order and no nil field. `GeoJS` exposes its three counts as methods, as every other part does, and `lookup` stops importing `metrics`. `metrics` drops the imports it holds only for a part's type. What remains is the types its event methods take, such as `requestlog.Line` and `ratelimit.Request`. The metric names stay in one file.
- **each part is a `prometheus.Collector`**: each part describes its own metrics, and `metrics` only registers them. Every part then imports the Prometheus client, and the metric names spread over a dozen files.
- Recommendation: **interfaces in `metrics`**.
Before (`internal/metrics/metrics.go`):
```go
func New(topN int, instanceName string) *Metrics
func (m *Metrics) AddBansAndClients(
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
)
// six more Add… methods
```
After (names are placeholders; new names are sneak's to approve):
```go
// banCounts is what the metrics read from the ban ledger.
type banCounts interface {
Made(cause string) int
Count(now time.Time) (active, permanent int)
}
type Params struct {
TopN int
InstanceName string
Now func() time.Time
Bans banCounts
Clients clientCounts
// one field per part; nil for a part that is off
}
func New(params Params) *Metrics
```
- This needs the parts built in one place first, which the comment on construction, start and stop proposes.
- `CODE_STYLEGUIDE_GO.md` asks for interfaces defined near the functions that use them. Production code has two interfaces today.
- Confidence: high.
### The loop that watches a directory exists three times
- What is there: `fsnotify.NewWatcher()` is called in `rules/rules.go`, `lookup/file.go` and `state/state.go`. In `rules` and `lookup`, `Watch` and `readAfterChanges` are the same code line for line: watch the directory, start a timer at once, restart it at each change, read again when it fires, log watcher errors. They differ in the directory, in which file names count, and in the messages. Both declare `const quietTime = 2 * time.Second`.
- Why it matters: the loop has three details that are easy to get wrong. It watches the directory and not the file, so that a rename is seen, as `GO_PACKAGE_DEFAULTS.md` records for `fsnotify`. It starts its wait at once, to catch an edit made before the watch began. It waits for quiet before reading. A fix to one copy has to be made in the others.
- Proposal: one function that both call. It takes the directory, a test for which changes count, and what to do once the directory has gone quiet. `readAgain` stays in each package, since what is read differs.
After (names are placeholders):
```go
// Watch calls changed once at the start and again each time dir has gone
// quiet after a change that counts, until ctx is done.
func Watch(ctx context.Context, dir string, counts func(fsnotify.Event) bool, changed func()) error
```
- It needs a home both packages can import. That is one new small package, whose name is sneak's to choose.
- The copy in `state` is left alone. The store spec removes `internal/state`, and keeps `fsnotify` for the rule files and the lookup database.
- Confidence: high on the finding. Medium on whether two copies of about 60 lines justify a package.
### Adding an app's exclusions to `coreruleset` edits about seven places
- Where: `internal/coreruleset/coreruleset.go` on `next` at `463a19e`.
- What is there:
- The `directives` template has one `%d` and nine `%s`. Seven of the `%s` are the per-app exclusions: gitea, WordPress, Drupal, MediaWiki, Mastodon, Vaultwarden, Nextcloud.
- Each app has a `//go:embed` variable. `New` passes them to `fmt.Sprintf` by position.
- The comment on `directives` maps each position to a variable in words: "giteaExclusions for the third, wordpressExclusions for the fourth".
- The comments number sww's changes to the Core Rule Set: "the eighth change" to "the fourteenth change", in the template and again on each variable.
- Why it matters: an eighth app means a new `.conf` file, a new variable, a new `%s`, a new template comment, a new argument in the correct position, a longer mapping comment and the next ordinal. Two arguments in the wrong order compile.
- Proposal: the per-app files are embedded as one directory with `embed.FS`, joined in name order, and take one `%s`. Adding an app is then adding its file. The comments name each change by what it does, not by its number.
Before:
```go
//go:embed gitea.conf
var giteaExclusions string
//go:embed wordpress.conf
var wordpressExclusions string
// five more
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
giteaExclusions, wordpressExclusions, drupalExclusions,
mediawikiExclusions, mastodonExclusions, vaultwardenExclusions,
nextcloudExclusions)
```
After (the names are placeholders):
```go
//go:embed apps/*.conf
var appExclusions embed.FS
// in New, with apps the files of appExclusions joined in name order:
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions, apps)
```
- Not checked: that the order of the per-app files among themselves has no effect. Name order differs from today's order.
- Confidence: medium, for that reason.
### Shorter notes
- `requestlog` holds three things: the two log line types and their writers, the words a decision is logged with (`Action…`, `Offence…`), and `NewProcessLogger`, the process's own `slog` logger. The conventions give the process logger a package of its own. Moving it is a small gain: `requestlog` would then be only the request log.
- One-symbol imports between parts, looked at and left:
- `anomaly` uses `ratelimit.Buckets` and `requestlog.Line` uses `ratelimit.Counts`. Both reuse a real type. The store spec reshapes `ratelimit`, so nothing is proposed.
- `remotelog` imports `requestlog` for `FormatTime`, one line. Left.
- `rproxy` imports `rules` for `rules.Path`, the path as the visitor sent it. `discriminator` uses it three times and `rproxy` twice. It is a fact about a request, not about rule files, and any other home trades this import for another. Left.
- Four types ask an outside service in the background and remember the answers: `lookup.GeoJS`, `reputation.DNSBL`, `reputation.AbuseIPDB`, `routewatch.Client`. Each has the same parts: a client, a mutex, a table of answers, a set of questions under way, a time to try again. No shared type is proposed: they differ in what they ask, when they wait and how they count failures, and a generic type would be harder to follow than four plain ones. One difference is worth removing: `GeoJS` copies nine of its params into fields of its own, where the other three keep `params` whole.
- `reputation.Lists` keeps every list in one map by URL. One `entries` struct has a field for each shape: netblocks for a blocklist or a tag list, percentages for the AS file, decisions for CrowdSec. It tells which a list is by comparing its URL with the settings (`reputation.go` lines 387, 469, 515). The store spec gives each shape a table of its own, so this is an observation only. Confidence: medium.
- `alerts.Alert.Detail` is a `map[string]any` read back by key. The store spec already replaces it with a typed `alerts.Detail`.
Model: fable-5-2
Summary: rproxy reads well from the top: ServeHTTP is one linear sequence. The hard part is the two time limits, which share one type and one file with everything else a request does. Three proposals, largest first, then what is left alone.
The time limits have no type of their own
What is there:
request has 25 fields. The last six are a mutex and the five fields it guards: timersStopped, requestTimer, responseTimer, requestEnd, answerStarted.
Twelve functions at lines 423 to 593 of request.go, the last third of the file, work on them: requestDeadline, stopReadingBody, startRequestTimer, requestTimedOut, gotFirstResponseByte, wroteRequest, responseTimedOut, responseReceived, setResponseDeadline, endRequest, stopTimers, stopTimer.
They are called from four kinds of goroutine: the handler's own, the transport's trace hooks, the two timers, and the read of the app's response body.
They read the clock directly, with time.Now and time.AfterFunc.
Why it matters:
This is the hardest code in rproxy: which side was being waited on when a limit passed decides between a 408 and a 504. It is methods on the same type as the path rewrite and the log line, so the type does not show which fields other goroutines touch.
It is tested only through whole requests over real connections with real waits. limits_test.go sleeps for 2.1 seconds in one test.
Proposal: the time limits become a type of their own, in a file of their own in internal/rproxy. It owns the mutex, the two timers and the two times. request holds one field of it.
After (names are placeholders; new names are sneak's to approve):
typerequeststruct{// nineteen fieldslimits*timeLimits}// timeLimits are one request's two time limits: SWRP_MAX_REQUEST_TIME_MS// and SWRP_MAX_RESPONSE_TIME_MS.typetimeLimitsstruct{maxRequestTimetime.DurationmaxResponseTimetime.Durationstarttime.Time// refuse is called with 408 or 504 when a limit passes.refusefunc(statusint)musync.MutexstoppedboolrequestTimer*time.TimerresponseTimer*time.TimerrequestEndtime.TimeanswerStartedtime.Time}
The catch: the limits are tied to the request in two more places. requestTimedOut reads whether the body is waiting for the visitor, and sets a read deadline on the visitor's connection while it holds the lock. The new type has to be given both. That is some connecting code, and the reason for medium confidence.
What it buys: the mutex and everything it guards sit in one type in one file. The timers can then be tested alone with testing/synctest, which the tests of seven other packages use and rproxy's do not.
Confidence: medium.
The body wrappers hold the whole request
What is there: requestBody and responseBody in bodies.go each hold rq *request. They are read on other goroutines than the handler's. requestBody uses the pointer for one call, rq.refuse(413). responseBody uses it for rq.refuse(502), rq.responseReceived() and rq.in.Context().
Why it matters: with the whole request in reach, the type does not say what another goroutine may touch. refuse is safe there because refused is atomic. The next field someone reads from b.rq may not be.
Proposal: each wrapper holds only what it calls. requestBody gets a refuse func(status int). responseBody gets that, a function to call once the app's whole answer has arrived, and the request's context.
Confidence: high. The gain is small and local.
Waiting for reports is three fields on handler
What is there: handler holds reports sync.WaitGroup, reportsMu and reportsStopped. sendReport in discriminator.go locks, tests the flag and starts the goroutine. Server.WaitForReports in rproxy.go reaches into s.h to lock, set the flag and wait.
Why it matters: the invariant, that no report starts once waiting has begun, is held by two functions in two files on two types.
Proposal: a small private type with two methods, one that starts a report unless stopped and one that stops and waits. About twenty lines, with the invariant in one place.
If the proposal on background work in the comment on construction, start and stop is taken, this becomes the one method of rproxy.Server that returns when its background work has ended.
Confidence: high.
Left alone
ServeHTTP in rproxy.go reads as SPEC's steps 1, 2 and 17 in order. Nothing is proposed for it.
allowedMethods is one string, the text of the Allow header, and ServeHTTP splits it for each request to test a method (rproxy.go:177). A slice declared once would read more directly. It would need two package variables and a linter suppression for each, which outweighs the gain. No change.
forward builds a new httputil.ReverseProxy for each request, so that its hooks are methods of that request. It is plain to read, and the transport underneath is shared. No change.
panic(http.ErrAbortHandler) for a close disposition is the way Go's server documents to drop a connection.
newQuestion, noteAnswer and postReport in rproxy/discriminator.go are covered by the comment on what rproxy and discriminator send each other.
rproxy never writes to disk and holds no state between requests. Nothing in these proposals changes that.
Not read: the bodies of rewrite, modifyResponse and answerError (request.go lines 200 to 283), and bodies.go after line 110.
Model: fable-5-2
## Inside `rproxy`
Where: `internal/rproxy/rproxy.go`, `request.go`, `bodies.go`, `discriminator.go`.
Summary: `rproxy` reads well from the top: `ServeHTTP` is one linear sequence. The hard part is the two time limits, which share one type and one file with everything else a request does. Three proposals, largest first, then what is left alone.
### The time limits have no type of their own
- What is there:
- `request` has 25 fields. The last six are a mutex and the five fields it guards: `timersStopped`, `requestTimer`, `responseTimer`, `requestEnd`, `answerStarted`.
- Twelve functions at lines 423 to 593 of `request.go`, the last third of the file, work on them: `requestDeadline`, `stopReadingBody`, `startRequestTimer`, `requestTimedOut`, `gotFirstResponseByte`, `wroteRequest`, `responseTimedOut`, `responseReceived`, `setResponseDeadline`, `endRequest`, `stopTimers`, `stopTimer`.
- They are called from four kinds of goroutine: the handler's own, the transport's trace hooks, the two timers, and the read of the app's response body.
- They read the clock directly, with `time.Now` and `time.AfterFunc`.
- Why it matters:
- This is the hardest code in `rproxy`: which side was being waited on when a limit passed decides between a 408 and a 504. It is methods on the same type as the path rewrite and the log line, so the type does not show which fields other goroutines touch.
- It is tested only through whole requests over real connections with real waits. `limits_test.go` sleeps for 2.1 seconds in one test.
- Proposal: the time limits become a type of their own, in a file of their own in `internal/rproxy`. It owns the mutex, the two timers and the two times. `request` holds one field of it.
Before (`request.go`):
```go
type request struct {
// nineteen fields
mu sync.Mutex
timersStopped bool
requestTimer *time.Timer
responseTimer *time.Timer
requestEnd time.Time
answerStarted time.Time
}
```
After (names are placeholders; new names are sneak's to approve):
```go
type request struct {
// nineteen fields
limits *timeLimits
}
// timeLimits are one request's two time limits: SWRP_MAX_REQUEST_TIME_MS
// and SWRP_MAX_RESPONSE_TIME_MS.
type timeLimits struct {
maxRequestTime time.Duration
maxResponseTime time.Duration
start time.Time
// refuse is called with 408 or 504 when a limit passes.
refuse func(status int)
mu sync.Mutex
stopped bool
requestTimer *time.Timer
responseTimer *time.Timer
requestEnd time.Time
answerStarted time.Time
}
```
- The catch: the limits are tied to the request in two more places. `requestTimedOut` reads whether the body is waiting for the visitor, and sets a read deadline on the visitor's connection while it holds the lock. The new type has to be given both. That is some connecting code, and the reason for medium confidence.
- What it buys: the mutex and everything it guards sit in one type in one file. The timers can then be tested alone with `testing/synctest`, which the tests of seven other packages use and `rproxy`'s do not.
- Confidence: medium.
### The body wrappers hold the whole request
- What is there: `requestBody` and `responseBody` in `bodies.go` each hold `rq *request`. They are read on other goroutines than the handler's. `requestBody` uses the pointer for one call, `rq.refuse(413)`. `responseBody` uses it for `rq.refuse(502)`, `rq.responseReceived()` and `rq.in.Context()`.
- Why it matters: with the whole request in reach, the type does not say what another goroutine may touch. `refuse` is safe there because `refused` is atomic. The next field someone reads from `b.rq` may not be.
- Proposal: each wrapper holds only what it calls. `requestBody` gets a `refuse func(status int)`. `responseBody` gets that, a function to call once the app's whole answer has arrived, and the request's context.
- Confidence: high. The gain is small and local.
### Waiting for reports is three fields on `handler`
- What is there: `handler` holds `reports sync.WaitGroup`, `reportsMu` and `reportsStopped`. `sendReport` in `discriminator.go` locks, tests the flag and starts the goroutine. `Server.WaitForReports` in `rproxy.go` reaches into `s.h` to lock, set the flag and wait.
- Why it matters: the invariant, that no report starts once waiting has begun, is held by two functions in two files on two types.
- Proposal: a small private type with two methods, one that starts a report unless stopped and one that stops and waits. About twenty lines, with the invariant in one place.
Before (`rproxy.go`):
```go
func (s *Server) WaitForReports() {
s.h.reportsMu.Lock()
s.h.reportsStopped = true
s.h.reportsMu.Unlock()
s.h.reports.Wait()
}
```
After (names are placeholders):
```go
func (s *Server) WaitForReports() {
s.h.reports.stopAndWait()
}
```
- If the proposal on background work in the comment on construction, start and stop is taken, this becomes the one method of `rproxy.Server` that returns when its background work has ended.
- Confidence: high.
### Left alone
- `ServeHTTP` in `rproxy.go` reads as SPEC's steps 1, 2 and 17 in order. Nothing is proposed for it.
- `allowedMethods` is one string, the text of the `Allow` header, and `ServeHTTP` splits it for each request to test a method (`rproxy.go:177`). A slice declared once would read more directly. It would need two package variables and a linter suppression for each, which outweighs the gain. No change.
- `forward` builds a new `httputil.ReverseProxy` for each request, so that its hooks are methods of that request. It is plain to read, and the transport underneath is shared. No change.
- `panic(http.ErrAbortHandler)` for a `close` disposition is the way Go's server documents to drop a connection.
- `newQuestion`, `noteAnswer` and `postReport` in `rproxy/discriminator.go` are covered by the comment on what `rproxy` and `discriminator` send each other.
- `rproxy` never writes to disk and holds no state between requests. Nothing in these proposals changes that.
Not read: the bodies of `rewrite`, `modifyResponse` and `answerError` (`request.go` lines 200 to 283), and `bodies.go` after line 110.
Model: fable-5-2
Constructors, file layout, the order of the proposals, and what was not checked
Where: across internal/.
Summary: three smaller points that belong to no one part, then what was looked at and left, how the proposals in the other comments depend on one another, and what this review did not check.
Three constructors take their arguments by position
What is there: fifteen constructors under internal/ take one Params struct. Three do not:
CODE_STYLEGUIDE_GO.md asks for a Params struct even for one argument.
Proposal: the three take Params, as the others do. For metrics.New that is part of the proposal in the comment on the smaller packages.
The pending SQLite state store spec (#217) writes ratelimit.New(limits, clients) and bans.New(rules, held) by position. That is the spec's to settle. It is noted so that ratelimit.New is not changed twice.
Confidence: high. The gain is consistency.
reputation and lookup hold several types, each with its own constructor
What is there:
reputation holds Lists (New, Params), DNSBL (NewDNSBL, DNSBLParams) and AbuseIPDB (NewAbuseIPDB, AbuseIPDBParams). The three share no state, and one helper of ten lines, raiseFailure.
lookup holds GeoJS (New, Params) and File (OpenFile, FileParams). The two share the Answer type.
CODE_STYLEGUIDE_GO.md prefers one type per package with a plain New(), and NewThing() only now and then.
The symbol names audits (#213) already propose names that say which New is which. That is not reopened here. This finding is the other answer to the same ambiguity, and choosing it would make those two renames unnecessary.
Options, for reputation only:
packages as they are: the names audit settles which constructor is which. Nothing moves.
one package per source: DNSBL and AbuseIPDB each move to a package of about 330 lines, with New and Params. raiseFailure is written three times, or alerts takes it.
Recommendation: packages as they are. The store spec changes how all three types hold what they know, and a split at the same time would make that change harder to review. If a split is wanted, it is cheapest after the store has landed.
lookup stays one package either way: its two types answer the same question and share the answer's type.
Confidence: medium.
internal/discriminator's files do not say what they hold
What is there:
proxy.go holds Params, Server, handler, newHandler, newReputation, NewCoreRuleSet and four of the endpoint paths. discriminator is not a proxy.
New is in discriminator.go, with the code that reads a question, writes the answer and takes a report.
Sixteen heuristics have a file each. The seventeenth, activeBan, is in bans.go with the three banFor… functions.
Five files hold code that heuristics share: bans.go, biased.go, client.go, lookup.go, reputation.go. Nothing in a file's name tells those from a heuristic's file. heuristics.go is the only index.
Why it matters: a newcomer looking for where the server is built opens discriminator.go and does not find it. CODE_STYLEGUIDE_GO.md asks for struct types near their constructors.
Proposal, moves only, with no identifier renamed:
Params, Server, New, handler and newHandler sit together in discriminator.go, the file named for the package.
The question, answer and report code leaves discriminator.go for a file of its own. If the package proposed in the comment on what rproxy and discriminator send each other is made, most of it goes there.
activeBan gets a file of its own, as the other sixteen have.
proxy.go goes away.
New file names are names too, and are sneak's to approve.
Confidence: high.
Looked at and left
Clocks. Every part that counts or stores a time takes Now func() time.Time in its Params. The real clock is read only to measure a duration, to stamp a line as it is sent, and to wait before trying again (alerts.go:693, remotelog.go:117 and :232, and the time limits in rproxy). The split is consistent. No change.
panic. Eight constructors call panic(err) after building an LRU, each with a comment that it fails only for a size below one. Seven sizes are constants. The eighth is SWW_MAX_TRACKED_CLIENTS in ratelimit.New, which config reads as a number that is not off. The guide says to avoid panic in library code. Six of the eight go with the parts' own tables under the store spec. No change now.
panic(http.ErrAbortHandler), five times, is how Go's server documents closing a connection without an answer.
Interfaces. Production code declares two: state.stateFile and discriminator.heuristic. The proposals add few: a handful of small ones in metrics, and one optional one among the heuristics. None is a layer of its own.
Test hooks. alerts, lookup and reputation each have an export_test.go with SetTransport or SetDial, so that tests can stand in for the network. An http.RoundTripper in each Params would do the same without the hook. It would also add a field production code never sets, so it is left.
Linter suppressions. Sixteen //nolint:tagliatelle lines in production code sit on JSON field names in snake_case. All of this project's JSON is snake_case, so the setting is what differs. That is a change to the canonical config in prompts, not to this repo.
One code comment carries an issue link: rproxy/discriminator.go:72. A comment that says what the code does not do yet, in words, outlives the tracker.
How the proposals depend on one another
Can be taken alone, in any order:
the package for the three messages;
the chi routes for discriminator's own endpoints;
the named types for values with a fixed set of choices;
one serve function, and start-up errors wrapped where they happen;
the one function that watches a directory;
the per-app exclusions as an embedded directory;
the three inside rproxy;
the file moves in internal/discriminator.
"The parts are built in one place" comes first for two others: metrics.Params with interfaces, and fx. fx with providers in internal/sww replaces the hand-written version of it.
Inside the heuristics, the order is: heuristics as built values, then typed facts on decision, then the status on the verdict. The one ban function and the plain statements after the answer are independent of those.
Everything in the comment on internal/config waits for the settings change through viper (#219) and for the issue that gives each daemon's settings one file (#227).
Anything that touches bans.Ledger, ratelimit or reputation is easier after the store spec is built, since it rewrites the same constructors.
The pair of ban methods meets the names audit on bans.Ledger and should be decided with it.
What was not checked
Nothing was built, tested or linted. This is a review by reading. The Before and After blocks were not compiled.
The code was read at 3b987b9. next moved to 463a19e during the work, and only internal/coreruleset/coreruleset.go was read again there.
Not read:
docs/;
SPEC.md outside "Architecture", "What happens to a request" and "Planned, not built";
.golangci.yml past its first 60 lines;
the tests, beyond their shape and what they import;
parts of the bodies of rules, anomaly, remotelog, routewatch, reputation and alerts, where only the types, constructors and exported functions were read;
config.go between its declarations and its helpers, beyond the one setting traced as an example;
in rproxy, the bodies of rewrite, modifyResponse and answerError, and bodies.go after line 110.
The diff of the settings change through viper was not read: only its description, its comments and its list of files.
Whether SPEC and the code agree was not checked, beyond the order of the eighteen steps.
Unverified claims, each also marked where it is made:
that the order of the per-app exclusion files among themselves has no effect;
that httpguts.ValidHeaderFieldName accepts exactly what config.IsHeaderName does;
that fx's start and stop work inside testing/synctest;
how BanForAdmin and WouldBePermanent fit the one ban method.
Judgement calls:
fx, chi and cobra are recommended on the strength of the two guides and GO_PACKAGE_DEFAULTS.md. No ruling on record applies them to this repo.
Where this repo differs from the guides by the owner's design (the health path, bearer tokens, the log level setting), that is listed as context and not as a finding.
No proposal changes a setting, renames an existing identifier, or touches internal/state or how a part stores what it knows. Where a proposal meets one of the four open decisions, the comment says so.
Tool failure: the gitea MCP tools could not be called in this session. The tracker's REST API was used in their place.
Model: fable-5-2
## Constructors, file layout, the order of the proposals, and what was not checked
Where: across `internal/`.
Summary: three smaller points that belong to no one part, then what was looked at and left, how the proposals in the other comments depend on one another, and what this review did not check.
### Three constructors take their arguments by position
- What is there: fifteen constructors under `internal/` take one `Params` struct. Three do not:
- `ratelimit.New(limits Limits, maxClients int)`
- `metrics.New(topN int, instanceName string)`
- `requestlog.NewProcessLogger(w io.Writer, instanceName string, level slog.Level)`
- `CODE_STYLEGUIDE_GO.md` asks for a `Params` struct even for one argument.
- Proposal: the three take `Params`, as the others do. For `metrics.New` that is part of the proposal in the comment on the smaller packages.
- The pending SQLite state store spec (https://git.eeqj.de/sneak/smallwebwaf/issues/217) writes `ratelimit.New(limits, clients)` and `bans.New(rules, held)` by position. That is the spec's to settle. It is noted so that `ratelimit.New` is not changed twice.
- Confidence: high. The gain is consistency.
### `reputation` and `lookup` hold several types, each with its own constructor
- What is there:
- `reputation` holds `Lists` (`New`, `Params`), `DNSBL` (`NewDNSBL`, `DNSBLParams`) and `AbuseIPDB` (`NewAbuseIPDB`, `AbuseIPDBParams`). The three share no state, and one helper of ten lines, `raiseFailure`.
- `lookup` holds `GeoJS` (`New`, `Params`) and `File` (`OpenFile`, `FileParams`). The two share the `Answer` type.
- `CODE_STYLEGUIDE_GO.md` prefers one type per package with a plain `New()`, and `NewThing()` only now and then.
- The symbol names audits (https://git.eeqj.de/sneak/smallwebwaf/issues/213) already propose names that say which `New` is which. That is not reopened here. This finding is the other answer to the same ambiguity, and choosing it would make those two renames unnecessary.
- Options, for `reputation` only:
- **packages as they are**: the names audit settles which constructor is which. Nothing moves.
- **one package per source**: `DNSBL` and `AbuseIPDB` each move to a package of about 330 lines, with `New` and `Params`. `raiseFailure` is written three times, or `alerts` takes it.
- Recommendation: **packages as they are**. The store spec changes how all three types hold what they know, and a split at the same time would make that change harder to review. If a split is wanted, it is cheapest after the store has landed.
- `lookup` stays one package either way: its two types answer the same question and share the answer's type.
- Confidence: medium.
### `internal/discriminator`'s files do not say what they hold
- What is there:
- `proxy.go` holds `Params`, `Server`, `handler`, `newHandler`, `newReputation`, `NewCoreRuleSet` and four of the endpoint paths. `discriminator` is not a proxy.
- `New` is in `discriminator.go`, with the code that reads a question, writes the answer and takes a report.
- Sixteen heuristics have a file each. The seventeenth, `activeBan`, is in `bans.go` with the three `banFor…` functions.
- Five files hold code that heuristics share: `bans.go`, `biased.go`, `client.go`, `lookup.go`, `reputation.go`. Nothing in a file's name tells those from a heuristic's file. `heuristics.go` is the only index.
- Why it matters: a newcomer looking for where the server is built opens `discriminator.go` and does not find it. `CODE_STYLEGUIDE_GO.md` asks for struct types near their constructors.
- Proposal, moves only, with no identifier renamed:
- `Params`, `Server`, `New`, `handler` and `newHandler` sit together in `discriminator.go`, the file named for the package.
- The question, answer and report code leaves `discriminator.go` for a file of its own. If the package proposed in the comment on what `rproxy` and `discriminator` send each other is made, most of it goes there.
- `activeBan` gets a file of its own, as the other sixteen have.
- `proxy.go` goes away.
- New file names are names too, and are sneak's to approve.
- Confidence: high.
### Looked at and left
- Clocks. Every part that counts or stores a time takes `Now func() time.Time` in its `Params`. The real clock is read only to measure a duration, to stamp a line as it is sent, and to wait before trying again (`alerts.go:693`, `remotelog.go:117` and `:232`, and the time limits in `rproxy`). The split is consistent. No change.
- `panic`. Eight constructors call `panic(err)` after building an LRU, each with a comment that it fails only for a size below one. Seven sizes are constants. The eighth is `SWW_MAX_TRACKED_CLIENTS` in `ratelimit.New`, which `config` reads as a number that is not off. The guide says to avoid `panic` in library code. Six of the eight go with the parts' own tables under the store spec. No change now.
- `panic(http.ErrAbortHandler)`, five times, is how Go's server documents closing a connection without an answer.
- Interfaces. Production code declares two: `state.stateFile` and `discriminator.heuristic`. The proposals add few: a handful of small ones in `metrics`, and one optional one among the heuristics. None is a layer of its own.
- Test hooks. `alerts`, `lookup` and `reputation` each have an `export_test.go` with `SetTransport` or `SetDial`, so that tests can stand in for the network. An `http.RoundTripper` in each `Params` would do the same without the hook. It would also add a field production code never sets, so it is left.
- Linter suppressions. Sixteen `//nolint:tagliatelle` lines in production code sit on JSON field names in snake_case. All of this project's JSON is snake_case, so the setting is what differs. That is a change to the canonical config in `prompts`, not to this repo.
- One code comment carries an issue link: `rproxy/discriminator.go:72`. A comment that says what the code does not do yet, in words, outlives the tracker.
### How the proposals depend on one another
- Can be taken alone, in any order:
- the package for the three messages;
- the `chi` routes for `discriminator`'s own endpoints;
- the named types for values with a fixed set of choices;
- one serve function, and start-up errors wrapped where they happen;
- the one function that watches a directory;
- the per-app exclusions as an embedded directory;
- the three inside `rproxy`;
- the file moves in `internal/discriminator`.
- "The parts are built in one place" comes first for two others: `metrics.Params` with interfaces, and fx. fx with providers in `internal/sww` replaces the hand-written version of it.
- Inside the heuristics, the order is: heuristics as built values, then typed facts on `decision`, then the status on the verdict. The one ban function and the plain statements after the answer are independent of those.
- Everything in the comment on `internal/config` waits for the settings change through viper (https://git.eeqj.de/sneak/smallwebwaf/pulls/219) and for the issue that gives each daemon's settings one file (https://git.eeqj.de/sneak/smallwebwaf/issues/227).
- Anything that touches `bans.Ledger`, `ratelimit` or `reputation` is easier after the store spec is built, since it rewrites the same constructors.
- The pair of ban methods meets the names audit on `bans.Ledger` and should be decided with it.
### What was not checked
- Nothing was built, tested or linted. This is a review by reading. The Before and After blocks were not compiled.
- The code was read at `3b987b9`. `next` moved to `463a19e` during the work, and only `internal/coreruleset/coreruleset.go` was read again there.
- Not read:
- `docs/`;
- `SPEC.md` outside "Architecture", "What happens to a request" and "Planned, not built";
- `.golangci.yml` past its first 60 lines;
- the tests, beyond their shape and what they import;
- parts of the bodies of `rules`, `anomaly`, `remotelog`, `routewatch`, `reputation` and `alerts`, where only the types, constructors and exported functions were read;
- `config.go` between its declarations and its helpers, beyond the one setting traced as an example;
- in `rproxy`, the bodies of `rewrite`, `modifyResponse` and `answerError`, and `bodies.go` after line 110.
- The diff of the settings change through viper was not read: only its description, its comments and its list of files.
- Whether SPEC and the code agree was not checked, beyond the order of the eighteen steps.
- Unverified claims, each also marked where it is made:
- that the order of the per-app exclusion files among themselves has no effect;
- that `httpguts.ValidHeaderFieldName` accepts exactly what `config.IsHeaderName` does;
- that fx's start and stop work inside `testing/synctest`;
- how `BanForAdmin` and `WouldBePermanent` fit the one ban method.
- Judgement calls:
- fx, `chi` and `cobra` are recommended on the strength of the two guides and `GO_PACKAGE_DEFAULTS.md`. No ruling on record applies them to this repo.
- Where this repo differs from the guides by the owner's design (the health path, bearer tokens, the log level setting), that is listed as context and not as a finding.
- No proposal changes a setting, renames an existing identifier, or touches `internal/state` or how a part stores what it knows. Where a proposal meets one of the four open decisions, the comment says so.
- Tool failure: the gitea MCP tools could not be called in this session. The tracker's REST API was used in their place.
Model: fable-5-2
sneak
was assigned by clawbot2026-11-11 02:58:08 +01:00
i was under the impression that rproxy sends the waf the request from the client nearly as-is (perhaps body truncated at max 1MB or something, as more than that isn't needed for deciding), with just a few headers added for the actual connecting ip and potentially a tls fingerprint later.
rproxy should drop connections when headers are >32KB, unconfigurable. nuanced decisions are for the waf, rproxy just needs basic sanity hardcoded in.
i was under the impression that rproxy sends the waf the request from the client nearly as-is (perhaps body truncated at max 1MB or something, as more than that isn't needed for deciding), with just a few headers added for the actual connecting ip and potentially a tls fingerprint later.
rproxy should drop connections when headers are >32KB, unconfigurable. nuanced decisions are for the waf, rproxy just needs basic sanity hardcoded in.
Filed from the two comments above: the fx wiring as recommended, #414, and the hard-coded 32 KB header limit in rproxy with a check of what it forwards to waf, #415. Workers start on both as soon as the machine's load allows.
Model: opus-5-5
Filed from the two comments above: the fx wiring as recommended, https://git.eeqj.de/sneak/smallwebwaf/issues/414, and the hard-coded 32 KB header limit in `rproxy` with a check of what it forwards to `waf`, https://git.eeqj.de/sneak/smallwebwaf/issues/415. Workers start on both as soon as the machine's load allows.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
sneak, in chat on 10 November:
> try a fable 5.2 genius bar over the general program architecture of smallwebwaf next. specifically i would like to know improvements to modularity, composability, oop best practices, code layout and clarity, comprehensibility, etc. output is a series of detailed comments on the issue ticket describing the layout and design, and proposed improvements. be specific.
For the genius-bar session (Fable 5.2): review the architecture of
smallwebwafas it stands onnext. This is analysis only: change no code, open no PR.Definition of done: a series of detailed comments on this issue that together:
rproxy,discriminator), the packages undercmd/andinternal/, what each package owns, how they depend on one another, how a request flows through them, and where state and settings live;Judge against the repo's own policies and
sneak/prompts' Go style guides (prompts/CODE_STYLEGUIDE_GO.md,prompts/GO_HTTP_SERVER_CONVENTIONS.md). Proposals are for sneak to choose from; nothing is built until he does.This repo's own rules, until its
AGENTS.mdholds them:?token=included.10.0.0.0/8,172.16.0.0/12,192.168.0.0/16).Open decisions that touch the design, to read and not to reopen: the SQLite state store spec (#217), the settings audit (#202), the symbol names audits (#213), and the settings change through viper (#219).
Model: opus-5-5
Layout and design as it stands
This is the first of nine comments, one per part of the system, so that each can be answered on its own. This one describes. The other eight propose. Analysis only: no code was changed.
Read at
3b987b9onnext.nexthas since moved to463a19e, which adds a per-app exclusions file tointernal/corerulesetand changes nothing else described here. Setting names are those onnexttoday; the settings change through viper (#219) renames them.Commands
One binary,
sww.cmd/sww/main.gois one call intointernal/sww, which picks the command from the first argument.sww rproxy: the reverse proxy in front of one app. It keeps nothing between requests and writes nothing to disk. Its settings start withSWRP_.sww discriminator: decides what happens to each request, and keeps the bans, counters and history. Its settings start withSWW_.sww healthcheck: the container's health check. It readsSWW_COMMANDto learn which of the two it is checking.Packages
Line counts are non-test Go: about 16,700 lines in all, beside about 35,800 lines of tests.
internal/sww(661): reads the command, builds every part of the program, starts and stops them.internal/config(2,769): both programs' settings: the two structs, the readers, about 60 parsers, the checks across settings.internal/rproxy(1,423): the proxy. It works out the visitor's address, asksdiscriminator, passes the request to the app or refuses it, applies its own size and time limits, writes its log line and sends the report.internal/discriminator(3,066): answers questions and takes reports. It holds the seventeen heuristics, sixteen of them in a file of their own, and the health, metrics and admin endpoints.internal/ratelimit(597): the table of clients: each client's counters against the limits, and its history.internal/bans(942): the table of bans: makes, finds and lifts them.internal/lookup(783): a client's AS number and country, from the GeoJS service (GeoJS) or from a.mmdbfile on disk whose contents are held in memory (File). Its sub-packagelookuptest(83) writes.mmdbfiles for tests, and only tests import it.internal/reputation(1,335): three sources.Listsfetches lists from URLs: blocklists, tag lists, the AS limit percentages and CrowdSec decisions.DNSBLasks DNSBL zones.AbuseIPDBchecks clients with AbuseIPDB.internal/routewatch(243): asks a routewatch instance for a client's announced netblock and AS. Memory only.internal/rules(500): reads the rule files, matches a request against them, watches their directory.internal/coreruleset(527): the OWASP Core Rule Set through Coraza, with sww's changes and the per-app exclusions.internal/anomaly(411): counters across clients that raise an alert over a threshold. It refuses nothing.internal/alerts(941): the alert queue: cooldown, hourly cap, and sending to a webhook, Slack and ntfy.internal/metrics(710): the Prometheus metrics and the handler that serves them.internal/requestlog(311): the JSON log line types and their writers, and the process logger.internal/remotelog(310): sends the log lines to a syslog server.internal/state(1,060): reads the five JSON state files at start, writes them, and takes in edits made to them while sww is up.Which package imports which
Production code only, internal packages only.
alerts,bans,coreruleset,ratelimit,routewatch.anomaly:alerts,ratelimit.config:alerts,anomaly,remotelog.requestlog:ratelimit.remotelog:requestlog.lookup:alerts,metrics.reputation:alerts,config.rules:alerts,config.metrics:alerts,bans,config,ratelimit,remotelog,reputation,requestlog,rules.state:alerts,anomaly,bans,lookup,metrics,ratelimit,reputation.discriminator:alerts,anomaly,bans,config,coreruleset,lookup,metrics,ratelimit,reputation,requestlog,routewatch,rules,state.rproxy:config,discriminator,requestlog,rules.sww:alerts,config,coreruleset,discriminator,lookup,remotelog,reputation,requestlog,rproxy,rules,state.Three shapes in this graph come up in the later comments:
configis imported by three of the parts (metrics,reputation,rules) and itself imports three others (alerts,anomaly,remotelog).metricsimports eight packages, and two of the packages it measures import it.rproxyimportsdiscriminator, for the names of the headers and paths the two send each other.How one request passes through
rproxy'sServeHTTP(internal/rproxy/rproxy.go) makes arequest, answers its own health path, refuses a method it does not pass on, and reads the first 64 KiB of the body.newQuestionandask(internal/rproxy/discriminator.go) send the visitor's request todiscriminatoras a question.discriminator'sServeHTTP(internal/discriminator/discriminator.go) tells a question from a request to its own endpoints by theX-Sww-Clientheader, and callsanswerQuestion.answerQuestionmakes its ownrequest(startRequest), thencheckandcheckClient(internal/discriminator/request.go) go through the heuristics in the orderheuristics.golists them: thebeforeLookuplist, thenlookUp, then theafterLookuplist. The first verdict ends the checks.allowordenywrites the answer with itsX-Sww-Disposition. TheafterAnswerlist (the refusal limit) follows, then the log line, thenholdForReportkeeps what the report will need, by request id.rproxyacts on the disposition:passToAppandforward(internal/rproxy/request.go) proxy to the app, or the visitor getsdiscriminator's refusal, or the connection is closed. With no answer in time,SWRP_DEFAULT_DISPOSITIONdecides.rproxywrites its log line andsendReportposts the report in the background.discriminator'stakeReportandfinishWithfind what was held, go through theonReportlist (the byte limits), add the request to the client's history and the anomaly counters, and write the second log line.Where state lives
All of it is in memory in
discriminator, each table inside the package that uses it:bans.Ledgerratelimit.Limiterlookup.GeoJSreputation.Lists,reputation.DNSBL,reputation.AbuseIPDBalerts.Queueanomaly.Countersroutewatch.Client, never written to diskawaitingReportindiscriminator'shandler, never written to disk.mmdbfile: read from disk, held in memory, never writteninternal/statewrites the first six groups to five JSON files inSWW_STATE_DIRand reads them back at start.rproxyholds nothing between requests.Where settings live
internal/configreads every setting once, at start, intoconfig.Config(fordiscriminator, about 110 fields) orconfig.RProxy.discriminatorandrproxyeach hold the whole struct. The other packages are given only the values they need, in aParamsstruct thatinternal/swwordiscriminatorfills in. Setting names also appear as string literals in 16 files outsideconfig, in log fields, error messages and metric help text.What open work will change
The later comments propose nothing in these areas and say so where they touch them.
internal/stateand takes the tables out of the packages listed under "Where state lives".internal/configby daemon.The other comments
rproxyanddiscriminatorsend each other: #283 (comment)internal/config, and values with a fixed set of choices: #283 (comment)metrics, the watchers,corerulesetand what is repeated: #283 (comment)rproxy: #283 (comment)Model: fable-5-2
Construction, start and stop
Where:
internal/sww/sww.go,internal/sww/rproxy.go,internal/discriminator/proxy.go.Summary: the program's parts are built in two packages and reached through public fields, and background work is started and waited for one named line at a time. Six proposals follow, then the question of fx, which would replace most of this code.
The parts are built in two places
internal/swwbuilds five parts: the remote log sender, the alert queue, the rule files, the Core Rule Set and the lookup file.newHandlerandnewReputationininternal/discriminator/proxy.gobuild nine more: the metrics, the limiter, the ledger, the anomaly counters,Lists,DNSBL,AbuseIPDB, the routewatch client andGeoJS.discriminator.Serverthen exports ten parts as public fields, so thatswwcan reach back in.loadStateFileshands eight of them tostate.Load.newServerandRunDiscriminatorcallserver.Metrics.Add…three times.servestartsserver.Lists.Runandserver.LookupFile.Watch.discriminator.Paramstakes two URLs (GeoJSURL,AbuseIPDBURL) in place of the parts they are for. A caller cannot hand in a part, and the package's tests read the public fields to see whatNewbuilt.proxy.golines 179, 180, 212, 215;sww.golines 169, 230, 233), in an order that only the comments onmetrics.AddRules,AddReputationandAddAbuseIPDBstate.Params.discriminator.Paramstakes every part already built.internal/swwbuilds them all in one function, in order.Serverexports no parts. A test builds the parts it wants to inspect and keeps them.Before (
internal/discriminator/proxy.go, shortened):After:
lookup.Params.Answeredishandler.addLookuptoday, which needs the limiter, the ledger and the ban netblock size. WithGeoJSbuilt outside, that callback becomes a functiondiscriminatorexports, or is set once both exist.Params.Storeand removes theServerfields that only fedstate.Load. This proposal finishes that move for the rest.Background work is started and waited for by name
servestarts six background loops throughinBackground, and later waits for six channels, one named line each:<-written,<-watched,<-rulesWatched,<-lookupFileWatched,<-listsFetched,<-alertsSent.startSending, with a stop function of its own.rproxyhas a third pattern:handler.reports(async.WaitGroup) andServer.WaitForReports.lookup/lookup.go:344,reputation/dnsbl.go:166,reputation/abuseipdb.go:157androutewatch/routewatch.go:142. Each isgo x.ask(context.WithoutCancel(ctx), …).serve's own comment records a second loss: a request cut off at the shutdown timeout can be missing fromclients.json.Store.Close. A goroutine nobody waits for can.rproxy'sWaitForReportsdoes.swwstarts the loops and waits for them througherrgroup(golang.org/x/syncis a decided default and already an indirect dependency), in place ofinBackgroundand the six channels. Acceptable is: an orderly stop loses nothing a part had started.Before (
internal/sww/sww.go):After:
serveandserveRProxyare one sequence written twiceShutdownwithshutdownTimeout, callClosewhen that times out, wait forServeto return, and check forhttp.ErrServerClosed. Then each does its own last step (files.WriteAll(),server.WaitForReports()) and logs "stopped".internal/swwthat takes the*http.Server, the listener and the process logger, and returns an error. Each command calls it, then does its own last step.Start-up errors are labelled away from where they happen
RunDiscriminatorhas six blocks of this form, at lines 114, 149, 156, 163, 174 and 181:newServerreturns is logged as "cannot use the lookup database", which is correct only while opening the.mmdbfile is the one thing innewServerthat can fail.RunDiscriminatorbecomes a function that returns anerror, plus a short wrapper that logs it once and returns the status. Each step wraps its own error with what it was doing:fmt.Errorf("read the rule files: %w", err).discriminator.NewCoreRuleSetis construction code in the decision packageinternal/discriminator/proxy.go:230.*config.Config, picks the Core Rule Set's settings out of it, and callscoreruleset.New. It returnsnil, niltwice when the Core Rule Set is off, each under//nolint:nilnil. It wraps one error with a setting's name. Its callers areswwand one test.internal/sww, besidenewAlertQueueandnewRemoteLogSender, which do the same job for their parts. With every part handed in,discriminatorhas no use for it.Commands are dispatched by hand
sww.Mainswitches onos.Args[1]and prints a usage text kept by hand.HealthCheckrejects arguments by hand.GO_PACKAGE_DEFAULTS.mdnamesgithub.com/spf13/cobraas the decided default for CLI commands.cobracommands ininternal/sww.cmd/sww/main.gostays one call.fx
CODE_STYLEGUIDE_GO.mdasks forgo.uber.org/fxin any project with more than two or three modules.GO_HTTP_SERVER_CONVENTIONS.mdlists it as mandatory.GO_PACKAGE_DEFAULTS.mdcalls it the standard app skeleton.internal/sww: the other packages keep theirParamsof plain values and import neither fx nor, where they do not today,config.internal/swwholds one small function per part that turns the settings into that part'sParamsand builds it, asnewAlertQueueandnewRemoteLogSenderalready do. fx calls those functions in dependency order. Each background loop is one lifecycle hook, registered where its part is built. This replacesnewServer,loadStateFiles, theinBackgroundlist and both serve functions.GO_HTTP_SERVER_CONVENTIONS.mdto the letter. Each package'sParamsembedsfx.Inand takes the whole config and the logger, and eachNew(lc fx.Lifecycle, params Params)registers its own start and stop. A part then no longer states which settings it uses, a test of one part needs fx's test lifecycle and a whole config, andconfigmust first stop importingalerts,anomalyandremotelog.internal/sww. It gives the guides' skeleton and keeps each part'sParamsas the full statement of what that part needs. That is how the five packages with no internal imports are built today.A sketch of the recommended option. The function names are placeholders; new names are sneak's to approve.
slog, since stdout carries JSON lines only.RunDiscriminator(ctx, params) intstays as the entry the tests call, using fx'sStartandStopand not its signal handling.servewrites out by hand, so the last state write stays last.Run, not inParams. That stays.func() time.Timeandio.Writerneed named types of their own, which are new names.internal/sww, and taking it out means writing the constructor calls back in one function.testing/synctest, which the tests ofinternal/swwuse.Model: fable-5-2
The decision pipeline: the heuristics
Where:
internal/discriminator:heuristics.go, the heuristic files,request.go,discriminator.go,bans.go,biased.go.Summary: the heuristics are split into files, and not yet into units. A heuristic cannot be built, read or tested without the whole discriminator, and adding one still means changes outside its file. That is the aim of the issue that ordered modular heuristics (#195): one file, one line in one place, no change to the others. Six findings follow, largest first.
What stays as it is: one small interface, the first verdict ends the checks, the name and figures go to the log line, and the tests that drive a whole discriminator.
A heuristic declares nothing it needs
type denyNets struct{}.check(ctx, rq *request, now)is handed the whole request. Throughrq.hit reaches the wholehandler: 17 collaborators and all ofconfig.Config.beforeLookup(),afterLookup(),afterAnswer(),onReport().handler, set innewHandler: a change outside its file.newHandlerbuilds the lists once.checkstops usingrq.h.Before (
denynets.go,heuristics.go):After:
denyNets{netblocks: …}, gives it a request with a client address, and checks the verdict. The whole-discriminator tests stay.newHandlerbecomes the one place that shows which settings each heuristic uses.Heuristics pass facts to each other through fields nobody lists
rq.line.Tags: written bytagLists,routeWatchTags,lookUp,crowdSec,dnsbl,abuseIPDB,ruleFilesandcoreRuleSet. Read bytagDenyand bylimitPercentagesinbiased.go.rq.blocklisted,rq.dnsblListed,rq.abuseIPDBHit: written byblocklists,dnsbl,abuseIPDB. Read bylimitPercentages.rq.counted,rq.rateLimitPercent,rq.bytesLimitPercent: written byrateLimits. Read bybyteLimitsandbanForLimit.rq.attack,rq.blockedByBlockRule,rq.blockedByCoreRuleSet: written bytrapPaths,ruleFiles,coreRuleSet. Read byrefusalLimitandaddToHistory.rq.line.ASN,rq.line.ASName,rq.line.Country: written bylookUp. Read bycountries,limitPercentages, the three ban functions and the alerts.decision.lookupAnswerholds the same three values and is not what they read.rq.line.WouldAction: written bycheck. Read bybyteLimitsandrefusalLimit.rq.line.Offence: written bybanForLimit. Read byaddToHistory.tagDenyis listed twice because tags are added on both sides of it.rateLimitsmust come after the three reputation heuristics because it reads their flags.request: again a change outside its file.decisionholds what heuristics tell each other, as typed fields of its own: the tags, the lookup answer it already has, what would have been done, the offence. The comment on each field names what sets it.decisionand never read back.Before (
tagdeny.go,bytelimits.go):After (field names are placeholders):
A verdict does not say what the visitor gets
verdictis{action string; figures map[string]any}. The action is a log word.checkinrequest.goworks out the status afterwards, by aswitchover that word: 403 for the two blocked actions, 431 for headers too large,SWW_BAN_RESPONSEfor the rest.refusalLimitrepeats part of thatswitch, overrq.line.WouldAction.afterAnswerandonReportlists the action is never read:refusalLimitandbyteLimitsreturn&verdict{figures: …}.switchincheck, outside its file. The status is decided by a string comparison far from the code that knows it.verdictcarries the status, set by the heuristic.checkthen only applies observe mode.refusalLimittakes the status from the verdict that refused the request.Before (
request.go):After:
Observe mode is decided in eleven places
rq.h.config.Observeis tested atrequest.go:186,discriminator.go:151,coreruleset.go:85, and eight times indiscriminator/bans.go(lines 34, 118, 142, 167, 178, 203, 216, 267).bans.Ledgerhas each way of making a ban twice:BanForLimitandWouldBanForLimit,BanForAttackandWouldBanForAttack,BanForCrowdSecandWouldBanForCrowdSec. All six call one privateban(…, keep bool).CheckandFindare a fourth pair.banForLimit,banForAttackandbanForCrowdSecindiscriminator/bans.gorepeat one skeleton: find the netblock; in observe mode stop unless an alert would be sent; fill the notes; in observe mode call theWould…method and alert; otherwise call the other, setBanExpiresand alert.bans.Ledgergets one method that makes a ban and one that reports what it would make. Both take the cause, the notes and, for CrowdSec, the end and the scenario, in one struct.discriminator/bans.gogets one function holding the skeleton. It is the one place that tests observe mode for a ban. The threebanFor…functions shrink to filling in what differs.Before (
internal/bans/bans.go):After (the struct's name is a placeholder):
CheckandFindon the same type. The two changes meet there and should be made together.BanForAdminandWouldBePermanentwere not traced through.The lists are not the whole order
heuristics.gosays the order is its lists.checkClientadds three things outside them: theSWW_ALLOW_NETSexemption before the first list,lookUpbetween the two lists, and reading the time again after the lookup.answerQuestionholds one more step, the 404 for a path under/_sww/.tagListsandrouteWatchTagsnever give a verdict. They only add tags.lookUpdoes the same kind of work, and is the only one written into the code and not into a list.tagListsis, andbeforeLookup()andafterLookup()become one list.heuristics.gothen reads top to bottom as SPEC's steps do.The steps after the answer are ordered by
deferanswerQuestionregisters fivedefers. Their reverse order is the order of the work, and two comments explain it ("The last deferred runs first").runAfterAnswer,finish,holdForReport. A refused one doesrunAfterAnswer,countAnomalies,addToHistory,finish,holdForReport.finishWithdoes the matching steps for a reported request, as plain statements: theonReportlist,countAnomalies,addToHistory. It does them on a secondrequestbuilt by hand around a stand-inresponseWriterthat carries the reported status and bytes.defers in their head, and the same steps are coded two ways.answerQuestionkeeps onedefer, for the log line and the hold for the report.Before (
discriminator.go):After:
deferdoes.One condition is written twice, and two more live elsewhere
refusedWhateverTheAnswer(discriminator.go:150) lists again the conditions of four heuristics: a CrowdSec decision, the header limits, a blocklist withdeny, a whole rate limit.lookUpuses it to skip the wait for GeoJS.Wait:value innewHandlerlists which settings need the lookup's answer.checkTagsandrequestCanGetTagininternal/confighold which setting can give a request which tag, and in what order tags are acted on.heuristics.goon adding a heuristic names none of them.heuristics.gonames the three places.refusedWhateverTheAnswerthen asks the list. The cost is one more interface.Model: fable-5-2
The HTTP surface
Where:
internal/discriminator/discriminator.go,admin.go,proxy.go;internal/rproxy/rproxy.go.Summary:
discriminator's own endpoints are matched by hand in three functions, with the token check written into one of them. Two proposals follow, then what is left alone and why.discriminator's own endpoints are routed by handServeHTTPtells a question from anything else by the presence ofX-Sww-Client. Anything else goes toanswerOwnEndpoint.answerOwnEndpointmatchesPOST /_sww/reportandGET /_sww/healthzwith twoifs, and hands the rest toanswerAdmin.answerAdmincallsendpoint(): aswitchon method and path that returns two things, the token the endpoint needs and a closure that answers it.strings.HasPrefix, andpathAddressremoves the prefix withstrings.TrimPrefix.request, the same type that carries a question's decision.startRequesttakes anisQuestion boolto tell the two uses apart.discriminatorserves.answerAdmin. An endpoint added one function earlier, beside the health endpoint, gets no check and nothing says so.CODE_STYLEGUIDE_GO.mdasks for handlers returned ashttp.HandlerFuncfrom a function, not methods used as handlers.GO_HTTP_SERVER_CONVENTIONS.mdasks for the routes inroutes.goon achirouter, and for middleware of the formfunc(http.Handler) http.Handler.internal/discriminator/routes.goholds one function that lists every own endpoint on achirouter.go-chi/chi/v5is the decided default inGO_PACKAGE_DEFAULTS.md.startRequest,defer rq.finish()) becomes a second middleware around the router.ServeHTTPkeeps its first test. A question never reaches the router, because it carries the visitor's own method and path.Before (
admin.go):After (
routes.go; the function names are placeholders, and new names are sneak's to approve):WWW-Authenticate: Bearer, compared in constant time.chianswers 405 unless its handler for that case is set to answer 404.Both
Servertypes embed*http.Serverdiscriminator.Serverandrproxy.Serverembed*http.Server. Every field and method of Go's server is then part of their own API:Handler,Addr,ListenAndServe,RegisterOnShutdownand the rest.internal/swwuses three:Serve,Shutdown,Close. The tests use a fourth:server.Handler.ServeHTTP.GO_HTTP_SERVER_CONVENTIONS.mdholds Go's server as a private field,httpServer *http.Server, and makesServerthe handler.ServergetsServeHTTPand whatinternal/swwneeds to start and stop it. With the one serve function proposed in the comment on construction, start and stop, that is one method perServer.Left alone
rproxy'sServeHTTPis one linear sequence with nothing to route: make the request, answer the health path, refuse a method, read the body, ask, act. It reads well as it is. No router is proposed for it./_sww/healthzand answersokin plain text. The guides put it at/.well-known/healthcheckwith a JSON body.globalspackage, no Sentry and no CORS. The log level isSWW_LOG_LEVEL, where the guides use aDEBUGvariable.Model: fable-5-2
What
rproxyanddiscriminatorsend each otherWhere:
internal/rproxy/discriminator.go,internal/rproxy/client.go,internal/discriminator/discriminator.go,internal/discriminator/client.go,internal/discriminator/lookup.go,internal/sww/healthcheck.go.Summary: SPEC defines three messages between the two programs: the question, the answer and the report. No package owns them. Each message is written in one package and read in the other, and
rproxyimports the decision package to get their names. One proposal.The messages have no package of their own
internal/discriminator.rproxyimports it for eleven of them:ClientHeader,PeerHeader,ContentLengthHeader,DispositionHeader,DispositionAllow,DispositionDeny,DispositionClose,QuestionBodyMaxBytes,ReportPath,HealthPathand the typeReport.internal/sww/healthcheck.goimports it forHealthPathalone.newQuestioninrproxywrites it.readQuestionandasTheVisitorSentItindiscriminatorread it.allow,denyandsetLookupHeadersindiscriminatorwrite it.noteAnswerinrproxyreads it.finishandpostReportinrproxywrite it.readReportindiscriminatorreads it.requestIDHeader,asnHeader,countryHeaderandownEndpointPrefix.discriminatormakesrproxydepend, through it, on every package the decisions use: the Core Rule Set, the ban ledger, the lookups, the state files.rproxyneeds only the message format, and its imports do not show that.newQuestionwithasTheVisitorSentItdirectly, and those two must mirror each other header for header.internal/owns the three messages. It holds:QuestionBodyMaxBytes,ReportPath,HealthPathand the/_sww/prefix;Report;rproxy,discriminatorandswwimport it.rproxystops importingdiscriminator.MetricsPath,BansPathandClientsPathstay indiscriminator:rproxynever uses them.Before (
internal/rproxy/discriminator.go, theninternal/discriminator/discriminator.go):After, both in the new package (names are placeholders; new names are sneak's to approve):
CODE_STYLEGUIDE_GO.mdforbids. It owns behaviour: how each message is put on the wire and taken off it.peerAddress,isInside,schemeand the tworesponseWritertypes also exist once in each package. Each is a few lines, the twoschemefunctions differ, and a shared package for them would be theutilpackage the guide bans.Model: fable-5-2
internal/config, and values with a fixed set of choicesWhere:
internal/config/config.go, and the packages that use it.Already ordered, and not touched here:
Everything below applies after that pull request lands. Line numbers and setting names are
next's today. That pull request's diff was not read, so sizes and line numbers will differ.configsits on both sides of the partsconfigimports three of the parts for their types:alerts.Events,anomaly.Thresholds,anomaly.NamedNetblock, and the threeremotelog.Scheme…constants.configfor helpers:reputation(TagList,ParseNetblock,ParseASN,ParsePercent,MaskZoneKey),rules(IsHeaderName) andmetrics(MaskZoneKey).Paramsstruct and do not importconfig.configimports can never use aconfighelper, and a part that importsconfigcan never giveconfiga type. The next helper or type lands wherever it compiles.configabove the parts: no part importsconfig. Onlysww,discriminatorandrproxyread it.TagListandMaskZoneKeymove toreputation, which owns tag lists and DNSBL zones.IsHeaderNamemoves torules, or both users callhttpguts.ValidHeaderFieldNamefromgolang.org/x/net, already an indirect dependency.ParseNetblock,ParseASNandParsePercentmove toreputation, the lower of their two users, andconfigimports them from there.configimports nothing internal:Events,Thresholds,NamedNetblockand the scheme constants move intoconfig, and every part that uses them importsconfig. Every part then depends on the settings package, and a part's types sit away from its behaviour, whichCODE_STYLEGUIDE_GO.mdadvises against.configabove the parts. It is the pattern most parts follow already, and it keeps each part testable with plain values.httpguts.ValidHeaderFieldNameaccepts exactly whatIsHeaderNamedoes: both read as the RFC 9110 token, and this was compared by reading only.One setting's code is in six places
config.gois 2,577 lines, ordered by kind of code: the struct, the constants, one block of errors,FromEnvironment, about 60 reader methods, the checks across settings, about 60 parsers.SWW_DNSBL_ZONES:DNSBLZonesat line 139;errNotZoneanderrZoneTooLongat 455 and 458;zonesat 986;parseZonesat 2410.Values with a fixed set of choices are bare strings
stringfields:LookupSource,BytesCount,BlocklistAction,ReputationAction.cfg.LookupSource == "file"atsww/sww.go:204,discriminator/request.go:369anddiscriminator/lookup.go:40;== "off"atdiscriminator/lookup.go:33;== "limit"atdiscriminator/biased.go:60and:65.fileSourceandoffinconfig,denyindiscriminator.bans.CauseLimit), limit kinds (ratelimit.KindBytes), alert events (alerts.EventBan), rule actions (rules.ActionBlock), the log line's actions and offences (requestlog.Action…), the dispositions. The fields and parameters that carry them arestring:ban(…, cause, reason string, …),verdict.action string.routewatch.Missingis a named type, andconfigexportsWAFModeOff,WAFModeDetectandWAFModeBlock.stringparameters can be swapped unnoticed. A reader cannot find the allowed values from the type.config.exhaustiveis already on in.golangci.yml. It then checks everyswitchover these types for a missing case.Before (
discriminator/lookup.go):After (names are placeholders; new names are sneak's to approve):
Setting names are written out again where they are logged
confighold a setting's name. They fall in three groups:discriminator/biased.gohas seven andcountries.go:32one: thesettingfigure of the log line and theLimitPercentSettingof a ban's notes.rules/rules.go:274,state/state.go:188,lookup/file.go:234and:239,sww/sww.go:181,sww/rproxy.go:41,remotelog/remotelog.go:246,discriminator/proxy.go:241, and five inalerts/alerts.go.internal/metrics.configexports a constant for the name of each setting that is written into a log line or a ban's notes.config's own reader anddiscriminatorboth use it.Model: fable-5-2
The smaller packages:
metrics, the watchers,corerulesetand what is repeatedWhere:
internal/metrics,internal/lookup,internal/rules,internal/coreruleset,internal/requestlog,internal/reputation.Summary:
metricsdepends on everything it measures, and is connected by seven calls in a fixed order. Two packages hold the same loop that watches a directory. Adding an app to the Core Rule Set exclusions edits about seven places. Proposals for those three, then shorter notes.metricsdepends on what it measuresmetricsimports eight internal packages. Six of them it needs for the concrete types its sevenAdd…methods take:AddBansAndClients(*bans.Ledger, *ratelimit.Limiter, …),AddRules(*rules.Files),AddRemoteLog(*remotelog.Sender),AddReputation(*reputation.Lists, *reputation.DNSBL),AddAbuseIPDB(*reputation.AbuseIPDB),AddAlerts(*alerts.Queue),AddLookupFile(…).AddRules"once, before RuleMatched",AddAbuseIPDB"once, after AddReputation". Until itsAdd…call, a part's metric fields are nil.lookupgoes the other way.GeoJSis handed the whole*metrics.Metricsand increments three of its public counters. Sometricscannot namelookup.File, andAddLookupFiletakes two closures instead. Its comment says so.statealso calls intometrics, for the state file counts. It goes away under the pending SQLite state store spec (#217).metricshas no test file. Every other package with production code has one.metrics, anAdd…method, and a call at the correct place in another file. A call out of order is a nil pointer when the program is up.metricswith stand-in parts, so nothing tests it alone.metrics:metricsdeclares, beside each group of metrics, the two or three methods it reads. All parts are handed in at once throughmetrics.Params, so there is no call order and no nil field.GeoJSexposes its three counts as methods, as every other part does, andlookupstops importingmetrics.metricsdrops the imports it holds only for a part's type. What remains is the types its event methods take, such asrequestlog.Lineandratelimit.Request. The metric names stay in one file.prometheus.Collector: each part describes its own metrics, andmetricsonly registers them. Every part then imports the Prometheus client, and the metric names spread over a dozen files.metrics.Before (
internal/metrics/metrics.go):After (names are placeholders; new names are sneak's to approve):
CODE_STYLEGUIDE_GO.mdasks for interfaces defined near the functions that use them. Production code has two interfaces today.The loop that watches a directory exists three times
fsnotify.NewWatcher()is called inrules/rules.go,lookup/file.goandstate/state.go. Inrulesandlookup,WatchandreadAfterChangesare the same code line for line: watch the directory, start a timer at once, restart it at each change, read again when it fires, log watcher errors. They differ in the directory, in which file names count, and in the messages. Both declareconst quietTime = 2 * time.Second.GO_PACKAGE_DEFAULTS.mdrecords forfsnotify. It starts its wait at once, to catch an edit made before the watch began. It waits for quiet before reading. A fix to one copy has to be made in the others.readAgainstays in each package, since what is read differs.After (names are placeholders):
stateis left alone. The store spec removesinternal/state, and keepsfsnotifyfor the rule files and the lookup database.Adding an app's exclusions to
corerulesetedits about seven placesinternal/coreruleset/coreruleset.goonnextat463a19e.directivestemplate has one%dand nine%s. Seven of the%sare the per-app exclusions: gitea, WordPress, Drupal, MediaWiki, Mastodon, Vaultwarden, Nextcloud.//go:embedvariable.Newpasses them tofmt.Sprintfby position.directivesmaps each position to a variable in words: "giteaExclusions for the third, wordpressExclusions for the fourth"..conffile, a new variable, a new%s, a new template comment, a new argument in the correct position, a longer mapping comment and the next ordinal. Two arguments in the wrong order compile.embed.FS, joined in name order, and take one%s. Adding an app is then adding its file. The comments name each change by what it does, not by its number.Before:
After (the names are placeholders):
Shorter notes
requestlogholds three things: the two log line types and their writers, the words a decision is logged with (Action…,Offence…), andNewProcessLogger, the process's ownsloglogger. The conventions give the process logger a package of its own. Moving it is a small gain:requestlogwould then be only the request log.anomalyusesratelimit.Bucketsandrequestlog.Lineusesratelimit.Counts. Both reuse a real type. The store spec reshapesratelimit, so nothing is proposed.remotelogimportsrequestlogforFormatTime, one line. Left.rproxyimportsrulesforrules.Path, the path as the visitor sent it.discriminatoruses it three times andrproxytwice. It is a fact about a request, not about rule files, and any other home trades this import for another. Left.lookup.GeoJS,reputation.DNSBL,reputation.AbuseIPDB,routewatch.Client. Each has the same parts: a client, a mutex, a table of answers, a set of questions under way, a time to try again. No shared type is proposed: they differ in what they ask, when they wait and how they count failures, and a generic type would be harder to follow than four plain ones. One difference is worth removing:GeoJScopies nine of its params into fields of its own, where the other three keepparamswhole.reputation.Listskeeps every list in one map by URL. Oneentriesstruct has a field for each shape: netblocks for a blocklist or a tag list, percentages for the AS file, decisions for CrowdSec. It tells which a list is by comparing its URL with the settings (reputation.golines 387, 469, 515). The store spec gives each shape a table of its own, so this is an observation only. Confidence: medium.alerts.Alert.Detailis amap[string]anyread back by key. The store spec already replaces it with a typedalerts.Detail.Model: fable-5-2
Inside
rproxyWhere:
internal/rproxy/rproxy.go,request.go,bodies.go,discriminator.go.Summary:
rproxyreads well from the top:ServeHTTPis one linear sequence. The hard part is the two time limits, which share one type and one file with everything else a request does. Three proposals, largest first, then what is left alone.The time limits have no type of their own
requesthas 25 fields. The last six are a mutex and the five fields it guards:timersStopped,requestTimer,responseTimer,requestEnd,answerStarted.request.go, the last third of the file, work on them:requestDeadline,stopReadingBody,startRequestTimer,requestTimedOut,gotFirstResponseByte,wroteRequest,responseTimedOut,responseReceived,setResponseDeadline,endRequest,stopTimers,stopTimer.time.Nowandtime.AfterFunc.rproxy: which side was being waited on when a limit passed decides between a 408 and a 504. It is methods on the same type as the path rewrite and the log line, so the type does not show which fields other goroutines touch.limits_test.gosleeps for 2.1 seconds in one test.internal/rproxy. It owns the mutex, the two timers and the two times.requestholds one field of it.Before (
request.go):After (names are placeholders; new names are sneak's to approve):
requestTimedOutreads whether the body is waiting for the visitor, and sets a read deadline on the visitor's connection while it holds the lock. The new type has to be given both. That is some connecting code, and the reason for medium confidence.testing/synctest, which the tests of seven other packages use andrproxy's do not.The body wrappers hold the whole request
requestBodyandresponseBodyinbodies.goeach holdrq *request. They are read on other goroutines than the handler's.requestBodyuses the pointer for one call,rq.refuse(413).responseBodyuses it forrq.refuse(502),rq.responseReceived()andrq.in.Context().refuseis safe there becauserefusedis atomic. The next field someone reads fromb.rqmay not be.requestBodygets arefuse func(status int).responseBodygets that, a function to call once the app's whole answer has arrived, and the request's context.Waiting for reports is three fields on
handlerhandlerholdsreports sync.WaitGroup,reportsMuandreportsStopped.sendReportindiscriminator.golocks, tests the flag and starts the goroutine.Server.WaitForReportsinrproxy.goreaches intos.hto lock, set the flag and wait.Before (
rproxy.go):After (names are placeholders):
rproxy.Serverthat returns when its background work has ended.Left alone
ServeHTTPinrproxy.goreads as SPEC's steps 1, 2 and 17 in order. Nothing is proposed for it.allowedMethodsis one string, the text of theAllowheader, andServeHTTPsplits it for each request to test a method (rproxy.go:177). A slice declared once would read more directly. It would need two package variables and a linter suppression for each, which outweighs the gain. No change.forwardbuilds a newhttputil.ReverseProxyfor each request, so that its hooks are methods of that request. It is plain to read, and the transport underneath is shared. No change.panic(http.ErrAbortHandler)for aclosedisposition is the way Go's server documents to drop a connection.newQuestion,noteAnswerandpostReportinrproxy/discriminator.goare covered by the comment on whatrproxyanddiscriminatorsend each other.rproxynever writes to disk and holds no state between requests. Nothing in these proposals changes that.Not read: the bodies of
rewrite,modifyResponseandanswerError(request.golines 200 to 283), andbodies.goafter line 110.Model: fable-5-2
Constructors, file layout, the order of the proposals, and what was not checked
Where: across
internal/.Summary: three smaller points that belong to no one part, then what was looked at and left, how the proposals in the other comments depend on one another, and what this review did not check.
Three constructors take their arguments by position
internal/take oneParamsstruct. Three do not:ratelimit.New(limits Limits, maxClients int)metrics.New(topN int, instanceName string)requestlog.NewProcessLogger(w io.Writer, instanceName string, level slog.Level)CODE_STYLEGUIDE_GO.mdasks for aParamsstruct even for one argument.Params, as the others do. Formetrics.Newthat is part of the proposal in the comment on the smaller packages.ratelimit.New(limits, clients)andbans.New(rules, held)by position. That is the spec's to settle. It is noted so thatratelimit.Newis not changed twice.reputationandlookuphold several types, each with its own constructorreputationholdsLists(New,Params),DNSBL(NewDNSBL,DNSBLParams) andAbuseIPDB(NewAbuseIPDB,AbuseIPDBParams). The three share no state, and one helper of ten lines,raiseFailure.lookupholdsGeoJS(New,Params) andFile(OpenFile,FileParams). The two share theAnswertype.CODE_STYLEGUIDE_GO.mdprefers one type per package with a plainNew(), andNewThing()only now and then.Newis which. That is not reopened here. This finding is the other answer to the same ambiguity, and choosing it would make those two renames unnecessary.reputationonly:DNSBLandAbuseIPDBeach move to a package of about 330 lines, withNewandParams.raiseFailureis written three times, oralertstakes it.lookupstays one package either way: its two types answer the same question and share the answer's type.internal/discriminator's files do not say what they holdproxy.goholdsParams,Server,handler,newHandler,newReputation,NewCoreRuleSetand four of the endpoint paths.discriminatoris not a proxy.Newis indiscriminator.go, with the code that reads a question, writes the answer and takes a report.activeBan, is inbans.gowith the threebanFor…functions.bans.go,biased.go,client.go,lookup.go,reputation.go. Nothing in a file's name tells those from a heuristic's file.heuristics.gois the only index.discriminator.goand does not find it.CODE_STYLEGUIDE_GO.mdasks for struct types near their constructors.Params,Server,New,handlerandnewHandlersit together indiscriminator.go, the file named for the package.discriminator.gofor a file of its own. If the package proposed in the comment on whatrproxyanddiscriminatorsend each other is made, most of it goes there.activeBangets a file of its own, as the other sixteen have.proxy.gogoes away.Looked at and left
Now func() time.Timein itsParams. The real clock is read only to measure a duration, to stamp a line as it is sent, and to wait before trying again (alerts.go:693,remotelog.go:117and:232, and the time limits inrproxy). The split is consistent. No change.panic. Eight constructors callpanic(err)after building an LRU, each with a comment that it fails only for a size below one. Seven sizes are constants. The eighth isSWW_MAX_TRACKED_CLIENTSinratelimit.New, whichconfigreads as a number that is not off. The guide says to avoidpanicin library code. Six of the eight go with the parts' own tables under the store spec. No change now.panic(http.ErrAbortHandler), five times, is how Go's server documents closing a connection without an answer.state.stateFileanddiscriminator.heuristic. The proposals add few: a handful of small ones inmetrics, and one optional one among the heuristics. None is a layer of its own.alerts,lookupandreputationeach have anexport_test.gowithSetTransportorSetDial, so that tests can stand in for the network. Anhttp.RoundTripperin eachParamswould do the same without the hook. It would also add a field production code never sets, so it is left.//nolint:tagliatellelines in production code sit on JSON field names in snake_case. All of this project's JSON is snake_case, so the setting is what differs. That is a change to the canonical config inprompts, not to this repo.rproxy/discriminator.go:72. A comment that says what the code does not do yet, in words, outlives the tracker.How the proposals depend on one another
chiroutes fordiscriminator's own endpoints;rproxy;internal/discriminator.metrics.Paramswith interfaces, and fx. fx with providers ininternal/swwreplaces the hand-written version of it.decision, then the status on the verdict. The one ban function and the plain statements after the answer are independent of those.internal/configwaits for the settings change through viper (#219) and for the issue that gives each daemon's settings one file (#227).bans.Ledger,ratelimitorreputationis easier after the store spec is built, since it rewrites the same constructors.bans.Ledgerand should be decided with it.What was not checked
3b987b9.nextmoved to463a19eduring the work, and onlyinternal/coreruleset/coreruleset.gowas read again there.docs/;SPEC.mdoutside "Architecture", "What happens to a request" and "Planned, not built";.golangci.ymlpast its first 60 lines;rules,anomaly,remotelog,routewatch,reputationandalerts, where only the types, constructors and exported functions were read;config.gobetween its declarations and its helpers, beyond the one setting traced as an example;rproxy, the bodies ofrewrite,modifyResponseandanswerError, andbodies.goafter line 110.httpguts.ValidHeaderFieldNameaccepts exactly whatconfig.IsHeaderNamedoes;testing/synctest;BanForAdminandWouldBePermanentfit the one ban method.chiandcobraare recommended on the strength of the two guides andGO_PACKAGE_DEFAULTS.md. No ruling on record applies them to this repo.internal/stateor how a part stores what it knows. Where a proposal meets one of the four open decisions, the comment says so.Model: fable-5-2
i was under the impression that rproxy sends the waf the request from the client nearly as-is (perhaps body truncated at max 1MB or something, as more than that isn't needed for deciding), with just a few headers added for the actual connecting ip and potentially a tls fingerprint later.
rproxy should drop connections when headers are >32KB, unconfigurable. nuanced decisions are for the waf, rproxy just needs basic sanity hardcoded in.
immediately begin the fx work on a branch using the recommended config. let's land it on next asap.
Filed from the two comments above: the fx wiring as recommended, #414, and the hard-coded 32 KB header limit in
rproxywith a check of what it forwards towaf, #415. Workers start on both as soon as the machine's load allows.Model: opus-5-5