Merge pull request 'Milestones 1 and 2: proxy with timeouts and size limits, rate limits, country lists and the image' (#40) from next into main

Reviewed-on: #40

docs/gitea.md

@@ -1,198 +0,0 @@
-# smallwebwaf in front of gitea
-
-These notes record what `smallwebwaf` does in front of gitea at the default
-settings. They are observations, not requirements; the requirements are in
-[`SPEC.md`](../SPEC.md). `script/gitea-requests` runs gitea behind the image and
-sends through it a sample of gitea's ordinary use and the requests named below.
-It reports each request the Core Rule Set refuses, so that these notes can be
-checked again when gitea or the Core Rule Set changes version.
-
-## What the defaults cover
-
-- SSH access to gitea does not pass through `smallwebwaf` and is not protected
-  by it.
-- A clone is a response. It fits the defaults of 30 minutes and 5 GiB for all
-  but the largest repositories and slowest links.
-- A push is a request, and so is every other upload: a git push, an LFS object,
-  a container image layer, a package file or a release attachment. At the
-  defaults, an upload larger than 100 MiB or taking more than 60 seconds is cut
-  off. The client is answered `413` for a body that is too large, before
-  anything reaches gitea when the request announces its size. It is answered
-  `408` for a body it sends too slowly, or `504` if gitea is too slow to take
-  it. The upload fails, and no one is banned for it. A gitea that takes large
-  uploads needs `SWWAF_REQUEST_MAX_BYTES`, `SWWAF_CLIENT_REQUEST_TIMEOUT` and
-  `SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised to fit. The Core Rule Set does not
-  read an upload's body, which streams through without being held in memory.
-- Archive download and blame or history pages are what scrapers hammer. The
-  request limits do most of the work there.
-- An Actions runner that reaches gitea through `smallwebwaf` sends requests all
-  day. One older than version 0.4 (April 2026) asks for work every 2 seconds,
-  43,200 requests a day, and reports a running job's log and state every second.
-  So it passes the day limit of 50,000 after about an hour and a quarter of
-  jobs. It is then banned, the job it is running fails, and each repeat within a
-  day makes the ban longer. Newer runners ask less often, but a busy one, or
-  several on one address, can still pass it. Put the runners' addresses in
-  `SWWAF_RATE_LIMIT_EXEMPT_NETS`. That takes them out of the request and byte
-  limits, while the error burst, attack detection and bans still apply.
-
-## What the Core Rule Set lets through
-
-At the defaults, the Core Rule Set lets gitea's ordinary use through, apart from
-the refusals in the next section:
-
-- browsing and views of files in a repository, with their history, blame and the
-  file tree;
-- diffs, including their hidden lines and large files, and pull request review;
-- git's clone, fetch and push over HTTP;
-- signing in, including the return to the page a visitor came from, and sign-in
-  with Git Credential Manager, git-credential-oauth or tea;
-- the API's calls for a file and its commits;
-- pushing and pulling container images and packages;
-- Actions runners, and artifacts uploaded with `actions/upload-artifact@v4`;
-- posting issues, pull requests, comments, wiki pages and files saved in the web
-  editor, code included, since no body is read;
-- the page gitea shows after a change, with its message naming what was done.
-
-## What the Core Rule Set still refuses
-
-Each refusal answers only that request, with `403`, and bans no one by itself.
-It counts toward the error burst, which a person does not reach this way. The
-request log names the rule in `waf_rule_ids`, which `SWWAF_WAF_DISABLED_RULES`
-can switch off.
-
-### A query string that reads as an attack
-
-Most often this is a search. Every query parameter is checked against the Core
-Rule Set's lists of system files and commands, except the ten named under
-"Attack detection" in `SPEC.md`. A release attachment uploaded through the API
-as `docker-compose.yml` is refused the same way. Examples:
-
-- a name on those lists, such as `package.json`, `.gitignore` or
-  `docker-compose.yml`;
-- text that starts with a command name, or whose first word has one right after
-  a `/`, such as `python3`, `ssh key` or `feat/docker-support`;
-- a shell command with its options or a system path (`ls -la`, `sed -i`,
-  `/bin/sh`), or a command in backticks;
-- script code (`fetch(`, `${VAR}`, `process.env`), HTML (`<img src=`) or an SQL
-  statement (`SELECT * FROM users WHERE`);
-- a URL naming an IP address or `localhost`.
-
-### A branch, tag or file name that starts with a command name
-
-The Core Rule Set's list of commands (932260) holds names such as `docker-`,
-`python3`, `ansible`, `base64` and `whoami`. A branch, tag or file name that
-starts with one is refused. So is one that has such a name right after a `/`,
-such as `feat/docker-support`, `fix/docker-build` or
-`renovate/docker-build-push-action-6.x`. gitea's own pages send the name in a
-query parameter that 930120, 932160 and 932260 still check:
-
-- `refSubUrl`, the branch or tag of a directory listing. It is sent when the
-  listing asks for its entries' last commits in a second request, which it does
-  whenever gitea takes more than a second to work them out;
-- `name`, when a branch is deleted or restored on the branches page;
-- `tag`, when a release is started from a tag;
-- `template`, the file of an issue template, when an issue is opened from it;
-- `rule_name`, when a branch protection rule is opened for editing in the
-  repository's settings or has just been saved.
-
-What a visitor sees, for a branch named `docker-build`:
-
-- A directory listing that makes that second request leaves those last commits
-  out and shows an error.
-- The branches page shows an error instead of deleting or restoring the branch.
-- A release started from such a tag gets the `403` answer of `smallwebwaf` in
-  place of its form. So does an issue opened from such a template, and such a
-  rule opened for editing.
-- Saving such a rule stores it. gitea then returns to the branch settings page,
-  which gets the `403` answer, so the save looks as if it failed.
-
-### A file name in a query parameter that trips the code rules
-
-The Core Rule Set's rules for Java, script, PowerShell and SQL find what they
-look for in some file names. This holds in the parameters `smallwebwaf` leaves
-out of 930120, 932160 and 932260 too (see the changes to the Core Rule Set under
-"Attack detection" in `SPEC.md`). Examples:
-
-- `runtime` in OpenJDK's `src/java.base/share/classes/java/lang/Runtime.java`
-  (944110);
-- a Java class, as in `java.lang.String.md` (944130);
-- `document.write.js` (941180);
-- `process.env.d.ts` (934100);
-- `__proto__` (934130);
-- the PowerShell command `Install-Script`, as in `install-script.sh` (932120);
-- a database's name, as in PostgreSQL's `information_schema.sql` (942140).
-
-For such a file, these requests are refused:
-
-- the API's list of its commits (`path`);
-- the API's call for the contents of files that names it in its query (`body`);
-- the commit graph of that file alone (`file`);
-- the hidden lines of its diff (`path`), which stay hidden;
-- the sign-in page gitea sends a visitor to on the way to a page of the file
-  that needs signing in, such as its edit page (`redirect_to`). The visitor can
-  still sign in by opening the sign-in page itself.
-
-### A path
-
-- A file name ending in `~`, or an `.xhtml` file whose path holds a space.
-- A path that holds what the Core Rule Set takes for a Java class (944130), as
-  `java.lang.String.md` and
-  `META-INF/services/org.apache.commons.logging.LogFactory` do.
-- A path that holds what it takes for Node.js code (934100), as
-  `process.env.d.ts` does.
-- A path with a directory named `...` (930100, 930110), such as `.../notes.txt`.
-
-For such a file, the page of the file, its raw file, its history, its blame and
-its edit page are refused. The directory that holds it lists it, and a clone
-carries it.
-
-### A container image push that reuses a layer
-
-A push can reuse a layer of an image docker has pushed to that gitea before.
-docker then asks gitea to take the layer from that image, naming it in the query
-parameter `from`, which 932260 checks. The request is refused when the image's
-owner, or its name after the owner, starts with an entry on the list of
-commands. Examples are `owner/python3-app`, `owner/docker-base`, and any image
-of an owner named `ansible` or `docker-team`.
-
-What happens next depends on docker's image store. A docker that keeps its
-images in containerd's image store fails the push, and fails each new try the
-same way. A new install of Docker Engine 29 does this. A docker that keeps them
-in its own older image store uploads the layer instead. The push then succeeds
-after one refusal for each layer it reuses, and each of those refusals counts
-toward the error burst.
-
-### Artifact uploads from `actions/upload-artifact@v3`
-
-This action sends the header `Content-Range`, which the Core Rule Set refuses
-(920450). The action sends two files at a time, does not retry a `403`, and
-sends no further file once one is refused. So an upload is refused once or
-twice, however many files it holds, and the step fails. One upload does not ban
-the runner. More than 30 such refusals from its address within a minute, as when
-many jobs upload at once, ban it through the error burst.
-`actions/upload-artifact@v4` does not send the header.
-
-## What `script/gitea-requests` sends
-
-Of the requests above, the script sends:
-
-- for each file named under "A file name in a query parameter that trips the
-  code rules", the API's list of its commits;
-- for `Runtime.java` alone, the commit graph of that file, the hidden lines of
-  its diff, and the API's call for the contents of files that names it;
-- the sign-in page gitea sends a visitor to on the way to the edit page of
-  `document.write.js`;
-- the pages of `java.lang.String.md`,
-  `META-INF/services/org.apache.commons.logging.LogFactory`, `process.env.d.ts`
-  and `.../notes.txt`;
-- one container image push that reuses a layer, which it does not try again.
-
-It does not send the raw file, history, blame or edit page of those four files,
-or the pages of the directories `META-INF/services` and `...`. It also sends a
-short sample of gitea's ordinary use:
-
-- signing in;
-- pages of files and commits, and a diff;
-- the API;
-- a push, and clone and fetch over HTTP;
-- two container image pushes.

internal/anomaly/anomaly.go

@@ -1,8 +1,8 @@
 // Package anomaly counts requests and bytes over a minute and an hour, per
 // client, per surrounding netblock, per AS number, for the whole service
 // and per named netblock, and raises an anomaly alert for a count over its
-// threshold, as "Anomaly thresholds" under "Settings" in SPEC.md
-// describes. It refuses and bans nothing. At most 20,000 counters
+// threshold, as "Anomaly thresholds" under "Configuration surface" in
+// SPEC.md describes. It refuses and bans nothing. At most 20,000 counters
 // are kept, in memory, and written to alerts.json and read from it by the
 // state package.
 package anomaly

internal/waf/waf.go

@@ -1,8 +1,8 @@
 // Package waf runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
 // method, the URL with its query and the headers of a request, and on its
 // body while SWWAF_WAF_BODY_LIMIT is set, with the six changes smallwebwaf
-// makes to it, as "Attack detection" under "Settings" in SPEC.md describes
-// them. It reads no response.
+// makes to it, as "Attack detection" under "Configuration surface" in
+// SPEC.md describes them. It reads no response.
 //
 // smallwebwaf writes only to its state directory, so Coraza is built with
 // its no_fs_access tag, as the Dockerfile and script/build build it: of a

script/gitea-requests

@@ -2,12 +2,11 @@
 # script/gitea-requests: build the image and run gitea 1.27.3 behind it,
 # with every setting of smallwebwaf at its default but SWWAF_UPSTREAM_URL,
 # which names gitea's container. Then send through it the gitea requests
-# that reading gitea's source found the Core Rule Set refuses, which
-# docs/gitea.md records, and a short sample of gitea's ordinary use:
-# signing in, pages of files and commits, a diff, the API, a push, clone
-# and fetch over HTTP, and two container image pushes. Last, print each
-# request smallwebwaf logged, with what it did with it and the ids of the
-# Core Rule Set's rules that matched it.
+# that reading gitea's source found the Core Rule Set refuses, and a short
+# sample of gitea's ordinary use: signing in, pages of files and commits,
+# a diff, the API, a push, clone and fetch over HTTP, and two container
+# image pushes. Last, print each request smallwebwaf logged, with what it
+# did with it and the ids of the Core Rule Set's rules that matched it.
 # Every request comes from one client, docker's gateway, so more than
 # SWWAF_ERROR_BURST_THRESHOLD, 30, refusals in a minute would ban it, and
 # the requests after would show as banned. The containers, the network,