script/gitea-requests, and the gitea refusals it confirms named in SPEC.md (closes #30)

script/gitea-requests runs gitea 1.27.3 behind the image at its
defaults, sends through it the requests that reading gitea's source
found the Core Rule Set refuses, with a short sample of ordinary use,
and prints each request with the rule ids that matched it. make
gitea-requests runs it; make check does not.

Every case still holds, and each fix would let through what SPEC.md
says stays refused, so the notes specific to gitea now name them with
how each refusal plays out: file names in query parameters that read as
Java, script, PowerShell or SQL, such paths, and an image push that
reuses a layer of an image named like a command.

Model: opus-5-5

Makefile

@@ -1,11 +1,12 @@
 .PHONY: bootstrap setup test lint fmt fmt-check tidy check docker hooks build \
-	run example-app
+	run example-app gitea-requests
 
 # Makefile targets are thin shims; the implementations live in script/
 # per the scripts-to-rule-them-all pattern (see the Entrypoints section
 # of README.md). tidy writes go.mod and go.sum as `go mod tidy` does,
 # which test checks. build and run are for working on the code by hand;
-# example-app checks the image with an app built on it.
+# example-app checks the image with an app built on it; gitea-requests
+# reports the gitea requests the Core Rule Set refuses at the defaults.
 
 bootstrap:
 	@script/bootstrap
@@ -45,3 +46,6 @@ run:
 
 example-app:
 	@script/example-app
+
+gitea-requests:
+	@script/gitea-requests

README.md

@@ -2374,6 +2374,19 @@ so that they run in minimal containers.
   the same volume still refuses the banned client; then removes the containers,
   the volume and both images. It needs network access, for nixpkgs' binary
   cache, and `script/check` does not run it; `make example-app` does.
+- `script/gitea-requests`: builds the image and runs gitea 1.27.3 behind it,
+  with every setting of `smallwebwaf` at its default but `SWWAF_UPSTREAM_URL`,
+  which names gitea's container. It sends through it the gitea requests that the
+  notes specific to gitea in [`SPEC.md`](SPEC.md) name for a file name in a
+  query parameter, for a path and for a container image push, and a short sample
+  of gitea's ordinary use: signing in, pages of files and commits, a diff, the
+  API, a push, clone and fetch over HTTP, and two container image pushes. It
+  prints a line for each request `smallwebwaf` logged, with what it did with it
+  and the ids of the Core Rule Set's rules that matched it, then removes the
+  containers, the network and the images it made, but gitea's own image. It
+  needs network access, and a docker daemon that pushes to a registry on
+  `localhost` over plain HTTP, as docker does by default; `script/check` does
+  not run it, and `make gitea-requests` does.
 
 ## TODO
 

SPEC.md

@@ -1477,8 +1477,42 @@ holds any token file.
           Saving such a rule stores it, but the branch settings page gitea then
           returns to gets the 403 answer of `smallwebwaf`, so the save looks as
           if it failed.
+        - A file name in a query parameter, those the fourth change leaves out
+          of 930120, 932160 and 932260 included, in which the Core Rule Set's
+          rules for Java, script, PowerShell and SQL find what they look for:
+          `runtime` in OpenJDK's
+          `src/java.base/share/classes/java/lang/Runtime.java` (944110), a Java
+          class, as in `java.lang.String.md` (944130), `document.write.js`
+          (941180), `process.env.d.ts` (934100), `__proto__` (934130), the
+          PowerShell command `Install-Script`, as in `install-script.sh`
+          (932120), or a database's name, as in PostgreSQL's
+          `information_schema.sql` (942140). For such a file, the API's list of
+          its commits (`path`) is refused, and so are the API's call for the
+          contents of files that names it in its query (`body`), the commit
+          graph of that file alone (`file`), and the hidden lines of its diff
+          (`path`), which stay hidden. A visitor whom gitea sends to sign in on
+          the way to a page of such a file that needs signing in, such as its
+          edit page, gets the 403 answer of `smallwebwaf` in place of the
+          sign-in page (`redirect_to`), and can still sign in by opening the
+          sign-in page itself.
         - A path: a file name ending in `~`, or an `.xhtml` file whose path
           holds a space.
+        - A path that holds what the Core Rule Set takes for a Java class
+          (944130), as `java.lang.String.md` and
+          `META-INF/services/org.apache.commons.logging.LogFactory` do, or for
+          Node.js code (934100), as `process.env.d.ts` does, or that has a
+          directory named `...` (930100, 930110), such as `.../notes.txt`. The
+          page of such a file, its raw file, its history, its blame and its edit
+          page are refused; the directory that holds it lists it, and a clone
+          carries it.
+        - A container image push that reuses a layer of an image docker has
+          pushed to that gitea before, when that image's name, after its owner,
+          starts with an entry on the list of commands (932260), such as
+          `owner/python3-app` or `owner/docker-base`. docker asks gitea to take
+          the layer from that image, naming it in the query parameter `from`,
+          which 932260 checks. The push fails with the 403 answer of
+          `smallwebwaf`, and so does each new try, since docker asks in the same
+          way.
     - Artifact uploads from `actions/upload-artifact@v3` send the header
       `Content-Range`, which the Core Rule Set refuses (920450). The action
       sends two files at a time, does not retry a 403 and sends no further file
@@ -1566,10 +1600,12 @@ holds any token file.
 - Core Rule Set false positives against real apps (a search for code, and any
   body once `SWWAF_WAF_BODY_LIMIT` is set): a match refuses only that request
   and bans no one by itself; the request log names the rule, and exclusions by
-  rule id and path fix it. Further gitea requests the Core Rule Set may refuse
-  at the defaults, found by reading gitea's source rather than a running gitea,
-  are collected in https://git.eeqj.de/sneak/smallwebwaf/issues/30 and checked
-  when milestone 1 runs in front of a real gitea.
+  rule id and path fix it. `script/gitea-requests` runs gitea behind the image
+  at the defaults, sends through it the requests the notes specific to gitea
+  name for a file name in a query parameter, for a path and for a container
+  image push, and a short sample of gitea's ordinary use, and reports each
+  request the Core Rule Set refuses, so that those notes can be checked again
+  when gitea or the Core Rule Set changes version.
 - Attacks carried in request bodies: not refused by default, since on a code
   forge bodies are full of code the Core Rule Set takes for attacks. Most of
   what shows in URLs and headers is still refused, the client that sends an

internal/proxy/lookup_test.go

@@ -86,6 +86,75 @@ func TestEveryClientIsLookedUpWithoutWaitingWhileNoSettingNeedsIt(t *testing.T)
 	}
 }
 
+func TestRequestThatWaitedForItsLookupIsCountedWhenItGoesOn(t *testing.T) {
+	t.Parallel()
+
+	// The country list has a new client's request wait for its answer,
+	// which the stand-in for GeoJS gives only once released.
+	geojsURL, asked, release := startHeldGeoJS(t)
+	app := startApp(t, func(http.ResponseWriter, *http.Request) {})
+	clk := &clock{now: time.Date(2026, 10, 6, 0, 0, 58, 0, time.UTC)}
+	addr, out, server := startProxyWithClock(t, app.URL, geojsURL, clk.Now,
+		map[string]string{
+			trustedProxies:  trustLocalhost,
+			lookupTimeout:   "1h",
+			deniedCountries: "kp",
+		})
+
+	requestFromDE := func() *http.Request {
+		req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
+		req.Header.Set(forwardedFor, fromDE)
+
+		return req
+	}
+
+	// fromDE's first request arrives two seconds before a minute ends,
+	// and waits for GeoJS.
+	first, httpClient := requestFromDE(), newClient(t)
+	firstEnded := make(chan error, 1)
+
+	go func() {
+		res, err := httpClient.Do(first)
+		if err == nil {
+			err = readAnswer(res).err
+		}
+
+		firstEnded <- err
+	}()
+
+	waitUntil(func() bool { return len(asked()) == 1 })
+
+	// Half a second into the next minute, the answer is kept, as an edit
+	// of lookups.json would keep it, while the first request still waits
+	// for GeoJS. The second request goes on at once, and is counted.
+	clk.advance(2500 * time.Millisecond)
+	server.GeoJS.Load([]lookup.Answer{{
+		Client: netip.MustParsePrefix(fromDE + "/32"), ASN: asnDE, ASName: asNameDE,
+		Country: "DE", Answered: clk.Now(), Used: clk.Now(),
+	}})
+	wantStatus(t, do(t, requestFromDE()), http.StatusOK)
+
+	// GeoJS answers, and the first request goes on.
+	release()
+
+	err := <-firstEnded
+	if err != nil {
+		t.Fatalf("first request: %v", err)
+	}
+
+	wantLine(t, out.requestLines(t, 2)[1], http.StatusOK, requestlog.ActionForward)
+
+	// The third request's count holds all three: the first was counted
+	// when it went on, in the minute under way, and the second's count
+	// was kept.
+	clk.advance(500 * time.Millisecond)
+	wantStatus(t, do(t, requestFromDE()), http.StatusOK)
+
+	if got := out.requestLines(t, 3)[2].Counts.Minute; got != 3 {
+		t.Errorf("the third request's count in the minute is %v, want 3", got)
+	}
+}
+
 func TestASNumberAndNameInTheLogLineTheHistoryTheBanNotesAndTheAlert(t *testing.T) {
 	t.Parallel()
 

internal/proxy/request.go

@@ -261,6 +261,12 @@ func (rq *request) checkClient(ctx context.Context) string {
 
 	rq.lookUp(ctx)
 
+	// The lookup can wait for GeoJS, so the checks after it read the time
+	// again: a request counted at the time it arrived could be counted
+	// after a later one of its client's, which the rate limits would take
+	// for the clock set back.
+	now = rq.h.now()
+
 	if rq.countryDenied() {
 		return requestlog.ActionCountryDenied
 	}

script/gitea-requests

@@ -0,0 +1,329 @@
+#!/bin/sh
+# script/gitea-requests: build the image and run gitea 1.27.3 behind it,
+# with every setting of smallwebwaf at its default but SWWAF_UPSTREAM_URL,
+# which names gitea's container. Then send through it the gitea requests
+# that reading gitea's source found the Core Rule Set refuses, and a short
+# sample of gitea's ordinary use: signing in, pages of files and commits,
+# a diff, the API, a push, clone and fetch over HTTP, and two container
+# image pushes. Last, print each request smallwebwaf logged, with what it
+# did with it and the ids of the Core Rule Set's rules that matched it.
+# Every request comes from one client, docker's gateway, so more than
+# SWWAF_ERROR_BURST_THRESHOLD, 30, refusals in a minute would ban it, and
+# the requests after would show as banned. The containers, the network, the images and the files the script makes
+# are removed however it ends; gitea's own image stays, as the base images
+# of a build do. It needs network access, and a docker daemon that pushes
+# to a registry on localhost over plain HTTP, as docker does by default.
+# script/check does not run this.
+set -eu
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
+
+# gitea 1.27.3, 2026-08-29
+GITEA_IMAGE="gitea/gitea@sha256:87a67ee09d3ae0d1df5fda5dcda3e2a1f9236a45b0a59025d6e00e46adc43bef"
+
+# Named after this run, so that runs in other clones on the same host
+# never touch each other's.
+NAME="$("$SCRIPT_DIR/projectname")-gitea-$$"
+IMAGE="$NAME-image"
+NETWORK="$NAME"
+SMALLWEBWAF="$NAME-smallwebwaf"
+GITEA="$NAME-gitea"
+
+# gitea's one user, who owns the repository and the container images.
+PASSWORD="owner-password"
+
+BROWSER="Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
+
+# The files the script makes, and smallwebwaf's address once it runs,
+# localhost and the port docker publishes it on: a host name, since the
+# Core Rule Set takes a Host that is an IP address for a warning (920350).
+WORK=""
+address=""
+
+cleanup() {
+    docker rm --force --volumes "$SMALLWEBWAF" "$GITEA" >/dev/null 2>&1 || true
+    docker network rm "$NETWORK" >/dev/null 2>&1 || true
+    docker rmi --force "$IMAGE" >/dev/null 2>&1 || true
+    if [ -n "$address" ]; then
+        docker rmi --force "$address/owner/python3-app:1" "$address/owner/app:1" \
+            >/dev/null 2>&1 || true
+    fi
+    rm -rf "$WORK"
+}
+
+fail() {
+    echo "gitea-requests: $*" >&2
+    report
+    exit 1
+}
+
+# wait_for <what fails> <command>...: run the command every second until
+# it succeeds, for at most a minute.
+wait_for() {
+    failure="$1"
+    shift
+    tries=0
+    until "$@"; do
+        tries=$((tries + 1))
+        [ "$tries" -lt 60 ] || fail "$failure"
+        sleep 1
+    done
+}
+
+healthy() {
+    status="$(docker inspect --format '{{.State.Health.Status}}' "$SMALLWEBWAF")"
+    [ "$status" = healthy ]
+}
+
+# logged <text>...: a line of smallwebwaf's output holds every text, in
+# any order.
+logged() {
+    lines="$(docker logs "$SMALLWEBWAF" 2>&1)"
+    for text in "$@"; do
+        lines="$(printf '%s\n' "$lines" | grep -F "$text")" || return 1
+    done
+}
+
+# send <label> <path> <curl option>...: send a request for path to
+# smallwebwaf, with label as its X-Request-ID, which the report gives,
+# and keep the body of the answer in $WORK/answer. The Core Rule Set
+# reads that header too, so a label names no file the cases name.
+send() {
+    label="$1"
+    path="$2"
+    shift 2
+    curl --silent --show-error --max-time 30 --output "$WORK/answer" \
+        --header "X-Request-ID: $label" "$@" "http://$address$path"
+}
+
+# page <label> <path> <curl option>...: send the request as a browser
+# does, with the cookies gitea has set. Its Referer is the repository's
+# page, which links to the others, and never a page the Core Rule Set
+# refused: its address would have the next request refused too.
+page() {
+    label="$1"
+    path="$2"
+    shift 2
+    send "$label" "$path" --compressed --user-agent "$BROWSER" \
+        --header "Accept: text/html,application/xhtml+xml,*/*;q=0.8" \
+        --header "Accept-Language: en-US,en;q=0.5" \
+        --cookie "$WORK/cookies" --cookie-jar "$WORK/cookies" \
+        --referer "http://$address/owner/repo" "$@"
+}
+
+# api <label> <path> <curl option>...: send the request as a client of
+# gitea's API does, as the owner.
+api() {
+    label="$1"
+    path="$2"
+    shift 2
+    send "$label" "$path" --user "owner:$PASSWORD" \
+        --header "Accept: application/json" "$@"
+}
+
+# repo_git <git argument>...: run git as the owner, without the host's
+# own git settings, so that nothing it does is kept outside $WORK.
+repo_git() {
+    GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 git \
+        -c user.name=owner -c user.email=owner@example.invalid "$@"
+}
+
+# lines: forty lines of text, enough for a diff to hide some of them.
+lines() {
+    i=1
+    while [ "$i" -le 40 ]; do
+        echo "line $i"
+        i=$((i + 1))
+    done
+}
+
+# make_repository: create the repository owner/repo, holding a file of
+# each name the cases name, and push it, which is the push over HTTP of
+# the sample. The second commit adds a line at the end of Runtime.java,
+# so that its diff hides the lines above.
+make_repository() {
+    status="$(api create-repository /api/v1/user/repos \
+        --header "Content-Type: application/json" --data '{"name":"repo"}' \
+        --write-out '%{http_code}')"
+    [ "$status" = 201 ] || fail "creating the repository was answered $status"
+
+    dir="$WORK/repo"
+    repo_git init --quiet --initial-branch=main "$dir"
+    for file in \
+        src/java.base/share/classes/java/lang/Runtime.java \
+        java.lang.String.md \
+        META-INF/services/org.apache.commons.logging.LogFactory \
+        document.write.js \
+        process.env.d.ts \
+        test/fixtures/__proto__.json \
+        install-script.sh \
+        src/backend/catalog/information_schema.sql \
+        .../notes.txt \
+        README.md; do
+        mkdir -p "$dir/$(dirname "$file")"
+        lines >"$dir/$file"
+        repo_git -C "$dir" add "$file"
+    done
+    repo_git -C "$dir" commit --quiet --message "Add the files"
+    echo "line 41" >>"$dir/src/java.base/share/classes/java/lang/Runtime.java"
+    repo_git -C "$dir" commit --quiet --all --message "Add a line"
+
+    repo_git -C "$dir" -c "http.extraHeader=X-Request-ID: git-push" push --quiet \
+        "http://owner:$PASSWORD@$address/owner/repo.git" main ||
+        fail "the push of the repository failed"
+}
+
+# browse: sign in, then open pages as a person does. The cases are those
+# of https://git.eeqj.de/sneak/smallwebwaf/issues/30.
+browse() {
+    # Case: gitea sends a visitor who is not signed in to the sign-in
+    # page, with the page asked for in redirect_to, when that page needs
+    # signing in, as a file's edit page does.
+    page issue-30-case /owner/repo/_edit/main/document.write.js --location
+
+    page sign-in /user/login
+    status="$(page sign-in /user/login --header "Origin: http://$address" \
+        --data-urlencode user_name=owner --data-urlencode "password=$PASSWORD" \
+        --write-out '%{http_code}')"
+    [ "$status" = 303 ] || fail "signing in was answered $status"
+
+    page browse /
+    page browse /owner/repo
+    page browse /owner/repo/src/branch/main/README.md
+    page browse /owner/repo/commits/branch/main
+    page browse "/owner/repo/commit/$(repo_git -C "$WORK/repo" rev-parse HEAD)"
+
+    # Case: expanding the lines the diff hides, by the link the diff's
+    # page gives for it.
+    excerpt="$(sed -n 's/.*data-fetch-url="\([^"]*blob_excerpt[^"]*\)".*/\1/p' \
+        "$WORK/answer" | head -n 1 | sed 's/&amp;/\&/g')"
+    [ -n "$excerpt" ] || fail "the diff's page has no link to its hidden lines"
+    page issue-30-case "$excerpt"
+
+    # Cases: the commit graph of one file, and the pages of files whose
+    # path the Core Rule Set takes for an attack.
+    page issue-30-case "/owner/repo/graph?file=src/java.base/share/classes/java/lang/Runtime.java"
+    page issue-30-case /owner/repo/src/branch/main/META-INF/services/org.apache.commons.logging.LogFactory
+    page issue-30-case /owner/repo/src/branch/main/java.lang.String.md
+    page issue-30-case /owner/repo/src/branch/main/process.env.d.ts
+    page issue-30-case /owner/repo/src/branch/main/.../notes.txt
+}
+
+# call_api: the API's calls the cases name: the commits of a file, for a
+# file of each name the Core Rule Set takes for an attack in the query
+# parameter path, and the file-contents call, which takes its list of
+# files as JSON in the query parameter body.
+call_api() {
+    for file in \
+        src/java.base/share/classes/java/lang/Runtime.java \
+        java.lang.String.md \
+        document.write.js \
+        process.env.d.ts \
+        test/fixtures/__proto__.json \
+        install-script.sh \
+        src/backend/catalog/information_schema.sql; do
+        api issue-30-case /api/v1/repos/owner/repo/commits --get --data-urlencode "path=$file"
+    done
+    api issue-30-case /api/v1/repos/owner/repo/file-contents --get \
+        --data-urlencode 'body={"files":["src/java.base/share/classes/java/lang/Runtime.java"]}'
+}
+
+# push_images: push a container image to owner/python3-app, then the same
+# image to owner/app. For the second, the case, docker asks gitea to take
+# its layer from owner/python3-app, naming it in the query parameter
+# from. docker sends no X-Request-ID, so its requests have the ids
+# smallwebwaf gives them. The image is one small file, imported rather
+# than built.
+push_images() {
+    echo "$NAME" >"$WORK/layer.txt"
+    tar -cf - -C "$WORK" layer.txt |
+        docker import - "$address/owner/python3-app:1" >/dev/null
+    docker tag "$address/owner/python3-app:1" "$address/owner/app:1"
+
+    echo "$PASSWORD" | DOCKER_CONFIG="$WORK/docker" docker login --username owner \
+        --password-stdin "$address" >/dev/null 2>&1 || fail "docker login failed"
+    for image in "$address/owner/python3-app:1" "$address/owner/app:1"; do
+        DOCKER_CONFIG="$WORK/docker" docker push --quiet "$image" >/dev/null ||
+            echo "gitea-requests: the push of $image failed" >&2
+    done
+}
+
+# clone_and_fetch: the clone and fetch over HTTP of the sample.
+clone_and_fetch() {
+    repo_git -c "http.extraHeader=X-Request-ID: git-clone" clone --quiet \
+        "http://owner:$PASSWORD@$address/owner/repo.git" "$WORK/clone" ||
+        fail "the clone failed"
+    repo_git -C "$WORK/clone" -c "http.extraHeader=X-Request-ID: git-fetch" fetch --quiet ||
+        fail "the fetch failed"
+}
+
+# field <name> <line>: the text of the field name in line, a line of the
+# request log, or nothing when it has none.
+field() {
+    printf '%s\n' "$2" | sed -n "s/.*\"$1\":\"\([^\"]*\)\".*/\1/p"
+}
+
+# report: print a line for each request smallwebwaf logged, but its
+# health checks: what it did with the request, the ids of the Core Rule
+# Set's rules that matched it, its X-Request-ID, and the request.
+# waf_blocked is a refusal by the Core Rule Set, and forward a request
+# passed on to gitea, whatever gitea answered.
+report() {
+    docker logs "$SMALLWEBWAF" 2>/dev/null | grep '"type":"request"' |
+        grep -v '"action":"admin"' | sed 's/\\u0026/\&/g' >"$WORK/requests"
+    while IFS= read -r line; do
+        ids="$(printf '%s\n' "$line" | sed -n 's/.*"waf_rule_ids":\[\([0-9,]*\)\].*/\1/p')"
+        query="$(field query "$line")"
+        printf '%-11s %-13s %-17s %s %s%s\n' "$(field action "$line")" "${ids:--}" \
+            "$(field request_id "$line")" "$(field method "$line")" \
+            "$(field path "$line")" "${query:+?$query}"
+    done <"$WORK/requests"
+    echo "gitea-requests: $(grep -c . "$WORK/requests") requests," \
+        "$(grep -c '"action":"waf_blocked"' "$WORK/requests") refused by the Core Rule Set"
+}
+
+main() {
+    cd "$ROOT"
+    trap cleanup EXIT
+    trap 'exit 1' HUP INT TERM
+    WORK="$(mktemp -d)"
+
+    docker build --no-cache -t "$IMAGE" .
+
+    # smallwebwaf first, so that gitea can be given the address it is
+    # reached at, and gitea's image pulled before, so that gitea starts
+    # within the minute smallwebwaf's health check waits for it.
+    docker pull --quiet "$GITEA_IMAGE" >/dev/null
+    docker network create "$NETWORK" >/dev/null
+    docker run --detach --name "$SMALLWEBWAF" --network "$NETWORK" \
+        --publish 127.0.0.1::8080 \
+        --env SWWAF_UPSTREAM_URL="http://$GITEA:3000" \
+        "$IMAGE" >/dev/null
+    published="$(docker port "$SMALLWEBWAF" 8080/tcp)"
+    address="localhost:${published##*:}"
+
+    docker run --detach --name "$GITEA" --network "$NETWORK" \
+        --env GITEA__security__INSTALL_LOCK=true \
+        --env GITEA__server__ROOT_URL="http://$address/" \
+        "$GITEA_IMAGE" >/dev/null
+    # smallwebwaf's health check passes once gitea takes connections.
+    wait_for "the health check did not pass" healthy
+    docker exec --user git "$GITEA" gitea admin user create --admin \
+        --username owner --password "$PASSWORD" --email owner@example.invalid \
+        --must-change-password=false >/dev/null
+
+    make_repository
+    browse
+    call_api
+    push_images
+    clone_and_fetch
+
+    # The fetch's last request is the last of all; once it is logged, so
+    # is every other.
+    wait_for "smallwebwaf logged no line for the fetch" \
+        logged '"request_id":"git-fetch"' '"method":"POST"'
+    report
+}
+
+main "$@"