script/gitea-requests, and the gitea refusals it confirms named in SPEC.md (closes #30)

script/gitea-requests runs gitea 1.27.3 behind the image at its
defaults, sends through it the requests that reading gitea's source
found the Core Rule Set refuses, with a short sample of ordinary use,
and prints each request with the rule ids that matched it. make
gitea-requests runs it; make check does not.

Every case still holds, and each fix would let through what SPEC.md
says stays refused, so the notes specific to gitea now name them with
how each refusal plays out: file names in query parameters that read as
Java, script, PowerShell or SQL, such paths, and an image push that
reuses a layer of an image named like a command.

Model: opus-5-5

Dockerfile

@@ -140,10 +140,14 @@ RUN mkdir /etc/sysusers.d \
 # apt also fetches the live archive's InRelease files, which change daily
 # and which the install does not use. The snapshot service is HTTPS only
 # and this image has no CA certificates yet, so this step uses the Go
-# image's.
+# image's. The snapshot service answers 503 at times; apt retries a
+# failed download, waiting twice as long before each retry, and this
+# step gives it five retries, about half a minute in all, where its
+# default of three gives up after seven seconds.
 RUN --mount=type=bind,from=builder,source=/etc/ssl/certs/ca-certificates.crt,target=/tmp/go-image-ca.crt \
     apt-get update --snapshot 20261001T000000Z \
         -o Acquire::https::CaInfo=/tmp/go-image-ca.crt \
+        -o Acquire::Retries=5 \
     && printf '%s\n' \
         '45f95ce276cdba3e41870516a130e03c58b8b7a79e9546b0efe9e526d255740c  snapshot.ubuntu.com_ubuntu_20261001T000000Z_dists_resolute_InRelease' \
         '802e675dd9de4c7f3916434a95e7c1d8eec0e82886622d7805ab19a2c6fe0365  snapshot.ubuntu.com_ubuntu_20261001T000000Z_dists_resolute-updates_InRelease' \
@@ -153,6 +157,7 @@ RUN --mount=type=bind,from=builder,source=/etc/ssl/certs/ca-certificates.crt,tar
     && DEBIAN_FRONTEND=noninteractive apt-get install --yes --no-install-recommends \
         --snapshot 20261001T000000Z \
         -o Acquire::https::CaInfo=/tmp/go-image-ca.crt \
+        -o Acquire::Retries=5 \
         ca-certificates nix-bin runit \
     && rm -rf /var/lib/apt/lists/*
 

README.md

@@ -2376,19 +2376,25 @@ so that they run in minimal containers.
   cache, and `script/check` does not run it; `make example-app` does.
 - `script/gitea-requests`: builds the image and runs gitea 1.27.3 behind it,
   with every setting of `smallwebwaf` at its default but `SWWAF_UPSTREAM_URL`,
-  which names gitea's container. It sends through it the gitea requests that
-  three of the notes specific to gitea in [`SPEC.md`](SPEC.md) name: a file name
-  in a query parameter that the Core Rule Set takes for Java, script, PowerShell
-  or SQL, a path it takes for a Java class or Node.js code or that has a
-  directory named `...`, and a container image push that reuses a layer. It also
-  sends a short sample of gitea's ordinary use: signing in, pages of files and
-  commits, a diff, the API, a push, clone and fetch over HTTP, and two container
-  image pushes. It prints a line for each request `smallwebwaf` logged, with
-  what it did with it and the ids of the Core Rule Set's rules that matched it,
-  then removes the containers, the network and the images it made, but gitea's
-  own image. It needs network access, and a docker daemon that pushes to a
-  registry on `localhost` over plain HTTP, as docker does by default;
-  `script/check` does not run it, and `make gitea-requests` does.
+  which names gitea's container. Of the requests that three of the notes
+  specific to gitea in [`SPEC.md`](SPEC.md) say are refused, it sends through
+  it: for each file the note on a file name in a query parameter names, the
+  API's list of its commits; for `Runtime.java` alone, the commit graph of that
+  file, the hidden lines of its diff and the API's call for the contents of
+  files that names it; the sign-in page gitea sends a visitor to on the way to
+  the edit page of `document.write.js`; the pages of `java.lang.String.md`,
+  `META-INF/services/org.apache.commons.logging.LogFactory`, `process.env.d.ts`
+  and `.../notes.txt`; and one container image push that reuses a layer. It does
+  not send the raw file, history, blame or edit page of those four files, the
+  pages of the directories `META-INF/services` and `...`, or the push again. It
+  also sends a short sample of gitea's ordinary use: signing in, pages of files
+  and commits, a diff, the API, a push, clone and fetch over HTTP, and two
+  container image pushes. It prints a line for each request `smallwebwaf`
+  logged, with what it did with it and the ids of the Core Rule Set's rules that
+  matched it, then removes the containers, the network and the images it made,
+  but gitea's own image. It needs network access, and a docker daemon that
+  pushes to a registry on `localhost` over plain HTTP, as docker does by
+  default; `script/check` does not run it, and `make gitea-requests` does.
 
 ## TODO
 

SPEC.md

@@ -1506,12 +1506,13 @@ holds any token file.
           page are refused; the directory that holds it lists it, and a clone
           carries it.
         - A container image push that reuses a layer of an image docker has
-          pushed to that gitea before, when that image's name, after its owner,
-          starts with an entry on the list of commands (932260), such as
-          `owner/python3-app` or `owner/docker-base`. docker asks gitea to take
-          the layer from that image, naming it in the query parameter `from`,
-          which 932260 checks, and gets the 403 answer of `smallwebwaf`. A
-          docker that keeps its images in containerd's image store, as a new
+          pushed to that gitea before, when that image's owner, or its name
+          after the owner, starts with an entry on the list of commands
+          (932260), such as `owner/python3-app`, `owner/docker-base` or any
+          image of an owner named `ansible` or `docker-team`. docker asks gitea
+          to take the layer from that image, naming it in the query parameter
+          `from`, which 932260 checks, and gets the 403 answer of `smallwebwaf`.
+          A docker that keeps its images in containerd's image store, as a new
           install of Docker Engine 29 does, then fails the push, and fails each
           new try the same way. A docker that keeps them in its own older image
           store uploads the layer instead, so the push succeeds after one
@@ -1606,12 +1607,17 @@ holds any token file.
   and bans no one by itself; the request log names the rule, and exclusions by
   rule id and path fix it. `script/gitea-requests` runs gitea behind the image
   at the defaults and sends through it a short sample of gitea's ordinary use
-  and the requests that three of the notes specific to gitea name: a file name
-  in a query parameter that the Core Rule Set takes for Java, script, PowerShell
-  or SQL, a path it takes for a Java class or Node.js code or that has a
-  directory named `...`, and a container image push that reuses a layer. It
-  reports each request the Core Rule Set refuses, so that those three notes can
-  be checked again when gitea or the Core Rule Set changes version.
+  and some of the requests that three of the notes specific to gitea say are
+  refused: for each file the note on a file name in a query parameter names, the
+  API's list of its commits; for `Runtime.java` alone, the commit graph of that
+  file, the hidden lines of its diff and the API's call for the contents of
+  files that names it; the sign-in page gitea sends a visitor to on the way to
+  the edit page of `document.write.js`; the pages of `java.lang.String.md`,
+  `META-INF/services/org.apache.commons.logging.LogFactory`, `process.env.d.ts`
+  and `.../notes.txt`; and one container image push that reuses a layer, which
+  it does not try again. It reports each request the Core Rule Set refuses, so
+  that those requests can be checked again when gitea or the Core Rule Set
+  changes version.
 - Attacks carried in request bodies: not refused by default, since on a code
   forge bodies are full of code the Core Rule Set takes for attacks. Most of
   what shows in URLs and headers is still refused, the client that sends an

internal/proxy/request.go

@@ -795,9 +795,19 @@ func (rq *request) responseTimedOut() {
 	}
 }
 
-// responseReceived is called once the app has sent its whole answer.
+// responseReceived is called once the app has sent its whole answer. An
+// app may answer before the client has sent its whole body, as with a 404
+// it sends without reading it. The timeouts then run on: until the
+// transport's read of the body returns, Go's server cannot send the
+// answer, and SWWAF_CLIENT_REQUEST_TIMEOUT is what ends that read for a
+// client that stops sending.
 func (rq *request) responseReceived() {
 	rq.complete = true
+
+	if rq.body != nil && !rq.body.received.Load() {
+		return
+	}
+
 	rq.stopTimers()
 }
 

internal/proxy/timeouts_test.go

@@ -194,6 +194,60 @@ func sendLargeBody(t *testing.T, addr string) net.Conn {
 	return conn
 }
 
+func TestClientStopsSendingAfterTheAppHasAnswered(t *testing.T) {
+	t.Parallel()
+
+	// answeredAt is when the app answered, or nil if the bytes the client
+	// sent never reached it.
+	var answeredAt atomic.Pointer[time.Time]
+
+	// The app answers without reading the whole body, as an app does for
+	// a path it does not serve, and only once smallwebwaf waits for the
+	// rest of it: the app takes the bytes the client sent, and then waits
+	// half the timeout, far longer than smallwebwaf takes to go on to
+	// wait.
+	app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
+		_, err := io.ReadFull(r.Body, make([]byte, len("the first bytes")))
+		if err != nil {
+			return
+		}
+
+		time.Sleep(shortTimeout / 2)
+		answeredAt.Store(new(time.Now()))
+		http.NotFound(w, r)
+	})
+	addr, out := startProxy(t, app.URL, map[string]string{
+		clientRequestTimeout: shortTimeoutSetting,
+		metricsToken:         token,
+	})
+
+	start := time.Now()
+	got := readResponse(t, sendPartOfBody(t, addr))
+	wantTimedOut(t, start)
+
+	// Close returns once the app has finished with the request.
+	app.Close()
+
+	// The app's answer came before the time ran out, so the client is
+	// sent that answer, not smallwebwaf's 408. Only a hold-up of the test
+	// process can make it come later, and SPEC.md then asks for 408, as
+	// smallwebwaf was waiting for the client, or for 504 if the bytes the
+	// client sent had not reached the app.
+	at := answeredAt.Load()
+	want := http.StatusNotFound
+
+	switch {
+	case at == nil:
+		want = http.StatusGatewayTimeout
+	case at.Sub(start) >= shortTimeout:
+		want = http.StatusRequestTimeout
+	}
+
+	wantStatus(t, got, want)
+	wantLine(t, out.requestLine(t), want, requestlog.ActionTimedOut)
+	wantLimitHits(t, addr, clientRequestTimeout, 1)
+}
+
 func TestAppTooSlowToAnswer(t *testing.T) {
 	t.Parallel()
 

script/gitea-requests

@@ -43,6 +43,8 @@ WORK=""
 address=""
 
 cleanup() {
+    # So that a signal cannot stop it partway through.
+    trap '' HUP INT TERM
     docker rm --force --volumes "$SMALLWEBWAF" "$GITEA" >/dev/null 2>&1 || true
     docker network rm "$NETWORK" >/dev/null 2>&1 || true
     docker rmi --force "$IMAGE" >/dev/null 2>&1 || true