Caches of outside APIs live in /var/cache/sww (closes #314)

The answers of GeoJS, the lists fetched by URL, the DNS blocklists,
AbuseIPDB and routewatch each go in a cache file in SWW_CACHE_DIR
(default /var/cache/sww): lookups.json, lists.json, dnsbl.json,
abuseipdb-scores.json and routewatch.json. reputation.json is gone. The
AbuseIPDB checks spent today are a count, not a cache, and stay in
SWW_STATE_DIR as abuseipdb-checks-spent.json.

A cache file never stops the start: one that does not parse is set
aside as an edit is, one that cannot be read is written over, and its
cache starts empty. An unwritable SWW_CACHE_DIR stops the start, as an
unwritable SWW_STATE_DIR does. Each directory is watched on its own.
The run script makes the cache directory and gives it to sww.

Model: opus-5-5

README.md

@@ -673,8 +673,8 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
   `internal/routewatch`, `internal/anomaly` and `internal/alerts`: one check or
   source each.
 - `internal/state`, `internal/requestlog`, `internal/remotelog` and
-  `internal/metrics`: the state and cache files, the log lines, the syslog
-  sender and the metrics.
+  `internal/metrics`: the state files, the log lines, the syslog sender and the
+  metrics.
 
 ## Rationale
 

SPEC.md

@@ -62,13 +62,13 @@ never as a time.
 - Packages:
     - `cmd/sww`: the binary, which only calls `internal/sww`.
     - `internal/sww`: the process, as the command it is given. `sww waf` reads
-      the settings, the rule files, the lookup database and the state and cache
-      files, listens, and serves until `SIGTERM` or `SIGINT`. Then it stops,
-      writing the state and cache files (see "`waf`"). `sww rproxy` reads only
-      that command's settings, and runs `internal/rproxy` until `SIGTERM` or
-      `SIGINT` (see "`rproxy`"). `sww healthcheck` is the image's health check.
-      With no command, or one it does not know, the binary prints the commands
-      and exits with status 2.
+      the settings, the rule files, the lookup database and the state files,
+      listens, and serves until `SIGTERM` or `SIGINT`. Then it stops, writing
+      the state files (see "`waf`"). `sww rproxy` reads only that command's
+      settings, and runs `internal/rproxy` until `SIGTERM` or `SIGINT` (see
+      "`rproxy`"). `sww healthcheck` is the image's health check. With no
+      command, or one it does not know, the binary prints the commands and exits
+      with status 2.
     - `internal/config`: reads the settings, the one place they are read.
     - `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds
       `waf`'s question, answer and report. It answers `waf`'s own endpoints in
@@ -106,7 +106,7 @@ never as a time.
       container, which `script/example-app` checks with `waf` in another.
 - Libraries, checked against the owner's Go dependency defaults:
     - the standard library for the reverse proxy, HTTP clients, DNS, logging
-      (`log/slog`) and the state and cache files (`encoding/json`, `os.Rename`);
+      (`log/slog`) and the state files (`encoding/json`, `os.Rename`);
     - `github.com/corazawaf/coraza/v3` and
       `github.com/corazawaf/coraza-coreruleset/v4` at `v4.25.0`, which carries
       the Core Rule Set 4.25.0. The changes `smallwebwaf` makes to it, and the
@@ -367,7 +367,7 @@ below are `waf`'s settings; "`rproxy`" gives the use of `rproxy`'s. Conventions:
   one, since `rproxy` never touches disk.
 - Settings, files included, are read once at start. Changing one means
   restarting the container. The files `waf` watches while it runs are its state
-  and cache files, its rule files and the lookup database.
+  files, its rule files and the lookup database.
 - An invalid setting stops the process at start with one log line, marked
   `"type":"process"` with `"msg":"invalid setting"`. The line names the variable
   and what is wrong with the value; "Each container's processes" says what
@@ -1932,10 +1932,10 @@ each app. `rproxy` asks `waf` about each visitor's request. `waf` answers
 This section is the agreement between the two. The header names, the fields and
 the statuses below are exact.
 
-`waf` reads the settings, the rule files, the lookup database and the state and
-cache files, and listens at `SWW_LISTEN_ADDR`. On `SIGTERM` or `SIGINT` it stops
+`waf` reads the settings, the rule files, the lookup database and the state
+files, and listens at `SWW_LISTEN_ADDR`. On `SIGTERM` or `SIGINT` it stops
 taking requests and gives those in progress five seconds to finish. It then
-writes the state and cache files.
+writes the state files.
 
 ### The question
 
@@ -2351,9 +2351,8 @@ but for these headers:
   on.
 - For a peer outside `SWRP_TRUSTED_PROXIES`, its forwarded headers are its own
   claims and are replaced. `X-Forwarded-For` names the peer, `X-Forwarded-Host`
-  the host it asked for, and `X-Forwarded-Proto` plain `http`. Every other
-  header whose name starts with `X-Forwarded-` is removed, and so are
-  `Forwarded` and `X-Real-IP`.
+  the host it asked for, and `X-Forwarded-Proto` plain `http`. The other headers
+  in which traefik tells an app about a visitor are removed.
 - `X-Request-ID`, which carries the fresh id.
 - `X-Client-ASN` and `X-Client-Country`, which carry what `waf` answered. A
   header it did not give is left out. Any of the two the visitor sent is

internal/requestlog/requestlog.go

@@ -5,7 +5,6 @@
 package requestlog
 
 import (
-	"bytes"
 	"encoding/json"
 	"fmt"
 	"io"
@@ -253,20 +252,14 @@ func WriteRProxyLine(w io.Writer, line *RProxyLine) error {
 	return writeJSONLine(w, line)
 }
 
-// writeJSONLine writes line to w as one JSON line. It writes "&", "<" and
-// ">" as they are, so a query string reads as it was sent.
+// writeJSONLine writes line to w as one JSON line.
 func writeJSONLine(w io.Writer, line any) error {
-	var encoded bytes.Buffer
-
-	encoder := json.NewEncoder(&encoded)
-	encoder.SetEscapeHTML(false)
-
-	err := encoder.Encode(line)
+	encoded, err := json.Marshal(line)
 	if err != nil {
 		return fmt.Errorf("encode the request log line: %w", err)
 	}
 
-	_, err = w.Write(encoded.Bytes())
+	_, err = w.Write(append(encoded, '\n'))
 	if err != nil {
 		return fmt.Errorf("write the request log line: %w", err)
 	}

internal/requestlog/requestlog_test.go

@@ -67,44 +67,6 @@ func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
 	}
 }
 
-func TestRequestLinesHoldTheQueryAndRefererAsSent(t *testing.T) {
-	t.Parallel()
-
-	var out bytes.Buffer
-
-	err := requestlog.Write(&out, &requestlog.Line{
-		Query:   "a=1&b=<2>",
-		Referer: "https://example.com/?x=1&y=<2>",
-	})
-	if err != nil {
-		t.Fatalf("write: %v", err)
-	}
-
-	err = requestlog.WriteRProxyLine(&out, &requestlog.RProxyLine{Query: "a=1&b=<2>"})
-	if err != nil {
-		t.Fatalf("write the rproxy line: %v", err)
-	}
-
-	lines := strings.Split(strings.TrimSuffix(out.String(), "\n"), "\n")
-	if len(lines) != 2 {
-		t.Fatalf("wrote %q, want two lines", out.String())
-	}
-
-	for _, line := range lines {
-		if !json.Valid([]byte(line)) {
-			t.Errorf("%s is not valid JSON", line)
-		}
-
-		if !strings.Contains(line, `"query":"a=1&b=<2>"`) {
-			t.Errorf("%s does not hold the query as sent", line)
-		}
-	}
-
-	if !strings.Contains(lines[0], `"referer":"https://example.com/?x=1&y=<2>"`) {
-		t.Errorf("%s does not hold the referer as sent", lines[0])
-	}
-}
-
 func TestWriteRProxyLineWritesOneJSONLineWithOnlyItsFields(t *testing.T) {
 	t.Parallel()
 

internal/rproxy/client.go

@@ -105,17 +105,17 @@ func setForwardedHeaders(in, out *http.Request, peer netip.Addr, trusted bool) {
 		return
 	}
 
-	// ReverseProxy has removed Forwarded and the three set below. Go's server
-	// gives header names in canonical form, so the prefix matches the name in
-	// any letter case.
-	for name := range out.Header {
-		if strings.HasPrefix(name, "X-Forwarded-") {
-			delete(out.Header, name)
-		}
+	// ReverseProxy has removed Forwarded and the three set below; these are the
+	// other headers in which traefik tells the app about the visitor and its
+	// request.
+	for _, name := range []string{
+		"X-Forwarded-Port", "X-Forwarded-Server", "X-Forwarded-Uri",
+		"X-Forwarded-Method", "X-Forwarded-Prefix", "X-Forwarded-Tls-Client-Cert",
+		"X-Forwarded-Tls-Client-Cert-Info", "X-Real-Ip",
+	} {
+		out.Header.Del(name)
 	}
 
-	out.Header.Del("X-Real-IP")
-
 	out.Header.Set("X-Forwarded-For", forwardedFor)
 	out.Header.Set("X-Forwarded-Host", in.Host)
 	out.Header.Set("X-Forwarded-Proto", "http")

internal/rproxy/limits_test.go

@@ -12,7 +12,6 @@ import (
 	"os"
 	"strconv"
 	"strings"
-	"sync"
 	"sync/atomic"
 	"testing"
 	"time"
@@ -808,10 +807,8 @@ func TestAVisitorThatStopsSendingAfterTheAppHasAnsweredGetsTheAnswer(t *testing.
 	app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
 		_, _ = io.WriteString(w, greeting)
 	})
-	// The request time limit is far longer than dial's deadline, so that a busy
-	// host cannot use it up before the app's answer arrives.
 	addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
-		maxRequestTime: "120000",
+		maxRequestTime: "3000",
 	})
 
 	// Far more announced than sent, so that Go's server answers for the app
@@ -872,11 +869,11 @@ func TestAVisitorThatStopsALittleShortGetsTheAppsAnswerAtOnce(t *testing.T) {
 
 	standIn := startWAF(t, allow)
 	app := startApp(t, answerBeforeTheBody)
-	// Both time limits are far longer than dial's deadline, so that the answer
-	// has to come before either runs out, and a busy host cannot use them up.
+	// The request time limit is longer than the 10 seconds the status and headers
+	// of an answer have to be sent, and than dial's deadline.
 	addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
-		maxRequestTime:  "120000",
-		maxResponseTime: "60000",
+		maxRequestTime:  "15000",
+		maxResponseTime: "300",
 	})
 
 	conn := dial(t, addr)
@@ -913,82 +910,6 @@ func TestAVisitorThatStopsALittleShortIsReadUntilTheRequestTimeLimit(t *testing.
 	}
 }
 
-func TestAVisitorThatStopsALittleShortHasTheConnectionClosedWhenTheAnswerIsCutOff(
-	t *testing.T,
-) {
-	t.Parallel()
-
-	const limit = 300 * time.Millisecond
-
-	// With two parts, rproxy fails to send the first and then waits for the
-	// second, which the app sends only once rproxy has ended the request to it.
-	for _, tc := range []struct {
-		name  string
-		parts []string
-	}{
-		{"with the body in one part", []string{greeting}},
-		{"with the body in two parts", []string{greeting[:5], greeting[5:]}},
-	} {
-		t.Run(tc.name, func(t *testing.T) {
-			t.Parallel()
-
-			headersRead := make(chan struct{})
-			// letAppSendBody also runs when the test ends, as the app's server waits
-			// for the app.
-			letAppSendBody := sync.OnceFunc(func() { close(headersRead) })
-			standIn := startWAF(t, allow)
-			// The app sends its body once the visitor has the status and headers and
-			// the response time limit has run out since, so that rproxy cannot send
-			// it. After each part, it reads the request's body, which ends when rproxy
-			// ends the request to it.
-			app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
-				rc := http.NewResponseController(w)
-				_ = rc.EnableFullDuplex()
-
-				w.Header().Set(contentLengthHeader, strconv.Itoa(len(greeting)))
-
-				_ = rc.Flush()
-
-				<-headersRead
-				time.Sleep(limit)
-
-				for _, part := range tc.parts {
-					_, _ = io.WriteString(w, part)
-					_ = rc.Flush()
-					_, _ = io.Copy(io.Discard, r.Body)
-				}
-			})
-
-			t.Cleanup(letAppSendBody)
-
-			// The request time limit is far longer than dial's deadline.
-			addr, _ := startRProxy(t, standIn.URL, app.URL, map[string]string{
-				maxRequestTime:  "120000",
-				maxResponseTime: strconv.FormatInt(limit.Milliseconds(), 10),
-			})
-
-			conn := dial(t, addr)
-			sendAllButALittle(t, conn)
-
-			res, err := http.ReadResponse(bufio.NewReader(conn), nil)
-			if err != nil {
-				t.Fatalf("read the answer: %v", err)
-			}
-
-			letAppSendBody()
-
-			// If the connection stays open, dial's deadline ends this read with a
-			// timeout.
-			body, err := io.ReadAll(res.Body)
-			_ = res.Body.Close()
-
-			if len(body) != 0 || !errors.Is(err, io.ErrUnexpectedEOF) {
-				t.Errorf("read %q (%v), want the connection closed", body, err)
-			}
-		})
-	}
-}
-
 func TestAVisitorThatStopsSendingWhileTheAppWorksGetsNoAnswer(t *testing.T) {
 	t.Parallel()
 

internal/rproxy/request.go

@@ -384,19 +384,8 @@ func (rq *request) refuse(status int) {
 func (rq *request) finish(ctx context.Context) {
 	rq.stopTimers()
 
-	// After a cut-off answer, the reading of a body that has not arrived whole
-	// ends now. Go's server would otherwise keep the connection open while it
-	// reads the rest. A flush fails if the time to send the answer ran out, and
-	// every flush after a failed one fails too. ReverseProxy ignores the failure,
-	// but panics if it was still waiting for more of the app's answer; finish runs
-	// either way. Before a status, a flush would send a 200; after a 101,
-	// ReverseProxy has taken over the connection.
-	switch {
-	case rq.out.status != 0 && rq.out.status != http.StatusSwitchingProtocols &&
-		errors.Is(rq.rc.Flush(), os.ErrDeadlineExceeded):
-		rq.stopPassingOnBody(time.Now())
-	case rq.refused.Load() == 0:
-		rq.stopPassingOnBody(rq.requestDeadline())
+	if rq.refused.Load() == 0 {
+		rq.stopPassingOnBody()
 	}
 
 	requestEnd := rq.endRequest()
@@ -484,10 +473,10 @@ func (rq *request) stopReadingBody(deadline time.Time) {
 }
 
 // stopPassingOnBody ends the transport's reading of a visitor's body that has
-// not arrived whole, and lets Go's server read the rest until deadline. Go's
-// server reads it once the handler returns, but with no deadline at all if
-// another read of it is still waiting.
-func (rq *request) stopPassingOnBody(deadline time.Time) {
+// not arrived whole, and lets Go's server read the rest until
+// SWRP_MAX_REQUEST_TIME_MS runs out. Go's server reads it once the handler
+// returns, but with no deadline at all if another read of it is still waiting.
+func (rq *request) stopPassingOnBody() {
 	if rq.body == nil || rq.body.received.Load() {
 		return
 	}
@@ -500,7 +489,7 @@ func (rq *request) stopPassingOnBody(deadline time.Time) {
 	rq.body.stopped = true
 	rq.body.mu.Unlock()
 
-	rq.stopReadingBody(deadline)
+	rq.stopReadingBody(rq.requestDeadline())
 }
 
 // startRequestTimer starts SWRP_MAX_REQUEST_TIME_MS, which runs from the

internal/rproxy/rproxy_test.go

@@ -559,8 +559,6 @@ func TestAPeerNotTrustedHasItsForwardedHeadersReplaced(t *testing.T) {
 		"X-Forwarded-Port", "X-Forwarded-Server", "X-Forwarded-Uri",
 		"X-Forwarded-Method", "X-Forwarded-Prefix", "X-Forwarded-Tls-Client-Cert",
 		"X-Forwarded-Tls-Client-Cert-Info", "X-Real-Ip",
-		"X-Forwarded-Scheme", "X-Forwarded-Ssl", "X-Forwarded-Protocol",
-		"X-Forwarded-Path", "X-Forwarded-Client-Cert",
 	}
 
 	standIn := startWAF(t, allow)
@@ -584,9 +582,6 @@ func TestAPeerNotTrustedHasItsForwardedHeadersReplaced(t *testing.T) {
 		req.Header.Set(name, "claimed")
 	}
 
-	// A made-up name, sent in lower case.
-	req.Header["x-forwarded-made-up"] = []string{"claimed"}
-
 	wantGreeting(t, visit(t, req))
 
 	header := <-seen
@@ -601,7 +596,7 @@ func TestAPeerNotTrustedHasItsForwardedHeadersReplaced(t *testing.T) {
 		}
 	}
 
-	for _, name := range append(removed, "X-Forwarded-Made-Up") {
+	for _, name := range removed {
 		if got := header.Values(name); len(got) != 0 {
 			t.Errorf("the app saw %s %q, want none", name, got)
 		}
@@ -627,7 +622,6 @@ func TestATrustedProxysForwardedHeadersPassOn(t *testing.T) {
 		"X-Forwarded-Host":   "public.example",
 		forwardedProtoHeader: "https",
 		"X-Real-Ip":          "198.51.100.1",
-		"X-Forwarded-Ssl":    "on",
 	}
 
 	req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)

internal/state/state.go

@@ -283,7 +283,7 @@ func Load(params Params) (*Files, error) {
 	f.readCache(abuseIPDBScoresJSON)
 	f.readCache(routeWatchJSON)
 
-	params.ProcessLog.Info("read the state and cache files", "directory", params.Dir,
+	params.ProcessLog.Info("read the state files", "directory", params.Dir,
 		"cache_directory", params.CacheDir, "bans", bansRead, "clients", clientsRead,
 		"lookups", lookupsRead, "lists", listsRead, "alerts_waiting", alertsRead)
 
@@ -344,7 +344,7 @@ func (f *Files) WriteAll() error {
 func (f *Files) Watch(ctx context.Context) {
 	watcher, err := fsnotify.NewWatcher()
 	if err != nil {
-		f.params.ProcessLog.Error("cannot watch the state and cache files for edits",
+		f.params.ProcessLog.Error("cannot watch the state files for edits",
 			"error", err.Error())
 
 		return
@@ -371,18 +371,12 @@ func (f *Files) Watch(ctx context.Context) {
 		watched = append(watched, dir.path)
 	}
 
-	watching := "watching the state and cache files for edits"
-
-	switch {
-	case len(watched) == 0:
+	if len(watched) == 0 {
 		return
-	case len(watched) == 1 && watched[0] == f.params.Dir:
-		watching = "watching the state files for edits"
-	case len(watched) == 1:
-		watching = "watching the cache files for edits"
 	}
 
-	f.params.ProcessLog.Info(watching, "directories", watched)
+	f.params.ProcessLog.Info("watching the state files for edits",
+		"directories", watched)
 
 	for {
 		select {
@@ -395,7 +389,7 @@ func (f *Files) Watch(ctx context.Context) {
 				f.fileChanged(name)
 			}
 		case err = <-watcher.Errors:
-			f.params.ProcessLog.Warn("watching the state and cache files failed",
+			f.params.ProcessLog.Warn("watching the state files failed",
 				"error", err.Error())
 		}
 	}
@@ -449,15 +443,11 @@ func (f *Files) takeInEdit(name string, data []byte) error {
 		return err
 	}
 
-	tookIn := "took in an edit of a state file"
-	if isCacheFile(name) {
-		tookIn = "took in an edit of a cache file"
-	}
-
 	// Counted before it is logged, so that the count is there once the
 	// log line is.
 	f.params.Metrics.StateFileEditTakenIn(name)
-	f.params.ProcessLog.Info(tookIn, "file", filepath.Join(f.dir(name), name))
+	f.params.ProcessLog.Info("took in an edit of a state file",
+		"file", filepath.Join(f.dir(name), name))
 
 	return nil
 }

internal/state/state_test.go

@@ -62,11 +62,10 @@ const (
 	// The AS number and AS name the tests' clients are looked up in.
 	asn    = "AS64496"
 	asName = "Example Net"
-	// What the process log says once Watch watches both directories, and as
-	// it takes in an edit of a state file, and of a cache file.
-	watching    = "watching the state and cache files for edits"
-	tookIn      = "took in an edit of a state file"
-	cacheTookIn = "took in an edit of a cache file"
+	// What the process log says once Watch watches the directory, and as
+	// it takes in an edit.
+	watching = "watching the state files for edits"
+	tookIn   = "took in an edit of a state file"
 	// maxLogLines is how many lines of the process log wait for a test to
 	// read them.
 	maxLogLines = 64
@@ -2064,18 +2063,11 @@ func TestDirectoryThatCannotBeWatchedLeavesTheOtherWatched(t *testing.T) {
 
 	for _, tc := range []struct {
 		// gone is the directory removed before Watch runs, and failed what
-		// the log says of it. watching is what the log says once Watch
-		// watches the other directory, and edited is a file in it.
-		gone, failed, watching, edited string
+		// the log says of it. edited is a file in the other directory.
+		gone, failed, edited string
 	}{
-		{
-			"state", "cannot watch the state files for edits",
-			"watching the cache files for edits", lookupsJSON,
-		},
-		{
-			"cache", "cannot watch the cache files for edits",
-			"watching the state files for edits", bansJSON,
-		},
+		{"state", "cannot watch the state files for edits", lookupsJSON},
+		{"cache", "cannot watch the cache files for edits", bansJSON},
 	} {
 		t.Run(tc.gone, func(t *testing.T) {
 			t.Parallel()
@@ -2103,7 +2095,7 @@ func TestDirectoryThatCannotBeWatchedLeavesTheOtherWatched(t *testing.T) {
 				t.Errorf("logged %v, want an error naming %s", line, gone)
 			}
 
-			lines.waitFor(t, tc.watching)
+			lines.waitFor(t, watching)
 			edit(t, other, tc.edited, `{"version": 1}`)
 			wantTakenIn(t, lines, other, tc.edited)
 		})
@@ -2468,21 +2460,12 @@ func (l processLog) waitFor(t *testing.T, msg string) map[string]any {
 	return nil
 }
 
-// wantTakenIn waits for the next edit taken in, of a state file or a cache
-// file as name is one, and checks that it is of the file name in dir.
+// wantTakenIn waits for the next edit taken in, and checks that it is of
+// the state file name in dir.
 func wantTakenIn(t *testing.T, lines processLog, dir, name string) {
 	t.Helper()
 
-	var msg string
-
-	switch name {
-	case lookupsJSON, listsJSON, dnsblJSON, abuseIPDBScoresJSON, routeWatchJSON:
-		msg = cacheTookIn
-	default:
-		msg = tookIn
-	}
-
-	line := lines.waitFor(t, msg)
+	line := lines.waitFor(t, tookIn)
 	if line["file"] != filepath.Join(dir, name) {
 		t.Fatalf("took in %v, want an edit of %s", line, name)
 	}

internal/waf/waf.go

@@ -213,10 +213,10 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
 
 // readQuestion returns the visitor's address and the body a question carries,
 // and whether waf takes the question. It answers a question it does not
-// take with 400 and no disposition. rproxy passes that 400 to the visitor,
-// and SWRP_DEFAULT_DISPOSITION does not apply. The id of a question waf takes
-// is held from here on, without a decision yet, so that another question with
-// that id is refused while this one is being handled.
+// take with 400 and no disposition, which rproxy treats as it treats no
+// answer. The id of a question it takes is held from here on, without a
+// decision yet, so that another question with that id is refused while this
+// one is being handled.
 //
 // The body is read before any check, since Go's server does not send an answer
 // until it has read the body. A question whose body stops arriving gets no