Blocklists and an AS percentage file fetched by URL (closes #29)

SWWAF_BLOCKLIST_URLS names lists of addresses and netblocks, fetched every
SWWAF_BLOCKLIST_REFRESH (24h, never under 1h); an IPv4-mapped line stands
for its IPv4 address or netblock. reputation.json keeps each list's last
try, failed or not, which a restart waits on as a running instance does,
and its last good copy, whole, used while a fetch fails.
SWWAF_BLOCKLIST_ACTION denies, limits or only logs a listed client; the log
line names the lists, each raises reputation_hit, and a failed fetch raises
source_failure. SWWAF_ASN_LIMIT_PERCENT_URL is fetched the same way and
counts as SWWAF_ASN_LIMIT_PERCENT does, the lower winning.

Judgement call: a failed fetch is retried after the refresh, not sooner.
Not done: ban notes do not name the lists yet.

Model: opus-5-5

README.md

@@ -132,15 +132,16 @@ in `bin/state` unless `SWWAF_STATE_DIR` is set, and the default rule file of
   `SWWAF_COUNTRY_LIMIT_PERCENT`, the percentage they give of every rate limit
   and byte limit, so that the same rules ban them after fewer requests, and,
   while `SWWAF_UNKNOWN_LIMIT_PERCENT` is below 100, every client without a
-  country that percentage. A client to which several apply gets the lowest.
-  `SWWAF_ASN_BYTES_PERCENT` and `SWWAF_COUNTRY_BYTES_PERCENT` give the AS
-  numbers and countries they list a percentage of the byte limits in place of
-  the other two. Each client is counted on its own, against its own lowered
-  limits: no budget is shared by a whole AS number or country, which one abuser
-  could use up and so lock out everyone else there. The log line of each request
-  the rate limits count gives its client's percentages below 100 and the
-  settings that gave them, and so do the notes of a ban for a lowered limit, and
-  its alert.
+  country that percentage. A client to which several apply gets the lowest. For
+  the byte limits, `SWWAF_ASN_BYTES_PERCENT` gives an AS number it lists a
+  percentage in place of those `SWWAF_ASN_LIMIT_PERCENT` and the file give it,
+  and `SWWAF_COUNTRY_BYTES_PERCENT` gives a country it lists one in place of the
+  one `SWWAF_COUNTRY_LIMIT_PERCENT` gives it. Each client is counted on its own,
+  against its own lowered limits: no budget is shared by a whole AS number or
+  country, which one abuser could use up and so lock out everyone else there.
+  The log line of each request the rate limits count gives its client's
+  percentages below 100 and the settings that gave them, and so do the notes of
+  a ban for a lowered limit, and its alert.
 - Bans a client that breaks a rate limit or a byte limit, as "Bans" in
   [`SPEC.md`](SPEC.md) describes: the first ban lasts an hour, and a limit
   broken again within a day of a ban ending bans for three times as long as that
@@ -985,11 +986,12 @@ with times in UTC.
   `grep` shows everything about one.
 - `lookups.json`: GeoJS's answers, one to a line, each with the client's AS
   number, AS name and country, when GeoJS gave it and when it was last used.
-- `reputation.json`: the last good copy of each list fetched from a URL (see
-  "Blocklists" below), indented to be read, under `lists`: its `url`, when it
-  was `fetched`, and its `lines`, as fetched, comment lines included, each on a
-  line of its own. As the file is read, the copies of lists the settings no
-  longer name are dropped.
+- `reputation.json`: each list fetched from a URL (see "Blocklists" below),
+  indented to be read, under `lists`: its `url`, when it was last `tried`, the
+  fetch failed or not, and its last good copy: when that was `fetched`, and its
+  `lines`, as fetched, comment lines included, each on a line of its own, both
+  left out while no fetch of it has succeeded. As the file is read, the lists
+  the settings no longer name are dropped.
 - `alerts.json`: the state of the alerts (see "Alerts" above), indented to be
   read: under `cooldowns`, for each event and netblock, with the `source` too
   for a `reputation_hit`, or event and `file` or `source`, or for an `anomaly`,
@@ -1612,24 +1614,28 @@ GeoJS.
 one to a line, written as the Spamhaus DROP list,
 `https://www.spamhaus.org/drop/drop.txt`, is. Anything after a `;` or a `#` on a
 line is left out, and so is a line left blank. A bare address stands for itself
-alone, as in the settings. None is named by default: a list judges a client by
-what others saw it do, while the defaults judge it by what it does to your
-service.
+alone, as in the settings. An IPv4-mapped address or netblock, such as
+`::ffff:192.0.2.0/120`, is read as the IPv4 one it stands for, here
+`192.0.2.0/24`, since a client's IPv4 address is checked as IPv4; a mapped
+netblock shorter than `/96` stands for none, and is not a netblock. None is
+named by default: a list judges a client by what others saw it do, while the
+defaults judge it by what it does to your service.
 
 `smallwebwaf` fetches each list, and the file `SWWAF_ASN_LIMIT_PERCENT_URL`
 names, `SWWAF_BLOCKLIST_REFRESH` after it last fetched it or tried to, 24 hours
-by default and never less than one, one list after another. At start it fetches
-at once a list it keeps no copy of, and one whose copy is that old; a younger
-copy waits its turn, so that restarts do not fetch a list more often. A fetch
-fails when the server answers other than `200`, when it does not finish within a
-minute, when the list is longer than 16 MiB, or when a line of it is not an
-address or a netblock, or for `SWWAF_ASN_LIMIT_PERCENT_URL`, not an AS number,
-`:` and a percentage. The copy fetched before then stays in use, and the failure
-is counted, logged and raised as a `source_failure` alert. The last good copy of
-each list is kept whole, comment lines included, in `reputation.json` (see
-"State files" above), so that a restart keeps it in use too. Each list is named
-by its URL, in the request log, the alerts and the metrics, so keep a secret out
-of it.
+by default and never less than one, one list after another. Since
+`reputation.json` keeps when each list was last tried, the fetch failed or not,
+at start `smallwebwaf` fetches at once only a list it has never tried, and one
+it last tried that long ago; any other waits its turn, so that restarts do not
+fetch a list more often. A fetch fails when the server answers other than `200`,
+when it does not finish within a minute, when the list is longer than 16 MiB, or
+when a line of it is not an address or a netblock, or for
+`SWWAF_ASN_LIMIT_PERCENT_URL`, not an AS number, `:` and a percentage. The copy
+fetched before then stays in use, and the failure is counted, logged and raised
+as a `source_failure` alert. The last good copy of each list is kept whole,
+comment lines included, in `reputation.json` (see "State files" above), so that
+a restart keeps it in use too. Each list is named by its URL, in the request
+log, the alerts and the metrics, so keep a secret out of it.
 
 A client in `SWWAF_ALLOW_NETS` is not checked. Any other is checked by its own
 address after the country lists, and `SWWAF_BLOCKLIST_ACTION` says what is done
@@ -1645,9 +1651,11 @@ and keep the list's date and copyright lines with the data, which the copy in
 from The Spamhaus Project, and should say so. They also ask that it be fetched
 automatically no more than once an hour, once a day being more than enough in
 most cases, and Spamhaus may block an address that fetches it more often. Each
-`smallwebwaf` fetches its own copy, so on a host where several apps run it,
-sharing one address, set `SWWAF_BLOCKLIST_REFRESH` long enough that their
-fetches come at least an hour apart.
+`smallwebwaf` fetches its own copy, on its own schedule, so on a host where
+several apps run it, sharing one address, their fetches can come less than an
+hour apart whatever `SWWAF_BLOCKLIST_REFRESH` is: each fetches a list again that
+long after its own last try, so those first started within the same hour, with
+the same refresh, keep fetching within the same hour.
 
 ## How the code is laid out
 

internal/reputation/reputation.go

@@ -2,8 +2,8 @@
 // blocklists of SWWAF_BLOCKLIST_URLS, and the file of AS:percent lines
 // SWWAF_ASN_LIMIT_PERCENT_URL names. It keeps the last good copy of each,
 // whole, comment lines included, which is used while a fetch fails, and
-// which the state package writes to reputation.json and reads from it, so
-// that a restart keeps it too.
+// when each was last tried, which the state package writes to
+// reputation.json and reads from it, so that a restart keeps them too.
 package reputation
 
 import (
@@ -29,6 +29,9 @@ const (
 	maxListBytes = 16 << 20
 	// fetchTimeout bounds one fetch of a list.
 	fetchTimeout = time.Minute
+	// mappedBits is the length of ::ffff:0.0.0.0/96, the netblock of every
+	// IPv4-mapped address.
+	mappedBits = 96
 )
 
 var (
@@ -39,13 +42,15 @@ var (
 		"is not an AS number, : and a percentage, such as AS64496:50")
 )
 
-// List is the last good copy of a list, as reputation.json holds it: the
-// URL it was fetched from, when it was fetched, and its lines, as fetched,
-// comment lines included.
+// List is a list as reputation.json holds it: the URL it is fetched from,
+// when it was last tried, the fetch failed or not, and its last good copy:
+// when that was fetched, and its lines, as fetched, comment lines
+// included, both left out while no fetch of it has succeeded.
 type List struct {
 	URL     string    `json:"url"`
-	Fetched time.Time `json:"fetched"`
-	Lines   []string  `json:"lines"`
+	Tried   time.Time `json:"tried"`
+	Fetched time.Time `json:"fetched,omitzero"`
+	Lines   []string  `json:"lines,omitzero"`
 }
 
 // Params are what New needs.
@@ -77,13 +82,12 @@ type Lists struct {
 	lists map[string]*list
 }
 
-// list is one list: its last good copy, what that says, when the list was
-// last fetched or tried, zero before the first try, and how many fetches
-// of it failed.
+// list is one list: what reputation.json keeps of it, its last try, zero
+// before the first, and its last good copy, what that copy says, and how
+// many fetches of it failed.
 type list struct {
 	kept     List
 	entries  entries
-	tried    time.Time
 	failures int
 }
 
@@ -101,7 +105,7 @@ func New(params Params) *Lists {
 	l := &Lists{params: params, httpClient: &http.Client{}, lists: map[string]*list{}}
 
 	for _, listURL := range l.URLs() {
-		l.lists[listURL] = &list{}
+		l.lists[listURL] = &list{kept: List{URL: listURL}}
 	}
 
 	return l
@@ -168,9 +172,9 @@ func (l *Lists) Failures(listURL string) int {
 }
 
 // Run fetches each list once Refresh has passed since it was last fetched
-// or tried, the later of the two, until ctx is done. A list without a copy
-// has never been fetched, so it is fetched at once, and so is one whose
-// copy, read from reputation.json, is that old.
+// or tried, the later of the two, until ctx is done. A list never tried is
+// fetched at once, and so is one whose last try or copy, read from
+// reputation.json, is that old.
 func (l *Lists) Run(ctx context.Context) {
 	if len(l.lists) == 0 {
 		return
@@ -189,35 +193,35 @@ func (l *Lists) Run(ctx context.Context) {
 	}
 }
 
-// Snapshot returns the copy of each list that has one, sorted by URL, as
-// reputation.json lists them.
+// Snapshot returns each list that has been tried, with its copy, if it
+// has one, sorted by URL, as reputation.json lists them.
 func (l *Lists) Snapshot() []List {
 	l.mu.Lock()
 
-	copies := make([]List, 0, len(l.lists))
+	tried := make([]List, 0, len(l.lists))
 
 	for _, held := range l.lists {
-		if !held.kept.Fetched.IsZero() {
-			copies = append(copies, held.kept)
+		if !held.kept.Tried.IsZero() {
+			tried = append(tried, held.kept)
 		}
 	}
 
 	l.mu.Unlock()
 
-	slices.SortFunc(copies, func(a, b List) int {
+	slices.SortFunc(tried, func(a, b List) int {
 		return strings.Compare(a.URL, b.URL)
 	})
 
-	return copies
+	return tried
 }
 
-// Load puts copies, read from reputation.json, in place of the copies
-// held. A copy of a list Params does not name is dropped. A copy with a
+// Load puts lists, read from reputation.json, in place of the last tries
+// and copies held. A list Params does not name is dropped. A copy with a
 // line that parse refuses is an error, and then nothing changes.
-func (l *Lists) Load(copies []List) error {
-	found := make(map[string]entries, len(copies))
+func (l *Lists) Load(lists []List) error {
+	found := make(map[string]entries, len(lists))
 
-	for _, kept := range copies {
+	for _, kept := range lists {
 		if _, named := l.lists[kept.URL]; !named {
 			continue
 		}
@@ -233,11 +237,11 @@ func (l *Lists) Load(copies []List) error {
 	l.mu.Lock()
 	defer l.mu.Unlock()
 
-	for _, held := range l.lists {
-		held.kept, held.entries = List{}, entries{}
+	for listURL, held := range l.lists {
+		held.kept, held.entries = List{URL: listURL}, entries{}
 	}
 
-	for _, kept := range copies {
+	for _, kept := range lists {
 		read, named := found[kept.URL]
 		if named {
 			l.lists[kept.URL].kept, l.lists[kept.URL].entries = kept, read
@@ -276,8 +280,8 @@ func (l *Lists) due(listURL string) time.Time {
 	held := l.lists[listURL]
 
 	last := held.kept.Fetched
-	if held.tried.After(last) {
-		last = held.tried
+	if held.kept.Tried.After(last) {
+		last = held.kept.Tried
 	}
 
 	return last.Add(l.params.Refresh)
@@ -304,10 +308,10 @@ func (l *Lists) fetch(ctx context.Context, listURL string) {
 	l.mu.Lock()
 
 	held := l.lists[listURL]
-	held.tried = now
+	held.kept.Tried = now
 
 	if err == nil {
-		held.kept = List{URL: listURL, Fetched: now, Lines: lines}
+		held.kept.Fetched, held.kept.Lines = now, lines
 		held.entries = found
 	} else {
 		held.failures++
@@ -399,8 +403,8 @@ func parseNetblocks(lines []string) (entries, error) {
 			continue
 		}
 
-		netblock, err := config.ParseNetblock(text)
-		if err != nil {
+		netblock, ok := parseNetblock(text)
+		if !ok {
 			return entries{}, fmt.Errorf("line %d %w", i+1, errNotNetblock)
 		}
 
@@ -414,6 +418,32 @@ func parseNetblocks(lines []string) (entries, error) {
 	return found, nil
 }
 
+// parseNetblock reads text, a line of a blocklist, and reports whether it
+// is an address or a netblock as the settings take them. A client's IPv4
+// address is checked as IPv4, never IPv4-mapped, so an IPv4-mapped line,
+// such as ::ffff:192.0.2.0/120, is read as the IPv4 address or netblock it
+// stands for, 192.0.2.0/24, and a mapped netblock shorter than /96, which
+// stands for none, is refused.
+func parseNetblock(text string) (netip.Prefix, bool) {
+	netblock, err := config.ParseNetblock(text)
+	if err != nil {
+		return netip.Prefix{}, false
+	}
+
+	// The address as written: ParseNetblock's has the bits past the
+	// netblock's length cleared, the ::ffff among them below /96.
+	written, _, _ := strings.Cut(text, "/")
+	if addr, _ := netip.ParseAddr(written); !addr.Is4In6() {
+		return netblock, true
+	}
+
+	if netblock.Bits() < mappedBits {
+		return netip.Prefix{}, false
+	}
+
+	return netip.PrefixFrom(netblock.Addr().Unmap(), netblock.Bits()-mappedBits), true
+}
+
 // parsePercents reads the lines of the file of AS:percent lines, each an
 // AS number, : and a percentage, as SWWAF_ASN_LIMIT_PERCENT takes them. An
 // AS number listed more than once gets the lowest of its percentages.

internal/reputation/reputation_test.go

@@ -72,6 +72,41 @@ func TestListedAddressesAndNetblocksWithTheCommentsLeftOut(t *testing.T) {
 	})
 }
 
+func TestIPv4MappedLineListsTheIPv4AddressOrNetblockItStandsFor(t *testing.T) {
+	t.Parallel()
+
+	now := time.Now()
+	lists := reputation.New(params(dropURL))
+
+	err := lists.Load([]reputation.List{{
+		URL: dropURL, Tried: now, Fetched: now,
+		Lines: []string{"::ffff:192.0.2.9", "::ffff:203.0.113.0/120"},
+	}})
+	if err != nil {
+		t.Fatalf("load: %v", err)
+	}
+
+	for addr, want := range map[string][]string{
+		"192.0.2.9":     {dropURL},
+		"203.0.113.255": {dropURL},
+		"192.0.2.8":     nil,
+		"203.0.114.0":   nil,
+	} {
+		wantListedBy(t, lists, addr, want...)
+	}
+
+	// A mapped netblock shorter than /96 stands for no IPv4 one.
+	err = lists.Load([]reputation.List{{
+		URL: dropURL, Tried: now, Fetched: now, Lines: []string{"::ffff:198.51.100.0/88"},
+	}})
+
+	const want = "the copy of " + dropURL +
+		": line 1 is not an address or a netblock, such as 192.0.2.0/24"
+	if err == nil || err.Error() != want {
+		t.Errorf("error %v, want %s", err, want)
+	}
+}
+
 func TestClientIsListedByEachBlocklistThatListsIt(t *testing.T) {
 	t.Parallel()
 
@@ -167,8 +202,11 @@ func TestFailedFetchKeepsTheLastGoodCopyAndAlertsOncePerCooldown(t *testing.T) {
 				wantFetches(t, servers, 3)
 				wantListedBy(t, lists, "203.0.113.9", dropURL)
 
-				if got := lists.Snapshot(); !reflect.DeepEqual(got, kept) {
-					t.Errorf("copies %+v, want the first %+v", got, kept)
+				want := kept[0]
+				want.Tried = time.Now()
+
+				if got := lists.Snapshot(); !reflect.DeepEqual(got, []reputation.List{want}) {
+					t.Errorf("lists %+v, want the first copy, last tried now, %+v", got, want)
 				}
 
 				if lists.Failures(dropURL) != 2 {
@@ -251,6 +289,43 @@ func TestKeptCopyIsFetchedAgainOnceRefreshHasPassedSinceItWasFetched(t *testing.
 	})
 }
 
+func TestRestartWaitsRefreshAfterTheLastTryEvenOneThatFailed(t *testing.T) {
+	t.Parallel()
+
+	synctest.Test(t, func(t *testing.T) {
+		// drop.txt is fetched, and a refresh later the fetch downloads it
+		// whole but fails on a line that does not read.
+		servers := &standIn{lists: map[string]string{dropURL: "198.51.100.1\n"}}
+		lists := start(t, servers, params(dropURL))
+
+		servers.set(dropURL, "198.51.100.2\n<html>\n")
+		time.Sleep(refresh)
+		wantFetches(t, servers, 2)
+
+		// Restarted with what reputation.json keeps, it waits a refresh
+		// after the failed try, as it does while it runs.
+		restarted := &standIn{lists: map[string]string{dropURL: "198.51.100.2\n"}}
+		again := reputation.New(params(dropURL))
+		again.SetTransport(restarted)
+
+		err := again.Load(lists.Snapshot())
+		if err != nil {
+			t.Fatalf("load: %v", err)
+		}
+
+		run(t, again)
+		wantFetches(t, restarted, 0)
+		wantListedBy(t, again, "198.51.100.1", dropURL)
+
+		time.Sleep(refresh - time.Nanosecond)
+		wantFetches(t, restarted, 0)
+
+		time.Sleep(time.Nanosecond)
+		wantFetches(t, restarted, 1)
+		wantListedBy(t, again, "198.51.100.2", dropURL)
+	})
+}
+
 func TestFetchCutOffAsItStopsIsNoFailure(t *testing.T) {
 	t.Parallel()
 
@@ -320,12 +395,14 @@ func TestLoadDropsCopiesOfListsNotNamedAndRefusesOnesThatDoNotRead(t *testing.T)
 	t.Parallel()
 
 	fetched := time.Date(2026, 10, 6, 0, 0, 0, 0, time.UTC)
-	kept := reputation.List{URL: dropURL, Fetched: fetched, Lines: []string{"203.0.113.9"}}
+	kept := reputation.List{
+		URL: dropURL, Tried: fetched, Fetched: fetched, Lines: []string{"203.0.113.9"},
+	}
 	lists := reputation.New(params(dropURL))
 
-	err := lists.Load([]reputation.List{
-		kept, {URL: torURL, Fetched: fetched, Lines: []string{"198.51.100.9"}},
-	})
+	err := lists.Load([]reputation.List{kept, {
+		URL: torURL, Tried: fetched, Fetched: fetched, Lines: []string{"198.51.100.9"},
+	}})
 	if err != nil {
 		t.Fatalf("load: %v", err)
 	}

internal/smallwebwaf/smallwebwaf_test.go

@@ -574,31 +574,39 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
 	}
 }
 
-func TestBlocklistCopyKeptInReputationJSONAcrossARestart(t *testing.T) {
+func TestBlocklistTriesAndCopiesKeptInReputationJSONAcrossRestarts(t *testing.T) {
 	t.Parallel()
 
 	const (
 		token    = "0123456789abcdef0123456789abcdef"
-		// drop is the blocklist, with the date and copyright line the
-		// Spamhaus DROP list starts with.
+		torPath  = "/tor.txt"
+		dropPath = "/drop.txt"
+		// copyright is the DROP list's date and copyright line.
 		copyright = "; Spamhaus DROP List 2026/10/07 - (c) 2026 The Spamhaus Project SLL"
-		drop      = copyright + "\n" + placed + " ; SBL1\n"
 	)
 
-	var failing atomic.Bool
+	failing, torFetches := new(atomic.Bool), new(atomic.Int32)
+	lists := map[string]string{
+		torPath:  "198.51.100.0/24\n",
+		dropPath: copyright + "\n" + placed + " ; SBL1\n",
+	}
+	server := httptest.NewServer(http.HandlerFunc(
+		func(w http.ResponseWriter, r *http.Request) {
+			if r.URL.Path == torPath {
+				torFetches.Add(1)
+			}
 
-	list := httptest.NewServer(http.HandlerFunc(
-		func(w http.ResponseWriter, _ *http.Request) {
 			if failing.Load() {
 				w.WriteHeader(http.StatusServiceUnavailable)
 
 				return
 			}
 
-			_, _ = io.WriteString(w, drop)
+			_, _ = io.WriteString(w, lists[r.URL.Path])
 		}))
-	t.Cleanup(list.Close)
+	t.Cleanup(server.Close)
 
+	torURL, dropURL := server.URL+torPath, server.URL+dropPath
 	dir := t.TempDir()
 	env := map[string]string{
 		listenAddr:             localhost + ":0",
@@ -608,43 +616,48 @@ func TestBlocklistCopyKeptInReputationJSONAcrossARestart(t *testing.T) {
 		trustedProxies:         localhost + "/32",
 		metricsToken:           token,
 		instanceName:           instance,
-		"SWWAF_BLOCKLIST_URLS": list.URL,
+		"SWWAF_BLOCKLIST_URLS": torURL,
 		// The requests sent until the list takes effect, and those for the
 		// metrics, must not break a rate limit, whose ban would refuse them
 		// too.
 		rateLimitExemptNets: placed + "," + localhost,
 	}
 	failures := `smallwebwaf_reputation_failures_total{instance="fsn1app1/gitea",` +
-		`source="` + list.URL + `"} `
+		`source="` + torURL + `"} `
 
-	// The list cannot be fetched at first, which leaves the client let
-	// through, and is counted.
+	// tor.txt cannot be fetched at first, which is counted.
 	failing.Store(true)
 	runUntilStopped(t, env, func(url string) {
 		metricsWith(t, url+"_smallwebwaf/metrics", token, failures+"1")
-		wantStatus(t, url, placed, http.StatusOK)
 	})
 
-	// With no copy kept, it is fetched as smallwebwaf starts again, and from
-	// then on the list refuses the client.
+	// Restarted with drop.txt named after it and the server answering, tor.txt
+	// waits SWWAF_BLOCKLIST_REFRESH after its failed try, kept in reputation.json,
+	// while drop.txt, never tried, is fetched at once. Lists are fetched in the
+	// order named, so once drop.txt refuses the client, tor.txt has had its turn.
 	failing.Store(false)
 
+	env["SWWAF_BLOCKLIST_URLS"] = torURL + "," + dropURL
 	out := runUntilStopped(t, env, func(url string) {
 		for statusFrom(t, url, placed) != http.StatusForbidden {
 			time.Sleep(pollInterval)
 		}
 	})
-	wantDeniedByList(t, out.line(t, "action", "denied"), list.URL)
+	wantDeniedByList(t, out.line(t, "action", "denied"), dropURL)
 
-	// After another restart, with the list's server failing, the copy kept
-	// in reputation.json, its copyright line included, refuses the client
-	// from the first request.
+	if fetches := torFetches.Load(); fetches != 1 {
+		t.Errorf("tor.txt fetched %d times, want once, before the restart", fetches)
+	}
+
+	// After another restart, with the server failing, the copy of drop.txt
+	// kept in reputation.json, its copyright line included, refuses the
+	// client from the first request.
 	failing.Store(true)
 
 	out = runUntilStopped(t, env, func(url string) {
 		wantStatus(t, url, placed, http.StatusForbidden)
 	})
-	wantDeniedByList(t, out.line(t, "type", "request"), list.URL)
+	wantDeniedByList(t, out.line(t, "type", "request"), dropURL)
 
 	path := filepath.Join(dir, "reputation.json")
 

internal/state/state.go

@@ -1,10 +1,10 @@
 // Package state keeps smallwebwaf's state in JSON files in
 // SWWAF_STATE_DIR, as the "Persistent state" section of SPEC.md describes:
 // bans.json holds the bans, clients.json each client's counters and
-// history, lookups.json GeoJS's answers, reputation.json the last good
-// copy of each list fetched from a URL, and alerts.json the cooldowns, the
-// hour under way, the alerts waiting for each destination and the anomaly
-// counters. Load
+// history, lookups.json GeoJS's answers, reputation.json the last try and
+// last good copy of each list fetched from a URL, and alerts.json the
+// cooldowns, the hour under way, the alerts waiting for each destination
+// and the anomaly counters. Load
 // reads them at start, Watch takes in an admin's edit of one while
 // smallwebwaf runs, and Run and WriteAll write them. The disk is read and
 // written outside the parts' locks, which are held only to take a
@@ -726,17 +726,20 @@ func (f *lookupsFile) check(data []byte) error {
 	return nil
 }
 
-// check refuses a list's copy without its URL, which would name no list,
-// the time it was fetched, which would have the list fetched at once, or
-// its lines, which hold the list.
+// check refuses a list without its URL, which would name no list, or the
+// time it was last tried, which would have it fetched at once, and a copy
+// of it without the time it was fetched, or without its lines, which hold
+// the list.
 func (f *reputationFile) check([]byte) error {
 	for i, kept := range f.Lists {
 		switch {
 		case kept.URL == "":
 			return missing(i, "url")
-		case kept.Fetched.IsZero():
+		case kept.Tried.IsZero():
+			return missing(i, "tried")
+		case kept.Fetched.IsZero() && kept.Lines != nil:
 			return missing(i, "fetched")
-		case kept.Lines == nil:
+		case kept.Lines == nil && !kept.Fetched.IsZero():
 			return missing(i, "lines")
 		}
 	}

internal/state/state_test.go

@@ -38,8 +38,9 @@ const (
 	lookupsJSON    = "lookups.json"
 	reputationJSON = "reputation.json"
 	alertsJSON     = "alerts.json"
-	// blocklistURL is the blocklist the tests' lists name.
+	// blocklistURL and torURL are the blocklists the tests' lists name.
 	blocklistURL = "https://lists.example/drop.txt"
+	torURL       = "https://lists.example/tor.txt"
 	// The AS number and AS name the tests' clients are looked up in.
 	asn    = "AS64496"
 	asName = "Example Net"
@@ -208,19 +209,24 @@ const filledAlertsJSON = `{
 }
 `
 
-// filledReputationJSON is reputation.json holding the copy of the
-// blocklist fill puts in, with its comment line.
+// filledReputationJSON is reputation.json holding the blocklists' last
+// tries and the copy of one, with its comment line, as fill puts them in.
 const filledReputationJSON = `{
     "version": 1,
     "lists": [
         {
             "url": "https://lists.example/drop.txt",
+            "tried": "2026-10-06T00:00:00Z",
             "fetched": "2026-10-05T23:00:00Z",
             "lines": [
                 "; Spamhaus DROP List 2026/10/05 - (c) 2026 The Spamhaus Project SLL",
                 "203.0.113.0/24 ; SBL1",
                 "2001:db8::/32 ; SBL2"
             ]
+        },
+        {
+            "url": "https://lists.example/tor.txt",
+            "tried": "2026-10-06T00:00:00Z"
         }
     ]
 }
@@ -479,7 +485,8 @@ func TestFileThatDoesNotParseStopsTheStart(t *testing.T) {
 		{
 			"a copy of a list with a line that does not read", reputationJSON,
 			`{"version": 1, "lists": [{"url": "` + blocklistURL + `", ` +
-				`"fetched": "2026-10-06T00:00:00Z", "lines": ["; DROP", "203.0.113.300"]}]}`,
+				`"tried": "2026-10-06T00:00:00Z", "fetched": "2026-10-06T00:00:00Z", ` +
+				`"lines": ["; DROP", "203.0.113.300"]}]}`,
 			`: the copy of ` + blocklistURL + `: line 2 is not an address or a netblock, ` +
 				`such as 192.0.2.0/24`,
 		},
@@ -581,7 +588,11 @@ func TestEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
 func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
 	t.Parallel()
 
-	const fetched = `"fetched": "2026-10-06T00:00:00Z"`
+	const (
+		drop    = `"url": "` + blocklistURL + `", `
+		tried   = `"tried": "2026-10-06T00:00:00Z", `
+		fetched = `"fetched": "2026-10-06T00:00:00Z"`
+	)
 
 	for _, tc := range []struct {
 		name, content string
@@ -589,20 +600,25 @@ func TestReputationJSONEntryWithoutAFieldItNeedsStopsTheStart(t *testing.T) {
 		want string
 	}{
 		{
-			"a copy of a list without its URL",
-			`{"version": 1, "lists": [{` + fetched + `, "lines": []}]}`,
+			"a list without its URL",
+			`{"version": 1, "lists": [{` + tried + fetched + `, "lines": []}]}`,
 			`: entry 1 has no "url"`,
 		},
+		{
+			"a list without the time it was last tried",
+			`{"version": 1, "lists": [{` + drop + fetched + `, "lines": []}]}`,
+			`: entry 1 has no "tried"`,
+		},
 		{
 			"a copy of a list without the time it was fetched",
-			`{"version": 1, "lists": [{"url": "` + blocklistURL + `", "lines": []}]}`,
+			`{"version": 1, "lists": [{` + drop + tried + `"lines": []}]}`,
 			`: entry 1 has no "fetched"`,
 		},
 		{
 			// An empty list has no lines, which is not having none.
 			"a copy of a list without its lines",
-			`{"version": 1, "lists": [{"url": "` + blocklistURL + `", ` + fetched +
-				`, "lines": []}, {"url": "https://lists.example/tor.txt", ` + fetched + `}]}`,
+			`{"version": 1, "lists": [{` + drop + tried + fetched + `, "lines": []}, ` +
+				`{"url": "` + torURL + `", ` + tried + fetched + `}]}`,
 			`: entry 2 has no "lines"`,
 		},
 	} {
@@ -1116,7 +1132,8 @@ func TestEditOfEachFileTakenIn(t *testing.T) {
 		[]lookup.Answer{{Client: client, Country: "FR", Answered: midnight()}})
 
 	edit(t, dir, reputationJSON, `{"version": 1, "lists": [{"url": "`+blocklistURL+`", `+
-		`"fetched": "2026-10-06T00:00:00Z", "lines": ["198.51.100.7"]}]}`)
+		`"tried": "2026-10-06T00:00:00Z", "fetched": "2026-10-06T00:00:00Z", `+
+		`"lines": ["198.51.100.7"]}]}`)
 	wantTakenIn(t, lines, dir, reputationJSON)
 
 	listedBy := params.Lists.ListedBy(client.Addr())
@@ -1506,8 +1523,8 @@ func midnight() time.Time {
 }
 
 // newParams returns Params for the state files in dir, with parts that
-// hold nothing yet. GeoJS is never asked, the lists, of which there is one
-// blocklist, are never fetched, and the alerts, at most two an hour, are
+// hold nothing yet. GeoJS is never asked, the lists, two blocklists, are
+// never fetched, and the alerts, at most two an hour, are
 // never sent. The anomaly counters count the scopes fill counts, with
 // thresholds fill does not reach.
 func newParams(dir string) state.Params {
@@ -1538,7 +1555,7 @@ func newParams(dir string) state.Params {
 			Now: midnight, ProcessLog: discard, Metrics: m,
 		}),
 		Lists: reputation.New(reputation.Params{
-			BlocklistURLs: []string{blocklistURL}, Refresh: 24 * time.Hour,
+			BlocklistURLs: []string{blocklistURL, torURL}, Refresh: 24 * time.Hour,
 			Now: midnight, ProcessLog: discard, Alerts: queue,
 		}),
 		Alerts: queue,
@@ -1565,9 +1582,9 @@ func office() netip.Prefix {
 
 // fill puts a permanent ban an admin made, a ban for a broken limit and
 // one for a clear sign of attack, clients with counts and histories,
-// GeoJS answers, a copy of the blocklist, as filledReputationJSON holds
-// it, and alerts and anomaly counters, as filledAlertsJSON holds them,
-// into the parts of params.
+// GeoJS answers, the blocklists' last tries and the copy of one, as
+// filledReputationJSON holds them, and alerts and anomaly counters, as
+// filledAlertsJSON holds them, into the parts of params.
 func fill(params state.Params) {
 	now := midnight()
 	client := netip.MustParsePrefix("203.0.113.9/32")
@@ -1600,14 +1617,16 @@ func fill(params state.Params) {
 		},
 	})
 
+	// The copy of drop.txt was fetched an hour ago, and the fetches of it
+	// and of tor.txt tried since failed.
 	err := params.Lists.Load([]reputation.List{{
-		URL: blocklistURL, Fetched: now.Add(-time.Hour),
+		URL: blocklistURL, Tried: now, Fetched: now.Add(-time.Hour),
 		Lines: []string{
 			"; Spamhaus DROP List 2026/10/05 - (c) 2026 The Spamhaus Project SLL",
 			"203.0.113.0/24 ; SBL1",
 			"2001:db8::/32 ; SBL2",
 		},
-	}})
+	}, {URL: torURL, Tried: now}})
 	if err != nil {
 		panic(err) // the copy reads
 	}