Rename blocklists to IP lists, add IP lookup flags, raise the download limit to 200 MiB (closes #233)
check / check (push) Failing after 10m45s

The lists are not only used for blocking, so the package, the `blocklists` field of the IP lookup answer and the state directory are now IP lists (`internal/iplist`, `ip_lists`, `iplists`). Every result gains a `flags` object with `is_tor` and `is_bogon`. The largest list file accepted is 200 MiB.

Judgement call: the dated 2026-10-27 `TODO.md` entry keeps the old word.
Rule suppressed: `tagliatelle` on the two result structs, for snake_case keys.

Model: opus-5-5
This commit was merged in pull request #241.
This commit is contained in:
clawbot committed 2026-11-10 10:48:23 +01:00
1 parent e58c6635e7
commit cf50d195ad
12 files changed
+375 -176

No files matched your search

+19 -16
View File
@@ -44,8 +44,7 @@ its abuse contact. RouteWatch keeps them in its database and fetches them again
once they are 30 days old. An IP lookup answers from the stored details and
never waits for WHOIS.
An IP lookup also says which public blocklists list the address (see
Blocklists).
An IP lookup also says which public IP lists hold the address (see IP lists).
## Getting started
@@ -65,7 +64,7 @@ docker run --memory=5g --memory-swap=5g -p 8080:8080 \
```
The memory limit is explained in Memory. The volume holds the database and the
copies of the blocklists.
copies of the IP lists.
To run the binary on the host instead, build it with `make build`. The build
needs Go and a C compiler, because the SQLite library is written in C:
@@ -284,8 +283,12 @@ address. Every entry has the same fields:
- `ip`: the address
- `ptr`: the address's PTR names, the host names that DNS gives for it, or an
empty list when it has none
- `blocklists`: the public blocklists the address is on (see Blocklists), or an
empty list when it is on none
- `flags`: yes-or-no facts about the address:
- `is_tor`: true when the address is on the `tor_exits` list
- `is_bogon`: true when the address is reserved, so that no network should
announce it on the internet
- `ip_lists`: the public IP lists the address is on (see IP lists), or an empty
list when it is on none
Every other field is about the prefix in the routing table that holds the
address. Each is null when no prefix in the routing table holds the address:
@@ -331,7 +334,7 @@ Each PTR lookup gets 2 seconds. An address whose lookup takes longer is answered
without PTR names.
A successful answer carries `Cache-Control` with a `max-age` of 15 minutes, the
shortest blocklist interval. The answer for the caller's own address (`/ip/`) is
shortest IP list interval. The answer for the caller's own address (`/ip/`) is
`private`, and the others are `public`.
### Errors
@@ -391,10 +394,10 @@ inline `<style>` and `<script>` blocks. Each answer's policy names a random
value, a nonce, made for that answer. The browser applies or runs only the
blocks that carry it.
## Blocklists
## IP lists
An IP lookup reports which public blocklists list each address. Each entry in
`blocklists` has:
An IP lookup reports which public IP lists hold each address. Each entry in
`ip_lists` has:
- `name`: the list's name, from the lists below
- `description`: what being on the list means
@@ -431,8 +434,8 @@ can be downloaded whole without an account or a key.
How RouteWatch keeps the lists:
- It holds each list in memory, so a lookup makes no outside request.
- It keeps a copy of each list in the `blocklists` directory of the state
directory (see Configuration). Each copy is a file named after its list.
- It keeps a copy of each list in the `iplists` directory of the state directory
(see Configuration). Each copy is a file named after its list.
- Beside each copy, a file ending in `.validators` keeps the `Last-Modified` and
`ETag` headers the publisher sent with that copy.
- At startup it loads each list from its copy. It downloads a list at once when
@@ -440,7 +443,7 @@ How RouteWatch keeps the lists:
- A download sends the copy's `Last-Modified` as `If-Modified-Since` and its
`ETag` as `If-None-Match`. The publisher then sends the list only when it has
changed.
- A download fails when it takes over a minute or its file is over 16 MiB.
- A download fails when it takes over a minute or its file is over 200 MiB.
- When a download fails, or its file does not parse, RouteWatch keeps the
previous copy and logs the failure. It tries again one interval later.
@@ -492,7 +495,7 @@ by the client. For a request from any other address, they use the address the
request came from, whatever headers it carries. `X-Real-IP` and `True-Client-IP`
are not read.
The state directory holds the database, `db.sqlite`, and the blocklist copies:
The state directory holds the database, `db.sqlite`, and the IP list copies:
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
- Linux and the BSDs, as root: `/var/lib/berlin.sneak.app.routewatch/`
@@ -647,7 +650,7 @@ The parts:
wait starts at 15 seconds. It halves after a success, down to 1 second, and
grows by half after a failure, up to a minute. An IP lookup that finds an AS
with old or missing details asks the fetcher for it at once, without the wait.
- The blocklists part downloads the blocklists and answers which lists hold an
- The IP lists part downloads the IP lists and answers which lists hold an
address.
- The database maintainer checkpoints the database every 5 seconds: it folds the
write-ahead log, where SQLite first writes changes, into the database file. It
@@ -756,7 +759,7 @@ On `SIGTERM` or `SIGINT` the daemon stops its parts in this order:
2. the handlers, each of which writes what it holds
3. the WHOIS fetcher
4. the database maintainer
5. the blocklist downloads
5. the IP list downloads
6. the HTTP server
Last, the daemon checkpoints and closes the database. It gives all this 8
@@ -790,10 +793,10 @@ routewatch/
├── docs/
│ └── message-examples.json # Examples of RIS Live messages
├── internal/
│ ├── blocklist/ # Public IP blocklists: downloads and lookups
│ ├── bogon/ # Reserved prefixes and AS numbers
│ ├── config/ # Settings from the environment
│ ├── database/ # SQLite storage
│ ├── iplist/ # Public IP lists: downloads and lookups
│ ├── logger/ # Structured logging
│ ├── metrics/ # Feed and route update figures
│ ├── ristypes/ # The RIS Live message types