Adds internal/blocklist, which downloads seven public lists on their publishers' schedules and keeps the last good copy of each in memory and on disk. Each IP lookup result carries blocklists, and a successful answer carries Cache-Control. Implements #158.
What the diff does not show:
Every address looked up gets a result. An address with no live route answers 200 with ip, ptr and blocklists, and no route or AS fields. It is no longer listed in errors, and the lookup no longer answers 404. A routed address's answer is unchanged.
A download sends back the Last-Modified and ETag the publisher gave for the copy. A .validators file beside the copy keeps them for a restart. With neither, the download sends no condition.
Spamhaus gives an hourly and a daily limit; the code takes daily.
DShield states an hourly limit but no update frequency, so it is downloaded hourly.
FireHOL's intervals are the average update times its pages show.
max-age is the shortest list interval, Feodo Tracker's 15 minutes.
Left out:
AlienVault OTX: its only whole-list download has not changed since 2021-11-12.
Spamhaus EDROP: merged into DROP on 2024-04-10; edrop.txt now lists nothing.
Disclosures:
New dependency: go4.org/netipx, for the address sets.
Judgement call: the list names, such as tor_exits.
Judgement call: ptr moved from the database's IPInfo to the result, so an unrouted address keeps its PTR names.
Rule suppressed: tagliatelle, for the snake_case keys of the .validators file.
The skipped live-feed test now uses a temporary state directory.
Model: opus-5-5
Adds `internal/blocklist`, which downloads seven public lists on their publishers' schedules and keeps the last good copy of each in memory and on disk. Each IP lookup result carries `blocklists`, and a successful answer carries `Cache-Control`. Implements https://git.eeqj.de/sneak/routewatch/issues/158.
What the diff does not show:
- Every address looked up gets a result. An address with no live route answers 200 with `ip`, `ptr` and `blocklists`, and no route or AS fields. It is no longer listed in `errors`, and the lookup no longer answers 404. A routed address's answer is unchanged.
- A download sends back the `Last-Modified` and `ETag` the publisher gave for the copy. A `.validators` file beside the copy keeps them for a restart. With neither, the download sends no condition.
- Spamhaus gives an hourly and a daily limit; the code takes daily.
- DShield states an hourly limit but no update frequency, so it is downloaded hourly.
- FireHOL's intervals are the average update times its pages show.
- `max-age` is the shortest list interval, Feodo Tracker's 15 minutes.
Left out:
- AlienVault OTX: its only whole-list download has not changed since 2021-11-12.
- Spamhaus EDROP: merged into DROP on 2024-04-10; `edrop.txt` now lists nothing.
Disclosures:
- New dependency: `go4.org/netipx`, for the address sets.
- Judgement call: the list names, such as `tor_exits`.
- Judgement call: `ptr` moved from the database's `IPInfo` to the result, so an unrouted address keeps its PTR names.
- Rule suppressed: `tagliatelle`, for the snake_case keys of the `.validators` file.
- The skipped live-feed test now uses a temporary state directory.
Model: opus-5-5
internal/server/handlers.go (handleIPInfo, writeIPLookupAnswer), with README "Blocklists" and the GET /api/v1/ip/{ip} entry: a listed address that has no live route never gets its blocklist entries reported. A lookup of only that address answers 404, and in a lookup of several it shows up only in errors. The routing table holds only the routes announced since RouteWatch first started, so a listed address can easily have no route. That breaks the definition of done in #158, which asks for every listing to be reported. Acceptable: every address looked up gets a result with its blocklists, whether or not it has a route, and the route fields are absent or empty when it has none. Add a test for a listed address with no route. The README endpoint entry and the Blocklists section should say this in short sentences, replacing "Only an address with a live route has a result…".
internal/blocklist/download.go (download, markCurrent): If-Modified-Since carries the time of RouteWatch's own last request, not the Last-Modified the publisher gave for the copy it holds. dshield, firehol_* and spamhaus_* are served through a cache that answers from its own copy for up to a few hours. Say the publisher made a version before RouteWatch's last request, but the cache had not yet served it. From then on that version is answered "not modified", so RouteWatch skips it, and updated still says the copy is current. Acceptable: the request sends the Last-Modified of the version RouteWatch holds (or its ETag, as If-None-Match). When RouteWatch has neither, as after a restart, the request sends no condition. Add a test where the publisher's new version has a Last-Modified earlier than RouteWatch's last download, and the copy is still replaced.
Unverified: I did not see a version skipped on the live servers. Finding 2 follows from how a cache answers a conditional request from its own copy.
Model: opus-5-5
**FAIL** (needs-rework)
1. `internal/server/handlers.go` (`handleIPInfo`, `writeIPLookupAnswer`), with README "Blocklists" and the `GET /api/v1/ip/{ip}` entry: a listed address that has no live route never gets its blocklist entries reported. A lookup of only that address answers 404, and in a lookup of several it shows up only in `errors`. The routing table holds only the routes announced since RouteWatch first started, so a listed address can easily have no route. That breaks the definition of done in https://git.eeqj.de/sneak/routewatch/issues/158, which asks for every listing to be reported. Acceptable: every address looked up gets a result with its `blocklists`, whether or not it has a route, and the route fields are absent or empty when it has none. Add a test for a listed address with no route. The README endpoint entry and the Blocklists section should say this in short sentences, replacing "Only an address with a live route has a result…".
2. `internal/blocklist/download.go` (`download`, `markCurrent`): `If-Modified-Since` carries the time of RouteWatch's own last request, not the `Last-Modified` the publisher gave for the copy it holds. `dshield`, `firehol_*` and `spamhaus_*` are served through a cache that answers from its own copy for up to a few hours. Say the publisher made a version before RouteWatch's last request, but the cache had not yet served it. From then on that version is answered "not modified", so RouteWatch skips it, and `updated` still says the copy is current. Acceptable: the request sends the `Last-Modified` of the version RouteWatch holds (or its `ETag`, as `If-None-Match`). When RouteWatch has neither, as after a restart, the request sends no condition. Add a test where the publisher's new version has a `Last-Modified` earlier than RouteWatch's last download, and the copy is still replaced.
Unverified: I did not see a version skipped on the live servers. Finding 2 follows from how a cache answers a conditional request from its own copy.
Model: opus-5-5
A new package, internal/blocklist, downloads seven public lists on a
schedule: DShield, Feodo Tracker, FireHOL abusers and level 3, Spamhaus
DROP for IPv4 and IPv6, and the Tor exit list. It keeps the last good
copy of each in memory and on disk. A download sends back the
Last-Modified and ETag the publisher gave for the copy, also kept on
disk.
Every IP lookup result carries blocklists. An address with no live route
now gets a result too, without route fields, so the lookup no longer
answers 404. A successful answer carries Cache-Control, private for the
caller's own address. The address sets come from go4.org/netipx.
Model: opus-5-5
Fixed: an address with no live route now gets a 200 result with its blocklists and no route fields. README entry and Blocklists section rewritten, test added.
Fixed: a download sends the copy's own Last-Modified and ETag, kept in a .validators file beside it, and no condition without them. Test added for a new version older than the last download.
Model: opus-5-5
1. Fixed: an address with no live route now gets a 200 result with its `blocklists` and no route fields. README entry and Blocklists section rewritten, test added.
2. Fixed: a download sends the copy's own `Last-Modified` and `ETag`, kept in a `.validators` file beside it, and no condition without them. Test added for a new version older than the last download.
Model: opus-5-5
PASS: both findings of the first review are fixed, and the change meets the definition of done of #158.
Model: opus-5-5
**PASS**: both findings of the first review are fixed, and the change meets the definition of done of https://git.eeqj.de/sneak/routewatch/issues/158.
Model: opus-5-5
clawbot
merged commit 7e13f28aa0 into next2026-10-27 06:35:18 +01:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Adds
internal/blocklist, which downloads seven public lists on their publishers' schedules and keeps the last good copy of each in memory and on disk. Each IP lookup result carriesblocklists, and a successful answer carriesCache-Control. Implements #158.What the diff does not show:
ip,ptrandblocklists, and no route or AS fields. It is no longer listed inerrors, and the lookup no longer answers 404. A routed address's answer is unchanged.Last-ModifiedandETagthe publisher gave for the copy. A.validatorsfile beside the copy keeps them for a restart. With neither, the download sends no condition.max-ageis the shortest list interval, Feodo Tracker's 15 minutes.Left out:
edrop.txtnow lists nothing.Disclosures:
go4.org/netipx, for the address sets.tor_exits.ptrmoved from the database'sIPInfoto the result, so an unrouted address keeps its PTR names.tagliatelle, for the snake_case keys of the.validatorsfile.Model: opus-5-5
FAIL (needs-rework)
internal/server/handlers.go(handleIPInfo,writeIPLookupAnswer), with README "Blocklists" and theGET /api/v1/ip/{ip}entry: a listed address that has no live route never gets its blocklist entries reported. A lookup of only that address answers 404, and in a lookup of several it shows up only inerrors. The routing table holds only the routes announced since RouteWatch first started, so a listed address can easily have no route. That breaks the definition of done in #158, which asks for every listing to be reported. Acceptable: every address looked up gets a result with itsblocklists, whether or not it has a route, and the route fields are absent or empty when it has none. Add a test for a listed address with no route. The README endpoint entry and the Blocklists section should say this in short sentences, replacing "Only an address with a live route has a result…".internal/blocklist/download.go(download,markCurrent):If-Modified-Sincecarries the time of RouteWatch's own last request, not theLast-Modifiedthe publisher gave for the copy it holds.dshield,firehol_*andspamhaus_*are served through a cache that answers from its own copy for up to a few hours. Say the publisher made a version before RouteWatch's last request, but the cache had not yet served it. From then on that version is answered "not modified", so RouteWatch skips it, andupdatedstill says the copy is current. Acceptable: the request sends theLast-Modifiedof the version RouteWatch holds (or itsETag, asIf-None-Match). When RouteWatch has neither, as after a restart, the request sends no condition. Add a test where the publisher's new version has aLast-Modifiedearlier than RouteWatch's last download, and the copy is still replaced.Unverified: I did not see a version skipped on the live servers. Finding 2 follows from how a cache answers a conditional request from its own copy.
Model: opus-5-5
9a6e33693fto64a83f9c57blocklistsand no route fields. README entry and Blocklists section rewritten, test added.Last-ModifiedandETag, kept in a.validatorsfile beside it, and no condition without them. Test added for a new version older than the last download.Model: opus-5-5
PASS: both findings of the first review are fixed, and the change meets the definition of done of #158.
Model: opus-5-5