Report which public blocklists list an address in the IP lookup (closes #158)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
A new package, internal/blocklist, downloads seven public lists on a schedule: DShield, Feodo Tracker, FireHOL abusers and level 3, Spamhaus DROP for IPv4 and IPv6, and the Tor exit list. It keeps the last good copy of each in memory and in the state directory's blocklists directory, and loads the copies at startup. Each list's interval follows its publisher's page, cited in the code. Each IP lookup result carries blocklists, one entry per list the address is on. A successful lookup answer carries Cache-Control, private for the caller's own address. The address sets come from go4.org/netipx. Model: opus-5-5
This commit is contained in:
@@ -27,6 +27,8 @@ route the feed has withdrawn.
|
||||
- Maintains live IPv4 and IPv6 routing tables
|
||||
- Tracks AS peering relationships
|
||||
- HTTP API for IP-to-AS lookups, prefix details, and AS information
|
||||
- IP lookups report the public blocklists an address is on, from copies kept in
|
||||
memory and on disk
|
||||
- Automatic reconnection with exponential backoff
|
||||
- Batched database writes for high performance
|
||||
- Backpressure handling to prevent memory exhaustion
|
||||
@@ -87,16 +89,18 @@ Every route answers `HEAD` as it answers `GET`, without the body.
|
||||
last full clock minute.
|
||||
- `GET /api/v1/ip/{ip}` - Looks up each address: its most specific live route
|
||||
(`netblock`), that route's origin AS with its WHOIS details, and the address's
|
||||
PTR names. `{ip}` is an IP address, a host name, or a comma-separated list of
|
||||
at most 10 of these. Of the addresses a host name resolves to, only public
|
||||
ones are looked up: a name with none, such as one that resolves to a private
|
||||
address, is reported as `host name has no public address`, without its
|
||||
addresses. `results` has one entry per address with a live route and `errors`
|
||||
one per address or name that failed. The answer is 400 when `{ip}` lists more
|
||||
than 10, when it comes to more than 10 addresses once its names are resolved,
|
||||
or when nothing in it is an IP address or a host name with a public address;
|
||||
and 404 when none of the addresses has a live route. A PTR lookup gets 2
|
||||
seconds; an address whose lookup takes longer is answered without PTR names.
|
||||
PTR names. Each result also lists the public blocklists the address is on, in
|
||||
`blocklists` (see Blocklists). `{ip}` is an IP address, a host name, or a
|
||||
comma-separated list of at most 10 of these. Of the addresses a host name
|
||||
resolves to, only public ones are looked up: a name with none, such as one
|
||||
that resolves to a private address, is reported as
|
||||
`host name has no public address`, without its addresses. `results` has one
|
||||
entry per address with a live route and `errors` one per address or name that
|
||||
failed. The answer is 400 when `{ip}` lists more than 10, when it comes to
|
||||
more than 10 addresses once its names are resolved, or when nothing in it is
|
||||
an IP address or a host name with a public address; and 404 when none of the
|
||||
addresses has a live route. A PTR lookup gets 2 seconds; an address whose
|
||||
lookup takes longer is answered without PTR names.
|
||||
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
|
||||
apart, each with when its newest route was updated. When the database has no
|
||||
record of the AS, its handle and description come from the AS list built into
|
||||
@@ -150,6 +154,63 @@ in a prefix's `origins`) means the route has no known origin: its AS path ends
|
||||
in an AS set of several members, any of which may have originated it. The pages
|
||||
show such a route as having no known origin.
|
||||
|
||||
## Blocklists
|
||||
|
||||
The IP lookup reports which public blocklists list each address. Each result has
|
||||
a `blocklists` list with one entry for each list the address is on:
|
||||
|
||||
- `name`: the list's name, from the lists below
|
||||
- `description`: what being on the list means
|
||||
- `updated`: when RouteWatch last downloaded the list, or last heard from its
|
||||
publisher that its copy was current
|
||||
|
||||
An address on no list has an empty `blocklists`. Only an address with a live
|
||||
route has a result, so an address without one gets no blocklists either.
|
||||
|
||||
The lists:
|
||||
|
||||
- `dshield`: the 20 networks that DShield saw scan the most targets in the last
|
||||
three days. From [DShield](https://isc.sans.edu/feeds_doc.html), downloaded
|
||||
every hour.
|
||||
- `feodo_tracker`: botnet command-and-control servers. From abuse.ch
|
||||
[Feodo Tracker](https://feodotracker.abuse.ch/blocklist/), downloaded every 15
|
||||
minutes.
|
||||
- `firehol_abusers_30d`: addresses reported for comment, form or forum spam in
|
||||
the last 30 days. From
|
||||
[FireHOL](https://iplists.firehol.org/?ipset=firehol_abusers_30d), downloaded
|
||||
every 5 hours.
|
||||
- `firehol_level3`: addresses reported for attacks, spyware or viruses in the
|
||||
last 30 days. From
|
||||
[FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every
|
||||
3 hours.
|
||||
- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks that Spamhaus
|
||||
says spammers or cyber criminals have hijacked or lease. From
|
||||
[Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/),
|
||||
downloaded once a day. Spamhaus merged its EDROP list into DROP.
|
||||
- `tor_exits`: Tor exit nodes, where traffic from the Tor network leaves. From
|
||||
the [Tor Project](https://check.torproject.org/torbulkexitlist), downloaded
|
||||
every hour.
|
||||
|
||||
No list is downloaded more often than its publisher updates it or allows. Each
|
||||
can be downloaded whole without an account or a key.
|
||||
|
||||
How RouteWatch keeps the lists:
|
||||
|
||||
- It holds each list in memory, so a lookup makes no outside request.
|
||||
- It keeps a copy of each list in the `blocklists` directory of the state
|
||||
directory (see Configuration). Each copy is a file named after its list.
|
||||
- At startup it loads each list from its copy. It downloads a list at once when
|
||||
the copy is older than the list's interval, or when there is no copy.
|
||||
- A download sends `If-Modified-Since`, so a publisher does not send a list that
|
||||
has not changed.
|
||||
- A download fails when it takes over a minute or its file is over 16 MiB.
|
||||
- When a download fails, or its file does not parse, RouteWatch keeps the
|
||||
previous copy and logs the failure. It tries again one interval later.
|
||||
|
||||
A successful IP lookup answer carries `Cache-Control` with a `max-age` of 15
|
||||
minutes, the shortest list interval. The answer for the caller's own address
|
||||
(`/ip/`) is `private`, and the others are `public`.
|
||||
|
||||
## Code Structure
|
||||
|
||||
```
|
||||
@@ -159,6 +220,7 @@ routewatch/
|
||||
│ ├── asinfo-gen/ # Utility to generate AS info data
|
||||
│ └── streamdumper/ # Debug utility for raw stream output
|
||||
├── internal/
|
||||
│ ├── blocklist/ # Public IP blocklists: downloads and lookups
|
||||
│ ├── routewatch/ # Core application logic
|
||||
│ ├── server/ # HTTP server and handlers
|
||||
│ ├── database/ # SQLite storage layer
|
||||
@@ -325,7 +387,7 @@ client sent. For a request from any other address they use the address the
|
||||
request came from, whatever forwarding headers it carries. `X-Real-IP` and
|
||||
`True-Client-IP` are not read.
|
||||
|
||||
State directory (database location):
|
||||
State directory (the database and the blocklist copies):
|
||||
|
||||
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
|
||||
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
|
||||
|
||||
Reference in New Issue
Block a user