From 64a83f9c573f3eda5207cb6371f818ebfaa65f1f Mon Sep 17 00:00:00 2001 From: sneak Date: Tue, 27 Oct 2026 04:50:03 +0000 Subject: [PATCH] Report which public blocklists list an address in the IP lookup (closes #158) A new package, internal/blocklist, downloads seven public lists on a schedule: DShield, Feodo Tracker, FireHOL abusers and level 3, Spamhaus DROP for IPv4 and IPv6, and the Tor exit list. It keeps the last good copy of each in memory and on disk. A download sends back the Last-Modified and ETag the publisher gave for the copy, also kept on disk. Every IP lookup result carries blocklists. An address with no live route now gets a result too, without route fields, so the lookup no longer answers 404. A successful answer carries Cache-Control, private for the caller's own address. The address sets come from go4.org/netipx. Model: opus-5-5 --- README.md | 97 +++- TODO.md | 7 + go.mod | 1 + go.sum | 2 + internal/blocklist/blocklist.go | 184 +++++++ internal/blocklist/blocklist_internal_test.go | 494 ++++++++++++++++++ internal/blocklist/download.go | 379 ++++++++++++++ internal/blocklist/lists.go | 93 ++++ internal/blocklist/parse.go | 158 ++++++ internal/database/models.go | 11 +- internal/routewatch/app.go | 34 +- .../app_integration_internal_test.go | 4 +- internal/server/handlers.go | 106 +++- internal/server/handlers_internal_test.go | 208 +++++++- internal/server/server.go | 15 + internal/templates/index.html | 4 +- 16 files changed, 1736 insertions(+), 61 deletions(-) create mode 100644 internal/blocklist/blocklist.go create mode 100644 internal/blocklist/blocklist_internal_test.go create mode 100644 internal/blocklist/download.go create mode 100644 internal/blocklist/lists.go create mode 100644 internal/blocklist/parse.go diff --git a/README.md b/README.md index e3d89a9..9a1a64f 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,8 @@ route the feed has withdrawn. - Maintains live IPv4 and IPv6 routing tables - Tracks AS peering relationships - HTTP API for IP-to-AS lookups, prefix details, and AS information +- IP lookups report the public blocklists an address is on, from copies kept in + memory and on disk - Automatic reconnection with exponential backoff - Batched database writes for high performance - Backpressure handling to prevent memory exhaustion @@ -85,18 +87,26 @@ Every route answers `HEAD` as it answers `GET`, without the body. collector, stream and handler queue statistics. `stream.route_churn_per_min` is the number of prefix announcements and withdrawals, from all peers, in the last full clock minute. -- `GET /api/v1/ip/{ip}` - Looks up each address: its most specific live route - (`netblock`), that route's origin AS with its WHOIS details, and the address's - PTR names. `{ip}` is an IP address, a host name, or a comma-separated list of - at most 10 of these. Of the addresses a host name resolves to, only public - ones are looked up: a name with none, such as one that resolves to a private - address, is reported as `host name has no public address`, without its - addresses. `results` has one entry per address with a live route and `errors` - one per address or name that failed. The answer is 400 when `{ip}` lists more - than 10, when it comes to more than 10 addresses once its names are resolved, - or when nothing in it is an IP address or a host name with a public address; - and 404 when none of the addresses has a live route. A PTR lookup gets 2 - seconds; an address whose lookup takes longer is answered without PTR names. +- `GET /api/v1/ip/{ip}` - Looks up each address. `{ip}` is an IP address, a host + name, or a comma-separated list of at most 10 of these. `results` has one + entry for each address, with: + - `ip` and `ptr`: the address and its PTR names + - `netblock` and the fields beside it: the address's most specific live + route, and that route's origin AS with its WHOIS details + - `blocklists`: the public blocklists the address is on (see Blocklists) + + An address with no live route has a result without the route and AS fields. + Of the addresses a host name resolves to, only public ones are looked up. A + name with none, such as one that resolves to a private address, is reported + as `host name has no public address`, without its addresses. `errors` has + one entry for each host name that failed. The answer is 400 when: + - `{ip}` lists more than 10 + - `{ip}` comes to more than 10 addresses once its names are resolved + - nothing in `{ip}` is an IP address or a host name with a public address + + A PTR lookup gets 2 seconds. An address whose lookup takes longer is + answered without PTR names. + - `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6 apart, each with when its newest route was updated. When the database has no record of the AS, its handle and description come from the AS list built into @@ -150,6 +160,66 @@ in a prefix's `origins`) means the route has no known origin: its AS path ends in an AS set of several members, any of which may have originated it. The pages show such a route as having no known origin. +## Blocklists + +The IP lookup reports which public blocklists list each address. Each result has +a `blocklists` list with one entry for each list the address is on: + +- `name`: the list's name, from the lists below +- `description`: what being on the list means +- `updated`: when RouteWatch last downloaded the list, or last heard from its + publisher that its copy was current + +An address on no list has an empty `blocklists`. An address with no live route +still has a result with its `blocklists`. + +The lists: + +- `dshield`: the 20 networks that DShield saw scan the most targets in the last + three days. From [DShield](https://isc.sans.edu/feeds_doc.html), downloaded + every hour. +- `feodo_tracker`: botnet command-and-control servers. From abuse.ch + [Feodo Tracker](https://feodotracker.abuse.ch/blocklist/), downloaded every 15 + minutes. +- `firehol_abusers_30d`: addresses reported for comment, form or forum spam in + the last 30 days. From + [FireHOL](https://iplists.firehol.org/?ipset=firehol_abusers_30d), downloaded + every 5 hours. +- `firehol_level3`: addresses reported for attacks, spyware or viruses in the + last 30 days. From + [FireHOL](https://iplists.firehol.org/?ipset=firehol_level3), downloaded every + 3 hours. +- `spamhaus_drop` and `spamhaus_dropv6`: IPv4 and IPv6 networks that Spamhaus + says spammers or cyber criminals have hijacked or lease. From + [Spamhaus DROP](https://www.spamhaus.org/blocklists/do-not-route-or-peer/), + downloaded once a day. Spamhaus merged its EDROP list into DROP. +- `tor_exits`: Tor exit nodes, where traffic from the Tor network leaves. From + the [Tor Project](https://check.torproject.org/torbulkexitlist), downloaded + every hour. + +No list is downloaded more often than its publisher updates it or allows. Each +can be downloaded whole without an account or a key. + +How RouteWatch keeps the lists: + +- It holds each list in memory, so a lookup makes no outside request. +- It keeps a copy of each list in the `blocklists` directory of the state + directory (see Configuration). Each copy is a file named after its list. +- Beside each copy, a file ending in `.validators` keeps the `Last-Modified` and + `ETag` the publisher sent with that copy. +- At startup it loads each list from its copy. It downloads a list at once when + the copy is older than the list's interval, or when there is no copy. +- A download sends the copy's `Last-Modified` as `If-Modified-Since` and its + `ETag` as `If-None-Match`. The publisher then sends the list only when it has + changed. Without either, the download asks for the whole list. +- A download fails when it takes over a minute or its file is over 16 MiB. +- When a download fails, or its file does not parse, RouteWatch keeps the + previous copy and logs the failure. It tries again one interval later. + +A successful IP lookup answer carries `Cache-Control` with a `max-age` of 15 +minutes, the shortest list interval. The answer for the caller's own address +(`/ip/`) is `private`, and the others are `public`. + ## Code Structure ``` @@ -159,6 +229,7 @@ routewatch/ │ ├── asinfo-gen/ # Utility to generate AS info data │ └── streamdumper/ # Debug utility for raw stream output ├── internal/ +│ ├── blocklist/ # Public IP blocklists: downloads and lookups │ ├── routewatch/ # Core application logic │ ├── server/ # HTTP server and handlers │ ├── database/ # SQLite storage layer @@ -325,7 +396,7 @@ client sent. For a request from any other address they use the address the request came from, whatever forwarding headers it carries. `X-Real-IP` and `True-Client-IP` are not read. -State directory (database location): +State directory (the database and the blocklist copies): - macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/` - Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise diff --git a/TODO.md b/TODO.md index 9b76298..35ecbbd 100644 --- a/TODO.md +++ b/TODO.md @@ -33,6 +33,13 @@ tested. # Completed Steps +- 2026-10-27: each result of the IP lookup lists the public blocklists the + address is on, in `blocklists`. The lists are DShield, Feodo Tracker, two + FireHOL lists, Spamhaus DROP and the Tor exit list. RouteWatch downloads each + list on its own schedule and answers from copies in memory. The copies are + also kept in the state directory's `blocklists` directory. IP lookup answers + carry `Cache-Control`. An address with no live route now has a result too, + without route fields, so the IP lookup no longer answers 404 (closes #158) - 2026-10-27: the home page has an IP information API card. It shows the `curl` command that looks up the visitor's own address at `/ip/{addr}`, and that command's live answer (closes #156) diff --git a/go.mod b/go.mod index e2f91ee..95b8520 100644 --- a/go.mod +++ b/go.mod @@ -9,6 +9,7 @@ require ( github.com/mattn/go-sqlite3 v1.14.29 github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 go.uber.org/fx v1.24.0 + go4.org/netipx v0.0.0-20260823151212-3075585bcbeb golang.org/x/term v0.33.0 golang.org/x/text v0.43.0 ) diff --git a/go.sum b/go.sum index 64fe7a1..20e604d 100644 --- a/go.sum +++ b/go.sum @@ -24,6 +24,8 @@ go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ= go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo= go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= +go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs= +go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y= golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA= golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg= diff --git a/internal/blocklist/blocklist.go b/internal/blocklist/blocklist.go new file mode 100644 index 0000000..9e16898 --- /dev/null +++ b/internal/blocklist/blocklist.go @@ -0,0 +1,184 @@ +// Package blocklist downloads public IP blocklists on a schedule, keeps the +// last good copy of each on disk and in memory, and answers which lists +// contain an address. +package blocklist + +import ( + "context" + "log/slog" + "net/netip" + "sync" + "time" + + "go4.org/netipx" +) + +// Format is how a list's file is written. +type Format int + +const ( + // FormatLines has one address or network per line, such as 192.0.2.1 or + // 192.0.2.0/24. Lines that start with # are comments. + FormatLines Format = iota + + // FormatSpamhausJSON has one JSON object per line, with a network in its + // cidr field. The last line holds the file's metadata and no network. + FormatSpamhausJSON + + // FormatDShield has one network per line, as tab-separated fields that + // start with the network's first and last address. Lines that start + // with # are comments. + FormatDShield +) + +// List is one public blocklist. +type List struct { + // Name names the list in lookup answers, and names its copy on disk. + Name string + + // Description says in one line what being on the list means. + Description string + + // URL is where the publisher serves the whole list. + URL string + + // Interval is how long a copy is used before the list is downloaded + // again. + Interval time.Duration + + // Format is how the list's file is written. + Format Format +} + +// Listing says that a list contains an address. +type Listing struct { + Name string `json:"name"` + Description string `json:"description"` + + // Updated is when routewatch last downloaded the list, or last heard from + // its publisher that its copy was still current. + Updated time.Time `json:"updated"` +} + +// listCopy is a list's last good copy in memory. +type listCopy struct { + addresses *netipx.IPSet + updated time.Time + + // validators identify the version of the list this copy holds. + validators validators +} + +// Params is what New needs. +type Params struct { + // Dir is the directory that holds the copies on disk. + Dir string + + // Lists are the lists to download and look addresses up in. + Lists []List + + Logger *slog.Logger +} + +// Blocklists keeps a copy of each list and answers which lists contain an +// address. Start loads the copies from disk and starts downloading the lists. +type Blocklists struct { + dir string + lists []List + logger *slog.Logger + + // copies holds each list's last good copy, by list name. A list with no + // good copy yet has none. + mu sync.RWMutex + copies map[string]listCopy + + // stop ends the goroutine Start runs, and done is closed when it returns. + stop context.CancelFunc + done chan struct{} +} + +// New returns a Blocklists with no copies. Start loads them. +func New(params Params) *Blocklists { + return &Blocklists{ + dir: params.Dir, + lists: params.Lists, + logger: params.Logger.With("component", "blocklists"), + copies: make(map[string]listCopy), + done: make(chan struct{}), + } +} + +// Lookup returns a Listing for each list whose copy contains addr, in the +// order of the lists given to New. It makes no outside request. When no list +// contains addr, it returns an empty list, not nil. +func (b *Blocklists) Lookup(addr netip.Addr) []Listing { + b.mu.RLock() + defer b.mu.RUnlock() + + listings := []Listing{} + + for _, list := range b.lists { + c, ok := b.copies[list.Name] + if ok && c.addresses.Contains(addr.Unmap()) { + listings = append(listings, Listing{ + Name: list.Name, + Description: list.Description, + Updated: c.updated, + }) + } + } + + return listings +} + +// ShortestInterval returns the shortest Interval of the lists. That is the +// soonest a copy can change, and so how long a lookup answer stays current. +func (b *Blocklists) ShortestInterval() time.Duration { + var shortest time.Duration + + for i, list := range b.lists { + if i == 0 || list.Interval < shortest { + shortest = list.Interval + } + } + + return shortest +} + +// updated returns when the named list's copy was last downloaded or found +// current, or the zero time when the list has no copy. +func (b *Blocklists) updated(name string) time.Time { + b.mu.RLock() + defer b.mu.RUnlock() + + return b.copies[name].updated +} + +// copyValidators returns the validators of the named list's copy. They are +// empty when the list has no copy. +func (b *Blocklists) copyValidators(name string) validators { + b.mu.RLock() + defer b.mu.RUnlock() + + return b.copies[name].validators +} + +// setCopy makes c the named list's copy. +func (b *Blocklists) setCopy(name string, c listCopy) { + b.mu.Lock() + defer b.mu.Unlock() + + b.copies[name] = c +} + +// setUpdated records that the named list's copy was found current at t. +func (b *Blocklists) setUpdated(name string, t time.Time) { + b.mu.Lock() + defer b.mu.Unlock() + + c, ok := b.copies[name] + if ok { + c.updated = t + b.copies[name] = c + } +} diff --git a/internal/blocklist/blocklist_internal_test.go b/internal/blocklist/blocklist_internal_test.go new file mode 100644 index 0000000..0ce9770 --- /dev/null +++ b/internal/blocklist/blocklist_internal_test.go @@ -0,0 +1,494 @@ +package blocklist + +import ( + "context" + "errors" + "io" + "io/fs" + "log/slog" + "net/http" + "net/http/httptest" + "net/netip" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" +) + +// The test list's name and description, the address its good version lists, +// and the address a newer version lists instead. +const ( + testListName = "test_list" + testListDescription = "On the test list" + listedAddr = "192.0.2.1" + goodListBody = "# A test list\n" + listedAddr + "\n" + newListedAddr = "192.0.2.2" + newListBody = "# A test list\n" + newListedAddr + "\n" +) + +// testETag is the ETag a publisher sends with a version of the test list. +const testETag = `"v1"` + +// maxWait is how long a test waits for the download goroutine. +const maxWait = 10 * time.Second + +// publisher stands in for a list's publisher. It answers every request with +// its status and body, and keeps the headers of each request. A 200 answer +// carries lastModified and etag, when set, as Last-Modified and ETag. As +// from a real publisher, a request whose If-None-Match or If-Modified-Since +// matches them is answered "not modified". +type publisher struct { + mu sync.Mutex + status int + body string + lastModified time.Time + etag string + headers []http.Header +} + +// newPublisher starts a publisher that answers 200 with body, and returns it +// and its URL. +func newPublisher(t *testing.T, body string) (*publisher, string) { + t.Helper() + + p := &publisher{status: http.StatusOK, body: body} + srv := httptest.NewServer(p) + t.Cleanup(srv.Close) + + return p, srv.URL +} + +func (p *publisher) ServeHTTP(w http.ResponseWriter, r *http.Request) { + p.mu.Lock() + defer p.mu.Unlock() + + p.headers = append(p.headers, r.Header.Clone()) + + if p.status != http.StatusOK { + w.WriteHeader(p.status) + _, _ = io.WriteString(w, p.body) + + return + } + + if p.etag != "" { + w.Header().Set(etagHeader, p.etag) + } + + // ServeContent sends Last-Modified unless lastModified is zero, and + // answers "not modified" when the request's conditions match. + http.ServeContent(w, r, testListName, p.lastModified, + strings.NewReader(p.body)) +} + +// answer makes the publisher answer later requests with status and body. +func (p *publisher) answer(status int, body string) { + p.mu.Lock() + defer p.mu.Unlock() + + p.status, p.body = status, body +} + +// publish makes the publisher answer later requests with a version of the +// list: body, with lastModified and etag. A zero lastModified or an empty +// etag is not sent. +func (p *publisher) publish(body string, lastModified time.Time, etag string) { + p.mu.Lock() + defer p.mu.Unlock() + + p.status, p.body = http.StatusOK, body + p.lastModified, p.etag = lastModified, etag +} + +// requests returns the headers of the requests the publisher has had. +func (p *publisher) requests() []http.Header { + p.mu.Lock() + defer p.mu.Unlock() + + return append([]http.Header(nil), p.headers...) +} + +// testList returns a list served at url, written in format. +func testList(url string, format Format) List { + return List{ + Name: testListName, + Description: testListDescription, + URL: url, + Interval: time.Hour, + Format: format, + } +} + +func newTestBlocklists(dir string, list List) *Blocklists { + return New(Params{ + Dir: dir, + Lists: []List{list}, + Logger: slog.New(slog.DiscardHandler), + }) +} + +// lookup looks addr up in b. +func lookup(t *testing.T, b *Blocklists, addr string) []Listing { + t.Helper() + + return b.Lookup(netip.MustParseAddr(addr)) +} + +// TestLookupFindsListedAddresses downloads a list in each format and looks +// up addresses on it, inside its networks, and off it, in IPv4 and IPv6. +// A listed address has one listing that names the list; any other has an +// empty list of listings. +func TestLookupFindsListedAddresses(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + format Format + body string + listed []string + unlisted []string + }{ + { + name: "single addresses", + format: FormatLines, + body: "# comment\n192.0.2.1\n\n2001:db8::1\n", + listed: []string{"192.0.2.1", "::ffff:192.0.2.1", "2001:db8::1"}, + unlisted: []string{"192.0.2.2", "2001:db8::2"}, + }, + { + name: "networks", + format: FormatLines, + body: "# comment\n198.51.100.0/24\n2001:db8:1::/48\n", + listed: []string{"198.51.100.0", "198.51.100.255", "2001:db8:1::5"}, + unlisted: []string{"198.51.101.0", "2001:db8:2::1"}, + }, + { + name: "Spamhaus networks", + format: FormatSpamhausJSON, + body: `{"cidr":"203.0.113.0/25","sblid":"SBL1","rir":"arin"}` + "\n" + + `{"cidr":"2001:db8:3::/48","sblid":"SBL2","rir":"ripencc"}` + "\n" + + `{"type":"metadata","timestamp":1793074442,"records":2}` + "\n", + listed: []string{"203.0.113.1", "2001:db8:3::1"}, + unlisted: []string{"203.0.113.200", "2001:db8:4::1"}, + }, + { + name: "DShield networks", + format: FormatDShield, + body: "# DShield.org Recommended Block List\n" + + "192.0.2.0\t192.0.2.255\t24\t327\tEXAMPLE\tUS\tNone\n", + listed: []string{"192.0.2.0", "192.0.2.77"}, + unlisted: []string{"192.0.3.1", "2001:db8::1"}, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + + _, url := newPublisher(t, tt.body) + b := newTestBlocklists(t.TempDir(), testList(url, tt.format)) + + err := b.download(t.Context(), b.lists[0]) + if err != nil { + t.Fatalf("download: %v", err) + } + + for _, addr := range tt.listed { + listings := lookup(t, b, addr) + if len(listings) != 1 || listings[0].Name != testListName || + listings[0].Description != testListDescription || + listings[0].Updated.IsZero() { + t.Errorf("%s: listings %+v, want one for %s", addr, + listings, testListName) + } + } + + for _, addr := range tt.unlisted { + listings := lookup(t, b, addr) + if listings == nil || len(listings) != 0 { + t.Errorf("%s: listings %#v, want an empty list", addr, listings) + } + } + }) + } +} + +// TestFailedDownloadKeepsThePreviousCopy downloads a good version of a list, +// then one that fails. The good copy stays in memory and on disk, and no +// part of the failed download is left beside it. +func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + status int + body string + }{ + {"server error", http.StatusInternalServerError, "unavailable"}, + {"not a list", http.StatusOK, "\n\n"}, + { + "over the size limit", http.StatusOK, + strings.Repeat(listedAddr+"\n", maxDownloadSize/len(listedAddr)), + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + p, url := newPublisher(t, goodListBody) + b := newTestBlocklists(dir, testList(url, FormatLines)) + + err := b.download(t.Context(), b.lists[0]) + if err != nil { + t.Fatalf("first download: %v", err) + } + + before := lookup(t, b, listedAddr) + + p.answer(tt.status, tt.body) + + err = b.download(t.Context(), b.lists[0]) + if err == nil { + t.Fatal("second download returned no error") + } + + after := lookup(t, b, listedAddr) + if len(after) != 1 || len(before) != 1 || + !after[0].Updated.Equal(before[0].Updated) { + t.Errorf("listings %+v after the failure, want %+v", after, before) + } + + copies := os.DirFS(dir) + + onDisk, err := fs.ReadFile(copies, testListName) + if err != nil || string(onDisk) != goodListBody { + t.Errorf("copy on disk %q (%v), want %q", onDisk, err, goodListBody) + } + + _, err = fs.Stat(copies, testListName+downloadSuffix) + if !errors.Is(err, fs.ErrNotExist) { + t.Errorf("the failed download's file is still there (%v)", err) + } + }) + } +} + +// TestRestartLoadsCopiesFromDisk downloads a list, then starts a new +// Blocklists on the same directory, as a restart does. The new one answers +// from the copy on disk at once, and does not download the list again while +// the copy is younger than the list's interval. +func TestRestartLoadsCopiesFromDisk(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + p, url := newPublisher(t, goodListBody) + list := testList(url, FormatLines) + + first := newTestBlocklists(dir, list) + + err := first.download(t.Context(), list) + if err != nil { + t.Fatalf("download: %v", err) + } + + downloaded := lookup(t, first, listedAddr)[0].Updated + + second := newTestBlocklists(dir, list) + second.Start(t.Context()) + second.Stop(t.Context()) + + listings := lookup(t, second, listedAddr) + if len(listings) != 1 || + listings[0].Updated.Sub(downloaded).Abs() > time.Second { + t.Errorf("after the restart, listings %+v, want one updated at %s", + listings, downloaded) + } + + if got := len(p.requests()); got != 1 { + t.Errorf("the publisher had %d requests, want 1", got) + } +} + +// TestStartRefreshesAnOldCopy starts on a copy older than its list's +// interval. The list is downloaded at once, with routewatch's User-Agent and +// the Last-Modified and ETag the publisher sent with the copy, kept on disk. +// The publisher answers that the copy is still current, so the copy is kept +// and marked as updated now. +func TestStartRefreshesAnOldCopy(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + p, url := newPublisher(t, goodListBody) + list := testList(url, FormatLines) + + lastModified := time.Now().Add(-3 * list.Interval).UTC().Truncate(time.Second) + p.publish(goodListBody, lastModified, testETag) + + err := newTestBlocklists(dir, list).download(t.Context(), list) + if err != nil { + t.Fatalf("download: %v", err) + } + + old := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second) + + err = os.Chtimes(filepath.Join(dir, testListName), old, old) + if err != nil { + t.Fatalf("Chtimes: %v", err) + } + + b := newTestBlocklists(dir, list) + b.Start(t.Context()) + + deadline := time.Now().Add(maxWait) + for !lookup(t, b, listedAddr)[0].Updated.After(old) { + if time.Now().After(deadline) { + t.Fatalf("the copy was not marked updated within %s", maxWait) + } + + time.Sleep(time.Millisecond) + } + + b.Stop(t.Context()) + + requests := p.requests() + last := requests[len(requests)-1] + + want := lastModified.Format(http.TimeFormat) + if got := last.Get(ifModifiedSinceHeader); got != want { + t.Errorf("If-Modified-Since %q, want %q", got, want) + } + + if got := last.Get(ifNoneMatchHeader); got != testETag { + t.Errorf("If-None-Match %q, want %q", got, testETag) + } + + if got := last.Get(userAgentHeader); got != userAgent { + t.Errorf("User-Agent %q, want %q", got, userAgent) + } +} + +// TestNewVersionOlderThanTheLastDownloadReplacesTheCopy downloads a list. +// The publisher then puts out a new version whose Last-Modified is earlier +// than that download, as when a cache in front of the publisher passes the +// version on late. The next download sends the Last-Modified of the copy, +// not the time of the download, so it gets the new version. +func TestNewVersionOlderThanTheLastDownloadReplacesTheCopy(t *testing.T) { + t.Parallel() + + p, url := newPublisher(t, goodListBody) + list := testList(url, FormatLines) + b := newTestBlocklists(t.TempDir(), list) + + firstVersion := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second) + p.publish(goodListBody, firstVersion, "") + + err := b.download(t.Context(), list) + if err != nil { + t.Fatalf("first download: %v", err) + } + + p.publish(newListBody, firstVersion.Add(list.Interval), "") + + err = b.download(t.Context(), list) + if err != nil { + t.Fatalf("second download: %v", err) + } + + if listings := lookup(t, b, newListedAddr); len(listings) != 1 { + t.Errorf("%s: listings %+v, want one from the new version", + newListedAddr, listings) + } + + want := firstVersion.Format(http.TimeFormat) + if got := p.requests()[1].Get(ifModifiedSinceHeader); got != want { + t.Errorf("If-Modified-Since %q, want %q", got, want) + } +} + +// TestDownloadWithoutLastModifiedOrETagSendsNoCondition downloads a list +// twice from a publisher that sends neither Last-Modified nor ETag. Neither +// request carries If-Modified-Since or If-None-Match, so the second download +// gets the whole list again. +func TestDownloadWithoutLastModifiedOrETagSendsNoCondition(t *testing.T) { + t.Parallel() + + p, url := newPublisher(t, goodListBody) + list := testList(url, FormatLines) + b := newTestBlocklists(t.TempDir(), list) + + err := b.download(t.Context(), list) + if err != nil { + t.Fatalf("first download: %v", err) + } + + err = b.download(t.Context(), list) + if err != nil { + t.Fatalf("second download: %v", err) + } + + for i, request := range p.requests() { + if request.Get(ifModifiedSinceHeader) != "" || + request.Get(ifNoneMatchHeader) != "" { + t.Errorf("request %d has a condition: %v", i, request) + } + } +} + +// TestStopCancelsADownloadInProgress stops while the publisher has not +// answered. Stop cancels the download and returns once the download +// goroutine has stopped. +func TestStopCancelsADownloadInProgress(t *testing.T) { + t.Parallel() + + arrived := make(chan struct{}) + + var once sync.Once + + srv := httptest.NewServer(http.HandlerFunc( + func(_ http.ResponseWriter, r *http.Request) { + once.Do(func() { close(arrived) }) + <-r.Context().Done() + })) + t.Cleanup(srv.Close) + + b := newTestBlocklists(t.TempDir(), testList(srv.URL, FormatLines)) + b.Start(t.Context()) + + select { + case <-arrived: + case <-time.After(maxWait): + t.Fatalf("no download began within %s", maxWait) + } + + ctx, cancel := context.WithTimeout(t.Context(), maxWait) + defer cancel() + + b.Stop(ctx) + + select { + case <-b.done: + default: + t.Error("Stop returned before the download goroutine stopped") + } +} + +// TestPublicListsAreComplete checks that every public list has what a +// download and a lookup answer need, and a unique name. +func TestPublicListsAreComplete(t *testing.T) { + t.Parallel() + + names := make(map[string]bool) + + for _, list := range Lists() { + if list.Name == "" || list.Description == "" || list.URL == "" || + list.Interval <= 0 || names[list.Name] { + t.Errorf("list %+v is incomplete or has a name used before", list) + } + + names[list.Name] = true + } +} diff --git a/internal/blocklist/download.go b/internal/blocklist/download.go new file mode 100644 index 0000000..dc6ac07 --- /dev/null +++ b/internal/blocklist/download.go @@ -0,0 +1,379 @@ +package blocklist + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "io/fs" + "net/http" + "os" + "slices" + "time" + + "go4.org/netipx" +) + +// Limits of one download. +const ( + // downloadTimeout is how long one download may take. + downloadTimeout = time.Minute + + // maxDownloadSize is the size of the largest list file accepted, in + // bytes. The largest list, FireHOL abusers, is about 2.5 MiB. + maxDownloadSize = 16 << 20 +) + +// The request headers a download sets. +const ( + userAgentHeader = "User-Agent" + ifModifiedSinceHeader = "If-Modified-Since" + ifNoneMatchHeader = "If-None-Match" + + // userAgent names routewatch to the publishers. + userAgent = "routewatch (+https://git.eeqj.de/sneak/routewatch)" +) + +// The response headers that identify the version of a list a publisher sent. +const ( + lastModifiedHeader = "Last-Modified" + etagHeader = "ETag" +) + +const ( + // downloadSuffix ends the name of the file a download is written to, + // beside the list's copy, until it has parsed. + downloadSuffix = ".download" + + // validatorsSuffix ends the name of the file beside a list's copy that + // keeps the copy's validators. + validatorsSuffix = ".validators" + + // dirPermissions are those of the directory that holds the copies. + dirPermissions = 0o750 + + // validatorsPermissions are those of the file that keeps a copy's + // validators. + validatorsPermissions = 0o600 +) + +// validators are the Last-Modified and ETag headers a publisher sent with a +// version of a list. A download sends them back, as If-Modified-Since and +// If-None-Match, so that the publisher sends the list only when it has +// changed. They are kept as the publisher wrote them. +// +//nolint:tagliatelle // snake_case keys, as the Go style guide asks +type validators struct { + LastModified string `json:"last_modified,omitempty"` + ETag string `json:"etag,omitempty"` +} + +var ( + errUnexpectedStatus = errors.New("unexpected HTTP status") + errTooLarge = errors.New("list is over the size limit") +) + +// Start loads each list's copy from disk. It then starts the goroutine that +// downloads each list once its copy is older than its Interval, and a list +// with no copy at once. The goroutine runs until Stop: the end of ctx does +// not stop it. +func (b *Blocklists) Start(ctx context.Context) { + err := os.MkdirAll(b.dir, dirPermissions) + if err != nil { + b.logger.Error("Failed to create the blocklists directory", + "dir", b.dir, "error", err) + } + + nextDownload := make([]time.Time, len(b.lists)) + + for i, list := range b.lists { + loadErr := b.loadCopy(list) + if loadErr != nil && !errors.Is(loadErr, fs.ErrNotExist) { + b.logger.Error("Failed to load a blocklist copy", + "list", list.Name, "error", loadErr) + } + + // A list with no copy has the zero time, so it is due at once. + nextDownload[i] = b.updated(list.Name).Add(list.Interval) + } + + ctx, b.stop = context.WithCancel(context.WithoutCancel(ctx)) + + go func() { + b.run(ctx, nextDownload) + }() + + b.logger.Info("Blocklist downloads started", "dir", b.dir) +} + +// Stop stops the downloads, cancelling one in progress, and waits until the +// goroutine has returned or ctx ends. The copies in memory stay, so Lookup +// still answers. +func (b *Blocklists) Stop(ctx context.Context) { + b.stop() + + select { + case <-b.done: + b.logger.Info("Blocklist downloads stopped") + case <-ctx.Done(): + b.logger.Warn("Stop deadline passed before the blocklist downloads " + + "stopped") + } +} + +// run downloads each list when nextDownload says, until ctx ends. After each +// try, whether it worked or not, the list's next download is an Interval +// later. +func (b *Blocklists) run(ctx context.Context, nextDownload []time.Time) { + defer close(b.done) + + if len(nextDownload) == 0 { + return + } + + for { + soonest := slices.MinFunc(nextDownload, time.Time.Compare) + timer := time.NewTimer(time.Until(soonest)) + + select { + case <-ctx.Done(): + timer.Stop() + + return + case <-timer.C: + } + + for i, list := range b.lists { + if time.Now().Before(nextDownload[i]) { + continue + } + + err := b.download(ctx, list) + if ctx.Err() != nil { + return + } + + if err != nil { + b.logger.Error("Blocklist download failed; keeping the "+ + "previous copy", "list", list.Name, "error", err) + } + + nextDownload[i] = time.Now().Add(list.Interval) + } + } +} + +// download asks the publisher for list, sending the validators of the list's +// copy when it has them. A new version that parses becomes the list's copy, +// on disk and in memory. An answer that the copy is still current marks the +// copy as updated now. Anything else leaves the copy as it is and returns an +// error. +func (b *Blocklists) download(ctx context.Context, list List) error { + ctx, cancel := context.WithTimeout(ctx, downloadTimeout) + defer cancel() + + req, err := http.NewRequestWithContext(ctx, http.MethodGet, list.URL, nil) + if err != nil { + return err + } + + req.Header.Set(userAgentHeader, userAgent) + + held := b.copyValidators(list.Name) + if held.LastModified != "" { + req.Header.Set(ifModifiedSinceHeader, held.LastModified) + } + + if held.ETag != "" { + req.Header.Set(ifNoneMatchHeader, held.ETag) + } + + resp, err := http.DefaultClient.Do(req) + if err != nil { + return err + } + + defer func() { _ = resp.Body.Close() }() + + switch resp.StatusCode { + case http.StatusOK: + return b.replaceCopy(list, resp.Body, validators{ + LastModified: resp.Header.Get(lastModifiedHeader), + ETag: resp.Header.Get(etagHeader), + }) + case http.StatusNotModified: + return b.markCurrent(list) + default: + return fmt.Errorf("%w: %s", errUnexpectedStatus, resp.Status) + } +} + +// replaceCopy writes a new version of list, read from body, beside the list's +// copy. Only when it parses does it replace the copy, on disk and in memory. +// sent are the validators the publisher sent with the new version. +func (b *Blocklists) replaceCopy(list List, body io.Reader, sent validators) error { + root, err := os.OpenRoot(b.dir) + if err != nil { + return err + } + + defer func() { _ = root.Close() }() + + downloadName := list.Name + downloadSuffix + + addresses, err := writeAndParse(root, downloadName, body, list.Format) + if err != nil { + _ = root.Remove(downloadName) + + return err + } + + // The old copy's validators are removed before the copy is replaced, so + // that they are never kept with the new copy. + err = root.Remove(list.Name + validatorsSuffix) + if err != nil && !errors.Is(err, fs.ErrNotExist) { + return err + } + + err = root.Rename(downloadName, list.Name) + if err != nil { + return err + } + + b.setCopy(list.Name, listCopy{ + addresses: addresses, updated: time.Now().UTC(), validators: sent, + }) + b.logger.Info("Downloaded a new version of a blocklist", "list", list.Name) + + // Without this file, the first download after a restart asks for the + // whole list. + err = writeValidators(root, list.Name, sent) + if err != nil { + b.logger.Error("Failed to keep the validators of a blocklist copy", + "list", list.Name, "error", err) + } + + return nil +} + +// writeAndParse writes body to the file name in root, up to maxDownloadSize +// bytes, and parses the file as format. +func writeAndParse( + root *os.Root, name string, body io.Reader, format Format, +) (*netipx.IPSet, error) { + f, err := root.Create(name) + if err != nil { + return nil, err + } + + defer func() { _ = f.Close() }() + + written, err := io.Copy(f, io.LimitReader(body, maxDownloadSize+1)) + if err != nil { + return nil, err + } + + if written > maxDownloadSize { + return nil, errTooLarge + } + + _, err = f.Seek(0, io.SeekStart) + if err != nil { + return nil, err + } + + return parse(f, format) +} + +// markCurrent records that the publisher says the list's copy is still +// current: the copy, on disk and in memory, is marked as updated now. +func (b *Blocklists) markCurrent(list List) error { + root, err := os.OpenRoot(b.dir) + if err != nil { + return err + } + + defer func() { _ = root.Close() }() + + now := time.Now().UTC() + + err = root.Chtimes(list.Name, now, now) + if err != nil { + return err + } + + b.setUpdated(list.Name, now) + b.logger.Debug("Blocklist copy is still current", "list", list.Name) + + return nil +} + +// loadCopy makes the list's copy on disk, with the validators kept beside it, +// its copy in memory. The file's modification time is when the copy was +// updated. +func (b *Blocklists) loadCopy(list List) error { + root, err := os.OpenRoot(b.dir) + if err != nil { + return err + } + + defer func() { _ = root.Close() }() + + f, err := root.Open(list.Name) + if err != nil { + return err + } + + defer func() { _ = f.Close() }() + + info, err := f.Stat() + if err != nil { + return err + } + + addresses, err := parse(f, list.Format) + if err != nil { + return err + } + + held, err := readValidators(root, list.Name) + if err != nil { + return err + } + + b.setCopy(list.Name, listCopy{ + addresses: addresses, updated: info.ModTime().UTC(), validators: held, + }) + + return nil +} + +// writeValidators keeps held in the file beside the named list's copy. +func writeValidators(root *os.Root, name string, held validators) error { + data, err := json.Marshal(held) + if err != nil { + return err + } + + return root.WriteFile(name+validatorsSuffix, data, validatorsPermissions) +} + +// readValidators returns the validators kept beside the named list's copy. +// They are empty when there is no such file. +func readValidators(root *os.Root, name string) (validators, error) { + var held validators + + data, err := root.ReadFile(name + validatorsSuffix) + if errors.Is(err, fs.ErrNotExist) { + return held, nil + } + + if err != nil { + return held, err + } + + err = json.Unmarshal(data, &held) + + return held, err +} diff --git a/internal/blocklist/lists.go b/internal/blocklist/lists.go new file mode 100644 index 0000000..2aa3507 --- /dev/null +++ b/internal/blocklist/lists.go @@ -0,0 +1,93 @@ +package blocklist + +import "time" + +// How often each list is downloaded. No list is downloaded more often than +// its publisher updates it or allows. +const ( + // DShield asks that its feeds be downloaded at most once an hour: + // https://isc.sans.edu/feeds_doc.html + dshieldInterval = time.Hour + + // abuse.ch makes the list every 5 minutes and recommends a download at + // least every 15: https://feodotracker.abuse.ch/blocklist/ + feodoTrackerInterval = 15 * time.Minute + + // FireHOL's page for each list gives how often the list changes on + // average: about every 5 hours for abusers and every 3 hours for level + // 3. FireHOL sets no download limit. + // https://iplists.firehol.org/?ipset=firehol_abusers_30d + // https://iplists.firehol.org/?ipset=firehol_level3 + fireHOLAbusersInterval = 5 * time.Hour + fireHOLLevel3Interval = 3 * time.Hour + + // Spamhaus asks that DROP be downloaded at most once a day: + // https://www.spamhaus.org/blocklists/do-not-route-or-peer/ + spamhausDROPInterval = 24 * time.Hour + + // Tor publishes a new exit list every hour and sets no download limit: + // https://collector.torproject.org/recent/exit-lists/ + torExitsInterval = time.Hour +) + +// Lists returns the public blocklists routewatch downloads. Each can be +// downloaded whole without an account or a key. Spamhaus merged its EDROP +// list into DROP, so DROP covers both. +func Lists() []List { + return []List{ + { + Name: "dshield", + Description: "In one of the 20 networks that DShield saw scan " + + "the most targets in the last three days", + URL: "https://feeds.dshield.org/block.txt", + Interval: dshieldInterval, + Format: FormatDShield, + }, + { + Name: "feodo_tracker", + Description: "A botnet command-and-control server", + URL: "https://feodotracker.abuse.ch/downloads/ipblocklist.txt", + Interval: feodoTrackerInterval, + Format: FormatLines, + }, + { + Name: "firehol_abusers_30d", + Description: "Reported for comment, form or forum spam in the " + + "last 30 days", + URL: "https://iplists.firehol.org/files/firehol_abusers_30d.netset", + Interval: fireHOLAbusersInterval, + Format: FormatLines, + }, + { + Name: "firehol_level3", + Description: "Reported for attacks, spyware or viruses in the " + + "last 30 days", + URL: "https://iplists.firehol.org/files/firehol_level3.netset", + Interval: fireHOLLevel3Interval, + Format: FormatLines, + }, + { + Name: "spamhaus_drop", + Description: "In an IPv4 network that Spamhaus says spammers " + + "or cyber criminals have hijacked or lease", + URL: "https://www.spamhaus.org/drop/drop_v4.json", + Interval: spamhausDROPInterval, + Format: FormatSpamhausJSON, + }, + { + Name: "spamhaus_dropv6", + Description: "In an IPv6 network that Spamhaus says spammers " + + "or cyber criminals have hijacked or lease", + URL: "https://www.spamhaus.org/drop/drop_v6.json", + Interval: spamhausDROPInterval, + Format: FormatSpamhausJSON, + }, + { + Name: "tor_exits", + Description: "A Tor exit node, where traffic from the Tor network leaves", + URL: "https://check.torproject.org/torbulkexitlist", + Interval: torExitsInterval, + Format: FormatLines, + }, + } +} diff --git a/internal/blocklist/parse.go b/internal/blocklist/parse.go new file mode 100644 index 0000000..7602cf0 --- /dev/null +++ b/internal/blocklist/parse.go @@ -0,0 +1,158 @@ +package blocklist + +import ( + "bufio" + "encoding/json" + "errors" + "fmt" + "io" + "net/netip" + "strings" + + "go4.org/netipx" +) + +const ( + // commentPrefix starts a comment line in a list file. + commentPrefix = "#" + + // networkSeparator separates the address and the mask length of a + // network, as in 192.0.2.0/24. + networkSeparator = "/" + + // dshieldFieldSeparator separates the fields of a DShield line. + dshieldFieldSeparator = "\t" + + // spamhausMetadataType is the type of the line that ends a Spamhaus + // file. That line holds the file's metadata and no network. + spamhausMetadataType = "metadata" +) + +var ( + errUnknownFormat = errors.New("unknown list format") + errNoNetwork = errors.New("no network on the line") + errBadRange = errors.New("first and last address make no range") +) + +// parse reads a list file written in format and returns the addresses it +// holds. A line it cannot read fails the whole file, so a file that is not +// the list, such as an error page, never replaces a good copy. +func parse(r io.Reader, format Format) (*netipx.IPSet, error) { + var builder netipx.IPSetBuilder + + scanner := bufio.NewScanner(r) + + for lineNumber := 1; scanner.Scan(); lineNumber++ { + line := strings.TrimSpace(scanner.Text()) + if line == "" || strings.HasPrefix(line, commentPrefix) { + continue + } + + err := addLine(&builder, line, format) + if err != nil { + return nil, fmt.Errorf("line %d: %w", lineNumber, err) + } + } + + err := scanner.Err() + if err != nil { + return nil, err + } + + return builder.IPSet() +} + +// addLine adds the address or network on one line of a list file to builder. +func addLine(builder *netipx.IPSetBuilder, line string, format Format) error { + switch format { + case FormatLines: + return addAddressOrNetwork(builder, line) + case FormatSpamhausJSON: + return addSpamhausNetwork(builder, line) + case FormatDShield: + return addDShieldNetwork(builder, line) + default: + return errUnknownFormat + } +} + +// addAddressOrNetwork adds an address, such as 192.0.2.1, or a network, such +// as 192.0.2.0/24, to builder. +func addAddressOrNetwork(builder *netipx.IPSetBuilder, text string) error { + if strings.Contains(text, networkSeparator) { + network, err := netip.ParsePrefix(text) + if err != nil { + return err + } + + builder.AddPrefix(network) + + return nil + } + + addr, err := netip.ParseAddr(text) + if err != nil { + return err + } + + builder.Add(addr) + + return nil +} + +// addSpamhausNetwork adds the network on a line of a Spamhaus file to +// builder. The metadata line adds nothing. +func addSpamhausNetwork(builder *netipx.IPSetBuilder, line string) error { + var entry struct { + CIDR string `json:"cidr"` + Type string `json:"type"` + } + + err := json.Unmarshal([]byte(line), &entry) + if err != nil { + return err + } + + if entry.Type == spamhausMetadataType { + return nil + } + + network, err := netip.ParsePrefix(entry.CIDR) + if err != nil { + return err + } + + builder.AddPrefix(network) + + return nil +} + +// addDShieldNetwork adds the network on a line of a DShield file to builder. +// The line's first two fields are the network's first and last address. +func addDShieldNetwork(builder *netipx.IPSetBuilder, line string) error { + firstField, rest, found := strings.Cut(line, dshieldFieldSeparator) + if !found { + return errNoNetwork + } + + lastField, _, _ := strings.Cut(rest, dshieldFieldSeparator) + + first, err := netip.ParseAddr(firstField) + if err != nil { + return err + } + + last, err := netip.ParseAddr(lastField) + if err != nil { + return err + } + + network := netipx.IPRangeFrom(first, last) + if !network.IsValid() { + return errBadRange + } + + builder.AddRange(network) + + return nil +} diff --git a/internal/database/models.go b/internal/database/models.go index dbb5309..8bdc4d6 100644 --- a/internal/database/models.go +++ b/internal/database/models.go @@ -92,12 +92,11 @@ type PrefixDistribution struct { // //nolint:tagliatelle // field names are the existing JSON API type IPInfo struct { - IP string `json:"ip"` - PTR []string `json:"ptr,omitempty"` - Netblock string `json:"netblock"` - MaskLength int `json:"mask_length"` - IPVersion int `json:"ip_version"` - NumPeers int `json:"num_peers"` + IP string `json:"ip"` + Netblock string `json:"netblock"` + MaskLength int `json:"mask_length"` + IPVersion int `json:"ip_version"` + NumPeers int `json:"num_peers"` // AS information ASN int `json:"asn"` ASName string `json:"as_name,omitempty"` diff --git a/internal/routewatch/app.go b/internal/routewatch/app.go index 1a4e925..9fa7d20 100644 --- a/internal/routewatch/app.go +++ b/internal/routewatch/app.go @@ -8,9 +8,11 @@ package routewatch import ( "context" "errors" + "path/filepath" "sync" "time" + "git.eeqj.de/sneak/routewatch/internal/blocklist" "git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/logger" @@ -52,8 +54,13 @@ type RouteWatch struct { peeringHandler *PeeringHandler asnFetcher *ASNFetcher dbMaintainer *DBMaintainer + blocklists *blocklist.Blocklists } +// blocklistsDirName is the directory in the state directory that holds the +// blocklist copies. +const blocklistsDirName = "blocklists" + // New creates a new RouteWatch instance func New(deps Dependencies) *RouteWatch { rw := &RouteWatch{ @@ -122,6 +129,18 @@ func (rw *RouteWatch) Run(ctx context.Context) error { rw.asnFetcher.Start(ctx) rw.server.SetASNFetcher(rw.asnFetcher) + // Load the blocklist copies from disk and start downloading the lists. + // The HTTP handlers read the server's blocklists without a lock, so they + // are set before the handlers run. + blocklists := blocklist.New(blocklist.Params{ + Dir: filepath.Join(rw.config.StateDir, blocklistsDirName), + Lists: blocklist.Lists(), + Logger: rw.logger.Logger, + }) + blocklists.Start(ctx) + rw.blocklists = blocklists + rw.server.SetBlocklists(blocklists) + // Start HTTP server err = rw.server.Start(ctx) if err != nil { @@ -139,11 +158,12 @@ func (rw *RouteWatch) Run(ctx context.Context) error { } // Shutdown stops all services in order: the streamer, the handlers, which -// write what they hold, the WHOIS fetcher, the database maintainer and the -// HTTP server, and last the database, which it checkpoints and closes. Each -// stage waits at most until ctx ends, so Shutdown returns soon after: a -// handler logs what it could not write by then, and the HTTP server closes -// the connections of requests still running. +// write what they hold, the WHOIS fetcher, the database maintainer, the +// blocklist downloads and the HTTP server, and last the database, which it +// checkpoints and closes. Each stage waits at most until ctx ends, so +// Shutdown returns soon after: a handler logs what it could not write by +// then, and the HTTP server closes the connections of requests still +// running. func (rw *RouteWatch) Shutdown(ctx context.Context) { rw.mu.Lock() if rw.shutdown { @@ -190,6 +210,10 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) { rw.dbMaintainer.Stop(ctx) } + if rw.blocklists != nil { + rw.blocklists.Stop(ctx) + } + err := rw.server.Stop(ctx) if err != nil { rw.logger.Error("Stopped the HTTP server before every request finished", diff --git a/internal/routewatch/app_integration_internal_test.go b/internal/routewatch/app_integration_internal_test.go index f467126..ad4f18a 100644 --- a/internal/routewatch/app_integration_internal_test.go +++ b/internal/routewatch/app_integration_internal_test.go @@ -466,9 +466,9 @@ func TestRouteWatchLiveFeed(t *testing.T) { // Create streamer s := streamer.New(logger, metricsTracker) - // Create test config with empty state dir (no snapshot loading) + // The state directory holds the blocklist copies Run downloads. cfg := &config.Config{ - StateDir: "", + StateDir: t.TempDir(), MaxRuntime: 5 * time.Second, EnableBatchedDatabaseWrites: true, } diff --git a/internal/server/handlers.go b/internal/server/handlers.go index 406a839..de829ce 100644 --- a/internal/server/handlers.go +++ b/internal/server/handlers.go @@ -15,6 +15,7 @@ import ( "strings" "time" + "git.eeqj.de/sneak/routewatch/internal/blocklist" "git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/ristypes" @@ -45,6 +46,15 @@ const ( ptrLookupTimeout = 2 * time.Second ) +// The Cache-Control header of an IP lookup answer, such as +// "public, max-age=900". +const ( + cacheControlHeader = "Cache-Control" + cacheControlPublic = "public" + cacheControlPrivate = "private" + cacheControlMaxAge = ", max-age=" +) + // Keys of the JSON envelope the handlers and middleware write, and the status // value of an error answer. const ( @@ -693,9 +703,24 @@ func (s *Server) handleIPLookup() http.HandlerFunc { // IPLookupResponse is the standard response for IP/hostname lookups. type IPLookupResponse struct { - Query string `json:"query"` - Results []*database.IPInfo `json:"results"` - Errors []string `json:"errors,omitempty"` + Query string `json:"query"` + Results []IPLookupResult `json:"results"` + Errors []string `json:"errors,omitempty"` +} + +// IPLookupResult is the answer for one address: its PTR names, its live route +// and that route's AS, and the blocklists that list it. IPInfo is nil when +// the address has no live route, and the answer then has no route or AS +// fields. IP hides the IPInfo field of the same name, so that an address with +// no route has it too. +type IPLookupResult struct { + *database.IPInfo + + IP string `json:"ip"` + PTR []string `json:"ptr,omitempty"` + + // Blocklists has one entry for each blocklist the address is on. + Blocklists []blocklist.Listing `json:"blocklists"` } // handleIPInfo returns a handler that provides comprehensive IP information. @@ -704,7 +729,7 @@ type IPLookupResponse struct { // Always returns the same response structure with PTR records for each IP. func (s *Server) handleIPInfo() http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { - target := lookupTarget(r) + target, ownAddress := lookupTarget(r) if target == "" { s.writeJSONError(w, http.StatusBadRequest, "Could not determine IP address") @@ -712,7 +737,7 @@ func (s *Server) handleIPInfo() http.HandlerFunc { } ctx := r.Context() - response := IPLookupResponse{Query: target, Results: make([]*database.IPInfo, 0)} + response := IPLookupResponse{Query: target, Results: make([]IPLookupResult, 0)} // Collect all IPs to look up ipsToLookup, resolveErrors, err := s.resolveLookupTargets(ctx, target) @@ -749,7 +774,10 @@ func (s *Server) handleIPInfo() http.HandlerFunc { } switch { - case errors.Is(err, database.ErrNoRoute), errors.Is(err, database.ErrInvalidIP): + case errors.Is(err, database.ErrNoRoute): + // The address still gets a result, without route fields. + ipInfo = nil + case errors.Is(err, database.ErrInvalidIP): response.Errors = append(response.Errors, ip+": "+err.Error()) continue @@ -768,13 +796,14 @@ func (s *Server) handleIPInfo() http.HandlerFunc { return } - if len(ptrs) > 0 { - ipInfo.PTR = ptrs - } + response.Results = append(response.Results, IPLookupResult{ + IP: ip, + PTR: ptrs, + IPInfo: ipInfo, + Blocklists: s.blocklistListings(ip), + }) - response.Results = append(response.Results, ipInfo) - - if ipInfo.NeedsWHOISRefresh { + if ipInfo != nil && ipInfo.NeedsWHOISRefresh { refreshASNs[ipInfo.ASN] = true } } @@ -782,10 +811,40 @@ func (s *Server) handleIPInfo() http.HandlerFunc { // Queue WHOIS refresh for stale ASNs (non-blocking) s.queueWHOISRefresh(refreshASNs) - s.writeIPLookupAnswer(w, response) + s.writeIPLookupAnswer(w, response, ownAddress) } } +// blocklistListings returns the blocklists that list ip. It returns an empty +// list, never nil, so that an answer's blocklists is [] rather than null. +func (s *Server) blocklistListings(ip string) []blocklist.Listing { + addr, err := netip.ParseAddr(ip) + if s.blocklists == nil || err != nil { + return []blocklist.Listing{} + } + + return s.blocklists.Lookup(addr) +} + +// setIPLookupCacheControl lets caches keep an IP lookup answer for the +// shortest blocklist interval, the soonest a blocklist copy can change. The +// answer about the caller's own address is private, so that a shared cache +// does not give it to another caller. Without blocklists it sets nothing. +func (s *Server) setIPLookupCacheControl(w http.ResponseWriter, ownAddress bool) { + if s.blocklists == nil { + return + } + + visibility := cacheControlPublic + if ownAddress { + visibility = cacheControlPrivate + } + + maxAgeSeconds := int(s.blocklists.ShortestInterval().Seconds()) + w.Header().Set(cacheControlHeader, + visibility+cacheControlMaxAge+strconv.Itoa(maxAgeSeconds)) +} + // noTargetsMessage returns the message of the 400 an IP lookup answers when // no address is left to look up, followed by the first of resolveErrors, which // says why, when there is one. @@ -799,16 +858,12 @@ func noTargetsMessage(resolveErrors []string) string { return message + ": " + resolveErrors[0] } -// writeIPLookupAnswer sends response, or 404 with its first error when no -// address had a route. +// writeIPLookupAnswer sends response with its Cache-Control header. +// ownAddress says the request asked about the caller's own address. func (s *Server) writeIPLookupAnswer( - w http.ResponseWriter, response IPLookupResponse, + w http.ResponseWriter, response IPLookupResponse, ownAddress bool, ) { - if len(response.Results) == 0 && len(response.Errors) > 0 { - s.writeJSONError(w, http.StatusNotFound, "No routes found: "+response.Errors[0]) - - return - } + s.setIPLookupCacheControl(w, ownAddress) err := writeJSONSuccess(w, response) if err != nil { @@ -817,8 +872,9 @@ func (s *Server) writeIPLookupAnswer( } // lookupTarget returns what an IP lookup request asks for: the ip or addr URL -// parameter, or else the client's own address. -func lookupTarget(r *http.Request) string { +// parameter, or else the client's own address. The bool is true for the +// client's own address. +func lookupTarget(r *http.Request) (string, bool) { target := chi.URLParam(r, "ip") if target == "" { target = chi.URLParam(r, "addr") @@ -826,10 +882,10 @@ func lookupTarget(r *http.Request) string { if target == "" { // Use client IP (RealIP middleware has already processed this) - target = extractClientIP(r) + return extractClientIP(r), true } - return target + return target, false } // queueWHOISRefresh asks the WHOIS fetcher, when there is one, to refresh diff --git a/internal/server/handlers_internal_test.go b/internal/server/handlers_internal_test.go index 434be48..cc7872e 100644 --- a/internal/server/handlers_internal_test.go +++ b/internal/server/handlers_internal_test.go @@ -5,6 +5,7 @@ import ( "encoding/json" "errors" "html" + "net" "net/http" "net/http/httptest" "net/netip" @@ -16,6 +17,7 @@ import ( "testing" "time" + "git.eeqj.de/sneak/routewatch/internal/blocklist" "git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/logger" @@ -1040,14 +1042,15 @@ func (failingIPLookupDB) GetIPInfoContext( // server's router with a store whose read of one address fails with the // driver's error. A lookup that asks for that address answers 500 with the // JSON error body, as the AS and prefix JSON routes do; an address with no -// route still answers 404. Before the fix the failed read was reported as if -// the address had no route: 404 for that address alone, and 200 beside a -// routed address (https://git.eeqj.de/sneak/routewatch/issues/109). +// route answers 200. Before the fix the failed read was reported as if the +// address had no route: 404 for that address alone, and 200 beside a routed +// address (https://git.eeqj.de/sneak/routewatch/issues/109). func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) { t.Parallel() s := New(failingIPLookupDB{}, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{}) + s.resolver = &fakeResolver{} tests := []struct { path string @@ -1063,7 +1066,7 @@ func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) { }, { path: "/api/v1/ip/203.0.113.1", - wantCode: http.StatusNotFound, + wantCode: http.StatusOK, }, } @@ -1077,7 +1080,9 @@ func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) { tt.path, rec.Code, tt.wantCode, rec.Body) } - checkJSONErrorCode(t, tt.path, rec, tt.wantCode) + if tt.wantCode != http.StatusOK { + checkJSONErrorCode(t, tt.path, rec, tt.wantCode) + } } } @@ -1165,10 +1170,10 @@ func TestIPLookupRefusesTooManyAddresses(t *testing.T) { }{ { strings.Join(addrs[:maxLookupTargets], ","), - http.StatusNotFound, maxLookupTargets, 0, + http.StatusOK, maxLookupTargets, 0, }, {strings.Join(addrs, ","), http.StatusBadRequest, 0, 0}, - {"most.test", http.StatusNotFound, maxLookupTargets, 1}, + {"most.test", http.StatusOK, maxLookupTargets, 1}, {"too-many.test", http.StatusBadRequest, 0, 1}, {strings.Join(names, ","), http.StatusBadRequest, 0, 0}, } @@ -1189,7 +1194,9 @@ func TestIPLookupRefusesTooManyAddresses(t *testing.T) { tt.target, rec.Code, tt.wantCode, rec.Body) } - checkJSONErrorCode(t, tt.target, rec, tt.wantCode) + if tt.wantCode != http.StatusOK { + checkJSONErrorCode(t, tt.target, rec, tt.wantCode) + } got := db.lookups.Load() if got != tt.wantLookups { @@ -1246,6 +1253,191 @@ func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) { } } +// addrInTestPrefixV6 is an address in testPrefixV6, which +// newServerWithTestRoutes stores a route for. +const addrInTestPrefixV6 = "2001:db8::1" + +// testBlocklistInterval is fakeBlocklists' shortest interval. +const testBlocklistInterval = 15 * time.Minute + +// ownAddressPath is the IP lookup of the caller's own address. +const ownAddressPath = "/ip/" + +// listedUnroutedAddr is an address that fakeBlocklists lists, outside the +// routes newServerWithTestRoutes stores. +const listedUnroutedAddr = "203.0.113.1" + +// fakeBlocklists has one list, which lists addrInTestPrefixV4 and +// listedUnroutedAddr and no other address. +type fakeBlocklists struct{} + +func testListing() blocklist.Listing { + return blocklist.Listing{ + Name: "test_list", + Description: "On the test list", + Updated: time.Date(2026, 10, 27, 4, 0, 0, 0, time.UTC), + } +} + +func (fakeBlocklists) Lookup(addr netip.Addr) []blocklist.Listing { + if addr == netip.MustParseAddr(addrInTestPrefixV4) || + addr == netip.MustParseAddr(listedUnroutedAddr) { + return []blocklist.Listing{testListing()} + } + + return []blocklist.Listing{} +} + +func (fakeBlocklists) ShortestInterval() time.Duration { + return testBlocklistInterval +} + +// TestIPLookupReportsBlocklists looks up two routed addresses, one of them on +// a blocklist. The listed address's result names the list, says what it means +// and when it was updated. The other's result has an empty list. +func TestIPLookupReportsBlocklists(t *testing.T) { + t.Parallel() + + s := newServerWithTestRoutes(t) + s.resolver = &fakeResolver{} + s.SetBlocklists(fakeBlocklists{}) + + rec := httptest.NewRecorder() + s.router.ServeHTTP(rec, httptest.NewRequestWithContext(t.Context(), + http.MethodGet, + "/api/v1/ip/"+addrInTestPrefixV4+","+addrInTestPrefixV6, nil)) + + if rec.Code != http.StatusOK { + t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body) + } + + var body struct { + Data IPLookupResponse `json:"data"` + } + + err := json.Unmarshal(rec.Body.Bytes(), &body) + if err != nil { + t.Fatalf("decoding the answer %q: %v", rec.Body, err) + } + + results := body.Data.Results + if len(results) != 2 { + t.Fatalf("answer %s, want two results", rec.Body) + } + + want := testListing() + + listed := results[0].Blocklists + if len(listed) != 1 || listed[0].Name != want.Name || + listed[0].Description != want.Description || + !listed[0].Updated.Equal(want.Updated) { + t.Errorf("%s: blocklists %+v, want [%+v]", results[0].IP, listed, want) + } + + if !strings.Contains(rec.Body.String(), `"blocklists":[]`) { + t.Errorf("answer %s, want an empty blocklists list for %s", + rec.Body, addrInTestPrefixV6) + } +} + +// TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute looks up a listed +// address that has no live route, alone and beside a routed address. Each +// answer is 200 and has a result for the address, with its blocklists and no +// route fields. Before the fix the lookup of that address alone answered +// 404, and beside a routed address it was only named in errors, so its +// blocklists were never reported. +func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) { + t.Parallel() + + s := newServerWithTestRoutes(t) + s.resolver = &fakeResolver{} + s.SetBlocklists(fakeBlocklists{}) + + for _, target := range []string{ + listedUnroutedAddr, + addrInTestPrefixV4 + "," + listedUnroutedAddr, + } { + rec := httptest.NewRecorder() + s.router.ServeHTTP(rec, httptest.NewRequestWithContext(t.Context(), + http.MethodGet, "/api/v1/ip/"+target, nil)) + + var body struct { + Data IPLookupResponse `json:"data"` + } + + err := json.Unmarshal(rec.Body.Bytes(), &body) + if rec.Code != http.StatusOK || err != nil { + t.Errorf("%s: status %d (%v), want %d; body %s", + target, rec.Code, err, http.StatusOK, rec.Body) + + continue + } + + i := slices.IndexFunc(body.Data.Results, func(r IPLookupResult) bool { + return r.IP == listedUnroutedAddr + }) + if i < 0 || len(body.Data.Errors) != 0 { + t.Errorf("%s: answer %s, want a result for %s and no errors", + target, rec.Body, listedUnroutedAddr) + + continue + } + + result := body.Data.Results[i] + if result.IPInfo != nil { + t.Errorf("%s: result for %s has route fields %+v", + target, listedUnroutedAddr, result.IPInfo) + } + + if len(result.Blocklists) != 1 || + result.Blocklists[0].Name != testListing().Name { + t.Errorf("%s: blocklists %+v, want [%+v]", + target, result.Blocklists, testListing()) + } + } +} + +// TestIPLookupAnswersMayBeCached checks the Cache-Control header of the IP +// lookup answers. Caches may keep them for the blocklists' shortest interval, +// and only the caller's own browser may keep the answer about the caller's +// own address. +func TestIPLookupAnswersMayBeCached(t *testing.T) { + t.Parallel() + + s := newServerWithTestRoutes(t) + s.resolver = &fakeResolver{} + s.SetBlocklists(fakeBlocklists{}) + + tests := []struct { + path string + want string + }{ + {"/api/v1/ip/" + addrInTestPrefixV4, "public, max-age=900"}, + {ownAddressPath + addrInTestPrefixV4, "public, max-age=900"}, + {ownAddressPath, "private, max-age=900"}, + } + + for _, tt := range tests { + req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, + tt.path, nil) + req.RemoteAddr = net.JoinHostPort(addrInTestPrefixV4, "1234") + + rec := httptest.NewRecorder() + s.router.ServeHTTP(rec, req) + + if rec.Code != http.StatusOK { + t.Errorf("%s: status %d, want %d; body %s", + tt.path, rec.Code, http.StatusOK, rec.Body) + + continue + } + + if got := rec.Header().Get(cacheControlHeader); got != tt.want { + t.Errorf("%s: Cache-Control %q, want %q", tt.path, got, tt.want) + } + } +} + // The host name TestIPLookupHidesPrivateAddresses resolves to a private // address, and that address. const ( diff --git a/internal/server/server.go b/internal/server/server.go index 58e9ffb..b0cb889 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -15,6 +15,7 @@ import ( "strconv" "time" + "git.eeqj.de/sneak/routewatch/internal/blocklist" "git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/logger" @@ -36,6 +37,13 @@ type ASNFetcher interface { GetStats() ASNFetcherStats } +// Blocklists tells which public blocklists list an address, from copies it +// keeps in memory, as *blocklist.Blocklists does. +type Blocklists interface { + Lookup(addr netip.Addr) []blocklist.Listing + ShortestInterval() time.Duration +} + // dnsResolver looks up the addresses of host names and the names of addresses // in DNS, as *net.Resolver does. type dnsResolver interface { @@ -52,6 +60,7 @@ type Server struct { port int srv *http.Server asnFetcher ASNFetcher + blocklists Blocklists // trustedProxies are the reverse proxies whose X-Forwarded-For header // RealIPMiddleware believes @@ -148,3 +157,9 @@ func (s *Server) Stop(ctx context.Context) error { func (s *Server) SetASNFetcher(fetcher ASNFetcher) { s.asnFetcher = fetcher } + +// SetBlocklists sets the blocklists the IP lookups report. Call it before +// Start: the handlers read the blocklists without a lock. +func (s *Server) SetBlocklists(blocklists Blocklists) { + s.blocklists = blocklists +} diff --git a/internal/templates/index.html b/internal/templates/index.html index 4fac31b..1760827 100644 --- a/internal/templates/index.html +++ b/internal/templates/index.html @@ -606,8 +606,8 @@ // IP information API card: show the curl command that asks /ip/ about // the visitor's own address, then make that same request and show the - // JSON it answers, an error answer such as a 404 included. The answer - // holds outside data, so it is set with textContent, never innerHTML. + // JSON it answers, an error answer included. The answer holds outside + // data, so it is set with textContent, never innerHTML. const ipAPICommand = document.getElementById('ip-api-command'); const ipAPIAnswer = document.getElementById('ip-api-answer'); const ipAPIURL = window.location.origin + '/ip/' + ipAPICommand.dataset.clientIp;