Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
ba74640c85 | ||
 clawbot
|
cf50d195ad |
No files matched your search
+3
-2
@@ -93,11 +93,12 @@
|
||||
**/*.dylib
|
||||
|
||||
# Local databases. The image carries a source archive of the whole build
|
||||
# context, so these would otherwise ship inside it. The daemon's database
|
||||
# is db.sqlite, with its -wal and -shm files while it runs.
|
||||
# context, so these would otherwise ship inside it. The daemon's databases
|
||||
# are db.sqlite and cache.db, with their -wal and -shm files while it runs.
|
||||
**/*.db
|
||||
**/*.db-journal
|
||||
**/*.db-wal
|
||||
**/*.db-shm
|
||||
**/*.sqlite
|
||||
**/*.sqlite-wal
|
||||
**/*.sqlite-shm
|
||||
|
||||
+3
-1
@@ -68,10 +68,12 @@ go.work
|
||||
go.work.sum
|
||||
|
||||
# Local databases, temporary files and debug output. The daemon's
|
||||
# database is db.sqlite, with its -wal and -shm files while it runs.
|
||||
# databases are db.sqlite and cache.db, with their -wal and -shm files
|
||||
# while it runs.
|
||||
*.db
|
||||
*.db-journal
|
||||
*.db-wal
|
||||
*.db-shm
|
||||
*.sqlite
|
||||
*.sqlite-wal
|
||||
*.sqlite-shm
|
||||
|
||||
@@ -44,8 +44,7 @@ its abuse contact. RouteWatch keeps them in its database and fetches them again
|
||||
once they are 30 days old. An IP lookup answers from the stored details and
|
||||
never waits for WHOIS.
|
||||
|
||||
An IP lookup also says which public blocklists list the address (see
|
||||
Blocklists).
|
||||
An IP lookup also says which public IP lists hold the address (see IP lists).
|
||||
|
||||
## Getting started
|
||||
|
||||
@@ -65,7 +64,7 @@ docker run --memory=5g --memory-swap=5g -p 8080:8080 \
|
||||
```
|
||||
|
||||
The memory limit is explained in Memory. The volume holds the database and the
|
||||
copies of the blocklists.
|
||||
copies of the IP lists.
|
||||
|
||||
To run the binary on the host instead, build it with `make build`. The build
|
||||
needs Go and a C compiler, because the SQLite library is written in C:
|
||||
@@ -282,10 +281,14 @@ answer holds `query`, the text looked up, and `results`, with one entry for each
|
||||
address. Every entry has the same fields:
|
||||
|
||||
- `ip`: the address
|
||||
- `ptr`: the address's PTR names, the host names that DNS gives for it, or an
|
||||
empty list when it has none
|
||||
- `blocklists`: the public blocklists the address is on (see Blocklists), or an
|
||||
empty list when it is on none
|
||||
- `ptr`: the address's PTR names, the host names that DNS gives for it (see PTR
|
||||
lookups), or an empty list when it has none
|
||||
- `flags`: yes-or-no facts about the address:
|
||||
- `is_tor`: true when the address is on the `tor_exits` list
|
||||
- `is_bogon`: true when the address is reserved, so that no network should
|
||||
announce it on the internet
|
||||
- `ip_lists`: the public IP lists the address is on (see IP lists), or an empty
|
||||
list when it is on none
|
||||
|
||||
Every other field is about the prefix in the routing table that holds the
|
||||
address. Each is null when no prefix in the routing table holds the address:
|
||||
@@ -327,11 +330,8 @@ address, such as one that resolves to a private address, is reported as
|
||||
- `{ip}` comes to more than 10 addresses once its names are resolved
|
||||
- nothing in `{ip}` is an IP address or a host name with a public address
|
||||
|
||||
Each PTR lookup gets 2 seconds. An address whose lookup takes longer is answered
|
||||
without PTR names.
|
||||
|
||||
A successful answer carries `Cache-Control` with a `max-age` of 15 minutes, the
|
||||
shortest blocklist interval. The answer for the caller's own address (`/ip/`) is
|
||||
shortest IP list interval. The answer for the caller's own address (`/ip/`) is
|
||||
`private`, and the others are `public`.
|
||||
|
||||
### Errors
|
||||
@@ -391,10 +391,10 @@ inline `<style>` and `<script>` blocks. Each answer's policy names a random
|
||||
value, a nonce, made for that answer. The browser applies or runs only the
|
||||
blocks that carry it.
|
||||
|
||||
## Blocklists
|
||||
## IP lists
|
||||
|
||||
An IP lookup reports which public blocklists list each address. Each entry in
|
||||
`blocklists` has:
|
||||
An IP lookup reports which public IP lists hold each address. Each entry in
|
||||
`ip_lists` has:
|
||||
|
||||
- `name`: the list's name, from the lists below
|
||||
- `description`: what being on the list means
|
||||
@@ -431,8 +431,8 @@ can be downloaded whole without an account or a key.
|
||||
How RouteWatch keeps the lists:
|
||||
|
||||
- It holds each list in memory, so a lookup makes no outside request.
|
||||
- It keeps a copy of each list in the `blocklists` directory of the state
|
||||
directory (see Configuration). Each copy is a file named after its list.
|
||||
- It keeps a copy of each list in the `iplists` directory of the state directory
|
||||
(see Configuration). Each copy is a file named after its list.
|
||||
- Beside each copy, a file ending in `.validators` keeps the `Last-Modified` and
|
||||
`ETag` headers the publisher sent with that copy.
|
||||
- At startup it loads each list from its copy. It downloads a list at once when
|
||||
@@ -440,10 +440,47 @@ How RouteWatch keeps the lists:
|
||||
- A download sends the copy's `Last-Modified` as `If-Modified-Since` and its
|
||||
`ETag` as `If-None-Match`. The publisher then sends the list only when it has
|
||||
changed.
|
||||
- A download fails when it takes over a minute or its file is over 16 MiB.
|
||||
- A download fails when it takes over a minute or its file is over 200 MiB.
|
||||
- When a download fails, or its file does not parse, RouteWatch keeps the
|
||||
previous copy and logs the failure. It tries again one interval later.
|
||||
|
||||
## PTR lookups
|
||||
|
||||
An IP lookup finds each address's PTR names over DNS-over-HTTPS. It asks these
|
||||
public resolvers in order, until one answers:
|
||||
|
||||
- Google: `https://dns.google/dns-query`
|
||||
- Cloudflare: `https://cloudflare-dns.com/dns-query`
|
||||
- AdGuard: `https://unfiltered.adguard-dns.com/dns-query`
|
||||
- Quad9: `https://dns.quad9.net/dns-query`
|
||||
|
||||
One address's lookup gets 2 seconds across all four. Each resolver gets an equal
|
||||
share of the time left, so one that does not answer leaves time for the next.
|
||||
When none answers in time, the address's `ptr` is an empty list.
|
||||
|
||||
A reserved address, one whose `is_bogon` flag is true, is not sent to the
|
||||
resolvers. Its `ptr` is an empty list.
|
||||
|
||||
What goes outside: each resolver asked receives the address being looked up. For
|
||||
`/ip/` and the home page, that address is the visitor's own. A lookup answered
|
||||
from the cache sends nothing.
|
||||
|
||||
How RouteWatch keeps the answers:
|
||||
|
||||
- It holds answers in memory. An answer that has not expired is served from
|
||||
there.
|
||||
- It writes new answers every 5 seconds to `cache.db` in the state directory
|
||||
(see Configuration), and loads them from there at startup.
|
||||
- An answer expires after the TTL of its records, the shortest one when there
|
||||
are several. RouteWatch keeps it for at least 60 seconds and at most 24 hours.
|
||||
- When an address has no PTR names, that answer expires after the SOA record's
|
||||
minimum field, or the record's own TTL when that is shorter. RouteWatch keeps
|
||||
it for at most 1 hour.
|
||||
- An answer with no PTR names and no SOA record is not kept. Neither is one
|
||||
whose SOA record has a TTL or minimum field of 0.
|
||||
- A failed lookup is not kept, so the next lookup of the address asks again.
|
||||
- Every 10 minutes it deletes expired answers from memory and from `cache.db`.
|
||||
|
||||
## Configuration
|
||||
|
||||
RouteWatch reads its settings from environment variables. It takes no
|
||||
@@ -492,7 +529,13 @@ by the client. For a request from any other address, they use the address the
|
||||
request came from, whatever headers it carries. `X-Real-IP` and `True-Client-IP`
|
||||
are not read.
|
||||
|
||||
The state directory holds the database, `db.sqlite`, and the blocklist copies:
|
||||
The state directory holds:
|
||||
|
||||
- `db.sqlite`: the database
|
||||
- `cache.db`: the answers of PTR lookups
|
||||
- `iplists`: the IP list copies
|
||||
|
||||
It is at:
|
||||
|
||||
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
|
||||
- Linux and the BSDs, as root: `/var/lib/berlin.sneak.app.routewatch/`
|
||||
@@ -647,8 +690,10 @@ The parts:
|
||||
wait starts at 15 seconds. It halves after a success, down to 1 second, and
|
||||
grows by half after a failure, up to a minute. An IP lookup that finds an AS
|
||||
with old or missing details asks the fetcher for it at once, without the wait.
|
||||
- The blocklists part downloads the blocklists and answers which lists hold an
|
||||
- The IP lists part downloads the IP lists and answers which lists hold an
|
||||
address.
|
||||
- The PTR cache looks up the PTR names of addresses and keeps the answers (see
|
||||
PTR lookups).
|
||||
- The database maintainer checkpoints the database every 5 seconds: it folds the
|
||||
write-ahead log, where SQLite first writes changes, into the database file. It
|
||||
reclaims free space every 10 minutes. Every hour it refreshes the statistics
|
||||
@@ -756,8 +801,9 @@ On `SIGTERM` or `SIGINT` the daemon stops its parts in this order:
|
||||
2. the handlers, each of which writes what it holds
|
||||
3. the WHOIS fetcher
|
||||
4. the database maintainer
|
||||
5. the blocklist downloads
|
||||
5. the IP list downloads
|
||||
6. the HTTP server
|
||||
7. the PTR cache, which writes what it holds
|
||||
|
||||
Last, the daemon checkpoints and closes the database. It gives all this 8
|
||||
seconds and then exits with status 0. A stop that is still running after 9
|
||||
@@ -778,6 +824,8 @@ What a stop can lose:
|
||||
- When the 8 seconds run out, the streamer and the handlers log what they could
|
||||
not hand over or write. The database is then not closed. When it next opens,
|
||||
SQLite rolls back any write the exit cut off.
|
||||
- When the 8 seconds run out, the PTR answers not yet in `cache.db` are lost.
|
||||
RouteWatch asks the resolvers for them again the next time they are needed.
|
||||
|
||||
### Code layout
|
||||
|
||||
@@ -790,12 +838,14 @@ routewatch/
|
||||
├── docs/
|
||||
│ └── message-examples.json # Examples of RIS Live messages
|
||||
├── internal/
|
||||
│ ├── blocklist/ # Public IP blocklists: downloads and lookups
|
||||
│ ├── bogon/ # Reserved prefixes and AS numbers
|
||||
│ ├── cachedb/ # cache.db, the caches of outside answers
|
||||
│ ├── config/ # Settings from the environment
|
||||
│ ├── database/ # SQLite storage
|
||||
│ ├── iplist/ # Public IP lists: downloads and lookups
|
||||
│ ├── logger/ # Structured logging
|
||||
│ ├── metrics/ # Feed and route update figures
|
||||
│ ├── ptr/ # PTR lookups over DNS-over-HTTPS, and their cache
|
||||
│ ├── ristypes/ # The RIS Live message types
|
||||
│ ├── routetable/ # The whole routing table in memory
|
||||
│ ├── routewatch/ # The daemon: startup, handlers, WHOIS fetcher
|
||||
@@ -844,6 +894,13 @@ announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
|
||||
|
||||
The indexes are in `internal/database/schema.sql`.
|
||||
|
||||
`cache.db` is a separate file, with one table:
|
||||
|
||||
```sql
|
||||
-- The answers of PTR lookups. names is empty for an address with no PTR names.
|
||||
hostnames(address PRIMARY KEY, names, expires_at)
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
The `make` targets:
|
||||
|
||||
@@ -41,9 +41,6 @@ https://git.eeqj.de/sneak/routewatch/issues/225, also on `next`, takes the peers
|
||||
whose sessions go down together out of the table in one walk. A 24-hour run of
|
||||
`9bd0e91` on the live feed is checking whether any handler still drops updates.
|
||||
|
||||
One question waits for sneak: which fields the IP lookup answer gains
|
||||
(https://git.eeqj.de/sneak/routewatch/issues/203).
|
||||
|
||||
The order of the project's remaining work is kept on
|
||||
https://git.eeqj.de/sneak/routewatch/issues/86.
|
||||
|
||||
@@ -56,6 +53,14 @@ problem the run finds.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-11-10: the IP lookup finds PTR names over DNS-over-HTTPS and caches them
|
||||
for the record's TTL, in memory and in `cache.db` (closes
|
||||
https://git.eeqj.de/sneak/routewatch/issues/238).
|
||||
- 2026-11-10: each result of the IP lookup has a `flags` object with `is_tor`
|
||||
and `is_bogon`. The public IP lists are named `ip_lists` in the answer,
|
||||
`internal/iplist` in the code and `iplists` in the state directory. A list
|
||||
file may be up to 200 MiB (closes
|
||||
https://git.eeqj.de/sneak/routewatch/issues/233).
|
||||
- 2026-11-05: the prefix page lists the prefix's origin changes, newest first.
|
||||
The routing table records each change, and the prefix handler writes them to a
|
||||
new `origin_changes` table. The database maintainer prunes each prefix to its
|
||||
|
||||
@@ -9,10 +9,11 @@ require (
|
||||
github.com/google/btree v1.1.3
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/mattn/go-sqlite3 v1.14.29
|
||||
github.com/miekg/dns v1.1.73
|
||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
|
||||
go.uber.org/fx v1.24.0
|
||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb
|
||||
golang.org/x/term v0.33.0
|
||||
golang.org/x/term v0.45.0
|
||||
golang.org/x/text v0.43.0
|
||||
)
|
||||
|
||||
@@ -20,5 +21,6 @@ require (
|
||||
go.uber.org/dig v1.19.0 // indirect
|
||||
go.uber.org/multierr v1.10.0 // indirect
|
||||
go.uber.org/zap v1.26.0 // indirect
|
||||
golang.org/x/sys v0.34.0 // indirect
|
||||
golang.org/x/net v0.57.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
)
|
||||
@@ -12,6 +12,8 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/mattn/go-sqlite3 v1.14.29 h1:1O6nRLJKvsi1H2Sj0Hzdfojwt8GiGKm+LOfLaBFaouQ=
|
||||
github.com/mattn/go-sqlite3 v1.14.29/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
|
||||
github.com/miekg/dns v1.1.73 h1:uhT8nJxmTrPJYClxVxTCX+CVn6qnzSiybRk72Z6DgrE=
|
||||
github.com/miekg/dns v1.1.73/go.mod h1:RW2Obtfd5NZHvOFe3zYG0W8koWOQtAzyHaLo8vASBuQ=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 h1:bsUq1dX0N8AOIL7EB/X911+m4EHsnWEHeJ0c+3TTBrg=
|
||||
@@ -30,10 +32,14 @@ go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
|
||||
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
|
||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb h1:XBM4hvfwGAttkkiTIFfeigdfcL1xIfdKXqFdgiHGtDs=
|
||||
go4.org/netipx v0.0.0-20260823151212-3075585bcbeb/go.mod h1:PLyyIXexvUFg3Owu6p/WfdlivPbZJsZdgWZlrGope/Y=
|
||||
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
|
||||
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||
golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg=
|
||||
golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0=
|
||||
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
|
||||
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
|
||||
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
|
||||
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
|
||||
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
|
||||
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
||||
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
// Package cachedb opens cache.db, the SQLite file in the state directory that
|
||||
// holds RouteWatch's caches of answers from outside services. It is separate
|
||||
// from the routes database so that a cache write never queues behind a write
|
||||
// to that database.
|
||||
package cachedb
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
_ "embed" // for schema.sql
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
|
||||
_ "github.com/mattn/go-sqlite3" // CGO SQLite driver
|
||||
)
|
||||
|
||||
const (
|
||||
// fileName is the database's file in the state directory.
|
||||
fileName = "cache.db"
|
||||
|
||||
// busyTimeoutMs is how long a write waits for another write to finish.
|
||||
busyTimeoutMs = 5000
|
||||
)
|
||||
|
||||
//go:embed schema.sql
|
||||
var schema string
|
||||
|
||||
// Open opens cache.db in stateDir, creating the file and its tables when they
|
||||
// do not exist. The caller closes the database.
|
||||
func Open(ctx context.Context, stateDir string) (*sql.DB, error) {
|
||||
dsn := fmt.Sprintf(
|
||||
"file:%s?_journal_mode=WAL&_synchronous=NORMAL&_busy_timeout=%d"+
|
||||
"&_txlock=immediate",
|
||||
filepath.Join(stateDir, fileName),
|
||||
busyTimeoutMs,
|
||||
)
|
||||
|
||||
db, err := sql.Open("sqlite3", dsn)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("opening the cache database: %w", err)
|
||||
}
|
||||
|
||||
_, err = db.ExecContext(ctx, schema)
|
||||
if err != nil {
|
||||
_ = db.Close()
|
||||
|
||||
return nil, fmt.Errorf("creating the cache database's tables: %w", err)
|
||||
}
|
||||
|
||||
return db, nil
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package cachedb
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestOpenCreatesTheFileInWALMode opens a new database, and then opens it
|
||||
// again, as a restart does.
|
||||
func TestOpenCreatesTheFileInWALMode(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
|
||||
for range 2 {
|
||||
db, err := Open(t.Context(), dir)
|
||||
if err != nil {
|
||||
t.Fatalf("Open: %v", err)
|
||||
}
|
||||
|
||||
var mode string
|
||||
|
||||
err = db.QueryRowContext(t.Context(), "PRAGMA journal_mode").Scan(&mode)
|
||||
if err != nil {
|
||||
t.Fatalf("reading the journal mode: %v", err)
|
||||
}
|
||||
|
||||
if mode != "wal" {
|
||||
t.Errorf("journal mode %q, want wal", mode)
|
||||
}
|
||||
|
||||
var rows int
|
||||
|
||||
err = db.QueryRowContext(t.Context(),
|
||||
"SELECT COUNT(*) FROM hostnames").Scan(&rows)
|
||||
if err != nil {
|
||||
t.Fatalf("reading the hostnames table: %v", err)
|
||||
}
|
||||
|
||||
err = db.Close()
|
||||
if err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
_, err := os.Stat(filepath.Join(dir, fileName))
|
||||
if err != nil {
|
||||
t.Errorf("the file is not in the state directory: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestOpenFailsWhenTheDirectoryDoesNotExist checks that a state directory that
|
||||
// is not there is reported, not ignored.
|
||||
func TestOpenFailsWhenTheDirectoryDoesNotExist(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db, err := Open(t.Context(), filepath.Join(t.TempDir(), "missing"))
|
||||
if err == nil {
|
||||
_ = db.Close()
|
||||
|
||||
t.Fatal("Open succeeded in a directory that does not exist")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
-- hostnames holds the PTR names of addresses. names has the names separated by
|
||||
-- spaces, and is empty for an address with no PTR names. expires_at is in
|
||||
-- seconds since 1970.
|
||||
CREATE TABLE IF NOT EXISTS hostnames (
|
||||
address TEXT PRIMARY KEY,
|
||||
names TEXT NOT NULL,
|
||||
expires_at INTEGER NOT NULL
|
||||
);
|
||||
@@ -1,4 +1,4 @@
|
||||
package blocklist
|
||||
package iplist
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -22,7 +22,7 @@ const (
|
||||
|
||||
// maxDownloadSize is the size of the largest list file accepted, in
|
||||
// bytes. The largest list, FireHOL abusers, is about 2.5 MiB.
|
||||
maxDownloadSize = 16 << 20
|
||||
maxDownloadSize = 200 << 20
|
||||
)
|
||||
|
||||
// The request headers a download sets.
|
||||
@@ -78,10 +78,10 @@ var (
|
||||
// downloads each list once its copy is older than its Interval, and a list
|
||||
// with no copy at once. The goroutine runs until Stop: the end of ctx does
|
||||
// not stop it.
|
||||
func (b *Blocklists) Start(ctx context.Context) {
|
||||
func (b *IPLists) Start(ctx context.Context) {
|
||||
err := os.MkdirAll(b.dir, dirPermissions)
|
||||
if err != nil {
|
||||
b.logger.Error("Failed to create the blocklists directory",
|
||||
b.logger.Error("Failed to create the IP lists directory",
|
||||
"dir", b.dir, "error", err)
|
||||
}
|
||||
|
||||
@@ -90,7 +90,7 @@ func (b *Blocklists) Start(ctx context.Context) {
|
||||
for i, list := range b.lists {
|
||||
loadErr := b.loadCopy(list)
|
||||
if loadErr != nil && !errors.Is(loadErr, fs.ErrNotExist) {
|
||||
b.logger.Error("Failed to load a blocklist copy",
|
||||
b.logger.Error("Failed to load an IP list copy",
|
||||
"list", list.Name, "error", loadErr)
|
||||
}
|
||||
|
||||
@@ -104,20 +104,20 @@ func (b *Blocklists) Start(ctx context.Context) {
|
||||
b.run(ctx, nextDownload)
|
||||
}()
|
||||
|
||||
b.logger.Info("Blocklist downloads started", "dir", b.dir)
|
||||
b.logger.Info("IP list downloads started", "dir", b.dir)
|
||||
}
|
||||
|
||||
// Stop stops the downloads, cancelling one in progress, and waits until the
|
||||
// goroutine has returned or ctx ends. The copies in memory stay, so Lookup
|
||||
// still answers.
|
||||
func (b *Blocklists) Stop(ctx context.Context) {
|
||||
func (b *IPLists) Stop(ctx context.Context) {
|
||||
b.stop()
|
||||
|
||||
select {
|
||||
case <-b.done:
|
||||
b.logger.Info("Blocklist downloads stopped")
|
||||
b.logger.Info("IP list downloads stopped")
|
||||
case <-ctx.Done():
|
||||
b.logger.Warn("Stop deadline passed before the blocklist downloads " +
|
||||
b.logger.Warn("Stop deadline passed before the IP list downloads " +
|
||||
"stopped")
|
||||
}
|
||||
}
|
||||
@@ -125,7 +125,7 @@ func (b *Blocklists) Stop(ctx context.Context) {
|
||||
// run downloads each list when nextDownload says, until ctx ends. After each
|
||||
// try, whether it worked or not, the list's next download is an Interval
|
||||
// later.
|
||||
func (b *Blocklists) run(ctx context.Context, nextDownload []time.Time) {
|
||||
func (b *IPLists) run(ctx context.Context, nextDownload []time.Time) {
|
||||
defer close(b.done)
|
||||
|
||||
if len(nextDownload) == 0 {
|
||||
@@ -155,7 +155,7 @@ func (b *Blocklists) run(ctx context.Context, nextDownload []time.Time) {
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
b.logger.Error("Blocklist download failed; keeping the "+
|
||||
b.logger.Error("IP list download failed; keeping the "+
|
||||
"previous copy", "list", list.Name, "error", err)
|
||||
}
|
||||
|
||||
@@ -169,7 +169,7 @@ func (b *Blocklists) run(ctx context.Context, nextDownload []time.Time) {
|
||||
// on disk and in memory. An answer that the copy is still current marks the
|
||||
// copy as updated now. Anything else leaves the copy as it is and returns an
|
||||
// error.
|
||||
func (b *Blocklists) download(ctx context.Context, list List) error {
|
||||
func (b *IPLists) download(ctx context.Context, list List) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, downloadTimeout)
|
||||
defer cancel()
|
||||
|
||||
@@ -212,7 +212,7 @@ func (b *Blocklists) download(ctx context.Context, list List) error {
|
||||
// replaceCopy writes a new version of list, read from body, beside the list's
|
||||
// copy. Only when it parses does it replace the copy, on disk and in memory.
|
||||
// sent are the validators the publisher sent with the new version.
|
||||
func (b *Blocklists) replaceCopy(list List, body io.Reader, sent validators) error {
|
||||
func (b *IPLists) replaceCopy(list List, body io.Reader, sent validators) error {
|
||||
root, err := os.OpenRoot(b.dir)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -244,13 +244,13 @@ func (b *Blocklists) replaceCopy(list List, body io.Reader, sent validators) err
|
||||
b.setCopy(list.Name, listCopy{
|
||||
addresses: addresses, updated: time.Now().UTC(), validators: sent,
|
||||
})
|
||||
b.logger.Info("Downloaded a new version of a blocklist", "list", list.Name)
|
||||
b.logger.Info("Downloaded a new version of an IP list", "list", list.Name)
|
||||
|
||||
// Without this file, the first download after a restart asks for the
|
||||
// whole list.
|
||||
err = writeValidators(root, list.Name, sent)
|
||||
if err != nil {
|
||||
b.logger.Error("Failed to keep the validators of a blocklist copy",
|
||||
b.logger.Error("Failed to keep the validators of an IP list copy",
|
||||
"list", list.Name, "error", err)
|
||||
}
|
||||
|
||||
@@ -288,7 +288,7 @@ func writeAndParse(
|
||||
|
||||
// markCurrent records that the publisher says the list's copy is still
|
||||
// current: the copy, on disk and in memory, is marked as updated now.
|
||||
func (b *Blocklists) markCurrent(list List) error {
|
||||
func (b *IPLists) markCurrent(list List) error {
|
||||
root, err := os.OpenRoot(b.dir)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -304,7 +304,7 @@ func (b *Blocklists) markCurrent(list List) error {
|
||||
}
|
||||
|
||||
b.setUpdated(list.Name, now)
|
||||
b.logger.Debug("Blocklist copy is still current", "list", list.Name)
|
||||
b.logger.Debug("IP list copy is still current", "list", list.Name)
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -312,7 +312,7 @@ func (b *Blocklists) markCurrent(list List) error {
|
||||
// loadCopy makes the list's copy on disk, with the validators kept beside it,
|
||||
// its copy in memory. The file's modification time is when the copy was
|
||||
// updated.
|
||||
func (b *Blocklists) loadCopy(list List) error {
|
||||
func (b *IPLists) loadCopy(list List) error {
|
||||
root, err := os.OpenRoot(b.dir)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -1,7 +1,7 @@
|
||||
// Package blocklist downloads public IP blocklists on a schedule, keeps the
|
||||
// last good copy of each on disk and in memory, and answers which lists
|
||||
// contain an address.
|
||||
package blocklist
|
||||
// Package iplist downloads public IP lists on a schedule, keeps the last good
|
||||
// copy of each on disk and in memory, and answers which lists contain an
|
||||
// address.
|
||||
package iplist
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -31,7 +31,7 @@ const (
|
||||
FormatDShield
|
||||
)
|
||||
|
||||
// List is one public blocklist.
|
||||
// List is one public IP list.
|
||||
type List struct {
|
||||
// Name names the list in lookup answers, and names its copy on disk.
|
||||
Name string
|
||||
@@ -80,9 +80,9 @@ type Params struct {
|
||||
Logger *slog.Logger
|
||||
}
|
||||
|
||||
// Blocklists keeps a copy of each list and answers which lists contain an
|
||||
// IPLists keeps a copy of each list and answers which lists contain an
|
||||
// address. Start loads the copies from disk and starts downloading the lists.
|
||||
type Blocklists struct {
|
||||
type IPLists struct {
|
||||
dir string
|
||||
lists []List
|
||||
logger *slog.Logger
|
||||
@@ -97,12 +97,12 @@ type Blocklists struct {
|
||||
done chan struct{}
|
||||
}
|
||||
|
||||
// New returns a Blocklists with no copies. Start loads them.
|
||||
func New(params Params) *Blocklists {
|
||||
return &Blocklists{
|
||||
// New returns an IPLists with no copies. Start loads them.
|
||||
func New(params Params) *IPLists {
|
||||
return &IPLists{
|
||||
dir: params.Dir,
|
||||
lists: params.Lists,
|
||||
logger: params.Logger.With("component", "blocklists"),
|
||||
logger: params.Logger.With("component", "iplists"),
|
||||
copies: make(map[string]listCopy),
|
||||
done: make(chan struct{}),
|
||||
}
|
||||
@@ -111,7 +111,7 @@ func New(params Params) *Blocklists {
|
||||
// Lookup returns a Listing for each list whose copy contains addr, in the
|
||||
// order of the lists given to New. It makes no outside request. When no list
|
||||
// contains addr, it returns an empty list, not nil.
|
||||
func (b *Blocklists) Lookup(addr netip.Addr) []Listing {
|
||||
func (b *IPLists) Lookup(addr netip.Addr) []Listing {
|
||||
b.mu.RLock()
|
||||
defer b.mu.RUnlock()
|
||||
|
||||
@@ -133,7 +133,7 @@ func (b *Blocklists) Lookup(addr netip.Addr) []Listing {
|
||||
|
||||
// ShortestInterval returns the shortest Interval of the lists. That is the
|
||||
// soonest a copy can change, and so how long a lookup answer stays current.
|
||||
func (b *Blocklists) ShortestInterval() time.Duration {
|
||||
func (b *IPLists) ShortestInterval() time.Duration {
|
||||
var shortest time.Duration
|
||||
|
||||
for i, list := range b.lists {
|
||||
@@ -147,7 +147,7 @@ func (b *Blocklists) ShortestInterval() time.Duration {
|
||||
|
||||
// updated returns when the named list's copy was last downloaded or found
|
||||
// current, or the zero time when the list has no copy.
|
||||
func (b *Blocklists) updated(name string) time.Time {
|
||||
func (b *IPLists) updated(name string) time.Time {
|
||||
b.mu.RLock()
|
||||
defer b.mu.RUnlock()
|
||||
|
||||
@@ -156,7 +156,7 @@ func (b *Blocklists) updated(name string) time.Time {
|
||||
|
||||
// copyValidators returns the validators of the named list's copy. They are
|
||||
// empty when the list has no copy.
|
||||
func (b *Blocklists) copyValidators(name string) validators {
|
||||
func (b *IPLists) copyValidators(name string) validators {
|
||||
b.mu.RLock()
|
||||
defer b.mu.RUnlock()
|
||||
|
||||
@@ -164,7 +164,7 @@ func (b *Blocklists) copyValidators(name string) validators {
|
||||
}
|
||||
|
||||
// setCopy makes c the named list's copy.
|
||||
func (b *Blocklists) setCopy(name string, c listCopy) {
|
||||
func (b *IPLists) setCopy(name string, c listCopy) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
|
||||
@@ -172,7 +172,7 @@ func (b *Blocklists) setCopy(name string, c listCopy) {
|
||||
}
|
||||
|
||||
// setUpdated records that the named list's copy was found current at t.
|
||||
func (b *Blocklists) setUpdated(name string, t time.Time) {
|
||||
func (b *IPLists) setUpdated(name string, t time.Time) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
|
||||
+107
-28
@@ -1,4 +1,4 @@
|
||||
package blocklist
|
||||
package iplist
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -121,7 +121,7 @@ func testList(url string, format Format) List {
|
||||
}
|
||||
}
|
||||
|
||||
func newTestBlocklists(dir string, list List) *Blocklists {
|
||||
func newTestIPLists(dir string, list List) *IPLists {
|
||||
return New(Params{
|
||||
Dir: dir,
|
||||
Lists: []List{list},
|
||||
@@ -130,7 +130,7 @@ func newTestBlocklists(dir string, list List) *Blocklists {
|
||||
}
|
||||
|
||||
// lookup looks addr up in b.
|
||||
func lookup(t *testing.T, b *Blocklists, addr string) []Listing {
|
||||
func lookup(t *testing.T, b *IPLists, addr string) []Listing {
|
||||
t.Helper()
|
||||
|
||||
return b.Lookup(netip.MustParseAddr(addr))
|
||||
@@ -188,7 +188,7 @@ func TestLookupFindsListedAddresses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, url := newPublisher(t, tt.body)
|
||||
b := newTestBlocklists(t.TempDir(), testList(url, tt.format))
|
||||
b := newTestIPLists(t.TempDir(), testList(url, tt.format))
|
||||
|
||||
err := b.download(t.Context(), b.lists[0])
|
||||
if err != nil {
|
||||
@@ -228,10 +228,6 @@ func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) {
|
||||
}{
|
||||
{"server error", http.StatusInternalServerError, "unavailable"},
|
||||
{"not a list", http.StatusOK, "<!DOCTYPE html>\n<html></html>\n"},
|
||||
{
|
||||
"over the size limit", http.StatusOK,
|
||||
strings.Repeat(listedAddr+"\n", maxDownloadSize/len(listedAddr)),
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -240,7 +236,7 @@ func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) {
|
||||
|
||||
dir := t.TempDir()
|
||||
p, url := newPublisher(t, goodListBody)
|
||||
b := newTestBlocklists(dir, testList(url, FormatLines))
|
||||
b := newTestIPLists(dir, testList(url, FormatLines))
|
||||
|
||||
err := b.download(t.Context(), b.lists[0])
|
||||
if err != nil {
|
||||
@@ -262,23 +258,106 @@ func TestFailedDownloadKeepsThePreviousCopy(t *testing.T) {
|
||||
t.Errorf("listings %+v after the failure, want %+v", after, before)
|
||||
}
|
||||
|
||||
copies := os.DirFS(dir)
|
||||
|
||||
onDisk, err := fs.ReadFile(copies, testListName)
|
||||
if err != nil || string(onDisk) != goodListBody {
|
||||
t.Errorf("copy on disk %q (%v), want %q", onDisk, err, goodListBody)
|
||||
}
|
||||
|
||||
_, err = fs.Stat(copies, testListName+downloadSuffix)
|
||||
if !errors.Is(err, fs.ErrNotExist) {
|
||||
t.Errorf("the failed download's file is still there (%v)", err)
|
||||
}
|
||||
checkGoodCopyOnDisk(t, dir)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// checkGoodCopyOnDisk checks that dir holds the good version of the test list
|
||||
// and no part of a failed download.
|
||||
func checkGoodCopyOnDisk(t *testing.T, dir string) {
|
||||
t.Helper()
|
||||
|
||||
copies := os.DirFS(dir)
|
||||
|
||||
onDisk, err := fs.ReadFile(copies, testListName)
|
||||
if err != nil || string(onDisk) != goodListBody {
|
||||
t.Errorf("copy on disk %q (%v), want %q", onDisk, err, goodListBody)
|
||||
}
|
||||
|
||||
_, err = fs.Stat(copies, testListName+downloadSuffix)
|
||||
if !errors.Is(err, fs.ErrNotExist) {
|
||||
t.Errorf("the failed download's file is still there (%v)", err)
|
||||
}
|
||||
}
|
||||
|
||||
// commentLines is an endless io.Reader of comment lines. It lets a test send
|
||||
// a large list file without holding the file in memory.
|
||||
type commentLines struct {
|
||||
offset int
|
||||
}
|
||||
|
||||
func (c *commentLines) Read(p []byte) (int, error) {
|
||||
const line = "# a comment line that pads a large list file\n"
|
||||
|
||||
for filled := 0; filled < len(p); {
|
||||
copied := copy(p[filled:], line[c.offset:])
|
||||
filled += copied
|
||||
c.offset = (c.offset + copied) % len(line)
|
||||
}
|
||||
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
// largeListFile returns a reader of a list file of size bytes: comment lines,
|
||||
// then newListedAddr on the last line.
|
||||
func largeListFile(size int64) io.Reader {
|
||||
lastLine := "\n" + newListedAddr + "\n"
|
||||
|
||||
return io.MultiReader(
|
||||
io.LimitReader(&commentLines{}, size-int64(len(lastLine))),
|
||||
strings.NewReader(lastLine),
|
||||
)
|
||||
}
|
||||
|
||||
// TestLargeFileUnderTheSizeLimitIsAccepted replaces a list with a 199 MiB
|
||||
// file. The address on its last line is listed, so the whole file was read.
|
||||
func TestLargeFileUnderTheSizeLimitIsAccepted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const size = 199 << 20
|
||||
|
||||
b := newTestIPLists(t.TempDir(), testList("", FormatLines))
|
||||
|
||||
err := b.replaceCopy(b.lists[0], largeListFile(size), validators{})
|
||||
if err != nil {
|
||||
t.Fatalf("replaceCopy: %v", err)
|
||||
}
|
||||
|
||||
if listings := lookup(t, b, newListedAddr); len(listings) != 1 {
|
||||
t.Errorf("listings %+v, want one", listings)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFileOverTheSizeLimitKeepsThePreviousCopy downloads a good version of a
|
||||
// list, then replaces it with a file one byte over the limit. The attempt
|
||||
// fails, and the good copy stays in memory and on disk.
|
||||
func TestFileOverTheSizeLimitKeepsThePreviousCopy(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
_, url := newPublisher(t, goodListBody)
|
||||
b := newTestIPLists(dir, testList(url, FormatLines))
|
||||
|
||||
err := b.download(t.Context(), b.lists[0])
|
||||
if err != nil {
|
||||
t.Fatalf("download: %v", err)
|
||||
}
|
||||
|
||||
err = b.replaceCopy(b.lists[0], largeListFile(maxDownloadSize+1), validators{})
|
||||
if !errors.Is(err, errTooLarge) {
|
||||
t.Fatalf("replaceCopy: %v, want %v", err, errTooLarge)
|
||||
}
|
||||
|
||||
if listings := lookup(t, b, listedAddr); len(listings) != 1 {
|
||||
t.Errorf("listings of %s %+v, want one", listedAddr, listings)
|
||||
}
|
||||
|
||||
checkGoodCopyOnDisk(t, dir)
|
||||
}
|
||||
|
||||
// TestRestartLoadsCopiesFromDisk downloads a list, then starts a new
|
||||
// Blocklists on the same directory, as a restart does. The new one answers
|
||||
// IPLists on the same directory, as a restart does. The new one answers
|
||||
// from the copy on disk at once, and does not download the list again while
|
||||
// the copy is younger than the list's interval.
|
||||
func TestRestartLoadsCopiesFromDisk(t *testing.T) {
|
||||
@@ -288,7 +367,7 @@ func TestRestartLoadsCopiesFromDisk(t *testing.T) {
|
||||
p, url := newPublisher(t, goodListBody)
|
||||
list := testList(url, FormatLines)
|
||||
|
||||
first := newTestBlocklists(dir, list)
|
||||
first := newTestIPLists(dir, list)
|
||||
|
||||
err := first.download(t.Context(), list)
|
||||
if err != nil {
|
||||
@@ -297,7 +376,7 @@ func TestRestartLoadsCopiesFromDisk(t *testing.T) {
|
||||
|
||||
downloaded := lookup(t, first, listedAddr)[0].Updated
|
||||
|
||||
second := newTestBlocklists(dir, list)
|
||||
second := newTestIPLists(dir, list)
|
||||
second.Start(t.Context())
|
||||
second.Stop(t.Context())
|
||||
|
||||
@@ -328,7 +407,7 @@ func TestStartRefreshesAnOldCopy(t *testing.T) {
|
||||
lastModified := time.Now().Add(-3 * list.Interval).UTC().Truncate(time.Second)
|
||||
p.publish(goodListBody, lastModified, testETag)
|
||||
|
||||
err := newTestBlocklists(dir, list).download(t.Context(), list)
|
||||
err := newTestIPLists(dir, list).download(t.Context(), list)
|
||||
if err != nil {
|
||||
t.Fatalf("download: %v", err)
|
||||
}
|
||||
@@ -340,7 +419,7 @@ func TestStartRefreshesAnOldCopy(t *testing.T) {
|
||||
t.Fatalf("Chtimes: %v", err)
|
||||
}
|
||||
|
||||
b := newTestBlocklists(dir, list)
|
||||
b := newTestIPLists(dir, list)
|
||||
b.Start(t.Context())
|
||||
|
||||
deadline := time.Now().Add(maxWait)
|
||||
@@ -381,7 +460,7 @@ func TestNewVersionOlderThanTheLastDownloadReplacesTheCopy(t *testing.T) {
|
||||
|
||||
p, url := newPublisher(t, goodListBody)
|
||||
list := testList(url, FormatLines)
|
||||
b := newTestBlocklists(t.TempDir(), list)
|
||||
b := newTestIPLists(t.TempDir(), list)
|
||||
|
||||
firstVersion := time.Now().Add(-2 * list.Interval).UTC().Truncate(time.Second)
|
||||
p.publish(goodListBody, firstVersion, "")
|
||||
@@ -418,7 +497,7 @@ func TestDownloadWithoutLastModifiedOrETagSendsNoCondition(t *testing.T) {
|
||||
|
||||
p, url := newPublisher(t, goodListBody)
|
||||
list := testList(url, FormatLines)
|
||||
b := newTestBlocklists(t.TempDir(), list)
|
||||
b := newTestIPLists(t.TempDir(), list)
|
||||
|
||||
err := b.download(t.Context(), list)
|
||||
if err != nil {
|
||||
@@ -455,7 +534,7 @@ func TestStopCancelsADownloadInProgress(t *testing.T) {
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
b := newTestBlocklists(t.TempDir(), testList(srv.URL, FormatLines))
|
||||
b := newTestIPLists(t.TempDir(), testList(srv.URL, FormatLines))
|
||||
b.Start(t.Context())
|
||||
|
||||
select {
|
||||
@@ -1,7 +1,10 @@
|
||||
package blocklist
|
||||
package iplist
|
||||
|
||||
import "time"
|
||||
|
||||
// TorExitsList is the name of the list of Tor exit nodes.
|
||||
const TorExitsList = "tor_exits"
|
||||
|
||||
// How often each list is downloaded. No list is downloaded more often than
|
||||
// its publisher updates it or allows.
|
||||
const (
|
||||
@@ -30,7 +33,7 @@ const (
|
||||
torExitsInterval = time.Hour
|
||||
)
|
||||
|
||||
// Lists returns the public blocklists routewatch downloads. Each can be
|
||||
// Lists returns the public IP lists routewatch downloads. Each can be
|
||||
// downloaded whole without an account or a key. Spamhaus merged its EDROP
|
||||
// list into DROP, so DROP covers both.
|
||||
func Lists() []List {
|
||||
@@ -83,7 +86,7 @@ func Lists() []List {
|
||||
Format: FormatSpamhausJSON,
|
||||
},
|
||||
{
|
||||
Name: "tor_exits",
|
||||
Name: TorExitsList,
|
||||
Description: "A Tor exit node, where traffic from the Tor network leaves",
|
||||
URL: "https://check.torproject.org/torbulkexitlist",
|
||||
Interval: torExitsInterval,
|
||||
@@ -1,4 +1,4 @@
|
||||
package blocklist
|
||||
package iplist
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
@@ -0,0 +1,313 @@
|
||||
// Package ptr looks up the PTR names of addresses over DNS-over-HTTPS. It
|
||||
// keeps each answer in memory for as long as DNS allows, and in cache.db so
|
||||
// that the answers survive a restart.
|
||||
package ptr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
||||
"github.com/miekg/dns"
|
||||
)
|
||||
|
||||
const (
|
||||
// flushInterval is how often new answers are written to cache.db.
|
||||
flushInterval = 5 * time.Second
|
||||
|
||||
// sweepInterval is how often expired answers are deleted.
|
||||
sweepInterval = 10 * time.Minute
|
||||
|
||||
// nameSeparator separates the names in a row of the hostnames table.
|
||||
nameSeparator = " "
|
||||
)
|
||||
|
||||
// Params is what New needs.
|
||||
type Params struct {
|
||||
// DB is the open cache.db.
|
||||
DB *sql.DB
|
||||
|
||||
// Resolvers are the URLs of the DNS-over-HTTPS resolvers to ask, in order.
|
||||
Resolvers []string
|
||||
|
||||
Logger *slog.Logger
|
||||
}
|
||||
|
||||
// entry is one address's cached answer.
|
||||
type entry struct {
|
||||
// names are the address's PTR names, nil when it has none.
|
||||
names []string
|
||||
expires time.Time
|
||||
}
|
||||
|
||||
// Cache answers PTR lookups from memory, and asks the resolvers for an address
|
||||
// it holds no current answer for. Start loads it from cache.db.
|
||||
type Cache struct {
|
||||
db *sql.DB
|
||||
resolvers []string
|
||||
timeLimit time.Duration
|
||||
logger *slog.Logger
|
||||
client *http.Client
|
||||
now func() time.Time
|
||||
|
||||
// entries holds every answer by address. pending holds those that
|
||||
// cache.db does not have yet.
|
||||
mu sync.RWMutex
|
||||
entries map[string]entry
|
||||
pending map[string]entry
|
||||
|
||||
// stop ends the goroutine Start runs, and done is closed when it returns.
|
||||
stop context.CancelFunc
|
||||
done chan struct{}
|
||||
}
|
||||
|
||||
// New returns an empty Cache. Start loads it.
|
||||
func New(params Params) *Cache {
|
||||
return &Cache{
|
||||
db: params.DB,
|
||||
resolvers: params.Resolvers,
|
||||
timeLimit: LookupTimeout,
|
||||
logger: params.Logger.With("component", "ptr"),
|
||||
client: &http.Client{},
|
||||
now: time.Now,
|
||||
entries: make(map[string]entry),
|
||||
pending: make(map[string]entry),
|
||||
done: make(chan struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
// LookupAddr returns the PTR names of addr, fully qualified, or none when it
|
||||
// has none. A reserved address has none, and is not sent to the resolvers. A
|
||||
// lookup takes at most LookupTimeout. A cached answer makes no outside request.
|
||||
// It returns an error when no resolver answered, and nothing is cached then.
|
||||
func (c *Cache) LookupAddr(ctx context.Context, addr string) ([]string, error) {
|
||||
parsed, err := netip.ParseAddr(addr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parsing the address: %w", err)
|
||||
}
|
||||
|
||||
parsed = parsed.Unmap()
|
||||
|
||||
_, reserved := bogon.Prefix(netip.PrefixFrom(parsed, parsed.BitLen()))
|
||||
if reserved {
|
||||
return []string{}, nil
|
||||
}
|
||||
|
||||
address := parsed.String()
|
||||
|
||||
cached, ok := c.current(address)
|
||||
if ok {
|
||||
return slices.Clone(cached.names), nil
|
||||
}
|
||||
|
||||
reverseName, err := dns.ReverseAddr(address)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("building the reverse name: %w", err)
|
||||
}
|
||||
|
||||
result, err := c.lookup(ctx, reverseName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if result.ttl > 0 {
|
||||
c.store(address, entry{names: result.names, expires: c.now().Add(result.ttl)})
|
||||
}
|
||||
|
||||
return slices.Clone(result.names), nil
|
||||
}
|
||||
|
||||
// Start loads the answers that have not expired from cache.db. It then starts
|
||||
// the goroutine that writes new answers there and deletes expired ones. The
|
||||
// goroutine runs until Stop: the end of ctx does not stop it.
|
||||
func (c *Cache) Start(ctx context.Context) error {
|
||||
err := c.load(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
c.logger.Info("PTR cache loaded", "entries", len(c.entries))
|
||||
|
||||
ctx, c.stop = context.WithCancel(context.WithoutCancel(ctx))
|
||||
|
||||
go c.run(ctx)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Stop ends the goroutine, which first writes the answers cache.db does not
|
||||
// have yet, and waits until it has returned or ctx ends.
|
||||
func (c *Cache) Stop(ctx context.Context) {
|
||||
c.stop()
|
||||
|
||||
select {
|
||||
case <-c.done:
|
||||
c.logger.Info("PTR cache stopped")
|
||||
case <-ctx.Done():
|
||||
c.logger.Warn("Stop deadline passed before the PTR cache was written")
|
||||
}
|
||||
}
|
||||
|
||||
// current returns the answer held for address, unless it has expired.
|
||||
func (c *Cache) current(address string) (entry, bool) {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
|
||||
found, ok := c.entries[address]
|
||||
if !ok || !c.now().Before(found.expires) {
|
||||
return entry{}, false
|
||||
}
|
||||
|
||||
return found, true
|
||||
}
|
||||
|
||||
// store keeps an answer in memory and queues it for cache.db.
|
||||
func (c *Cache) store(address string, answer entry) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
c.entries[address] = answer
|
||||
c.pending[address] = answer
|
||||
}
|
||||
|
||||
// run writes new answers and deletes expired ones until ctx ends.
|
||||
func (c *Cache) run(ctx context.Context) {
|
||||
defer close(c.done)
|
||||
|
||||
flushes := time.NewTicker(flushInterval)
|
||||
defer flushes.Stop()
|
||||
|
||||
sweeps := time.NewTicker(sweepInterval)
|
||||
defer sweeps.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
c.flush(context.WithoutCancel(ctx))
|
||||
|
||||
return
|
||||
case <-flushes.C:
|
||||
c.flush(ctx)
|
||||
case <-sweeps.C:
|
||||
c.sweep(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// load reads the rows that have not expired into memory.
|
||||
func (c *Cache) load(ctx context.Context) error {
|
||||
rows, err := c.db.QueryContext(ctx,
|
||||
"SELECT address, names, expires_at FROM hostnames WHERE expires_at > ?",
|
||||
c.now().Unix())
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading the PTR cache: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
address, names string
|
||||
expires int64
|
||||
)
|
||||
|
||||
err = rows.Scan(&address, &names, &expires)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading a PTR cache row: %w", err)
|
||||
}
|
||||
|
||||
c.entries[address] = entry{
|
||||
names: strings.Fields(names),
|
||||
expires: time.Unix(expires, 0).UTC(),
|
||||
}
|
||||
}
|
||||
|
||||
err = rows.Err()
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading the PTR cache: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// flush writes the pending answers to cache.db. Those it cannot write stay
|
||||
// pending, unless a newer answer for the address has arrived meanwhile.
|
||||
func (c *Cache) flush(ctx context.Context) {
|
||||
c.mu.Lock()
|
||||
batch := c.pending
|
||||
c.pending = make(map[string]entry)
|
||||
c.mu.Unlock()
|
||||
|
||||
if len(batch) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
err := c.writeRows(ctx, batch)
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
|
||||
c.logger.Error("Failed to write the PTR cache", "error", err)
|
||||
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
for address, answer := range batch {
|
||||
_, newer := c.pending[address]
|
||||
if !newer {
|
||||
c.pending[address] = answer
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// writeRows writes batch to cache.db in one transaction.
|
||||
func (c *Cache) writeRows(ctx context.Context, batch map[string]entry) error {
|
||||
tx, err := c.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
for address, answer := range batch {
|
||||
_, err = tx.ExecContext(ctx,
|
||||
"INSERT OR REPLACE INTO hostnames (address, names, expires_at) "+
|
||||
"VALUES (?, ?, ?)",
|
||||
address,
|
||||
strings.Join(answer.names, nameSeparator),
|
||||
answer.expires.Unix())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// sweep deletes the expired answers from memory and cache.db.
|
||||
func (c *Cache) sweep(ctx context.Context) {
|
||||
now := c.now()
|
||||
expired := func(_ string, answer entry) bool {
|
||||
return !now.Before(answer.expires)
|
||||
}
|
||||
|
||||
c.mu.Lock()
|
||||
maps.DeleteFunc(c.entries, expired)
|
||||
maps.DeleteFunc(c.pending, expired)
|
||||
c.mu.Unlock()
|
||||
|
||||
_, err := c.db.ExecContext(ctx,
|
||||
"DELETE FROM hostnames WHERE expires_at <= ?", now.Unix())
|
||||
if err != nil {
|
||||
c.logger.Error("Failed to delete expired PTR cache rows", "error", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,882 @@
|
||||
package ptr
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strconv"
|
||||
"sync"
|
||||
"testing"
|
||||
"testing/synctest"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/cachedb"
|
||||
"github.com/miekg/dns"
|
||||
)
|
||||
|
||||
// These addresses are public, because a reserved address is not sent to the
|
||||
// resolvers. That rules out the documentation ranges.
|
||||
const (
|
||||
testAddress = "8.8.4.4"
|
||||
testReverse = "4.4.8.8.in-addr.arpa."
|
||||
mappedAddress = "::ffff:" + testAddress
|
||||
otherAddress = "8.8.8.8"
|
||||
testV6Address = "2606:4700:4700::1111"
|
||||
testV6Reverse = "1.1.1.1.0.0.0.0.0.0.0.0.0.0.0.0." +
|
||||
"0.0.0.0.0.0.7.4.0.0.7.4.6.0.6.2.ip6.arpa."
|
||||
)
|
||||
|
||||
const (
|
||||
testName = "host.example.com."
|
||||
otherName = "other.example.com."
|
||||
|
||||
// testTimeLimit replaces LookupTimeout in the tests that use a real
|
||||
// server. A stall on a busy host cannot use it up, and no test waits for
|
||||
// it.
|
||||
testTimeLimit = 2 * time.Minute
|
||||
)
|
||||
|
||||
// classlessOwner is the name that holds the PTR records of a CNAME chain.
|
||||
const classlessOwner = "4.0-25.4.8.8.in-addr.arpa."
|
||||
|
||||
// fixedTime returns the time a test's clock starts at.
|
||||
func fixedTime() time.Time {
|
||||
return time.Date(2026, time.January, 2, 3, 4, 5, 0, time.UTC)
|
||||
}
|
||||
|
||||
// fakeResolver is a DNS-over-HTTPS server. It answers each query with reply's
|
||||
// result, and counts its requests.
|
||||
type fakeResolver struct {
|
||||
server *httptest.Server
|
||||
|
||||
mu sync.Mutex
|
||||
requests int
|
||||
reply func(http.ResponseWriter, *http.Request, *dns.Msg)
|
||||
}
|
||||
|
||||
// newFakeResolver starts a fake resolver that answers with reply.
|
||||
func newFakeResolver(
|
||||
t *testing.T,
|
||||
reply func(http.ResponseWriter, *http.Request, *dns.Msg),
|
||||
) *fakeResolver {
|
||||
t.Helper()
|
||||
|
||||
resolver := &fakeResolver{reply: reply}
|
||||
resolver.server = httptest.NewServer(http.HandlerFunc(resolver.serve))
|
||||
t.Cleanup(resolver.server.Close)
|
||||
|
||||
return resolver
|
||||
}
|
||||
|
||||
// serve checks that the request is RFC 8484 wire format and passes the
|
||||
// question on to the current reply.
|
||||
func (f *fakeResolver) serve(w http.ResponseWriter, r *http.Request) {
|
||||
f.mu.Lock()
|
||||
f.requests++
|
||||
reply := f.reply
|
||||
f.mu.Unlock()
|
||||
|
||||
body, err := io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
var query dns.Msg
|
||||
|
||||
err = query.Unpack(body)
|
||||
if err != nil || r.Header.Get("Content-Type") != dnsMessageType ||
|
||||
r.Header.Get("Accept") != dnsMessageType || r.Method != http.MethodPost {
|
||||
http.Error(w, "not an RFC 8484 request", http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
reply(w, r, &query)
|
||||
}
|
||||
|
||||
func (f *fakeResolver) setReply(
|
||||
reply func(http.ResponseWriter, *http.Request, *dns.Msg),
|
||||
) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
|
||||
f.reply = reply
|
||||
}
|
||||
|
||||
func (f *fakeResolver) requestCount() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
|
||||
return f.requests
|
||||
}
|
||||
|
||||
// writeReply packs reply and writes it as the answer to query.
|
||||
func writeReply(w http.ResponseWriter, query, reply *dns.Msg) {
|
||||
reply.SetReply(query)
|
||||
|
||||
wire, err := reply.Pack()
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", dnsMessageType)
|
||||
_, _ = w.Write(wire)
|
||||
}
|
||||
|
||||
// ptrReply answers with one PTR record for each name, each with the TTL ttl.
|
||||
func ptrReply(
|
||||
ttl uint32, names ...string,
|
||||
) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
||||
return func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
||||
reply := new(dns.Msg)
|
||||
|
||||
for _, name := range names {
|
||||
reply.Answer = append(reply.Answer, &dns.PTR{
|
||||
Hdr: dns.RR_Header{
|
||||
Name: query.Question[0].Name, Rrtype: dns.TypePTR,
|
||||
Class: dns.ClassINET, Ttl: ttl,
|
||||
},
|
||||
Ptr: name,
|
||||
})
|
||||
}
|
||||
|
||||
writeReply(w, query, reply)
|
||||
}
|
||||
}
|
||||
|
||||
// rcodeReply answers with the response code and nothing else.
|
||||
func rcodeReply(rcode int) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
||||
return func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
||||
reply := new(dns.Msg)
|
||||
reply.SetRcode(query, rcode)
|
||||
|
||||
wire, err := reply.Pack()
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
_, _ = w.Write(wire)
|
||||
}
|
||||
}
|
||||
|
||||
// soaReply answers with the response code and a SOA record with these TTLs.
|
||||
func soaReply(
|
||||
rcode int, soaTTL, minimum uint32,
|
||||
) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
||||
return func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
||||
reply := new(dns.Msg)
|
||||
reply.SetRcode(query, rcode)
|
||||
reply.Ns = []dns.RR{&dns.SOA{
|
||||
Hdr: dns.RR_Header{
|
||||
Name: "arpa.", Rrtype: dns.TypeSOA, Class: dns.ClassINET, Ttl: soaTTL,
|
||||
},
|
||||
Ns: "ns.example.com.", Mbox: "hostmaster.example.com.", Minttl: minimum,
|
||||
}}
|
||||
|
||||
wire, err := reply.Pack()
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
_, _ = w.Write(wire)
|
||||
}
|
||||
}
|
||||
|
||||
// hangingReply never answers: it waits until the client gives up.
|
||||
func hangingReply(_ http.ResponseWriter, r *http.Request, _ *dns.Msg) {
|
||||
<-r.Context().Done()
|
||||
}
|
||||
|
||||
// statusReply answers with an HTTP status.
|
||||
func statusReply(status int) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
||||
return func(w http.ResponseWriter, _ *http.Request, _ *dns.Msg) {
|
||||
w.WriteHeader(status)
|
||||
}
|
||||
}
|
||||
|
||||
// openTestDB opens a cache.db in dir.
|
||||
func openTestDB(t *testing.T, dir string) *sql.DB {
|
||||
t.Helper()
|
||||
|
||||
db, err := cachedb.Open(t.Context(), dir)
|
||||
if err != nil {
|
||||
t.Fatalf("opening the cache database: %v", err)
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
return db
|
||||
}
|
||||
|
||||
// testClock is a clock a test moves by hand.
|
||||
type testClock struct {
|
||||
mu sync.Mutex
|
||||
now time.Time
|
||||
}
|
||||
|
||||
func (c *testClock) time() time.Time {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
return c.now
|
||||
}
|
||||
|
||||
func (c *testClock) advance(d time.Duration) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
c.now = c.now.Add(d)
|
||||
}
|
||||
|
||||
// newTestCache returns a Cache that asks the resolvers in order, reads the
|
||||
// time from a clock the test holds, has testTimeLimit, and is not started.
|
||||
func newTestCache(
|
||||
t *testing.T, db *sql.DB, resolvers ...*fakeResolver,
|
||||
) (*Cache, *testClock) {
|
||||
t.Helper()
|
||||
|
||||
urls := make([]string, 0, len(resolvers))
|
||||
for _, resolver := range resolvers {
|
||||
urls = append(urls, resolver.server.URL)
|
||||
}
|
||||
|
||||
cache := New(Params{DB: db, Resolvers: urls, Logger: slog.Default()})
|
||||
clock := &testClock{now: fixedTime()}
|
||||
cache.now = clock.time
|
||||
cache.timeLimit = testTimeLimit
|
||||
|
||||
return cache, clock
|
||||
}
|
||||
|
||||
// handlerTransport hands each request to the fake resolver for its host, with
|
||||
// no network between them. A test in a synctest bubble needs this: the bubble's
|
||||
// clock stands still while one of its goroutines waits on a real connection. As
|
||||
// over a network, a request whose time is up is not sent.
|
||||
type handlerTransport map[string]*fakeResolver
|
||||
|
||||
func (h handlerTransport) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
recorder := httptest.NewRecorder()
|
||||
|
||||
if r.Context().Err() == nil {
|
||||
h[r.URL.Host].serve(recorder, r)
|
||||
}
|
||||
|
||||
err := r.Context().Err()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return recorder.Result(), nil
|
||||
}
|
||||
|
||||
// newBubbleCache returns a Cache with the real time limit, for a test in a
|
||||
// synctest bubble. It has one fake resolver for each reply, in order, reached
|
||||
// through a handlerTransport.
|
||||
func newBubbleCache(
|
||||
replies ...func(http.ResponseWriter, *http.Request, *dns.Msg),
|
||||
) (*Cache, []*fakeResolver) {
|
||||
transport := make(handlerTransport, len(replies))
|
||||
resolvers := make([]*fakeResolver, 0, len(replies))
|
||||
urls := make([]string, 0, len(replies))
|
||||
|
||||
for i, reply := range replies {
|
||||
host := "resolver" + strconv.Itoa(i) + ".example.com"
|
||||
resolver := &fakeResolver{reply: reply}
|
||||
|
||||
transport[host] = resolver
|
||||
resolvers = append(resolvers, resolver)
|
||||
urls = append(urls, "https://"+host+"/dns-query")
|
||||
}
|
||||
|
||||
cache := New(Params{Resolvers: urls, Logger: slog.Default()})
|
||||
cache.client = &http.Client{Transport: transport}
|
||||
|
||||
return cache, resolvers
|
||||
}
|
||||
|
||||
// lookupOK looks up address and fails the test on an error.
|
||||
func lookupOK(t *testing.T, cache *Cache, address string) []string {
|
||||
t.Helper()
|
||||
|
||||
names, err := cache.LookupAddr(t.Context(), address)
|
||||
if err != nil {
|
||||
t.Fatalf("LookupAddr(%s): %v", address, err)
|
||||
}
|
||||
|
||||
return names
|
||||
}
|
||||
|
||||
func countRows(t *testing.T, db *sql.DB) int {
|
||||
t.Helper()
|
||||
|
||||
var rows int
|
||||
|
||||
err := db.QueryRowContext(t.Context(),
|
||||
"SELECT COUNT(*) FROM hostnames").Scan(&rows)
|
||||
if err != nil {
|
||||
t.Fatalf("counting rows: %v", err)
|
||||
}
|
||||
|
||||
return rows
|
||||
}
|
||||
|
||||
func TestLookupAsksForTheReverseName(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
address string
|
||||
want string
|
||||
}{
|
||||
{testAddress, testReverse},
|
||||
{mappedAddress, testReverse},
|
||||
{testV6Address, testV6Reverse},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
var asked string
|
||||
|
||||
resolver := newFakeResolver(t,
|
||||
func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
||||
asked = query.Question[0].Name
|
||||
if query.Question[0].Qtype != dns.TypePTR {
|
||||
t.Errorf("query type %d, want PTR", query.Question[0].Qtype)
|
||||
}
|
||||
|
||||
ptrReply(300, testName)(w, r, query)
|
||||
})
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
names := lookupOK(t, cache, tt.address)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("%s: names %v, want %s", tt.address, names, testName)
|
||||
}
|
||||
|
||||
if asked != tt.want {
|
||||
t.Errorf("%s: asked for %s, want %s", tt.address, asked, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestOnlyPublicAddressesAreSentToTheResolvers gives every resolver an answer
|
||||
// to send, which a reserved address must not get.
|
||||
func TestOnlyPublicAddressesAreSentToTheResolvers(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
address string
|
||||
want []string
|
||||
requests int
|
||||
}{
|
||||
{"private IPv4", "10.0.0.1", []string{}, 0},
|
||||
{"documentation IPv4", "192.0.2.1", []string{}, 0},
|
||||
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", []string{}, 0},
|
||||
{"unique-local IPv6", "fd00::1", []string{}, 0},
|
||||
{"documentation IPv6", "2001:db8::1", []string{}, 0},
|
||||
{"public IPv4", testAddress, []string{testName}, 1},
|
||||
{"public IPv4 in IPv6 form", mappedAddress, []string{testName}, 1},
|
||||
{"public IPv6", testV6Address, []string{testName}, 1},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
names := lookupOK(t, cache, tt.address)
|
||||
if names == nil || !slices.Equal(names, tt.want) {
|
||||
t.Errorf("%s: names %#v, want %#v", tt.name, names, tt.want)
|
||||
}
|
||||
|
||||
if got := resolver.requestCount(); got != tt.requests {
|
||||
t.Errorf("%s: %d requests, want %d", tt.name, got, tt.requests)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolversAreTriedInOrderUntilOneAnswers has the first two resolvers fail,
|
||||
// in different ways, and the third answer. The fourth is not asked.
|
||||
func TestResolversAreTriedInOrderUntilOneAnswers(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
order []int
|
||||
)
|
||||
|
||||
asked := func(
|
||||
index int, reply func(http.ResponseWriter, *http.Request, *dns.Msg),
|
||||
) func(http.ResponseWriter, *http.Request, *dns.Msg) {
|
||||
return func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
||||
mu.Lock()
|
||||
|
||||
order = append(order, index)
|
||||
|
||||
mu.Unlock()
|
||||
|
||||
reply(w, r, query)
|
||||
}
|
||||
}
|
||||
|
||||
resolvers := []*fakeResolver{
|
||||
newFakeResolver(t, asked(0, statusReply(http.StatusServiceUnavailable))),
|
||||
newFakeResolver(t, asked(1, rcodeReply(dns.RcodeServerFailure))),
|
||||
newFakeResolver(t, asked(2, ptrReply(300, testName))),
|
||||
newFakeResolver(t, asked(3, ptrReply(300, otherName))),
|
||||
}
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolvers...)
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("names %v, want %s", names, testName)
|
||||
}
|
||||
|
||||
if !slices.Equal(order, []int{0, 1, 2}) {
|
||||
t.Errorf("resolvers asked %v, want [0 1 2]", order)
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolverThatNeverAnswersLeavesTimeForTheNext would fail if the first
|
||||
// resolver could take the whole time limit. The first gets half of it, the
|
||||
// share of one resolver in two. The test runs in a synctest bubble, whose clock
|
||||
// is fake, so a stall on the host uses none of the limit.
|
||||
func TestResolverThatNeverAnswersLeavesTimeForTheNext(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
cache, _ := newBubbleCache(hangingReply, ptrReply(300, testName))
|
||||
|
||||
start := time.Now()
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
elapsed := time.Since(start)
|
||||
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("names %v, want %s", names, testName)
|
||||
}
|
||||
|
||||
if elapsed != time.Second {
|
||||
t.Errorf("took %s, want 1s", elapsed)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestLookupStopsAtTheTimeLimit has no resolver answer. The lookup fails when
|
||||
// 2 seconds have passed, having asked every resolver. The test runs in a
|
||||
// synctest bubble, whose clock is fake, so a stall on the host uses none of the
|
||||
// limit.
|
||||
func TestLookupStopsAtTheTimeLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
synctest.Test(t, func(t *testing.T) {
|
||||
cache, resolvers := newBubbleCache(
|
||||
hangingReply, hangingReply, hangingReply, hangingReply)
|
||||
|
||||
start := time.Now()
|
||||
_, err := cache.LookupAddr(t.Context(), testAddress)
|
||||
elapsed := time.Since(start)
|
||||
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Errorf("error %v, want %v", err, context.DeadlineExceeded)
|
||||
}
|
||||
|
||||
if elapsed != 2*time.Second {
|
||||
t.Errorf("took %s, want 2s", elapsed)
|
||||
}
|
||||
|
||||
for i, resolver := range resolvers {
|
||||
if resolver.requestCount() != 1 {
|
||||
t.Errorf("resolver %d asked %d times, want once",
|
||||
i, resolver.requestCount())
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestAnswerIsServedFromMemoryUntilItExpires makes a second lookup within the
|
||||
// TTL, which asks nobody, and a third after it, which asks again.
|
||||
func TestAnswerIsServedFromMemoryUntilItExpires(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, clock := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("cached names %v, want %s", names, testName)
|
||||
}
|
||||
|
||||
if got := resolver.requestCount(); got != 1 {
|
||||
t.Errorf("%d requests after the second lookup, want 1", got)
|
||||
}
|
||||
|
||||
clock.advance(300 * time.Second)
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
if got := resolver.requestCount(); got != 2 {
|
||||
t.Errorf("%d requests after the TTL passed, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAnAddressHasOneCacheEntryHoweverItIsWritten looks up an IPv4 address in
|
||||
// its IPv4-mapped IPv6 form, then in its usual form.
|
||||
func TestAnAddressHasOneCacheEntryHoweverItIsWritten(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
lookupOK(t, cache, mappedAddress)
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
if got := resolver.requestCount(); got != 1 {
|
||||
t.Errorf("%d requests, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCallerCannotChangeACachedAnswer changes the list each lookup returned, as
|
||||
// the server does when it removes the trailing dots. The second lookup is the
|
||||
// first answered from the cache. The third shows that the cached list did not
|
||||
// change with the list the second returned.
|
||||
func TestCallerCannotChangeACachedAnswer(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
for range 3 {
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("names %v, want %s", names, testName)
|
||||
}
|
||||
|
||||
names[0] = "changed"
|
||||
}
|
||||
}
|
||||
|
||||
func TestTTLIsClamped(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
ttl uint32
|
||||
want time.Duration
|
||||
}{
|
||||
{"below the minimum", 1, time.Minute},
|
||||
{"zero", 0, time.Minute},
|
||||
{"at the minimum", 60, time.Minute},
|
||||
{"in range", 300, 300 * time.Second},
|
||||
{"at the maximum", 86400, 24 * time.Hour},
|
||||
{"above the maximum", 604800, 24 * time.Hour},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
resolver := newFakeResolver(t, ptrReply(tt.ttl, testName))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
got := cache.entries[testAddress].expires.Sub(fixedTime())
|
||||
if got != tt.want {
|
||||
t.Errorf("%s: kept for %s, want %s", tt.name, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSmallestTTLOfTheChainCounts answers with a CNAME record and two PTR
|
||||
// records, whose TTLs all differ.
|
||||
func TestSmallestTTLOfTheChainCounts(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
reply := func(w http.ResponseWriter, _ *http.Request, query *dns.Msg) {
|
||||
owner := query.Question[0].Name
|
||||
answer := new(dns.Msg)
|
||||
answer.Answer = []dns.RR{
|
||||
&dns.CNAME{
|
||||
Hdr: dns.RR_Header{
|
||||
Name: owner, Rrtype: dns.TypeCNAME, Class: dns.ClassINET, Ttl: 200,
|
||||
},
|
||||
Target: classlessOwner,
|
||||
},
|
||||
&dns.PTR{
|
||||
Hdr: dns.RR_Header{
|
||||
Name: classlessOwner, Rrtype: dns.TypePTR,
|
||||
Class: dns.ClassINET, Ttl: 900,
|
||||
},
|
||||
Ptr: testName,
|
||||
},
|
||||
&dns.PTR{
|
||||
Hdr: dns.RR_Header{
|
||||
Name: classlessOwner, Rrtype: dns.TypePTR,
|
||||
Class: dns.ClassINET, Ttl: 600,
|
||||
},
|
||||
Ptr: otherName,
|
||||
},
|
||||
}
|
||||
|
||||
writeReply(w, query, answer)
|
||||
}
|
||||
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), newFakeResolver(t, reply))
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName, otherName}) {
|
||||
t.Errorf("names %v, want both PTR names", names)
|
||||
}
|
||||
|
||||
got := cache.entries[testAddress].expires.Sub(fixedTime())
|
||||
if got != 200*time.Second {
|
||||
t.Errorf("kept for %s, want 200s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNegativeTTL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
rcode int
|
||||
soaTTL uint32
|
||||
minimum uint32
|
||||
want time.Duration
|
||||
}{
|
||||
{"name does not exist", dns.RcodeNameError, 3600, 300, 300 * time.Second},
|
||||
{"no PTR record", dns.RcodeSuccess, 3600, 300, 300 * time.Second},
|
||||
{"limited to an hour", dns.RcodeNameError, 86400, 7200, time.Hour},
|
||||
{"SOA's own TTL is smaller", dns.RcodeNameError, 100, 300, 100 * time.Second},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
resolver := newFakeResolver(t, soaReply(tt.rcode, tt.soaTTL, tt.minimum))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if len(names) != 0 {
|
||||
t.Errorf("%s: names %v, want none", tt.name, names)
|
||||
}
|
||||
|
||||
got := cache.entries[testAddress].expires.Sub(fixedTime())
|
||||
if got != tt.want {
|
||||
t.Errorf("%s: kept for %s, want %s", tt.name, got, tt.want)
|
||||
}
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
if count := resolver.requestCount(); count != 1 {
|
||||
t.Errorf("%s: %d requests, want 1", tt.name, count)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestAnswerWithoutNamesOrSOAIsNotCached has nothing to say how long the
|
||||
// answer may be kept.
|
||||
func TestAnswerWithoutNamesOrSOAIsNotCached(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t, rcodeReply(dns.RcodeNameError))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
if got := resolver.requestCount(); got != 2 {
|
||||
t.Errorf("%d requests, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailedQueryIsNotCached has the only resolver fail, and then recover.
|
||||
func TestFailedQueryIsNotCached(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t, rcodeReply(dns.RcodeServerFailure))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
_, err := cache.LookupAddr(t.Context(), testAddress)
|
||||
if !errors.Is(err, errBadResponse) {
|
||||
t.Fatalf("error %v, want %v", err, errBadResponse)
|
||||
}
|
||||
|
||||
if len(cache.entries)+len(cache.pending) != 0 {
|
||||
t.Error("a failed query left an entry in the cache")
|
||||
}
|
||||
|
||||
resolver.setReply(ptrReply(300, testName))
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("names %v, want %s", names, testName)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNamesThatAreNotHostNamesAreDropped answers with a name that holds markup
|
||||
// beside a good one, as a PTR record can.
|
||||
func TestNamesThatAreNotHostNamesAreDropped(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resolver := newFakeResolver(t,
|
||||
ptrReply(300, "bad<script>.example.com.", testName, "."))
|
||||
cache, _ := newTestCache(t, openTestDB(t, t.TempDir()), resolver)
|
||||
|
||||
names := lookupOK(t, cache, testAddress)
|
||||
if !slices.Equal(names, []string{testName}) {
|
||||
t.Errorf("names %v, want only %s", names, testName)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCacheSurvivesARestart stops a cache that holds a positive and a negative
|
||||
// answer, and starts a new one on the reopened file. It makes no request.
|
||||
func TestCacheSurvivesARestart(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
db := openTestDB(t, dir)
|
||||
|
||||
resolver := newFakeResolver(t,
|
||||
func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
||||
if query.Question[0].Name == testReverse {
|
||||
ptrReply(300, testName, otherName)(w, r, query)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
soaReply(dns.RcodeNameError, 3600, 300)(w, r, query)
|
||||
})
|
||||
|
||||
cache, _ := newTestCache(t, db, resolver)
|
||||
|
||||
err := cache.Start(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("Start: %v", err)
|
||||
}
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
lookupOK(t, cache, otherAddress)
|
||||
cache.Stop(t.Context())
|
||||
|
||||
err = db.Close()
|
||||
if err != nil {
|
||||
t.Fatalf("closing the database: %v", err)
|
||||
}
|
||||
|
||||
restarted, _ := newTestCache(t, openTestDB(t, dir), resolver)
|
||||
|
||||
err = restarted.Start(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("Start after the restart: %v", err)
|
||||
}
|
||||
|
||||
defer restarted.Stop(t.Context())
|
||||
|
||||
requestsBefore := resolver.requestCount()
|
||||
|
||||
names := lookupOK(t, restarted, testAddress)
|
||||
if !slices.Equal(names, []string{testName, otherName}) {
|
||||
t.Errorf("names after restart %v", names)
|
||||
}
|
||||
|
||||
if names := lookupOK(t, restarted, otherAddress); len(names) != 0 {
|
||||
t.Errorf("names of the address with none after restart: %v", names)
|
||||
}
|
||||
|
||||
if got := resolver.requestCount(); got != requestsBefore {
|
||||
t.Errorf("%d requests after the restart, want none",
|
||||
got-requestsBefore)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSweepDeletesExpiredRows also checks that a restart does not load a row
|
||||
// that has expired since it was written.
|
||||
func TestSweepDeletesExpiredRows(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := openTestDB(t, t.TempDir())
|
||||
resolver := newFakeResolver(t,
|
||||
func(w http.ResponseWriter, r *http.Request, query *dns.Msg) {
|
||||
ttl := uint32(300)
|
||||
if query.Question[0].Name == testReverse {
|
||||
ttl = 60
|
||||
}
|
||||
|
||||
ptrReply(ttl, testName)(w, r, query)
|
||||
})
|
||||
cache, clock := newTestCache(t, db, resolver)
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
lookupOK(t, cache, otherAddress)
|
||||
cache.flush(t.Context())
|
||||
|
||||
if got := countRows(t, db); got != 2 {
|
||||
t.Fatalf("%d rows before the sweep, want 2", got)
|
||||
}
|
||||
|
||||
clock.advance(90 * time.Second)
|
||||
|
||||
loaded, loadedClock := newTestCache(t, db, resolver)
|
||||
loadedClock.advance(90 * time.Second)
|
||||
|
||||
err := loaded.load(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("load: %v", err)
|
||||
}
|
||||
|
||||
if len(loaded.entries) != 1 {
|
||||
t.Errorf("loaded %d entries, want the 1 not expired", len(loaded.entries))
|
||||
}
|
||||
|
||||
cache.sweep(t.Context())
|
||||
|
||||
if got := countRows(t, db); got != 1 {
|
||||
t.Errorf("%d rows after the sweep, want 1", got)
|
||||
}
|
||||
|
||||
if len(cache.entries) != 1 {
|
||||
t.Errorf("%d entries in memory after the sweep, want 1", len(cache.entries))
|
||||
}
|
||||
}
|
||||
|
||||
// TestFlushKeepsAnswersItCouldNotWrite makes one write fail by hiding the
|
||||
// table. The next write, once the table is back, stores the answer.
|
||||
func TestFlushKeepsAnswersItCouldNotWrite(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := openTestDB(t, t.TempDir())
|
||||
resolver := newFakeResolver(t, ptrReply(300, testName))
|
||||
cache, _ := newTestCache(t, db, resolver)
|
||||
|
||||
lookupOK(t, cache, testAddress)
|
||||
|
||||
_, err := db.ExecContext(t.Context(), "ALTER TABLE hostnames RENAME TO hidden")
|
||||
if err != nil {
|
||||
t.Fatalf("renaming the table: %v", err)
|
||||
}
|
||||
|
||||
cache.flush(t.Context())
|
||||
|
||||
_, err = db.ExecContext(t.Context(), "ALTER TABLE hidden RENAME TO hostnames")
|
||||
if err != nil {
|
||||
t.Fatalf("renaming the table back: %v", err)
|
||||
}
|
||||
|
||||
if got := countRows(t, db); got != 0 {
|
||||
t.Fatalf("%d rows after the failed write, want 0", got)
|
||||
}
|
||||
|
||||
cache.flush(t.Context())
|
||||
|
||||
if got := countRows(t, db); got != 1 {
|
||||
t.Errorf("%d rows after the second write, want 1", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,200 @@
|
||||
package ptr
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/miekg/dns"
|
||||
)
|
||||
|
||||
const (
|
||||
// LookupTimeout limits one address's lookup, every resolver included.
|
||||
LookupTimeout = 2 * time.Second
|
||||
|
||||
// minTTL and maxTTL limit how long PTR names are kept.
|
||||
minTTL = time.Minute
|
||||
maxTTL = 24 * time.Hour
|
||||
|
||||
// maxNegativeTTL limits how long an address with no PTR names is
|
||||
// remembered as having none.
|
||||
maxNegativeTTL = time.Hour
|
||||
|
||||
// dnsMessageType is the content type of RFC 8484 requests and answers.
|
||||
dnsMessageType = "application/dns-message"
|
||||
|
||||
// maxMessageBytes is the largest possible DNS message.
|
||||
maxMessageBytes = 65535
|
||||
)
|
||||
|
||||
var (
|
||||
errBadStatus = errors.New("unexpected HTTP status")
|
||||
errBadResponse = errors.New("resolver reported an error")
|
||||
)
|
||||
|
||||
// Resolvers returns the URLs of the public DNS-over-HTTPS resolvers, in the
|
||||
// order they are tried.
|
||||
func Resolvers() []string {
|
||||
return []string{
|
||||
"https://dns.google/dns-query",
|
||||
"https://cloudflare-dns.com/dns-query",
|
||||
"https://unfiltered.adguard-dns.com/dns-query",
|
||||
"https://dns.quad9.net/dns-query",
|
||||
}
|
||||
}
|
||||
|
||||
// answer is a resolver's answer for one address.
|
||||
type answer struct {
|
||||
// names are the address's PTR names, nil when it has none.
|
||||
names []string
|
||||
|
||||
// ttl is how long the answer may be kept. Zero means it is not kept.
|
||||
ttl time.Duration
|
||||
}
|
||||
|
||||
// lookup asks the resolvers in turn for the PTR names of the reverse name
|
||||
// until one answers. It fails when none has answered within the time limit.
|
||||
func (c *Cache) lookup(ctx context.Context, reverseName string) (answer, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, c.timeLimit)
|
||||
defer cancel()
|
||||
|
||||
query := new(dns.Msg).SetQuestion(reverseName, dns.TypePTR)
|
||||
query.Id = 0 // RFC 8484 asks for 0, so that equal queries can be cached.
|
||||
|
||||
wire, err := query.Pack()
|
||||
if err != nil {
|
||||
return answer{}, fmt.Errorf("packing the query: %w", err)
|
||||
}
|
||||
|
||||
var failures []error
|
||||
|
||||
for i, resolver := range c.resolvers {
|
||||
// Each resolver gets an equal share of the time left, so that one
|
||||
// that never answers leaves time for the next.
|
||||
deadline, _ := ctx.Deadline()
|
||||
share := time.Until(deadline) / time.Duration(len(c.resolvers)-i)
|
||||
|
||||
result, err := c.ask(ctx, resolver, wire, share)
|
||||
if err == nil {
|
||||
return result, nil
|
||||
}
|
||||
|
||||
failures = append(failures, fmt.Errorf("%s: %w", resolver, err))
|
||||
}
|
||||
|
||||
return answer{}, errors.Join(failures...)
|
||||
}
|
||||
|
||||
// ask sends the query to one resolver and waits at most timeout for its answer.
|
||||
func (c *Cache) ask(
|
||||
ctx context.Context, resolver string, query []byte, timeout time.Duration,
|
||||
) (answer, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
request, err := http.NewRequestWithContext(
|
||||
ctx, http.MethodPost, resolver, bytes.NewReader(query))
|
||||
if err != nil {
|
||||
return answer{}, err
|
||||
}
|
||||
|
||||
request.Header.Set("Content-Type", dnsMessageType)
|
||||
request.Header.Set("Accept", dnsMessageType)
|
||||
|
||||
response, err := c.client.Do(request)
|
||||
if err != nil {
|
||||
return answer{}, err
|
||||
}
|
||||
|
||||
defer func() { _ = response.Body.Close() }()
|
||||
|
||||
if response.StatusCode != http.StatusOK {
|
||||
return answer{}, fmt.Errorf("%w: %d", errBadStatus, response.StatusCode)
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(io.LimitReader(response.Body, maxMessageBytes))
|
||||
if err != nil {
|
||||
return answer{}, err
|
||||
}
|
||||
|
||||
var reply dns.Msg
|
||||
|
||||
err = reply.Unpack(body)
|
||||
if err != nil {
|
||||
return answer{}, err
|
||||
}
|
||||
|
||||
return answerFrom(&reply)
|
||||
}
|
||||
|
||||
// answerFrom reads a resolver's reply. A name that does not exist and a name
|
||||
// with no PTR records are answers. Every other response code is a failure.
|
||||
func answerFrom(reply *dns.Msg) (answer, error) {
|
||||
if reply.Rcode != dns.RcodeSuccess && reply.Rcode != dns.RcodeNameError {
|
||||
return answer{}, fmt.Errorf("%w: %s",
|
||||
errBadResponse, dns.RcodeToString[reply.Rcode])
|
||||
}
|
||||
|
||||
var names []string
|
||||
|
||||
seconds := uint32(math.MaxUint32)
|
||||
|
||||
for _, record := range reply.Answer {
|
||||
// The chain to a PTR record can hold CNAME records, and the answer
|
||||
// is good only as long as every record in it.
|
||||
seconds = min(seconds, record.Header().Ttl)
|
||||
|
||||
ptr, ok := record.(*dns.PTR)
|
||||
if ok && isHostName(ptr.Ptr) {
|
||||
names = append(names, ptr.Ptr)
|
||||
}
|
||||
}
|
||||
|
||||
if len(names) == 0 {
|
||||
return answer{ttl: negativeTTL(reply.Ns)}, nil
|
||||
}
|
||||
|
||||
ttl := time.Duration(seconds) * time.Second
|
||||
|
||||
return answer{names: names, ttl: min(max(ttl, minTTL), maxTTL)}, nil
|
||||
}
|
||||
|
||||
// negativeTTL returns how long a reply with no PTR names may be kept: the
|
||||
// smaller of the SOA record's own TTL and its minimum field (RFC 2308), at most
|
||||
// maxNegativeTTL. Without a SOA record it is zero.
|
||||
func negativeTTL(authority []dns.RR) time.Duration {
|
||||
for _, record := range authority {
|
||||
soa, ok := record.(*dns.SOA)
|
||||
if ok {
|
||||
ttl := time.Duration(min(soa.Hdr.Ttl, soa.Minttl)) * time.Second
|
||||
|
||||
return min(ttl, maxNegativeTTL)
|
||||
}
|
||||
}
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
// isHostName reports whether name is a host name. A PTR record can hold any
|
||||
// bytes, and the names go into web pages, so anything else is dropped, as
|
||||
// net.Resolver.LookupAddr drops it. The root name is not a host name.
|
||||
func isHostName(name string) bool {
|
||||
_, ok := dns.IsDomainName(name)
|
||||
|
||||
return ok && name != "." && !strings.ContainsFunc(name, notInHostName)
|
||||
}
|
||||
|
||||
// notInHostName reports whether r cannot be part of a host name.
|
||||
func notInHostName(r rune) bool {
|
||||
return !strings.ContainsRune(hostNameCharacters, r)
|
||||
}
|
||||
|
||||
// hostNameCharacters are the characters of a host name, in presentation format.
|
||||
const hostNameCharacters = "abcdefghijklmnopqrstuvwxyz" +
|
||||
"ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_."
|
||||
+78
-17
@@ -9,17 +9,20 @@ package routewatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/cachedb"
|
||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||
"git.eeqj.de/sneak/routewatch/internal/ptr"
|
||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||
"git.eeqj.de/sneak/routewatch/internal/server"
|
||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||
@@ -61,12 +64,14 @@ type RouteWatch struct {
|
||||
peeringHandler *PeeringHandler
|
||||
asnFetcher *ASNFetcher
|
||||
dbMaintainer *DBMaintainer
|
||||
blocklists *blocklist.Blocklists
|
||||
ipLists *iplist.IPLists
|
||||
cacheDB *sql.DB
|
||||
ptrCache *ptr.Cache
|
||||
}
|
||||
|
||||
// blocklistsDirName is the directory in the state directory that holds the
|
||||
// blocklist copies.
|
||||
const blocklistsDirName = "blocklists"
|
||||
// ipListsDirName is the directory in the state directory that holds the IP
|
||||
// list copies.
|
||||
const ipListsDirName = "iplists"
|
||||
|
||||
// New creates a new RouteWatch instance
|
||||
func New(deps Dependencies) *RouteWatch {
|
||||
@@ -138,17 +143,22 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
||||
rw.asnFetcher.Start(ctx)
|
||||
rw.server.SetASNFetcher(rw.asnFetcher)
|
||||
|
||||
// Load the blocklist copies from disk and start downloading the lists.
|
||||
// The HTTP handlers read the server's blocklists without a lock, so they
|
||||
// Load the IP list copies from disk and start downloading the lists.
|
||||
// The HTTP handlers read the server's IP lists without a lock, so they
|
||||
// are set before the handlers run.
|
||||
blocklists := blocklist.New(blocklist.Params{
|
||||
Dir: filepath.Join(rw.config.StateDir, blocklistsDirName),
|
||||
Lists: blocklist.Lists(),
|
||||
ipLists := iplist.New(iplist.Params{
|
||||
Dir: filepath.Join(rw.config.StateDir, ipListsDirName),
|
||||
Lists: iplist.Lists(),
|
||||
Logger: rw.logger.Logger,
|
||||
})
|
||||
blocklists.Start(ctx)
|
||||
rw.blocklists = blocklists
|
||||
rw.server.SetBlocklists(blocklists)
|
||||
ipLists.Start(ctx)
|
||||
rw.ipLists = ipLists
|
||||
rw.server.SetIPLists(ipLists)
|
||||
|
||||
err = rw.startPTRCache(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Start HTTP server
|
||||
err = rw.server.Start(ctx)
|
||||
@@ -168,8 +178,8 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
|
||||
|
||||
// Shutdown stops all services in order: the streamer, the handlers, which
|
||||
// write what they hold, the WHOIS fetcher, the database maintainer, the
|
||||
// blocklist downloads and the HTTP server, and last the database, which it
|
||||
// checkpoints and closes. Each stage waits at most until ctx ends, so
|
||||
// IP list downloads, the HTTP server and the PTR cache, and last the database,
|
||||
// which it checkpoints and closes. Each stage waits at most until ctx ends, so
|
||||
// Shutdown returns soon after: a handler logs what it could not write by
|
||||
// then, and the HTTP server closes the connections of requests still
|
||||
// running.
|
||||
@@ -219,8 +229,8 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
||||
rw.dbMaintainer.Stop(ctx)
|
||||
}
|
||||
|
||||
if rw.blocklists != nil {
|
||||
rw.blocklists.Stop(ctx)
|
||||
if rw.ipLists != nil {
|
||||
rw.ipLists.Stop(ctx)
|
||||
}
|
||||
|
||||
err := rw.server.Stop(ctx)
|
||||
@@ -229,6 +239,8 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
||||
"error", err)
|
||||
}
|
||||
|
||||
rw.stopPTRCache(ctx)
|
||||
|
||||
rw.closeDatabase(ctx)
|
||||
|
||||
// Log final metrics
|
||||
@@ -242,6 +254,55 @@ func (rw *RouteWatch) Shutdown(ctx context.Context) {
|
||||
)
|
||||
}
|
||||
|
||||
// startPTRCache opens cache.db, loads the PTR cache from it and gives the
|
||||
// server the cache. The HTTP handlers read it without a lock, so it is set
|
||||
// before they run.
|
||||
func (rw *RouteWatch) startPTRCache(ctx context.Context) error {
|
||||
cacheDB, err := cachedb.Open(ctx, rw.config.StateDir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
ptrCache := ptr.New(ptr.Params{
|
||||
DB: cacheDB,
|
||||
Resolvers: ptr.Resolvers(),
|
||||
Logger: rw.logger.Logger,
|
||||
})
|
||||
|
||||
err = ptrCache.Start(ctx)
|
||||
if err != nil {
|
||||
_ = cacheDB.Close()
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
rw.cacheDB = cacheDB
|
||||
rw.ptrCache = ptrCache
|
||||
rw.server.SetPTRResolver(ptrCache)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// stopPTRCache stops the PTR cache, which writes the answers cache.db does not
|
||||
// have yet, and closes cache.db, unless ctx has ended and a write may still be
|
||||
// running.
|
||||
func (rw *RouteWatch) stopPTRCache(ctx context.Context) {
|
||||
if rw.ptrCache == nil {
|
||||
return
|
||||
}
|
||||
|
||||
rw.ptrCache.Stop(ctx)
|
||||
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
|
||||
err := rw.cacheDB.Close()
|
||||
if err != nil {
|
||||
rw.logger.Error("Failed to close the cache database", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// closeDatabase checkpoints and closes the database, unless ctx has ended: a
|
||||
// write may then still be running, and both would wait for it. SQLite rolls
|
||||
// back a write cut off by the exit when the database next opens.
|
||||
|
||||
@@ -418,7 +418,7 @@ func TestRouteWatchLiveFeed(t *testing.T) {
|
||||
// Create streamer
|
||||
s := streamer.New(logger, metricsTracker)
|
||||
|
||||
// The state directory holds the blocklist copies Run downloads.
|
||||
// The state directory holds the IP list copies Run downloads.
|
||||
cfg := &config.Config{
|
||||
StateDir: t.TempDir(),
|
||||
MaxRuntime: 5 * time.Second,
|
||||
|
||||
+84
-34
@@ -15,8 +15,9 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/bogon"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||
@@ -38,12 +39,9 @@ const (
|
||||
const (
|
||||
// maxLookupTargets is the most addresses and host names an IP lookup may
|
||||
// list, and the most addresses it may have once the names are resolved:
|
||||
// each address costs two database reads and a PTR lookup.
|
||||
// each address costs two database reads and a PTR lookup. Ten PTR lookups
|
||||
// that run to ptr.LookupTimeout fit in the request's 30-second limit.
|
||||
maxLookupTargets = 10
|
||||
|
||||
// ptrLookupTimeout is each PTR lookup's own time limit, so that ten
|
||||
// addresses whose PTR lookups never answer fit in the 30-second limit.
|
||||
ptrLookupTimeout = 2 * time.Second
|
||||
)
|
||||
|
||||
// The Cache-Control header of an IP lookup answer, such as
|
||||
@@ -765,9 +763,12 @@ type IPLookupResponse struct {
|
||||
Errors []string `json:"errors,omitempty"`
|
||||
}
|
||||
|
||||
// IPLookupResult is the answer for one address: its PTR names, its live route
|
||||
// and that route's AS, and the blocklists that list it. Every result has every
|
||||
// field. The fields of IPInfo are null when the address has no live route.
|
||||
// IPLookupResult is the answer for one address: its PTR names, its flags, its
|
||||
// live route and that route's AS, and the IP lists that contain it. Every
|
||||
// result has every field. The fields of IPInfo are null when the address has
|
||||
// no live route.
|
||||
//
|
||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
||||
type IPLookupResult struct {
|
||||
IPInfo
|
||||
|
||||
@@ -775,8 +776,22 @@ type IPLookupResult struct {
|
||||
// PTR is empty, never nil, when the address has no PTR names.
|
||||
PTR []string `json:"ptr"`
|
||||
|
||||
// Blocklists has one entry for each blocklist the address is on.
|
||||
Blocklists []blocklist.Listing `json:"blocklists"`
|
||||
Flags IPLookupFlags `json:"flags"`
|
||||
|
||||
// IPLists has one entry for each IP list the address is on.
|
||||
IPLists []iplist.Listing `json:"ip_lists"`
|
||||
}
|
||||
|
||||
// IPLookupFlags are the yes-or-no facts about an address. Every result has
|
||||
// them, whether or not the address has a live route.
|
||||
//
|
||||
//nolint:tagliatelle // snake_case keys, as the Go style guide asks
|
||||
type IPLookupFlags struct {
|
||||
// IsTor says that the address is on the Tor exit list.
|
||||
IsTor bool `json:"is_tor"`
|
||||
|
||||
// IsBogon says that the address is reserved, as package bogon tells.
|
||||
IsBogon bool `json:"is_bogon"`
|
||||
}
|
||||
|
||||
// handleIPInfo returns a handler that provides comprehensive IP information.
|
||||
@@ -852,12 +867,7 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
response.Results = append(response.Results, IPLookupResult{
|
||||
IP: ip,
|
||||
PTR: ptrs,
|
||||
IPInfo: ipInfo,
|
||||
Blocklists: s.blocklistListings(ip),
|
||||
})
|
||||
response.Results = append(response.Results, s.ipLookupResult(ip, ptrs, ipInfo))
|
||||
|
||||
if ipInfo.NeedsWHOISRefresh {
|
||||
refreshASNs[*ipInfo.ASN] = true
|
||||
@@ -958,23 +968,61 @@ func ipVersionOf(prefix netip.Prefix) int {
|
||||
return ipVersionV6
|
||||
}
|
||||
|
||||
// blocklistListings returns the blocklists that list ip. It returns an empty
|
||||
// list, never nil, so that an answer's blocklists is [] rather than null.
|
||||
func (s *Server) blocklistListings(ip string) []blocklist.Listing {
|
||||
// ipLookupResult returns the result for ip, adding its flags and IP lists to
|
||||
// its PTR names and route.
|
||||
func (s *Server) ipLookupResult(ip string, ptrs []string, info IPInfo) IPLookupResult {
|
||||
listings := s.ipListListings(ip)
|
||||
|
||||
return IPLookupResult{
|
||||
IP: ip,
|
||||
PTR: ptrs,
|
||||
IPInfo: info,
|
||||
Flags: ipLookupFlags(ip, listings),
|
||||
IPLists: listings,
|
||||
}
|
||||
}
|
||||
|
||||
// ipListListings returns the IP lists that contain ip. It returns an empty
|
||||
// list, never nil, so that an answer's ip_lists is [] rather than null.
|
||||
func (s *Server) ipListListings(ip string) []iplist.Listing {
|
||||
addr, err := netip.ParseAddr(ip)
|
||||
if s.blocklists == nil || err != nil {
|
||||
return []blocklist.Listing{}
|
||||
if s.ipLists == nil || err != nil {
|
||||
return []iplist.Listing{}
|
||||
}
|
||||
|
||||
return s.blocklists.Lookup(addr)
|
||||
return s.ipLists.Lookup(addr)
|
||||
}
|
||||
|
||||
// ipLookupFlags returns the flags of ip, whose IP list entries are listings.
|
||||
func ipLookupFlags(ip string, listings []iplist.Listing) IPLookupFlags {
|
||||
return IPLookupFlags{
|
||||
IsTor: slices.ContainsFunc(listings, func(l iplist.Listing) bool {
|
||||
return l.Name == iplist.TorExitsList
|
||||
}),
|
||||
IsBogon: isBogon(ip),
|
||||
}
|
||||
}
|
||||
|
||||
// isBogon says whether the address ip is reserved. An IPv4 address written in
|
||||
// IPv6 form counts as the IPv4 address it holds.
|
||||
func isBogon(ip string) bool {
|
||||
addr, err := netip.ParseAddr(ip)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
addr = addr.Unmap()
|
||||
_, reserved := bogon.Prefix(netip.PrefixFrom(addr, addr.BitLen()))
|
||||
|
||||
return reserved
|
||||
}
|
||||
|
||||
// setIPLookupCacheControl lets caches keep an IP lookup answer for the
|
||||
// shortest blocklist interval, the soonest a blocklist copy can change. The
|
||||
// shortest IP list interval, the soonest an IP list copy can change. The
|
||||
// answer about the caller's own address is private, so that a shared cache
|
||||
// does not give it to another caller. Without blocklists it sets nothing.
|
||||
// does not give it to another caller. Without IP lists it sets nothing.
|
||||
func (s *Server) setIPLookupCacheControl(w http.ResponseWriter, ownAddress bool) {
|
||||
if s.blocklists == nil {
|
||||
if s.ipLists == nil {
|
||||
return
|
||||
}
|
||||
|
||||
@@ -983,7 +1031,7 @@ func (s *Server) setIPLookupCacheControl(w http.ResponseWriter, ownAddress bool)
|
||||
visibility = cacheControlPrivate
|
||||
}
|
||||
|
||||
maxAgeSeconds := int(s.blocklists.ShortestInterval().Seconds())
|
||||
maxAgeSeconds := int(s.ipLists.ShortestInterval().Seconds())
|
||||
w.Header().Set(cacheControlHeader,
|
||||
visibility+cacheControlMaxAge+strconv.Itoa(maxAgeSeconds))
|
||||
}
|
||||
@@ -1143,15 +1191,17 @@ func publicAddresses(addrs []string) []string {
|
||||
}
|
||||
|
||||
// lookupPTRNames returns the PTR records of ip without their trailing dots. It
|
||||
// returns an empty list, never nil, when there are none, when the lookup fails
|
||||
// and when it runs past ptrLookupTimeout, so that an answer's ptr is [] rather
|
||||
// than null.
|
||||
// returns an empty list, never nil, when there are none and when the lookup
|
||||
// fails, so that an answer's ptr is [] rather than null.
|
||||
func (s *Server) lookupPTRNames(ctx context.Context, ip string) []string {
|
||||
ctx, cancel := context.WithTimeout(ctx, ptrLookupTimeout)
|
||||
defer cancel()
|
||||
ptrs, err := s.ptrResolver.LookupAddr(ctx, ip)
|
||||
if err != nil {
|
||||
s.logger.Warn("PTR lookup failed", "ip", ip, "error", err)
|
||||
|
||||
ptrs, err := s.resolver.LookupAddr(ctx, ip)
|
||||
if err != nil || ptrs == nil {
|
||||
return []string{}
|
||||
}
|
||||
|
||||
if ptrs == nil {
|
||||
return []string{}
|
||||
}
|
||||
|
||||
|
||||
@@ -19,11 +19,12 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
||||
"git.eeqj.de/sneak/routewatch/internal/ptr"
|
||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||
@@ -523,8 +524,11 @@ func newServerWithTable(t *testing.T, table *routetable.Table) *Server {
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
return New(db, table, streamer.New(logger.New(), metrics.New()), logger.New(),
|
||||
s := New(db, table, streamer.New(logger.New(), metrics.New()), logger.New(),
|
||||
&config.Config{})
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
// TestPrefixDetailHandlersParsePrefix checks that both prefix pages find a
|
||||
@@ -1213,7 +1217,7 @@ func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) {
|
||||
|
||||
s := New(failingIPLookupDB{}, newTestTable(),
|
||||
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
tests := []struct {
|
||||
path string
|
||||
@@ -1255,8 +1259,8 @@ var errNoSuchHost = errors.New("no such host")
|
||||
|
||||
// fakeResolver answers a host name lookup with the addresses hosts lists for
|
||||
// the name, and a PTR lookup with no names or, when ptrNeverAnswers is set,
|
||||
// not at all: it waits until the lookup's context ends. hostLookups counts the
|
||||
// host name lookups.
|
||||
// with a failure after ptr.LookupTimeout, as the real one does when no
|
||||
// resolver answers. hostLookups counts the host name lookups.
|
||||
type fakeResolver struct {
|
||||
hosts map[string][]string
|
||||
ptrNeverAnswers bool
|
||||
@@ -1277,6 +1281,9 @@ func (r *fakeResolver) LookupHost(_ context.Context, host string) ([]string, err
|
||||
|
||||
func (r *fakeResolver) LookupAddr(ctx context.Context, _ string) ([]string, error) {
|
||||
if r.ptrNeverAnswers {
|
||||
ctx, cancel := context.WithTimeout(ctx, ptr.LookupTimeout)
|
||||
defer cancel()
|
||||
|
||||
<-ctx.Done()
|
||||
|
||||
return nil, ctx.Err()
|
||||
@@ -1285,6 +1292,12 @@ func (r *fakeResolver) LookupAddr(ctx context.Context, _ string) ([]string, erro
|
||||
return nil, errNoSuchHost
|
||||
}
|
||||
|
||||
// useFakeResolver makes r answer the server's host name and PTR lookups.
|
||||
func useFakeResolver(s *Server, r *fakeResolver) {
|
||||
s.resolver = r
|
||||
s.ptrResolver = r
|
||||
}
|
||||
|
||||
// countingIPLookupDB embeds database.Store (left nil) and answers the IP
|
||||
// lookup's reads with nothing stored. It counts the lookups, as the reads of a
|
||||
// prefix's first-seen time. Every other Store method is unused and would panic
|
||||
@@ -1353,7 +1366,7 @@ func TestIPLookupRefusesTooManyAddresses(t *testing.T) {
|
||||
s := New(db, newTestTable(), streamer.New(logger.New(), metrics.New()),
|
||||
logger.New(), &config.Config{})
|
||||
resolver := &fakeResolver{hosts: hosts}
|
||||
s.resolver = resolver
|
||||
useFakeResolver(s, resolver)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
@@ -1395,10 +1408,10 @@ func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{ptrNeverAnswers: true}
|
||||
useFakeResolver(s, &fakeResolver{ptrNeverAnswers: true})
|
||||
|
||||
// The request's time limit passes after the PTR lookup's.
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 2*ptrLookupTimeout)
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 2*ptr.LookupTimeout)
|
||||
defer cancel()
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
@@ -1428,50 +1441,112 @@ func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
|
||||
// newServerWithTestRoutes stores a route for.
|
||||
const addrInTestPrefixV6 = "2001:db8::1"
|
||||
|
||||
// testBlocklistInterval is fakeBlocklists' shortest interval.
|
||||
const testBlocklistInterval = 15 * time.Minute
|
||||
// testIPListInterval is fakeIPLists' shortest interval.
|
||||
const testIPListInterval = 15 * time.Minute
|
||||
|
||||
// ownAddressPath is the IP lookup of the caller's own address.
|
||||
const ownAddressPath = "/ip/"
|
||||
|
||||
// listedUnroutedAddr is an address that fakeBlocklists lists, outside the
|
||||
// routes newServerWithTestRoutes stores.
|
||||
// listedUnroutedAddr is an address that fakeIPLists lists, outside the routes
|
||||
// newServerWithTestRoutes stores.
|
||||
const listedUnroutedAddr = "203.0.113.1"
|
||||
|
||||
// fakeBlocklists has one list, which lists addrInTestPrefixV4 and
|
||||
// listedUnroutedAddr and no other address.
|
||||
type fakeBlocklists struct{}
|
||||
// torExitAddr is the address that fakeIPLists lists on the Tor exit list.
|
||||
const torExitAddr = "203.0.113.2"
|
||||
|
||||
func testListing() blocklist.Listing {
|
||||
return blocklist.Listing{
|
||||
// fakeIPLists has two lists. The test list holds addrInTestPrefixV4 and
|
||||
// listedUnroutedAddr, and the Tor exit list holds torExitAddr.
|
||||
type fakeIPLists struct{}
|
||||
|
||||
func testListing() iplist.Listing {
|
||||
return iplist.Listing{
|
||||
Name: "test_list",
|
||||
Description: "On the test list",
|
||||
Updated: time.Date(2026, 10, 27, 4, 0, 0, 0, time.UTC),
|
||||
}
|
||||
}
|
||||
|
||||
func (fakeBlocklists) Lookup(addr netip.Addr) []blocklist.Listing {
|
||||
func (fakeIPLists) Lookup(addr netip.Addr) []iplist.Listing {
|
||||
if addr == netip.MustParseAddr(addrInTestPrefixV4) ||
|
||||
addr == netip.MustParseAddr(listedUnroutedAddr) {
|
||||
return []blocklist.Listing{testListing()}
|
||||
return []iplist.Listing{testListing()}
|
||||
}
|
||||
|
||||
return []blocklist.Listing{}
|
||||
if addr == netip.MustParseAddr(torExitAddr) {
|
||||
return []iplist.Listing{{Name: iplist.TorExitsList}}
|
||||
}
|
||||
|
||||
return []iplist.Listing{}
|
||||
}
|
||||
|
||||
func (fakeBlocklists) ShortestInterval() time.Duration {
|
||||
return testBlocklistInterval
|
||||
func (fakeIPLists) ShortestInterval() time.Duration {
|
||||
return testIPListInterval
|
||||
}
|
||||
|
||||
// TestIPLookupReportsBlocklists looks up two routed addresses, one of them on
|
||||
// a blocklist. The listed address's result names the list, says what it means
|
||||
// and when it was updated. The other's result has an empty list.
|
||||
func TestIPLookupReportsBlocklists(t *testing.T) {
|
||||
// TestIPLookupFlags looks up addresses with and without a route. Every result
|
||||
// has both flags, under the JSON keys the API documents.
|
||||
func TestIPLookupFlags(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
s.SetBlocklists(fakeBlocklists{})
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetIPLists(fakeIPLists{})
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
addr string
|
||||
tor bool
|
||||
bogon bool
|
||||
}{
|
||||
{"on the Tor exit list", torExitAddr, true, true},
|
||||
{"on another list", listedUnroutedAddr, false, true},
|
||||
{"routed IPv4", addrInTestPrefixV4, false, true},
|
||||
{"routed IPv6", addrInTestPrefixV6, false, true},
|
||||
{"private IPv4", "10.0.0.1", false, true},
|
||||
{"loopback IPv4", "127.0.0.1", false, true},
|
||||
{"public IPv4", "1.1.1.1", false, false},
|
||||
{"private IPv6", "fd00::1", false, true},
|
||||
{"loopback IPv6", "::1", false, true},
|
||||
{"public IPv6", "2606:4700:4700::1111", false, false},
|
||||
{"public IPv4 in IPv6 form", "::ffff:1.1.1.1", false, false},
|
||||
{"private IPv4 in IPv6 form", "::ffff:10.0.0.1", false, true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := "/api/v1/ip/" + tt.addr
|
||||
|
||||
results := resultFields(t, get(t, s, path).Body.Bytes())
|
||||
if len(results) != 1 {
|
||||
t.Fatalf("%s: %d results, want 1", path, len(results))
|
||||
}
|
||||
|
||||
var got map[string]bool
|
||||
|
||||
err := json.Unmarshal(results[0]["flags"], &got)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: decoding flags %s: %v", path, results[0]["flags"], err)
|
||||
}
|
||||
|
||||
want := map[string]bool{"is_tor": tt.tor, "is_bogon": tt.bogon}
|
||||
if !maps.Equal(got, want) {
|
||||
t.Errorf("%s: flags %v, want %v", path, got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestIPLookupReportsIPLists looks up two routed addresses, one of them on an
|
||||
// IP list. The listed address's result names the list, says what it means and
|
||||
// when it was updated. The other's result has an empty list.
|
||||
func TestIPLookupReportsIPLists(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetIPLists(fakeIPLists{})
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(t.Context(),
|
||||
@@ -1498,15 +1573,15 @@ func TestIPLookupReportsBlocklists(t *testing.T) {
|
||||
|
||||
want := testListing()
|
||||
|
||||
listed := results[0].Blocklists
|
||||
listed := results[0].IPLists
|
||||
if len(listed) != 1 || listed[0].Name != want.Name ||
|
||||
listed[0].Description != want.Description ||
|
||||
!listed[0].Updated.Equal(want.Updated) {
|
||||
t.Errorf("%s: blocklists %+v, want [%+v]", results[0].IP, listed, want)
|
||||
t.Errorf("%s: ip_lists %+v, want [%+v]", results[0].IP, listed, want)
|
||||
}
|
||||
|
||||
if !strings.Contains(rec.Body.String(), `"blocklists":[]`) {
|
||||
t.Errorf("answer %s, want an empty blocklists list for %s",
|
||||
if !strings.Contains(rec.Body.String(), `"ip_lists":[]`) {
|
||||
t.Errorf("answer %s, want an empty ip_lists list for %s",
|
||||
rec.Body, addrInTestPrefixV6)
|
||||
}
|
||||
}
|
||||
@@ -1536,20 +1611,19 @@ func resultFields(t *testing.T, answer []byte) []map[string]json.RawMessage {
|
||||
return body.Data.Results
|
||||
}
|
||||
|
||||
// TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute looks up a listed
|
||||
// address that has no live route, alone and beside a routed address. Each
|
||||
// answer is 200 and has a result for the address, with its blocklists and
|
||||
// every route field set to null. Before the fix the lookup of that address
|
||||
// alone answered 404, and beside a routed address it was only named in
|
||||
// errors, so its blocklists were never reported. After that fix the result
|
||||
// still left out the route fields
|
||||
// (https://git.eeqj.de/sneak/routewatch/issues/200).
|
||||
func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
|
||||
// TestIPLookupReportsIPListsOfAnAddressWithNoRoute looks up a listed address
|
||||
// that has no live route, alone and beside a routed address. Each answer is
|
||||
// 200 and has a result for the address, with its IP lists and every route
|
||||
// field set to null. Before the fix the lookup of that address alone answered
|
||||
// 404, and beside a routed address it was only named in errors, so its IP
|
||||
// lists were never reported. After that fix the result still left out the
|
||||
// route fields (https://git.eeqj.de/sneak/routewatch/issues/200).
|
||||
func TestIPLookupReportsIPListsOfAnAddressWithNoRoute(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
s.SetBlocklists(fakeBlocklists{})
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetIPLists(fakeIPLists{})
|
||||
|
||||
// routeFields are the fields of a result that come from the address's
|
||||
// route and that route's origin AS.
|
||||
@@ -1605,10 +1679,10 @@ func TestIPLookupReportsBlocklistsOfAnAddressWithNoRoute(t *testing.T) {
|
||||
}
|
||||
|
||||
result := body.Data.Results[i]
|
||||
if len(result.Blocklists) != 1 ||
|
||||
result.Blocklists[0].Name != testListing().Name {
|
||||
t.Errorf("%s: blocklists %+v, want [%+v]",
|
||||
target, result.Blocklists, testListing())
|
||||
if len(result.IPLists) != 1 ||
|
||||
result.IPLists[0].Name != testListing().Name {
|
||||
t.Errorf("%s: ip_lists %+v, want [%+v]",
|
||||
target, result.IPLists, testListing())
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1626,10 +1700,10 @@ func TestIPLookupResultsHaveTheSameFields(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{hosts: map[string][]string{
|
||||
useFakeResolver(s, &fakeResolver{hosts: map[string][]string{
|
||||
twoAddressHostName: {addrInTestPrefixV4, listedUnroutedAddr},
|
||||
}}
|
||||
s.SetBlocklists(fakeBlocklists{})
|
||||
}})
|
||||
s.SetIPLists(fakeIPLists{})
|
||||
|
||||
// want is the fields of the first result.
|
||||
var want []string
|
||||
@@ -1661,7 +1735,7 @@ func TestIPLookupOfAnAddressWithoutPTRNamesOrStoredNetblock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
path := "/api/v1/ip/" + addrInTestPrefixV4
|
||||
|
||||
@@ -1684,15 +1758,15 @@ func TestIPLookupOfAnAddressWithoutPTRNamesOrStoredNetblock(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestIPLookupAnswersMayBeCached checks the Cache-Control header of the IP
|
||||
// lookup answers. Caches may keep them for the blocklists' shortest interval,
|
||||
// lookup answers. Caches may keep them for the IP lists' shortest interval,
|
||||
// and only the caller's own browser may keep the answer about the caller's
|
||||
// own address.
|
||||
func TestIPLookupAnswersMayBeCached(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
s.SetBlocklists(fakeBlocklists{})
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetIPLists(fakeIPLists{})
|
||||
|
||||
tests := []struct {
|
||||
path string
|
||||
@@ -1742,10 +1816,10 @@ func TestIPLookupHidesPrivateAddresses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{hosts: map[string][]string{
|
||||
useFakeResolver(s, &fakeResolver{hosts: map[string][]string{
|
||||
internalHostName: {internalHostAddr},
|
||||
"mixed.test": {internalHostAddr, addrInTestPrefixV4},
|
||||
}}
|
||||
}})
|
||||
|
||||
notPublic := internalHostName + ": " + hostNotPublicMessage
|
||||
|
||||
|
||||
@@ -85,7 +85,7 @@ func TestIPLookupGivesTheOriginMostPeersCarry(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := newServerWithTable(t, newTwoOriginTable())
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
firstSeen := testRouteTime(-1)
|
||||
|
||||
|
||||
@@ -114,7 +114,7 @@ func TestJSONAnswersPutNoUnitInAString(t *testing.T) {
|
||||
`^([0-9]+(\.[0-9]+)? ?(ns|µs|ms|s|m|h|B|kB|MB|GB|TB))+$`)
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.SetASNFetcher(fixedIntervalASNFetcher{interval: time.Second})
|
||||
|
||||
paths := []string{
|
||||
@@ -174,7 +174,7 @@ func TestJSONAnswersCarryMeta(t *testing.T) {
|
||||
)
|
||||
|
||||
s := newServerWithTestRoutes(t)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
s.streamer.GetMetricsTracker().SetConnected(true)
|
||||
s.router.Get(writesNothingPath, JSONValidationMiddleware(http.HandlerFunc(
|
||||
func(http.ResponseWriter, *http.Request) {})).ServeHTTP)
|
||||
@@ -239,7 +239,7 @@ func TestJSONAnswersCarryMeta(t *testing.T) {
|
||||
// handlers of the other routes 500.
|
||||
failing := New(failingDB{}, newTestTable(),
|
||||
streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
||||
failing.resolver = &fakeResolver{}
|
||||
useFakeResolver(failing, &fakeResolver{})
|
||||
|
||||
checkAnswerCarriesMeta(t, failing, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, healthCheckPath, nil),
|
||||
@@ -514,7 +514,7 @@ func newServerWithConfig(
|
||||
|
||||
s := New(db, newTestTable(), streamer.New(logger.New(), metrics.New()),
|
||||
&logger.Logger{Logger: slog.New(slog.NewJSONHandler(requestLog, nil))}, cfg)
|
||||
s.resolver = &fakeResolver{}
|
||||
useFakeResolver(s, &fakeResolver{})
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
+26
-13
@@ -22,9 +22,9 @@ import (
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"git.eeqj.de/sneak/routewatch/internal/blocklist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||
"git.eeqj.de/sneak/routewatch/internal/iplist"
|
||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||
"git.eeqj.de/sneak/routewatch/internal/routetable"
|
||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||
@@ -45,17 +45,21 @@ type ASNFetcher interface {
|
||||
GetStats() ASNFetcherStats
|
||||
}
|
||||
|
||||
// Blocklists tells which public blocklists list an address, from copies it
|
||||
// keeps in memory, as *blocklist.Blocklists does.
|
||||
type Blocklists interface {
|
||||
Lookup(addr netip.Addr) []blocklist.Listing
|
||||
// IPLists tells which public IP lists contain an address, from copies it keeps
|
||||
// in memory, as *iplist.IPLists does.
|
||||
type IPLists interface {
|
||||
Lookup(addr netip.Addr) []iplist.Listing
|
||||
ShortestInterval() time.Duration
|
||||
}
|
||||
|
||||
// dnsResolver looks up the addresses of host names and the names of addresses
|
||||
// in DNS, as *net.Resolver does.
|
||||
// dnsResolver looks up the addresses of host names in DNS, as *net.Resolver
|
||||
// does.
|
||||
type dnsResolver interface {
|
||||
LookupHost(ctx context.Context, host string) ([]string, error)
|
||||
}
|
||||
|
||||
// ptrResolver looks up the PTR names of addresses, as *ptr.Cache does.
|
||||
type ptrResolver interface {
|
||||
LookupAddr(ctx context.Context, addr string) ([]string, error)
|
||||
}
|
||||
|
||||
@@ -69,16 +73,19 @@ type Server struct {
|
||||
port int
|
||||
srv *http.Server
|
||||
asnFetcher ASNFetcher
|
||||
blocklists Blocklists
|
||||
ipLists IPLists
|
||||
|
||||
// trustedProxies are the reverse proxies whose X-Forwarded-For header
|
||||
// RealIPMiddleware believes
|
||||
trustedProxies []netip.Prefix
|
||||
|
||||
// resolver makes the IP lookup's DNS lookups: net.DefaultResolver, or a
|
||||
// resolver looks up the IP lookup's host names: net.DefaultResolver, or a
|
||||
// fake in tests
|
||||
resolver dnsResolver
|
||||
|
||||
// ptrResolver looks up the IP lookup's PTR names. SetPTRResolver sets it.
|
||||
ptrResolver ptrResolver
|
||||
|
||||
// cards is the copy of the home page's cards that the pages serve, nil
|
||||
// until the server first works them out
|
||||
cards atomic.Pointer[homeCards]
|
||||
@@ -196,14 +203,20 @@ func (s *Server) Stop(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetPTRResolver sets what looks up the IP lookup's PTR names. Call it before
|
||||
// Start: the handlers read it without a lock.
|
||||
func (s *Server) SetPTRResolver(resolver ptrResolver) {
|
||||
s.ptrResolver = resolver
|
||||
}
|
||||
|
||||
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups. Call it
|
||||
// before Start: the handlers read the fetcher without a lock.
|
||||
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
|
||||
s.asnFetcher = fetcher
|
||||
}
|
||||
|
||||
// SetBlocklists sets the blocklists the IP lookups report. Call it before
|
||||
// Start: the handlers read the blocklists without a lock.
|
||||
func (s *Server) SetBlocklists(blocklists Blocklists) {
|
||||
s.blocklists = blocklists
|
||||
// SetIPLists sets the IP lists the IP lookups report. Call it before Start:
|
||||
// the handlers read the IP lists without a lock.
|
||||
func (s *Server) SetIPLists(ipLists IPLists) {
|
||||
s.ipLists = ipLists
|
||||
}
|
||||
Reference in new issue
Block a user