Resolve PTR names over DNS-over-HTTPS and cache them for the record TTL (closes #238)
check / check (push) Successful in 1m58s
check / check (push) Successful in 1m58s
`internal/ptr` asks Google, Cloudflare, AdGuard and Quad9 in turn, 2 seconds in all per address. It keeps answers in memory and writes them through to `cache.db`, opened by the new `internal/cachedb`. A reserved address is not sent to the resolvers. The cache holds at most 250,000 addresses. Judgement calls: - Each resolver gets an equal share of the time left, so a hung resolver does not use up the 2 seconds. - The negative TTL is the smaller of the SOA minimum and the SOA's own TTL (RFC 2308). - Names that are not host names are dropped, as `net.Resolver.LookupAddr` does. Model: opus-5-5
This commit is contained in:
sneak committed
1 parent
641798bb22
commit
848271b5c8
21 files changed
+1973
-69
No files matched your search
@@ -281,8 +281,8 @@ answer holds `query`, the text looked up, and `results`, with one entry for each
|
||||
address. Every entry has the same fields:
|
||||
|
||||
- `ip`: the address
|
||||
- `ptr`: the address's PTR names, the host names that DNS gives for it, or an
|
||||
empty list when it has none
|
||||
- `ptr`: the address's PTR names, the host names that DNS gives for it (see PTR
|
||||
lookups), or an empty list when it has none
|
||||
- `flags`: yes-or-no facts about the address:
|
||||
- `is_tor`: true when the address is on the `tor_exits` list
|
||||
- `is_vpn`: true when the address is on any list of VPN servers (see IP
|
||||
@@ -334,9 +334,6 @@ address, such as one that resolves to a private address, is reported as
|
||||
- `{ip}` comes to more than 10 addresses once its names are resolved
|
||||
- nothing in `{ip}` is an IP address or a host name with a public address
|
||||
|
||||
Each PTR lookup gets 2 seconds. An address whose lookup takes longer is answered
|
||||
without PTR names.
|
||||
|
||||
A successful answer carries `Cache-Control` with a `max-age` of 15 minutes, the
|
||||
shortest IP list interval. The answer for the caller's own address (`/ip/`) is
|
||||
`private`, and the others are `public`.
|
||||
@@ -465,6 +462,45 @@ How RouteWatch keeps the lists:
|
||||
- When a download fails, or its file does not parse, RouteWatch keeps the
|
||||
previous copy and logs the failure. It tries again one interval later.
|
||||
|
||||
## PTR lookups
|
||||
|
||||
An IP lookup finds each address's PTR names over DNS-over-HTTPS. It asks these
|
||||
public resolvers in order, until one answers:
|
||||
|
||||
- Google: `https://dns.google/dns-query`
|
||||
- Cloudflare: `https://cloudflare-dns.com/dns-query`
|
||||
- AdGuard: `https://unfiltered.adguard-dns.com/dns-query`
|
||||
- Quad9: `https://dns.quad9.net/dns-query`
|
||||
|
||||
One address's lookup gets 2 seconds across all four. Each resolver gets an equal
|
||||
share of the time left, so one that does not answer leaves time for the next.
|
||||
When none answers in time, the address's `ptr` is an empty list.
|
||||
|
||||
A reserved address, one whose `is_bogon` flag is true, is not sent to the
|
||||
resolvers. Its `ptr` is an empty list.
|
||||
|
||||
What goes outside: each resolver asked receives the address being looked up. For
|
||||
`/ip/` and the home page, that address is the visitor's own. A lookup answered
|
||||
from the cache sends nothing.
|
||||
|
||||
How RouteWatch keeps the answers:
|
||||
|
||||
- It holds answers in memory. An answer that has not expired is served from
|
||||
there.
|
||||
- It writes new answers every 5 seconds to `cache.db` in the state directory
|
||||
(see Configuration), and loads them from there at startup.
|
||||
- An answer expires after the TTL of its records, the shortest one when there
|
||||
are several. RouteWatch keeps it for at least 60 seconds and at most 24 hours.
|
||||
- When an address has no PTR names, that answer expires after the SOA record's
|
||||
minimum field, or the record's own TTL when that is shorter. RouteWatch keeps
|
||||
it for at most 1 hour.
|
||||
- An answer with no PTR names and no SOA record is not kept. Neither is one
|
||||
whose SOA record has a TTL or minimum field of 0.
|
||||
- A failed lookup is not kept, so the next lookup of the address asks again.
|
||||
- Every 10 minutes it deletes expired answers from memory and from `cache.db`.
|
||||
- It keeps answers for at most 250,000 addresses, and keeps no answer for a new
|
||||
address while it holds that many.
|
||||
|
||||
## Configuration
|
||||
|
||||
RouteWatch reads its settings from environment variables. It takes no
|
||||
@@ -513,7 +549,13 @@ by the client. For a request from any other address, they use the address the
|
||||
request came from, whatever headers it carries. `X-Real-IP` and `True-Client-IP`
|
||||
are not read.
|
||||
|
||||
The state directory holds the database, `db.sqlite`, and the IP list copies:
|
||||
The state directory holds:
|
||||
|
||||
- `db.sqlite`: the database
|
||||
- `cache.db`: the answers of PTR lookups
|
||||
- `iplists`: the IP list copies
|
||||
|
||||
It is at:
|
||||
|
||||
- macOS: `~/Library/Application Support/berlin.sneak.app.routewatch/`
|
||||
- Linux and the BSDs, as root: `/var/lib/berlin.sneak.app.routewatch/`
|
||||
@@ -670,6 +712,8 @@ The parts:
|
||||
with old or missing details asks the fetcher for it at once, without the wait.
|
||||
- The IP lists part downloads the IP lists and answers which lists hold an
|
||||
address.
|
||||
- The PTR cache looks up the PTR names of addresses and keeps the answers (see
|
||||
PTR lookups).
|
||||
- The database maintainer checkpoints the database every 5 seconds: it folds the
|
||||
write-ahead log, where SQLite first writes changes, into the database file. It
|
||||
reclaims free space every 10 minutes. Every hour it refreshes the statistics
|
||||
@@ -779,6 +823,7 @@ On `SIGTERM` or `SIGINT` the daemon stops its parts in this order:
|
||||
4. the database maintainer
|
||||
5. the IP list downloads
|
||||
6. the HTTP server
|
||||
7. the PTR cache, which writes what it holds
|
||||
|
||||
Last, the daemon checkpoints and closes the database. It gives all this 8
|
||||
seconds and then exits with status 0. A stop that is still running after 9
|
||||
@@ -799,6 +844,8 @@ What a stop can lose:
|
||||
- When the 8 seconds run out, the streamer and the handlers log what they could
|
||||
not hand over or write. The database is then not closed. When it next opens,
|
||||
SQLite rolls back any write the exit cut off.
|
||||
- When the 8 seconds run out, the PTR answers not yet in `cache.db` are lost.
|
||||
RouteWatch asks the resolvers for them again the next time they are needed.
|
||||
|
||||
### Code layout
|
||||
|
||||
@@ -812,11 +859,13 @@ routewatch/
|
||||
│ └── message-examples.json # Examples of RIS Live messages
|
||||
├── internal/
|
||||
│ ├── bogon/ # Reserved prefixes and AS numbers
|
||||
│ ├── cachedb/ # cache.db, the caches of outside answers
|
||||
│ ├── config/ # Settings from the environment
|
||||
│ ├── database/ # SQLite storage
|
||||
│ ├── iplist/ # Public IP lists: downloads and lookups
|
||||
│ ├── logger/ # Structured logging
|
||||
│ ├── metrics/ # Feed and route update figures
|
||||
│ ├── ptr/ # PTR lookups over DNS-over-HTTPS, and their cache
|
||||
│ ├── ristypes/ # The RIS Live message types
|
||||
│ ├── routetable/ # The whole routing table in memory
|
||||
│ ├── routewatch/ # The daemon: startup, handlers, WHOIS fetcher
|
||||
@@ -865,6 +914,13 @@ announcements(id PRIMARY KEY, prefix_id, peer_asn, origin_asn, path,
|
||||
|
||||
The indexes are in `internal/database/schema.sql`.
|
||||
|
||||
`cache.db` is a separate file, with one table:
|
||||
|
||||
```sql
|
||||
-- The answers of PTR lookups. names is empty for an address with no PTR names.
|
||||
hostnames(address PRIMARY KEY, names, expires_at)
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
The `make` targets:
|
||||
|
||||
Reference in new issue
Block a user