Commit Graph
26 Commits
Author SHA1 Message Date
sneak 133d2ffc66 Take Brazilian AS details from NIC.br's WHOIS server (closes #174)
check / check (push) Waiting to run
LACNIC's answer for an AS that NIC.br registers is NIC.br's answer with
each accented letter replaced by U+FFFD, so a name such as AS52751's
cannot be recovered from it. When LACNIC's answer carries NIC.br's
copyright notice, the client now also asks whois.registro.br for the AS
and takes the fields from that answer, which is Latin-1 and decoded as
before. When that query fails, when the answer has no fields, or when
NIC.br says the client is over its query limit, LACNIC's fields are kept
and a warning is logged. After an over-limit answer the client skips
NIC.br for an hour. The tests use recorded answers for AS52751.

Model: opus-5-5
2026-10-27 02:44:50 +00:00
clawbot cd1b9b94a4 Stop asinfo-gen when the as.csv download breaks off (closes #177)
check / check (push) Canceled after 0s
parseCSV skipped a record on any read error and asked for the next one.
When the download of as.csv broke off, the response body returned the
same error on every read, so the generator logged it forever and make
asupdate never finished.

parseCSV now skips only a record that does not parse, a csv.ParseError.
Any other read error ends it with that error, so the generator exits
non-zero and script/asupdate keeps the old list.

Model: opus-5-5
2026-10-27 03:10:40 +01:00
clawbot 023c3f9b41 Keep the AS list when the asupdate generator fails (closes #173)
check / check (push) Canceled after 0s
make asupdate piped the generator into gzip, so the target saw gzip's
exit status. A failed download replaced pkg/asinfo/asdata.json.gz with
an empty file and the target still exited 0.

The target now runs the new script/asupdate. It writes the output of
cmd/asinfo-gen to a temporary file. Only when the generator succeeds
does it compress the list and move it into place. Otherwise it leaves
the file alone and exits non-zero. The generator now fails when as.csv
holds no AS.

The generator reads as.csv from ipverse/as-metadata, the source's new
name. The list it builds there is unchanged, so no new list is
committed.

Model: opus-5-5
2026-10-27 02:43:35 +01:00
clawbot 57469b6370 Read only the AS's objects in RIPE, APNIC and AFRINIC answers (closes #168)
check / check (push) Canceled after 0s
APNIC and RIPE start their answer for an AS with the as-block object of
the block the AS is in. The parser kept the first value of each field
across the whole answer, so it stored the block's dates as the AS's.

The parser now reads an answer from RIPE, APNIC or AFRINIC object by
object. It takes the fields from the aut-num object and the organisation
object it names, and leaves out every other object. The irt object's
address therefore no longer joins the organisation's. ARIN and LACNIC
answers are parsed as before.

Model: opus-5-5
2026-10-27 02:12:48 +01:00
clawbot 6d55d39737 Keep accented letters in AS names (closes #165)
check / check (push) Canceled after 0s
The built-in AS list had no letter outside ASCII, and make asupdate
rejected today's as.csv, which added a country code column.

The generator now checks only the first three header columns and reads
past the rest; the parsing moved into parseCSV. The list is regenerated.
The new source spells some names differently and no longer lists AS 0
or the documentation AS numbers, so the tests that relied on those
entries changed.

A WHOIS answer that is not valid UTF-8 is now decoded as Latin-1, which
RIPE and APNIC send, before it is parsed and stored. LACNIC answers in
UTF-8.

Model: opus-5-5
2026-10-27 01:50:34 +01:00
clawbot 86e1f20e11 Show ages on the AS and prefix pages in words (closes #163)
check / check (push) Canceled after 0s
The Age column printed Go's duration format for every plural age, such
as "5m0s ago" or "360h0m0s ago".

The templates now register humanize.Time from go-humanize, which the
server already uses, as the timeSince function, and the hand-written
function is gone. Ages read "5 minutes ago", "3 hours ago" or "2 weeks
ago". An age under a minute reads "now" or in seconds instead of "just
now". An age over a month reads in months or years instead of a date.

Model: opus-5-5
2026-10-27 01:23:50 +01:00
clawbot a1de67ebe1 Never move last seen back or first seen forward (closes #162)
check / check (push) Canceled after 0s
Some RIS collectors send updates days old while they catch up on a
backlog. Every update to a stored AS, peering or peer set last_seen to
the update's time, even an older one, and never lowered first_seen. An
AS page could then list a peer first seen after it was last seen.

The update statements for asns, peerings and bgp_peers now keep the
earlier of the stored and new first_seen and the later of the stored
and new last_seen, with MIN and MAX in the SQL. The batch and the
single-row writes share each statement. GetOrCreateASN returns the
times the row now holds. Route times are unchanged.

Model: opus-5-5
2026-10-27 01:10:01 +01:00
clawbot 355cd37496 Show route churn as a whole count per minute (closes #160)
check / check (push) Canceled after 0s
The status page showed route churn per second as a one-minute moving
average, so a quiet feed gave a fraction such as 0.1. The average was
correct.

The metrics tracker now also counts prefix announcements and
withdrawals, from all peers, in each clock minute. /api/v1/stats
reports the count for the last full minute as route_churn_per_min in
stream, in place of route_churn_per_sec. The status page shows it as
"Route Churn/min".

The status and home pages round messages and updates per second to
whole numbers.

Model: opus-5-5
2026-10-27 00:14:24 +01:00
clawbot 087b8de9ae Back off WHOIS lookups more slowly and stop asking for personal data (closes #159)
check / check (push) Canceled after 0s
A failed lookup now makes the fetcher's wait 1.5 times as long, up to
one minute instead of five. A success still halves it.

RIPE counts the person and role objects an answer holds against a daily
limit per IP address, and refuses an address for good that passes it
too often. RIPE, APNIC and AFRINIC are now asked with --no-personal,
which leaves those objects out but keeps the organisation object.

An answer with RIPE's access-denied or limit-reached error line is now
reported as a refusal holding that line, not as an answer with no
fields.

Model: opus-5-5
2026-10-26 23:40:13 +01:00
clawbot 99c7efc156 Make the home page's AS name search work (closes #151)
check / check (push) Canceled after 0s
The AS Name Search card on the home page said the search was coming soon.
It now asks the new GET /api/v1/search/as?q=<text> and lists the matches
as links to their AS pages. asinfo.Search ignores case, returns the first
matches up to a limit in AS number order, and counts all matches. An
empty q answers 400.

When the database has no record of an AS, its page and /api/v1/as/{asn}
take its handle and description from asinfo and still list its prefixes.
They answer 404 only for an AS with no record, no asinfo entry and no
prefix. AS 0 means no known origin: it always answers 404, and the search
never lists it.

Model: opus-5-5
2026-10-26 22:51:12 +01:00
clawbot ef32082339 Bring README and TODO.md up to date with the code (closes #72)
check / check (push) Canceled after 0s
README "HTTP Endpoints" now lists every route the server registers with
what it answers, errors included: / is a home page and /ip/{addr} answers
JSON, neither redirects; /ip/ looks up the caller; the IP endpoints take
host names and comma-separated lists; 400, 404 and 504 are described, and
the health check's envelope. "Database Schema" matches schema.sql, and
other sentences the code contradicted are corrected. TODO.md gets a dated
line for each issue landed since 2026-10-03, with Status and Next Step
current.

Model: opus-5-5
2026-10-19 16:09:22 +02:00
clawbot b89987e356 Re-vendor the shared files from sneak/prompts at 0653579 (closes #52)
check / check (push) Canceled after 0s
Takes .golangci.yml, .dockerignore, .gitignore, .editorconfig, the
workflow and REPO_POLICIES.md from sneak/prompts 0653579, with this
repository's own entries after the shared content. Lint (golangci-lint
v2.14.0) and test are Dockerfile phases that script/lint and script/test
build alone with a new CACHEBUST value, keeping Go's and golangci-lint's
caches; the image build depends on both. make fmt and make fmt-check also
format the markdown with prettier, pinned in yarn.lock. The agent
guidance moves to AGENTS.md. The Go code is fixed for what the new
configuration finds, and the AS and prefix detail reads now return the
rows' error after their loops.

Model: opus-5-5
2026-10-19 12:10:11 +02:00
clawbot a9b68608c6 Serve the /api/v1/stats prefix distribution from memory (closes #30)
check / check (push) Successful in 3m19s
The prefix distribution was the last query on the stats path. It read one index entry per live route on every request, so on a large database it took the whole 4-second deadline and /api/v1/stats answered 500.

The distribution now lives next to the in-memory counts: seeded once at startup from the same query, then kept exact by every live-route write. A new route whose prefix had no live route adds one at its mask length; a delete that leaves a prefix with no live route takes one away. Each check is one lookup on the prefix index, within the same write. The single-route delete now runs its delete and that lookup in one transaction.

Model: opus-5-5
2026-10-03 18:43:58 +02:00
clawbot 44a5f4cdca Create new databases with auto_vacuum incremental (closes #43)
check / check (push) Successful in 2m50s
SQLite only accepts auto_vacuum before the database file is first written. The connection switched to WAL first, which writes the file, so the PRAGMA auto_vacuum in Initialize came too late and was ignored. The setting now goes in the connection string, which the driver applies on open before the journal mode, and the late PRAGMA is removed.

Vacuum now reads every row PRAGMA incremental_vacuum returns: SQLite frees one page per row, and the single step ExecContext took freed only one page per call.

Tests check that every pooled connection sees auto_vacuum incremental on a new database and that one Vacuum call frees every page left by deleting routes.

Model: opus-5-5
2026-10-03 17:09:29 +02:00
clawbot 32704c601e Look up IP addresses by prefix at each mask length (closes #48)
check / check (push) Successful in 3m13s
Looking up an address on /ip/ and /api/v1/ip/ read every IPv6 live route and about half of the IPv4 range index, so on a day-sized database it passed the 30-second request timeout. The lookup is now one function for both families: from the longest mask length down, it looks up the address's network prefix on the existing prefix index, and the first live route wins.

The feed sends IPv6 withdrawals uncompressed, so they never matched the stored compressed prefix and never removed a route. Prefixes are now stored in the text form net/netip prints, so IPv6 withdrawals take effect.

ip_start, ip_end and GetASInfoForIP are removed; nothing read them. A database from before this change must be deleted.

Model: opus-5-5
2026-10-03 16:04:38 +02:00
clawbot 7322f936e4 Stamp the git tag or short commit in a plain docker build (closes #46)
check / check (push) Successful in 3m6s
A plain `docker build .` stamped `unknown` into the page footer:
.dockerignore left out .git, and the Dockerfile built without the -X
flags. The context now carries .git without .git/config, which can hold
a credential. The build stage takes the VERSION build argument when one
is given, otherwise `git describe --tags --always`, fails the build if
.git is present and no version comes out, and stamps it together with
the full commit the footer links to. `make build` now uses
`git describe` as well, and script/docker is the current shared copy.

Model: opus-5-5
2026-10-02 08:17:35 +02:00
clawbot 6422d9fa0c Container makes its data directory usable itself (closes #42)
check / check (push) Successful in 8s
sneak's standing rule: the container makes its data directory usable itself, with no step on the host. entrypoint.sh now creates /var/lib/berlin.sneak.app.routewatch if it is missing and stops the start when any step fails (set -euo pipefail); before, a failed cd went on to change the ownership of whatever directory the script was in, and a failed chown still started the daemon. Taking ownership of the directory and switching to the routewatch user through setpriv are unchanged. The README's upaas volume line now says only which path to mount.

The empty-directory and other-uid cases were run by hand on the built image with upaas-style bind mounts, not added as an automated test.

Model: opus-5-5
2026-09-29 12:23:42 +02:00
clawbot 057e0bd9a9 Add a .dockerignore (closes #39)
check / check (push) Successful in 6s
Adds a .dockerignore, which REPO_POLICIES.md lists among the files every repo has. Until now every docker build sent the whole working tree as its build context, and the image's source archive is made from that context. It now keeps out .git, local build and test output, archives, local databases, .env and a local Go workspace. Every tracked file stays in, so the format check, the linter, the tests, the build and the source archive read the same files as before.

Without .git in the context the binary's build info records no git revision; nothing in routewatch reads it.

Model: opus-5-5
2026-09-29 07:42:20 +02:00
clawbot 87a3bc115e README: license and author up front; bring TODO.md up to date (closes #38)
check / check (push) Successful in 3m14s
Brings README.md and TODO.md in line with REPO_POLICIES.md and with what is on next. The README's first line now names routewatch, what it is, its MIT licence and its author, @sneak; the License section says MIT and links LICENSE; a new Author section names @sneak.

TODO.md's Status, Next Step and Future Steps now describe next as it is: it waits for sneak to merge the milestone PR, after which the upaas deploy and the run under a real 5 GiB limit are his. Completed Steps gains a line for each issue that landed without one.

Docs only. make fmt formats only Go here, so the markdown was wrapped by hand.

Model: opus-5-5
2026-09-29 07:06:30 +02:00
clawbot 1ac24669d3 Stop the streamer without sending on closed queues (closes #34)
check / check (push) Successful in 2m38s
Stopping the daemon while the RIS Live feed was flowing could panic with "send on closed channel" and skip the rest of the shutdown. Stop cancels the stream and closes the handler queues under the streamer's write lock, but the read loop checked for a stop only before parsing each line. It now checks again under the read lock it already takes just before handing a message to the queues, so it never sends to a closed queue. Stop also clears its cancel function and returns early when there is none, so a second call no longer closes the queues again. A test forces both cases.

Behaviour change: Stop before Start now does nothing.

Model: opus-5-5
2026-09-28 21:42:33 +02:00
clawbot 6187ac8503 Let the daemon receive docker stop's signal itself (closes #33)
check / check (push) Successful in 2m42s
entrypoint.sh now switches to the routewatch user (UID 1000) with setpriv instead of runuser. setpriv replaces itself with the daemon, so the daemon is the container's main process and receives docker stop's signal itself. runuser stayed in between, passed the signal on and killed the daemon 2 seconds later, so every stop ended with exit 143. The daemon now gets the whole wait the caller allows, up to its own 60-second limit, and a clean stop exits 0. Taking ownership of the state directory and the MALLOC_ARENA_MAX check still run as root first.

Not fixed here: a stop while the feed is flowing can still panic (#34).

Model: opus-5-5
2026-09-28 21:08:33 +02:00
clawbot f2a9e90625 Refuse invalid settings at start, health check follows PORT (closes #31)
check / check (push) Successful in 3m17s
A set but invalid PORT (anything but plain digits from 1 to 65535) or a relative XDG_DATA_HOME now stops the start before the database opens; before, a bad PORT left the daemon running without HTTP. entrypoint.sh refuses a MALLOC_ARENA_MAX that is not a positive whole number, since glibc ignores a bad one silently. The HEALTHCHECK probes the port PORT names, 8080 when unset. PORT is now read in internal/config, so server.New takes the config.

The README gives the real Linux state directory, lists XDG_DATA_HOME, and adds "Running under upaas": port, volume, environment, the 5g memory limit and the health check.

Unverified: the 5g memory limit could not be exercised on the build host.

Model: opus-5-5
2026-09-28 20:08:42 +02:00
clawbot df9e23d503 Serve /api/v1/stats counts from realtime in-memory counters (closes #27)
check / check (push) Successful in 2m36s
Realtime in-memory counters seeded at startup and adjusted on every insert, update and delete; no periodic recompute. Independent review passed: #29 (comment)

model: claude-opus-4-8 (implementation and review); merged by claude-fable-5
2026-09-22 09:41:16 +02:00
sneak 583c16a65c Adopt scripts-to-rule-them-all: script/ entrypoints, Makefile shims 2026-07-07 01:53:25 +02:00
sneak 0747165ab1 Add standard Workflow section to TODO.md 2026-07-06 21:06:41 +02:00
sneak b144e1515b Add TODO.md 2026-07-06 20:35:48 +02:00