Author SHA1 Message Date
sneak 7de2066bf2 WIP: re-vendor canonical files and gate phases
check / check (push) Canceled after 0s
Model: opus-5-5
2026-10-14 12:35:07 +00:00
sneak 00e51d284f Merge pull request 'Next milestone: production memory under 5 GiB, ready for upaas' (#6) from next into main
check / check (push) Canceled after 0s
Reviewed-on: #6
2026-10-06 00:34:48 +02:00
clawbot a9b68608c6 Serve the /api/v1/stats prefix distribution from memory (closes #30)
check / check (push) Successful in 3m19s
The prefix distribution was the last query on the stats path. It read one index entry per live route on every request, so on a large database it took the whole 4-second deadline and /api/v1/stats answered 500.

The distribution now lives next to the in-memory counts: seeded once at startup from the same query, then kept exact by every live-route write. A new route whose prefix had no live route adds one at its mask length; a delete that leaves a prefix with no live route takes one away. Each check is one lookup on the prefix index, within the same write. The single-route delete now runs its delete and that lookup in one transaction.

Model: opus-5-5
2026-10-03 18:43:58 +02:00
clawbot 44a5f4cdca Create new databases with auto_vacuum incremental (closes #43)
check / check (push) Successful in 2m50s
SQLite only accepts auto_vacuum before the database file is first written. The connection switched to WAL first, which writes the file, so the PRAGMA auto_vacuum in Initialize came too late and was ignored. The setting now goes in the connection string, which the driver applies on open before the journal mode, and the late PRAGMA is removed.

Vacuum now reads every row PRAGMA incremental_vacuum returns: SQLite frees one page per row, and the single step ExecContext took freed only one page per call.

Tests check that every pooled connection sees auto_vacuum incremental on a new database and that one Vacuum call frees every page left by deleting routes.

Model: opus-5-5
2026-10-03 17:09:29 +02:00
clawbot 32704c601e Look up IP addresses by prefix at each mask length (closes #48)
check / check (push) Successful in 3m13s
Looking up an address on /ip/ and /api/v1/ip/ read every IPv6 live route and about half of the IPv4 range index, so on a day-sized database it passed the 30-second request timeout. The lookup is now one function for both families: from the longest mask length down, it looks up the address's network prefix on the existing prefix index, and the first live route wins.

The feed sends IPv6 withdrawals uncompressed, so they never matched the stored compressed prefix and never removed a route. Prefixes are now stored in the text form net/netip prints, so IPv6 withdrawals take effect.

ip_start, ip_end and GetASInfoForIP are removed; nothing read them. A database from before this change must be deleted.

Model: opus-5-5
2026-10-03 16:04:38 +02:00
clawbot 7322f936e4 Stamp the git tag or short commit in a plain docker build (closes #46)
check / check (push) Successful in 3m6s
A plain `docker build .` stamped `unknown` into the page footer:
.dockerignore left out .git, and the Dockerfile built without the -X
flags. The context now carries .git without .git/config, which can hold
a credential. The build stage takes the VERSION build argument when one
is given, otherwise `git describe --tags --always`, fails the build if
.git is present and no version comes out, and stamps it together with
the full commit the footer links to. `make build` now uses
`git describe` as well, and script/docker is the current shared copy.

Model: opus-5-5
2026-10-02 08:17:35 +02:00
clawbot 6422d9fa0c Container makes its data directory usable itself (closes #42)
check / check (push) Successful in 8s
sneak's standing rule: the container makes its data directory usable itself, with no step on the host. entrypoint.sh now creates /var/lib/berlin.sneak.app.routewatch if it is missing and stops the start when any step fails (set -euo pipefail); before, a failed cd went on to change the ownership of whatever directory the script was in, and a failed chown still started the daemon. Taking ownership of the directory and switching to the routewatch user through setpriv are unchanged. The README's upaas volume line now says only which path to mount.

The empty-directory and other-uid cases were run by hand on the built image with upaas-style bind mounts, not added as an automated test.

Model: opus-5-5
2026-09-29 12:23:42 +02:00
clawbot 057e0bd9a9 Add a .dockerignore (closes #39)
check / check (push) Successful in 6s
Adds a .dockerignore, which REPO_POLICIES.md lists among the files every repo has. Until now every docker build sent the whole working tree as its build context, and the image's source archive is made from that context. It now keeps out .git, local build and test output, archives, local databases, .env and a local Go workspace. Every tracked file stays in, so the format check, the linter, the tests, the build and the source archive read the same files as before.

Without .git in the context the binary's build info records no git revision; nothing in routewatch reads it.

Model: opus-5-5
2026-09-29 07:42:20 +02:00
clawbot 87a3bc115e README: license and author up front; bring TODO.md up to date (closes #38)
check / check (push) Successful in 3m14s
Brings README.md and TODO.md in line with REPO_POLICIES.md and with what is on next. The README's first line now names routewatch, what it is, its MIT licence and its author, @sneak; the License section says MIT and links LICENSE; a new Author section names @sneak.

TODO.md's Status, Next Step and Future Steps now describe next as it is: it waits for sneak to merge the milestone PR, after which the upaas deploy and the run under a real 5 GiB limit are his. Completed Steps gains a line for each issue that landed without one.

Docs only. make fmt formats only Go here, so the markdown was wrapped by hand.

Model: opus-5-5
2026-09-29 07:06:30 +02:00
clawbot d28a59023a Add MIT LICENSE (closes #1)
check / check (push) Successful in 4m5s
Adds LICENSE with the MIT licence sneak chose for routewatch, copied byte for byte from sneak/webhooker's LICENSE: copyright 2026, Jeffrey Paul. The rest of the issue, .editorconfig and the Gitea check workflow, was already on next, and the README already points at LICENSE.

Judgement call: our repos write the copyright holder in several different ways and sneak named no form; webhooker's gives his full name and email address, the most explicit of them.

Model: opus-5-5
2026-09-29 05:05:35 +02:00
clawbot 1ac24669d3 Stop the streamer without sending on closed queues (closes #34)
check / check (push) Successful in 2m38s
Stopping the daemon while the RIS Live feed was flowing could panic with "send on closed channel" and skip the rest of the shutdown. Stop cancels the stream and closes the handler queues under the streamer's write lock, but the read loop checked for a stop only before parsing each line. It now checks again under the read lock it already takes just before handing a message to the queues, so it never sends to a closed queue. Stop also clears its cancel function and returns early when there is none, so a second call no longer closes the queues again. A test forces both cases.

Behaviour change: Stop before Start now does nothing.

Model: opus-5-5
2026-09-28 21:42:33 +02:00
clawbot 6187ac8503 Let the daemon receive docker stop's signal itself (closes #33)
check / check (push) Successful in 2m42s
entrypoint.sh now switches to the routewatch user (UID 1000) with setpriv instead of runuser. setpriv replaces itself with the daemon, so the daemon is the container's main process and receives docker stop's signal itself. runuser stayed in between, passed the signal on and killed the daemon 2 seconds later, so every stop ended with exit 143. The daemon now gets the whole wait the caller allows, up to its own 60-second limit, and a clean stop exits 0. Taking ownership of the state directory and the MALLOC_ARENA_MAX check still run as root first.

Not fixed here: a stop while the feed is flowing can still panic (#34).

Model: opus-5-5
2026-09-28 21:08:33 +02:00
clawbot f2a9e90625 Refuse invalid settings at start, health check follows PORT (closes #31)
check / check (push) Successful in 3m17s
A set but invalid PORT (anything but plain digits from 1 to 65535) or a relative XDG_DATA_HOME now stops the start before the database opens; before, a bad PORT left the daemon running without HTTP. entrypoint.sh refuses a MALLOC_ARENA_MAX that is not a positive whole number, since glibc ignores a bad one silently. The HEALTHCHECK probes the port PORT names, 8080 when unset. PORT is now read in internal/config, so server.New takes the config.

The README gives the real Linux state directory, lists XDG_DATA_HOME, and adds "Running under upaas": port, volume, environment, the 5g memory limit and the health check.

Unverified: the 5g memory limit could not be exercised on the build host.

Model: opus-5-5
2026-09-28 20:08:42 +02:00
clawbot df9e23d503 Serve /api/v1/stats counts from realtime in-memory counters (closes #27)
check / check (push) Successful in 2m36s
Realtime in-memory counters seeded at startup and adjusted on every insert, update and delete; no periodic recompute. Independent review passed: #29 (comment)

model: claude-opus-4-8 (implementation and review); merged by claude-fable-5
2026-09-22 09:41:16 +02:00
38 changed files with 2622 additions and 1175 deletions

No files matched your search

+97
View File
@@ -0,0 +1,97 @@
# .dockerignore does NOT use .gitignore semantics. Docker matches with
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
# `/` and an unprefixed pattern is anchored at the context root. Every
# depth-independent pattern therefore needs `**/`, or `config/.env` and
# `certs/server.key` still ship while this file reads as solved. Only
# genuinely root-anchored entries go unprefixed. Never transplant these
# into .gitignore, where `**/` is wrong.
#
# Matching is case-sensitive, so secrets use character ranges rather
# than an ALL-CAPS twin, which would still miss `Server.Key`.
#
# Extend with this repo's own host-built artifacts, written anchored:
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
# Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
# KNOWN GAP: a repo running agents in subdirectories still ships
# `services/api/.claude/` and must add its own anchored entry.
.claude
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Re-include a committed template with a negation if the
# build needs one: `!docs/example.env`.
**/*.[eE][nN][vV]
**/.[eE][nN][vV].*
**/.[eE][nN][vV][rR][cC]
# Private keys and the bundles carrying them. Public certificates
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
**/*.[pP][eE][mM]
**/*.[kK][eE][yY]
**/*.[pP]12
**/*.[pP][fF][xX]
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in.
**/node_modules
# OS metadata.
**/.DS_Store
**/Thumbs.db
# Editor state: never a build input, and it churns COPY.
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea
**/.vscode
**/*.sublime-*
# This repository's host-built artifacts: `make build` writes bin/, `go
# build` in the root the binaries, `make run` log.txt, `make asupdate` its
# .tmp file; test binaries, coverage profiles and source archives.
/bin
/routewatch
/asinfo-gen
/streamdumper
/log.txt
/out
/pkg/asinfo/asdata.json
**/*.tar.zst
**/*.test
**/*.out
**/*.tmp
# Local databases. The image carries a source archive of the whole build
# context, so these would otherwise ship inside it.
**/*.db
**/*.db-journal
**/*.db-wal
# A local Go workspace points at directories outside the build context.
/go.work
/go.work.sum
+3
View File
@@ -10,3 +10,6 @@ insert_final_newline = true
[Makefile] [Makefile]
indent_style = tab indent_style = tab
[*.go]
indent_style = tab
+1 -1
View File
@@ -4,6 +4,6 @@ jobs:
check: check:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
# actions/checkout v4.2.2, 2026-02-28 # actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- run: script/cibuild - run: script/cibuild
+63 -27
View File
@@ -1,41 +1,77 @@
# Binaries for programs and plugins # OS
.DS_Store
Thumbs.db
# Editors
*.swp
*.swo
*~
*.bak
.idea/
.vscode/
*.sublime-*
# Agent scratch (worktrees of this repo, created and destroyed by
# in-flight tooling). Unanchored: .gitignore patterns already match at
# every depth, so no prefix is wanted here. This is not a .dockerignore
# entry and must not be given a `**/` prefix on the way into one.
.claude/
# Node
node_modules/
# Secrets. Unanchored like every entry above, so each matches at every
# depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, kept after the canonical content above.
# Go: logs, test binaries, coverage profiles, and the binaries `make build`
# and `go build` in the repository root write.
*.log
*.out
*.test
*.exe *.exe
*.exe~ *.exe~
*.dll *.dll
*.so *.so
*.dylib *.dylib
/bin/ /bin/
/routewatch
.DS_Store /asinfo-gen
log.txt /streamdumper
# Test binary, built with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# Dependency directories (remove the comment below to include it)
# vendor/
# Go workspace file
go.work go.work
go.work.sum go.work.sum
# env file # Local databases, temporary files and debug output.
.env
# Database files
*.db *.db
*.db-journal *.db-journal
*.db-wal *.db-wal
# Temporary files
*.tmp *.tmp
# Raw AS data (we only commit the gzipped version)
pkg/asinfo/asdata.json
# Debug output files
out
log.txt log.txt
out
# Raw AS data (only the gzipped version is committed).
pkg/asinfo/asdata.json
+87 -80
View File
@@ -1,92 +1,99 @@
version: "2" version: "2"
# Config schema uses the golangci-lint v2 layout (settings live under
# linters.settings, not top-level linters-settings) so that the
# thresholds below are actually applied by golangci-lint >= v2.
run: run:
go: "1.24" timeout: 5m
tests: false modules-download-mode: readonly
linters: linters:
default: all
enable: enable:
# Additional linters requested # Successor to the deprecated gomodguard. Named explicitly, rather than
- testifylint # Checks usage of github.com/stretchr/testify # left to `default: all`, because it carries the module policy below.
- usetesting # usetesting is an analyzer that detects using os.Setenv instead of t.Setenv since Go 1.17 - gomodguard_v2
# - tagliatelle # Disabled: we need snake_case for external API compatibility disable:
- nlreturn # nlreturn checks for a new line before return and branch statements # Genuinely incompatible with project patterns
- nilnil # Checks that there is no simultaneous return of nil error and an invalid value - exhaustruct # Requires all struct fields
- nestif # Reports deeply nested if statements - exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
- mnd # An analyzer to detect magic numbers - godot # Requires comments to end with periods
- lll # Reports long lines - wrapcheck # Too verbose for internal packages
- intrange # intrange is a linter to find places where for loops could make use of an integer range - varnamelen # Short names like db, id are idiomatic Go
- gochecknoglobals # Check that no global variables exist # Deprecated: the warning is attached to the old name, so it is
# silenced by disabling that name, not by enabling the successor.
# Default/existing linters that are commonly useful - wsl # Deprecated, replaced by wsl_v5
- govet - gomodguard # Deprecated, replaced by gomodguard_v2
- errcheck settings:
- staticcheck
- unused
- ineffassign
- misspell
- revive
- gosec
- unconvert
- unparam
linters-settings:
lll: lll:
line-length: 120 line-length: 88
funlen:
nestif: lines: 80
min-complexity: 4 statements: 50
cyclop:
nlreturn: max-complexity: 15
block-size: 2 dupl:
threshold: 100
revive: depguard:
# Test-support code must not be compiled into the shipped binary. A
# test-support package exists to hand a test privileges the program
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name
# ends in `test`, are where that code belongs, and are exempt.
#
# The deny list below is the one part of this file a repository is
# expected to extend, and the only part it may. depguard matches an
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support
# packages have to be named here one at a time, by full import path,
# under a module path that differs from repository to repository. Add
# them; change nothing else.
rules: rules:
- name: var-naming test-support:
arguments: list-mode: lax
- [] files:
- [] - "$all"
- "upperCaseConst=true" - "!$test"
- "!**/*test/**"
tagliatelle: deny:
case: - pkg: net/http/httptest
rules: desc: >-
json: snake Test-support code belongs in test files and in packages whose
yaml: snake directory name ends in test, not in the shipped binary.
xml: snake # Only decisions already recorded in the Go package defaults are
bson: snake # listed here. Every entry matches the module path exactly.
gomodguard_v2:
testifylint: blocked:
enable-all: true - module: github.com/rs/zerolog
recommendations:
usetesting: {} - log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues: issues:
max-issues-per-linter: 0 max-issues-per-linter: 0
max-same-issues: 0 max-same-issues: 0
exclude-rules:
# Exclude unused parameter warnings for cobra command signatures
- text: "parameter '(args|cmd)' seems to be unused"
linters:
- revive
# Allow ALL_CAPS constant names
- text: "don't use ALL_CAPS in Go names"
linters:
- revive
# Allow snake_case JSON tags for external API compatibility
- path: "internal/types/ris.go"
linters:
- tagliatelle
# Allow snake_case JSON tags for database models
- path: "internal/database/models.go"
linters:
- tagliatelle
# Allow generic package name for types that define data structures
- path: "internal/types/"
text: "avoid meaningless package names"
linters:
- revive
View File
File renamed without changes.
+50 -24
View File
@@ -1,22 +1,37 @@
# Lint stage — fast feedback on formatting and lint issues. # Lint phase, built alone by script/lint. The linter is invoked directly
# The golangci-lint image bundles Go, gcc and make, so it can run go vet on # rather than through `make lint`, which is itself a docker build and
# the CGO sqlite package and golangci-lint without extra installs. # would recurse into a daemon that does not exist in a build step. The
# golangci/golangci-lint:v2.7.2 (Go 1.25.5), 2026-09-21 # image is Debian-based and has Go and gcc, so it can check the cgo
FROM golangci/golangci-lint@sha256:5d6d5c70a61f1356adfd9dd6316ce286799fefc9d743421356ff1b00842368ba AS lint # SQLite package without extra installs.
# golangci/golangci-lint:v2.14.0 (Go 1.27.0), 2026-10-14
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
RUN golangci-lint run --config .golangci.yml ./...
RUN make fmt-check # Test phase, built alone by script/test. -race needs cgo and so a C
RUN make lint # compiler, which the Debian Go image ships. -short skips the live RIPE RIS
# feed test, so the suite needs no network. The rerun with -v only shows
# details: the build fails however it ends, because the first run failed.
# golang:1.24-bookworm, 2026-09-21
FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -short -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -short -timeout 90s -race -v ./...; exit 1; }
# Build stage # Build stage. Nothing is wanted from the lint and test phases; the copies
# are what make BuildKit build them first, so this stage cannot run unless
# lint and test passed.
# golang:1.24-bookworm, 2026-09-21 # golang:1.24-bookworm, 2026-09-21
FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS builder FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
# Install build dependencies (zstd for archive, gcc for CGO/sqlite3) # Install build dependencies (zstd for archive, gcc for CGO/sqlite3)
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -25,10 +40,10 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libc6-dev \ libc6-dev \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
WORKDIR /src # A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
# Force BuildKit to run the lint stage before compiling or testing. WORKDIR /src
COPY --from=lint /src/go.sum /dev/null
# Copy everything # Copy everything
COPY . . COPY . .
@@ -36,12 +51,23 @@ COPY . .
# Vendor dependencies (must be after copying source) # Vendor dependencies (must be after copying source)
RUN go mod download && go mod vendor RUN go mod download && go mod vendor
# Run the test suite in the build stage: -race needs cgo and the C compiler # Build the binary with CGO enabled (required for sqlite3). The version the
# installed above. The suite is offline (the live-feed test is opt-in). # page footer shows is the VERSION build argument when one is given, otherwise
RUN make test # `git describe --tags --always` of the .git in the build context (git comes
# with this image): the tag on a tagged commit, tag-N-gHASH after one, the
# Build the binary with CGO enabled (required for sqlite3) # short commit when no tag is reachable. A context that carries .git and still
RUN CGO_ENABLED=1 GOOS=linux go build -o /routewatch ./cmd/routewatch # yields no version fails the build. The footer links to the full commit.
ARG VERSION
RUN version="${VERSION:-$(git describe --tags --always || echo unknown)}"; \
if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
[ "$version" = unknown ]; }; then \
echo "no version could be derived although the build context carries .git" >&2; \
exit 1; \
fi; \
CGO_ENABLED=1 GOOS=linux go build -o /routewatch -ldflags "\
-X git.eeqj.de/sneak/routewatch/internal/version.GitRevision=$(git rev-parse --verify HEAD || echo unknown) \
-X git.eeqj.de/sneak/routewatch/internal/version.GitRevisionShort=$version" \
./cmd/routewatch
# Create source archive with vendored dependencies # Create source archive with vendored dependencies
RUN tar --zstd -cf /routewatch-source.tar.zst \ RUN tar --zstd -cf /routewatch-source.tar.zst \
@@ -79,8 +105,8 @@ RUN chown -R routewatch:routewatch /app
ENV XDG_DATA_HOME=/var/lib ENV XDG_DATA_HOME=/var/lib
# Cap the Go heap at 1.5 GiB so the runtime collects harder before the # Cap the Go heap at 1.5 GiB so the runtime collects harder before the
# container's memory limit is reached. runuser preserves this the way it does # container's memory limit is reached. setpriv in the entrypoint preserves this
# XDG_DATA_HOME above. # the way it does XDG_DATA_HOME above.
ENV GOMEMLIMIT=1536MiB ENV GOMEMLIMIT=1536MiB
# Cap glibc's malloc arenas. The SQLite C library allocates and frees millions # Cap glibc's malloc arenas. The SQLite C library allocates and frees millions
@@ -96,8 +122,8 @@ EXPOSE 8080
COPY ./entrypoint.sh /entrypoint.sh COPY ./entrypoint.sh /entrypoint.sh
# Health check using the health endpoint # Health check using the health endpoint, on the port PORT names
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD curl -sf http://localhost:8080/.well-known/healthcheck.json || exit 1 CMD curl -sf "http://localhost:${PORT:-8080}/.well-known/healthcheck.json" || exit 1
ENTRYPOINT ["/bin/bash", "/entrypoint.sh" ] ENTRYPOINT ["/bin/bash", "/entrypoint.sh" ]
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Jeffrey Paul <sneak@sneak.berlin>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+1 -1
View File
@@ -2,7 +2,7 @@ export DEBUG = routewatch
# Git revision for version embedding # Git revision for version embedding
GIT_REVISION := $(shell git rev-parse HEAD 2>/dev/null || echo "unknown") GIT_REVISION := $(shell git rev-parse HEAD 2>/dev/null || echo "unknown")
GIT_REVISION_SHORT := $(shell git rev-parse --short HEAD 2>/dev/null || echo "unknown") GIT_REVISION_SHORT := $(shell git describe --tags --always 2>/dev/null || echo "unknown")
VERSION_PKG := git.eeqj.de/sneak/routewatch/internal/version VERSION_PKG := git.eeqj.de/sneak/routewatch/internal/version
LDFLAGS := -X $(VERSION_PKG).GitRevision=$(GIT_REVISION) -X $(VERSION_PKG).GitRevisionShort=$(GIT_REVISION_SHORT) LDFLAGS := -X $(VERSION_PKG).GitRevision=$(GIT_REVISION) -X $(VERSION_PKG).GitRevisionShort=$(GIT_REVISION_SHORT)
+40 -7
View File
@@ -1,6 +1,9 @@
# RouteWatch # RouteWatch
RouteWatch is a real-time BGP routing table monitor that streams BGP UPDATE messages from the RIPE RIS Live service, maintains a live routing table in SQLite, and provides HTTP APIs for querying routing information. RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing
table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
service, maintains a live routing table in SQLite, and provides HTTP APIs for
querying routing information.
## Features ## Features
@@ -139,7 +142,10 @@ routewatch/
- **Backpressure**: Probabilistic message dropping when queues exceed 50% capacity - **Backpressure**: Probabilistic message dropping when queues exceed 50% capacity
- **Graceful Shutdown**: 60-second timeout, flushes all pending batches - **Graceful Shutdown**: 60-second timeout, flushes all pending batches
- **Reconnection**: Exponential backoff (5s-320s) with reset after 30s of stable connection - **Reconnection**: Exponential backoff (5s-320s) with reset after 30s of stable connection
- **IPv4 Optimization**: IP ranges stored as uint32 for O(1) lookups - **IP Lookup**: the most specific live route is found by looking up the
address's prefix at each mask length, longest first, in the prefix index
(at most 33 lookups for IPv4, 129 for IPv6); prefixes are stored in the
text form Go's `net/netip` prints
### Database Schema ### Database Schema
@@ -151,7 +157,7 @@ prefixes_v6(id, prefix, mask_length, first_seen, last_seen)
-- Live routing tables (one per IP version) -- Live routing tables (one per IP version)
live_routes_v4(id, prefix, mask_length, origin_asn, peer_ip, as_path, live_routes_v4(id, prefix, mask_length, origin_asn, peer_ip, as_path,
next_hop, last_updated, v4_ip_start, v4_ip_end) next_hop, last_updated)
live_routes_v6(id, prefix, mask_length, origin_asn, peer_ip, as_path, live_routes_v6(id, prefix, mask_length, origin_asn, peer_ip, as_path,
next_hop, last_updated) next_hop, last_updated)
@@ -166,14 +172,21 @@ Configuration is handled via environment variables and OS-specific paths:
| Variable | Default | Description | | Variable | Default | Description |
|----------|----------|-------------| |----------|----------|-------------|
| `PORT` | `8080` | HTTP server port | | `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 |
| `DEBUG` | (empty) | Set to `routewatch` for debug logging | | `DEBUG` | (empty) | Set to `routewatch` for debug logging |
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory | | `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap; see Memory | | `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory |
A variable that is set to an invalid value stops the start with an error and a
non-zero exit. An empty variable counts as unset.
State directory (database location): State directory (database location):
- macOS: `~/Library/Application Support/routewatch/` - macOS: `~/Library/Application Support/routewatch/`
- Linux: `/var/lib/routewatch/` or `~/.local/share/routewatch/` - Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root,
otherwise `$XDG_DATA_HOME/berlin.sneak.app.routewatch/` (with `XDG_DATA_HOME`
unset, `~/.local/share/berlin.sneak.app.routewatch/`). In the Docker image
this is `/var/lib/berlin.sneak.app.routewatch/`.
## Memory ## Memory
@@ -220,6 +233,22 @@ What happens at each limit:
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
with the goroutine count and Go memory figures. with the goroutine count and Go memory figures.
## Running under upaas
What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
- Container port: `8080`.
- Volume: one, at container path `/var/lib/berlin.sneak.app.routewatch`.
- Environment: nothing is required. Leave `XDG_DATA_HOME`, `GOMEMLIMIT` and
`MALLOC_ARENA_MAX` at the image's values. `DEBUG=routewatch` is optional and
adds the `System stats` memory line to the log.
- Memory Limit: `5g`, the 5 GiB limit from Memory above. upaas sets no swap
limit, so on a host with swap Docker allows the same amount of swap again.
- Health check: the image's `HEALTHCHECK` requests
`/.well-known/healthcheck.json` on the container port. upaas reads the
container's health 60 seconds after a deploy and fails the deploy unless it
is `healthy`.
## Development ## Development
```bash ```bash
@@ -274,4 +303,8 @@ invoke `go test` directly without `-short`:
## License ## License
See LICENSE file. MIT. See [`LICENSE`](LICENSE).
## Author
[@sneak](https://sneak.berlin)
+355 -84
View File
@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-07-06 last_modified: 2026-10-04
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -60,17 +60,28 @@ style conventions are in separate documents:
prerequisite since nvm requires bash. yarn is then pinned via prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts"; `corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root and runs `docker build .`; the Gitea workflow calls it. Four further repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
scripts are our own extensions to the standard: `script/check` runs with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is `script/bootstrap` first**, because the workflow checks out the repo and runs
what the git pre-commit hook runs, and it calls `script/check`; nothing else, while `script/fmt-check` runs the formatter on the host: on a
`script/install-precommit` installs the git pre-commit hook (the `make hooks` pristine checkout with nothing installed the run dies there, after the
target shims to it); and `script/projectname` (literally that filename) simply containerised gates have passed. **The bootstrap alone is not enough**:
outputs the project's name. Scripts that need the name call `script/bootstrap` installs node and yarn under nvm and leaves neither on the
`script/projectname` — e.g. `script/docker` assembles its image tag from it — `PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
so those scripts stay byte-identical across all repos. Repo-type-specific entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in source nvm for the pinned node version before invoking it, exactly as
`script/precommit`, not in the hook itself. Model scripts are at `script/bootstrap`'s own install step does. A runner carrying nothing but
docker and git then gets through `script/check`. Four further scripts are our
own extensions to the standard: `script/check` runs `script/test`,
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/projectname` (literally that filename) simply outputs the project's
name. Scripts that need the name call `script/projectname` — e.g.
`script/docker` assembles its image tag from it — so those scripts stay
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the must document the provided scripts in an **Entrypoints** section (see the
README requirements below). README requirements below).
@@ -89,87 +100,198 @@ style conventions are in separate documents:
contributor should be able to understand the entire development workflow by contributor should be able to understand the entire development workflow by
reading the Makefile. reading the Makefile.
- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check` - Every repo should have a `Dockerfile`, and it carries the repo's gates: a
as a build step so the build fails if the branch is not green. For non-server `lint` phase and a `test` phase, with the final stage depending on both so the
repos, the Dockerfile should bring up a development environment and run image cannot be built unless they pass. For non-server repos the final stage
`make check`. For server repos, `make check` should run as an early build brings up a development environment; for server repos it is the runtime image.
stage before the final image is assembled. Dockerfiles install development The gate phases and the build stage start from their pinned base images and
prerequisites by running `script/bootstrap` rather than duplicating installs install what those images lack either inline, as the canonical Go `Dockerfile`
inline; COPY `script/` and the dependency manifests (`package.json` + below does for `git`, or by running `script/bootstrap`, as the `prompts`
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap repo's own `Dockerfile` does for its yarn packages. The development
layer stays cached until dependencies change. environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go - **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
repos use a multistage build where linting runs in an independent stage based no separate lint file. `script/lint` and `script/test` each build one phase
on the `golangci/golangci-lint` image (pinned by hash). This stage runs and nothing else:
`make fmt-check` and `make lint` before the full build begins. The build stage
then declares an explicit dependency on the lint stage via
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
linting before proceeding to compilation and tests. This ensures lint failures
surface in seconds rather than minutes, without blocking on dependency
download or compilation in the build stage.
The standard pattern for a Go repo Dockerfile is: ```sh
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test -t "$(script/projectname)-test" .
```
**A stage that is not the last one in the file is built only when the final
stage's chain depends on it, or when `--target` names it.** That is why the
two gates are always invoked by name here, and why the final stage carries a
`COPY --from=` of a harmless file from each of them: without that edge a
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
themselves a `docker build` and would recurse into a daemon that does not
exist in a build step. Formatting is the exception and stays on the host:
`script/fmt` writes the working tree, and `script/fmt-check` is its
read-only twin.
**No lint verdict may come from a host invocation of the linter.** On a
shared host golangci-lint reads a result cache keyed on file content rather
than location, so a second checkout of the same content is served the first
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`:
```dockerfile ```dockerfile
# Lint stage — fast feedback on formatting and lint issues # Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD # golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
RUN make fmt-check RUN golangci-lint run --config .golangci.yml ./...
RUN make lint
# Build stage # Test phase. -race needs cgo and so a C compiler, which the Debian Go
# golang:1.x-alpine, YYYY-MM-DD # image ships and the alpine one does not.
FROM golang@sha256:... AS builder # golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
WORKDIR /src WORKDIR /src
# Force BuildKit to run the lint stage before proceeding
COPY --from=lint /src/go.sum /dev/null
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
RUN make test RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
ARG VERSION=dev # Build stage. Nothing is wanted from either phase above; the copies
RUN CGO_ENABLED=0 go build -trimpath \ # are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \ -ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/ -o /app ./cmd/app/
# Runtime stage # Runtime stage, and the last one
FROM alpine@sha256:... FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"] ENTRYPOINT ["app"]
``` ```
Key points: Key points:
- The lint stage uses the `golangci/golangci-lint` image directly (it - The lint phase uses the `golangci/golangci-lint` image directly (it has
includes both Go and the linter), so there is no need to install the both Go and the linter), so nothing needs installing.
linter separately. - `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates purpose is the ordering edge. BuildKit runs stages in parallel by default,
a stage dependency. BuildKit runs stages in parallel by default; without and a stage nothing depends on is not built at all, so without these two
this line, the build stage would not wait for lint to finish and a lint lines a red gate would not fail the build.
failure might not fail the overall build. - Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- If the project uses `//go:embed` directives that reference build artifacts - If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint stage must (e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example: create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`. `RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
The lint stage should not depend on the actual build output — it exists to - If the project requires CGO or system libraries for linting, install them
fail fast. in the lint phase. The `golangci/golangci-lint` image is Debian-based and
- If the project requires CGO or system libraries for linting (e.g. has no `apk`, so install with `apt-get` under the Debian package name
`vips-dev`), install them in the lint stage with `apk add`. (`libvips-dev`, where alpine says `vips-dev`), and delete the package
- The build stage runs `make test` after compilation setup. Tests run in the lists in the same `RUN`, so the layer does not keep them:
build stage, not the lint stage, because they may require compiled
artifacts or heavier dependencies. ```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` (which runs `docker build .`) on push. Since the runs `script/cibuild` on push, and checks out the repo as its only other step.
Dockerfile already runs `make check`, a successful build implies all checks That script bootstraps, runs the gate phases, and then builds the image, so a
pass. successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for - Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -189,14 +311,21 @@ style conventions are in separate documents:
module under test to verify it compiles/parses. There is no excuse for module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op. `make test` to be a no-op.
- `make test` must complete in under 20 seconds. Add a 30-second timeout in the - `make test` must complete in under 60 seconds. That is the hard cap, and a
Makefile. suite that exceeds it fails. Under 20 seconds is the target. A suite between
20 and 60 seconds is still green, but the overage must be filed as an
improvement bug against that repo. Add a 90-second timeout to the test
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
hard cap so that it catches a genuinely hung test rather than a merely slow
one.
- **`make test` should use the conditional verbose rerun pattern.** Run tests - **The test command should use the conditional verbose rerun pattern.** Run
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to tests without `-v` (verbose) first. If tests fail, automatically rerun with
show full output. This keeps CI logs and `docker build` output clean on `-v` to show full output. This keeps CI logs and `docker build` output clean
success (just package/suite summaries) while providing full diagnostic detail on success (just package/suite summaries) while providing full diagnostic
on failure (every test case, every assertion). The general shell pattern: detail on failure (every test case, every assertion). The command lives in the
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
Makefile form below is the same pattern for any repo-local invocation:
```makefile ```makefile
test: test:
@@ -209,11 +338,26 @@ style conventions are in separate documents:
```makefile ```makefile
test: test:
@go test -timeout 30s -race -cover ./... || \ @go test -count=1 -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \ { echo "--- Rerunning with -v for details ---"; \
go test -timeout 30s -race -v ./...; exit 1; } go test -count=1 -timeout 90s -race -v ./...; exit 1; }
``` ```
`-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite
and no recompilation.
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. That matters on a developer's machine, where this
target runs and the directory lasts from one run to the next. The `test`
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example: Python example:
```makefile ```makefile
@@ -239,10 +383,84 @@ style conventions are in separate documents:
must be in `.gitignore`. No exceptions. must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`), - `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`. editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
Fetch the standard `.gitignore` from language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
a new repo. setting up a new repo. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
across unmodified leaves secrets in the build context.** Docker matches with
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
therefore excludes only the copies at the repository root, while `config/.env`
and `certs/server.key` still reach the context and can land in an image layer
— which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
also catches something the build needs, re-include it with a negation
(`!docs/example.env`); deleting the pattern reopens the exposure for every
other file it covers. Fetch the standard `.dockerignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
it with the repo's own artifacts.
- **In-repo agent scratch belongs in both files, written to each file's own
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
additional checkout of the repo — so under `COPY . .` the build context
inflates by a multiple of the repo and another session's unreviewed work can
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
prefix, because the prefixed form would also delete any nested directory of
that name from the build. Anchoring carries a known gap that the canonical
`.dockerignore` states in its own comment, since consuming repos receive the
file and not the tracker: the directory is created in the agent's working
directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there.
- **A plain `docker build .` of a clone stamps the version that
`git describe --tags --always` gives**, derived from the `.git` in the build
context as the canonical `Dockerfile` above shows. Without its failure check,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
and the build would still exit 0. `script/docker` and `script/cibuild` pass
the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh
# Own line: a failing command substitution inside an argument does not
# trip `set -e`, so the inline form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$(script/projectname)" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
than failing, and the `[ -n "$version" ]` line is the single place the
fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
a probe image that does `COPY . .`, and list what actually landed
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
size is not a substitute: a nested secret is a few bytes, and BuildKit
transfers only the delta from the previous build.
- **No build artifacts in version control.** Code-derived data (compiled - **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the bundles, minified output, generated assets) must never be committed to the
@@ -258,9 +476,56 @@ style conventions are in separate documents:
- Make all changes on a feature branch. You can do whatever you want on a - Make all changes on a feature branch. You can do whatever you want on a
feature branch. feature branch.
- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only - `.golangci.yml` is standardized. The vendored copy in a consuming repo must
manually by the user. Fetch from _NEVER_ be modified by an agent: fetch it from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`. `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
byte-identical, so that no repo can quietly loosen its own linting. Linter
configuration changes are made to the canonical copy in the `prompts` repo and
reach consuming repos by re-vendoring; an agent may open a PR against
canonical, which only the user merges. One list is exempt from byte-identity,
because it cannot be written once for every repo: the `deny` list of the
`test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
directive must not name a newer Go minor version than the one golangci-lint
was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
`PATH` only, so on an already-provisioned machine the pin is inert and a
version bump is a silent no-op — while the Dockerfile, installing into a clean
image, gets the pinned version, so a local `make check` and `make docker` can
disagree about what the tool even is. The canonical form:
- compares the installed version against the pin over the **whole** version
token; a parser that stops at the first `-` reports `2.12.2` for a host
running `2.12.2-rc1` and skips the install;
- treats absent, non-zero, empty or unrecognised `--version` output as a
mismatch, so the failure direction is a redundant install and never a
skipped one;
- after installing, re-resolves the binary the way callers do — `hash -r`,
then through `PATH`, not through the directory the installer wrote to —
and fails naming the resolved path, since an install that a shadowing
binary hides succeeds while changing nothing any caller sees;
- is actually called, and prints the version on both success paths: a
function defined and never invoked has the same exit status and the same
empty output as one that worked.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference - When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD). with the version and date (YYYY-MM-DD).
@@ -374,12 +639,14 @@ style conventions are in separate documents:
settings. settings.
- Avoid putting files in the repo root unless necessary. Root should contain - Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `Makefile`, `Dockerfile`, only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and `Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
language-specific config). Everything else goes in a subdirectory. Canonical and language-specific config). Everything else goes in a subdirectory.
subdirectory names: Canonical subdirectory names:
- `bin/` — executable scripts and tools - `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints - `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in
`cmd/`
- `configs/` — configuration templates and examples - `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform) - `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root) - `docs/` — documentation and markdown (README.md stays in root)
@@ -406,3 +673,7 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml` - Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore` - JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml` - Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
+86 -12
View File
@@ -10,19 +10,95 @@
# Status # Status
pre-1.0. No git tags. Runs in production-style Docker deployment, but pre-1.0. No git tags. The Docker build runs the format check, the linter
the policy compliance branch (repo-policies-compliance, make check and the tests, and the Gitea workflow runs that build on every push. The
passing, clean tree) is unmerged to main and the CI workflow is missing. image sets memory ceilings for a 5 GiB container (README "Memory") and the
README says how to run it under upaas (README "Running under upaas"). A
35-hour run of `3898daa` on the live feed peaked at about 1 GiB, without a
container memory limit.
# Next Step # Next Step
Merge repo-policies-compliance into main (3 commits: policy files and `next` waits for sneak to merge it to `main` through
.gitignore, Makefile targets fmt-check/check/docker/hooks, gofmt pass), https://git.eeqj.de/sneak/routewatch/pulls/6. After that, setting
then add .gitea/workflows/check.yml as a small follow-up commit so CI routewatch up under upaas on fsn1app1 and deploying it are his
runs make check on main. (https://git.eeqj.de/sneak/routewatch/issues/31), and so is the run under a
real 5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3).
# Completed Steps # Completed Steps
- 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory,
seeded at startup and adjusted on every live-route write, so a request no
longer reads every live route (closes #30)
- 2026-10-03: a new database is created with `auto_vacuum` set to
incremental, through the connection string so it is set before the file
is first written, and each periodic incremental vacuum now returns up to
1000 free pages; the late `PRAGMA auto_vacuum` in `Initialize`, which
SQLite ignored, is gone (closes #43)
- 2026-10-03: looking up an IP address no longer reads every IPv6 route:
both families find the most specific live route with at most 33 or 129
lookups on the prefix index, and the IPv4 range columns are gone. Prefixes
from the feed are stored in one text form, so an IPv6 withdrawal, which the
feed sends uncompressed, now removes its route (closes #48)
- 2026-10-02: a plain `docker build .` stamps the commit's tag or short
commit (`git describe --tags --always`) into the page footer instead of
`unknown`: `.dockerignore` sends `.git` without `.git/config`, a `VERSION`
build argument takes precedence, and `make build` stamps the same value
(closes #46)
- 2026-09-29: the entrypoint creates the data directory if it is missing and
stops the start if a step fails; README "Running under upaas" no longer
asks for the host directory to be created first (closes #42)
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local
databases and `.env` out of the Docker build context, and so out of the
source archive in the image (closes #39)
- 2026-09-29: README first line names the MIT license and the author;
the License section now says MIT and links `LICENSE`, and an Author
section was added; this file brought up to date (closes #38)
- 2026-09-29: MIT `LICENSE` (closes #1)
- 2026-09-28: stopping the daemon while the feed is flowing no longer
panics with "send on closed channel": the read loop checks for a stop
just before handing a message to the handler queues, and a second
`Stop` no longer closes the queues again (closes #34)
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the
stop signal: the entrypoint switches to the `routewatch` user with
`setpriv` instead of `runuser`, so the daemon receives the signal itself
and gets the whole wait `docker stop` allows, up to its own 60-second
limit (closes #33)
- 2026-09-28: ready to run under upaas: a set but invalid `PORT`,
`XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start, the health
check follows `PORT`, README "Running under upaas" section (closes
#31)
- 2026-09-22: realtime in-memory database statistics: counts seeded at
startup and adjusted on every write, oldest/newest route timestamps via
index-end lookups; `/api/v1/stats` no longer scans the tables (closes
#27)
- 2026-09-21: batch writes take the write lock when their transaction
begins (`_txlock=immediate`), so they wait out a WAL checkpoint instead
of failing with "database is locked" (closes #25)
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps glibc malloc arenas,
so memory outside the Go runtime no longer grows with the core count
(closes #23)
- 2026-09-21: `GOMEMLIMIT=1536MiB` in the image; README Memory section
with the memory budget and the 5 GiB container limit (closes #13)
- 2026-09-21: the four handler queues hold at most 20,000 messages each,
down from 100,000 (closes #11)
- 2026-09-21: two goroutine leaks fixed: the stats handlers after a
timeout and the streamer's tickers on every reconnect (closes #12)
- 2026-09-21: parsed RIS messages no longer keep the unused `Community`
and `Raw` fields (closes #9)
- 2026-09-21: `.editorconfig`, and a Gitea workflow that runs
`script/cibuild` on every push (closes #14)
- 2026-09-21: the peering handler's AS-path map holds at most 500,000
paths and is swapped for an empty one every 30 seconds instead of copied
(closes #10)
- 2026-09-21: SQLite memory bounded across the whole connection pool: a
64 MiB page cache on each connection, 1 GiB soft and 1.5 GiB hard heap
limits (closes #8)
- 2026-09-21: the Docker build runs the format check, the linter and the
tests, linting in a separate stage on a golangci-lint image pinned by
digest (closes #5)
- 2026-09-21: `make test` skips the live-network feed test (`-short`), so
`make check` no longer depends on the network (closes #2)
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, - 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section Makefile shims, README Entrypoints section
- 2026-02-22: repo policy compliance: required policy files, .gitignore - 2026-02-22: repo policy compliance: required policy files, .gitignore
@@ -41,8 +117,6 @@ runs make check on main.
# Future Steps # Future Steps
- Verify main is green after the merge: make check locally and the new - Production memory under 5 GiB: whether to test under a real 5 GiB
CI workflow passing container limit on fsn1app1 is open for sneak
- Review stale remote branches fix-min-time-calculation and (https://git.eeqj.de/sneak/routewatch/issues/3)
optimize-sqlite-settings: land or delete
- Clean up the tmp/ directory at the repo root: gitignore or remove
+14 -1
View File
@@ -1,7 +1,20 @@
#!/bin/bash #!/bin/bash
set -euo pipefail
# glibc silently ignores a malformed MALLOC_ARENA_MAX, so refuse it here.
if [[ -n "${MALLOC_ARENA_MAX:-}" && ! "$MALLOC_ARENA_MAX" =~ ^[1-9][0-9]*$ ]]; then
echo "MALLOC_ARENA_MAX must be a positive whole number, got '$MALLOC_ARENA_MAX'" >&2
exit 1
fi
# Give the data directory to the routewatch user before the daemon starts,
# whether it is missing, an empty root-owned mount, or holds another uid's files.
mkdir -p /var/lib/berlin.sneak.app.routewatch
cd /var/lib/berlin.sneak.app.routewatch cd /var/lib/berlin.sneak.app.routewatch
chown -R routewatch:routewatch . chown -R routewatch:routewatch .
chmod 700 . chmod 700 .
exec runuser -u routewatch -- /app/routewatch # setpriv replaces itself with the daemon, so the daemon receives the stop
# signal directly. runuser would stay in between and kill the daemon 2 seconds
# after passing the signal on.
exec setpriv --reuid=routewatch --regid=routewatch --init-groups -- /app/routewatch
+40 -1
View File
@@ -6,6 +6,7 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"runtime" "runtime"
"strconv"
"time" "time"
) )
@@ -18,6 +19,12 @@ const (
// defaultRouteExpirationMinutes is the default route expiration timeout in minutes // defaultRouteExpirationMinutes is the default route expiration timeout in minutes
defaultRouteExpirationMinutes = 5 defaultRouteExpirationMinutes = 5
// defaultPort is the HTTP port used when PORT is not set
defaultPort = 8080
// maxPort is the highest TCP port number
maxPort = 65535
) )
// Config holds configuration for the entire application // Config holds configuration for the entire application
@@ -25,6 +32,9 @@ type Config struct {
// StateDir is the directory for all application state (database, snapshots) // StateDir is the directory for all application state (database, snapshots)
StateDir string StateDir string
// Port is the TCP port the HTTP server listens on
Port int
// MaxRuntime is the maximum runtime (0 = run forever) // MaxRuntime is the maximum runtime (0 = run forever)
MaxRuntime time.Duration MaxRuntime time.Duration
@@ -43,8 +53,14 @@ func New() (*Config, error) {
return nil, fmt.Errorf("failed to determine state directory: %w", err) return nil, fmt.Errorf("failed to determine state directory: %w", err)
} }
port, err := getPort()
if err != nil {
return nil, err
}
return &Config{ return &Config{
StateDir: stateDir, StateDir: stateDir,
Port: port,
MaxRuntime: 0, // Run forever by default MaxRuntime: 0, // Run forever by default
EnableBatchedDatabaseWrites: true, // Enable batching by default EnableBatchedDatabaseWrites: true, // Enable batching by default
RouteExpirationTimeout: defaultRouteExpirationMinutes * time.Minute, // For active route monitoring RouteExpirationTimeout: defaultRouteExpirationMinutes * time.Minute, // For active route monitoring
@@ -69,13 +85,20 @@ func getStateDirectory() (string, error) {
return filepath.Join(home, "Library", "Application Support", AppIdentifier), nil return filepath.Join(home, "Library", "Application Support", AppIdentifier), nil
case "linux", "freebsd", "openbsd", "netbsd": case "linux", "freebsd", "openbsd", "netbsd":
// The XDG spec requires an absolute path; a relative one would put
// the database somewhere unexpected.
xdgData := os.Getenv("XDG_DATA_HOME")
if xdgData != "" && !filepath.IsAbs(xdgData) {
return "", fmt.Errorf("XDG_DATA_HOME must be an absolute path, got %q", xdgData)
}
// Unix-like: /var/lib/berlin.sneak.app.routewatch if root, else XDG_DATA_HOME // Unix-like: /var/lib/berlin.sneak.app.routewatch if root, else XDG_DATA_HOME
if os.Geteuid() == 0 { if os.Geteuid() == 0 {
return filepath.Join("/var/lib", AppIdentifier), nil return filepath.Join("/var/lib", AppIdentifier), nil
} }
// Check XDG_DATA_HOME first // Check XDG_DATA_HOME first
if xdgData := os.Getenv("XDG_DATA_HOME"); xdgData != "" { if xdgData != "" {
return filepath.Join(xdgData, AppIdentifier), nil return filepath.Join(xdgData, AppIdentifier), nil
} }
@@ -92,6 +115,22 @@ func getStateDirectory() (string, error) {
} }
} }
// getPort returns the HTTP port from PORT, or defaultPort when PORT is not set
func getPort() (int, error) {
value := os.Getenv("PORT")
if value == "" {
return defaultPort, nil
}
// ParseUint, unlike Atoi, refuses a sign: the health check URL cannot use "+9090"
port, err := strconv.ParseUint(value, 10, 0)
if err != nil || port < 1 || port > maxPort {
return 0, fmt.Errorf("PORT must be a whole number from 1 to %d, got %q", maxPort, value)
}
return int(port), nil
}
// EnsureDirectories creates all necessary directories if they don't exist // EnsureDirectories creates all necessary directories if they don't exist
func (c *Config) EnsureDirectories() error { func (c *Config) EnsureDirectories() error {
// Ensure state directory exists // Ensure state directory exists
+62
View File
@@ -0,0 +1,62 @@
package config
import (
"runtime"
"testing"
)
func TestNewReadsPort(t *testing.T) {
tests := map[string]int{
"": defaultPort,
"1": 1,
"9090": 9090,
"65535": 65535,
}
for value, want := range tests {
t.Run(value, func(t *testing.T) {
t.Setenv("PORT", value)
t.Setenv("XDG_DATA_HOME", "")
cfg, err := New()
if err != nil {
t.Fatalf("New() with PORT=%q: %v", value, err)
}
if cfg.Port != want {
t.Errorf("New() with PORT=%q: Port = %d, want %d", value, cfg.Port, want)
}
})
}
}
func TestNewRefusesInvalidPort(t *testing.T) {
for _, value := range []string{"0", "65536", "-1", "+9090", "http", "80.5"} {
t.Run(value, func(t *testing.T) {
t.Setenv("PORT", value)
t.Setenv("XDG_DATA_HOME", "")
if _, err := New(); err == nil {
t.Errorf("New() with PORT=%q returned no error", value)
}
})
}
}
func TestNewRefusesRelativeXDGDataHome(t *testing.T) {
if runtime.GOOS == "darwin" {
t.Skip("macOS does not read XDG_DATA_HOME")
}
t.Setenv("PORT", "")
t.Setenv("XDG_DATA_HOME", "relative/path")
if _, err := New(); err == nil {
t.Error("New() with a relative XDG_DATA_HOME returned no error")
}
t.Setenv("XDG_DATA_HOME", "/var/lib")
if _, err := New(); err != nil {
t.Errorf("New() with XDG_DATA_HOME=/var/lib: %v", err)
}
}
+203
View File
@@ -0,0 +1,203 @@
package database
import (
"context"
"fmt"
"sync"
)
// liveCounts holds the running row counts and the prefix distribution that the
// stats endpoints report. They are seeded once at startup from the tables and
// then adjusted on every write, so a stats read serves them from memory instead
// of running a query over the tables. The COUNT(*) scans (issue 27) and then
// the prefix distribution query (issue 30) each grew with the database until
// they took the whole request timeout and made /api/v1/stats return 500.
//
// A single mutex guards all fields so the stats reader takes a consistent
// snapshot at one instant and writers, which already run under the database
// write lock, adjust the counts after their transaction commits.
type liveCounts struct {
mu sync.RWMutex
asns int
prefixesV4 int
prefixesV6 int
peerings int
peers int
routesV4 int
routesV6 int
// The prefix distribution: for each mask length, the number of distinct
// prefixes that have at least one live route.
distributionV4 [ipv4Bits + 1]int
distributionV6 [ipv6Bits + 1]int
}
// seed sets every count to the value read from the tables at startup. It runs
// before any writer, so it needs no coordination with the adjust methods.
func (c *liveCounts) seed(asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6 int,
distributionV4, distributionV6 []PrefixDistribution) {
c.mu.Lock()
defer c.mu.Unlock()
c.asns = asns
c.prefixesV4 = prefixesV4
c.prefixesV6 = prefixesV6
c.peerings = peerings
c.peers = peers
c.routesV4 = routesV4
c.routesV6 = routesV6
for _, entry := range distributionV4 {
addAtMaskLength(c.distributionV4[:], entry.MaskLength, entry.Count)
}
for _, entry := range distributionV6 {
addAtMaskLength(c.distributionV6[:], entry.MaskLength, entry.Count)
}
}
// addASNs adds n to the ASN count.
func (c *liveCounts) addASNs(n int) {
c.mu.Lock()
c.asns += n
c.mu.Unlock()
}
// addPrefixes adds to the IPv4 and IPv6 prefix counts.
func (c *liveCounts) addPrefixes(v4, v6 int) {
c.mu.Lock()
c.prefixesV4 += v4
c.prefixesV6 += v6
c.mu.Unlock()
}
// addPeerings adds n to the peering count.
func (c *liveCounts) addPeerings(n int) {
c.mu.Lock()
c.peerings += n
c.mu.Unlock()
}
// addPeers adds n to the BGP peer count.
func (c *liveCounts) addPeers(n int) {
c.mu.Lock()
c.peers += n
c.mu.Unlock()
}
// addRoutes adds to the IPv4 and IPv6 live-route counts. Deletions pass
// negative values.
func (c *liveCounts) addRoutes(v4, v6 int) {
c.mu.Lock()
c.routesV4 += v4
c.routesV6 += v6
c.mu.Unlock()
}
// addToDistribution adds n to the IPv4 and IPv6 prefix distributions once for
// each listed mask length. A write lists the mask lengths of the prefixes it
// gave their first live route with n = 1, and of the prefixes it left with no
// live route with n = -1.
func (c *liveCounts) addToDistribution(maskLengthsV4, maskLengthsV6 []int, n int) {
c.mu.Lock()
defer c.mu.Unlock()
for _, maskLength := range maskLengthsV4 {
addAtMaskLength(c.distributionV4[:], maskLength, n)
}
for _, maskLength := range maskLengthsV6 {
addAtMaskLength(c.distributionV6[:], maskLength, n)
}
}
// addAtMaskLength adds n to counts[maskLength]. A mask length the array has no
// entry for is ignored, so a malformed route cannot crash the daemon.
func addAtMaskLength(counts []int, maskLength, n int) {
if maskLength >= 0 && maskLength < len(counts) {
counts[maskLength] += n
}
}
// fill copies the counts into a Stats, including the derived totals, under a
// single read lock so the reader sees one consistent snapshot.
func (c *liveCounts) fill(s *Stats) {
c.mu.RLock()
defer c.mu.RUnlock()
s.ASNs = c.asns
s.IPv4Prefixes = c.prefixesV4
s.IPv6Prefixes = c.prefixesV6
s.Prefixes = c.prefixesV4 + c.prefixesV6
s.Peerings = c.peerings
s.Peers = c.peers
s.IPv4Routes = c.routesV4
s.IPv6Routes = c.routesV6
s.LiveRoutes = c.routesV4 + c.routesV6
s.IPv4PrefixDistribution = distributionList(c.distributionV4[:])
s.IPv6PrefixDistribution = distributionList(c.distributionV6[:])
}
// distributionList lists the mask lengths that have at least one prefix, in
// ascending order, the way the distribution query returns them.
func distributionList(counts []int) []PrefixDistribution {
var list []PrefixDistribution
for maskLength, count := range counts {
if count > 0 {
list = append(list, PrefixDistribution{MaskLength: maskLength, Count: count})
}
}
return list
}
// countRows returns the number of rows in the named table. It is used only at
// startup to seed the in-memory counters, so a full COUNT(*) is acceptable.
func (d *Database) countRows(ctx context.Context, table string) (int, error) {
var n int
// table is one of a fixed set of literals below, never external input.
if err := d.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM "+table).Scan(&n); err != nil {
return 0, fmt.Errorf("failed to count %s: %w", table, err)
}
return n, nil
}
// seedCounts reads the current row counts and prefix distribution from the
// tables into the in-memory counters. It runs once at startup, before the
// streamer begins writing.
func (d *Database) seedCounts(ctx context.Context) error {
asns, err := d.countRows(ctx, "asns")
if err != nil {
return err
}
prefixesV4, err := d.countRows(ctx, "prefixes_v4")
if err != nil {
return err
}
prefixesV6, err := d.countRows(ctx, "prefixes_v6")
if err != nil {
return err
}
peerings, err := d.countRows(ctx, "peerings")
if err != nil {
return err
}
peers, err := d.countRows(ctx, "bgp_peers")
if err != nil {
return err
}
routesV4, err := d.countRows(ctx, "live_routes_v4")
if err != nil {
return err
}
routesV6, err := d.countRows(ctx, "live_routes_v6")
if err != nil {
return err
}
distributionV4, distributionV6, err := d.GetPrefixDistributionContext(ctx)
if err != nil {
return err
}
d.counts.seed(asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6,
distributionV4, distributionV6)
return nil
}
+511
View File
@@ -0,0 +1,511 @@
package database
import (
"context"
"slices"
"sync"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
"github.com/google/uuid"
)
// mkV4Route builds an IPv4 live route with its mask length taken from the
// prefix.
func mkV4Route(t *testing.T, prefix string, asn int, ts time.Time) *LiveRoute {
t.Helper()
maskLength, err := prefixMaskLength(prefix)
if err != nil {
t.Fatalf("prefixMaskLength(%s): %v", prefix, err)
}
return &LiveRoute{
ID: uuid.New(),
Prefix: prefix,
MaskLength: maskLength,
IPVersion: ipVersionV4,
OriginASN: asn,
PeerIP: "192.0.2.1",
ASPath: []int{asn},
NextHop: "192.0.2.254",
LastUpdated: ts,
}
}
// mkV6Route builds an IPv6 live route.
func mkV6Route(prefix string, asn int, ts time.Time) *LiveRoute {
return &LiveRoute{
ID: uuid.New(),
Prefix: prefix,
MaskLength: 32,
IPVersion: ipVersionV6,
OriginASN: asn,
PeerIP: "2001:db8::1",
ASPath: []int{asn},
NextHop: "2001:db8::ffff",
LastUpdated: ts,
}
}
// TestLiveCountsTrackWritesInRealtime checks that the stats counts start at
// zero, reflect each write the moment it commits (no recompute, no timer), do
// not move when a route is merely re-announced, and drop when a route is
// deleted. These counts are what /api/v1/stats reports; before this change the
// endpoint recomputed them with a COUNT(*) over each table on every request.
func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
empty, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext on empty database: %v", err)
}
if empty.ASNs != 0 || empty.Prefixes != 0 || empty.Peerings != 0 ||
empty.Peers != 0 || empty.LiveRoutes != 0 {
t.Fatalf("empty database counts nonzero: %+v", empty)
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
if err := db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts}); err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
if err := db.UpdatePrefixesBatch(map[string]time.Time{
"198.51.100.0/24": ts,
"2001:db8::/32": ts,
}); err != nil {
t.Fatalf("UpdatePrefixesBatch: %v", err)
}
if err := db.UpdatePeerBatch(map[string]PeerUpdate{
"192.0.2.1": {PeerIP: "192.0.2.1", PeerASN: 64500, MessageType: "UPDATE", Timestamp: ts},
}); err != nil {
t.Fatalf("UpdatePeerBatch: %v", err)
}
if err := db.RecordPeering(64500, 64501, ts); err != nil {
t.Fatalf("RecordPeering: %v", err)
}
routes := []*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, ts),
mkV4Route(t, "203.0.113.0/24", 64501, ts.Add(time.Minute)),
mkV6Route("2001:db8::/32", 64502, ts.Add(2*time.Minute)),
}
if err := db.UpsertLiveRouteBatch(routes); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
assertCounts(t, "after inserts", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
})
// Re-announcing the same routes is an update, not an insert: counts hold.
if err := db.UpsertLiveRouteBatch(routes); err != nil {
t.Fatalf("UpsertLiveRouteBatch (re-announce): %v", err)
}
stats, err = db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
assertCounts(t, "after re-announce", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
})
// A withdrawal removes one route.
if err := db.DeleteLiveRouteBatch([]LiveRouteDeletion{
{Prefix: "203.0.113.0/24", OriginASN: 64501, PeerIP: "192.0.2.1", IPVersion: ipVersionV4},
}); err != nil {
t.Fatalf("DeleteLiveRouteBatch: %v", err)
}
stats, err = db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
assertCounts(t, "after delete", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 1, ipv6Routes: 1, liveRoutes: 2,
})
}
// TestLiveCountsSeededFromDatabaseAtStartup writes rows, reopens the same
// database file, and checks the counts and the prefix distribution come back
// from the seed scan rather than starting at zero.
func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
if err := db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts, 64502: ts}); err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
if err := db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, ts),
mkV6Route("2001:db8::/32", 64502, ts),
}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
if err := db.Close(); err != nil {
t.Fatalf("Close: %v", err)
}
reopened, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to reopen database: %v", err)
}
defer func() { _ = reopened.Close() }()
stats, err := reopened.GetStatsContext(context.Background())
if err != nil {
t.Fatalf("GetStatsContext after reopen: %v", err)
}
if stats.ASNs != 3 {
t.Errorf("seeded ASNs = %d, want 3", stats.ASNs)
}
if stats.IPv4Routes != 1 || stats.IPv6Routes != 1 || stats.LiveRoutes != 2 {
t.Errorf("seeded routes = (v4 %d, v6 %d, total %d), want (1, 1, 2)",
stats.IPv4Routes, stats.IPv6Routes, stats.LiveRoutes)
}
assertDistribution(t, "seeded IPv4 distribution", stats.IPv4PrefixDistribution,
[]PrefixDistribution{{MaskLength: 24, Count: 1}})
assertDistribution(t, "seeded IPv6 distribution", stats.IPv6PrefixDistribution,
[]PrefixDistribution{{MaskLength: 32, Count: 1}})
}
// TestPrefixDistributionTracksWrites checks that the prefix distribution the
// stats read reports stays exact across each kind of live-route write, and that
// after every step it equals what the distribution query reads from the route
// tables. The steps run once through the batch methods the prefix handler uses
// and once through the single-route methods.
func TestPrefixDistributionTracksWrites(t *testing.T) {
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
shared := mkV4Route(t, "198.51.100.0/24", 64500, ts)
sharedSecondPeer := mkV4Route(t, "198.51.100.0/24", 64500, ts)
sharedSecondPeer.PeerIP = "192.0.2.2"
other := mkV4Route(t, "203.0.113.0/24", 64501, ts)
wide := mkV4Route(t, "172.16.0.0/16", 64502, ts)
v6 := mkV6Route("2001:db8::/32", 64503, ts)
v6SecondPeer := mkV6Route("2001:db8::/32", 64503, ts)
v6SecondPeer.PeerIP = "2001:db8::2"
all := []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 2}}
v6Only := []PrefixDistribution{{MaskLength: 32, Count: 1}}
steps := []struct {
name string
announce []*LiveRoute
withdraw []*LiveRoute
wantV4 []PrefixDistribution
wantV6 []PrefixDistribution
}{
{
name: "new routes",
announce: []*LiveRoute{shared, other, wide, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "re-announcement",
announce: []*LiveRoute{shared, other, wide, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "second peer announces a prefix that has a live route",
announce: []*LiveRoute{sharedSecondPeer, v6SecondPeer},
wantV4: all, wantV6: v6Only,
},
{
name: "withdrawal of a route that is not the last for its prefix",
withdraw: []*LiveRoute{shared, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "withdrawal of the last route for a prefix",
withdraw: []*LiveRoute{sharedSecondPeer, v6SecondPeer},
wantV4: []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 1}},
},
{
name: "withdrawal of every remaining route",
withdraw: []*LiveRoute{other, wide},
},
{
name: "two peers announce a new prefix together",
announce: []*LiveRoute{shared, sharedSecondPeer, v6, v6SecondPeer},
wantV4: []PrefixDistribution{{MaskLength: 24, Count: 1}},
wantV6: v6Only,
},
{
name: "both routes for a prefix withdrawn together",
withdraw: []*LiveRoute{shared, sharedSecondPeer, v6, v6SecondPeer},
},
// The feed often withdraws a route that is not live. That must not take
// the prefix out of the distribution. A count wrongly taken below zero is
// left out of the answer, so the next step shows it: its announcement
// at the same mask length would then not be counted.
{
name: "withdrawal of routes that are not live",
withdraw: []*LiveRoute{other, v6},
},
{
name: "announcement after a withdrawal of routes that are not live",
announce: []*LiveRoute{other, v6},
wantV4: []PrefixDistribution{{MaskLength: 24, Count: 1}},
wantV6: v6Only,
},
}
for _, batch := range []bool{true, false} {
name := "single-route writes"
if batch {
name = "batch writes"
}
t.Run(name, func(t *testing.T) {
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
for _, step := range steps {
if err := announceRoutes(db, batch, step.announce); err != nil {
t.Fatalf("%s: announce: %v", step.name, err)
}
if err := withdrawRoutes(db, batch, step.withdraw); err != nil {
t.Fatalf("%s: withdraw: %v", step.name, err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("%s: GetStatsContext: %v", step.name, err)
}
queryV4, queryV6, err := db.GetPrefixDistributionContext(ctx)
if err != nil {
t.Fatalf("%s: GetPrefixDistributionContext: %v", step.name, err)
}
assertDistribution(t, step.name+": IPv4 distribution", stats.IPv4PrefixDistribution, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution", stats.IPv6PrefixDistribution, step.wantV6)
assertDistribution(t, step.name+": IPv4 distribution query", queryV4, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution query", queryV6, step.wantV6)
}
})
}
}
// announceRoutes writes routes in one UpsertLiveRouteBatch, or with one
// UpsertLiveRoute each.
func announceRoutes(db *Database, batch bool, routes []*LiveRoute) error {
if batch {
return db.UpsertLiveRouteBatch(routes)
}
for _, route := range routes {
if err := db.UpsertLiveRoute(route); err != nil {
return err
}
}
return nil
}
// withdrawRoutes removes routes in one DeleteLiveRouteBatch, or with one
// DeleteLiveRoute each. It names each route by prefix and peer only, with no
// origin ASN, as a withdrawal from the feed does when its message carries no AS
// path.
func withdrawRoutes(db *Database, batch bool, routes []*LiveRoute) error {
if !batch {
for _, route := range routes {
if err := db.DeleteLiveRoute(route.Prefix, 0, route.PeerIP); err != nil {
return err
}
}
return nil
}
deletions := make([]LiveRouteDeletion, 0, len(routes))
for _, route := range routes {
deletions = append(deletions, LiveRouteDeletion{
Prefix: route.Prefix, PeerIP: route.PeerIP, IPVersion: route.IPVersion,
})
}
return db.DeleteLiveRouteBatch(deletions)
}
// TestStatsRouteTimestamps checks the oldest/newest route timestamps are read
// from the right rows across both tables and parse into time.Time. The old
// MIN/MAX union query read its result into *time.Time, which the driver could
// not parse, so it logged a warning every call and left both timestamps nil.
func TestStatsRouteTimestamps(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
empty, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext on empty database: %v", err)
}
if empty.OldestRoute != nil || empty.NewestRoute != nil {
t.Fatalf("empty database timestamps = (%v, %v), want (nil, nil)",
empty.OldestRoute, empty.NewestRoute)
}
base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
oldest := base
newest := base.Add(2 * time.Minute)
if err := db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, base.Add(time.Minute)),
mkV4Route(t, "203.0.113.0/24", 64501, oldest),
mkV6Route("2001:db8::/32", 64502, newest),
}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
if stats.OldestRoute == nil || !stats.OldestRoute.Equal(oldest) {
t.Errorf("OldestRoute = %v, want %v", stats.OldestRoute, oldest)
}
if stats.NewestRoute == nil || !stats.NewestRoute.Equal(newest) {
t.Errorf("NewestRoute = %v, want %v", stats.NewestRoute, newest)
}
}
// TestLiveCountsConcurrentReadWrite runs writers and stats readers at once so
// the race detector proves the counters are safe under concurrent use.
func TestLiveCountsConcurrentReadWrite(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
const writers = 4
var wg sync.WaitGroup
wg.Add(writers)
for w := range writers {
go func(base int) {
defer wg.Done()
for i := range 25 {
asn := 65000 + base*100 + i
route := mkV6Route("2001:db8::/32", asn, ts)
route.PeerIP = "2001:db8::" + uuid.NewString()
if err := db.UpsertLiveRoute(route); err != nil {
t.Errorf("UpsertLiveRoute: %v", err)
return
}
}
}(w)
}
var readerWG sync.WaitGroup
readerWG.Add(1)
stop := make(chan struct{})
go func() {
defer readerWG.Done()
for {
select {
case <-stop:
return
default:
if _, err := db.GetStatsContext(context.Background()); err != nil {
t.Errorf("GetStatsContext: %v", err)
return
}
}
}
}()
wg.Wait()
close(stop)
readerWG.Wait()
stats, err := db.GetStatsContext(context.Background())
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
if want := writers * 25; stats.IPv6Routes != want {
t.Errorf("IPv6Routes = %d, want %d", stats.IPv6Routes, want)
}
// Every writer announced the same prefix, so it counts once.
assertDistribution(t, "IPv6 distribution", stats.IPv6PrefixDistribution,
[]PrefixDistribution{{MaskLength: 32, Count: 1}})
}
type wantCounts struct {
asns int
prefixes int
peerings int
peers int
ipv4Routes int
ipv6Routes int
liveRoutes int
}
func assertCounts(t *testing.T, when string, got Stats, want wantCounts) {
t.Helper()
if got.ASNs != want.asns {
t.Errorf("%s: ASNs = %d, want %d", when, got.ASNs, want.asns)
}
if got.Prefixes != want.prefixes {
t.Errorf("%s: Prefixes = %d, want %d", when, got.Prefixes, want.prefixes)
}
if got.Peerings != want.peerings {
t.Errorf("%s: Peerings = %d, want %d", when, got.Peerings, want.peerings)
}
if got.Peers != want.peers {
t.Errorf("%s: Peers = %d, want %d", when, got.Peers, want.peers)
}
if got.IPv4Routes != want.ipv4Routes {
t.Errorf("%s: IPv4Routes = %d, want %d", when, got.IPv4Routes, want.ipv4Routes)
}
if got.IPv6Routes != want.ipv6Routes {
t.Errorf("%s: IPv6Routes = %d, want %d", when, got.IPv6Routes, want.ipv6Routes)
}
if got.LiveRoutes != want.liveRoutes {
t.Errorf("%s: LiveRoutes = %d, want %d", when, got.LiveRoutes, want.liveRoutes)
}
}
func assertDistribution(t *testing.T, what string, got, want []PrefixDistribution) {
t.Helper()
if !slices.Equal(got, want) {
t.Errorf("%s = %v, want %v", what, got, want)
}
}
File diff suppressed because it is too large. Load diff
+148 -273
View File
@@ -3,23 +3,36 @@ package database
import ( import (
"context" "context"
"database/sql" "database/sql"
"net" "errors"
"fmt"
"net/netip"
"sync" "sync"
"testing" "testing"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/config" "git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"github.com/google/uuid"
) )
// tempStoreMemory is the PRAGMA temp_store value meaning "hold temp B-trees in // tempStoreMemory is the PRAGMA temp_store value meaning "hold temp B-trees in
// memory"; the DSN change must leave temp_store below this so they spill to disk. // memory"; the DSN change must leave temp_store below this so they spill to disk.
const tempStoreMemory = 2 const tempStoreMemory = 2
// autoVacuumIncremental is the PRAGMA auto_vacuum value meaning "incremental".
const autoVacuumIncremental = 2
// vacuumTestRoutes is how many routes the vacuum test writes and then deletes,
// enough to leave many free pages in the file.
const vacuumTestRoutes = 2000
// heldConnections is how many pooled connections the pragma test holds open at // heldConnections is how many pooled connections the pragma test holds open at
// once so each is a distinct SQLite connection that parsed the DSN. // once so each is a distinct SQLite connection that parsed the DSN.
const heldConnections = 5 const heldConnections = 5
// testPeerIP is the peer every route in the IP lookup test is learned from.
const testPeerIP = "192.0.2.254"
// Parameters for the checkpoint-contention regression test. // Parameters for the checkpoint-contention regression test.
const ( const (
// contentionIterations is how many batch writes race the checkpoint loop. // contentionIterations is how many batch writes race the checkpoint loop.
@@ -32,286 +45,98 @@ const (
asnSecondBand = 100 asnSecondBand = 100
) )
func TestIPToUint32(t *testing.T) { // TestGetIPInfoFindsMostSpecificLiveRoute stores nested live prefixes for both
tests := []struct { // families and checks that a lookup returns the most specific one covering the
name string // address, ErrNoRoute when none covers it, and the next less specific prefix
ip string // once the only route of the most specific one is withdrawn.
expected uint32 func TestGetIPInfoFindsMostSpecificLiveRoute(t *testing.T) {
}{ cfg := &config.Config{StateDir: t.TempDir()}
{
name: "Simple IP",
ip: "192.168.1.1",
expected: 3232235777, // 192<<24 + 168<<16 + 1<<8 + 1
},
{
name: "Minimum IP",
ip: "0.0.0.0",
expected: 0,
},
{
name: "Maximum IP",
ip: "255.255.255.255",
expected: 4294967295,
},
{
name: "10.0.0.0",
ip: "10.0.0.0",
expected: 167772160,
},
{
name: "172.16.0.0",
ip: "172.16.0.0",
expected: 2886729728,
},
{
name: "8.8.8.8",
ip: "8.8.8.8",
expected: 134744072,
},
{
name: "1.2.3.4",
ip: "1.2.3.4",
expected: 16909060,
},
}
for _, tt := range tests { db, err := New(cfg, logger.New())
t.Run(tt.name, func(t *testing.T) { if err != nil {
ip := net.ParseIP(tt.ip) t.Fatalf("failed to create database: %v", err)
if ip == nil {
t.Fatalf("Failed to parse IP: %s", tt.ip)
} }
defer func() { _ = db.Close() }()
result := ipToUint32(ip) // Nested live prefixes, each originated by its own AS.
if result != tt.expected { origins := map[string]int{
t.Errorf("ipToUint32(%s) = %d, want %d", tt.ip, result, tt.expected) "10.0.0.0/8": 64500,
} "10.1.0.0/16": 64501,
"10.1.2.0/24": 64502,
// Test with IPv4-mapped IPv6 address "2001:db8::/32": 64500,
ip6 := net.ParseIP(tt.ip).To16() "2001:db8:1::/48": 64501,
if ip6 != nil { "2001:db8:1:2::/64": 64502,
result6 := ipToUint32(ip6)
if result6 != tt.expected {
t.Errorf("ipToUint32(%s as IPv6) = %d, want %d", tt.ip, result6, tt.expected)
}
} }
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
routes := make([]*LiveRoute, 0, len(origins))
for prefix, asn := range origins {
routes = append(routes, &LiveRoute{
ID: uuid.New(),
Prefix: prefix,
MaskLength: netip.MustParsePrefix(prefix).Bits(),
IPVersion: detectIPVersion(prefix),
OriginASN: asn,
PeerIP: testPeerIP,
ASPath: []int{asn},
NextHop: testPeerIP,
LastUpdated: ts,
}) })
} }
if err := db.UpsertLiveRouteBatch(routes); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
} }
func TestCalculateIPv4Range(t *testing.T) { // lookup checks that ip resolves to the live prefix want, or to ErrNoRoute
tests := []struct { // when want is empty.
name string lookup := func(ip, want string) {
cidr string t.Helper()
wantStart uint32
wantEnd uint32 info, err := db.GetIPInfo(ip)
wantErr bool if want == "" {
}{ if !errors.Is(err, ErrNoRoute) {
{ t.Errorf("GetIPInfo(%s) = %+v, %v; want ErrNoRoute", ip, info, err)
name: "Single IP /32",
cidr: "192.168.1.1/32",
wantStart: 3232235777,
wantEnd: 3232235777,
},
{
name: "Class C /24",
cidr: "192.168.1.0/24",
wantStart: 3232235776, // 192.168.1.0
wantEnd: 3232236031, // 192.168.1.255
},
{
name: "Class B /16",
cidr: "192.168.0.0/16",
wantStart: 3232235520, // 192.168.0.0
wantEnd: 3232301055, // 192.168.255.255
},
{
name: "Class A /8",
cidr: "10.0.0.0/8",
wantStart: 167772160, // 10.0.0.0
wantEnd: 184549375, // 10.255.255.255
},
{
name: "Entire IPv4 space /0",
cidr: "0.0.0.0/0",
wantStart: 0,
wantEnd: 4294967295,
},
{
name: "Small subnet /30",
cidr: "192.168.1.0/30",
wantStart: 3232235776, // 192.168.1.0
wantEnd: 3232235779, // 192.168.1.3
},
{
name: "Medium subnet /20",
cidr: "172.16.0.0/20",
wantStart: 2886729728, // 172.16.0.0
wantEnd: 2886733823, // 172.16.15.255
},
{
name: "Private range 172.16/12",
cidr: "172.16.0.0/12",
wantStart: 2886729728, // 172.16.0.0
wantEnd: 2887778303, // 172.31.255.255
},
{
name: "Google DNS /29",
cidr: "8.8.8.8/29",
wantStart: 134744072, // 8.8.8.8 (network is actually 8.8.8.8 with /29)
wantEnd: 134744079, // 8.8.8.15
},
{
name: "Non-zero host bits",
cidr: "192.168.1.5/24",
wantStart: 3232235776, // 192.168.1.0 (network address)
wantEnd: 3232236031, // 192.168.1.255
},
{
name: "Invalid CIDR",
cidr: "192.168.1.1/33",
wantErr: true,
},
{
name: "Invalid IP",
cidr: "256.256.256.256/24",
wantErr: true,
},
{
name: "IPv6 CIDR",
cidr: "2001:db8::/32",
wantErr: true,
},
{
name: "Empty CIDR",
cidr: "",
wantErr: true,
},
{
name: "Missing mask",
cidr: "192.168.1.1",
wantErr: true,
},
} }
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
start, end, err := CalculateIPv4Range(tt.cidr)
if tt.wantErr {
if err == nil {
t.Errorf("CalculateIPv4Range(%s) expected error, got nil", tt.cidr)
}
return return
} }
if err != nil { if err != nil {
t.Errorf("CalculateIPv4Range(%s) unexpected error: %v", tt.cidr, err) t.Errorf("GetIPInfo(%s): %v", ip, err)
return return
} }
if info.Netblock != want || info.MaskLength != netip.MustParsePrefix(want).Bits() ||
if start != tt.wantStart { info.ASN != origins[want] {
t.Errorf("CalculateIPv4Range(%s) start = %d, want %d", tt.cidr, start, tt.wantStart) t.Errorf("GetIPInfo(%s) = %s (mask %d) AS%d, want %s AS%d",
ip, info.Netblock, info.MaskLength, info.ASN, want, origins[want])
}
} }
if end != tt.wantEnd { lookup("10.1.2.3", "10.1.2.0/24")
t.Errorf("CalculateIPv4Range(%s) end = %d, want %d", tt.cidr, end, tt.wantEnd) lookup("::ffff:10.1.2.3", "10.1.2.0/24")
} lookup("10.1.3.4", "10.1.0.0/16")
lookup("10.2.0.1", "10.0.0.0/8")
lookup("192.0.2.1", "")
lookup("2001:db8:1:2::3", "2001:db8:1:2::/64")
lookup("2001:db8:1:3::4", "2001:db8:1::/48")
lookup("2001:db8:2::1", "2001:db8::/32")
lookup("2001:db9::1", "")
// Verify that start <= end err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{
if start > end { {Prefix: "10.1.2.0/24", OriginASN: 64502, PeerIP: testPeerIP, IPVersion: ipVersionV4},
t.Errorf("CalculateIPv4Range(%s) start (%d) > end (%d)", tt.cidr, start, end) {Prefix: "2001:db8:1:2::/64", OriginASN: 64502, PeerIP: testPeerIP, IPVersion: ipVersionV6},
}
// Verify the range size matches the CIDR mask
if !tt.wantErr && tt.cidr != "" {
_, ipNet, _ := net.ParseCIDR(tt.cidr)
if ipNet != nil {
ones, bits := ipNet.Mask.Size()
expectedSize := uint32(1) << uint(bits-ones)
actualSize := end - start + 1
if actualSize != expectedSize {
t.Errorf("CalculateIPv4Range(%s) range size = %d, want %d", tt.cidr, actualSize, expectedSize)
}
}
}
}) })
}
}
func TestIPv4RangeIntegration(t *testing.T) {
// Test that our functions work correctly together
tests := []struct {
name string
cidr string
testIPs []string
shouldContain []bool
}{
{
name: "192.168.1.0/24",
cidr: "192.168.1.0/24",
testIPs: []string{
"192.168.1.0",
"192.168.1.1",
"192.168.1.255",
"192.168.0.255",
"192.168.2.0",
},
shouldContain: []bool{true, true, true, false, false},
},
{
name: "10.0.0.0/8",
cidr: "10.0.0.0/8",
testIPs: []string{
"10.0.0.0",
"10.255.255.255",
"10.1.2.3",
"9.255.255.255",
"11.0.0.0",
},
shouldContain: []bool{true, true, true, false, false},
},
{
name: "172.16.0.0/12",
cidr: "172.16.0.0/12",
testIPs: []string{
"172.16.0.0",
"172.31.255.255",
"172.20.1.1",
"172.15.255.255",
"172.32.0.0",
},
shouldContain: []bool{true, true, true, false, false},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
start, end, err := CalculateIPv4Range(tt.cidr)
if err != nil { if err != nil {
t.Fatalf("Failed to calculate range for %s: %v", tt.cidr, err) t.Fatalf("DeleteLiveRouteBatch: %v", err)
} }
for i, testIP := range tt.testIPs { lookup("10.1.2.3", "10.1.0.0/16")
ip := net.ParseIP(testIP) lookup("2001:db8:1:2::3", "2001:db8:1::/48")
if ip == nil {
t.Fatalf("Failed to parse test IP: %s", testIP)
}
ipUint := ipToUint32(ip)
contained := ipUint >= start && ipUint <= end
if contained != tt.shouldContain[i] {
t.Errorf("IP %s in range %s: got %v, want %v", testIP, tt.cidr, contained, tt.shouldContain[i])
}
}
})
}
} }
// TestConnectionPoolPragmas holds several pooled connections open at once and // TestConnectionPoolPragmas holds several pooled connections open at once and
// checks each one carries the per-connection settings from the DSN, plus the // checks each one carries the per-connection settings from the DSN, plus the
// process-wide hard heap limit. // process-wide hard heap limit, and sees the new file with auto_vacuum
// incremental.
func TestConnectionPoolPragmas(t *testing.T) { func TestConnectionPoolPragmas(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()} cfg := &config.Config{StateDir: t.TempDir()}
@@ -372,6 +197,74 @@ func TestConnectionPoolPragmas(t *testing.T) {
if hardHeapLimit != sqliteHardHeapLimitBytes { if hardHeapLimit != sqliteHardHeapLimitBytes {
t.Errorf("conn %d: hard_heap_limit = %d, want %d", i, hardHeapLimit, sqliteHardHeapLimitBytes) t.Errorf("conn %d: hard_heap_limit = %d, want %d", i, hardHeapLimit, sqliteHardHeapLimitBytes)
} }
var autoVacuum int
if err := c.QueryRowContext(ctx, "PRAGMA auto_vacuum").Scan(&autoVacuum); err != nil {
t.Fatalf("conn %d: failed to read auto_vacuum: %v", i, err)
}
if autoVacuum != autoVacuumIncremental {
t.Errorf("conn %d: auto_vacuum = %d, want %d (incremental)", i, autoVacuum, autoVacuumIncremental)
}
}
}
// TestVacuumReturnsFreePages checks that after routes are deleted from a new
// database, one Vacuum call returns every page they used. The deletes leave
// fewer free pages than the 1000 Vacuum frees per call, so none may remain.
// With auto_vacuum off (issue https://git.eeqj.de/sneak/routewatch/issues/43)
// the free pages stayed in the file, and with the PRAGMA run by ExecContext
// Vacuum freed only one page per call.
func TestVacuumReturnsFreePages(t *testing.T) {
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
ts := time.Now().UTC()
const asn = 64500
routes := make([]*LiveRoute, 0, vacuumTestRoutes)
deletions := make([]LiveRouteDeletion, 0, vacuumTestRoutes)
for i := range vacuumTestRoutes {
route := mkV4Route(t, fmt.Sprintf("10.%d.%d.0/24", i/256, i%256), asn, ts)
routes = append(routes, route)
deletions = append(deletions, LiveRouteDeletion{
Prefix: route.Prefix,
OriginASN: asn,
PeerIP: route.PeerIP,
IPVersion: ipVersionV4,
})
}
if err := db.UpsertLiveRouteBatch(routes); err != nil {
t.Fatalf("failed to write routes: %v", err)
}
if err := db.DeleteLiveRouteBatch(deletions); err != nil {
t.Fatalf("failed to delete routes: %v", err)
}
var before int
if err := db.db.QueryRowContext(ctx, "PRAGMA freelist_count").Scan(&before); err != nil {
t.Fatalf("failed to read freelist_count: %v", err)
}
if before == 0 {
t.Fatalf("no free pages after deleting %d routes", vacuumTestRoutes)
}
if err := db.Vacuum(ctx); err != nil {
t.Fatalf("Vacuum failed: %v", err)
}
var after int
if err := db.db.QueryRowContext(ctx, "PRAGMA freelist_count").Scan(&after); err != nil {
t.Fatalf("failed to read freelist_count: %v", err)
}
if after != 0 {
t.Errorf("free pages after Vacuum = %d of %d, want 0", after, before)
} }
} }
@@ -424,21 +317,3 @@ func TestBatchWriteDuringCheckpoint(t *testing.T) {
cancel() cancel()
wg.Wait() wg.Wait()
} }
func BenchmarkIPToUint32(b *testing.B) {
ip := net.ParseIP("192.168.1.1")
b.ResetTimer()
for i := 0; i < b.N; i++ {
_ = ipToUint32(ip)
}
}
func BenchmarkCalculateIPv4Range(b *testing.B) {
cidr := "192.168.0.0/16"
b.ResetTimer()
for i := 0; i < b.N; i++ {
_, _, _ = CalculateIPv4Range(cidr)
}
}
+2 -2
View File
@@ -18,6 +18,8 @@ type Stats struct {
Peers int Peers int
FileSizeBytes int64 FileSizeBytes int64
LiveRoutes int LiveRoutes int
IPv4Routes int
IPv6Routes int
OldestRoute *time.Time OldestRoute *time.Time
NewestRoute *time.Time NewestRoute *time.Time
IPv4PrefixDistribution []PrefixDistribution IPv4PrefixDistribution []PrefixDistribution
@@ -61,8 +63,6 @@ type Store interface {
GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, ipv6Count int, err error) GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, ipv6Count int, err error)
// IP lookup operations // IP lookup operations
GetASInfoForIP(ip string) (*ASInfo, error)
GetASInfoForIPContext(ctx context.Context, ip string) (*ASInfo, error)
GetIPInfo(ip string) (*IPInfo, error) GetIPInfo(ip string) (*IPInfo, error)
GetIPInfoContext(ctx context.Context, ip string) (*IPInfo, error) GetIPInfoContext(ctx context.Context, ip string) (*IPInfo, error)
-13
View File
@@ -77,9 +77,6 @@ type LiveRoute struct {
ASPath []int `json:"as_path"` ASPath []int `json:"as_path"`
NextHop string `json:"next_hop"` NextHop string `json:"next_hop"`
LastUpdated time.Time `json:"last_updated"` LastUpdated time.Time `json:"last_updated"`
// IPv4 range fields for fast lookups (nil for IPv6)
V4IPStart *uint32 `json:"v4_ip_start,omitempty"`
V4IPEnd *uint32 `json:"v4_ip_end,omitempty"`
} }
// PrefixDistribution represents the distribution of prefixes by mask length // PrefixDistribution represents the distribution of prefixes by mask length
@@ -88,16 +85,6 @@ type PrefixDistribution struct {
Count int `json:"count"` Count int `json:"count"`
} }
// ASInfo represents AS information for an IP lookup (legacy format)
type ASInfo struct {
ASN int `json:"asn"`
Handle string `json:"handle"`
Description string `json:"description"`
Prefix string `json:"prefix"`
LastUpdated time.Time `json:"last_updated"`
Age string `json:"age"`
}
// IPInfo represents comprehensive IP information for the /ip endpoint // IPInfo represents comprehensive IP information for the /ip endpoint
type IPInfo struct { type IPInfo struct {
IP string `json:"ip"` IP string `json:"ip"`
-6
View File
@@ -107,9 +107,6 @@ CREATE TABLE IF NOT EXISTS live_routes_v4 (
as_path TEXT NOT NULL, -- JSON array as_path TEXT NOT NULL, -- JSON array
next_hop TEXT NOT NULL, next_hop TEXT NOT NULL,
last_updated DATETIME NOT NULL, last_updated DATETIME NOT NULL,
-- IPv4 range columns for fast lookups
ip_start INTEGER NOT NULL, -- Start of IPv4 range as 32-bit unsigned int
ip_end INTEGER NOT NULL, -- End of IPv4 range as 32-bit unsigned int
UNIQUE(prefix, origin_asn, peer_ip) UNIQUE(prefix, origin_asn, peer_ip)
); );
@@ -123,7 +120,6 @@ CREATE TABLE IF NOT EXISTS live_routes_v6 (
as_path TEXT NOT NULL, -- JSON array as_path TEXT NOT NULL, -- JSON array
next_hop TEXT NOT NULL, next_hop TEXT NOT NULL,
last_updated DATETIME NOT NULL, last_updated DATETIME NOT NULL,
-- Note: IPv6 doesn't use integer range columns
UNIQUE(prefix, origin_asn, peer_ip) UNIQUE(prefix, origin_asn, peer_ip)
); );
@@ -132,8 +128,6 @@ CREATE INDEX IF NOT EXISTS idx_live_routes_v4_prefix ON live_routes_v4(prefix);
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_length ON live_routes_v4(mask_length); CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_length ON live_routes_v4(mask_length);
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_origin_asn ON live_routes_v4(origin_asn); CREATE INDEX IF NOT EXISTS idx_live_routes_v4_origin_asn ON live_routes_v4(origin_asn);
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_last_updated ON live_routes_v4(last_updated); CREATE INDEX IF NOT EXISTS idx_live_routes_v4_last_updated ON live_routes_v4(last_updated);
-- Indexes for IPv4 range queries
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_ip_range ON live_routes_v4(ip_start, ip_end);
-- Index to optimize prefix distribution queries -- Index to optimize prefix distribution queries
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_prefix ON live_routes_v4(mask_length, prefix); CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_prefix ON live_routes_v4(mask_length, prefix);
+13
View File
@@ -1,6 +1,8 @@
package database package database
import ( import (
"fmt"
"net"
"strings" "strings"
"github.com/google/uuid" "github.com/google/uuid"
@@ -18,3 +20,14 @@ func detectIPVersion(prefix string) int {
return ipVersionV4 return ipVersionV4
} }
// prefixMaskLength returns the mask length of a prefix such as 192.0.2.0/24.
func prefixMaskLength(prefix string) (int, error) {
_, network, err := net.ParseCIDR(prefix)
if err != nil {
return 0, fmt.Errorf("invalid prefix %s: %w", prefix, err)
}
maskLength, _ := network.Mask.Size()
return maskLength, nil
}
+1 -20
View File
@@ -232,25 +232,6 @@ func (m *mockStore) GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, i
return m.GetLiveRouteCounts() return m.GetLiveRouteCounts()
} }
// GetASInfoForIP mock implementation
func (m *mockStore) GetASInfoForIP(ip string) (*database.ASInfo, error) {
// Simple mock - return a test AS
now := time.Now()
return &database.ASInfo{
ASN: 15169,
Handle: "GOOGLE",
Description: "Google LLC",
Prefix: "8.8.8.0/24",
LastUpdated: now.Add(-5 * time.Minute),
Age: "5m0s",
}, nil
}
// GetASInfoForIPContext mock implementation with context support
func (m *mockStore) GetASInfoForIPContext(ctx context.Context, ip string) (*database.ASInfo, error) {
return m.GetASInfoForIP(ip)
}
// GetASDetails mock implementation // GetASDetails mock implementation
func (m *mockStore) GetASDetails(asn int) (*database.ASN, []database.LiveRoute, error) { func (m *mockStore) GetASDetails(asn int) (*database.ASN, []database.LiveRoute, error) {
m.mu.Lock() m.mu.Lock()
@@ -450,7 +431,7 @@ func TestRouteWatchLiveFeed(t *testing.T) {
} }
// Create server // Create server
srv := server.New(mockDB, s, logger) srv := server.New(mockDB, s, logger, cfg)
// Create RouteWatch with 5 second limit // Create RouteWatch with 5 second limit
deps := Dependencies{ deps := Dependencies{
+18 -44
View File
@@ -2,6 +2,7 @@ package routewatch
import ( import (
"net" "net"
"net/netip"
"strings" "strings"
"sync" "sync"
"time" "time"
@@ -108,7 +109,7 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
for _, announcement := range msg.Announcements { for _, announcement := range msg.Announcements {
for _, prefix := range announcement.Prefixes { for _, prefix := range announcement.Prefixes {
h.batch = append(h.batch, prefixUpdate{ h.batch = append(h.batch, prefixUpdate{
prefix: prefix, prefix: canonicalPrefix(prefix),
originASN: originASN, originASN: originASN,
peer: msg.Peer, peer: msg.Peer,
messageType: "announcement", messageType: "announcement",
@@ -125,7 +126,7 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
// Process withdrawals // Process withdrawals
for _, prefix := range msg.Withdrawals { for _, prefix := range msg.Withdrawals {
h.batch = append(h.batch, prefixUpdate{ h.batch = append(h.batch, prefixUpdate{
prefix: prefix, prefix: canonicalPrefix(prefix),
originASN: originASN, // Use the originASN from path if available originASN: originASN, // Use the originASN from path if available
peer: msg.Peer, peer: msg.Peer,
messageType: "withdrawal", messageType: "withdrawal",
@@ -264,6 +265,21 @@ func (h *PrefixHandler) flushBatchLocked() {
h.lastFlush = time.Now() h.lastFlush = time.Now()
} }
// canonicalPrefix returns prefix in the text form net/netip prints, the form
// the IP lookup builds when it looks a prefix up. The feed sends IPv6
// announcements compressed ("2001:db8::/32") but IPv6 withdrawals uncompressed
// ("2001:db8:0:0:0:0:0:0/32"); stored as received, a withdrawal would not match
// the route its announcement stored. A prefix that does not parse is returned
// unchanged, and the batch flush reports it.
func canonicalPrefix(prefix string) string {
p, err := netip.ParsePrefix(prefix)
if err != nil {
return prefix
}
return p.Masked().String()
}
// parseCIDR extracts the mask length and IP version from a prefix string // parseCIDR extracts the mask length and IP version from a prefix string
func parseCIDR(prefix string) (maskLength int, ipVersion int, err error) { func parseCIDR(prefix string) (maskLength int, ipVersion int, err error) {
_, ipNet, err := net.ParseCIDR(prefix) _, ipNet, err := net.ParseCIDR(prefix)
@@ -315,20 +331,6 @@ func (h *PrefixHandler) processAnnouncement(_ *database.Prefix, update prefixUpd
LastUpdated: update.timestamp, LastUpdated: update.timestamp,
} }
// For IPv4, calculate the IP range
if ipVersion == ipv4Version {
start, end, err := database.CalculateIPv4Range(update.prefix)
if err == nil {
liveRoute.V4IPStart = &start
liveRoute.V4IPEnd = &end
} else {
h.logger.Error("Failed to calculate IPv4 range",
"prefix", update.prefix,
"error", err,
)
}
}
if err := h.db.UpsertLiveRoute(liveRoute); err != nil { if err := h.db.UpsertLiveRoute(liveRoute); err != nil {
h.logger.Error("Failed to upsert live route", h.logger.Error("Failed to upsert live route",
"prefix", update.prefix, "prefix", update.prefix,
@@ -372,20 +374,6 @@ func (h *PrefixHandler) createLiveRoute(update prefixUpdate) *database.LiveRoute
LastUpdated: update.timestamp, LastUpdated: update.timestamp,
} }
// For IPv4, calculate the IP range
if ipVersion == ipv4Version {
start, end, err := database.CalculateIPv4Range(update.prefix)
if err == nil {
liveRoute.V4IPStart = &start
liveRoute.V4IPEnd = &end
} else {
h.logger.Error("Failed to calculate IPv4 range",
"prefix", update.prefix,
"error", err,
)
}
}
return liveRoute return liveRoute
} }
@@ -425,20 +413,6 @@ func (h *PrefixHandler) processAnnouncementDirect(update prefixUpdate) {
LastUpdated: update.timestamp, LastUpdated: update.timestamp,
} }
// For IPv4, calculate the IP range
if ipVersion == ipv4Version {
start, end, err := database.CalculateIPv4Range(update.prefix)
if err == nil {
liveRoute.V4IPStart = &start
liveRoute.V4IPEnd = &end
} else {
h.logger.Error("Failed to calculate IPv4 range",
"prefix", update.prefix,
"error", err,
)
}
}
if err := h.db.UpsertLiveRoute(liveRoute); err != nil { if err := h.db.UpsertLiveRoute(liveRoute); err != nil {
h.logger.Error("Failed to upsert live route", h.logger.Error("Failed to upsert live route",
"prefix", update.prefix, "prefix", update.prefix,
+74
View File
@@ -0,0 +1,74 @@
package routewatch
import (
"errors"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
const testPeerIP = "2001:db8:ffff::1"
// TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and
// withdrawals through the prefix handler into a real database and looks the
// addresses up. The prefixes are written the way the feed sends them: IPv6
// announcements compressed, IPv6 withdrawals uncompressed. The withdrawal must
// remove the route the announcement stored, and the IP lookup must find the
// stored prefix.
func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
h := &PrefixHandler{db: db, logger: logger.New()}
flush := func() {
h.mu.Lock()
defer h.mu.Unlock()
h.flushBatchLocked()
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
h.HandleMessage(&ristypes.RISMessage{
Peer: testPeerIP,
Path: ristypes.ASPath{testASNA, testASNB},
ParsedTimestamp: ts,
Announcements: []ristypes.RISAnnouncement{{
NextHop: testPeerIP,
Prefixes: []string{"2001:db8:1::/48", "192.0.2.0/24"},
}},
})
flush()
for ip, want := range map[string]string{
"2001:db8:1::1": "2001:db8:1::/48",
"192.0.2.1": "192.0.2.0/24",
} {
info, err := db.GetIPInfo(ip)
if err != nil {
t.Fatalf("GetIPInfo(%s) after announcement: %v", ip, err)
}
if info.Netblock != want || info.ASN != testASNB {
t.Errorf("GetIPInfo(%s) = %s AS%d, want %s AS%d", ip, info.Netblock, info.ASN, want, testASNB)
}
}
h.HandleMessage(&ristypes.RISMessage{
Peer: testPeerIP,
ParsedTimestamp: ts.Add(time.Minute),
Withdrawals: []string{"2001:db8:1:0:0:0:0:0/48", "192.0.2.0/24"},
})
flush()
for _, ip := range []string{"2001:db8:1::1", "192.0.2.1"} {
if info, err := db.GetIPInfo(ip); !errors.Is(err, database.ErrNoRoute) {
t.Errorf("GetIPInfo(%s) after withdrawal = %+v, %v; want ErrNoRoute", ip, info, err)
}
}
}
+4 -18
View File
@@ -219,13 +219,6 @@ func (s *Server) handleStatusJSON() http.HandlerFunc {
const bitsPerMegabit = 1000000.0 const bitsPerMegabit = 1000000.0
// Get route counts from database
ipv4Routes, ipv6Routes, err := s.db.GetLiveRouteCountsContext(ctx)
if err != nil {
s.logger.Warn("Failed to get live route counts", "error", err)
// Continue with zero counts
}
// Get route update metrics // Get route update metrics
routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics() routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics()
@@ -259,8 +252,8 @@ func (s *Server) handleStatusJSON() http.HandlerFunc {
Peers: dbStats.Peers, Peers: dbStats.Peers,
DatabaseSizeBytes: dbStats.FileSizeBytes, DatabaseSizeBytes: dbStats.FileSizeBytes,
LiveRoutes: dbStats.LiveRoutes, LiveRoutes: dbStats.LiveRoutes,
IPv4Routes: ipv4Routes, IPv4Routes: dbStats.IPv4Routes,
IPv6Routes: ipv6Routes, IPv6Routes: dbStats.IPv6Routes,
OldestRoute: dbStats.OldestRoute, OldestRoute: dbStats.OldestRoute,
NewestRoute: dbStats.NewestRoute, NewestRoute: dbStats.NewestRoute,
IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec, IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec,
@@ -439,13 +432,6 @@ func (s *Server) handleStats() http.HandlerFunc {
const bitsPerMegabit = 1000000.0 const bitsPerMegabit = 1000000.0
// Get route counts from database
ipv4Routes, ipv6Routes, err := s.db.GetLiveRouteCountsContext(ctx)
if err != nil {
s.logger.Warn("Failed to get live route counts", "error", err)
// Continue with zero counts
}
// Get route update metrics // Get route update metrics
routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics() routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics()
@@ -537,8 +523,8 @@ func (s *Server) handleStats() http.HandlerFunc {
Peers: dbStats.Peers, Peers: dbStats.Peers,
DatabaseSizeBytes: dbStats.FileSizeBytes, DatabaseSizeBytes: dbStats.FileSizeBytes,
LiveRoutes: dbStats.LiveRoutes, LiveRoutes: dbStats.LiveRoutes,
IPv4Routes: ipv4Routes, IPv4Routes: dbStats.IPv4Routes,
IPv6Routes: ipv6Routes, IPv6Routes: dbStats.IPv6Routes,
OldestRoute: dbStats.OldestRoute, OldestRoute: dbStats.OldestRoute,
NewestRoute: dbStats.NewestRoute, NewestRoute: dbStats.NewestRoute,
IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec, IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec,
+79 -1
View File
@@ -2,16 +2,20 @@ package server
import ( import (
"context" "context"
"encoding/json"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"runtime" "runtime"
"slices"
"testing" "testing"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/streamer"
"github.com/google/uuid"
) )
// blockingStatsDB embeds database.Store (left nil) and overrides only // blockingStatsDB embeds database.Store (left nil) and overrides only
@@ -38,7 +42,7 @@ func (d blockingStatsDB) GetStatsContext(_ context.Context) (database.Stats, err
func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) { func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) {
release := make(chan struct{}) release := make(chan struct{})
db := blockingStatsDB{release: release} db := blockingStatsDB{release: release}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New()) s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
handlers := map[string]http.HandlerFunc{ handlers := map[string]http.HandlerFunc{
"status.json": s.handleStatusJSON(), "status.json": s.handleStatusJSON(),
@@ -70,6 +74,80 @@ func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) {
} }
} }
// TestStatsHandlersAnswerFromMemory checks that both stats handlers answer 200
// with the live route counts and the prefix distribution while the database is
// closed, so that any query would fail: the request path reads them from
// memory. The prefix distribution query it used to run read every live route
// and, on a large database, took the whole 4-second deadline, so
// /api/v1/stats answered 500 (https://git.eeqj.de/sneak/routewatch/issues/30).
// The oldest and newest route times still come from one-row lookups at the ends
// of an index; with the database closed they are left out of the answer.
func TestStatsHandlersAnswerFromMemory(t *testing.T) {
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
if err := db.UpsertLiveRouteBatch([]*database.LiveRoute{
{
ID: uuid.New(), Prefix: "198.51.100.0/24", MaskLength: 24, IPVersion: 4,
OriginASN: 64500, PeerIP: "192.0.2.1", ASPath: []int{64500}, NextHop: "192.0.2.1",
LastUpdated: ts,
},
{
ID: uuid.New(), Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: "2001:db8::1", ASPath: []int{64501}, NextHop: "2001:db8::1",
LastUpdated: ts,
},
}); err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
if err := db.Close(); err != nil {
t.Fatalf("Close: %v", err)
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
handlers := map[string]http.HandlerFunc{
"status.json": s.handleStatusJSON(),
"stats": s.handleStats(),
}
for name, handler := range handlers {
rec := httptest.NewRecorder()
handler(rec, httptest.NewRequest(http.MethodGet, "/", nil))
if rec.Code != http.StatusOK {
t.Errorf("%s: status %d, want %d; body %s", name, rec.Code, http.StatusOK, rec.Body)
continue
}
var body struct {
Data struct {
IPv4Routes int `json:"ipv4_routes"`
IPv6Routes int `json:"ipv6_routes"`
IPv4PrefixDistribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"`
IPv6PrefixDistribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"`
} `json:"data"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
t.Fatalf("%s: decoding the answer: %v", name, err)
}
if body.Data.IPv4Routes != 1 || body.Data.IPv6Routes != 1 {
t.Errorf("%s: routes = (v4 %d, v6 %d), want (1, 1)", name, body.Data.IPv4Routes, body.Data.IPv6Routes)
}
wantV4 := []database.PrefixDistribution{{MaskLength: 24, Count: 1}}
if !slices.Equal(body.Data.IPv4PrefixDistribution, wantV4) {
t.Errorf("%s: IPv4 distribution = %v, want %v", name, body.Data.IPv4PrefixDistribution, wantV4)
}
wantV6 := []database.PrefixDistribution{{MaskLength: 32, Count: 1}}
if !slices.Equal(body.Data.IPv6PrefixDistribution, wantV6) {
t.Errorf("%s: IPv6 distribution = %v, want %v", name, body.Data.IPv6PrefixDistribution, wantV6)
}
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the // settledGoroutineCount lets transient goroutines finish, then reports the
// current count. // current count.
func settledGoroutineCount() int { func settledGoroutineCount() int {
+7 -9
View File
@@ -4,9 +4,10 @@ package server
import ( import (
"context" "context"
"net/http" "net/http"
"os" "strconv"
"time" "time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database" "git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/streamer" "git.eeqj.de/sneak/routewatch/internal/streamer"
@@ -33,16 +34,18 @@ type Server struct {
db database.Store db database.Store
streamer *streamer.Streamer streamer *streamer.Streamer
logger *logger.Logger logger *logger.Logger
port int
srv *http.Server srv *http.Server
asnFetcher ASNFetcher asnFetcher ASNFetcher
} }
// New creates a new HTTP server // New creates a new HTTP server
func New(db database.Store, streamer *streamer.Streamer, logger *logger.Logger) *Server { func New(db database.Store, streamer *streamer.Streamer, logger *logger.Logger, cfg *config.Config) *Server {
s := &Server{ s := &Server{
db: db, db: db,
streamer: streamer, streamer: streamer,
logger: logger, logger: logger,
port: cfg.Port,
} }
s.setupRoutes() s.setupRoutes()
@@ -52,11 +55,6 @@ func New(db database.Store, streamer *streamer.Streamer, logger *logger.Logger)
// Start starts the HTTP server // Start starts the HTTP server
func (s *Server) Start() error { func (s *Server) Start() error {
port := os.Getenv("PORT")
if port == "" {
port = "8080"
}
const ( const (
readHeaderTimeout = 40 * time.Second readHeaderTimeout = 40 * time.Second
readTimeout = 60 * time.Second readTimeout = 60 * time.Second
@@ -65,7 +63,7 @@ func (s *Server) Start() error {
) )
s.srv = &http.Server{ s.srv = &http.Server{
Addr: ":" + port, Addr: ":" + strconv.Itoa(s.port),
Handler: s.router, Handler: s.router,
ReadHeaderTimeout: readHeaderTimeout, ReadHeaderTimeout: readHeaderTimeout,
ReadTimeout: readTimeout, ReadTimeout: readTimeout,
@@ -73,7 +71,7 @@ func (s *Server) Start() error {
IdleTimeout: idleTimeout, IdleTimeout: idleTimeout,
} }
s.logger.Info("Starting HTTP server", "port", port, "addr", s.srv.Addr) s.logger.Info("Starting HTTP server", "port", s.port, "addr", s.srv.Addr)
// Start in goroutine but log when actually listening // Start in goroutine but log when actually listening
go func() { go func() {
+15 -3
View File
@@ -210,9 +210,14 @@ func (s *Streamer) Start() error {
// the connection status in metrics. This method is safe to call multiple times. // the connection status in metrics. This method is safe to call multiple times.
func (s *Streamer) Stop() { func (s *Streamer) Stop() {
s.mu.Lock() s.mu.Lock()
if s.cancel != nil { if s.cancel == nil {
s.cancel() // Not started, or already stopped: closing the queues again would panic.
s.mu.Unlock()
return
} }
s.cancel()
s.cancel = nil
// Close all handler queues to signal workers to stop // Close all handler queues to signal workers to stop
for _, info := range s.handlers { for _, info := range s.handlers {
close(info.queue) close(info.queue)
@@ -660,8 +665,15 @@ func (s *Streamer) stream(ctx context.Context) error {
continue continue
} }
// Dispatch to interested handlers // Dispatch to interested handlers. Stop cancels ctx and closes the
// queues under the write lock, so if ctx is cancelled here, under the
// read lock, the queues are closed and must not be sent to.
s.mu.RLock() s.mu.RLock()
if ctx.Err() != nil {
s.mu.RUnlock()
return ctx.Err()
}
for _, info := range s.handlers { for _, info := range s.handlers {
if !info.handler.WantsMessage(msg.Type) { if !info.handler.WantsMessage(msg.Type) {
continue continue
+42
View File
@@ -2,6 +2,8 @@ package streamer
import ( import (
"context" "context"
"errors"
"io"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"runtime" "runtime"
@@ -10,6 +12,7 @@ import (
"git.eeqj.de/sneak/routewatch/internal/logger" "git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics" "git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
) )
func TestNewStreamer(t *testing.T) { func TestNewStreamer(t *testing.T) {
@@ -75,6 +78,45 @@ func TestStreamDoesNotLeakTickersAcrossReconnects(t *testing.T) {
} }
} }
// updateHandler wants UPDATE messages and does nothing with them.
type updateHandler struct{}
func (updateHandler) WantsMessage(messageType string) bool { return messageType == "UPDATE" }
func (updateHandler) HandleMessage(*ristypes.RISMessage) {}
func (updateHandler) QueueCapacity() int { return 10 }
// TestStopBeforeMessageReachesQueues stops the streamer after the read loop
// has checked for cancellation but before it hands the message to the handler
// queues. That is the gap a stop from another goroutine can land in, and it
// used to end in "send on closed channel". The raw handler runs in that gap on
// the read loop itself, so calling Stop from it hits the gap every time.
func TestStopBeforeMessageReachesQueues(t *testing.T) {
const line = `{"type":"ris_message","data":{"type":"UPDATE","peer":"192.0.2.1",` +
`"peer_asn":"64496","timestamp":1700000000}}` + "\n"
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = io.WriteString(w, line)
}))
defer srv.Close()
s := New(logger.New(), metrics.New())
s.url = srv.URL
s.RegisterHandler(updateHandler{})
s.RegisterRawHandler(func(string) { s.Stop() })
// Start would run the stream in the background, where the test cannot
// wait for it. Setting cancel as Start does lets Stop cancel the stream
// run here instead.
ctx, cancel := context.WithCancel(context.Background())
s.cancel = cancel
if err := s.stream(ctx); !errors.Is(err, context.Canceled) {
t.Fatalf("stream returned %v, want %v", err, context.Canceled)
}
// A second Stop must not close the queues again.
s.Stop()
}
// settledGoroutineCount lets transient goroutines finish, then reports the // settledGoroutineCount lets transient goroutines finish, then reports the
// current count. // current count.
func settledGoroutineCount() int { func settledGoroutineCount() int {
+2 -1
View File
@@ -7,7 +7,8 @@ package version
var ( var (
// GitRevision is the git commit hash // GitRevision is the git commit hash
GitRevision = "unknown" GitRevision = "unknown"
// GitRevisionShort is the short git commit hash (7 chars) // GitRevisionShort is the version the page footer shows: the tag or
// short commit hash from `git describe --tags --always`
GitRevisionShort = "unknown" GitRevisionShort = "unknown"
) )
+6 -6
View File
@@ -3,7 +3,8 @@
# this repo. Idempotent: every install is guarded by a check so already # this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew, # installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes NOTHING is present (not git, # or apk (detected in that order); assumes NOTHING is present (not git,
# make, or go). # make, or go). The linter and the tests run only in Docker, as phases of
# the Dockerfile, so nothing installs golangci-lint here.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
@@ -58,11 +59,10 @@ main() {
# Go toolchain # Go toolchain
if missing go; then pkg_install go golang go go; fi if missing go; then pkg_install go golang go go; fi
# golangci-lint: packaged in nix, brew, and apk. There is no apt # Warn, don't fail: make build works without it.
# package; on apt systems install it manually from a hash-verified if missing docker; then
# GitHub release archive (never curl | sh). echo "bootstrap: WARNING: docker not found; install it to run" \
if missing golangci-lint; then "make test, make lint, make check and make docker." >&2
pkg_install golangci-lint golangci-lint golangci-lint golangci-lint
fi fi
go mod download go mod download
+3 -2
View File
@@ -1,7 +1,8 @@
#!/bin/sh #!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own # script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. Must not modify any files. # extension to scripts-to-rule-them-all. test and lint are Docker
# Generic: usually needs no adaptation. # phases; fmt-check is native, because a formatter writes the working
# tree. Must not modify any files.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+19 -5
View File
@@ -1,14 +1,28 @@
#!/bin/sh #!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs script/check # script/cibuild: run the CI build. It bootstraps first: a CI runner
# (via make check), so a successful build implies all checks pass. # checks out and runs this and nothing else, and script/fmt-check runs
# Generic: needs no adaptation. The Gitea workflow runs this on push. # the formatter on the host, which a pristine checkout cannot do.
# --no-cache for the same reason as script/docker: the gate phases the
# final stage depends on are RUN steps, and a cached one is a check that
# did not run.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build . "$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"
+11 -2
View File
@@ -1,7 +1,8 @@
#!/bin/sh #!/bin/sh
# script/docker: build the Docker image tagged with the project name. # script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname. # Identical in all repos; the tag comes from script/projectname.
# Generic: needs no adaptation. # --no-cache because the gate phases the final stage depends on are RUN
# steps, and a cached one is a check that did not run.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -9,7 +10,15 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build -t "$("$SCRIPT_DIR/projectname")" . # Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"
+14 -4
View File
@@ -1,13 +1,23 @@
#!/bin/sh #!/bin/sh
# script/lint: run the linters. # script/lint: run the linter. Linting is a phase of the Dockerfile and
# this builds that phase alone; the linter is never installed or run on
# a developer host, where a shared result cache and a host-global lock
# make its answer untrustworthy.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. The tag makes each build replace the previous image
# instead of leaving a dangling one behind.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
go vet ./... docker build --no-cache \
golangci-lint run --target lint \
-t "$("$SCRIPT_DIR/projectname")-lint" .
} }
main "$@" main "$@"
+10 -8
View File
@@ -1,17 +1,19 @@
#!/bin/sh #!/bin/sh
# script/test: run the test suite. On failure, rerun verbosely so the # script/test: run the test suite. Testing is a phase of the Dockerfile
# failing tests are visible in the output. # and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when
# named, --no-cache because a cached test layer is a test that did not
# run, and a tag so each build replaces the previous image.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
go test -short -timeout 30s -race -cover ./... || { docker build --no-cache \
echo "--- Rerunning with -v for details ---" --target test \
go test -short -timeout 30s -race -v ./... -t "$("$SCRIPT_DIR/projectname")-test" .
exit 1
}
} }
main "$@" main "$@"