Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
793c75751d |
No files matched your search
@@ -13,6 +13,3 @@ indent_style = tab
|
|||||||
|
|
||||||
[*.go]
|
[*.go]
|
||||||
indent_style = tab
|
indent_style = tab
|
||||||
|
|
||||||
# This repository's own sections, such as one for another language it
|
|
||||||
# uses, go below this comment, and a re-vendor keeps them.
|
|
||||||
@@ -1,20 +1,9 @@
|
|||||||
name: check
|
name: check
|
||||||
on: [push]
|
on: [push]
|
||||||
# Free the shared runner: a new push cancels only the same branch's older run.
|
|
||||||
concurrency:
|
|
||||||
group: ${{ github.workflow }}-${{ github.ref }}
|
|
||||||
cancel-in-progress: true
|
|
||||||
jobs:
|
jobs:
|
||||||
check:
|
check:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
# Free the shared runner from a hung build.
|
|
||||||
timeout-minutes: 20
|
|
||||||
steps:
|
steps:
|
||||||
# actions/checkout v4.2.2, 2026-02-22
|
# actions/checkout v4.2.2, 2026-02-22
|
||||||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
||||||
# script/cibuild needs no token, so none is left in .git/config.
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
# All history and tags, so git describe finds the version tag.
|
|
||||||
fetch-depth: 0
|
|
||||||
- run: script/cibuild
|
- run: script/cibuild
|
||||||
+2
-4
@@ -27,7 +27,7 @@ node_modules/
|
|||||||
# Environment files. `*.env` covers bare `.env` and the `prod.env`
|
# Environment files. `*.env` covers bare `.env` and the `prod.env`
|
||||||
# convention. Only the templates `example.env` and `sample.env` are
|
# convention. Only the templates `example.env` and `sample.env` are
|
||||||
# re-included below. A repository that commits any other template adds
|
# re-included below. A repository that commits any other template adds
|
||||||
# its own negation at the end of this file, for example `!.env.example`.
|
# its own negation after these lines, for example `!.env.example`.
|
||||||
*.[eE][nN][vV]
|
*.[eE][nN][vV]
|
||||||
.[eE][nN][vV].*
|
.[eE][nN][vV].*
|
||||||
.[eE][nN][vV][rR][cC]
|
.[eE][nN][vV][rR][cC]
|
||||||
@@ -46,9 +46,7 @@ node_modules/
|
|||||||
[iI][dD]_[eE][dD]25519
|
[iI][dD]_[eE][dD]25519
|
||||||
[iI][dD]_[eE][dD]25519_[sS][kK]
|
[iI][dD]_[eE][dD]25519_[sS][kK]
|
||||||
|
|
||||||
# This repository's own entries, such as its build outputs, go below
|
# This repository's own entries, kept after the canonical content above.
|
||||||
# this comment, and a re-vendor keeps them. Anchor a binary built at the
|
|
||||||
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
|
|
||||||
|
|
||||||
# Go: logs, test binaries, coverage profiles, and the binaries `make build`
|
# Go: logs, test binaries, coverage profiles, and the binaries `make build`
|
||||||
# and `go build` in the repository root write.
|
# and `go build` in the repository root write.
|
||||||
|
|||||||
@@ -25,8 +25,6 @@ linters:
|
|||||||
# silenced by disabling that name, not by enabling the successor.
|
# silenced by disabling that name, not by enabling the successor.
|
||||||
- wsl # Deprecated, replaced by wsl_v5
|
- wsl # Deprecated, replaced by wsl_v5
|
||||||
- gomodguard # Deprecated, replaced by gomodguard_v2
|
- gomodguard # Deprecated, replaced by gomodguard_v2
|
||||||
# Misses findings at random in v2.14.0; back once a pinned release fixes it
|
|
||||||
- canonicalheader
|
|
||||||
settings:
|
settings:
|
||||||
lll:
|
lll:
|
||||||
line-length: 88
|
line-length: 88
|
||||||
|
|||||||
@@ -1,95 +1,97 @@
|
|||||||
# IMPORTANT RULES
|
# IMPORTANT RULES
|
||||||
|
|
||||||
- Claude is an inanimate tool. The spam that Claude attempts to insert into
|
* Claude is an inanimate tool. The spam that Claude attempts to insert into
|
||||||
commit messages (which it erroneously refers to as "attribution") is not
|
commit messages (which it erroneously refers to as "attribution") is not
|
||||||
attribution, as I am the sole author of code created using Claude. It is
|
attribution, as I am the sole author of code created using Claude. It is
|
||||||
corporate advertising for Anthropic and is therefore completely unacceptable
|
corporate advertising for Anthropic and is therefore completely
|
||||||
in commit messages.
|
unacceptable in commit messages.
|
||||||
|
|
||||||
- Tests should always be run before committing code. No commits should be made
|
* Tests should always be run before committing code. No commits should be
|
||||||
that do not pass tests.
|
made that do not pass tests.
|
||||||
|
|
||||||
- Code should always be formatted before committing. Do not commit unformatted
|
* Code should always be formatted before committing. Do not commit
|
||||||
code.
|
unformatted code.
|
||||||
|
|
||||||
- Code should always be linted and linter errors fixed before committing. NEVER
|
* Code should always be linted and linter errors fixed before committing.
|
||||||
commit code that does not pass the linter. DO NOT modify the linter config
|
NEVER commit code that does not pass the linter. DO NOT modify the linter
|
||||||
unless specifically instructed.
|
config unless specifically instructed.
|
||||||
|
|
||||||
- The test suite is fast and local. When running tests, NEVER run individual
|
* The test suite is fast and local. When running tests, NEVER run
|
||||||
parts of the test suite, always run the whole thing by running "make test".
|
individual parts of the test suite, always run the whole thing by running
|
||||||
|
"make test".
|
||||||
|
|
||||||
- Do not stop working on a task until you have reached the definition of done
|
* Do not stop working on a task until you have reached the definition of
|
||||||
provided to you in the initial instruction. Don't do part or most of the work,
|
done provided to you in the initial instruction. Don't do part or most of
|
||||||
do all of the work until the criteria for done are met.
|
the work, do all of the work until the criteria for done are met.
|
||||||
|
|
||||||
- When you complete each task, if the tests are passing and the code is
|
* When you complete each task, if the tests are passing and the code is
|
||||||
formatted and there are no linter errors, always commit and push your work.
|
formatted and there are no linter errors, always commit and push your
|
||||||
Use a good commit message and don't mention any author or co-author
|
work. Use a good commit message and don't mention any author or co-author
|
||||||
attribution.
|
attribution.
|
||||||
|
|
||||||
- Do not create additional files in the root directory of the project without
|
* Do not create additional files in the root directory of the project
|
||||||
asking permission first. Configuration files, documentation, and build files
|
without asking permission first. Configuration files, documentation, and
|
||||||
are acceptable in the root, but source code and other files should be
|
build files are acceptable in the root, but source code and other files
|
||||||
organized in appropriate subdirectories.
|
should be organized in appropriate subdirectories.
|
||||||
|
|
||||||
- Do not use bare strings or numbers in code, especially if they appear anywhere
|
* Do not use bare strings or numbers in code, especially if they appear
|
||||||
more than once. Always define a constant (usually at the top of the file) and
|
anywhere more than once. Always define a constant (usually at the top of
|
||||||
give it a descriptive name, then use that constant in the code instead of the
|
the file) and give it a descriptive name, then use that constant in the
|
||||||
bare string or number.
|
code instead of the bare string or number.
|
||||||
|
|
||||||
- If you are fixing a bug, write a test first that reproduces the bug and fails,
|
* If you are fixing a bug, write a test first that reproduces the bug and
|
||||||
and then fix the bug in the code, using the test to verify that the fix
|
fails, and then fix the bug in the code, using the test to verify that the
|
||||||
worked.
|
fix worked.
|
||||||
|
|
||||||
- When implementing new features, be aware of potential side-effects (such as
|
* When implementing new features, be aware of potential side-effects (such
|
||||||
state files on disk, data in the database, etc.) and ensure that it is
|
as state files on disk, data in the database, etc.) and ensure that it is
|
||||||
possible to mock or stub these side-effects in tests when designing an API.
|
possible to mock or stub these side-effects in tests when designing an
|
||||||
|
API.
|
||||||
|
|
||||||
- When dealing with dates and times or timestamps, always use, display, and
|
* When dealing with dates and times or timestamps, always use, display, and
|
||||||
store UTC. Set the local timezone to UTC on startup. If the user needs to see
|
store UTC. Set the local timezone to UTC on startup. If the user needs
|
||||||
the time in a different timezone, store the user's timezone in a separate
|
to see the time in a different timezone, store the user's timezone in a
|
||||||
field and convert the UTC time to the user's timezone when displaying it. For
|
separate field and convert the UTC time to the user's timezone when
|
||||||
internal use and internal applications and administrative purposes, always
|
displaying it. For internal use and internal applications and
|
||||||
display UTC.
|
administrative purposes, always display UTC.
|
||||||
|
|
||||||
- When implementing programs, put the main.go in ./cmd/<program_name>/main.go
|
* When implementing programs, put the main.go in
|
||||||
and put the program's code in ./internal/<program_name>/. This allows for
|
./cmd/<program_name>/main.go and put the program's code in
|
||||||
multiple programs to be implemented in the same repository without cluttering
|
./internal/<program_name>/. This allows for multiple programs to be
|
||||||
the root directory. main.go should simply import and call
|
implemented in the same repository without cluttering the root directory.
|
||||||
<program_name>.CLIEntry(). The full implementation should be in
|
main.go should simply import and call <program_name>.CLIEntry(). The
|
||||||
./internal/<program_name>/.
|
full implementation should be in ./internal/<program_name>/.
|
||||||
|
|
||||||
- When you are instructed to make the tests pass, DO NOT delete tests, skip
|
* When you are instructed to make the tests pass, DO NOT delete tests, skip
|
||||||
tests, or change the tests specifically to make them pass (unless there is a
|
tests, or change the tests specifically to make them pass (unless there
|
||||||
bug in the test). This is cheating, and it is bad. You should only be
|
is a bug in the test). This is cheating, and it is bad. You should only
|
||||||
modifying the test if it is incorrect or if the test is no longer relevant. In
|
be modifying the test if it is incorrect or if the test is no longer
|
||||||
almost all cases, you should be fixing the code that is being tested, or
|
relevant. In almost all cases, you should be fixing the code that is
|
||||||
updating the tests to match a refactored implementation.
|
being tested, or updating the tests to match a refactored implementation.
|
||||||
|
|
||||||
- Always write a `Makefile` with the default target being `test`, and with a
|
* Always write a `Makefile` with the default target being `test`, and with a
|
||||||
`fmt` target that formats the code. The `test` target should run all tests in
|
`fmt` target that formats the code. The `test` target should run all
|
||||||
the project, and the `fmt` target should format the code. `test` should also
|
tests in the project, and the `fmt` target should format the code. `test`
|
||||||
have a prerequisite target `lint` that should run any linters that are
|
should also have a prerequisite target `lint` that should run any linters
|
||||||
configured for the project.
|
that are configured for the project.
|
||||||
|
|
||||||
- After each completed bugfix or feature, the code must be committed. Do all of
|
* After each completed bugfix or feature, the code must be committed. Do
|
||||||
the pre-commit checks (test, lint, fmt) before committing, of course. After
|
all of the pre-commit checks (test, lint, fmt) before committing, of
|
||||||
each commit, push to the remote.
|
course. After each commit, push to the remote.
|
||||||
|
|
||||||
- Always write tests, even if they are extremely simple and just check for
|
* Always write tests, even if they are extremely simple and just check for
|
||||||
correct syntax (ability to compile/import). If you are writing a new feature,
|
correct syntax (ability to compile/import). If you are writing a new
|
||||||
write a test for it. You don't need to target complete coverage, but you
|
feature, write a test for it. You don't need to target complete coverage,
|
||||||
should at least test any new functionality you add.
|
but you should at least test any new functionality you add.
|
||||||
|
|
||||||
- Always use structured logging. Log any relevant state/context with the
|
* Always use structured logging. Log any relevant state/context with the
|
||||||
messages (but do not log secrets). If stdout is not a terminal, output the
|
messages (but do not log secrets). If stdout is not a terminal, output
|
||||||
structured logs in jsonl format. Use go's log/slog.
|
the structured logs in jsonl format. Use go's log/slog.
|
||||||
|
|
||||||
- You do not need to summarize your changes in the chat after making them.
|
* You do not need to summarize your changes in the chat after making them.
|
||||||
Making the changes and committing them is sufficient. If anything out of the
|
Making the changes and committing them is sufficient. If anything out of
|
||||||
ordinary happened, please explain it, but in the normal case where you found
|
the ordinary happened, please explain it, but in the normal case where you
|
||||||
and fixed the bug, or implemented the feature, there is no need for the
|
found and fixed the bug, or implemented the feature, there is no need for
|
||||||
end-of-change summary.
|
the end-of-change summary.
|
||||||
|
|
||||||
- When testing daemons, use a 15 second timeout always.
|
* When testing daemons, use a 15 second timeout always.
|
||||||
+37
-54
@@ -1,46 +1,37 @@
|
|||||||
# In every stage ARG CACHEBUST comes right after the steps that install
|
# Lint phase, built alone by script/lint. The linter is invoked directly
|
||||||
# system packages and download Go's modules, or right after FROM in a
|
# rather than through `make lint`, which is itself a docker build and
|
||||||
# stage with neither, and every step that lints, tests or compiles
|
# would recurse into a daemon that does not exist in a build step. The
|
||||||
# comes after it. script/ passes a new value on every build, which
|
# image is Debian-based and has Go and gcc, so it can check the cgo
|
||||||
# reruns every RUN step after it. Docker keeps the layers above it
|
# SQLite package without extra installs.
|
||||||
# until the base image, their own lines or go.mod and go.sum change.
|
# golangci/golangci-lint:v2.14.0 (Go 1.27.0), 2026-10-14
|
||||||
# The steps that compile mount Go's build cache, and golangci-lint run
|
|
||||||
# also mounts golangci-lint's; both survive from build to build.
|
|
||||||
|
|
||||||
# Lint phase. The image is Debian-based and has gcc, so it can check the
|
|
||||||
# cgo SQLite package without extra installs.
|
|
||||||
# golangci/golangci-lint:v2.14.0 (Go 1.27.0), 2026-10-19
|
|
||||||
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
|
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
ARG CACHEBUST
|
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN golangci-lint run --config .golangci.yml ./...
|
||||||
--mount=type=cache,target=/root/.cache/golangci-lint \
|
|
||||||
golangci-lint run --config .golangci.yml ./...
|
|
||||||
|
|
||||||
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
# Test phase, built alone by script/test. -race needs cgo and so a C
|
||||||
# image ships and the alpine one does not. -count=1 because the mounted
|
# compiler, which the Debian Go image ships. -short skips the live RIPE RIS
|
||||||
# cache also holds test results, which go test would otherwise replay.
|
# feed test, so the suite needs no network. The rerun with -v only shows
|
||||||
# -short skips the live RIS Live feed test, so the suite needs no network.
|
# details: the build fails however it ends, because the first run failed.
|
||||||
# golang:1.24-bookworm, 2026-09-21
|
# golang:1.24-bookworm, 2026-09-21
|
||||||
FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS test
|
FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS test
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
ARG CACHEBUST
|
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN go test -short -timeout 90s -race -cover ./... || \
|
||||||
go test -short -count=1 -timeout 90s -race -cover ./... || \
|
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
go test -short -count=1 -timeout 90s -race -v ./...; exit 1; }
|
go test -short -timeout 90s -race -v ./...; exit 1; }
|
||||||
|
|
||||||
# Build stage. Nothing is wanted from either phase above; the copies
|
# Build stage. Nothing is wanted from the lint and test phases; the copies
|
||||||
# are what make BuildKit build them first, so nothing after them can
|
# are what make BuildKit build them first, so this stage cannot run unless
|
||||||
# run unless lint and test passed.
|
# lint and test passed.
|
||||||
# golang:1.24-bookworm, 2026-09-21
|
# golang:1.24-bookworm, 2026-09-21
|
||||||
FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS builder
|
FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS builder
|
||||||
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
|
COPY --from=test /src/go.sum /dev/null
|
||||||
|
|
||||||
# Install build dependencies (zstd for archive, gcc for CGO/sqlite3)
|
# Install build dependencies (zstd for archive, gcc for CGO/sqlite3)
|
||||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||||
@@ -49,39 +40,33 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
|||||||
libc6-dev \
|
libc6-dev \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
WORKDIR /src
|
|
||||||
COPY go.mod go.sum ./
|
|
||||||
RUN go mod download
|
|
||||||
ARG CACHEBUST
|
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
|
||||||
COPY --from=test /src/go.sum /dev/null
|
|
||||||
# A tar-stream context keeps the sender's file owners, which git refuses.
|
# A tar-stream context keeps the sender's file owners, which git refuses.
|
||||||
RUN git config --system --add safe.directory /src
|
RUN git config --system --add safe.directory /src
|
||||||
|
|
||||||
|
WORKDIR /src
|
||||||
|
|
||||||
|
# Copy everything
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# Vendor dependencies for the source archive (must be after copying source)
|
# Vendor dependencies (must be after copying source)
|
||||||
RUN go mod vendor
|
RUN go mod download && go mod vendor
|
||||||
|
|
||||||
# Build the binary with CGO enabled (required for sqlite3). The version the
|
# Build the binary with CGO enabled (required for sqlite3). The version the
|
||||||
# page footer shows is the VERSION build arg when one is given, otherwise
|
# page footer shows is the VERSION build argument when one is given, otherwise
|
||||||
# `git describe --tags --always` on the .git in the build context: the tag
|
# `git describe --tags --always` of the .git in the build context (git comes
|
||||||
# on a tagged commit, tag-N-gHASH after one, the short commit when no tag is
|
# with this image): the tag on a tagged commit, tag-N-gHASH after one, the
|
||||||
# reachable. With .git present, a version that is still empty, dev or
|
# short commit when no tag is reachable. A context that carries .git and still
|
||||||
# unknown fails the build: git is missing or could not read the checkout.
|
# yields no version fails the build. The footer links to the full commit.
|
||||||
# Without .git the footer says unknown. The footer links to the full commit.
|
|
||||||
ARG VERSION
|
ARG VERSION
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN version="${VERSION:-$(git describe --tags --always || echo unknown)}"; \
|
||||||
VERSION="${VERSION:-$(git describe --tags --always)}"; \
|
if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
|
||||||
if [ -e .git ]; then \
|
[ "$version" = unknown ]; }; then \
|
||||||
case "$VERSION" in ""|dev|unknown) \
|
echo "no version could be derived although the build context carries .git" >&2; \
|
||||||
echo "version is '$VERSION' although .git is present" >&2; \
|
exit 1; \
|
||||||
exit 1 ;; \
|
|
||||||
esac; \
|
|
||||||
fi; \
|
fi; \
|
||||||
REVISION="$(git rev-parse --verify HEAD || echo unknown)"; \
|
|
||||||
CGO_ENABLED=1 GOOS=linux go build -o /routewatch -ldflags "\
|
CGO_ENABLED=1 GOOS=linux go build -o /routewatch -ldflags "\
|
||||||
-X git.eeqj.de/sneak/routewatch/internal/version.GitRevision=$REVISION \
|
-X git.eeqj.de/sneak/routewatch/internal/version.GitRevision=$(git rev-parse --verify HEAD || echo unknown) \
|
||||||
-X git.eeqj.de/sneak/routewatch/internal/version.GitRevisionShort=${VERSION:-unknown}" \
|
-X git.eeqj.de/sneak/routewatch/internal/version.GitRevisionShort=$version" \
|
||||||
./cmd/routewatch
|
./cmd/routewatch
|
||||||
|
|
||||||
# Create source archive with vendored dependencies
|
# Create source archive with vendored dependencies
|
||||||
@@ -90,7 +75,7 @@ RUN tar --zstd -cf /routewatch-source.tar.zst \
|
|||||||
--exclude='*.tar.zst' \
|
--exclude='*.tar.zst' \
|
||||||
.
|
.
|
||||||
|
|
||||||
# Runtime stage, and the last one
|
# Runtime stage
|
||||||
# debian:bookworm-slim, 2026-09-21
|
# debian:bookworm-slim, 2026-09-21
|
||||||
FROM debian@sha256:3783cc01769c7b2b1b83a5c5ad96c815348e28ed7da68e2e3687004faa906251
|
FROM debian@sha256:3783cc01769c7b2b1b83a5c5ad96c815348e28ed7da68e2e3687004faa906251
|
||||||
|
|
||||||
@@ -102,8 +87,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
|||||||
curl \
|
curl \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
ARG CACHEBUST
|
|
||||||
|
|
||||||
# Create non-root user
|
# Create non-root user
|
||||||
RUN useradd -r -u 1000 -m routewatch
|
RUN useradd -r -u 1000 -m routewatch
|
||||||
|
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
|
|||||||
service, maintains a live routing table in SQLite, and provides HTTP APIs for
|
service, maintains a live routing table in SQLite, and provides HTTP APIs for
|
||||||
querying routing information.
|
querying routing information.
|
||||||
|
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
- Real-time streaming of BGP updates from RIPE RIS Live
|
- Real-time streaming of BGP updates from RIPE RIS Live
|
||||||
@@ -37,7 +38,6 @@ DEBUG=routewatch ./routewatch
|
|||||||
## HTTP Endpoints
|
## HTTP Endpoints
|
||||||
|
|
||||||
### Web Interface
|
### Web Interface
|
||||||
|
|
||||||
- `GET /` - Redirects to /status
|
- `GET /` - Redirects to /status
|
||||||
- `GET /status` - HTML status dashboard
|
- `GET /status` - HTML status dashboard
|
||||||
- `GET /status.json` - JSON statistics
|
- `GET /status.json` - JSON statistics
|
||||||
@@ -48,7 +48,6 @@ DEBUG=routewatch ./routewatch
|
|||||||
- `GET /ip/{ip}` - Redirects to prefix containing the IP
|
- `GET /ip/{ip}` - Redirects to prefix containing the IP
|
||||||
|
|
||||||
### API v1
|
### API v1
|
||||||
|
|
||||||
- `GET /api/v1/stats` - Detailed statistics with handler metrics
|
- `GET /api/v1/stats` - Detailed statistics with handler metrics
|
||||||
- `GET /api/v1/ip/{ip}` - Look up AS information for an IP address
|
- `GET /api/v1/ip/{ip}` - Look up AS information for an IP address
|
||||||
- `GET /api/v1/as/{asn}` - Get prefixes announced by an AS
|
- `GET /api/v1/as/{asn}` - Get prefixes announced by an AS
|
||||||
@@ -108,54 +107,49 @@ routewatch/
|
|||||||
### Execution Flow
|
### Execution Flow
|
||||||
|
|
||||||
1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
|
1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
|
||||||
- Uber fx dependency injection initializes all components
|
- Uber fx dependency injection initializes all components
|
||||||
- Signal handlers registered for graceful shutdown
|
- Signal handlers registered for graceful shutdown
|
||||||
|
|
||||||
2. **Initialization** (`internal/routewatch/app.go`)
|
2. **Initialization** (`internal/routewatch/app.go`)
|
||||||
- Database created with SQLite schema (WAL mode, 3GB cache)
|
- Database created with SQLite schema (WAL mode, 3GB cache)
|
||||||
- Message handlers registered with the streamer
|
- Message handlers registered with the streamer
|
||||||
- HTTP server started on configured port
|
- HTTP server started on configured port
|
||||||
|
|
||||||
3. **Message Processing Pipeline**
|
3. **Message Processing Pipeline**
|
||||||
|
```
|
||||||
```
|
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite
|
||||||
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite
|
```
|
||||||
```
|
- Streamer connects to `ris-live.ripe.net` via HTTP
|
||||||
|
- Parses BGP UPDATE messages from JSON stream
|
||||||
- Streamer connects to `ris-live.ripe.net` via HTTP
|
- Dispatches to registered handlers based on message type
|
||||||
- Parses BGP UPDATE messages from JSON stream
|
- Each handler has its own queue with backpressure handling
|
||||||
- Dispatches to registered handlers based on message type
|
- Handlers batch writes for efficiency (25K-30K ops, 1-2s timeout)
|
||||||
- Each handler has its own queue with backpressure handling
|
|
||||||
- Handlers batch writes for efficiency (25K-30K ops, 1-2s timeout)
|
|
||||||
|
|
||||||
4. **Handler Details**
|
4. **Handler Details**
|
||||||
- **ASHandler**: Tracks all ASNs seen in AS paths
|
- **ASHandler**: Tracks all ASNs seen in AS paths
|
||||||
- **PeerHandler**: Records BGP peer information
|
- **PeerHandler**: Records BGP peer information
|
||||||
- **PrefixHandler**: Maintains live routing table (upserts on announcement,
|
- **PrefixHandler**: Maintains live routing table (upserts on announcement, deletes on withdrawal)
|
||||||
deletes on withdrawal)
|
- **PeeringHandler**: Extracts AS peering relationships from AS paths
|
||||||
- **PeeringHandler**: Extracts AS peering relationships from AS paths
|
|
||||||
|
|
||||||
5. **HTTP Request Flow**
|
5. **HTTP Request Flow**
|
||||||
```
|
```
|
||||||
Request → Chi Router → Middleware (timeout, logging) → Handler → Database Query → Response
|
Request → Chi Router → Middleware (timeout, logging) → Handler → Database Query → Response
|
||||||
```
|
```
|
||||||
|
|
||||||
### Key Design Patterns
|
### Key Design Patterns
|
||||||
|
|
||||||
- **Batched Writes**: All database operations are batched for performance
|
- **Batched Writes**: All database operations are batched for performance
|
||||||
- **Backpressure**: Probabilistic message dropping when queues exceed 50%
|
- **Backpressure**: Probabilistic message dropping when queues exceed 50% capacity
|
||||||
capacity
|
|
||||||
- **Graceful Shutdown**: 60-second timeout, flushes all pending batches
|
- **Graceful Shutdown**: 60-second timeout, flushes all pending batches
|
||||||
- **Reconnection**: Any end of the stream other than shutdown, including RIS
|
- **Reconnection**: Any end of the stream other than shutdown, including RIS Live
|
||||||
Live ending the response normally, marks the feed disconnected and reconnects
|
ending the response normally, marks the feed disconnected and reconnects with
|
||||||
with exponential backoff (5s-320s). The backoff resets after a stream that
|
exponential backoff (5s-320s). The backoff resets after a stream that lasted
|
||||||
lasted more than 30s. A stream with no line for 60s is dropped and
|
more than 30s. A stream with no line for 60s is dropped and reconnected.
|
||||||
reconnected.
|
|
||||||
- **IP Lookup**: the most specific live route is found by looking up the
|
- **IP Lookup**: the most specific live route is found by looking up the
|
||||||
address's prefix at each mask length, longest first, in the prefix index (at
|
address's prefix at each mask length, longest first, in the prefix index
|
||||||
most 25 lookups for IPv4, 113 for IPv6); prefixes are stored in the text form
|
(at most 25 lookups for IPv4, 113 for IPv6); prefixes are stored in the
|
||||||
Go's `net/netip` prints. Routes shorter than /8 for IPv4 or /16 for IPv6, such
|
text form Go's `net/netip` prints. Routes shorter than /8 for IPv4 or /16
|
||||||
as a default route, are not used.
|
for IPv6, such as a default route, are not used.
|
||||||
|
|
||||||
### Database Schema
|
### Database Schema
|
||||||
|
|
||||||
@@ -180,24 +174,23 @@ bgp_peers(id, peer_ip, peer_asn, last_message_type, last_seen)
|
|||||||
|
|
||||||
Configuration is handled via environment variables and OS-specific paths:
|
Configuration is handled via environment variables and OS-specific paths:
|
||||||
|
|
||||||
| Variable | Default | Description |
|
| Variable | Default | Description |
|
||||||
| ------------------ | -------------------------------- | ----------------------------------------------------------- |
|
|----------|----------|-------------|
|
||||||
| `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 |
|
| `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 |
|
||||||
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
|
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
|
||||||
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
|
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
|
||||||
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
|
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
|
||||||
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory |
|
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory |
|
||||||
|
|
||||||
A variable that is set to an invalid value stops the start with an error and a
|
A variable that is set to an invalid value stops the start with an error and a
|
||||||
non-zero exit. An empty variable counts as unset.
|
non-zero exit. An empty variable counts as unset.
|
||||||
|
|
||||||
State directory (database location):
|
State directory (database location):
|
||||||
|
|
||||||
- macOS: `~/Library/Application Support/routewatch/`
|
- macOS: `~/Library/Application Support/routewatch/`
|
||||||
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root, otherwise
|
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root,
|
||||||
`$XDG_DATA_HOME/berlin.sneak.app.routewatch/` (with `XDG_DATA_HOME` unset,
|
otherwise `$XDG_DATA_HOME/berlin.sneak.app.routewatch/` (with `XDG_DATA_HOME`
|
||||||
`~/.local/share/berlin.sneak.app.routewatch/`). In the Docker image this is
|
unset, `~/.local/share/berlin.sneak.app.routewatch/`). In the Docker image
|
||||||
`/var/lib/berlin.sneak.app.routewatch/`.
|
this is `/var/lib/berlin.sneak.app.routewatch/`.
|
||||||
|
|
||||||
## Memory
|
## Memory
|
||||||
|
|
||||||
@@ -205,7 +198,6 @@ The daemon holds a live routing table, so its memory grows with the size of the
|
|||||||
data it tracks. The image sets ceilings that keep it inside a 5 GiB container.
|
data it tracks. The image sets ceilings that keep it inside a 5 GiB container.
|
||||||
|
|
||||||
Budget:
|
Budget:
|
||||||
|
|
||||||
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image).
|
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image).
|
||||||
- SQLite: at most 640 MiB of page cache across the connection pool (64 MiB per
|
- SQLite: at most 640 MiB of page cache across the connection pool (64 MiB per
|
||||||
connection, 10 connections) and a 1.5 GiB hard heap limit for the C library.
|
connection, 10 connections) and a 1.5 GiB hard heap limit for the C library.
|
||||||
@@ -228,12 +220,11 @@ or the equivalent in your deployment tool. This leaves headroom above the
|
|||||||
ceilings for spikes and the kernel page cache.
|
ceilings for spikes and the kernel page cache.
|
||||||
|
|
||||||
Override the Go soft limit by setting `GOMEMLIMIT` in the environment (for
|
Override the Go soft limit by setting `GOMEMLIMIT` in the environment (for
|
||||||
example `-e GOMEMLIMIT=1GiB`); this replaces the image default.
|
example `-e GOMEMLIMIT=1GiB`); this replaces the image default. `MALLOC_ARENA_MAX`
|
||||||
`MALLOC_ARENA_MAX` can be overridden the same way, but raising it lets RSS climb
|
can be overridden the same way, but raising it lets RSS climb again on a
|
||||||
again on a many-core host.
|
many-core host.
|
||||||
|
|
||||||
What happens at each limit:
|
What happens at each limit:
|
||||||
|
|
||||||
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
|
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
|
||||||
collection more aggressively rather than growing further.
|
collection more aggressively rather than growing further.
|
||||||
- SQLite: at a 1 GiB soft heap limit it recycles its page cache instead of
|
- SQLite: at a 1 GiB soft heap limit it recycles its page cache instead of
|
||||||
@@ -242,15 +233,6 @@ What happens at each limit:
|
|||||||
keeps running.
|
keeps running.
|
||||||
- Handler queues: each of the four handler queues holds at most 20,000 messages.
|
- Handler queues: each of the four handler queues holds at most 20,000 messages.
|
||||||
When a queue fills, the streamer drops messages instead of blocking.
|
When a queue fills, the streamer drops messages instead of blocking.
|
||||||
- Handler batches: while the AS, peer or prefix handler writes a batch to the
|
|
||||||
database, the updates that arrive collect in its next batch, which stops at
|
|
||||||
ten times the batch size: 300,000 updates for the AS handler
|
|
||||||
(`asnBatchMaxSize`, about 10 MiB), 100,000 for the peer handler
|
|
||||||
(`peerBatchMaxSize`, about 10 MiB) and 250,000 for the prefix handler
|
|
||||||
(`prefixBatchMaxSize`, about 60 MiB). At that size the handler takes nothing
|
|
||||||
more off its queue until the write ends, so the queue fills and the streamer
|
|
||||||
drops messages as above. With the batches being written, the three handlers'
|
|
||||||
batches take at most about 0.25 GiB of the Go heap's 1.5 GiB.
|
|
||||||
|
|
||||||
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
|
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
|
||||||
with the goroutine count and Go memory figures.
|
with the goroutine count and Go memory figures.
|
||||||
@@ -268,8 +250,8 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
|
|||||||
limit, so on a host with swap Docker allows the same amount of swap again.
|
limit, so on a host with swap Docker allows the same amount of swap again.
|
||||||
- Health check: the image's `HEALTHCHECK` requests
|
- Health check: the image's `HEALTHCHECK` requests
|
||||||
`/.well-known/healthcheck.json` on the container port. upaas reads the
|
`/.well-known/healthcheck.json` on the container port. upaas reads the
|
||||||
container's health 60 seconds after a deploy and fails the deploy unless it is
|
container's health 60 seconds after a deploy and fails the deploy unless it
|
||||||
`healthy`.
|
is `healthy`.
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
@@ -292,40 +274,41 @@ make
|
|||||||
This repository adheres to the
|
This repository adheres to the
|
||||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||||
standard: normalized scripts in `script/` are the entrypoints for the
|
standard: normalized scripts in `script/` are the entrypoints for the
|
||||||
development workflow, and the Makefile targets are thin shims that call them. We
|
development workflow, and the Makefile targets are thin shims that call
|
||||||
provide:
|
them. We provide:
|
||||||
|
|
||||||
- `script/bootstrap` — install all development dependencies (git, make, go, the
|
- `script/bootstrap` — install all development dependencies (git, make, go, Go
|
||||||
pinned node and yarn, prettier from `yarn.lock`, Go module download)
|
module download); warns when docker is missing
|
||||||
- `script/setup` — make a fresh clone ready for development: runs
|
- `script/setup` — make a fresh clone ready for development: runs
|
||||||
`script/bootstrap`, then `script/install-precommit`
|
`script/bootstrap`, then `script/install-precommit`
|
||||||
- `script/projectname` — print the project name (used for the Docker image tag)
|
- `script/projectname` — print the project name (used for the Docker
|
||||||
- `script/test` — build the `test` phase of the `Dockerfile` alone, which runs
|
image tag)
|
||||||
`go test -short -count=1 -timeout 90s -race -cover ./...` (verbose rerun on
|
- `script/test` — build the `test` phase alone, which runs
|
||||||
failure). The `-short` flag skips the live-network integration test so the run
|
`go test -short -timeout 90s -race -cover ./...` (verbose rerun on failure).
|
||||||
is deterministic and offline.
|
The `-short` flag skips the live-network integration test so the run is
|
||||||
- `script/lint` — build the `lint` phase of the `Dockerfile` alone, which runs
|
deterministic and offline.
|
||||||
golangci-lint v2.14.0 with `.golangci.yml`
|
- `script/lint` — build the `lint` phase alone, which runs golangci-lint
|
||||||
- `script/fmt` — format the Go code with `go fmt` and the markdown with prettier
|
v2.14.0 with `.golangci.yml`
|
||||||
(writes)
|
- `script/fmt` — format all code (writes)
|
||||||
- `script/fmt-check` — check formatting (read-only)
|
- `script/fmt-check` — check formatting (read-only)
|
||||||
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
|
- `script/check` — run `script/test`, `script/lint`, and
|
||||||
- `script/docker` — build the Docker image tagged via `script/projectname`
|
`script/fmt-check`
|
||||||
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
|
- `script/docker` — build the Docker image tagged via
|
||||||
then builds the image
|
`script/projectname`
|
||||||
- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must not
|
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and
|
||||||
change files), then `script/check`
|
`script/check`, then builds the image
|
||||||
- `script/install-precommit` — install the git pre-commit hook that runs
|
- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
|
||||||
`script/precommit`
|
not change files), then `script/check`
|
||||||
|
- `script/install-precommit` — install the git pre-commit hook that
|
||||||
|
runs `script/precommit`
|
||||||
|
|
||||||
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
|
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
|
||||||
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
|
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
|
||||||
in `script/` passes a new `CACHEBUST` value, so each run executes every check,
|
in `script/` passes `--no-cache`, so each run executes every check.
|
||||||
while Go's build cache and golangci-lint's cache are kept between builds.
|
|
||||||
|
|
||||||
The live-network integration test `TestRouteWatchLiveFeed` streams the RIPE RIS
|
The live-network integration test `TestRouteWatchLiveFeed` streams the RIPE
|
||||||
feed for a few seconds and is skipped in short mode. To run it on demand, invoke
|
RIS feed for a few seconds and is skipped in short mode. To run it on demand,
|
||||||
`go test` directly without `-short`:
|
invoke `go test` directly without `-short`:
|
||||||
`go test -run TestRouteWatchLiveFeed ./internal/routewatch/`.
|
`go test -run TestRouteWatchLiveFeed ./internal/routewatch/`.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|||||||
+86
-196
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Repository Policies
|
title: Repository Policies
|
||||||
last_modified: 2026-10-18
|
last_modified: 2026-10-04
|
||||||
---
|
---
|
||||||
|
|
||||||
This document covers repository structure, tooling, and workflow standards. Code
|
This document covers repository structure, tooling, and workflow standards. Code
|
||||||
@@ -54,39 +54,34 @@ style conventions are in separate documents:
|
|||||||
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
|
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
|
||||||
assumes nothing is present: base tools come from nix, apt, brew, or apk
|
assumes nothing is present: base tools come from nix, apt, brew, or apk
|
||||||
(detected in that order; apt runs noninteractive). For node it uses the
|
(detected in that order; apt runs noninteractive). For node it uses the
|
||||||
installed node only when its major version is the pinned one; otherwise (node
|
installed node if present; otherwise it installs a PINNED node version via
|
||||||
missing, or another major, such as the node 24 the CI runner image ships) it
|
nvm, first installing nvm itself if missing — from a hash-verified GitHub
|
||||||
installs the PINNED node version via nvm and installs yarn and the packages
|
release archive (never `curl | sh`), with bash installed as an explicit
|
||||||
under it. nvm itself is installed first if missing, from a hash-verified
|
prerequisite since nvm requires bash. yarn is then pinned via
|
||||||
GitHub release archive (never `curl | sh`), with bash installed as an explicit
|
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
|
||||||
prerequisite since nvm requires bash. Only the major version is compared
|
always exact versions. `script/cibuild` runs the CI build: it changes to the
|
||||||
because the `Dockerfile` stages start from a node image whose exact version is
|
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
|
||||||
not the pin, and nvm cannot install a prebuilt node on alpine. yarn is pinned
|
with the version; the Gitea workflow calls it. **`script/cibuild` runs
|
||||||
via `corepack prepare yarn@<version> --activate`. Never install "latest" or
|
|
||||||
"lts"; always exact versions. `script/cibuild` runs the CI build: it changes
|
|
||||||
to the repo root, runs `script/bootstrap`, runs `script/check`, and builds the
|
|
||||||
image with the version; the Gitea workflow calls it. **`script/cibuild` runs
|
|
||||||
`script/bootstrap` first**, because the workflow checks out the repo and runs
|
`script/bootstrap` first**, because the workflow checks out the repo and runs
|
||||||
nothing else, while `script/fmt-check` runs the formatter on the host: on a
|
nothing else, while `script/fmt-check` runs the formatter on the host: on a
|
||||||
pristine checkout with nothing installed the run dies there, after the
|
pristine checkout with nothing installed the run dies there, after the
|
||||||
containerised gates have passed. **The bootstrap alone is not enough**: when
|
containerised gates have passed. **The bootstrap alone is not enough**:
|
||||||
`script/bootstrap` installs node and yarn under nvm it leaves neither on the
|
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
|
||||||
`PATH` of the shell that called it, so a bare `yarn` either exits 127 or runs
|
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
|
||||||
under another node. The host entrypoints that need yarn — `script/fmt` and
|
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
|
||||||
`script/fmt-check` — therefore use the same test as `script/bootstrap`: when
|
source nvm for the pinned node version before invoking it, exactly as
|
||||||
the node on `PATH` has the pinned major version they run the `yarn` on `PATH`,
|
`script/bootstrap`'s own install step does. A runner carrying nothing but
|
||||||
and otherwise they load nvm through `$HOME/.nvm/nvm.sh` and run yarn under the
|
docker and git then gets through `script/check`. Four further scripts are our
|
||||||
pinned node. A runner carrying nothing but docker and git then gets through
|
own extensions to the standard: `script/check` runs `script/test`,
|
||||||
`script/check`. Four further scripts are our own extensions to the standard:
|
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
|
||||||
`script/check` runs `script/test`, `script/lint` and `script/fmt-check`;
|
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
|
||||||
`script/precommit` is what the git pre-commit hook runs, and it calls
|
installs the git pre-commit hook (the `make hooks` target shims to it); and
|
||||||
`script/check`; `script/install-precommit` installs the git pre-commit hook
|
`script/projectname` (literally that filename) simply outputs the project's
|
||||||
(the `make hooks` target shims to it); and `script/projectname` (literally
|
name. Scripts that need the name call `script/projectname` — e.g.
|
||||||
that filename) simply outputs the project's name. Scripts that need the name
|
`script/docker` assembles its image tag from it — so those scripts stay
|
||||||
call `script/projectname` — e.g. `script/docker` assembles its image tag from
|
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
|
||||||
it — so those scripts stay byte-identical across all repos. Repo-type-specific
|
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
|
||||||
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
|
the hook itself. Model scripts are at
|
||||||
`script/precommit`, not in the hook itself. Model scripts are at
|
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
||||||
must document the provided scripts in an **Entrypoints** section (see the
|
must document the provided scripts in an **Entrypoints** section (see the
|
||||||
README requirements below).
|
README requirements below).
|
||||||
@@ -123,10 +118,8 @@ style conventions are in separate documents:
|
|||||||
and nothing else:
|
and nothing else:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
|
||||||
--target lint --output type=cacheonly .
|
docker build --no-cache --target test -t "$(script/projectname)-test" .
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--target test --output type=cacheonly .
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**A stage that is not the last one in the file is built only when the final
|
**A stage that is not the last one in the file is built only when the final
|
||||||
@@ -136,15 +129,10 @@ style conventions are in separate documents:
|
|||||||
plain `docker build .` builds the last stage alone and exits 0 having linted
|
plain `docker build .` builds the last stage alone and exits 0 having linted
|
||||||
and tested nothing.
|
and tested nothing.
|
||||||
|
|
||||||
**The gate builds write no image.** With `--output type=cacheonly` the phase
|
**Every `docker build` in `script/` is tagged**, here and in
|
||||||
runs and a failing step fails the build, but the result is not exported.
|
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
|
||||||
Nothing uses those images, and writing one out is slow: a Go test phase's
|
image behind on every invocation, on every developer host and every CI
|
||||||
image holds the toolchain. A build given neither `--output` nor `-t` writes
|
runner; a tagged one replaces the previous image.
|
||||||
an untagged image and leaves it dangling, on every developer host and every
|
|
||||||
CI runner. `script/cibuild` and `script/docker` build the image that ships
|
|
||||||
and tag it, so each build replaces the previous image; each assigns the tag
|
|
||||||
on its own line before the build, so `set -e` stops it where
|
|
||||||
`script/projectname` fails.
|
|
||||||
|
|
||||||
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
|
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
|
||||||
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
|
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
|
||||||
@@ -159,42 +147,22 @@ style conventions are in separate documents:
|
|||||||
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
|
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
|
||||||
exit non-zero with `parallel golangci-lint is running` — a status a caller
|
exit non-zero with `parallel golangci-lint is running` — a status a caller
|
||||||
cannot tell from real findings. Both have produced wrong verdicts in this
|
cannot tell from real findings. Both have produced wrong verdicts in this
|
||||||
org, in both directions. A container has its own `TMPDIR`, so the lock is
|
org, in both directions. A container has its own cache, its own `TMPDIR` and
|
||||||
out of reach. The lint phase does keep golangci-lint's cache between check
|
a digest-pinned binary, so neither is reachable.
|
||||||
builds (below), but it lints every checkout at `/src`, and a finding is
|
|
||||||
taken from that cache only for the same files, settings and golangci-lint
|
|
||||||
version.
|
|
||||||
|
|
||||||
- **Any build that runs checks is given a new `CACHEBUST` value.** Docker
|
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
|
||||||
invalidates a `COPY` layer only when the copied content changes, so on an
|
a `COPY` layer only when the copied content changes, so on an unchanged tree
|
||||||
unchanged tree the check `RUN` is served from cache, nothing executes, and the
|
the check `RUN` is served from cache, nothing executes, and the build still
|
||||||
build still exits 0. Every `docker build` in `script/` therefore passes
|
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
|
||||||
`--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the script's process
|
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
|
||||||
id, so no two builds share a value, not even the three in one
|
four, and there is no fifth — `script/check` runs the two gate phases and
|
||||||
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
|
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
|
||||||
`script/docker` are the four, and there is no fifth — `script/check` runs the
|
not evidence that anything ran: a sub-second build reporting success is a
|
||||||
two gate phases and `script/fmt-check`, and builds no image of its own. In
|
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
|
||||||
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
|
and friends destroy a build cache shared with every other build on the host.
|
||||||
that install system packages and download Go's modules, or right after `FROM`
|
When a check is added or changed, prove it works by planting a defect it must
|
||||||
in a stage with neither, and every step that lints, tests or compiles comes
|
catch and watching the run fail on it, then revert the defect. A green run
|
||||||
after it. A step that installs anything else as well, such as a
|
alone shows neither that the check ran nor that it covers what it should.
|
||||||
`script/bootstrap` that also installs node packages, comes after it too. A new
|
|
||||||
value reruns every `RUN` step after that line; `COPY` steps are compared by
|
|
||||||
content and need nothing. Docker keeps the layers above the line until the
|
|
||||||
base image, their own lines or the files they copy change. A stage without the
|
|
||||||
`ARG` line is served from cache on an unchanged tree, so a repository
|
|
||||||
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
|
|
||||||
not pass `--no-cache`: it reruns every step too, but it also builds the layers
|
|
||||||
above the line again and empties every cache mount, so Go's build cache and
|
|
||||||
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
|
|
||||||
survive between check builds, would start empty every time. A bare
|
|
||||||
`docker build .` is not evidence that anything ran: a sub-second build
|
|
||||||
reporting success is a cache hit, not a result. Never invalidate by pruning —
|
|
||||||
`docker builder prune` and friends destroy a build cache shared with every
|
|
||||||
other build on the host. When a check is added or changed, prove it works by
|
|
||||||
planting a defect it must catch and watching the run fail on it, then revert
|
|
||||||
the defect. A green run alone shows neither that the check ran nor that it
|
|
||||||
covers what it should.
|
|
||||||
|
|
||||||
- **The gate phases are separate stages, and the build stage depends on both.**
|
- **The gate phases are separate stages, and the build stage depends on both.**
|
||||||
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
||||||
@@ -203,56 +171,40 @@ style conventions are in separate documents:
|
|||||||
`Dockerfile`:
|
`Dockerfile`:
|
||||||
|
|
||||||
```dockerfile
|
```dockerfile
|
||||||
# In every stage ARG CACHEBUST comes right after the steps that install
|
|
||||||
# system packages and download Go's modules, or right after FROM in a
|
|
||||||
# stage with neither, and every step that lints, tests or compiles
|
|
||||||
# comes after it. script/ passes a new value on every build, which
|
|
||||||
# reruns every RUN step after it. Docker keeps the layers above it
|
|
||||||
# until the base image, their own lines or go.mod and go.sum change.
|
|
||||||
# The steps that compile mount Go's build cache, and golangci-lint run
|
|
||||||
# also mounts golangci-lint's; both survive from build to build.
|
|
||||||
|
|
||||||
# Lint phase
|
# Lint phase
|
||||||
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
||||||
FROM golangci/golangci-lint@sha256:... AS lint
|
FROM golangci/golangci-lint@sha256:... AS lint
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
ARG CACHEBUST
|
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN golangci-lint run --config .golangci.yml ./...
|
||||||
--mount=type=cache,target=/root/.cache/golangci-lint \
|
|
||||||
golangci-lint run --config .golangci.yml ./...
|
|
||||||
|
|
||||||
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
||||||
# image ships and the alpine one does not. -count=1 because the mounted
|
# image ships and the alpine one does not.
|
||||||
# cache also holds test results, which go test would otherwise replay.
|
|
||||||
# golang:1.x, YYYY-MM-DD
|
# golang:1.x, YYYY-MM-DD
|
||||||
FROM golang@sha256:... AS test
|
FROM golang@sha256:... AS test
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
ARG CACHEBUST
|
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN go test -timeout 90s -race -cover ./... || \
|
||||||
go test -count=1 -timeout 90s -race -cover ./... || \
|
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
|
go test -timeout 90s -race -v ./...; exit 1; }
|
||||||
|
|
||||||
# Build stage. Nothing is wanted from either phase above; the copies
|
# Build stage. Nothing is wanted from either phase above; the copies
|
||||||
# are what make BuildKit build them first, so nothing after them can
|
# are what make BuildKit build them first, so this stage cannot run
|
||||||
# run unless lint and test passed.
|
# unless lint and test passed.
|
||||||
# golang:1.x-alpine, YYYY-MM-DD
|
# golang:1.x-alpine, YYYY-MM-DD
|
||||||
FROM golang@sha256:... AS builder
|
FROM golang@sha256:... AS builder
|
||||||
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
|
COPY --from=test /src/go.sum /dev/null
|
||||||
RUN apk add --no-cache git
|
RUN apk add --no-cache git
|
||||||
|
# A tar-stream context keeps the sender's file owners, which git refuses.
|
||||||
|
RUN git config --system --add safe.directory /src
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
ARG CACHEBUST
|
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
|
||||||
COPY --from=test /src/go.sum /dev/null
|
|
||||||
# A tar-stream context keeps the sender's file owners, which git refuses.
|
|
||||||
RUN git config --system --add safe.directory /src
|
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# The VERSION build arg when one is given, otherwise
|
# The VERSION build arg when one is given, otherwise
|
||||||
@@ -260,8 +212,7 @@ style conventions are in separate documents:
|
|||||||
# .git present, a version that is still empty, dev or unknown fails the
|
# .git present, a version that is still empty, dev or unknown fails the
|
||||||
# build: git is missing or could not read the checkout.
|
# build: git is missing or could not read the checkout.
|
||||||
ARG VERSION
|
ARG VERSION
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
|
||||||
VERSION="${VERSION:-$(git describe --tags --always)}"; \
|
|
||||||
if [ -e .git ]; then \
|
if [ -e .git ]; then \
|
||||||
case "$VERSION" in ""|dev|unknown) \
|
case "$VERSION" in ""|dev|unknown) \
|
||||||
echo "version is '$VERSION' although .git is present" >&2; \
|
echo "version is '$VERSION' although .git is present" >&2; \
|
||||||
@@ -274,7 +225,6 @@ style conventions are in separate documents:
|
|||||||
|
|
||||||
# Runtime stage, and the last one
|
# Runtime stage, and the last one
|
||||||
FROM alpine@sha256:...
|
FROM alpine@sha256:...
|
||||||
ARG CACHEBUST
|
|
||||||
COPY --from=builder /app /usr/local/bin/app
|
COPY --from=builder /app /usr/local/bin/app
|
||||||
ENTRYPOINT ["app"]
|
ENTRYPOINT ["app"]
|
||||||
```
|
```
|
||||||
@@ -289,26 +239,6 @@ style conventions are in separate documents:
|
|||||||
- Keep the runtime stage last, and if you add a stage after it, give it the
|
- Keep the runtime stage last, and if you add a stage after it, give it the
|
||||||
same two copies. A plain `docker build .` builds the last stage's chain
|
same two copies. A plain `docker build .` builds the last stage's chain
|
||||||
and nothing else.
|
and nothing else.
|
||||||
- Check builds keep these, and none of them can hand back a stale result:
|
|
||||||
- The layers above `ARG CACHEBUST`: the system packages and Go's
|
|
||||||
modules. Docker installs the packages again when the base image or the
|
|
||||||
install line changes, and downloads the modules again when `go.mod` or
|
|
||||||
`go.sum` changes. Go checks every module against `go.sum` as it
|
|
||||||
downloads it, and no later build can change a kept layer. The modules
|
|
||||||
are kept in a layer rather than a cache mount because a cache mount is
|
|
||||||
shared by every build on the host, and Go builds from whatever files
|
|
||||||
it holds without checking them again (only `go mod verify` does), so a
|
|
||||||
file changed there would go unnoticed.
|
|
||||||
- Go's build cache (`/root/.cache/go-build`, where Go keeps it for
|
|
||||||
root), mounted on the steps that compile: `golangci-lint run`, both
|
|
||||||
`go test` runs and `go build`. Go reuses a compiled package only when
|
|
||||||
all its inputs are unchanged. The cache also holds test results, which
|
|
||||||
is why both `go test` runs carry `-count=1`.
|
|
||||||
- golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned
|
|
||||||
image keeps it), mounted on `golangci-lint run`. golangci-lint reuses
|
|
||||||
a package's findings only when the package's files and those of every
|
|
||||||
package it imports, `go.mod`, the linters and their settings in
|
|
||||||
`.golangci.yml`, and the golangci-lint version are all unchanged.
|
|
||||||
- If the project uses `//go:embed` directives that reference build artifacts
|
- If the project uses `//go:embed` directives that reference build artifacts
|
||||||
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
||||||
create placeholder files so the embed directives resolve. Example:
|
create placeholder files so the embed directives resolve. Example:
|
||||||
@@ -317,20 +247,12 @@ style conventions are in separate documents:
|
|||||||
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
|
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
|
||||||
has no `apk`, so install with `apt-get` under the Debian package name
|
has no `apk`, so install with `apt-get` under the Debian package name
|
||||||
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
|
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
|
||||||
lists in the same `RUN`, so the layer does not keep them. The test phase,
|
lists in the same `RUN`, so the layer does not keep them:
|
||||||
on the Debian Go image, installs the libraries its tests need the same
|
|
||||||
way. Either install goes right after `FROM`, above `ARG CACHEBUST`, so
|
|
||||||
Docker keeps it until the base image or the install line changes:
|
|
||||||
|
|
||||||
```dockerfile
|
```dockerfile
|
||||||
FROM golangci/golangci-lint@sha256:... AS lint
|
|
||||||
RUN apt-get update \
|
RUN apt-get update \
|
||||||
&& apt-get install -y --no-install-recommends libvips-dev \
|
&& apt-get install -y --no-install-recommends libvips-dev \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
WORKDIR /src
|
|
||||||
COPY go.mod go.sum ./
|
|
||||||
RUN go mod download
|
|
||||||
ARG CACHEBUST
|
|
||||||
```
|
```
|
||||||
|
|
||||||
- `.dockerignore` lets `.git` into the build context. It keeps out every git
|
- `.dockerignore` lets `.git` into the build context. It keeps out every git
|
||||||
@@ -356,39 +278,20 @@ style conventions are in separate documents:
|
|||||||
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
|
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
|
||||||
`docker build .` with no build arguments must succeed; a Dockerfile that
|
`docker build .` with no build arguments must succeed; a Dockerfile that
|
||||||
refuses an empty build argument drops that refusal and keeps the argument.
|
refuses an empty build argument drops that refusal and keeps the argument.
|
||||||
A checkout whose `.git` is a file (a linked worktree, or a repository
|
|
||||||
checked out as a submodule) is the exception: that file points to a git
|
|
||||||
directory outside the build context, so the build cannot read the version
|
|
||||||
and a plain `docker build .` fails; pass the version with
|
|
||||||
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
|
|
||||||
do.
|
|
||||||
|
|
||||||
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
||||||
runs `script/cibuild` on push, and checks out the repo as its only other step,
|
runs `script/cibuild` on push, and checks out the repo as its only other step.
|
||||||
with `persist-credentials: false`: `script/cibuild` needs no token, and
|
That script bootstraps, runs the gate phases, and then builds the image, so a
|
||||||
without it the checkout leaves the job's token in `.git/config` for every
|
successful run means every check passed; a bare `docker build .` does not
|
||||||
later step. The checkout step also sets `fetch-depth: 0`, which fetches the
|
|
||||||
tags `git describe` needs: by default it clones shallow with no tags, and a
|
|
||||||
tagged repository's CI build would stamp a bare short commit id. The
|
|
||||||
workflow's `concurrency` block groups runs by workflow and branch
|
|
||||||
(`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`,
|
|
||||||
so a new push cancels the older run on the same branch, queued or running, and
|
|
||||||
no other: runs for replaced commits do not hold up the shared runner.
|
|
||||||
`script/cibuild` bootstraps, runs the gate phases, and then builds the image,
|
|
||||||
so a successful run means every check passed; a bare `docker build .` does not
|
|
||||||
carry the same guarantee, because its gate phases may come from the cache. The
|
carry the same guarantee, because its gate phases may come from the cache. The
|
||||||
image build gets a `CACHEBUST` value of its own and so runs the gate phases a
|
image build is uncached and so runs the gate phases a second time. That is the
|
||||||
second time. That is the price of the rule above, and it is worth paying: the
|
price of the rule above, and it is worth paying: the image that ships is built
|
||||||
image that ships is built from a run of its own gates rather than from a cache
|
from a run of its own gates rather than from a cache entry. A separate
|
||||||
entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the
|
workflow limited to `main` by a `branches` list under `on: push` cannot be
|
||||||
shared runner after 20 minutes. That allows for the three Docker builds
|
checked by review: to try a change to it, add the feature branch to that list
|
||||||
described above (the test phase, the lint phase, then the image), each held to
|
and push, then remove the branch from the list again before merging. Keep any
|
||||||
the 5-minute Docker build limit below, plus the bootstrap. A separate workflow
|
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
|
||||||
limited to `main` by a `branches` list under `on: push` cannot be checked by
|
from the feature branch publishes nothing.
|
||||||
review: to try a change to it, add the feature branch to that list and push,
|
|
||||||
then remove the branch from the list again before merging. Keep any job in it
|
|
||||||
that publishes behind `if: github.ref_name == 'main'`, so the run from the
|
|
||||||
feature branch publishes nothing.
|
|
||||||
|
|
||||||
- Use platform-standard formatters: `black` for Python, `prettier` for
|
- Use platform-standard formatters: `black` for Python, `prettier` for
|
||||||
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
||||||
@@ -448,11 +351,12 @@ style conventions are in separate documents:
|
|||||||
That cache is Go's own, separate from Docker's layer cache. Go stores a
|
That cache is Go's own, separate from Docker's layer cache. Go stores a
|
||||||
passing result in its cache directory (`GOCACHE`), and when the same tests
|
passing result in its cache directory (`GOCACHE`), and when the same tests
|
||||||
run again on unchanged code it prints that result, marked `(cached)`,
|
run again on unchanged code it prints that result, marked `(cached)`,
|
||||||
without running them. The directory lasts from one run to the next on a
|
without running them. That matters on a developer's machine, where this
|
||||||
developer's machine, where this target runs, and in the `test` phase of the
|
target runs and the directory lasts from one run to the next. The `test`
|
||||||
`Dockerfile`, which mounts it to keep compiled packages between check
|
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
|
||||||
builds, so the phase's `go test` runs carry `-count=1` too. A new
|
result for this repo's tests and nothing before its `go test` step runs a
|
||||||
`CACHEBUST` value (above) is what makes that step run on an unchanged tree.
|
test, so there is nothing to replay. `--no-cache` (above) is what makes that
|
||||||
|
step run on an unchanged tree.
|
||||||
|
|
||||||
Python example:
|
Python example:
|
||||||
|
|
||||||
@@ -480,12 +384,9 @@ style conventions are in separate documents:
|
|||||||
|
|
||||||
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
||||||
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
|
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
|
||||||
`node_modules/`, and the repo's own build outputs. Fetch the standard
|
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
|
||||||
`.gitignore` from
|
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
|
setting up a new repo. These patterns are written to `.gitignore`'s own
|
||||||
a new repo. A repo's `.gitignore` is the standard file followed by the repo's
|
|
||||||
own entries, such as its binaries; a re-vendor replaces the standard part and
|
|
||||||
keeps those entries. These patterns are written to `.gitignore`'s own
|
|
||||||
semantics, in which an unanchored pattern already matches at every depth; they
|
semantics, in which an unanchored pattern already matches at every depth; they
|
||||||
are not a `.dockerignore` and must not be transplanted into one unmodified.
|
are not a `.dockerignore` and must not be transplanted into one unmodified.
|
||||||
|
|
||||||
@@ -533,15 +434,13 @@ style conventions are in separate documents:
|
|||||||
byte-identically across repos:
|
byte-identically across repos:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# The version and the tag each get their own line: a failing command
|
# Own line: a failing command substitution inside an argument does not
|
||||||
# substitution inside an argument does not trip `set -e`, so the inline
|
# trip `set -e`, so the inline form degrades to an empty constant.
|
||||||
# form degrades to an empty constant.
|
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
tag="$(script/projectname)"
|
docker build --no-cache \
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$tag" .
|
-t "$(script/projectname)" .
|
||||||
```
|
```
|
||||||
|
|
||||||
`--always` makes an untagged repo yield an abbreviated commit hash rather
|
`--always` makes an untagged repo yield an abbreviated commit hash rather
|
||||||
@@ -553,8 +452,8 @@ style conventions are in separate documents:
|
|||||||
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
||||||
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
||||||
the scripts stay byte-identical. One consequence for CI: the standard
|
the scripts stay byte-identical. One consequence for CI: the standard
|
||||||
checkout action clones shallow and fetches no tags, so the canonical
|
checkout action clones shallow and fetches no tags, so a repo that embeds a
|
||||||
`.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step.
|
tag-derived version must set `fetch-depth: 0` on its checkout step.
|
||||||
|
|
||||||
- **Verify `.dockerignore` by enumerating the image, not by reading the
|
- **Verify `.dockerignore` by enumerating the image, not by reading the
|
||||||
patterns.** Plant files at the root _and_ at least two directories deep, build
|
patterns.** Plant files at the root _and_ at least two directories deep, build
|
||||||
@@ -621,11 +520,6 @@ style conventions are in separate documents:
|
|||||||
function defined and never invoked has the same exit status and the same
|
function defined and never invoked has the same exit status and the same
|
||||||
empty output as one that worked.
|
empty output as one that worked.
|
||||||
|
|
||||||
Node is the exception to the whole-token comparison: the canonical
|
|
||||||
`script/bootstrap` compares only its major version, because the node 22
|
|
||||||
alpine image the `Dockerfile` stages start from is not the exact pin, and
|
|
||||||
nvm cannot install a prebuilt node on alpine.
|
|
||||||
|
|
||||||
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
||||||
|
|
||||||
A Go tool a repo needs on the host is installed with `go install` pinned to
|
A Go tool a repo needs on the host is installed with `go install` pinned to
|
||||||
@@ -742,11 +636,7 @@ style conventions are in separate documents:
|
|||||||
Never edit existing migrations after release.
|
Never edit existing migrations after release.
|
||||||
|
|
||||||
- All repos should have an `.editorconfig` enforcing the project's indentation
|
- All repos should have an `.editorconfig` enforcing the project's indentation
|
||||||
settings: the standard file from
|
settings.
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, which sets
|
|
||||||
tabs for `Makefile` and Go files, followed by the repo's own sections, such as
|
|
||||||
one for another language it uses. A re-vendor replaces the standard part and
|
|
||||||
keeps those sections.
|
|
||||||
|
|
||||||
- Avoid putting files in the repo root unless necessary. Root should contain
|
- Avoid putting files in the repo root unless necessary. Root should contain
|
||||||
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
|
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
|
||||||
|
|||||||
@@ -1,121 +1,127 @@
|
|||||||
# Workflow
|
# Workflow
|
||||||
|
|
||||||
- branch (from `main`)
|
* branch (from `main`)
|
||||||
- do the work in Next Step
|
* do the work in Next Step
|
||||||
- move Next Step to the top of Completed Steps
|
* move Next Step to the top of Completed Steps
|
||||||
- move the top item of Future Steps into Next Step
|
* move the top item of Future Steps into Next Step
|
||||||
- commit (`TODO.md` changes in the same commit as the work)
|
* commit (`TODO.md` changes in the same commit as the work)
|
||||||
- merge to `main` if the branch is not protected, otherwise open a PR
|
* merge to `main` if the branch is not protected, otherwise open a PR
|
||||||
- push
|
* push
|
||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
pre-1.0. No git tags. The linter and the tests are phases of the Docker build,
|
pre-1.0. No git tags. The linter and the tests are phases of the Docker
|
||||||
and the Gitea workflow runs `script/cibuild` (the format check of the Go code
|
build, and the Gitea workflow runs `script/cibuild` (the format check, both
|
||||||
and the markdown, both phases, then the image) on every push. The image sets
|
phases, then the image) on every push. The
|
||||||
memory ceilings for a 5 GiB container (README "Memory") and the README says how
|
image sets memory ceilings for a 5 GiB container (README "Memory") and the
|
||||||
to run it under upaas (README "Running under upaas"). A 35-hour run of `3898daa`
|
README says how to run it under upaas (README "Running under upaas"). A
|
||||||
on the live feed peaked at about 1 GiB, without a container memory limit.
|
35-hour run of `3898daa` on the live feed peaked at about 1 GiB, without a
|
||||||
|
container memory limit.
|
||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
`next` waits for sneak to merge it to `main` through
|
`next` waits for sneak to merge it to `main` through
|
||||||
https://git.eeqj.de/sneak/routewatch/pulls/6. After that, setting routewatch up
|
https://git.eeqj.de/sneak/routewatch/pulls/6. After that, setting
|
||||||
under upaas on fsn1app1 and deploying it are his
|
routewatch up under upaas on fsn1app1 and deploying it are his
|
||||||
(https://git.eeqj.de/sneak/routewatch/issues/31), and so is the run under a real
|
(https://git.eeqj.de/sneak/routewatch/issues/31), and so is the run under a
|
||||||
5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3).
|
real 5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3).
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-10-19: the shared files are re-vendored from `sneak/prompts` at
|
- 2026-10-14: the shared files are re-vendored from `sneak/prompts` at
|
||||||
`0653579`: golangci-lint v2.14.0 and its configuration, lint and test as
|
`dd4027b`: golangci-lint v2.14.0 and its configuration, lint and test as
|
||||||
`Dockerfile` phases that keep Go's and golangci-lint's caches, markdown
|
separate uncached Dockerfile phases, agent guidance in `AGENTS.md`; the code
|
||||||
formatted with prettier, agent guidance in `AGENTS.md`; the code is fixed for
|
is fixed for what the new linter configuration finds (closes #52)
|
||||||
what the new linter configuration finds (closes #52)
|
- 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory,
|
||||||
- 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory, seeded
|
seeded at startup and adjusted on every live-route write, so a request no
|
||||||
at startup and adjusted on every live-route write, so a request no longer
|
longer reads every live route (closes #30)
|
||||||
reads every live route (closes #30)
|
- 2026-10-03: a new database is created with `auto_vacuum` set to
|
||||||
- 2026-10-03: a new database is created with `auto_vacuum` set to incremental,
|
incremental, through the connection string so it is set before the file
|
||||||
through the connection string so it is set before the file is first written,
|
is first written, and each periodic incremental vacuum now returns up to
|
||||||
and each periodic incremental vacuum now returns up to 1000 free pages; the
|
1000 free pages; the late `PRAGMA auto_vacuum` in `Initialize`, which
|
||||||
late `PRAGMA auto_vacuum` in `Initialize`, which SQLite ignored, is gone
|
SQLite ignored, is gone (closes #43)
|
||||||
(closes #43)
|
- 2026-10-03: looking up an IP address no longer reads every IPv6 route:
|
||||||
- 2026-10-03: looking up an IP address no longer reads every IPv6 route: both
|
both families find the most specific live route with at most 33 or 129
|
||||||
families find the most specific live route with at most 33 or 129 lookups on
|
lookups on the prefix index, and the IPv4 range columns are gone. Prefixes
|
||||||
the prefix index, and the IPv4 range columns are gone. Prefixes from the feed
|
from the feed are stored in one text form, so an IPv6 withdrawal, which the
|
||||||
are stored in one text form, so an IPv6 withdrawal, which the feed sends
|
feed sends uncompressed, now removes its route (closes #48)
|
||||||
uncompressed, now removes its route (closes #48)
|
- 2026-10-02: a plain `docker build .` stamps the commit's tag or short
|
||||||
- 2026-10-02: a plain `docker build .` stamps the commit's tag or short commit
|
commit (`git describe --tags --always`) into the page footer instead of
|
||||||
(`git describe --tags --always`) into the page footer instead of `unknown`:
|
`unknown`: `.dockerignore` sends `.git` without `.git/config`, a `VERSION`
|
||||||
`.dockerignore` sends `.git` without `.git/config`, a `VERSION` build argument
|
build argument takes precedence, and `make build` stamps the same value
|
||||||
takes precedence, and `make build` stamps the same value (closes #46)
|
(closes #46)
|
||||||
- 2026-09-29: the entrypoint creates the data directory if it is missing and
|
- 2026-09-29: the entrypoint creates the data directory if it is missing and
|
||||||
stops the start if a step fails; README "Running under upaas" no longer asks
|
stops the start if a step fails; README "Running under upaas" no longer
|
||||||
for the host directory to be created first (closes #42)
|
asks for the host directory to be created first (closes #42)
|
||||||
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local databases
|
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local
|
||||||
and `.env` out of the Docker build context, and so out of the source archive
|
databases and `.env` out of the Docker build context, and so out of the
|
||||||
in the image (closes #39)
|
source archive in the image (closes #39)
|
||||||
- 2026-09-29: README first line names the MIT license and the author; the
|
- 2026-09-29: README first line names the MIT license and the author;
|
||||||
License section now says MIT and links `LICENSE`, and an Author section was
|
the License section now says MIT and links `LICENSE`, and an Author
|
||||||
added; this file brought up to date (closes #38)
|
section was added; this file brought up to date (closes #38)
|
||||||
- 2026-09-29: MIT `LICENSE` (closes #1)
|
- 2026-09-29: MIT `LICENSE` (closes #1)
|
||||||
- 2026-09-28: stopping the daemon while the feed is flowing no longer panics
|
- 2026-09-28: stopping the daemon while the feed is flowing no longer
|
||||||
with "send on closed channel": the read loop checks for a stop just before
|
panics with "send on closed channel": the read loop checks for a stop
|
||||||
handing a message to the handler queues, and a second `Stop` no longer closes
|
just before handing a message to the handler queues, and a second
|
||||||
the queues again (closes #34)
|
`Stop` no longer closes the queues again (closes #34)
|
||||||
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the stop
|
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the
|
||||||
signal: the entrypoint switches to the `routewatch` user with `setpriv`
|
stop signal: the entrypoint switches to the `routewatch` user with
|
||||||
instead of `runuser`, so the daemon receives the signal itself and gets the
|
`setpriv` instead of `runuser`, so the daemon receives the signal itself
|
||||||
whole wait `docker stop` allows, up to its own 60-second limit (closes #33)
|
and gets the whole wait `docker stop` allows, up to its own 60-second
|
||||||
|
limit (closes #33)
|
||||||
- 2026-09-28: ready to run under upaas: a set but invalid `PORT`,
|
- 2026-09-28: ready to run under upaas: a set but invalid `PORT`,
|
||||||
`XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start, the health check
|
`XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start, the health
|
||||||
follows `PORT`, README "Running under upaas" section (closes #31)
|
check follows `PORT`, README "Running under upaas" section (closes
|
||||||
- 2026-09-22: realtime in-memory database statistics: counts seeded at startup
|
#31)
|
||||||
and adjusted on every write, oldest/newest route timestamps via index-end
|
- 2026-09-22: realtime in-memory database statistics: counts seeded at
|
||||||
lookups; `/api/v1/stats` no longer scans the tables (closes #27)
|
startup and adjusted on every write, oldest/newest route timestamps via
|
||||||
- 2026-09-21: batch writes take the write lock when their transaction begins
|
index-end lookups; `/api/v1/stats` no longer scans the tables (closes
|
||||||
(`_txlock=immediate`), so they wait out a WAL checkpoint instead of failing
|
#27)
|
||||||
with "database is locked" (closes #25)
|
- 2026-09-21: batch writes take the write lock when their transaction
|
||||||
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps glibc malloc arenas, so
|
begins (`_txlock=immediate`), so they wait out a WAL checkpoint instead
|
||||||
memory outside the Go runtime no longer grows with the core count (closes #23)
|
of failing with "database is locked" (closes #25)
|
||||||
- 2026-09-21: `GOMEMLIMIT=1536MiB` in the image; README Memory section with the
|
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps glibc malloc arenas,
|
||||||
memory budget and the 5 GiB container limit (closes #13)
|
so memory outside the Go runtime no longer grows with the core count
|
||||||
- 2026-09-21: the four handler queues hold at most 20,000 messages each, down
|
(closes #23)
|
||||||
from 100,000 (closes #11)
|
- 2026-09-21: `GOMEMLIMIT=1536MiB` in the image; README Memory section
|
||||||
- 2026-09-21: two goroutine leaks fixed: the stats handlers after a timeout and
|
with the memory budget and the 5 GiB container limit (closes #13)
|
||||||
the streamer's tickers on every reconnect (closes #12)
|
- 2026-09-21: the four handler queues hold at most 20,000 messages each,
|
||||||
- 2026-09-21: parsed RIS messages no longer keep the unused `Community` and
|
down from 100,000 (closes #11)
|
||||||
`Raw` fields (closes #9)
|
- 2026-09-21: two goroutine leaks fixed: the stats handlers after a
|
||||||
- 2026-09-21: `.editorconfig`, and a Gitea workflow that runs `script/cibuild`
|
timeout and the streamer's tickers on every reconnect (closes #12)
|
||||||
on every push (closes #14)
|
- 2026-09-21: parsed RIS messages no longer keep the unused `Community`
|
||||||
- 2026-09-21: the peering handler's AS-path map holds at most 500,000 paths and
|
and `Raw` fields (closes #9)
|
||||||
is swapped for an empty one every 30 seconds instead of copied (closes #10)
|
- 2026-09-21: `.editorconfig`, and a Gitea workflow that runs
|
||||||
- 2026-09-21: SQLite memory bounded across the whole connection pool: a 64 MiB
|
`script/cibuild` on every push (closes #14)
|
||||||
page cache on each connection, 1 GiB soft and 1.5 GiB hard heap limits (closes
|
- 2026-09-21: the peering handler's AS-path map holds at most 500,000
|
||||||
#8)
|
paths and is swapped for an empty one every 30 seconds instead of copied
|
||||||
- 2026-09-21: the Docker build runs the format check, the linter and the tests,
|
(closes #10)
|
||||||
linting in a separate stage on a golangci-lint image pinned by digest (closes
|
- 2026-09-21: SQLite memory bounded across the whole connection pool: a
|
||||||
#5)
|
64 MiB page cache on each connection, 1 GiB soft and 1.5 GiB hard heap
|
||||||
|
limits (closes #8)
|
||||||
|
- 2026-09-21: the Docker build runs the format check, the linter and the
|
||||||
|
tests, linting in a separate stage on a golangci-lint image pinned by
|
||||||
|
digest (closes #5)
|
||||||
- 2026-09-21: `make test` skips the live-network feed test (`-short`), so
|
- 2026-09-21: `make test` skips the live-network feed test (`-short`), so
|
||||||
`make check` no longer depends on the network (closes #2)
|
`make check` no longer depends on the network (closes #2)
|
||||||
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
|
||||||
shims, README Entrypoints section
|
Makefile shims, README Entrypoints section
|
||||||
- 2026-02-22: repo policy compliance: required policy files, .gitignore update,
|
- 2026-02-22: repo policy compliance: required policy files, .gitignore
|
||||||
Makefile fmt-check/check/docker/hooks targets, gofmt pass
|
update, Makefile fmt-check/check/docker/hooks targets, gofmt pass
|
||||||
(repo-policies-compliance, unmerged)
|
(repo-policies-compliance, unmerged)
|
||||||
- 2026-01-01: WAL checkpointing: periodic (5s), on startup with logging,
|
- 2026-01-01: WAL checkpointing: periodic (5s), on startup with logging,
|
||||||
TRUNCATE mode; fixed slow queries
|
TRUNCATE mode; fixed slow queries
|
||||||
- 2025-12-31: status page: navbar, home page with search, oldest/newest route
|
- 2025-12-31: status page: navbar, home page with search, oldest/newest
|
||||||
timestamps, NULL handling in WHOIS stats; container runs as routewatch user
|
route timestamps, NULL handling in WHOIS stats; container runs as
|
||||||
with proper state dir
|
routewatch user with proper state dir
|
||||||
- 2025-12-30: structured HTTP request logging, increased timeouts, CIDR prefix
|
- 2025-12-30: structured HTTP request logging, increased timeouts,
|
||||||
URL routing fixes, status page metrics and footer
|
CIDR prefix URL routing fixes, status page metrics and footer
|
||||||
- 2025-12-29: Dockerfile multi-stage build with source archive; SQLite
|
- 2025-12-29: Dockerfile multi-stage build with source archive; SQLite
|
||||||
incremental vacuum for non-blocking space reclamation
|
incremental vacuum for non-blocking space reclamation
|
||||||
- 2025-07-27: initial RouteWatch BGP stream monitor
|
- 2025-07-27: initial RouteWatch BGP stream monitor
|
||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
- Production memory under 5 GiB: whether to test under a real 5 GiB container
|
- Production memory under 5 GiB: whether to test under a real 5 GiB
|
||||||
limit on fsn1app1 is open for sneak
|
container limit on fsn1app1 is open for sneak
|
||||||
(https://git.eeqj.de/sneak/routewatch/issues/3)
|
(https://git.eeqj.de/sneak/routewatch/issues/3)
|
||||||
@@ -15,7 +15,7 @@ import (
|
|||||||
// testV4PeerIP is the peer the IPv4 routes in these tests are learned from.
|
// testV4PeerIP is the peer the IPv4 routes in these tests are learned from.
|
||||||
const testV4PeerIP = "192.0.2.1"
|
const testV4PeerIP = "192.0.2.1"
|
||||||
|
|
||||||
// testV6Prefix is the prefix of the IPv6 routes in these tests.
|
// testV6Prefix is the prefix of every IPv6 route in these tests.
|
||||||
const testV6Prefix = "2001:db8::/32"
|
const testV6Prefix = "2001:db8::/32"
|
||||||
|
|
||||||
// mkV4Route builds an IPv4 live route with its mask length taken from the
|
// mkV4Route builds an IPv4 live route with its mask length taken from the
|
||||||
@@ -41,11 +41,11 @@ func mkV4Route(t *testing.T, prefix string, asn int, ts time.Time) *LiveRoute {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// mkV6Route builds an IPv6 live route.
|
// mkV6Route builds an IPv6 live route for testV6Prefix.
|
||||||
func mkV6Route(prefix string, asn int, ts time.Time) *LiveRoute {
|
func mkV6Route(asn int, ts time.Time) *LiveRoute {
|
||||||
return &LiveRoute{
|
return &LiveRoute{
|
||||||
ID: uuid.New(),
|
ID: uuid.New(),
|
||||||
Prefix: prefix,
|
Prefix: testV6Prefix,
|
||||||
MaskLength: 32,
|
MaskLength: 32,
|
||||||
IPVersion: ipVersionV6,
|
IPVersion: ipVersionV6,
|
||||||
OriginASN: asn,
|
OriginASN: asn,
|
||||||
@@ -91,7 +91,7 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
|
|||||||
routes := []*LiveRoute{
|
routes := []*LiveRoute{
|
||||||
mkV4Route(t, "198.51.100.0/24", 64500, ts),
|
mkV4Route(t, "198.51.100.0/24", 64500, ts),
|
||||||
mkV4Route(t, "203.0.113.0/24", 64501, ts.Add(time.Minute)),
|
mkV4Route(t, "203.0.113.0/24", 64501, ts.Add(time.Minute)),
|
||||||
mkV6Route(testV6Prefix, 64502, ts.Add(2*time.Minute)),
|
mkV6Route(64502, ts.Add(2*time.Minute)),
|
||||||
}
|
}
|
||||||
|
|
||||||
err = db.UpsertLiveRouteBatch(routes)
|
err = db.UpsertLiveRouteBatch(routes)
|
||||||
@@ -126,9 +126,10 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
|
|||||||
})
|
})
|
||||||
|
|
||||||
// A withdrawal removes one route.
|
// A withdrawal removes one route.
|
||||||
err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{
|
err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{{
|
||||||
{Prefix: "203.0.113.0/24", PeerIP: testV4PeerIP, IPVersion: ipVersionV4},
|
Prefix: "203.0.113.0/24", OriginASN: 64501,
|
||||||
})
|
PeerIP: testV4PeerIP, IPVersion: ipVersionV4,
|
||||||
|
}})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("DeleteLiveRouteBatch: %v", err)
|
t.Fatalf("DeleteLiveRouteBatch: %v", err)
|
||||||
}
|
}
|
||||||
@@ -200,7 +201,7 @@ func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
|
|||||||
|
|
||||||
err = db.UpsertLiveRouteBatch([]*LiveRoute{
|
err = db.UpsertLiveRouteBatch([]*LiveRoute{
|
||||||
mkV4Route(t, "198.51.100.0/24", 64500, ts),
|
mkV4Route(t, "198.51.100.0/24", 64500, ts),
|
||||||
mkV6Route(testV6Prefix, 64502, ts),
|
mkV6Route(64502, ts),
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
||||||
@@ -320,8 +321,8 @@ func prefixDistributionSteps(t *testing.T) []prefixDistributionStep {
|
|||||||
sharedSecondPeer.PeerIP = "192.0.2.2"
|
sharedSecondPeer.PeerIP = "192.0.2.2"
|
||||||
other := mkV4Route(t, "203.0.113.0/24", 64501, ts)
|
other := mkV4Route(t, "203.0.113.0/24", 64501, ts)
|
||||||
wide := mkV4Route(t, "172.16.0.0/16", 64502, ts)
|
wide := mkV4Route(t, "172.16.0.0/16", 64502, ts)
|
||||||
v6 := mkV6Route(testV6Prefix, 64503, ts)
|
v6 := mkV6Route(64503, ts)
|
||||||
v6SecondPeer := mkV6Route(testV6Prefix, 64503, ts)
|
v6SecondPeer := mkV6Route(64503, ts)
|
||||||
v6SecondPeer.PeerIP = "2001:db8::2"
|
v6SecondPeer.PeerIP = "2001:db8::2"
|
||||||
|
|
||||||
all := []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 2}}
|
all := []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 2}}
|
||||||
@@ -404,11 +405,13 @@ func announceRoutes(db *Database, batch bool, routes []*LiveRoute) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// withdrawRoutes removes routes in one DeleteLiveRouteBatch, or with one
|
// withdrawRoutes removes routes in one DeleteLiveRouteBatch, or with one
|
||||||
// DeleteLiveRoute each.
|
// DeleteLiveRoute each. It names each route by prefix and peer only, with no
|
||||||
|
// origin ASN, as a withdrawal from the feed does when its message carries no AS
|
||||||
|
// path.
|
||||||
func withdrawRoutes(db *Database, batch bool, routes []*LiveRoute) error {
|
func withdrawRoutes(db *Database, batch bool, routes []*LiveRoute) error {
|
||||||
if !batch {
|
if !batch {
|
||||||
for _, route := range routes {
|
for _, route := range routes {
|
||||||
err := db.DeleteLiveRoute(route.Prefix, route.PeerIP)
|
err := db.DeleteLiveRoute(route.Prefix, 0, route.PeerIP)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -461,7 +464,7 @@ func TestStatsRouteTimestamps(t *testing.T) {
|
|||||||
err = db.UpsertLiveRouteBatch([]*LiveRoute{
|
err = db.UpsertLiveRouteBatch([]*LiveRoute{
|
||||||
mkV4Route(t, "198.51.100.0/24", 64500, base.Add(time.Minute)),
|
mkV4Route(t, "198.51.100.0/24", 64500, base.Add(time.Minute)),
|
||||||
mkV4Route(t, "203.0.113.0/24", 64501, oldest),
|
mkV4Route(t, "203.0.113.0/24", 64501, oldest),
|
||||||
mkV6Route(testV6Prefix, 64502, newest),
|
mkV6Route(64502, newest),
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
||||||
@@ -507,7 +510,7 @@ func TestLiveCountsConcurrentReadWrite(t *testing.T) {
|
|||||||
|
|
||||||
for i := range 25 {
|
for i := range 25 {
|
||||||
asn := 65000 + base*100 + i
|
asn := 65000 + base*100 + i
|
||||||
route := mkV6Route(testV6Prefix, asn, ts)
|
route := mkV6Route(asn, ts)
|
||||||
route.PeerIP = "2001:db8::" + uuid.NewString()
|
route.PeerIP = "2001:db8::" + uuid.NewString()
|
||||||
|
|
||||||
err := db.UpsertLiveRoute(route)
|
err := db.UpsertLiveRoute(route)
|
||||||
|
|||||||
+142
-268
@@ -7,7 +7,6 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"math/rand/v2"
|
|
||||||
"net"
|
"net"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"os"
|
"os"
|
||||||
@@ -68,17 +67,6 @@ const (
|
|||||||
sqliteHardHeapLimitBytes = 1610612736
|
sqliteHardHeapLimitBytes = 1610612736
|
||||||
)
|
)
|
||||||
|
|
||||||
// Bounds on the work GetRandomPrefixesByLength does to pick its sample.
|
|
||||||
const (
|
|
||||||
// maxPrefixesToShuffle is the most prefixes of one mask length that are
|
|
||||||
// read in full and shuffled to pick the sample. A mask length with more is
|
|
||||||
// sampled with random lookups.
|
|
||||||
maxPrefixesToShuffle = 5000
|
|
||||||
// sampleLookupsPerPrefix is the most random lookups made for each prefix
|
|
||||||
// asked for.
|
|
||||||
sampleLookupsPerPrefix = 4
|
|
||||||
)
|
|
||||||
|
|
||||||
// Common errors
|
// Common errors
|
||||||
var (
|
var (
|
||||||
// ErrInvalidIP is returned when an IP address is malformed
|
// ErrInvalidIP is returned when an IP address is malformed
|
||||||
@@ -224,25 +212,24 @@ func (d *Database) Close() error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// A live-route upsert is an UPDATE followed, only when no row matched, by an
|
// A live-route upsert is an UPDATE followed, only when no row matched, by an
|
||||||
// INSERT. A live route is keyed by prefix and peer, so an announcement replaces
|
// INSERT. The UPDATE's rows-affected count (1 for an existing key, 0 for a new
|
||||||
// the peer's route for the prefix whatever its origin AS. The UPDATE's
|
// one) is what lets the in-memory route counters stay exact without a COUNT(*).
|
||||||
// rows-affected count (1 for an existing key, 0 for a new one) is what lets the
|
// Callers hold the database write lock, so no other writer can insert the same
|
||||||
// in-memory route counters stay exact without a COUNT(*). Callers hold the
|
// key between the two statements. The id column is set only on INSERT, so an
|
||||||
// database write lock, so no other writer can insert the same key between the
|
// updated route keeps its original id, exactly as the previous ON CONFLICT
|
||||||
// two statements. The id column is set only on INSERT, so an updated route
|
// upsert did.
|
||||||
// keeps its original id, exactly as the previous ON CONFLICT upsert did.
|
|
||||||
const (
|
const (
|
||||||
updateLiveRouteV4SQL = `UPDATE live_routes_v4
|
updateLiveRouteV4SQL = `UPDATE live_routes_v4
|
||||||
SET mask_length = ?, origin_asn = ?, as_path = ?, next_hop = ?,
|
SET mask_length = ?, as_path = ?, next_hop = ?, last_updated = ?
|
||||||
last_updated = ? WHERE prefix = ? AND peer_ip = ?`
|
WHERE prefix = ? AND origin_asn = ? AND peer_ip = ?`
|
||||||
|
|
||||||
insertLiveRouteV4SQL = `INSERT INTO live_routes_v4
|
insertLiveRouteV4SQL = `INSERT INTO live_routes_v4
|
||||||
(id, prefix, mask_length, origin_asn, peer_ip, as_path, next_hop,
|
(id, prefix, mask_length, origin_asn, peer_ip, as_path, next_hop,
|
||||||
last_updated) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`
|
last_updated) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`
|
||||||
|
|
||||||
updateLiveRouteV6SQL = `UPDATE live_routes_v6
|
updateLiveRouteV6SQL = `UPDATE live_routes_v6
|
||||||
SET mask_length = ?, origin_asn = ?, as_path = ?, next_hop = ?,
|
SET mask_length = ?, as_path = ?, next_hop = ?, last_updated = ?
|
||||||
last_updated = ? WHERE prefix = ? AND peer_ip = ?`
|
WHERE prefix = ? AND origin_asn = ? AND peer_ip = ?`
|
||||||
|
|
||||||
insertLiveRouteV6SQL = `INSERT INTO live_routes_v6
|
insertLiveRouteV6SQL = `INSERT INTO live_routes_v6
|
||||||
(id, prefix, mask_length, origin_asn, peer_ip, as_path, next_hop,
|
(id, prefix, mask_length, origin_asn, peer_ip, as_path, next_hop,
|
||||||
@@ -259,11 +246,18 @@ const (
|
|||||||
(SELECT 1 FROM live_routes_v6 WHERE prefix = ?)`
|
(SELECT 1 FROM live_routes_v6 WHERE prefix = ?)`
|
||||||
)
|
)
|
||||||
|
|
||||||
// A deletion removes the live route the peer has for the prefix, if any.
|
// A deletion that names an origin ASN removes at most one row. One without an
|
||||||
|
// origin ASN removes the prefix's routes from that peer whatever their origin,
|
||||||
|
// which can be several rows.
|
||||||
const (
|
const (
|
||||||
deleteLiveRouteV4SQL = `DELETE FROM live_routes_v4
|
deleteLiveRouteV4SQL = `DELETE FROM live_routes_v4
|
||||||
|
WHERE prefix = ? AND origin_asn = ? AND peer_ip = ?`
|
||||||
|
deleteLiveRouteAnyOriginV4SQL = `DELETE FROM live_routes_v4
|
||||||
WHERE prefix = ? AND peer_ip = ?`
|
WHERE prefix = ? AND peer_ip = ?`
|
||||||
|
|
||||||
deleteLiveRouteV6SQL = `DELETE FROM live_routes_v6
|
deleteLiveRouteV6SQL = `DELETE FROM live_routes_v6
|
||||||
|
WHERE prefix = ? AND origin_asn = ? AND peer_ip = ?`
|
||||||
|
deleteLiveRouteAnyOriginV6SQL = `DELETE FROM live_routes_v6
|
||||||
WHERE prefix = ? AND peer_ip = ?`
|
WHERE prefix = ? AND peer_ip = ?`
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -274,8 +268,8 @@ func upsertRouteRow(
|
|||||||
ctx context.Context, upd, ins, has *sql.Stmt, route *LiveRoute,
|
ctx context.Context, upd, ins, has *sql.Stmt, route *LiveRoute,
|
||||||
pathJSON string,
|
pathJSON string,
|
||||||
) (bool, bool, error) {
|
) (bool, bool, error) {
|
||||||
res, err := upd.ExecContext(ctx, route.MaskLength, route.OriginASN, pathJSON,
|
res, err := upd.ExecContext(ctx, route.MaskLength, pathJSON, route.NextHop,
|
||||||
route.NextHop, route.LastUpdated, route.Prefix, route.PeerIP)
|
route.LastUpdated, route.Prefix, route.OriginASN, route.PeerIP)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, false, err
|
return false, false, err
|
||||||
}
|
}
|
||||||
@@ -424,20 +418,33 @@ func (d *Database) UpsertLiveRouteBatch(routes []*LiveRoute) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// deleteRouteRow runs one deletion with the delete statement of its IP
|
// deleteRouteRows runs one deletion with the delete statements of its IP
|
||||||
// version's table. It returns how many rows it removed and whether that left
|
// version's table: withOrigin when it names an origin ASN, otherwise
|
||||||
// the prefix with no live route.
|
// withoutOrigin. It returns how many rows it removed and whether that left the
|
||||||
func deleteRouteRow(
|
// prefix with no live route.
|
||||||
ctx context.Context, deleteStmt, has *sql.Stmt, del LiveRouteDeletion,
|
func deleteRouteRows(
|
||||||
|
ctx context.Context, withOrigin, withoutOrigin, has *sql.Stmt,
|
||||||
|
del LiveRouteDeletion,
|
||||||
) (int64, bool, error) {
|
) (int64, bool, error) {
|
||||||
res, err := deleteStmt.ExecContext(ctx, del.Prefix, del.PeerIP)
|
var (
|
||||||
|
res sql.Result
|
||||||
|
err error
|
||||||
|
)
|
||||||
|
|
||||||
|
if del.OriginASN == 0 {
|
||||||
|
res, err = withoutOrigin.ExecContext(ctx, del.Prefix, del.PeerIP)
|
||||||
|
} else {
|
||||||
|
res, err = withOrigin.ExecContext(ctx, del.Prefix, del.OriginASN,
|
||||||
|
del.PeerIP)
|
||||||
|
}
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, false, fmt.Errorf("failed to delete route %s: %w",
|
return 0, false, fmt.Errorf("failed to delete route %s: %w",
|
||||||
del.Prefix, err)
|
del.Prefix, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The feed often withdraws a route that is not live, which deletes
|
// A deletion with no origin ASN can remove several rows, so use the
|
||||||
// nothing, so the in-memory route counters use the rows-affected count.
|
// exact rows-affected count to keep the in-memory route counters right.
|
||||||
affected, err := res.RowsAffected()
|
affected, err := res.RowsAffected()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, false, fmt.Errorf("failed to count deleted route %s: %w",
|
return 0, false, fmt.Errorf("failed to count deleted route %s: %w",
|
||||||
@@ -471,19 +478,29 @@ func deleteRoutesInTable(
|
|||||||
return 0, nil, nil
|
return 0, nil, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
family, deleteSQL, lookupSQL := ipv4Family, deleteLiveRouteV4SQL,
|
family, withOriginSQL, withoutOriginSQL, lookupSQL := ipv4Family,
|
||||||
|
deleteLiveRouteV4SQL, deleteLiveRouteAnyOriginV4SQL,
|
||||||
prefixHasLiveRouteV4SQL
|
prefixHasLiveRouteV4SQL
|
||||||
if ipVersion == ipVersionV6 {
|
if ipVersion == ipVersionV6 {
|
||||||
family, deleteSQL, lookupSQL = ipv6Family, deleteLiveRouteV6SQL,
|
family, withOriginSQL, withoutOriginSQL, lookupSQL = ipv6Family,
|
||||||
|
deleteLiveRouteV6SQL, deleteLiveRouteAnyOriginV6SQL,
|
||||||
prefixHasLiveRouteV6SQL
|
prefixHasLiveRouteV6SQL
|
||||||
}
|
}
|
||||||
|
|
||||||
deleteStmt, err := tx.PrepareContext(ctx, deleteSQL)
|
withOrigin, err := tx.PrepareContext(ctx, withOriginSQL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, nil, fmt.Errorf("failed to prepare %s delete statement: %w",
|
return 0, nil, fmt.Errorf(
|
||||||
|
"failed to prepare %s delete with origin statement: %w", family, err)
|
||||||
|
}
|
||||||
|
defer func() { _ = withOrigin.Close() }()
|
||||||
|
|
||||||
|
withoutOrigin, err := tx.PrepareContext(ctx, withoutOriginSQL)
|
||||||
|
if err != nil {
|
||||||
|
return 0, nil, fmt.Errorf(
|
||||||
|
"failed to prepare %s delete without origin statement: %w",
|
||||||
family, err)
|
family, err)
|
||||||
}
|
}
|
||||||
defer func() { _ = deleteStmt.Close() }()
|
defer func() { _ = withoutOrigin.Close() }()
|
||||||
|
|
||||||
has, err := tx.PrepareContext(ctx, lookupSQL)
|
has, err := tx.PrepareContext(ctx, lookupSQL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -497,7 +514,8 @@ func deleteRoutesInTable(
|
|||||||
var gonePrefixMaskLengths []int
|
var gonePrefixMaskLengths []int
|
||||||
|
|
||||||
for _, del := range deletions {
|
for _, del := range deletions {
|
||||||
affected, prefixGone, err := deleteRouteRow(ctx, deleteStmt, has, del)
|
affected, prefixGone, err := deleteRouteRows(ctx,
|
||||||
|
withOrigin, withoutOrigin, has, del)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, nil, err
|
return 0, nil, err
|
||||||
}
|
}
|
||||||
@@ -1330,8 +1348,30 @@ func (d *Database) UpsertLiveRoute(route *LiveRoute) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// DeleteLiveRoute deletes the live route the peer has for the prefix
|
// deleteLiveRouteRows deletes the live routes for prefix and peerIP from the
|
||||||
func (d *Database) DeleteLiveRoute(prefix, peerIP string) error {
|
// table of the prefix's IP version, only those from originASN unless it is 0.
|
||||||
|
// Literal per-table queries (rather than one formatted with the table name)
|
||||||
|
// keep the delete free of dynamically built SQL.
|
||||||
|
func deleteLiveRouteRows(
|
||||||
|
tx *loggingTx, isV4 bool, prefix string, originASN int, peerIP string,
|
||||||
|
) (sql.Result, error) {
|
||||||
|
switch {
|
||||||
|
case isV4 && originASN == 0:
|
||||||
|
return tx.Exec(deleteLiveRouteAnyOriginV4SQL, prefix, peerIP)
|
||||||
|
case isV4:
|
||||||
|
return tx.Exec(deleteLiveRouteV4SQL, prefix, originASN, peerIP)
|
||||||
|
case originASN == 0:
|
||||||
|
return tx.Exec(deleteLiveRouteAnyOriginV6SQL, prefix, peerIP)
|
||||||
|
default:
|
||||||
|
return tx.Exec(deleteLiveRouteV6SQL, prefix, originASN, peerIP)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeleteLiveRoute deletes a live route
|
||||||
|
// If originASN is 0, deletes all routes for the prefix/peer combination
|
||||||
|
func (d *Database) DeleteLiveRoute(
|
||||||
|
prefix string, originASN int, peerIP string,
|
||||||
|
) error {
|
||||||
d.lock("DeleteLiveRoute")
|
d.lock("DeleteLiveRoute")
|
||||||
defer d.unlock()
|
defer d.unlock()
|
||||||
|
|
||||||
@@ -1351,14 +1391,7 @@ func (d *Database) DeleteLiveRoute(prefix, peerIP string) error {
|
|||||||
}
|
}
|
||||||
defer d.rollback(tx)
|
defer d.rollback(tx)
|
||||||
|
|
||||||
// Literal per-table queries (rather than one formatted with the table name)
|
res, err := deleteLiveRouteRows(tx, isV4, prefix, originASN, peerIP)
|
||||||
// so the delete carries no dynamically built SQL.
|
|
||||||
deleteSQL := deleteLiveRouteV6SQL
|
|
||||||
if isV4 {
|
|
||||||
deleteSQL = deleteLiveRouteV4SQL
|
|
||||||
}
|
|
||||||
|
|
||||||
res, err := tx.Exec(deleteSQL, prefix, peerIP)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -1757,44 +1790,80 @@ func (d *Database) GetRandomPrefixesByLength(
|
|||||||
maskLength, ipVersion, limit)
|
maskLength, ipVersion, limit)
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetRandomPrefixesByLengthContext returns limit prefixes with the given mask
|
// GetRandomPrefixesByLengthContext returns a random sample of prefixes with
|
||||||
// length picked at random, or all of them when there are fewer, each once with
|
// context support
|
||||||
// its most recently updated live route, for a limit up to
|
|
||||||
// maxPrefixesToShuffle. However many routes there are, it makes at most
|
|
||||||
// maxPrefixesToShuffle+1 index searches to list prefixes, at most
|
|
||||||
// sampleLookupsPerPrefix random lookups for each prefix asked for, and one
|
|
||||||
// index search for each listed prefix's newest route.
|
|
||||||
func (d *Database) GetRandomPrefixesByLengthContext(
|
func (d *Database) GetRandomPrefixesByLengthContext(
|
||||||
ctx context.Context, maskLength, ipVersion, limit int) ([]LiveRoute, error) {
|
ctx context.Context, maskLength, ipVersion, limit int,
|
||||||
|
) ([]LiveRoute, error) {
|
||||||
|
// Select unique prefixes with their most recent route information
|
||||||
tableName := tableLiveRoutesV4
|
tableName := tableLiveRoutesV4
|
||||||
if ipVersion == ipVersionV6 {
|
if ipVersion == ipVersionV6 {
|
||||||
tableName = tableLiveRoutesV6
|
tableName = tableLiveRoutesV6
|
||||||
}
|
}
|
||||||
|
|
||||||
// Reading one prefix more than maxPrefixesToShuffle tells whether these
|
query := fmt.Sprintf(`
|
||||||
// are all the prefixes with this mask length.
|
WITH unique_prefixes AS (
|
||||||
prefixes, err := d.firstPrefixesByLength(ctx, tableName, maskLength,
|
SELECT prefix, MAX(last_updated) as max_updated
|
||||||
maxPrefixesToShuffle+1)
|
FROM %s
|
||||||
|
WHERE mask_length = ?
|
||||||
|
GROUP BY prefix
|
||||||
|
ORDER BY RANDOM()
|
||||||
|
LIMIT ?
|
||||||
|
)
|
||||||
|
SELECT lr.prefix, lr.mask_length, lr.origin_asn, lr.as_path,
|
||||||
|
lr.peer_ip, lr.last_updated
|
||||||
|
FROM %s lr
|
||||||
|
INNER JOIN unique_prefixes up
|
||||||
|
ON lr.prefix = up.prefix AND lr.last_updated = up.max_updated
|
||||||
|
WHERE lr.mask_length = ?
|
||||||
|
`, tableName, tableName)
|
||||||
|
|
||||||
|
rows, err := d.db.QueryContext(ctx, query, maskLength, limit, maskLength)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, fmt.Errorf("failed to query random prefixes: %w", err)
|
||||||
}
|
}
|
||||||
|
defer func() {
|
||||||
|
_ = rows.Close()
|
||||||
|
}()
|
||||||
|
|
||||||
//nolint:gosec // The sample needs variety, not unpredictability.
|
var routes []LiveRoute
|
||||||
rand.Shuffle(len(prefixes), func(i, j int) {
|
|
||||||
prefixes[i], prefixes[j] = prefixes[j], prefixes[i]
|
for rows.Next() {
|
||||||
})
|
var (
|
||||||
|
route LiveRoute
|
||||||
|
pathJSON string
|
||||||
|
)
|
||||||
|
|
||||||
|
err := rows.Scan(
|
||||||
|
&route.Prefix,
|
||||||
|
&route.MaskLength,
|
||||||
|
&route.OriginASN,
|
||||||
|
&pathJSON,
|
||||||
|
&route.PeerIP,
|
||||||
|
&route.LastUpdated,
|
||||||
|
)
|
||||||
|
// Set IP version based on which table we queried
|
||||||
|
route.IPVersion = ipVersion
|
||||||
|
|
||||||
if len(prefixes) > maxPrefixesToShuffle {
|
|
||||||
prefixes, err = d.samplePrefixesByLength(ctx, tableName, maskLength,
|
|
||||||
limit, prefixes)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Decode AS path
|
||||||
|
err = json.Unmarshal([]byte(pathJSON), &route.ASPath)
|
||||||
|
if err != nil {
|
||||||
|
route.ASPath = []int{}
|
||||||
|
}
|
||||||
|
|
||||||
|
routes = append(routes, route)
|
||||||
}
|
}
|
||||||
|
|
||||||
prefixes = prefixes[:min(limit, len(prefixes))]
|
err = rows.Err()
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to read random prefixes: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
return d.newestRouteOfEachPrefix(ctx, tableName, ipVersion, prefixes)
|
return routes, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetNextStaleASN returns a random ASN that needs WHOIS data refresh.
|
// GetNextStaleASN returns a random ASN that needs WHOIS data refresh.
|
||||||
@@ -2292,198 +2361,3 @@ func (d *Database) originPrefixes(
|
|||||||
|
|
||||||
return routes, nil
|
return routes, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// firstPrefixesByLength returns the first n prefixes, in text order, with the
|
|
||||||
// given mask length. Each prefix is one search of the mask length and prefix
|
|
||||||
// index for the prefix after the previous one, so the read does not grow with
|
|
||||||
// the number of peers that carry a prefix, as a SELECT DISTINCT would.
|
|
||||||
func (d *Database) firstPrefixesByLength(
|
|
||||||
ctx context.Context, tableName string, maskLength, n int) ([]string, error) {
|
|
||||||
//nolint:gosec // Table name is hardcoded based on IP version
|
|
||||||
query := fmt.Sprintf(`
|
|
||||||
SELECT prefix FROM %s
|
|
||||||
WHERE mask_length = ? AND prefix > ?
|
|
||||||
ORDER BY prefix
|
|
||||||
LIMIT 1
|
|
||||||
`, tableName)
|
|
||||||
|
|
||||||
stmt, err := d.db.PrepareContext(ctx, query)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to prepare prefix query: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = stmt.Close() }()
|
|
||||||
|
|
||||||
var prefixes []string
|
|
||||||
|
|
||||||
previous := "" // every prefix sorts after the empty string
|
|
||||||
|
|
||||||
for len(prefixes) < n {
|
|
||||||
var prefix string
|
|
||||||
|
|
||||||
err := stmt.QueryRowContext(ctx, maskLength, previous).Scan(&prefix)
|
|
||||||
if errors.Is(err, sql.ErrNoRows) {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to list prefixes: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
prefixes = append(prefixes, prefix)
|
|
||||||
previous = prefix
|
|
||||||
}
|
|
||||||
|
|
||||||
return prefixes, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// samplePrefixesByLength picks limit distinct prefixes with the given mask
|
|
||||||
// length at random, in at most sampleLookupsPerPrefix lookups for each prefix
|
|
||||||
// asked for. ORDER BY RANDOM() would read every route with that mask length.
|
|
||||||
// Instead each lookup draws a row id between those of the first and last route
|
|
||||||
// with that mask length and takes the prefix of the first such route at or
|
|
||||||
// after it. SQLite keeps the row id at the end of every index entry, so the
|
|
||||||
// mask length index is ordered by mask length, then row id, and the lookup is
|
|
||||||
// one search of it.
|
|
||||||
//
|
|
||||||
// The sample is uneven, as the page needs a varied sample, not an even one: a
|
|
||||||
// route is picked in proportion to the row ids between it and the previous
|
|
||||||
// route with that mask length, including the row ids of routes with other mask
|
|
||||||
// lengths and of withdrawn routes. A route that follows a long run of
|
|
||||||
// withdrawn routes is picked far more often than one that directly follows
|
|
||||||
// another route with that mask length.
|
|
||||||
//
|
|
||||||
// Should the lookups keep landing on prefixes already picked, the rest are
|
|
||||||
// taken from shuffled, prefixes of that length in random order.
|
|
||||||
func (d *Database) samplePrefixesByLength(
|
|
||||||
ctx context.Context, tableName string, maskLength, limit int,
|
|
||||||
shuffled []string,
|
|
||||||
) ([]string, error) {
|
|
||||||
//nolint:gosec // Table name is hardcoded based on IP version
|
|
||||||
rangeQuery := fmt.Sprintf(`
|
|
||||||
SELECT
|
|
||||||
(SELECT MIN(rowid) FROM %[1]s WHERE mask_length = ?),
|
|
||||||
(SELECT MAX(rowid) FROM %[1]s WHERE mask_length = ?)
|
|
||||||
`, tableName)
|
|
||||||
|
|
||||||
var first, last sql.NullInt64
|
|
||||||
|
|
||||||
err := d.db.QueryRowContext(ctx, rangeQuery, maskLength, maskLength).
|
|
||||||
Scan(&first, &last)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to read row id range: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !first.Valid {
|
|
||||||
// Every route with this mask length was withdrawn since they were listed.
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
//nolint:gosec // Table name is hardcoded based on IP version
|
|
||||||
lookupQuery := fmt.Sprintf(`
|
|
||||||
SELECT prefix FROM %s
|
|
||||||
WHERE mask_length = ? AND rowid >= ?
|
|
||||||
ORDER BY rowid
|
|
||||||
LIMIT 1
|
|
||||||
`, tableName)
|
|
||||||
|
|
||||||
stmt, err := d.db.PrepareContext(ctx, lookupQuery)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to prepare prefix lookup: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = stmt.Close() }()
|
|
||||||
|
|
||||||
picked := make(map[string]bool, limit)
|
|
||||||
prefixes := make([]string, 0, limit)
|
|
||||||
pick := func(prefix string) {
|
|
||||||
if !picked[prefix] {
|
|
||||||
picked[prefix] = true
|
|
||||||
prefixes = append(prefixes, prefix)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
maxLookups := limit * sampleLookupsPerPrefix
|
|
||||||
for lookups := 0; lookups < maxLookups && len(prefixes) < limit; lookups++ {
|
|
||||||
//nolint:gosec // The sample needs variety, not unpredictability.
|
|
||||||
rowID := first.Int64 + rand.Int64N(last.Int64-first.Int64+1)
|
|
||||||
|
|
||||||
var prefix string
|
|
||||||
|
|
||||||
err := stmt.QueryRowContext(ctx, maskLength, rowID).Scan(&prefix)
|
|
||||||
if errors.Is(err, sql.ErrNoRows) {
|
|
||||||
// The routes from rowID on were withdrawn since the range was read.
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to look up prefix: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
pick(prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, prefix := range shuffled {
|
|
||||||
if len(prefixes) == limit {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
|
|
||||||
pick(prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
return prefixes, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// newestRouteOfEachPrefix returns the most recently updated live route of each
|
|
||||||
// prefix, one search of the prefix and last_updated index per prefix however
|
|
||||||
// many peers carry it. A prefix whose routes were all withdrawn since it was
|
|
||||||
// picked is left out.
|
|
||||||
func (d *Database) newestRouteOfEachPrefix(
|
|
||||||
ctx context.Context, tableName string, ipVersion int, prefixes []string,
|
|
||||||
) ([]LiveRoute, error) {
|
|
||||||
//nolint:gosec // Table name is hardcoded based on IP version
|
|
||||||
query := fmt.Sprintf(`
|
|
||||||
SELECT prefix, mask_length, origin_asn, as_path, peer_ip, last_updated
|
|
||||||
FROM %s
|
|
||||||
WHERE prefix = ?
|
|
||||||
ORDER BY last_updated DESC
|
|
||||||
LIMIT 1
|
|
||||||
`, tableName)
|
|
||||||
|
|
||||||
stmt, err := d.db.PrepareContext(ctx, query)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to prepare route query: %w", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = stmt.Close() }()
|
|
||||||
|
|
||||||
routes := make([]LiveRoute, 0, len(prefixes))
|
|
||||||
|
|
||||||
for _, prefix := range prefixes {
|
|
||||||
route := LiveRoute{IPVersion: ipVersion}
|
|
||||||
|
|
||||||
var pathJSON string
|
|
||||||
|
|
||||||
err := stmt.QueryRowContext(ctx, prefix).Scan(
|
|
||||||
&route.Prefix,
|
|
||||||
&route.MaskLength,
|
|
||||||
&route.OriginASN,
|
|
||||||
&pathJSON,
|
|
||||||
&route.PeerIP,
|
|
||||||
&route.LastUpdated,
|
|
||||||
)
|
|
||||||
if errors.Is(err, sql.ErrNoRows) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to query route of %s: %w", prefix, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = json.Unmarshal([]byte(pathJSON), &route.ASPath)
|
|
||||||
if err != nil {
|
|
||||||
route.ASPath = []int{}
|
|
||||||
}
|
|
||||||
|
|
||||||
routes = append(routes, route)
|
|
||||||
}
|
|
||||||
|
|
||||||
return routes, nil
|
|
||||||
}
|
|
||||||
@@ -6,7 +6,6 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"slices"
|
|
||||||
"sync"
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -14,7 +13,6 @@ import (
|
|||||||
"git.eeqj.de/sneak/routewatch/internal/config"
|
"git.eeqj.de/sneak/routewatch/internal/config"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
"git.eeqj.de/sneak/routewatch/internal/logger"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/mattn/go-sqlite3"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// tempStoreMemory is the PRAGMA temp_store value meaning "hold temp B-trees in
|
// tempStoreMemory is the PRAGMA temp_store value meaning "hold temp B-trees in
|
||||||
@@ -47,32 +45,6 @@ const (
|
|||||||
asnSecondBand = 100
|
asnSecondBand = 100
|
||||||
)
|
)
|
||||||
|
|
||||||
// Parameters for the prefix-length sample tests.
|
|
||||||
const (
|
|
||||||
// sampleLimit is how many prefixes the tests ask the sample for.
|
|
||||||
sampleLimit = 10
|
|
||||||
// sampledPrefixCount is how many prefixes the sampling test stores, more
|
|
||||||
// than sampleLimit so the sample has to choose among them.
|
|
||||||
sampledPrefixCount = 30
|
|
||||||
// secondV4PeerIP and secondV6PeerIP are the second peer each prefix is
|
|
||||||
// learned from, besides the one mkV4Route and mkV6Route set.
|
|
||||||
secondV4PeerIP = "192.0.2.2"
|
|
||||||
secondV6PeerIP = "2001:db8::2"
|
|
||||||
// largeSamplePrefixCount is how many prefixes the test of a mask length
|
|
||||||
// with more than maxPrefixesToShuffle prefixes stores: twice that, so about
|
|
||||||
// half of them come after the first maxPrefixesToShuffle+1 in text order.
|
|
||||||
largeSamplePrefixCount = 2 * maxPrefixesToShuffle
|
|
||||||
// largeSampleLimit is how many prefixes that test asks for. Each random
|
|
||||||
// lookup lands after the first maxPrefixesToShuffle+1 in text order about
|
|
||||||
// half the time, so none of them doing so is about as likely as
|
|
||||||
// largeSampleLimit coin tosses all coming up heads.
|
|
||||||
largeSampleLimit = 30
|
|
||||||
// liveRoutesV4Table and liveRoutesV6Table are the live route tables the
|
|
||||||
// tests pass to the sampling steps.
|
|
||||||
liveRoutesV4Table = "live_routes_v4"
|
|
||||||
liveRoutesV6Table = "live_routes_v6"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestGetIPInfoFindsMostSpecificLiveRoute stores nested live prefixes for both
|
// TestGetIPInfoFindsMostSpecificLiveRoute stores nested live prefixes for both
|
||||||
// families and checks that a lookup returns the most specific one covering the
|
// families and checks that a lookup returns the most specific one covering the
|
||||||
// address, ErrNoRoute when none covers it, and the next less specific prefix
|
// address, ErrNoRoute when none covers it, and the next less specific prefix
|
||||||
@@ -137,8 +109,14 @@ func TestGetIPInfoFindsMostSpecificLiveRoute(t *testing.T) {
|
|||||||
lookup("2001:db9::1", "")
|
lookup("2001:db9::1", "")
|
||||||
|
|
||||||
err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{
|
err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{
|
||||||
{Prefix: "10.1.2.0/24", PeerIP: testPeerIP, IPVersion: ipVersionV4},
|
{
|
||||||
{Prefix: "2001:db8:1:2::/64", PeerIP: testPeerIP, IPVersion: ipVersionV6},
|
Prefix: "10.1.2.0/24", OriginASN: 64502,
|
||||||
|
PeerIP: testPeerIP, IPVersion: ipVersionV4,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Prefix: "2001:db8:1:2::/64", OriginASN: 64502,
|
||||||
|
PeerIP: testPeerIP, IPVersion: ipVersionV6,
|
||||||
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("DeleteLiveRouteBatch: %v", err)
|
t.Fatalf("DeleteLiveRouteBatch: %v", err)
|
||||||
@@ -213,55 +191,6 @@ func TestGetIPInfoIgnoresShortRoutes(t *testing.T) {
|
|||||||
lookup("2001:db8::1", "2001:db8::/32", 64502)
|
lookup("2001:db8::1", "2001:db8::/32", 64502)
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestLiveRouteTablesHoldOneRoutePerPrefixAndPeer checks that each live route
|
|
||||||
// table refuses a second route from a peer for a prefix, even with another
|
|
||||||
// origin AS. An upsert updates the peer's route before it would insert one, so
|
|
||||||
// only an insert straight into the table shows what the table allows.
|
|
||||||
func TestLiveRouteTablesHoldOneRoutePerPrefixAndPeer(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
const otherOriginASN = 64501
|
|
||||||
|
|
||||||
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
|
|
||||||
for _, tt := range []struct {
|
|
||||||
route *LiveRoute
|
|
||||||
insertSQL string
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
route: mkV4Route(t, "198.51.100.0/24", 64500, ts),
|
|
||||||
insertSQL: insertLiveRouteV4SQL,
|
|
||||||
},
|
|
||||||
{
|
|
||||||
route: mkV6Route(testV6Prefix, 64500, ts),
|
|
||||||
insertSQL: insertLiveRouteV6SQL,
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
err := db.UpsertLiveRoute(tt.route)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("UpsertLiveRoute(%s): %v", tt.route.Prefix, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = db.db.ExecContext(t.Context(), tt.insertSQL, uuid.NewString(),
|
|
||||||
tt.route.Prefix, tt.route.MaskLength, otherOriginASN,
|
|
||||||
tt.route.PeerIP, "[]", tt.route.NextHop, ts)
|
|
||||||
|
|
||||||
var sqliteErr sqlite3.Error
|
|
||||||
if !errors.As(err, &sqliteErr) ||
|
|
||||||
sqliteErr.ExtendedCode != sqlite3.ErrConstraintUnique {
|
|
||||||
t.Errorf("second route from peer %s for %s: error %v, "+
|
|
||||||
"want a UNIQUE constraint error",
|
|
||||||
tt.route.PeerIP, tt.route.Prefix, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// writeOriginRoutes writes one live route from testPeerIP for each prefix in
|
// writeOriginRoutes writes one live route from testPeerIP for each prefix in
|
||||||
// origins, originated by the AS it maps to.
|
// origins, originated by the AS it maps to.
|
||||||
func writeOriginRoutes(t *testing.T, db *Database, origins map[string]int) {
|
func writeOriginRoutes(t *testing.T, db *Database, origins map[string]int) {
|
||||||
@@ -404,6 +333,7 @@ func TestVacuumReturnsFreePages(t *testing.T) {
|
|||||||
routes = append(routes, route)
|
routes = append(routes, route)
|
||||||
deletions = append(deletions, LiveRouteDeletion{
|
deletions = append(deletions, LiveRouteDeletion{
|
||||||
Prefix: route.Prefix,
|
Prefix: route.Prefix,
|
||||||
|
OriginASN: asn,
|
||||||
PeerIP: route.PeerIP,
|
PeerIP: route.PeerIP,
|
||||||
IPVersion: ipVersionV4,
|
IPVersion: ipVersionV4,
|
||||||
})
|
})
|
||||||
@@ -502,346 +432,3 @@ func TestBatchWriteDuringCheckpoint(t *testing.T) {
|
|||||||
cancel()
|
cancel()
|
||||||
wg.Wait()
|
wg.Wait()
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each
|
|
||||||
// family learned from two peers in the same second and checks that the
|
|
||||||
// prefix-length sample lists each of them once.
|
|
||||||
func TestGetRandomPrefixesByLengthListsPrefixOnce(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := &config.Config{StateDir: t.TempDir()}
|
|
||||||
|
|
||||||
db, err := New(cfg, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
v4First := mkV4Route(t, "198.51.100.0/24", 64500, ts)
|
|
||||||
v4Second := mkV4Route(t, "198.51.100.0/24", 64500, ts)
|
|
||||||
v4Second.PeerIP = secondV4PeerIP
|
|
||||||
v6First := mkV6Route(testV6Prefix, 64500, ts)
|
|
||||||
v6Second := mkV6Route(testV6Prefix, 64500, ts)
|
|
||||||
v6Second.PeerIP = secondV6PeerIP
|
|
||||||
|
|
||||||
err = db.UpsertLiveRouteBatch(
|
|
||||||
[]*LiveRoute{v4First, v4Second, v6First, v6Second})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, want := range []*LiveRoute{v4First, v6First} {
|
|
||||||
routes, err := db.GetRandomPrefixesByLength(want.MaskLength,
|
|
||||||
want.IPVersion, sampleLimit)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetRandomPrefixesByLength(%d, %d): %v",
|
|
||||||
want.MaskLength, want.IPVersion, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(routes) != 1 || routes[0].Prefix != want.Prefix {
|
|
||||||
t.Errorf("GetRandomPrefixesByLength(%d, %d) = %+v, want %s listed once",
|
|
||||||
want.MaskLength, want.IPVersion, routes, want.Prefix)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestGetRandomPrefixesByLengthListsNewestRoute stores a prefix learned from
|
|
||||||
// two peers at different times, the older route first, and checks that the
|
|
||||||
// prefix-length sample lists the newer route.
|
|
||||||
func TestGetRandomPrefixesByLengthListsNewestRoute(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := &config.Config{StateDir: t.TempDir()}
|
|
||||||
|
|
||||||
db, err := New(cfg, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
older := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
olderRoute := mkV4Route(t, "198.51.100.0/24", 64500, older)
|
|
||||||
newerRoute := mkV4Route(t, "198.51.100.0/24", 64500, older.Add(time.Minute))
|
|
||||||
newerRoute.PeerIP = secondV4PeerIP
|
|
||||||
|
|
||||||
err = db.UpsertLiveRouteBatch([]*LiveRoute{olderRoute, newerRoute})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
routes, err := db.GetRandomPrefixesByLength(newerRoute.MaskLength,
|
|
||||||
ipVersionV4, sampleLimit)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetRandomPrefixesByLength: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(routes) != 1 || routes[0].PeerIP != newerRoute.PeerIP {
|
|
||||||
t.Errorf("GetRandomPrefixesByLength = %+v, want only the newer route, "+
|
|
||||||
"from %s", routes, newerRoute.PeerIP)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestGetRandomPrefixesByLengthSamplesManyPrefixes stores more /24 prefixes
|
|
||||||
// than the sample asks for, each followed by a /16 so that routes of another
|
|
||||||
// mask length sit between the /24 rows, and each prefix learned from two peers
|
|
||||||
// in the same second. It checks both the sample the pages get, picked from the
|
|
||||||
// shuffled prefixes, and the random lookups used for a mask length with more
|
|
||||||
// than maxPrefixesToShuffle prefixes.
|
|
||||||
func TestGetRandomPrefixesByLengthSamplesManyPrefixes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := &config.Config{StateDir: t.TempDir()}
|
|
||||||
|
|
||||||
db, err := New(cfg, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
|
|
||||||
var routes []*LiveRoute
|
|
||||||
|
|
||||||
stored := make([]string, 0, sampledPrefixCount)
|
|
||||||
|
|
||||||
for i := range sampledPrefixCount {
|
|
||||||
prefix := fmt.Sprintf("10.0.%d.0/24", i)
|
|
||||||
for _, p := range []string{prefix, fmt.Sprintf("172.%d.0.0/16", i)} {
|
|
||||||
first := mkV4Route(t, p, 64500, ts)
|
|
||||||
second := mkV4Route(t, p, 64500, ts)
|
|
||||||
second.PeerIP = secondV4PeerIP
|
|
||||||
routes = append(routes, first, second)
|
|
||||||
}
|
|
||||||
|
|
||||||
stored = append(stored, prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = db.UpsertLiveRouteBatch(routes)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
slices.Sort(stored)
|
|
||||||
|
|
||||||
maskLength := routes[0].MaskLength
|
|
||||||
|
|
||||||
sample, err := db.GetRandomPrefixesByLength(maskLength, ipVersionV4, sampleLimit)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetRandomPrefixesByLength: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
listed := make([]string, 0, len(sample))
|
|
||||||
for _, route := range sample {
|
|
||||||
listed = append(listed, route.Prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkPrefixSample(t, "GetRandomPrefixesByLength", listed, stored)
|
|
||||||
|
|
||||||
looked, err := db.samplePrefixesByLength(
|
|
||||||
context.Background(), liveRoutesV4Table, maskLength, sampleLimit, nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("samplePrefixesByLength: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkPrefixSample(t, "samplePrefixesByLength", looked, stored)
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkPrefixSample checks that sample lists sampleLimit prefixes, each once
|
|
||||||
// and each one of stored, and that it is not just the first sampleLimit of
|
|
||||||
// stored, which is in text order.
|
|
||||||
func checkPrefixSample(t *testing.T, name string, sample, stored []string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
if len(sample) != sampleLimit {
|
|
||||||
t.Errorf("%s listed %d prefixes, want %d", name, len(sample), sampleLimit)
|
|
||||||
}
|
|
||||||
|
|
||||||
listed := make(map[string]bool)
|
|
||||||
|
|
||||||
for _, prefix := range sample {
|
|
||||||
if listed[prefix] || !slices.Contains(stored, prefix) {
|
|
||||||
t.Errorf("%s lists %s twice or it is not one of the /24s", name, prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
listed[prefix] = true
|
|
||||||
}
|
|
||||||
|
|
||||||
if slices.Equal(slices.Sorted(slices.Values(sample)), stored[:sampleLimit]) {
|
|
||||||
t.Errorf("%s lists just the first %d prefixes in text order",
|
|
||||||
name, sampleLimit)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestGetRandomPrefixesByLengthSamplesPastFirstPrefixes stores more prefixes
|
|
||||||
// of one mask length than maxPrefixesToShuffle. It checks that
|
|
||||||
// firstPrefixesByLength reads just the first maxPrefixesToShuffle+1 in text
|
|
||||||
// order, and that the sample lists the prefixes asked for, each once, some of
|
|
||||||
// them after those first ones, which only the random lookups reach.
|
|
||||||
func TestGetRandomPrefixesByLengthSamplesPastFirstPrefixes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := &config.Config{StateDir: t.TempDir()}
|
|
||||||
|
|
||||||
db, err := New(cfg, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
routes := make([]*LiveRoute, 0, largeSamplePrefixCount)
|
|
||||||
stored := make([]string, 0, largeSamplePrefixCount)
|
|
||||||
|
|
||||||
for i := range largeSamplePrefixCount {
|
|
||||||
route := mkV6Route(fmt.Sprintf("2001:%x::/32", i), 64500, ts)
|
|
||||||
routes = append(routes, route)
|
|
||||||
stored = append(stored, route.Prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = db.UpsertLiveRouteBatch(routes)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
slices.Sort(stored)
|
|
||||||
firstPrefixes := stored[:maxPrefixesToShuffle+1]
|
|
||||||
maskLength := routes[0].MaskLength
|
|
||||||
|
|
||||||
read, err := db.firstPrefixesByLength(context.Background(),
|
|
||||||
liveRoutesV6Table, maskLength, maxPrefixesToShuffle+1)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("firstPrefixesByLength: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(read, firstPrefixes) {
|
|
||||||
t.Errorf("firstPrefixesByLength read %d prefixes, want the first %d "+
|
|
||||||
"in text order", len(read), len(firstPrefixes))
|
|
||||||
}
|
|
||||||
|
|
||||||
sample, err := db.GetRandomPrefixesByLength(maskLength, ipVersionV6,
|
|
||||||
largeSampleLimit)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetRandomPrefixesByLength: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(sample) != largeSampleLimit {
|
|
||||||
t.Errorf("GetRandomPrefixesByLength listed %d prefixes, want %d",
|
|
||||||
len(sample), largeSampleLimit)
|
|
||||||
}
|
|
||||||
|
|
||||||
listed := make(map[string]bool)
|
|
||||||
pastFirstPrefixes := 0
|
|
||||||
|
|
||||||
for _, route := range sample {
|
|
||||||
if listed[route.Prefix] {
|
|
||||||
t.Errorf("GetRandomPrefixesByLength lists %s twice", route.Prefix)
|
|
||||||
}
|
|
||||||
|
|
||||||
listed[route.Prefix] = true
|
|
||||||
|
|
||||||
if !slices.Contains(firstPrefixes, route.Prefix) {
|
|
||||||
pastFirstPrefixes++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if pastFirstPrefixes == 0 {
|
|
||||||
t.Errorf("GetRandomPrefixesByLength lists only prefixes among the first "+
|
|
||||||
"%d in text order", len(firstPrefixes))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSamplePrefixesByLengthFillsUpFromShuffled stores a single prefix, so
|
|
||||||
// every random lookup lands on it, and checks that the sample still lists
|
|
||||||
// sampleLimit prefixes: the stored one, then the shuffled ones in order. The
|
|
||||||
// stored prefix is the second shuffled one, so the fill-up must skip it.
|
|
||||||
func TestSamplePrefixesByLengthFillsUpFromShuffled(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := &config.Config{StateDir: t.TempDir()}
|
|
||||||
|
|
||||||
db, err := New(cfg, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
shuffled := make([]string, 0, sampledPrefixCount)
|
|
||||||
for i := range sampledPrefixCount {
|
|
||||||
shuffled = append(shuffled, fmt.Sprintf("10.0.%d.0/24", i))
|
|
||||||
}
|
|
||||||
|
|
||||||
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
route := mkV4Route(t, shuffled[1], 64500, ts)
|
|
||||||
|
|
||||||
err = db.UpsertLiveRouteBatch([]*LiveRoute{route})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
sample, err := db.samplePrefixesByLength(context.Background(),
|
|
||||||
liveRoutesV4Table, route.MaskLength, sampleLimit, shuffled)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("samplePrefixesByLength: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := append([]string{shuffled[1], shuffled[0]}, shuffled[2:sampleLimit]...)
|
|
||||||
if !slices.Equal(sample, want) {
|
|
||||||
t.Errorf("samplePrefixesByLength = %v, want %v", sample, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestGetRandomPrefixesByLengthSkipsWithdrawnRoutes withdraws a route after
|
|
||||||
// storing it, as can happen between the steps of GetRandomPrefixesByLength. It
|
|
||||||
// checks that the random lookups pick nothing for a mask length with no route
|
|
||||||
// left, and that only the prefixes that still have a route get their newest
|
|
||||||
// route listed.
|
|
||||||
func TestGetRandomPrefixesByLengthSkipsWithdrawnRoutes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cfg := &config.Config{StateDir: t.TempDir()}
|
|
||||||
|
|
||||||
db, err := New(cfg, logger.New())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create database: %v", err)
|
|
||||||
}
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
kept := mkV4Route(t, "198.51.100.0/24", 64500, ts)
|
|
||||||
withdrawn := mkV4Route(t, "172.16.0.0/16", 64500, ts)
|
|
||||||
|
|
||||||
err = db.UpsertLiveRouteBatch([]*LiveRoute{kept, withdrawn})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("UpsertLiveRouteBatch: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = db.DeleteLiveRoute(withdrawn.Prefix, withdrawn.PeerIP)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("DeleteLiveRoute: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
|
|
||||||
sample, err := db.samplePrefixesByLength(ctx, liveRoutesV4Table,
|
|
||||||
withdrawn.MaskLength, sampleLimit, []string{withdrawn.Prefix})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("samplePrefixesByLength: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(sample) != 0 {
|
|
||||||
t.Errorf("samplePrefixesByLength = %v, want none for a mask length "+
|
|
||||||
"with no route left", sample)
|
|
||||||
}
|
|
||||||
|
|
||||||
routes, err := db.newestRouteOfEachPrefix(ctx, liveRoutesV4Table,
|
|
||||||
ipVersionV4, []string{withdrawn.Prefix, kept.Prefix})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("newestRouteOfEachPrefix: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(routes) != 1 || routes[0].Prefix != kept.Prefix {
|
|
||||||
t.Errorf("newestRouteOfEachPrefix = %+v, want only the route of %s",
|
|
||||||
routes, kept.Prefix)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -57,7 +57,7 @@ type Store interface {
|
|||||||
// Live route operations
|
// Live route operations
|
||||||
UpsertLiveRoute(route *LiveRoute) error
|
UpsertLiveRoute(route *LiveRoute) error
|
||||||
UpsertLiveRouteBatch(routes []*LiveRoute) error
|
UpsertLiveRouteBatch(routes []*LiveRoute) error
|
||||||
DeleteLiveRoute(prefix, peerIP string) error
|
DeleteLiveRoute(prefix string, originASN int, peerIP string) error
|
||||||
DeleteLiveRouteBatch(deletions []LiveRouteDeletion) error
|
DeleteLiveRouteBatch(deletions []LiveRouteDeletion) error
|
||||||
GetPrefixDistribution() (
|
GetPrefixDistribution() (
|
||||||
ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error)
|
ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error)
|
||||||
|
|||||||
@@ -121,6 +121,7 @@ type IPInfo struct {
|
|||||||
// LiveRouteDeletion represents parameters for deleting a live route
|
// LiveRouteDeletion represents parameters for deleting a live route
|
||||||
type LiveRouteDeletion struct {
|
type LiveRouteDeletion struct {
|
||||||
Prefix string
|
Prefix string
|
||||||
|
OriginASN int
|
||||||
PeerIP string
|
PeerIP string
|
||||||
IPVersion int
|
IPVersion int
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -107,7 +107,7 @@ CREATE TABLE IF NOT EXISTS live_routes_v4 (
|
|||||||
as_path TEXT NOT NULL, -- JSON array
|
as_path TEXT NOT NULL, -- JSON array
|
||||||
next_hop TEXT NOT NULL,
|
next_hop TEXT NOT NULL,
|
||||||
last_updated DATETIME NOT NULL,
|
last_updated DATETIME NOT NULL,
|
||||||
UNIQUE(prefix, peer_ip) -- a peer has at most one route for a prefix
|
UNIQUE(prefix, origin_asn, peer_ip)
|
||||||
);
|
);
|
||||||
|
|
||||||
-- IPv6 routing table maintained by PrefixHandler
|
-- IPv6 routing table maintained by PrefixHandler
|
||||||
@@ -120,12 +120,11 @@ CREATE TABLE IF NOT EXISTS live_routes_v6 (
|
|||||||
as_path TEXT NOT NULL, -- JSON array
|
as_path TEXT NOT NULL, -- JSON array
|
||||||
next_hop TEXT NOT NULL,
|
next_hop TEXT NOT NULL,
|
||||||
last_updated DATETIME NOT NULL,
|
last_updated DATETIME NOT NULL,
|
||||||
UNIQUE(prefix, peer_ip) -- a peer has at most one route for a prefix
|
UNIQUE(prefix, origin_asn, peer_ip)
|
||||||
);
|
);
|
||||||
|
|
||||||
-- Indexes for live_routes_v4 table
|
-- Indexes for live_routes_v4 table
|
||||||
-- Finds a prefix's newest route in one search, whatever the number of peers.
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_prefix ON live_routes_v4(prefix);
|
||||||
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_prefix_last_updated ON live_routes_v4(prefix, last_updated);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_length ON live_routes_v4(mask_length);
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_length ON live_routes_v4(mask_length);
|
||||||
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_origin_asn ON live_routes_v4(origin_asn);
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_origin_asn ON live_routes_v4(origin_asn);
|
||||||
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_last_updated ON live_routes_v4(last_updated);
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_last_updated ON live_routes_v4(last_updated);
|
||||||
@@ -133,8 +132,7 @@ CREATE INDEX IF NOT EXISTS idx_live_routes_v4_last_updated ON live_routes_v4(las
|
|||||||
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_prefix ON live_routes_v4(mask_length, prefix);
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_prefix ON live_routes_v4(mask_length, prefix);
|
||||||
|
|
||||||
-- Indexes for live_routes_v6 table
|
-- Indexes for live_routes_v6 table
|
||||||
-- Finds a prefix's newest route in one search, whatever the number of peers.
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_prefix ON live_routes_v6(prefix);
|
||||||
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_prefix_last_updated ON live_routes_v6(prefix, last_updated);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_mask_length ON live_routes_v6(mask_length);
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_mask_length ON live_routes_v6(mask_length);
|
||||||
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_origin_asn ON live_routes_v6(origin_asn);
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_origin_asn ON live_routes_v6(origin_asn);
|
||||||
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_last_updated ON live_routes_v6(last_updated);
|
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_last_updated ON live_routes_v6(last_updated);
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ func logSlowQuery(logger *logger.Logger, query string, start time.Time) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// queryRow wraps QueryRow with slow query logging
|
||||||
|
//
|
||||||
|
//nolint:unused // kept for consistency with other query wrappers
|
||||||
|
func (d *Database) queryRow(query string, args ...any) *sql.Row {
|
||||||
|
start := time.Now()
|
||||||
|
defer logSlowQuery(d.logger, query, start)
|
||||||
|
|
||||||
|
return d.db.QueryRowContext(context.Background(), query, args...)
|
||||||
|
}
|
||||||
|
|
||||||
// exec wraps Exec with slow query logging
|
// exec wraps Exec with slow query logging
|
||||||
func (d *Database) exec(query string, args ...any) error {
|
func (d *Database) exec(query string, args ...any) error {
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
|
|||||||
@@ -210,7 +210,7 @@ func (m *mockStore) UpsertLiveRoute(_ *database.LiveRoute) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// DeleteLiveRoute mock implementation
|
// DeleteLiveRoute mock implementation
|
||||||
func (m *mockStore) DeleteLiveRoute(_, _ string) error {
|
func (m *mockStore) DeleteLiveRoute(_ string, _ int, _ string) error {
|
||||||
// Simple mock - just return nil
|
// Simple mock - just return nil
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,12 +19,6 @@ const (
|
|||||||
// asnBatchSize is the number of ASN operations to batch together
|
// asnBatchSize is the number of ASN operations to batch together
|
||||||
asnBatchSize = 30000
|
asnBatchSize = 30000
|
||||||
|
|
||||||
// asnBatchMaxSize is the most ASN operations the batch holds while the
|
|
||||||
// flush goroutine writes the previous one: ten batches, about 10 MiB
|
|
||||||
// (README "Memory"). At this size HandleMessage waits until the flush
|
|
||||||
// goroutine takes the batch.
|
|
||||||
asnBatchMaxSize = 300000
|
|
||||||
|
|
||||||
// asnBatchTimeout is the maximum time to wait before flushing a batch
|
// asnBatchTimeout is the maximum time to wait before flushing a batch
|
||||||
// DO NOT reduce this timeout - larger batches are more efficient
|
// DO NOT reduce this timeout - larger batches are more efficient
|
||||||
asnBatchTimeout = 2 * time.Second
|
asnBatchTimeout = 2 * time.Second
|
||||||
@@ -38,18 +32,12 @@ type ASHandler struct {
|
|||||||
db database.Store
|
db database.Store
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
|
|
||||||
// Batching. mu guards batch only: the flush goroutine takes the batch
|
// Batching
|
||||||
// under mu and writes it after releasing mu, so HandleMessage waits for
|
mu sync.Mutex
|
||||||
// a database write only while batch is at asnBatchMaxSize.
|
batch []asnOp
|
||||||
mu sync.Mutex
|
lastFlush time.Time
|
||||||
batch []asnOp
|
stopCh chan struct{}
|
||||||
// batchTaken is signalled when the flush goroutine takes batch; its L is
|
wg sync.WaitGroup
|
||||||
// &mu.
|
|
||||||
batchTaken sync.Cond
|
|
||||||
batchFullCh chan struct{} // wakes the flush goroutine when batch is full
|
|
||||||
lastFlush time.Time // used only by the flush goroutine
|
|
||||||
stopCh chan struct{}
|
|
||||||
wg sync.WaitGroup
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type asnOp struct {
|
type asnOp struct {
|
||||||
@@ -64,14 +52,12 @@ type asnOp struct {
|
|||||||
// background goroutine is terminated.
|
// background goroutine is terminated.
|
||||||
func NewASHandler(db database.Store, logger *logger.Logger) *ASHandler {
|
func NewASHandler(db database.Store, logger *logger.Logger) *ASHandler {
|
||||||
h := &ASHandler{
|
h := &ASHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
batch: make([]asnOp, 0, asnBatchSize),
|
batch: make([]asnOp, 0, asnBatchSize),
|
||||||
batchFullCh: make(chan struct{}, 1),
|
lastFlush: time.Now(),
|
||||||
lastFlush: time.Now(),
|
stopCh: make(chan struct{}),
|
||||||
stopCh: make(chan struct{}),
|
|
||||||
}
|
}
|
||||||
h.batchTaken.L = &h.mu
|
|
||||||
|
|
||||||
// Start the flush timer goroutine
|
// Start the flush timer goroutine
|
||||||
h.wg.Add(1)
|
h.wg.Add(1)
|
||||||
@@ -99,8 +85,8 @@ func (h *ASHandler) QueueCapacity() int {
|
|||||||
// HandleMessage processes a RIS Live BGP message by extracting all ASNs from
|
// HandleMessage processes a RIS Live BGP message by extracting all ASNs from
|
||||||
// the AS path and queuing them for batch insertion into the database. The
|
// the AS path and queuing them for batch insertion into the database. The
|
||||||
// origin ASN (last element in the path) and all transit ASNs are recorded
|
// origin ASN (last element in the path) and all transit ASNs are recorded
|
||||||
// with their associated timestamps. When the batch reaches the configured size
|
// with their associated timestamps. The batch is automatically flushed when
|
||||||
// threshold, the flush goroutine is woken to write it.
|
// it reaches the configured size threshold.
|
||||||
func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
|
func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||||
// Use the pre-parsed timestamp
|
// Use the pre-parsed timestamp
|
||||||
timestamp := msg.ParsedTimestamp
|
timestamp := msg.ParsedTimestamp
|
||||||
@@ -114,13 +100,6 @@ func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
defer h.mu.Unlock()
|
defer h.mu.Unlock()
|
||||||
|
|
||||||
// While the batch is at its maximum size, wait until the flush goroutine
|
|
||||||
// takes it. Meanwhile the handler takes nothing off its queue, which
|
|
||||||
// fills, and the streamer drops messages for it.
|
|
||||||
for len(h.batch) >= asnBatchMaxSize {
|
|
||||||
h.batchTaken.Wait()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Queue origin ASN operation
|
// Queue origin ASN operation
|
||||||
if originASN > 0 {
|
if originASN > 0 {
|
||||||
h.batch = append(h.batch, asnOp{
|
h.batch = append(h.batch, asnOp{
|
||||||
@@ -139,14 +118,9 @@ func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
// Check if we need to flush
|
||||||
// waits in batchFullCh and the batch keeps growing until the write ends
|
|
||||||
// or the batch reaches asnBatchMaxSize.
|
|
||||||
if len(h.batch) >= asnBatchSize {
|
if len(h.batch) >= asnBatchSize {
|
||||||
select {
|
h.flushBatchLocked()
|
||||||
case h.batchFullCh <- struct{}{}:
|
|
||||||
default: // already woken
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -160,8 +134,7 @@ func (h *ASHandler) Stop() {
|
|||||||
h.wg.Wait()
|
h.wg.Wait()
|
||||||
}
|
}
|
||||||
|
|
||||||
// flushLoop runs in a goroutine, the only one that writes the batch: it
|
// flushLoop runs in a goroutine and periodically flushes batches
|
||||||
// flushes when the batch is full, periodically, and once more on Stop.
|
|
||||||
func (h *ASHandler) flushLoop() {
|
func (h *ASHandler) flushLoop() {
|
||||||
defer h.wg.Done()
|
defer h.wg.Done()
|
||||||
|
|
||||||
@@ -171,48 +144,32 @@ func (h *ASHandler) flushLoop() {
|
|||||||
for {
|
for {
|
||||||
select {
|
select {
|
||||||
case <-ticker.C:
|
case <-ticker.C:
|
||||||
|
h.mu.Lock()
|
||||||
if time.Since(h.lastFlush) >= asnBatchTimeout {
|
if time.Since(h.lastFlush) >= asnBatchTimeout {
|
||||||
h.flushBatch()
|
h.flushBatchLocked()
|
||||||
}
|
}
|
||||||
case <-h.batchFullCh:
|
h.mu.Unlock()
|
||||||
h.flushBatch()
|
|
||||||
case <-h.stopCh:
|
case <-h.stopCh:
|
||||||
// Final flush
|
// Final flush
|
||||||
h.flushBatch()
|
h.mu.Lock()
|
||||||
|
h.flushBatchLocked()
|
||||||
|
h.mu.Unlock()
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// takeBatch returns the pending operations and leaves an empty batch in their
|
// flushBatchLocked flushes the ASN batch to the database (must be called with
|
||||||
// place for HandleMessage to fill.
|
// mutex held)
|
||||||
func (h *ASHandler) takeBatch() []asnOp {
|
func (h *ASHandler) flushBatchLocked() {
|
||||||
h.mu.Lock()
|
|
||||||
defer h.mu.Unlock()
|
|
||||||
|
|
||||||
if len(h.batch) == 0 {
|
if len(h.batch) == 0 {
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
batch := h.batch
|
|
||||||
h.batch = make([]asnOp, 0, asnBatchSize)
|
|
||||||
h.batchTaken.Broadcast()
|
|
||||||
|
|
||||||
return batch
|
|
||||||
}
|
|
||||||
|
|
||||||
// flushBatch takes the pending batch and writes it to the database, without
|
|
||||||
// holding the mutex during the write.
|
|
||||||
func (h *ASHandler) flushBatch() {
|
|
||||||
batch := h.takeBatch()
|
|
||||||
if len(batch) == 0 {
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// Process ASNs first (deduped)
|
// Process ASNs first (deduped)
|
||||||
asnMap := make(map[int]time.Time)
|
asnMap := make(map[int]time.Time)
|
||||||
for _, op := range batch {
|
for _, op := range h.batch {
|
||||||
if existing, ok := asnMap[op.number]; !ok || op.timestamp.After(existing) {
|
if existing, ok := asnMap[op.number]; !ok || op.timestamp.After(existing) {
|
||||||
asnMap[op.number] = op.timestamp
|
asnMap[op.number] = op.timestamp
|
||||||
}
|
}
|
||||||
@@ -225,5 +182,7 @@ func (h *ASHandler) flushBatch() {
|
|||||||
"error", err, "count", len(asnMap))
|
"error", err, "count", len(asnMap))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Clear batch
|
||||||
|
h.batch = h.batch[:0]
|
||||||
h.lastFlush = time.Now()
|
h.lastFlush = time.Now()
|
||||||
}
|
}
|
||||||
@@ -1,425 +0,0 @@
|
|||||||
package routewatch
|
|
||||||
|
|
||||||
import (
|
|
||||||
"slices"
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/logger"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// handleMessageTimeout is how long a test waits for HandleMessage to
|
|
||||||
// return, or for a write to start, before it fails.
|
|
||||||
handleMessageTimeout = 2 * time.Second
|
|
||||||
|
|
||||||
// stillWaitingAfter is how long a test watches a HandleMessage that must
|
|
||||||
// wait. One that does not wait returns well within it.
|
|
||||||
stillWaitingAfter = 100 * time.Millisecond
|
|
||||||
)
|
|
||||||
|
|
||||||
// blockingStore is a mockStore whose batch writes for the AS, peer and prefix
|
|
||||||
// handlers wait until the test calls releaseWrites. writeStarted is closed when
|
|
||||||
// the first of them starts. It records what each write was given.
|
|
||||||
type blockingStore struct {
|
|
||||||
*mockStore
|
|
||||||
|
|
||||||
release chan struct{}
|
|
||||||
releaseOnce sync.Once
|
|
||||||
writeStarted chan struct{}
|
|
||||||
startOnce sync.Once
|
|
||||||
|
|
||||||
mu sync.Mutex
|
|
||||||
prefixes map[string]bool
|
|
||||||
asns map[int]bool
|
|
||||||
peers map[string]bool
|
|
||||||
}
|
|
||||||
|
|
||||||
func newBlockingStore() *blockingStore {
|
|
||||||
return &blockingStore{
|
|
||||||
mockStore: newMockStore(),
|
|
||||||
release: make(chan struct{}),
|
|
||||||
writeStarted: make(chan struct{}),
|
|
||||||
prefixes: make(map[string]bool),
|
|
||||||
asns: make(map[int]bool),
|
|
||||||
peers: make(map[string]bool),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *blockingStore) UpsertLiveRouteBatch(routes []*database.LiveRoute) error {
|
|
||||||
s.waitForRelease()
|
|
||||||
s.mu.Lock()
|
|
||||||
defer s.mu.Unlock()
|
|
||||||
|
|
||||||
for _, route := range routes {
|
|
||||||
s.prefixes[route.Prefix] = true
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *blockingStore) GetOrCreateASNBatch(asns map[int]time.Time) error {
|
|
||||||
s.waitForRelease()
|
|
||||||
s.mu.Lock()
|
|
||||||
defer s.mu.Unlock()
|
|
||||||
|
|
||||||
for asn := range asns {
|
|
||||||
s.asns[asn] = true
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *blockingStore) UpdatePeerBatch(peers map[string]database.PeerUpdate) error {
|
|
||||||
s.waitForRelease()
|
|
||||||
s.mu.Lock()
|
|
||||||
defer s.mu.Unlock()
|
|
||||||
|
|
||||||
for peerIP := range peers {
|
|
||||||
s.peers[peerIP] = true
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *blockingStore) waitForRelease() {
|
|
||||||
s.startOnce.Do(func() { close(s.writeStarted) })
|
|
||||||
<-s.release
|
|
||||||
}
|
|
||||||
|
|
||||||
// releaseWrites lets the held writes, and all later ones, finish. A second call
|
|
||||||
// does nothing.
|
|
||||||
func (s *blockingStore) releaseWrites() {
|
|
||||||
s.releaseOnce.Do(func() { close(s.release) })
|
|
||||||
}
|
|
||||||
|
|
||||||
// returnsInTime reports whether f returns within handleMessageTimeout.
|
|
||||||
func returnsInTime(f func()) bool {
|
|
||||||
done := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
f()
|
|
||||||
close(done)
|
|
||||||
}()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-done:
|
|
||||||
return true
|
|
||||||
case <-time.After(handleMessageTimeout):
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// startBlockedWrite hands h the messages in fill, which fill its batch so that
|
|
||||||
// the handler writes it to store, and waits until that write starts. store
|
|
||||||
// holds it until the test calls releaseWrites.
|
|
||||||
func startBlockedWrite(
|
|
||||||
t *testing.T,
|
|
||||||
store *blockingStore,
|
|
||||||
h interface {
|
|
||||||
HandleMessage(msg *ristypes.RISMessage)
|
|
||||||
},
|
|
||||||
fill []*ristypes.RISMessage,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
if !returnsInTime(func() {
|
|
||||||
for _, msg := range fill {
|
|
||||||
h.HandleMessage(msg)
|
|
||||||
}
|
|
||||||
}) {
|
|
||||||
t.Fatal("HandleMessage did not return after filling the batch: " +
|
|
||||||
"it waits for the database write")
|
|
||||||
}
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-store.writeStarted:
|
|
||||||
case <-time.After(handleMessageTimeout):
|
|
||||||
t.Fatal("the handler did not write its full batch")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleWhileAWriteIsBlocked starts a blocked write with the messages in fill
|
|
||||||
// and hands h the message extra while store holds the write. HandleMessage must
|
|
||||||
// return each time. Then it lets the writes finish and stops h, which writes
|
|
||||||
// what it holds.
|
|
||||||
func handleWhileAWriteIsBlocked(
|
|
||||||
t *testing.T,
|
|
||||||
store *blockingStore,
|
|
||||||
h interface {
|
|
||||||
HandleMessage(msg *ristypes.RISMessage)
|
|
||||||
Stop()
|
|
||||||
},
|
|
||||||
fill []*ristypes.RISMessage,
|
|
||||||
extra *ristypes.RISMessage,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
defer h.Stop()
|
|
||||||
defer store.releaseWrites()
|
|
||||||
|
|
||||||
startBlockedWrite(t, store, h, fill)
|
|
||||||
|
|
||||||
if !returnsInTime(func() { h.HandleMessage(extra) }) {
|
|
||||||
t.Fatal("HandleMessage did not return while a database write was blocked")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleAtMaxBatchSize starts a blocked write with the messages in fill, then,
|
|
||||||
// while store holds the write, hands h the messages in toMax, which bring its
|
|
||||||
// next batch to its maximum size, and the message extra. HandleMessage for
|
|
||||||
// extra must wait until the test lets the write finish, and return after. Then
|
|
||||||
// it stops h, which writes what it holds.
|
|
||||||
func handleAtMaxBatchSize(
|
|
||||||
t *testing.T,
|
|
||||||
store *blockingStore,
|
|
||||||
h interface {
|
|
||||||
HandleMessage(msg *ristypes.RISMessage)
|
|
||||||
Stop()
|
|
||||||
},
|
|
||||||
fill, toMax []*ristypes.RISMessage,
|
|
||||||
extra *ristypes.RISMessage,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
defer h.Stop()
|
|
||||||
defer store.releaseWrites()
|
|
||||||
|
|
||||||
startBlockedWrite(t, store, h, fill)
|
|
||||||
|
|
||||||
for _, msg := range toMax {
|
|
||||||
h.HandleMessage(msg)
|
|
||||||
}
|
|
||||||
|
|
||||||
returned := make(chan struct{})
|
|
||||||
|
|
||||||
go func() {
|
|
||||||
h.HandleMessage(extra)
|
|
||||||
close(returned)
|
|
||||||
}()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-returned:
|
|
||||||
t.Fatal("HandleMessage returned while the batch was at its maximum " +
|
|
||||||
"size and a write was blocked")
|
|
||||||
case <-time.After(stillWaitingAfter):
|
|
||||||
}
|
|
||||||
|
|
||||||
store.releaseWrites()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-returned:
|
|
||||||
case <-time.After(handleMessageTimeout):
|
|
||||||
t.Fatal("HandleMessage did not return after the database write finished")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// prefixAnnouncement returns a message in which testPeerIP announces prefixes
|
|
||||||
// with origin AS testASNA.
|
|
||||||
func prefixAnnouncement(prefixes []string) *ristypes.RISMessage {
|
|
||||||
return &ristypes.RISMessage{
|
|
||||||
Peer: testPeerIP,
|
|
||||||
Path: ristypes.ASPath{testASNA},
|
|
||||||
Announcements: []ristypes.RISAnnouncement{
|
|
||||||
{NextHop: testPeerIP, Prefixes: prefixes},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestPrefixHandlerHandlesMessagesWhileAWriteIsBlocked(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const extraPrefix = "198.51.100.0/24"
|
|
||||||
|
|
||||||
store := newBlockingStore()
|
|
||||||
h := NewPrefixHandler(store, logger.New())
|
|
||||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
|
||||||
handleWhileAWriteIsBlocked(t, store, h,
|
|
||||||
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
|
||||||
prefixAnnouncement([]string{extraPrefix}))
|
|
||||||
|
|
||||||
store.mu.Lock()
|
|
||||||
defer store.mu.Unlock()
|
|
||||||
|
|
||||||
if !store.prefixes[extraPrefix] {
|
|
||||||
t.Errorf("Stop did not write %s, handled while a write was blocked",
|
|
||||||
extraPrefix)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestASHandlerHandlesMessagesWhileAWriteIsBlocked(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
store := newBlockingStore()
|
|
||||||
h := NewASHandler(store, logger.New())
|
|
||||||
fill := slices.Repeat(ristypes.ASPath{testASNA}, asnBatchSize)
|
|
||||||
handleWhileAWriteIsBlocked(t, store, h,
|
|
||||||
[]*ristypes.RISMessage{{Path: fill}},
|
|
||||||
&ristypes.RISMessage{Path: ristypes.ASPath{testASNB}})
|
|
||||||
|
|
||||||
store.mu.Lock()
|
|
||||||
defer store.mu.Unlock()
|
|
||||||
|
|
||||||
if !store.asns[testASNB] {
|
|
||||||
t.Errorf("Stop did not write AS%d, handled while a write was blocked",
|
|
||||||
testASNB)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestPeerHandlerHandlesMessagesWhileAWriteIsBlocked(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const otherPeerIP = "2001:db8:ffff::2"
|
|
||||||
|
|
||||||
store := newBlockingStore()
|
|
||||||
h := NewPeerHandler(store, logger.New())
|
|
||||||
handleWhileAWriteIsBlocked(t, store, h,
|
|
||||||
slices.Repeat([]*ristypes.RISMessage{{Peer: testPeerIP}}, peerBatchSize),
|
|
||||||
&ristypes.RISMessage{Peer: otherPeerIP})
|
|
||||||
|
|
||||||
store.mu.Lock()
|
|
||||||
defer store.mu.Unlock()
|
|
||||||
|
|
||||||
if !store.peers[otherPeerIP] {
|
|
||||||
t.Errorf("Stop did not write peer %s, handled while a write was blocked",
|
|
||||||
otherPeerIP)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestPrefixHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const extraPrefix = "198.51.100.0/24"
|
|
||||||
|
|
||||||
store := newBlockingStore()
|
|
||||||
h := NewPrefixHandler(store, logger.New())
|
|
||||||
fill := slices.Repeat([]string{testV4Prefix}, prefixBatchSize)
|
|
||||||
toMax := slices.Repeat([]string{testV4Prefix}, prefixBatchMaxSize)
|
|
||||||
handleAtMaxBatchSize(t, store, h,
|
|
||||||
[]*ristypes.RISMessage{prefixAnnouncement(fill)},
|
|
||||||
[]*ristypes.RISMessage{prefixAnnouncement(toMax)},
|
|
||||||
prefixAnnouncement([]string{extraPrefix}))
|
|
||||||
|
|
||||||
store.mu.Lock()
|
|
||||||
defer store.mu.Unlock()
|
|
||||||
|
|
||||||
if !store.prefixes[extraPrefix] {
|
|
||||||
t.Errorf("Stop did not write %s, handled after the batch reached its "+
|
|
||||||
"maximum size", extraPrefix)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestASHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
store := newBlockingStore()
|
|
||||||
h := NewASHandler(store, logger.New())
|
|
||||||
fill := slices.Repeat(ristypes.ASPath{testASNA}, asnBatchSize)
|
|
||||||
toMax := slices.Repeat(ristypes.ASPath{testASNA}, asnBatchMaxSize)
|
|
||||||
handleAtMaxBatchSize(t, store, h,
|
|
||||||
[]*ristypes.RISMessage{{Path: fill}},
|
|
||||||
[]*ristypes.RISMessage{{Path: toMax}},
|
|
||||||
&ristypes.RISMessage{Path: ristypes.ASPath{testASNB}})
|
|
||||||
|
|
||||||
store.mu.Lock()
|
|
||||||
defer store.mu.Unlock()
|
|
||||||
|
|
||||||
if !store.asns[testASNB] {
|
|
||||||
t.Errorf("Stop did not write AS%d, handled after the batch reached its "+
|
|
||||||
"maximum size", testASNB)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestPeerHandlerWaitsAtTheMaximumBatchSize(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const otherPeerIP = "2001:db8:ffff::2"
|
|
||||||
|
|
||||||
store := newBlockingStore()
|
|
||||||
h := NewPeerHandler(store, logger.New())
|
|
||||||
message := []*ristypes.RISMessage{{Peer: testPeerIP}}
|
|
||||||
handleAtMaxBatchSize(t, store, h,
|
|
||||||
slices.Repeat(message, peerBatchSize),
|
|
||||||
slices.Repeat(message, peerBatchMaxSize),
|
|
||||||
&ristypes.RISMessage{Peer: otherPeerIP})
|
|
||||||
|
|
||||||
store.mu.Lock()
|
|
||||||
defer store.mu.Unlock()
|
|
||||||
|
|
||||||
if !store.peers[otherPeerIP] {
|
|
||||||
t.Errorf("Stop did not write peer %s, handled after the batch reached "+
|
|
||||||
"its maximum size", otherPeerIP)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull hands each handler, built
|
|
||||||
// without its flush goroutine so that nothing takes a wake-up, one update short
|
|
||||||
// of a full batch, and a second one exactly a full batch. Only the full batch
|
|
||||||
// leaves a wake-up in batchFullCh.
|
|
||||||
func TestHandlersWakeTheFlushGoroutineWhenTheBatchIsFull(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
batchSize int
|
|
||||||
// handle hands a new handler n updates and returns its batchFullCh.
|
|
||||||
handle func(n int) chan struct{}
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
name: "prefix",
|
|
||||||
batchSize: prefixBatchSize,
|
|
||||||
handle: func(n int) chan struct{} {
|
|
||||||
h := &PrefixHandler{batchFullCh: make(chan struct{}, 1)}
|
|
||||||
h.HandleMessage(prefixAnnouncement(
|
|
||||||
slices.Repeat([]string{testV4Prefix}, n)))
|
|
||||||
|
|
||||||
return h.batchFullCh
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "AS",
|
|
||||||
batchSize: asnBatchSize,
|
|
||||||
handle: func(n int) chan struct{} {
|
|
||||||
h := &ASHandler{batchFullCh: make(chan struct{}, 1)}
|
|
||||||
// The origin AS and each AS in the path are one update each.
|
|
||||||
h.HandleMessage(&ristypes.RISMessage{
|
|
||||||
Path: slices.Repeat(ristypes.ASPath{testASNA}, n-1),
|
|
||||||
})
|
|
||||||
|
|
||||||
return h.batchFullCh
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "peer",
|
|
||||||
batchSize: peerBatchSize,
|
|
||||||
handle: func(n int) chan struct{} {
|
|
||||||
h := &PeerHandler{batchFullCh: make(chan struct{}, 1)}
|
|
||||||
for range n {
|
|
||||||
h.HandleMessage(&ristypes.RISMessage{Peer: testPeerIP})
|
|
||||||
}
|
|
||||||
|
|
||||||
return h.batchFullCh
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
got := len(tt.handle(tt.batchSize - 1))
|
|
||||||
if got != 0 {
|
|
||||||
t.Errorf("one update short of a full batch left %d wake-ups, "+
|
|
||||||
"want none", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
got = len(tt.handle(tt.batchSize))
|
|
||||||
if got != 1 {
|
|
||||||
t.Errorf("a full batch left %d wake-ups, want 1", got)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -24,12 +24,6 @@ const (
|
|||||||
// peerBatchSize is the number of peer updates to batch together
|
// peerBatchSize is the number of peer updates to batch together
|
||||||
peerBatchSize = 10000
|
peerBatchSize = 10000
|
||||||
|
|
||||||
// peerBatchMaxSize is the most peer updates the batch holds while the
|
|
||||||
// flush goroutine writes the previous one: ten batches, about 10 MiB
|
|
||||||
// (README "Memory"). At this size HandleMessage waits until the flush
|
|
||||||
// goroutine takes the batch.
|
|
||||||
peerBatchMaxSize = 100000
|
|
||||||
|
|
||||||
// peerBatchTimeout is the maximum time to wait before flushing a batch
|
// peerBatchTimeout is the maximum time to wait before flushing a batch
|
||||||
peerBatchTimeout = 2 * time.Second
|
peerBatchTimeout = 2 * time.Second
|
||||||
)
|
)
|
||||||
@@ -42,18 +36,12 @@ type PeerHandler struct {
|
|||||||
db database.Store
|
db database.Store
|
||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
|
|
||||||
// Batching. mu guards peerBatch only: the flush goroutine takes the batch
|
// Batching
|
||||||
// under mu and writes it after releasing mu, so HandleMessage waits for
|
|
||||||
// a database write only while peerBatch is at peerBatchMaxSize.
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
peerBatch []peerUpdate
|
peerBatch []peerUpdate
|
||||||
// batchTaken is signalled when the flush goroutine takes peerBatch; its L
|
lastFlush time.Time
|
||||||
// is &mu.
|
stopCh chan struct{}
|
||||||
batchTaken sync.Cond
|
wg sync.WaitGroup
|
||||||
batchFullCh chan struct{} // wakes the flush goroutine when peerBatch is full
|
|
||||||
lastFlush time.Time // used only by the flush goroutine
|
|
||||||
stopCh chan struct{}
|
|
||||||
wg sync.WaitGroup
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type peerUpdate struct {
|
type peerUpdate struct {
|
||||||
@@ -69,14 +57,12 @@ type peerUpdate struct {
|
|||||||
// should be stopped by calling Stop when it is no longer needed.
|
// should be stopped by calling Stop when it is no longer needed.
|
||||||
func NewPeerHandler(db database.Store, logger *logger.Logger) *PeerHandler {
|
func NewPeerHandler(db database.Store, logger *logger.Logger) *PeerHandler {
|
||||||
h := &PeerHandler{
|
h := &PeerHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
peerBatch: make([]peerUpdate, 0, peerBatchSize),
|
peerBatch: make([]peerUpdate, 0, peerBatchSize),
|
||||||
batchFullCh: make(chan struct{}, 1),
|
lastFlush: time.Now(),
|
||||||
lastFlush: time.Now(),
|
stopCh: make(chan struct{}),
|
||||||
stopCh: make(chan struct{}),
|
|
||||||
}
|
}
|
||||||
h.batchTaken.L = &h.mu
|
|
||||||
|
|
||||||
// Start the flush timer goroutine
|
// Start the flush timer goroutine
|
||||||
h.wg.Add(1)
|
h.wg.Add(1)
|
||||||
@@ -102,8 +88,8 @@ func (h *PeerHandler) QueueCapacity() int {
|
|||||||
|
|
||||||
// HandleMessage processes a RIS message to track peer information.
|
// HandleMessage processes a RIS message to track peer information.
|
||||||
// It extracts the peer IP address and ASN from the message and adds
|
// It extracts the peer IP address and ASN from the message and adds
|
||||||
// the update to an internal batch. The flush goroutine writes the batch to the
|
// the update to an internal batch. When the batch reaches peerBatchSize
|
||||||
// database when it reaches peerBatchSize or the batch timeout expires.
|
// or the batch timeout expires, the batch is flushed to the database.
|
||||||
func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
|
func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
|
||||||
// Parse peer ASN from string
|
// Parse peer ASN from string
|
||||||
peerASN := 0
|
peerASN := 0
|
||||||
@@ -118,13 +104,6 @@ func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
defer h.mu.Unlock()
|
defer h.mu.Unlock()
|
||||||
|
|
||||||
// While the batch is at its maximum size, wait until the flush goroutine
|
|
||||||
// takes it. Meanwhile the handler takes nothing off its queue, which
|
|
||||||
// fills, and the streamer drops messages for it.
|
|
||||||
for len(h.peerBatch) >= peerBatchMaxSize {
|
|
||||||
h.batchTaken.Wait()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Add to batch
|
// Add to batch
|
||||||
h.peerBatch = append(h.peerBatch, peerUpdate{
|
h.peerBatch = append(h.peerBatch, peerUpdate{
|
||||||
peerIP: msg.Peer,
|
peerIP: msg.Peer,
|
||||||
@@ -133,14 +112,9 @@ func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
timestamp: msg.ParsedTimestamp,
|
timestamp: msg.ParsedTimestamp,
|
||||||
})
|
})
|
||||||
|
|
||||||
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
// Check if we need to flush
|
||||||
// waits in batchFullCh and the batch keeps growing until the write ends
|
|
||||||
// or the batch reaches peerBatchMaxSize.
|
|
||||||
if len(h.peerBatch) >= peerBatchSize {
|
if len(h.peerBatch) >= peerBatchSize {
|
||||||
select {
|
h.flushBatchLocked()
|
||||||
case h.batchFullCh <- struct{}{}:
|
|
||||||
default: // already woken
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -150,8 +124,7 @@ func (h *PeerHandler) Stop() {
|
|||||||
h.wg.Wait()
|
h.wg.Wait()
|
||||||
}
|
}
|
||||||
|
|
||||||
// flushLoop runs in a goroutine, the only one that writes the batch: it
|
// flushLoop runs in a goroutine and periodically flushes batches
|
||||||
// flushes when the batch is full, periodically, and once more on Stop.
|
|
||||||
func (h *PeerHandler) flushLoop() {
|
func (h *PeerHandler) flushLoop() {
|
||||||
defer h.wg.Done()
|
defer h.wg.Done()
|
||||||
|
|
||||||
@@ -161,48 +134,32 @@ func (h *PeerHandler) flushLoop() {
|
|||||||
for {
|
for {
|
||||||
select {
|
select {
|
||||||
case <-ticker.C:
|
case <-ticker.C:
|
||||||
|
h.mu.Lock()
|
||||||
if time.Since(h.lastFlush) >= peerBatchTimeout {
|
if time.Since(h.lastFlush) >= peerBatchTimeout {
|
||||||
h.flushBatch()
|
h.flushBatchLocked()
|
||||||
}
|
}
|
||||||
case <-h.batchFullCh:
|
h.mu.Unlock()
|
||||||
h.flushBatch()
|
|
||||||
case <-h.stopCh:
|
case <-h.stopCh:
|
||||||
// Final flush
|
// Final flush
|
||||||
h.flushBatch()
|
h.mu.Lock()
|
||||||
|
h.flushBatchLocked()
|
||||||
|
h.mu.Unlock()
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// takeBatch returns the pending updates and leaves an empty batch in their
|
// flushBatchLocked flushes the peer batch to the database (must be called
|
||||||
// place for HandleMessage to fill.
|
// with mutex held)
|
||||||
func (h *PeerHandler) takeBatch() []peerUpdate {
|
func (h *PeerHandler) flushBatchLocked() {
|
||||||
h.mu.Lock()
|
|
||||||
defer h.mu.Unlock()
|
|
||||||
|
|
||||||
if len(h.peerBatch) == 0 {
|
if len(h.peerBatch) == 0 {
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
batch := h.peerBatch
|
|
||||||
h.peerBatch = make([]peerUpdate, 0, peerBatchSize)
|
|
||||||
h.batchTaken.Broadcast()
|
|
||||||
|
|
||||||
return batch
|
|
||||||
}
|
|
||||||
|
|
||||||
// flushBatch takes the pending batch and writes it to the database, without
|
|
||||||
// holding the mutex during the write.
|
|
||||||
func (h *PeerHandler) flushBatch() {
|
|
||||||
batch := h.takeBatch()
|
|
||||||
if len(batch) == 0 {
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// Deduplicate by peer IP, keeping the latest update for each peer
|
// Deduplicate by peer IP, keeping the latest update for each peer
|
||||||
peerMap := make(map[string]peerUpdate)
|
peerMap := make(map[string]peerUpdate)
|
||||||
for _, update := range batch {
|
for _, update := range h.peerBatch {
|
||||||
existing, ok := peerMap[update.peerIP]
|
existing, ok := peerMap[update.peerIP]
|
||||||
if !ok || update.timestamp.After(existing.timestamp) {
|
if !ok || update.timestamp.After(existing.timestamp) {
|
||||||
peerMap[update.peerIP] = update
|
peerMap[update.peerIP] = update
|
||||||
@@ -227,5 +184,7 @@ func (h *PeerHandler) flushBatch() {
|
|||||||
"error", err, "count", len(dbPeerMap))
|
"error", err, "count", len(dbPeerMap))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Clear batch
|
||||||
|
h.peerBatch = h.peerBatch[:0]
|
||||||
h.lastFlush = time.Now()
|
h.lastFlush = time.Now()
|
||||||
}
|
}
|
||||||
@@ -24,12 +24,6 @@ const (
|
|||||||
// prefixBatchSize is the number of prefix updates to batch together
|
// prefixBatchSize is the number of prefix updates to batch together
|
||||||
prefixBatchSize = 25000
|
prefixBatchSize = 25000
|
||||||
|
|
||||||
// prefixBatchMaxSize is the most prefix updates the batch holds while the
|
|
||||||
// flush goroutine writes the previous one: ten batches, about 60 MiB
|
|
||||||
// (README "Memory"). At this size HandleMessage waits until the flush
|
|
||||||
// goroutine takes the batch.
|
|
||||||
prefixBatchMaxSize = 250000
|
|
||||||
|
|
||||||
// prefixBatchTimeout is the maximum time to wait before flushing a batch
|
// prefixBatchTimeout is the maximum time to wait before flushing a batch
|
||||||
// DO NOT reduce this timeout - larger batches are more efficient
|
// DO NOT reduce this timeout - larger batches are more efficient
|
||||||
prefixBatchTimeout = 1 * time.Second
|
prefixBatchTimeout = 1 * time.Second
|
||||||
@@ -46,18 +40,12 @@ type PrefixHandler struct {
|
|||||||
logger *logger.Logger
|
logger *logger.Logger
|
||||||
metrics *metrics.Tracker
|
metrics *metrics.Tracker
|
||||||
|
|
||||||
// Batching. mu guards batch only: the flush goroutine takes the batch
|
// Batching
|
||||||
// under mu and writes it after releasing mu, so HandleMessage waits for
|
mu sync.Mutex
|
||||||
// a database write only while batch is at prefixBatchMaxSize.
|
batch []prefixUpdate
|
||||||
mu sync.Mutex
|
lastFlush time.Time
|
||||||
batch []prefixUpdate
|
stopCh chan struct{}
|
||||||
// batchTaken is signalled when the flush goroutine takes batch; its L is
|
wg sync.WaitGroup
|
||||||
// &mu.
|
|
||||||
batchTaken sync.Cond
|
|
||||||
batchFullCh chan struct{} // wakes the flush goroutine when batch is full
|
|
||||||
lastFlush time.Time // used only by the flush goroutine
|
|
||||||
stopCh chan struct{}
|
|
||||||
wg sync.WaitGroup
|
|
||||||
}
|
}
|
||||||
|
|
||||||
type prefixUpdate struct {
|
type prefixUpdate struct {
|
||||||
@@ -72,14 +60,12 @@ type prefixUpdate struct {
|
|||||||
// NewPrefixHandler creates a new batched prefix tracking handler
|
// NewPrefixHandler creates a new batched prefix tracking handler
|
||||||
func NewPrefixHandler(db database.Store, logger *logger.Logger) *PrefixHandler {
|
func NewPrefixHandler(db database.Store, logger *logger.Logger) *PrefixHandler {
|
||||||
h := &PrefixHandler{
|
h := &PrefixHandler{
|
||||||
db: db,
|
db: db,
|
||||||
logger: logger,
|
logger: logger,
|
||||||
batch: make([]prefixUpdate, 0, prefixBatchSize),
|
batch: make([]prefixUpdate, 0, prefixBatchSize),
|
||||||
batchFullCh: make(chan struct{}, 1),
|
lastFlush: time.Now(),
|
||||||
lastFlush: time.Now(),
|
stopCh: make(chan struct{}),
|
||||||
stopCh: make(chan struct{}),
|
|
||||||
}
|
}
|
||||||
h.batchTaken.L = &h.mu
|
|
||||||
|
|
||||||
// Start the flush timer goroutine
|
// Start the flush timer goroutine
|
||||||
h.wg.Add(1)
|
h.wg.Add(1)
|
||||||
@@ -120,24 +106,17 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
h.mu.Lock()
|
h.mu.Lock()
|
||||||
defer h.mu.Unlock()
|
defer h.mu.Unlock()
|
||||||
|
|
||||||
// While the batch is at its maximum size, wait until the flush goroutine
|
|
||||||
// takes it. Meanwhile the handler takes nothing off its queue, which
|
|
||||||
// fills, and the streamer drops messages for it.
|
|
||||||
for len(h.batch) >= prefixBatchMaxSize {
|
|
||||||
h.batchTaken.Wait()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Withdrawals are queued before announcements, because the batch keeps
|
// Withdrawals are queued before announcements, because the batch keeps
|
||||||
// each peer's last update for a prefix: a message that lists a prefix as
|
// each peer's last update for a prefix: a message that lists a prefix as
|
||||||
// both withdrawn and announced keeps the route (RFC 4271 section 4.3).
|
// both withdrawn and announced keeps the route (RFC 4271 section 4.3).
|
||||||
// A withdrawal names only the prefix and peer: the message's AS path, and
|
|
||||||
// so its origin AS, belongs to the announced prefixes.
|
|
||||||
for _, prefix := range msg.Withdrawals {
|
for _, prefix := range msg.Withdrawals {
|
||||||
h.batch = append(h.batch, prefixUpdate{
|
h.batch = append(h.batch, prefixUpdate{
|
||||||
prefix: canonicalPrefix(prefix),
|
prefix: canonicalPrefix(prefix),
|
||||||
|
originASN: originASN, // Use the originASN from path if available
|
||||||
peer: msg.Peer,
|
peer: msg.Peer,
|
||||||
messageType: "withdrawal",
|
messageType: "withdrawal",
|
||||||
timestamp: timestamp,
|
timestamp: timestamp,
|
||||||
|
path: msg.Path,
|
||||||
})
|
})
|
||||||
// Record withdrawal in metrics
|
// Record withdrawal in metrics
|
||||||
if h.metrics != nil {
|
if h.metrics != nil {
|
||||||
@@ -162,14 +141,9 @@ func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// A full batch wakes the flush goroutine. If it is writing, the wake-up
|
// Check if we need to flush
|
||||||
// waits in batchFullCh and the batch keeps growing until the write ends
|
|
||||||
// or the batch reaches prefixBatchMaxSize.
|
|
||||||
if len(h.batch) >= prefixBatchSize {
|
if len(h.batch) >= prefixBatchSize {
|
||||||
select {
|
h.flushBatchLocked()
|
||||||
case h.batchFullCh <- struct{}{}:
|
|
||||||
default: // already woken
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -179,8 +153,7 @@ func (h *PrefixHandler) Stop() {
|
|||||||
h.wg.Wait()
|
h.wg.Wait()
|
||||||
}
|
}
|
||||||
|
|
||||||
// flushLoop runs in a goroutine, the only one that writes the batch: it
|
// flushLoop runs in a goroutine and periodically flushes batches
|
||||||
// flushes when the batch is full, periodically, and once more on Stop.
|
|
||||||
func (h *PrefixHandler) flushLoop() {
|
func (h *PrefixHandler) flushLoop() {
|
||||||
defer h.wg.Done()
|
defer h.wg.Done()
|
||||||
|
|
||||||
@@ -190,47 +163,31 @@ func (h *PrefixHandler) flushLoop() {
|
|||||||
for {
|
for {
|
||||||
select {
|
select {
|
||||||
case <-ticker.C:
|
case <-ticker.C:
|
||||||
|
h.mu.Lock()
|
||||||
if time.Since(h.lastFlush) >= prefixBatchTimeout {
|
if time.Since(h.lastFlush) >= prefixBatchTimeout {
|
||||||
h.flushBatch()
|
h.flushBatchLocked()
|
||||||
}
|
}
|
||||||
case <-h.batchFullCh:
|
h.mu.Unlock()
|
||||||
h.flushBatch()
|
|
||||||
case <-h.stopCh:
|
case <-h.stopCh:
|
||||||
// Final flush
|
// Final flush
|
||||||
h.flushBatch()
|
h.mu.Lock()
|
||||||
|
h.flushBatchLocked()
|
||||||
|
h.mu.Unlock()
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// takeBatch returns the pending updates and leaves an empty batch in their
|
// flushBatchLocked flushes the prefix batch to the database (must be called
|
||||||
// place for HandleMessage to fill.
|
// with mutex held)
|
||||||
func (h *PrefixHandler) takeBatch() []prefixUpdate {
|
func (h *PrefixHandler) flushBatchLocked() {
|
||||||
h.mu.Lock()
|
|
||||||
defer h.mu.Unlock()
|
|
||||||
|
|
||||||
if len(h.batch) == 0 {
|
if len(h.batch) == 0 {
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
batch := h.batch
|
|
||||||
h.batch = make([]prefixUpdate, 0, prefixBatchSize)
|
|
||||||
h.batchTaken.Broadcast()
|
|
||||||
|
|
||||||
return batch
|
|
||||||
}
|
|
||||||
|
|
||||||
// flushBatch takes the pending batch and writes it to the database, without
|
|
||||||
// holding the mutex during the write.
|
|
||||||
func (h *PrefixHandler) flushBatch() {
|
|
||||||
batch := h.takeBatch()
|
|
||||||
if len(batch) == 0 {
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
startTime := time.Now()
|
startTime := time.Now()
|
||||||
batchSize := len(batch)
|
batchSize := len(h.batch)
|
||||||
|
|
||||||
// For each prefix and peer, keep only the last update the feed delivered.
|
// For each prefix and peer, keep only the last update the feed delivered.
|
||||||
// Updates from different peers for the same prefix are all kept. Arrival
|
// Updates from different peers for the same prefix are all kept. Arrival
|
||||||
@@ -240,7 +197,7 @@ func (h *PrefixHandler) flushBatch() {
|
|||||||
|
|
||||||
lastUpdates := make(map[prefixAndPeer]prefixUpdate)
|
lastUpdates := make(map[prefixAndPeer]prefixUpdate)
|
||||||
|
|
||||||
for _, update := range batch {
|
for _, update := range h.batch {
|
||||||
lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update
|
lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -279,6 +236,7 @@ func (h *PrefixHandler) flushBatch() {
|
|||||||
// Create deletion record for batch delete
|
// Create deletion record for batch delete
|
||||||
routesToDelete = append(routesToDelete, database.LiveRouteDeletion{
|
routesToDelete = append(routesToDelete, database.LiveRouteDeletion{
|
||||||
Prefix: update.prefix,
|
Prefix: update.prefix,
|
||||||
|
OriginASN: update.originASN,
|
||||||
PeerIP: update.peer,
|
PeerIP: update.peer,
|
||||||
IPVersion: ipVersion,
|
IPVersion: ipVersion,
|
||||||
})
|
})
|
||||||
@@ -295,6 +253,8 @@ func (h *PrefixHandler) flushBatch() {
|
|||||||
"duration_ms", elapsed.Milliseconds(),
|
"duration_ms", elapsed.Milliseconds(),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Clear batch
|
||||||
|
h.batch = h.batch[:0]
|
||||||
h.lastFlush = time.Now()
|
h.lastFlush = time.Now()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package routewatch
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
|
||||||
"slices"
|
"slices"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -14,12 +13,9 @@ import (
|
|||||||
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
"git.eeqj.de/sneak/routewatch/internal/ristypes"
|
||||||
)
|
)
|
||||||
|
|
||||||
const testPeerIP = "2001:db8:ffff::1"
|
|
||||||
|
|
||||||
// An IPv4 and an IPv6 prefix the tests announce.
|
|
||||||
const (
|
const (
|
||||||
|
testPeerIP = "2001:db8:ffff::1"
|
||||||
testV4Prefix = "192.0.2.0/24"
|
testV4Prefix = "192.0.2.0/24"
|
||||||
testV6Prefix = "2001:db8:1::/48"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and
|
// TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and
|
||||||
@@ -31,10 +27,20 @@ const (
|
|||||||
func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
|
func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create database: %v", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = db.Close() }()
|
||||||
|
|
||||||
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
|
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
|
||||||
h := &PrefixHandler{db: db, logger: logger.New()}
|
h := &PrefixHandler{db: db, logger: logger.New()}
|
||||||
|
flush := func() {
|
||||||
|
h.mu.Lock()
|
||||||
|
defer h.mu.Unlock()
|
||||||
|
|
||||||
|
h.flushBatchLocked()
|
||||||
|
}
|
||||||
|
|
||||||
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||||
h.HandleMessage(&ristypes.RISMessage{
|
h.HandleMessage(&ristypes.RISMessage{
|
||||||
@@ -43,13 +49,13 @@ func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
|
|||||||
ParsedTimestamp: ts,
|
ParsedTimestamp: ts,
|
||||||
Announcements: []ristypes.RISAnnouncement{{
|
Announcements: []ristypes.RISAnnouncement{{
|
||||||
NextHop: testPeerIP,
|
NextHop: testPeerIP,
|
||||||
Prefixes: []string{testV6Prefix, testV4Prefix},
|
Prefixes: []string{"2001:db8:1::/48", testV4Prefix},
|
||||||
}},
|
}},
|
||||||
})
|
})
|
||||||
h.flushBatch()
|
flush()
|
||||||
|
|
||||||
for ip, want := range map[string]string{
|
for ip, want := range map[string]string{
|
||||||
"2001:db8:1::1": testV6Prefix,
|
"2001:db8:1::1": "2001:db8:1::/48",
|
||||||
"192.0.2.1": testV4Prefix,
|
"192.0.2.1": testV4Prefix,
|
||||||
} {
|
} {
|
||||||
info, err := db.GetIPInfo(ip)
|
info, err := db.GetIPInfo(ip)
|
||||||
@@ -68,7 +74,7 @@ func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
|
|||||||
ParsedTimestamp: ts.Add(time.Minute),
|
ParsedTimestamp: ts.Add(time.Minute),
|
||||||
Withdrawals: []string{"2001:db8:1:0:0:0:0:0/48", testV4Prefix},
|
Withdrawals: []string{"2001:db8:1:0:0:0:0:0/48", testV4Prefix},
|
||||||
})
|
})
|
||||||
h.flushBatch()
|
flush()
|
||||||
|
|
||||||
for _, ip := range []string{"2001:db8:1::1", "192.0.2.1"} {
|
for _, ip := range []string{"2001:db8:1::1", "192.0.2.1"} {
|
||||||
info, err := db.GetIPInfo(ip)
|
info, err := db.GetIPInfo(ip)
|
||||||
@@ -106,8 +112,9 @@ func TestPrefixHandlerAppliesEachPeersLastUpdateInABatch(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
withdraw := func(peer string) *ristypes.RISMessage {
|
withdraw := func(peer string) *ristypes.RISMessage {
|
||||||
return &ristypes.RISMessage{Peer: peer, ParsedTimestamp: ts,
|
return &ristypes.RISMessage{
|
||||||
Withdrawals: []string{prefix}}
|
Peer: peer, ParsedTimestamp: ts, Withdrawals: []string{prefix},
|
||||||
|
}
|
||||||
}
|
}
|
||||||
announceAndWithdraw := func(peer string) *ristypes.RISMessage {
|
announceAndWithdraw := func(peer string) *ristypes.RISMessage {
|
||||||
msg := announce(peer)
|
msg := announce(peer)
|
||||||
@@ -124,10 +131,8 @@ func TestPrefixHandlerAppliesEachPeersLastUpdateInABatch(t *testing.T) {
|
|||||||
wantPeers []string
|
wantPeers []string
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "two peers announce the prefix",
|
name: "two peers announce the prefix",
|
||||||
batches: [][]*ristypes.RISMessage{
|
batches: [][]*ristypes.RISMessage{{announce(testPeerIP), announce(otherPeerIP)}},
|
||||||
{announce(testPeerIP), announce(otherPeerIP)},
|
|
||||||
},
|
|
||||||
wantPeers: []string{testPeerIP, otherPeerIP},
|
wantPeers: []string{testPeerIP, otherPeerIP},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -166,106 +171,17 @@ func TestPrefixHandlerAppliesEachPeersLastUpdateInABatch(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
db := handleBatches(t, tt.batches)
|
db := handleBatches(t, tt.batches)
|
||||||
checkLivePeers(t, db, prefix, prefixMaskLength, tt.wantPeers)
|
checkPrefixRoutes(t, db, prefix, prefixMaskLength, tt.wantPeers)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestPrefixHandlerKeepsOneRoutePerPrefixAndPeer runs updates from one peer
|
// handleBatches runs each batch of messages through a prefix handler on a new
|
||||||
// through the prefix handler into a real database, flushing each message as its
|
// database, flushing after each batch, and returns the database. The database
|
||||||
// own batch, once with IPv4 prefixes and once with IPv6 prefixes. A peer has at
|
// is closed when the test ends.
|
||||||
// most one route for a prefix: a later announcement replaces it whatever its
|
func handleBatches(
|
||||||
// origin AS, and a withdrawal removes it whatever AS the message's AS path ends
|
t *testing.T, batches [][]*ristypes.RISMessage,
|
||||||
// in. The in-memory route count and prefix distribution must match the route
|
) *database.Database {
|
||||||
// table.
|
|
||||||
func TestPrefixHandlerKeepsOneRoutePerPrefixAndPeer(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
families := []struct {
|
|
||||||
ipVersion int
|
|
||||||
prefix, otherPrefix string
|
|
||||||
// The mask length of both prefixes.
|
|
||||||
maskLength int
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
ipVersion: ipv4Version, maskLength: 24,
|
|
||||||
prefix: "198.51.100.0/24", otherPrefix: "203.0.113.0/24",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
ipVersion: ipv6Version, maskLength: 48,
|
|
||||||
prefix: testV6Prefix, otherPrefix: "2001:db8:2::/48",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
announce := func(path ristypes.ASPath, announced string) *ristypes.RISMessage {
|
|
||||||
return &ristypes.RISMessage{
|
|
||||||
Peer: testPeerIP,
|
|
||||||
Path: path,
|
|
||||||
ParsedTimestamp: ts,
|
|
||||||
Announcements: []ristypes.RISAnnouncement{
|
|
||||||
{NextHop: testPeerIP, Prefixes: []string{announced}},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, family := range families {
|
|
||||||
prefix, otherPrefix := family.prefix, family.otherPrefix
|
|
||||||
|
|
||||||
// One message that announces otherPrefix with origin AS testASNB and
|
|
||||||
// withdraws prefix.
|
|
||||||
announceOtherAndWithdraw := announce(
|
|
||||||
ristypes.ASPath{testASNA, testASNB}, otherPrefix)
|
|
||||||
announceOtherAndWithdraw.Withdrawals = []string{prefix}
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
// Each message is handled and then flushed as its own batch.
|
|
||||||
batches [][]*ristypes.RISMessage
|
|
||||||
// The origin AS of the peer's live route for each prefix. A prefix
|
|
||||||
// left out must have no live route.
|
|
||||||
wantOrigins map[string]int
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
name: "the peer announces the prefix with a new origin AS",
|
|
||||||
batches: [][]*ristypes.RISMessage{
|
|
||||||
{announce(ristypes.ASPath{testASNA}, prefix)},
|
|
||||||
{announce(ristypes.ASPath{testASNA, testASNB}, prefix)},
|
|
||||||
},
|
|
||||||
wantOrigins: map[string]int{prefix: testASNB},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "one message announces a prefix with another origin AS " +
|
|
||||||
"and withdraws the prefix",
|
|
||||||
batches: [][]*ristypes.RISMessage{
|
|
||||||
{announce(ristypes.ASPath{testASNA}, prefix)},
|
|
||||||
{announceOtherAndWithdraw},
|
|
||||||
},
|
|
||||||
wantOrigins: map[string]int{otherPrefix: testASNB},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
name := fmt.Sprintf("IPv%d %s", family.ipVersion, tt.name)
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := handleBatches(t, tt.batches)
|
|
||||||
checkLiveOrigins(t, db, tt.wantOrigins, prefix, otherPrefix)
|
|
||||||
|
|
||||||
wantDistribution := []database.PrefixDistribution{
|
|
||||||
{MaskLength: family.maskLength, Count: len(tt.wantOrigins)},
|
|
||||||
}
|
|
||||||
assertCounts(t, db, family.ipVersion, len(tt.wantOrigins),
|
|
||||||
wantDistribution)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// newTestDatabase opens a database in a new temporary directory. It is closed
|
|
||||||
// when the test ends.
|
|
||||||
func newTestDatabase(t *testing.T) *database.Database {
|
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
|
||||||
@@ -275,18 +191,6 @@ func newTestDatabase(t *testing.T) *database.Database {
|
|||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
|
||||||
return db
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleBatches runs each batch of messages through a prefix handler into a
|
|
||||||
// new database, flushing after each batch, and returns the database.
|
|
||||||
func handleBatches(
|
|
||||||
t *testing.T, batches [][]*ristypes.RISMessage,
|
|
||||||
) *database.Database {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
|
|
||||||
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
|
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
|
||||||
h := &PrefixHandler{db: db, logger: logger.New()}
|
h := &PrefixHandler{db: db, logger: logger.New()}
|
||||||
|
|
||||||
@@ -295,24 +199,28 @@ func handleBatches(
|
|||||||
h.HandleMessage(msg)
|
h.HandleMessage(msg)
|
||||||
}
|
}
|
||||||
|
|
||||||
h.flushBatch()
|
h.mu.Lock()
|
||||||
|
h.flushBatchLocked()
|
||||||
|
h.mu.Unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
return db
|
return db
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkLivePeers checks that exactly wantPeers have a live route for the IPv4
|
// checkPrefixRoutes checks that exactly wantPeers have a live route for the
|
||||||
// prefix, and that the in-memory route count and prefix distribution match
|
// IPv4 prefix, and that the in-memory route count and prefix distribution
|
||||||
// the route table.
|
// match the route table.
|
||||||
func checkLivePeers(
|
func checkPrefixRoutes(
|
||||||
t *testing.T, db *database.Database, prefix string, maskLength int,
|
t *testing.T, db *database.Database, prefix string, maskLength int,
|
||||||
wantPeers []string,
|
wantPeers []string,
|
||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
routes, _, err := db.GetPrefixDetailsContext(context.Background(), prefix)
|
ctx := context.Background()
|
||||||
|
|
||||||
|
routes, _, err := db.GetPrefixDetailsContext(ctx, prefix)
|
||||||
if err != nil && !errors.Is(err, database.ErrNoRoute) {
|
if err != nil && !errors.Is(err, database.ErrNoRoute) {
|
||||||
t.Fatalf("GetPrefixDetailsContext(%s): %v", prefix, err)
|
t.Fatalf("GetPrefixDetailsContext: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
gotPeers := make([]string, 0, len(routes))
|
gotPeers := make([]string, 0, len(routes))
|
||||||
@@ -325,6 +233,21 @@ func checkLivePeers(
|
|||||||
prefix, gotPeers, wantPeers)
|
prefix, gotPeers, wantPeers)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
stats, err := db.GetStatsContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetStatsContext: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
tableRoutesV4, _, err := db.GetLiveRouteCountsContext(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GetLiveRouteCountsContext: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if stats.IPv4Routes != len(wantPeers) || tableRoutesV4 != len(wantPeers) {
|
||||||
|
t.Errorf("IPv4 live routes: %d in memory, %d in the table, want %d",
|
||||||
|
stats.IPv4Routes, tableRoutesV4, len(wantPeers))
|
||||||
|
}
|
||||||
|
|
||||||
var wantDistribution []database.PrefixDistribution
|
var wantDistribution []database.PrefixDistribution
|
||||||
if len(wantPeers) > 0 {
|
if len(wantPeers) > 0 {
|
||||||
wantDistribution = []database.PrefixDistribution{
|
wantDistribution = []database.PrefixDistribution{
|
||||||
@@ -332,86 +255,14 @@ func checkLivePeers(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
assertCounts(t, db, ipv4Version, len(wantPeers), wantDistribution)
|
queryDistributionV4, _, err := db.GetPrefixDistributionContext(ctx)
|
||||||
}
|
|
||||||
|
|
||||||
// checkLiveOrigins checks that each of prefixes has one live route, with the
|
|
||||||
// origin AS wantOrigins gives for it, or none when wantOrigins leaves it out.
|
|
||||||
func checkLiveOrigins(
|
|
||||||
t *testing.T, db *database.Database, wantOrigins map[string]int,
|
|
||||||
prefixes ...string,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for _, prefix := range prefixes {
|
|
||||||
routes, _, err := db.GetPrefixDetailsContext(context.Background(), prefix)
|
|
||||||
if err != nil && !errors.Is(err, database.ErrNoRoute) {
|
|
||||||
t.Fatalf("GetPrefixDetailsContext(%s): %v", prefix, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
gotOrigins := make([]int, 0, len(routes))
|
|
||||||
for _, route := range routes {
|
|
||||||
gotOrigins = append(gotOrigins, route.OriginASN)
|
|
||||||
}
|
|
||||||
|
|
||||||
var want []int
|
|
||||||
if origin, ok := wantOrigins[prefix]; ok {
|
|
||||||
want = []int{origin}
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(gotOrigins, want) {
|
|
||||||
t.Errorf("origin ASes of the live routes for %s = %v, want %v",
|
|
||||||
prefix, gotOrigins, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// assertCounts checks that the live route count and prefix distribution of one
|
|
||||||
// IP version are wantRoutes and wantDistribution, both as held in memory and as
|
|
||||||
// read from the route table.
|
|
||||||
func assertCounts(
|
|
||||||
t *testing.T, db *database.Database, ipVersion, wantRoutes int,
|
|
||||||
wantDistribution []database.PrefixDistribution,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
|
|
||||||
stats, err := db.GetStatsContext(ctx)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetStatsContext: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
tableRoutesV4, tableRoutesV6, err := db.GetLiveRouteCountsContext(ctx)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GetLiveRouteCountsContext: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
queryDistributionV4, queryDistributionV6, err :=
|
|
||||||
db.GetPrefixDistributionContext(ctx)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("GetPrefixDistributionContext: %v", err)
|
t.Fatalf("GetPrefixDistributionContext: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
memoryRoutes, tableRoutes := stats.IPv4Routes, tableRoutesV4
|
if !slices.Equal(stats.IPv4PrefixDistribution, wantDistribution) ||
|
||||||
memoryDistribution := stats.IPv4PrefixDistribution
|
!slices.Equal(queryDistributionV4, wantDistribution) {
|
||||||
queryDistribution := queryDistributionV4
|
t.Errorf("IPv4 prefix distribution: %v in memory, %v from the table, want %v",
|
||||||
|
stats.IPv4PrefixDistribution, queryDistributionV4, wantDistribution)
|
||||||
if ipVersion == ipv6Version {
|
|
||||||
memoryRoutes, tableRoutes = stats.IPv6Routes, tableRoutesV6
|
|
||||||
memoryDistribution = stats.IPv6PrefixDistribution
|
|
||||||
queryDistribution = queryDistributionV6
|
|
||||||
}
|
|
||||||
|
|
||||||
if memoryRoutes != wantRoutes || tableRoutes != wantRoutes {
|
|
||||||
t.Errorf("IPv%d live routes: %d in memory, %d in the table, want %d",
|
|
||||||
ipVersion, memoryRoutes, tableRoutes, wantRoutes)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !slices.Equal(memoryDistribution, wantDistribution) ||
|
|
||||||
!slices.Equal(queryDistribution, wantDistribution) {
|
|
||||||
t.Errorf("IPv%d prefix distribution: %v in memory, %v from the table, "+
|
|
||||||
"want %v", ipVersion, memoryDistribution, queryDistribution,
|
|
||||||
wantDistribution)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
+111
-209
@@ -15,7 +15,6 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/database"
|
"git.eeqj.de/sneak/routewatch/internal/database"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/metrics"
|
|
||||||
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
"git.eeqj.de/sneak/routewatch/internal/streamer"
|
||||||
"git.eeqj.de/sneak/routewatch/internal/templates"
|
"git.eeqj.de/sneak/routewatch/internal/templates"
|
||||||
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
|
asinfo "git.eeqj.de/sneak/routewatch/pkg/asinfo"
|
||||||
@@ -242,9 +241,9 @@ func (s *Server) handleStatusJSON() http.HandlerFunc {
|
|||||||
metrics := s.streamer.GetMetrics()
|
metrics := s.streamer.GetMetrics()
|
||||||
|
|
||||||
dbStats, err := s.databaseStats(ctx)
|
dbStats, err := s.databaseStats(ctx)
|
||||||
if deadlinePassed(ctx) {
|
if err != nil && ctx.Err() != nil {
|
||||||
s.logger.Error("Database stats timeout in status.json")
|
s.logger.Error("Database stats timeout in status.json")
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
s.writeJSONError(w, http.StatusRequestTimeout, "Database timeout")
|
||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -266,15 +265,7 @@ func (s *Server) handleStatusJSON() http.HandlerFunc {
|
|||||||
runtime.ReadMemStats(&memStats)
|
runtime.ReadMemStats(&memStats)
|
||||||
|
|
||||||
// Get WHOIS stats if fetcher is available
|
// Get WHOIS stats if fetcher is available
|
||||||
var whoisStats *WHOISStatsInfo
|
whoisStats := s.getWHOISStats(ctx)
|
||||||
if s.asnFetcher != nil {
|
|
||||||
whoisStats = s.getWHOISStats(ctx)
|
|
||||||
if deadlinePassed(ctx) {
|
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stats := Stats{
|
stats := Stats{
|
||||||
Uptime: streamUptime(metrics.ConnectedSince),
|
Uptime: streamUptime(metrics.ConnectedSince),
|
||||||
@@ -314,8 +305,13 @@ func (s *Server) handleStatusJSON() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// getWHOISStats builds WHOIS statistics from database and fetcher.
|
// getWHOISStats builds WHOIS statistics from database and fetcher, or returns
|
||||||
|
// nil when there is no fetcher.
|
||||||
func (s *Server) getWHOISStats(ctx context.Context) *WHOISStatsInfo {
|
func (s *Server) getWHOISStats(ctx context.Context) *WHOISStatsInfo {
|
||||||
|
if s.asnFetcher == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// Get database WHOIS stats
|
// Get database WHOIS stats
|
||||||
dbStats, err := s.db.GetWHOISStats(ctx, whoisStaleThreshold)
|
dbStats, err := s.db.GetWHOISStats(ctx, whoisStaleThreshold)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -523,7 +519,7 @@ func (s *Server) handleStats() http.HandlerFunc {
|
|||||||
// Check if context is already cancelled
|
// Check if context is already cancelled
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
http.Error(w, "Request timeout", http.StatusRequestTimeout)
|
||||||
|
|
||||||
return
|
return
|
||||||
default:
|
default:
|
||||||
@@ -532,10 +528,10 @@ func (s *Server) handleStats() http.HandlerFunc {
|
|||||||
metrics := s.streamer.GetMetrics()
|
metrics := s.streamer.GetMetrics()
|
||||||
|
|
||||||
dbStats, err := s.databaseStats(ctx)
|
dbStats, err := s.databaseStats(ctx)
|
||||||
if deadlinePassed(ctx) {
|
if err != nil && ctx.Err() != nil {
|
||||||
s.logger.Error("Database stats timeout")
|
s.logger.Error("Database stats timeout")
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
|
||||||
|
|
||||||
|
// Don't write response here - timeout middleware already handles it
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -546,79 +542,62 @@ func (s *Server) handleStats() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// Get WHOIS stats if fetcher is available
|
const bitsPerMegabit = 1000000.0
|
||||||
var whoisStats *WHOISStatsInfo
|
|
||||||
if s.asnFetcher != nil {
|
|
||||||
whoisStats = s.getWHOISStats(ctx)
|
|
||||||
if deadlinePassed(ctx) {
|
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
|
||||||
|
|
||||||
return
|
// Get route update metrics
|
||||||
}
|
routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics()
|
||||||
|
|
||||||
|
// Get memory stats
|
||||||
|
var memStats runtime.MemStats
|
||||||
|
runtime.ReadMemStats(&memStats)
|
||||||
|
|
||||||
|
// Get WHOIS stats if fetcher is available
|
||||||
|
whoisStats := s.getWHOISStats(ctx)
|
||||||
|
|
||||||
|
stats := StatsResponse{
|
||||||
|
Uptime: streamUptime(metrics.ConnectedSince),
|
||||||
|
TotalMessages: metrics.TotalMessages,
|
||||||
|
TotalBytes: metrics.TotalBytes,
|
||||||
|
TotalWireBytes: metrics.TotalWireBytes,
|
||||||
|
MessagesPerSec: metrics.MessagesPerSec,
|
||||||
|
MbitsPerSec: metrics.BitsPerSec / bitsPerMegabit,
|
||||||
|
WireMbitsPerSec: metrics.WireBitsPerSec / bitsPerMegabit,
|
||||||
|
Connected: metrics.Connected,
|
||||||
|
ConnectionDuration: connectionDuration(
|
||||||
|
metrics.Connected, metrics.ConnectedSince),
|
||||||
|
ReconnectCount: metrics.ReconnectCount,
|
||||||
|
GoVersion: runtime.Version(),
|
||||||
|
Goroutines: runtime.NumGoroutine(),
|
||||||
|
MemoryUsage: humanize.Bytes(memStats.Alloc),
|
||||||
|
GC: gcStats(&memStats),
|
||||||
|
Stream: s.streamStats(),
|
||||||
|
ASNs: dbStats.ASNs,
|
||||||
|
Prefixes: dbStats.Prefixes,
|
||||||
|
IPv4Prefixes: dbStats.IPv4Prefixes,
|
||||||
|
IPv6Prefixes: dbStats.IPv6Prefixes,
|
||||||
|
Peerings: dbStats.Peerings,
|
||||||
|
Peers: dbStats.Peers,
|
||||||
|
DatabaseSizeBytes: dbStats.FileSizeBytes,
|
||||||
|
LiveRoutes: dbStats.LiveRoutes,
|
||||||
|
IPv4Routes: dbStats.IPv4Routes,
|
||||||
|
IPv6Routes: dbStats.IPv6Routes,
|
||||||
|
OldestRoute: dbStats.OldestRoute,
|
||||||
|
NewestRoute: dbStats.NewestRoute,
|
||||||
|
IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec,
|
||||||
|
IPv6UpdatesPerSec: routeMetrics.IPv6UpdatesPerSec,
|
||||||
|
HandlerStats: handlerStatsInfo(s.streamer.GetHandlerStats()),
|
||||||
|
IPv4PrefixDistribution: dbStats.IPv4PrefixDistribution,
|
||||||
|
IPv6PrefixDistribution: dbStats.IPv6PrefixDistribution,
|
||||||
|
WHOISStats: whoisStats,
|
||||||
}
|
}
|
||||||
|
|
||||||
err = writeJSONSuccess(w, s.statsResponse(metrics, dbStats, whoisStats))
|
err = writeJSONSuccess(w, stats)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Error("Failed to encode stats", "error", err)
|
s.logger.Error("Failed to encode stats", "error", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// statsResponse builds the /api/v1/stats answer from the stream metrics, the
|
|
||||||
// database stats and the WHOIS stats, with the route update rates, memory use
|
|
||||||
// and handler queues as they are now.
|
|
||||||
func (s *Server) statsResponse(
|
|
||||||
streamMetrics metrics.StreamMetrics,
|
|
||||||
dbStats database.Stats,
|
|
||||||
whoisStats *WHOISStatsInfo,
|
|
||||||
) StatsResponse {
|
|
||||||
const bitsPerMegabit = 1000000.0
|
|
||||||
|
|
||||||
// Get route update metrics
|
|
||||||
routeMetrics := s.streamer.GetMetricsTracker().GetRouteMetrics()
|
|
||||||
|
|
||||||
// Get memory stats
|
|
||||||
var memStats runtime.MemStats
|
|
||||||
runtime.ReadMemStats(&memStats)
|
|
||||||
|
|
||||||
return StatsResponse{
|
|
||||||
Uptime: streamUptime(streamMetrics.ConnectedSince),
|
|
||||||
TotalMessages: streamMetrics.TotalMessages,
|
|
||||||
TotalBytes: streamMetrics.TotalBytes,
|
|
||||||
TotalWireBytes: streamMetrics.TotalWireBytes,
|
|
||||||
MessagesPerSec: streamMetrics.MessagesPerSec,
|
|
||||||
MbitsPerSec: streamMetrics.BitsPerSec / bitsPerMegabit,
|
|
||||||
WireMbitsPerSec: streamMetrics.WireBitsPerSec / bitsPerMegabit,
|
|
||||||
Connected: streamMetrics.Connected,
|
|
||||||
ConnectionDuration: connectionDuration(
|
|
||||||
streamMetrics.Connected, streamMetrics.ConnectedSince),
|
|
||||||
ReconnectCount: streamMetrics.ReconnectCount,
|
|
||||||
GoVersion: runtime.Version(),
|
|
||||||
Goroutines: runtime.NumGoroutine(),
|
|
||||||
MemoryUsage: humanize.Bytes(memStats.Alloc),
|
|
||||||
GC: gcStats(&memStats),
|
|
||||||
Stream: s.streamStats(),
|
|
||||||
ASNs: dbStats.ASNs,
|
|
||||||
Prefixes: dbStats.Prefixes,
|
|
||||||
IPv4Prefixes: dbStats.IPv4Prefixes,
|
|
||||||
IPv6Prefixes: dbStats.IPv6Prefixes,
|
|
||||||
Peerings: dbStats.Peerings,
|
|
||||||
Peers: dbStats.Peers,
|
|
||||||
DatabaseSizeBytes: dbStats.FileSizeBytes,
|
|
||||||
LiveRoutes: dbStats.LiveRoutes,
|
|
||||||
IPv4Routes: dbStats.IPv4Routes,
|
|
||||||
IPv6Routes: dbStats.IPv6Routes,
|
|
||||||
OldestRoute: dbStats.OldestRoute,
|
|
||||||
NewestRoute: dbStats.NewestRoute,
|
|
||||||
IPv4UpdatesPerSec: routeMetrics.IPv4UpdatesPerSec,
|
|
||||||
IPv6UpdatesPerSec: routeMetrics.IPv6UpdatesPerSec,
|
|
||||||
HandlerStats: handlerStatsInfo(s.streamer.GetHandlerStats()),
|
|
||||||
IPv4PrefixDistribution: dbStats.IPv4PrefixDistribution,
|
|
||||||
IPv6PrefixDistribution: dbStats.IPv6PrefixDistribution,
|
|
||||||
WHOISStats: whoisStats,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// handleStatusHTML returns a handler that serves the HTML status page,
|
// handleStatusHTML returns a handler that serves the HTML status page,
|
||||||
// which displays real-time statistics fetched via JavaScript.
|
// which displays real-time statistics fetched via JavaScript.
|
||||||
func (s *Server) handleStatusHTML() http.HandlerFunc {
|
func (s *Server) handleStatusHTML() http.HandlerFunc {
|
||||||
@@ -653,7 +632,17 @@ type IPLookupResponse struct {
|
|||||||
// Always returns the same response structure with PTR records for each IP.
|
// Always returns the same response structure with PTR records for each IP.
|
||||||
func (s *Server) handleIPInfo() http.HandlerFunc {
|
func (s *Server) handleIPInfo() http.HandlerFunc {
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
target := lookupTarget(r)
|
// Get IP/hostname from URL param, falling back to client IP
|
||||||
|
target := chi.URLParam(r, "ip")
|
||||||
|
if target == "" {
|
||||||
|
target = chi.URLParam(r, "addr")
|
||||||
|
}
|
||||||
|
|
||||||
|
if target == "" {
|
||||||
|
// Use client IP (RealIP middleware has already processed this)
|
||||||
|
target = extractClientIP(r)
|
||||||
|
}
|
||||||
|
|
||||||
if target == "" {
|
if target == "" {
|
||||||
s.writeJSONError(w, http.StatusBadRequest, "Could not determine IP address")
|
s.writeJSONError(w, http.StatusBadRequest, "Could not determine IP address")
|
||||||
|
|
||||||
@@ -668,12 +657,6 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
|
|
||||||
// Collect all IPs to look up
|
// Collect all IPs to look up
|
||||||
ipsToLookup, resolveErrors := resolveLookupTargets(ctx, target)
|
ipsToLookup, resolveErrors := resolveLookupTargets(ctx, target)
|
||||||
if deadlinePassed(ctx) {
|
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
response.Errors = append(response.Errors, resolveErrors...)
|
response.Errors = append(response.Errors, resolveErrors...)
|
||||||
|
|
||||||
if len(ipsToLookup) == 0 {
|
if len(ipsToLookup) == 0 {
|
||||||
@@ -688,12 +671,6 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
// Look up each IP
|
// Look up each IP
|
||||||
for _, ip := range ipsToLookup {
|
for _, ip := range ipsToLookup {
|
||||||
ipInfo, err := s.db.GetIPInfoContext(ctx, ip)
|
ipInfo, err := s.db.GetIPInfoContext(ctx, ip)
|
||||||
if deadlinePassed(ctx) {
|
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
response.Errors = append(response.Errors, ip+": "+err.Error())
|
response.Errors = append(response.Errors, ip+": "+err.Error())
|
||||||
|
|
||||||
@@ -702,12 +679,6 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
|
|
||||||
// Do PTR lookup for this IP
|
// Do PTR lookup for this IP
|
||||||
ptrs := lookupPTRNames(ctx, ip)
|
ptrs := lookupPTRNames(ctx, ip)
|
||||||
if deadlinePassed(ctx) {
|
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(ptrs) > 0 {
|
if len(ptrs) > 0 {
|
||||||
ipInfo.PTR = ptrs
|
ipInfo.PTR = ptrs
|
||||||
}
|
}
|
||||||
@@ -720,7 +691,11 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Queue WHOIS refresh for stale ASNs (non-blocking)
|
// Queue WHOIS refresh for stale ASNs (non-blocking)
|
||||||
s.queueWHOISRefresh(refreshASNs)
|
if s.asnFetcher != nil {
|
||||||
|
for asn := range refreshASNs {
|
||||||
|
s.asnFetcher.QueueImmediate(asn)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Return response (even if no results, include errors)
|
// Return response (even if no results, include errors)
|
||||||
if len(response.Results) == 0 && len(response.Errors) > 0 {
|
if len(response.Results) == 0 && len(response.Errors) > 0 {
|
||||||
@@ -736,38 +711,9 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// lookupTarget returns what an IP lookup request asks for: the ip or addr URL
|
|
||||||
// parameter, or else the client's own address.
|
|
||||||
func lookupTarget(r *http.Request) string {
|
|
||||||
target := chi.URLParam(r, "ip")
|
|
||||||
if target == "" {
|
|
||||||
target = chi.URLParam(r, "addr")
|
|
||||||
}
|
|
||||||
|
|
||||||
if target == "" {
|
|
||||||
// Use client IP (RealIP middleware has already processed this)
|
|
||||||
target = extractClientIP(r)
|
|
||||||
}
|
|
||||||
|
|
||||||
return target
|
|
||||||
}
|
|
||||||
|
|
||||||
// queueWHOISRefresh asks the WHOIS fetcher, when there is one, to refresh
|
|
||||||
// each of asns now.
|
|
||||||
func (s *Server) queueWHOISRefresh(asns map[int]bool) {
|
|
||||||
if s.asnFetcher == nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
for asn := range asns {
|
|
||||||
s.asnFetcher.QueueImmediate(asn)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// resolveLookupTargets splits a comma-separated list of IP addresses and
|
// resolveLookupTargets splits a comma-separated list of IP addresses and
|
||||||
// hostnames and returns the IP addresses to look up, resolving each hostname.
|
// hostnames and returns the IP addresses to look up, resolving each hostname.
|
||||||
// A hostname that does not resolve is reported in the returned errors instead.
|
// A hostname that does not resolve is reported in the returned errors instead.
|
||||||
// It stops once ctx's deadline has passed; the caller then answers 504.
|
|
||||||
func resolveLookupTargets(ctx context.Context, target string) ([]string, []string) {
|
func resolveLookupTargets(ctx context.Context, target string) ([]string, []string) {
|
||||||
var ipsToLookup, resolveErrors []string
|
var ipsToLookup, resolveErrors []string
|
||||||
|
|
||||||
@@ -786,10 +732,6 @@ func resolveLookupTargets(ctx context.Context, target string) ([]string, []strin
|
|||||||
|
|
||||||
// It's a hostname - resolve it
|
// It's a hostname - resolve it
|
||||||
resolved, err := net.DefaultResolver.LookupHost(ctx, t)
|
resolved, err := net.DefaultResolver.LookupHost(ctx, t)
|
||||||
if deadlinePassed(ctx) {
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
resolveErrors = append(resolveErrors, t+": "+err.Error())
|
resolveErrors = append(resolveErrors, t+": "+err.Error())
|
||||||
|
|
||||||
@@ -844,12 +786,6 @@ func (s *Server) handleASDetailJSON() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
|
|
||||||
asInfo, prefixes, err := s.db.GetASDetailsContext(r.Context(), asn)
|
asInfo, prefixes, err := s.db.GetASDetailsContext(r.Context(), asn)
|
||||||
if deadlinePassed(r.Context()) {
|
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if errors.Is(err, database.ErrNoRoute) {
|
if errors.Is(err, database.ErrNoRoute) {
|
||||||
s.writeJSONError(w, http.StatusNotFound, err.Error())
|
s.writeJSONError(w, http.StatusNotFound, err.Error())
|
||||||
@@ -941,12 +877,6 @@ func (s *Server) handlePrefixDetailJSON() http.HandlerFunc {
|
|||||||
prefix := parsed.Masked().String()
|
prefix := parsed.Masked().String()
|
||||||
|
|
||||||
routes, _, err := s.db.GetPrefixDetailsContext(r.Context(), prefix)
|
routes, _, err := s.db.GetPrefixDetailsContext(r.Context(), prefix)
|
||||||
if deadlinePassed(r.Context()) {
|
|
||||||
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if errors.Is(err, database.ErrNoRoute) {
|
if errors.Is(err, database.ErrNoRoute) {
|
||||||
s.writeJSONError(w, http.StatusNotFound, err.Error())
|
s.writeJSONError(w, http.StatusNotFound, err.Error())
|
||||||
@@ -991,12 +921,6 @@ func (s *Server) handleASDetail() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
|
|
||||||
asInfo, prefixes, err := s.db.GetASDetailsContext(r.Context(), asn)
|
asInfo, prefixes, err := s.db.GetASDetailsContext(r.Context(), asn)
|
||||||
if deadlinePassed(r.Context()) {
|
|
||||||
http.Error(w, requestTimeoutMessage, http.StatusGatewayTimeout)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if errors.Is(err, database.ErrNoRoute) {
|
if errors.Is(err, database.ErrNoRoute) {
|
||||||
http.Error(w, "AS not found", http.StatusNotFound)
|
http.Error(w, "AS not found", http.StatusNotFound)
|
||||||
@@ -1010,12 +934,6 @@ func (s *Server) handleASDetail() http.HandlerFunc {
|
|||||||
|
|
||||||
// Get peers
|
// Get peers
|
||||||
peers, err := s.db.GetASPeersContext(r.Context(), asn)
|
peers, err := s.db.GetASPeersContext(r.Context(), asn)
|
||||||
if deadlinePassed(r.Context()) {
|
|
||||||
http.Error(w, requestTimeoutMessage, http.StatusGatewayTimeout)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Error("Failed to get AS peers", "error", err)
|
s.logger.Error("Failed to get AS peers", "error", err)
|
||||||
// Continue without peers rather than failing the whole request
|
// Continue without peers rather than failing the whole request
|
||||||
@@ -1051,9 +969,11 @@ func (s *Server) handleASDetail() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Check if context is still valid before writing response
|
// Check if context is still valid before writing response
|
||||||
if r.Context().Err() != nil {
|
select {
|
||||||
|
case <-r.Context().Done():
|
||||||
// Request was cancelled, don't write response
|
// Request was cancelled, don't write response
|
||||||
return
|
return
|
||||||
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||||
@@ -1093,12 +1013,6 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc {
|
|||||||
prefix := parsed.Masked().String()
|
prefix := parsed.Masked().String()
|
||||||
|
|
||||||
routes, originNames, err := s.db.GetPrefixDetailsContext(r.Context(), prefix)
|
routes, originNames, err := s.db.GetPrefixDetailsContext(r.Context(), prefix)
|
||||||
if deadlinePassed(r.Context()) {
|
|
||||||
http.Error(w, requestTimeoutMessage, http.StatusGatewayTimeout)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if errors.Is(err, database.ErrNoRoute) {
|
if errors.Is(err, database.ErrNoRoute) {
|
||||||
http.Error(w, "Prefix not found", http.StatusNotFound)
|
http.Error(w, "Prefix not found", http.StatusNotFound)
|
||||||
@@ -1111,7 +1025,7 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Group by origin AS and collect unique AS info
|
// Group by origin AS and collect unique AS info
|
||||||
origins := originASNs(routes, originNames)
|
originMap := originASNs(routes, originNames)
|
||||||
|
|
||||||
// Get the first route to extract some common info
|
// Get the first route to extract some common info
|
||||||
var maskLength, ipVersion int
|
var maskLength, ipVersion int
|
||||||
@@ -1119,6 +1033,12 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc {
|
|||||||
maskLength, ipVersion = prefixMaskLengthAndIPVersion(prefix)
|
maskLength, ipVersion = prefixMaskLengthAndIPVersion(prefix)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Convert origin map to sorted slice
|
||||||
|
var origins []*ASNInfo
|
||||||
|
for _, origin := range originMap {
|
||||||
|
origins = append(origins, origin)
|
||||||
|
}
|
||||||
|
|
||||||
// Create enhanced routes with AS path handles
|
// Create enhanced routes with AS path handles
|
||||||
enhancedRoutes := routesWithASPathHandles(routes)
|
enhancedRoutes := routesWithASPathHandles(routes)
|
||||||
|
|
||||||
@@ -1138,7 +1058,7 @@ func (s *Server) handlePrefixDetail() http.HandlerFunc {
|
|||||||
Routes: enhancedRoutes,
|
Routes: enhancedRoutes,
|
||||||
Origins: origins,
|
Origins: origins,
|
||||||
PeerCount: len(routes),
|
PeerCount: len(routes),
|
||||||
OriginCount: len(origins),
|
OriginCount: len(originMap),
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check if context is still valid before writing response
|
// Check if context is still valid before writing response
|
||||||
@@ -1171,11 +1091,11 @@ type ASNInfo struct {
|
|||||||
|
|
||||||
// originASNs groups routes by origin AS, with each AS's handle and description
|
// originASNs groups routes by origin AS, with each AS's handle and description
|
||||||
// from names, as the prefix query returned them, and the number of routes it
|
// from names, as the prefix query returned them, and the number of routes it
|
||||||
// originates. The origin ASes come in no particular order.
|
// originates.
|
||||||
func originASNs(
|
func originASNs(
|
||||||
routes []database.LiveRoute,
|
routes []database.LiveRoute,
|
||||||
names map[int]database.ASN,
|
names map[int]database.ASN,
|
||||||
) []*ASNInfo {
|
) map[int]*ASNInfo {
|
||||||
originMap := make(map[int]*ASNInfo)
|
originMap := make(map[int]*ASNInfo)
|
||||||
|
|
||||||
for _, route := range routes {
|
for _, route := range routes {
|
||||||
@@ -1191,12 +1111,7 @@ func originASNs(
|
|||||||
originMap[route.OriginASN].PeerCount++
|
originMap[route.OriginASN].PeerCount++
|
||||||
}
|
}
|
||||||
|
|
||||||
origins := make([]*ASNInfo, 0, len(originMap))
|
return originMap
|
||||||
for _, origin := range originMap {
|
|
||||||
origins = append(origins, origin)
|
|
||||||
}
|
|
||||||
|
|
||||||
return origins
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ASPathEntry is one AS of a route's AS path, with its handle.
|
// ASPathEntry is one AS of a route's AS path, with its handle.
|
||||||
@@ -1295,12 +1210,6 @@ func (s *Server) prefixLengthHandler(ipVersion, maxMaskLength int) http.HandlerF
|
|||||||
|
|
||||||
prefixes, err := s.db.GetRandomPrefixesByLengthContext(
|
prefixes, err := s.db.GetRandomPrefixesByLengthContext(
|
||||||
r.Context(), maskLength, ipVersion, maxPrefixes)
|
r.Context(), maskLength, ipVersion, maxPrefixes)
|
||||||
if deadlinePassed(r.Context()) {
|
|
||||||
http.Error(w, requestTimeoutMessage, http.StatusGatewayTimeout)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.logger.Error("Failed to get prefixes by length", "error", err)
|
s.logger.Error("Failed to get prefixes by length", "error", err)
|
||||||
http.Error(w, "Internal server error", http.StatusInternalServerError)
|
http.Error(w, "Internal server error", http.StatusInternalServerError)
|
||||||
@@ -1318,11 +1227,32 @@ func (s *Server) prefixLengthHandler(ipVersion, maxMaskLength int) http.HandlerF
|
|||||||
return prefixes[i].Prefix < prefixes[j].Prefix
|
return prefixes[i].Prefix < prefixes[j].Prefix
|
||||||
})
|
})
|
||||||
|
|
||||||
|
// Create enhanced prefixes with AS descriptions
|
||||||
|
type EnhancedPrefix struct {
|
||||||
|
database.LiveRoute
|
||||||
|
|
||||||
|
OriginASDescription string
|
||||||
|
Age string
|
||||||
|
}
|
||||||
|
|
||||||
|
enhancedPrefixes := make([]EnhancedPrefix, len(prefixes))
|
||||||
|
for i, prefix := range prefixes {
|
||||||
|
enhancedPrefixes[i] = EnhancedPrefix{
|
||||||
|
LiveRoute: prefix,
|
||||||
|
Age: formatAge(prefix.LastUpdated),
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get AS description
|
||||||
|
if asInfo, ok := asinfo.Get(prefix.OriginASN); ok {
|
||||||
|
enhancedPrefixes[i].OriginASDescription = asInfo.Description
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Render template
|
// Render template
|
||||||
data := map[string]any{
|
data := map[string]any{
|
||||||
"MaskLength": maskLength,
|
"MaskLength": maskLength,
|
||||||
"IPVersion": ipVersion,
|
"IPVersion": ipVersion,
|
||||||
"Prefixes": prefixesWithASDescriptions(prefixes),
|
"Prefixes": enhancedPrefixes,
|
||||||
"Count": len(prefixes),
|
"Count": len(prefixes),
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1344,34 +1274,6 @@ func (s *Server) prefixLengthHandler(ipVersion, maxMaskLength int) http.HandlerF
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// EnhancedPrefix is a live route on the prefix length page, with the
|
|
||||||
// description of its origin AS and its age.
|
|
||||||
type EnhancedPrefix struct {
|
|
||||||
database.LiveRoute
|
|
||||||
|
|
||||||
OriginASDescription string
|
|
||||||
Age string
|
|
||||||
}
|
|
||||||
|
|
||||||
// prefixesWithASDescriptions looks up the description of the origin AS of
|
|
||||||
// each prefix and gives each one's age.
|
|
||||||
func prefixesWithASDescriptions(prefixes []database.LiveRoute) []EnhancedPrefix {
|
|
||||||
enhancedPrefixes := make([]EnhancedPrefix, len(prefixes))
|
|
||||||
for i, prefix := range prefixes {
|
|
||||||
enhancedPrefixes[i] = EnhancedPrefix{
|
|
||||||
LiveRoute: prefix,
|
|
||||||
Age: formatAge(prefix.LastUpdated),
|
|
||||||
}
|
|
||||||
|
|
||||||
// Get AS description
|
|
||||||
if asInfo, ok := asinfo.Get(prefix.OriginASN); ok {
|
|
||||||
enhancedPrefixes[i].OriginASDescription = asInfo.Description
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return enhancedPrefixes
|
|
||||||
}
|
|
||||||
|
|
||||||
// handlePrefixLength6 shows a random sample of IPv6 prefixes with the
|
// handlePrefixLength6 shows a random sample of IPv6 prefixes with the
|
||||||
// specified mask length
|
// specified mask length
|
||||||
func (s *Server) handlePrefixLength6() http.HandlerFunc {
|
func (s *Server) handlePrefixLength6() http.HandlerFunc {
|
||||||
|
|||||||
@@ -29,12 +29,6 @@ const (
|
|||||||
testPeerV6 = "2001:db8::1"
|
testPeerV6 = "2001:db8::1"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The names the stats handler tests report the two stats handlers by.
|
|
||||||
const (
|
|
||||||
statusJSONHandlerName = "status.json"
|
|
||||||
statsHandlerName = "stats"
|
|
||||||
)
|
|
||||||
|
|
||||||
// blockingStatsDB embeds database.Store (left nil) and overrides only
|
// blockingStatsDB embeds database.Store (left nil) and overrides only
|
||||||
// GetStatsContext, which blocks until release is closed. The stats handlers
|
// GetStatsContext, which blocks until release is closed. The stats handlers
|
||||||
// call it in a goroutine; every other Store method is unused on the timeout
|
// call it in a goroutine; every other Store method is unused on the timeout
|
||||||
@@ -65,8 +59,8 @@ func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) {
|
|||||||
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
||||||
|
|
||||||
handlers := map[string]http.HandlerFunc{
|
handlers := map[string]http.HandlerFunc{
|
||||||
statusJSONHandlerName: s.handleStatusJSON(),
|
"status.json": s.handleStatusJSON(),
|
||||||
statsHandlerName: s.handleStats(),
|
"stats": s.handleStats(),
|
||||||
}
|
}
|
||||||
|
|
||||||
baseline := settledGoroutineCount()
|
baseline := settledGoroutineCount()
|
||||||
@@ -135,8 +129,8 @@ func TestStatsHandlersAnswerFromMemory(t *testing.T) {
|
|||||||
|
|
||||||
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
|
||||||
handlers := map[string]http.HandlerFunc{
|
handlers := map[string]http.HandlerFunc{
|
||||||
statusJSONHandlerName: s.handleStatusJSON(),
|
"status.json": s.handleStatusJSON(),
|
||||||
statsHandlerName: s.handleStats(),
|
"stats": s.handleStats(),
|
||||||
}
|
}
|
||||||
|
|
||||||
for name, handler := range handlers {
|
for name, handler := range handlers {
|
||||||
@@ -331,309 +325,6 @@ func TestPrefixDetailShowsOriginNames(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// shortRequestDeadline is the deadline the tests below give a request. The
|
|
||||||
// server's 30-second limit and the stats handlers' 4-second limit inherit it,
|
|
||||||
// so the tests do not wait for either.
|
|
||||||
const shortRequestDeadline = 50 * time.Millisecond
|
|
||||||
|
|
||||||
// TestRequestPastTimeLimitAnswers504 sends a request through the server's
|
|
||||||
// router to a page and to a JSON route whose handlers wait for the time limit
|
|
||||||
// to pass and return without answering. The page answers 504; the JSON route
|
|
||||||
// answers 504 with the JSON error body. Before the fix both answered 408,
|
|
||||||
// written by the time limit middleware from its own goroutine while the
|
|
||||||
// handler could still be writing.
|
|
||||||
func TestRequestPastTimeLimitAnswers504(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := New(nil, streamer.New(logger.New(), metrics.New()), logger.New(),
|
|
||||||
&config.Config{})
|
|
||||||
|
|
||||||
waitForTimeLimit := func(_ http.ResponseWriter, r *http.Request) {
|
|
||||||
<-r.Context().Done()
|
|
||||||
}
|
|
||||||
// Registered the way setupRoutes registers pages and JSON routes.
|
|
||||||
s.router.Get("/slow-page", waitForTimeLimit)
|
|
||||||
s.router.Get("/slow.json",
|
|
||||||
JSONValidationMiddleware(http.HandlerFunc(waitForTimeLimit)).ServeHTTP)
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
path string
|
|
||||||
wantJSONBody bool
|
|
||||||
}{
|
|
||||||
{path: "/slow-page"},
|
|
||||||
{path: "/slow.json", wantJSONBody: true},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
ctx, cancel := context.WithTimeout(context.Background(),
|
|
||||||
shortRequestDeadline)
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec,
|
|
||||||
httptest.NewRequestWithContext(ctx, http.MethodGet, tt.path, nil))
|
|
||||||
cancel()
|
|
||||||
|
|
||||||
if rec.Code != http.StatusGatewayTimeout {
|
|
||||||
t.Errorf("%s: status %d, want %d; body %s",
|
|
||||||
tt.path, rec.Code, http.StatusGatewayTimeout, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
if tt.wantJSONBody {
|
|
||||||
checkJSONErrorCode(t, tt.path, rec, http.StatusGatewayTimeout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestStatsHandlersAnswer504WhenDatabaseReadRunsOut calls each stats handler
|
|
||||||
// with a store that does not answer before the handler's 4-second limit runs
|
|
||||||
// out, and checks that the handler itself answers 504 with the JSON error
|
|
||||||
// body. Before the fix status.json answered 408, and /api/v1/stats wrote
|
|
||||||
// nothing, which the JSON check on its route turned into 500.
|
|
||||||
func TestStatsHandlersAnswer504WhenDatabaseReadRunsOut(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
release := make(chan struct{})
|
|
||||||
defer close(release)
|
|
||||||
|
|
||||||
db := blockingStatsDB{release: release}
|
|
||||||
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(),
|
|
||||||
&config.Config{})
|
|
||||||
|
|
||||||
handlers := map[string]http.HandlerFunc{
|
|
||||||
statusJSONHandlerName: s.handleStatusJSON(),
|
|
||||||
statsHandlerName: s.handleStats(),
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, handler := range handlers {
|
|
||||||
ctx, cancel := context.WithTimeout(context.Background(),
|
|
||||||
shortRequestDeadline)
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
handler(rec, httptest.NewRequestWithContext(ctx, http.MethodGet, "/", nil))
|
|
||||||
cancel()
|
|
||||||
|
|
||||||
if rec.Code != http.StatusGatewayTimeout {
|
|
||||||
t.Errorf("%s: status %d, want %d; body %s",
|
|
||||||
name, rec.Code, http.StatusGatewayTimeout, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
checkJSONErrorCode(t, name, rec, http.StatusGatewayTimeout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// slowDB embeds database.Store (left nil) and overrides the reads the routes
|
|
||||||
// in TestRoutesAnswer504WhenDatabaseReadIsCutOff make. Each read waits until
|
|
||||||
// its context ends and then answers as the real store does when the time
|
|
||||||
// limit cuts it off: the IP read with the context's error, the prefix read
|
|
||||||
// that there is no route, and the others with what they had read, here
|
|
||||||
// nothing, and no error. Every other Store method is unused and would panic
|
|
||||||
// if called.
|
|
||||||
type slowDB struct {
|
|
||||||
database.Store
|
|
||||||
}
|
|
||||||
|
|
||||||
func (slowDB) GetStatsContext(ctx context.Context) (database.Stats, error) {
|
|
||||||
<-ctx.Done()
|
|
||||||
|
|
||||||
return database.Stats{}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (slowDB) GetIPInfoContext(
|
|
||||||
ctx context.Context, _ string,
|
|
||||||
) (*database.IPInfo, error) {
|
|
||||||
<-ctx.Done()
|
|
||||||
|
|
||||||
return nil, ctx.Err()
|
|
||||||
}
|
|
||||||
|
|
||||||
func (slowDB) GetASDetailsContext(
|
|
||||||
ctx context.Context, asn int,
|
|
||||||
) (*database.ASN, []database.LiveRoute, error) {
|
|
||||||
<-ctx.Done()
|
|
||||||
|
|
||||||
return &database.ASN{ASN: asn}, nil, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (slowDB) GetPrefixDetailsContext(
|
|
||||||
ctx context.Context, _ string,
|
|
||||||
) ([]database.LiveRoute, map[int]database.ASN, error) {
|
|
||||||
<-ctx.Done()
|
|
||||||
|
|
||||||
return nil, nil, database.ErrNoRoute
|
|
||||||
}
|
|
||||||
|
|
||||||
func (slowDB) GetRandomPrefixesByLengthContext(
|
|
||||||
ctx context.Context, _, _, _ int,
|
|
||||||
) ([]database.LiveRoute, error) {
|
|
||||||
<-ctx.Done()
|
|
||||||
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRoutesAnswer504WhenDatabaseReadIsCutOff sends a request through the
|
|
||||||
// server's router for each route that reads the database, with a store whose
|
|
||||||
// reads are cut off when the request's time limit passes. Each route answers
|
|
||||||
// 504 with the time limit message. Without its check after the read, a route
|
|
||||||
// answers with what the cut-off read returned: 500, 404, 200, or, on a page
|
|
||||||
// that stops before rendering once the request has ended, 504 with an empty
|
|
||||||
// body. The health check is left out: it answers 503 when its own 2-second
|
|
||||||
// ping runs out.
|
|
||||||
func TestRoutesAnswer504WhenDatabaseReadIsCutOff(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := New(slowDB{}, streamer.New(logger.New(), metrics.New()), logger.New(),
|
|
||||||
&config.Config{})
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
path string
|
|
||||||
wantJSONBody bool
|
|
||||||
}{
|
|
||||||
{path: "/as/64500"},
|
|
||||||
{path: "/prefix/192.0.2.0/24"},
|
|
||||||
{path: "/prefixlength/24"},
|
|
||||||
{path: "/prefixlength6/48"},
|
|
||||||
{path: "/status.json", wantJSONBody: true},
|
|
||||||
{path: "/ip/192.0.2.1", wantJSONBody: true},
|
|
||||||
{path: "/api/v1/stats", wantJSONBody: true},
|
|
||||||
{path: "/api/v1/ip/192.0.2.1", wantJSONBody: true},
|
|
||||||
{path: "/api/v1/as/64500", wantJSONBody: true},
|
|
||||||
{path: "/api/v1/prefix/192.0.2.0/24", wantJSONBody: true},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
checkRouteAnswers504(t, s, tt.path, tt.wantJSONBody)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// slowSecondReadDB embeds database.Store (left nil) and overrides the reads
|
|
||||||
// of the routes that read the database twice. The first read answers at once;
|
|
||||||
// the second, the AS page's peers read or the stats routes' WHOIS read, waits
|
|
||||||
// until its context ends and then answers with the context's error, as the
|
|
||||||
// real store does.
|
|
||||||
type slowSecondReadDB struct {
|
|
||||||
database.Store
|
|
||||||
}
|
|
||||||
|
|
||||||
func (slowSecondReadDB) GetASDetailsContext(
|
|
||||||
_ context.Context, asn int,
|
|
||||||
) (*database.ASN, []database.LiveRoute, error) {
|
|
||||||
return &database.ASN{ASN: asn}, nil, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (slowSecondReadDB) GetASPeersContext(
|
|
||||||
ctx context.Context, _ int,
|
|
||||||
) ([]database.ASPeer, error) {
|
|
||||||
<-ctx.Done()
|
|
||||||
|
|
||||||
return nil, ctx.Err()
|
|
||||||
}
|
|
||||||
|
|
||||||
func (slowSecondReadDB) GetStatsContext(_ context.Context) (database.Stats, error) {
|
|
||||||
return database.Stats{}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (slowSecondReadDB) GetWHOISStats(
|
|
||||||
ctx context.Context, _ time.Duration,
|
|
||||||
) (*database.WHOISStats, error) {
|
|
||||||
<-ctx.Done()
|
|
||||||
|
|
||||||
return nil, ctx.Err()
|
|
||||||
}
|
|
||||||
|
|
||||||
// idleASNFetcher is a WHOIS fetcher that does nothing. The stats handlers
|
|
||||||
// read the WHOIS stats only when the server has a fetcher.
|
|
||||||
type idleASNFetcher struct{}
|
|
||||||
|
|
||||||
func (idleASNFetcher) QueueImmediate(int) {}
|
|
||||||
|
|
||||||
func (idleASNFetcher) GetStats() ASNFetcherStats {
|
|
||||||
return ASNFetcherStats{}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRoutesAnswer504WhenSecondDatabaseReadIsCutOff sends a request through
|
|
||||||
// the server's router for each route that reads the database twice, with a
|
|
||||||
// store that answers the first read and cuts off the second when the
|
|
||||||
// request's time limit passes. Each route answers 504 with the time limit
|
|
||||||
// message. Without the check after the second read, the AS page answers 504
|
|
||||||
// with an empty body and the stats routes answer 200 without the WHOIS stats.
|
|
||||||
func TestRoutesAnswer504WhenSecondDatabaseReadIsCutOff(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := New(slowSecondReadDB{}, streamer.New(logger.New(), metrics.New()),
|
|
||||||
logger.New(), &config.Config{})
|
|
||||||
s.SetASNFetcher(idleASNFetcher{})
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
path string
|
|
||||||
wantJSONBody bool
|
|
||||||
}{
|
|
||||||
{path: "/as/64500"},
|
|
||||||
{path: "/status.json", wantJSONBody: true},
|
|
||||||
{path: "/api/v1/stats", wantJSONBody: true},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
checkRouteAnswers504(t, s, tt.path, tt.wantJSONBody)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkRouteAnswers504 sends a request for path through the server's router
|
|
||||||
// with a short deadline and checks that it answers 504 with the time limit
|
|
||||||
// message: the JSON error body on JSON routes, the plain message on pages.
|
|
||||||
func checkRouteAnswers504(
|
|
||||||
t *testing.T, s *Server, path string, wantJSONBody bool,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), shortRequestDeadline)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.router.ServeHTTP(rec,
|
|
||||||
httptest.NewRequestWithContext(ctx, http.MethodGet, path, nil))
|
|
||||||
|
|
||||||
if rec.Code != http.StatusGatewayTimeout {
|
|
||||||
t.Errorf("%s: status %d, want %d; body %s",
|
|
||||||
path, rec.Code, http.StatusGatewayTimeout, rec.Body)
|
|
||||||
}
|
|
||||||
|
|
||||||
if wantJSONBody {
|
|
||||||
checkJSONErrorCode(t, path, rec, http.StatusGatewayTimeout)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
body := strings.TrimSpace(rec.Body.String())
|
|
||||||
if body != requestTimeoutMessage {
|
|
||||||
t.Errorf("%s: body %q, want %q", path, body, requestTimeoutMessage)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkJSONErrorCode checks that rec holds the JSON error body with the given
|
|
||||||
// code.
|
|
||||||
func checkJSONErrorCode(
|
|
||||||
t *testing.T, name string, rec *httptest.ResponseRecorder, want int,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var body struct {
|
|
||||||
Status string `json:"status"`
|
|
||||||
Error struct {
|
|
||||||
Code int `json:"code"`
|
|
||||||
} `json:"error"`
|
|
||||||
}
|
|
||||||
|
|
||||||
err := json.Unmarshal(rec.Body.Bytes(), &body)
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("%s: decoding the answer %q: %v", name, rec.Body, err)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if body.Status != jsonStatusError || body.Error.Code != want {
|
|
||||||
t.Errorf("%s: answer %s, want status \"error\" and error code %d",
|
|
||||||
name, rec.Body, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// settledGoroutineCount lets transient goroutines finish, then reports the
|
// settledGoroutineCount lets transient goroutines finish, then reports the
|
||||||
// current count.
|
// current count.
|
||||||
func settledGoroutineCount() int {
|
func settledGoroutineCount() int {
|
||||||
|
|||||||
+136
-21
@@ -4,7 +4,6 @@ import (
|
|||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
|
||||||
"fmt"
|
"fmt"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
@@ -113,28 +112,148 @@ func JSONResponseMiddleware(next http.Handler) http.Handler {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// requestTimeoutMessage is the error message sent with 504 Gateway Timeout.
|
// timeoutWriter wraps ResponseWriter to prevent concurrent writes after timeout
|
||||||
const requestTimeoutMessage = "Request timeout"
|
type timeoutWriter struct {
|
||||||
|
http.ResponseWriter
|
||||||
|
|
||||||
// deadlinePassed reports whether ctx ended because its deadline passed.
|
mu sync.Mutex
|
||||||
|
written bool
|
||||||
|
header http.Header // cached header to prevent concurrent access
|
||||||
|
}
|
||||||
|
|
||||||
|
func (tw *timeoutWriter) Write(b []byte) (int, error) {
|
||||||
|
tw.mu.Lock()
|
||||||
|
defer tw.mu.Unlock()
|
||||||
|
|
||||||
|
if tw.written {
|
||||||
|
return 0, nil // Discard writes after timeout
|
||||||
|
}
|
||||||
|
|
||||||
|
return tw.ResponseWriter.Write(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (tw *timeoutWriter) WriteHeader(statusCode int) {
|
||||||
|
tw.mu.Lock()
|
||||||
|
defer tw.mu.Unlock()
|
||||||
|
|
||||||
|
if tw.written {
|
||||||
|
return // Discard writes after timeout
|
||||||
|
}
|
||||||
|
|
||||||
|
tw.ResponseWriter.WriteHeader(statusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (tw *timeoutWriter) Header() http.Header {
|
||||||
|
tw.mu.Lock()
|
||||||
|
defer tw.mu.Unlock()
|
||||||
|
|
||||||
|
if tw.written {
|
||||||
|
// Return a copy to prevent modifications after timeout
|
||||||
|
if tw.header == nil {
|
||||||
|
tw.header = make(http.Header)
|
||||||
|
}
|
||||||
|
|
||||||
|
return tw.header
|
||||||
|
}
|
||||||
|
|
||||||
|
return tw.ResponseWriter.Header()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (tw *timeoutWriter) markWritten() {
|
||||||
|
tw.mu.Lock()
|
||||||
|
defer tw.mu.Unlock()
|
||||||
|
|
||||||
|
tw.written = true
|
||||||
|
}
|
||||||
|
|
||||||
|
// TimeoutMiddleware creates an HTTP middleware that enforces a request timeout.
|
||||||
|
// If the handler does not complete within the specified duration, the middleware
|
||||||
|
// returns a JSON error response with HTTP status 408 (Request Timeout).
|
||||||
//
|
//
|
||||||
// A handler calls it right after each database read or host name lookup,
|
// The timeout parameter specifies the maximum duration allowed for request processing.
|
||||||
// before it looks at what came back, error or not, and answers 504 if it
|
// The returned middleware handles panics from the wrapped handler by re-panicking
|
||||||
// reports true. A read cut off by the time limit can return an error, or part
|
// after cleanup, and prevents concurrent writes to the response after timeout occurs.
|
||||||
// of its rows or none of them without one, so only this check tells a cut-off
|
func TimeoutMiddleware(
|
||||||
// read from a complete one.
|
timeout time.Duration,
|
||||||
func deadlinePassed(ctx context.Context) bool {
|
logger *slog.Logger,
|
||||||
return errors.Is(ctx.Err(), context.DeadlineExceeded)
|
) func(http.Handler) http.Handler {
|
||||||
|
return func(next http.Handler) http.Handler {
|
||||||
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
startTime := time.Now()
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(r.Context(), timeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
tw := &timeoutWriter{
|
||||||
|
ResponseWriter: w,
|
||||||
|
header: make(http.Header),
|
||||||
|
}
|
||||||
|
|
||||||
|
done := make(chan struct{})
|
||||||
|
panicChan := make(chan any, 1)
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
defer func() {
|
||||||
|
if p := recover(); p != nil {
|
||||||
|
panicChan <- p
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
next.ServeHTTP(tw, r.WithContext(ctx))
|
||||||
|
close(done)
|
||||||
|
}()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case p := <-panicChan:
|
||||||
|
panic(p)
|
||||||
|
case <-done:
|
||||||
|
return
|
||||||
|
case <-ctx.Done():
|
||||||
|
tw.markWritten() // Prevent the handler from writing after timeout
|
||||||
|
|
||||||
|
execTime := time.Since(startTime)
|
||||||
|
|
||||||
|
// Log the timeout as a warning
|
||||||
|
logger.Warn("Request timeout",
|
||||||
|
"method", r.Method,
|
||||||
|
"path", r.URL.Path,
|
||||||
|
"duration_ms", execTime.Milliseconds(),
|
||||||
|
"remote_addr", r.RemoteAddr,
|
||||||
|
)
|
||||||
|
|
||||||
|
// Write directly to the underlying writer since we've marked tw as written
|
||||||
|
// This is safe because markWritten() prevents the handler from writing
|
||||||
|
tw.mu.Lock()
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusRequestTimeout)
|
||||||
|
tw.mu.Unlock()
|
||||||
|
|
||||||
|
response := map[string]any{
|
||||||
|
jsonKeyStatus: jsonStatusError,
|
||||||
|
jsonKeyError: map[string]any{
|
||||||
|
jsonKeyMsg: "Request timeout",
|
||||||
|
jsonKeyCode: http.StatusRequestTimeout,
|
||||||
|
},
|
||||||
|
"@meta": map[string]any{
|
||||||
|
"time_zone": "UTC",
|
||||||
|
"api_version": 1,
|
||||||
|
"execution_time": fmt.Sprintf("%d ms", execTime.Milliseconds()),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
_ = json.NewEncoder(w).Encode(response)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// JSONValidationMiddleware is an HTTP middleware that validates JSON API responses.
|
// JSONValidationMiddleware is an HTTP middleware that validates JSON API responses.
|
||||||
// It ensures that responses with Content-Type "application/json" contain valid JSON.
|
// It ensures that responses with Content-Type "application/json" contain valid JSON.
|
||||||
//
|
//
|
||||||
// If a response is not valid JSON or is empty when JSON is expected, the middleware
|
// If a response is not valid JSON or is empty when JSON is expected, the middleware
|
||||||
// returns a properly formatted JSON error response. If the request's time limit
|
// returns a properly formatted JSON error response. For timeout errors (status 408),
|
||||||
// has passed, it answers 504 with the error message "Request timeout". For other
|
// the error message will be "Request timeout". For other errors, it returns
|
||||||
// errors, it returns "Internal server error" with status 500 if the original
|
// "Internal server error" with status 500 if the original status was 200.
|
||||||
// status was 200.
|
|
||||||
//
|
//
|
||||||
// Non-JSON responses are passed through unchanged.
|
// Non-JSON responses are passed through unchanged.
|
||||||
func JSONValidationMiddleware(next http.Handler) http.Handler {
|
func JSONValidationMiddleware(next http.Handler) http.Handler {
|
||||||
@@ -186,15 +305,11 @@ func JSONValidationMiddleware(next http.Handler) http.Handler {
|
|||||||
statusCode = http.StatusInternalServerError
|
statusCode = http.StatusInternalServerError
|
||||||
}
|
}
|
||||||
|
|
||||||
if deadlinePassed(r.Context()) {
|
|
||||||
statusCode = http.StatusGatewayTimeout
|
|
||||||
}
|
|
||||||
|
|
||||||
w.WriteHeader(statusCode)
|
w.WriteHeader(statusCode)
|
||||||
|
|
||||||
errorMsg := "Internal server error"
|
errorMsg := "Internal server error"
|
||||||
if statusCode == http.StatusGatewayTimeout {
|
if statusCode == http.StatusRequestTimeout {
|
||||||
errorMsg = requestTimeoutMessage
|
errorMsg = "Request timeout"
|
||||||
}
|
}
|
||||||
|
|
||||||
response := map[string]any{
|
response := map[string]any{
|
||||||
|
|||||||
@@ -16,12 +16,11 @@ func (s *Server) setupRoutes() {
|
|||||||
r.Use(middleware.RealIP)
|
r.Use(middleware.RealIP)
|
||||||
r.Use(RequestLoggerMiddleware(s.logger.Logger)) // Structured request logging
|
r.Use(RequestLoggerMiddleware(s.logger.Logger)) // Structured request logging
|
||||||
r.Use(middleware.Recoverer)
|
r.Use(middleware.Recoverer)
|
||||||
r.Use(JSONResponseMiddleware)
|
|
||||||
// Timeout must come after JSONResponseMiddleware, which sends its answer
|
|
||||||
// when the handler returns and would send a 200 ahead of Timeout's 504 for
|
|
||||||
// a handler that wrote nothing.
|
|
||||||
const requestTimeout = 30 * time.Second // Increased from 8s for slow queries
|
const requestTimeout = 30 * time.Second // Increased from 8s for slow queries
|
||||||
r.Use(middleware.Timeout(requestTimeout))
|
|
||||||
|
r.Use(TimeoutMiddleware(requestTimeout, s.logger.Logger))
|
||||||
|
r.Use(JSONResponseMiddleware)
|
||||||
|
|
||||||
// Routes
|
// Routes
|
||||||
r.Get("/", s.handleIndex())
|
r.Get("/", s.handleIndex())
|
||||||
|
|||||||
@@ -1,6 +0,0 @@
|
|||||||
{
|
|
||||||
"private": true,
|
|
||||||
"devDependencies": {
|
|
||||||
"prettier": "3.8.1"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+11
-109
@@ -2,27 +2,15 @@
|
|||||||
# script/bootstrap: install all dependencies needed to build and develop
|
# script/bootstrap: install all dependencies needed to build and develop
|
||||||
# this repo. Idempotent: every install is guarded by a check so already
|
# this repo. Idempotent: every install is guarded by a check so already
|
||||||
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
||||||
# or apk (detected in that order); assumes nothing is present. The
|
# or apk (detected in that order); assumes NOTHING is present (not git,
|
||||||
# installed node is used only when its major version is the pinned one;
|
# make, or go). The linter and the tests run only in Docker, as phases of
|
||||||
# otherwise (node missing, or another major) the pinned version is
|
# the Dockerfile, so nothing installs golangci-lint here.
|
||||||
# installed via nvm (installing nvm itself first, from a hash-verified
|
|
||||||
# release archive, never curl | sh), and yarn and the packages are
|
|
||||||
# installed under it. Go formats the code and builds it on the host; the
|
|
||||||
# linter and the tests run only in Docker, as phases of the Dockerfile.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
# Pinned versions, 2026-07-06
|
|
||||||
NODE_VERSION="22.17.0"
|
|
||||||
NVM_VERSION="0.40.3"
|
|
||||||
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
|
|
||||||
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
|
|
||||||
YARN_VERSION="1.22.22"
|
|
||||||
|
|
||||||
PKGMGR=""
|
PKGMGR=""
|
||||||
SUDO=""
|
SUDO=""
|
||||||
APT_UPDATED=""
|
|
||||||
|
|
||||||
detect_pkgmgr() {
|
detect_pkgmgr() {
|
||||||
[ -n "$PKGMGR" ] && return 0
|
[ -n "$PKGMGR" ] && return 0
|
||||||
@@ -51,14 +39,7 @@ pkg_install() {
|
|||||||
detect_pkgmgr
|
detect_pkgmgr
|
||||||
case "$PKGMGR" in
|
case "$PKGMGR" in
|
||||||
nix) nix-env -iA "nixpkgs.$1" ;;
|
nix) nix-env -iA "nixpkgs.$1" ;;
|
||||||
apt)
|
apt) $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2" ;;
|
||||||
# Package lists may be empty (fresh images); refresh once per run.
|
|
||||||
if [ -z "$APT_UPDATED" ]; then
|
|
||||||
$SUDO env DEBIAN_FRONTEND=noninteractive apt-get update
|
|
||||||
APT_UPDATED=1
|
|
||||||
fi
|
|
||||||
$SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2"
|
|
||||||
;;
|
|
||||||
brew) brew install "$3" ;;
|
brew) brew install "$3" ;;
|
||||||
apk) apk add --no-cache "$4" ;;
|
apk) apk add --no-cache "$4" ;;
|
||||||
esac
|
esac
|
||||||
@@ -68,100 +49,21 @@ missing() {
|
|||||||
! command -v "$1" >/dev/null 2>&1
|
! command -v "$1" >/dev/null 2>&1
|
||||||
}
|
}
|
||||||
|
|
||||||
# verify_sha256 <file> <expected-hash>
|
|
||||||
verify_sha256() {
|
|
||||||
if command -v sha256sum >/dev/null 2>&1; then
|
|
||||||
actual="$(sha256sum "$1" | cut -d' ' -f1)"
|
|
||||||
else
|
|
||||||
actual="$(shasum -a 256 "$1" | cut -d' ' -f1)"
|
|
||||||
fi
|
|
||||||
if [ "$actual" != "$2" ]; then
|
|
||||||
echo "bootstrap: sha256 mismatch for $1" >&2
|
|
||||||
echo " expected: $2" >&2
|
|
||||||
echo " actual: $actual" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# nvm is a bash script; run a command in a bash with nvm loaded
|
|
||||||
nvm_sh() {
|
|
||||||
bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*"
|
|
||||||
}
|
|
||||||
|
|
||||||
ensure_nvm() {
|
|
||||||
[ -s "$HOME/.nvm/nvm.sh" ] && return 0
|
|
||||||
# nvm prerequisites; nvm itself requires bash
|
|
||||||
if missing bash; then pkg_install bash bash bash bash; fi
|
|
||||||
if missing curl; then pkg_install curl curl curl curl; fi
|
|
||||||
if missing git; then pkg_install git git git git; fi
|
|
||||||
tmp="$(mktemp -d)"
|
|
||||||
curl -fsSL -o "$tmp/nvm.tar.gz" \
|
|
||||||
"https://github.com/nvm-sh/nvm/archive/refs/tags/v${NVM_VERSION}.tar.gz"
|
|
||||||
verify_sha256 "$tmp/nvm.tar.gz" "$NVM_SHA256"
|
|
||||||
mkdir -p "$HOME/.nvm"
|
|
||||||
tar -xzf "$tmp/nvm.tar.gz" -C "$HOME/.nvm" --strip-components=1
|
|
||||||
rm -rf "$tmp"
|
|
||||||
}
|
|
||||||
|
|
||||||
# True when the node on PATH has the pinned major version. Only the
|
|
||||||
# major is compared, as in the shared script/bootstrap of sneak/prompts.
|
|
||||||
# Another major is not used: the pinned yarn 1 prints a deprecation
|
|
||||||
# warning under node 24. script/fmt and script/fmt-check carry this
|
|
||||||
# function unchanged, to run the yarn installed here.
|
|
||||||
node_is_pinned_major() {
|
|
||||||
if ! command -v node >/dev/null 2>&1; then return 1; fi
|
|
||||||
installed="$(node --version)"
|
|
||||||
installed="${installed#v}"
|
|
||||||
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
|
|
||||||
}
|
|
||||||
|
|
||||||
ensure_node() {
|
|
||||||
if node_is_pinned_major; then
|
|
||||||
echo "bootstrap: using node $(node --version)"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
ensure_nvm
|
|
||||||
nvm_sh "nvm install $NODE_VERSION"
|
|
||||||
nvm_sh "nvm use $NODE_VERSION >/dev/null && \
|
|
||||||
echo \"bootstrap: using node \$(node --version)\""
|
|
||||||
}
|
|
||||||
|
|
||||||
ensure_yarn() {
|
|
||||||
if ! node_is_pinned_major; then
|
|
||||||
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
|
|
||||||
corepack prepare yarn@$YARN_VERSION --activate"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
if ! missing yarn; then return 0; fi
|
|
||||||
if ! missing corepack; then
|
|
||||||
corepack enable
|
|
||||||
corepack prepare "yarn@$YARN_VERSION" --activate
|
|
||||||
else
|
|
||||||
npm install -g "yarn@$YARN_VERSION"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
install_js_deps() {
|
|
||||||
if node_is_pinned_major; then
|
|
||||||
yarn install --frozen-lockfile
|
|
||||||
else
|
|
||||||
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
|
|
||||||
yarn install --frozen-lockfile"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
|
|
||||||
if missing make; then pkg_install gnumake make make make; fi
|
# Base tooling (every repo)
|
||||||
if missing git; then pkg_install git git git git; fi
|
if missing git; then pkg_install git git git git; fi
|
||||||
|
if missing make; then pkg_install gnumake make make make; fi
|
||||||
|
|
||||||
# Go toolchain
|
# Go toolchain
|
||||||
if missing go; then pkg_install go golang go go; fi
|
if missing go; then pkg_install go golang go go; fi
|
||||||
|
|
||||||
ensure_node
|
# Warn, don't fail: make build works without it.
|
||||||
ensure_yarn
|
if missing docker; then
|
||||||
install_js_deps
|
echo "bootstrap: WARNING: docker not found; install it to run" \
|
||||||
|
"make test, make lint, make check and make docker." >&2
|
||||||
|
fi
|
||||||
|
|
||||||
go mod download
|
go mod download
|
||||||
|
|
||||||
|
|||||||
+9
-14
@@ -2,12 +2,9 @@
|
|||||||
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
||||||
# checks out and runs this and nothing else, and script/fmt-check runs
|
# checks out and runs this and nothing else, and script/fmt-check runs
|
||||||
# the formatter on the host, which a pristine checkout cannot do.
|
# the formatter on the host, which a pristine checkout cannot do.
|
||||||
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST
|
# --no-cache for the same reason as script/docker: the gate phases the
|
||||||
# line, since a cached gate phase is a check that did not run; unlike
|
# final stage depends on are RUN steps, and a cached one is a check that
|
||||||
# --no-cache, it keeps the layers above that line and the cache mounts.
|
# did not run.
|
||||||
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
|
|
||||||
# before every step that lints, tests or compiles; a stage without that
|
|
||||||
# line is served from cache on an unchanged tree, with no warning.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -17,17 +14,15 @@ main() {
|
|||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
"$SCRIPT_DIR/bootstrap"
|
"$SCRIPT_DIR/bootstrap"
|
||||||
"$SCRIPT_DIR/check"
|
"$SCRIPT_DIR/check"
|
||||||
# The version and the tag each get their own line: a failing
|
# Own line: a failing command substitution inside an argument does
|
||||||
# command substitution inside an argument does not trip `set -e`,
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
# so the inline form degrades silently to an empty constant. The
|
# empty constant. The VERSION build argument takes precedence over
|
||||||
# VERSION build argument takes precedence over the version a build
|
# the version a build stage derives from the .git in the context.
|
||||||
# stage derives from the .git in the context.
|
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
tag="$("$SCRIPT_DIR/projectname")"
|
docker build --no-cache \
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$tag" .
|
-t "$("$SCRIPT_DIR/projectname")" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
+8
-14
@@ -1,12 +1,8 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/docker: build the Docker image tagged with the project name.
|
# script/docker: build the Docker image tagged with the project name.
|
||||||
# Identical in all repos; the tag comes from script/projectname.
|
# Identical in all repos; the tag comes from script/projectname.
|
||||||
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST
|
# --no-cache because the gate phases the final stage depends on are RUN
|
||||||
# line, since a cached gate phase is a check that did not run; unlike
|
# steps, and a cached one is a check that did not run.
|
||||||
# --no-cache, it keeps the layers above that line and the cache mounts.
|
|
||||||
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
|
|
||||||
# before every step that lints, tests or compiles; a stage without that
|
|
||||||
# line is served from cache on an unchanged tree, with no warning.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -14,17 +10,15 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
# The version and the tag each get their own line: a failing
|
# Own line: a failing command substitution inside an argument does
|
||||||
# command substitution inside an argument does not trip `set -e`,
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
# so the inline form degrades silently to an empty constant. The
|
# empty constant. The VERSION build argument takes precedence over
|
||||||
# VERSION build argument takes precedence over the version a build
|
# the version a build stage derives from the .git in the context.
|
||||||
# stage derives from the .git in the context.
|
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
tag="$("$SCRIPT_DIR/projectname")"
|
docker build --no-cache \
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$tag" .
|
-t "$("$SCRIPT_DIR/projectname")" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
+1
-32
@@ -1,43 +1,12 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/fmt: format all files (writes): Go with go fmt, markdown with
|
# script/fmt: format all files (writes).
|
||||||
# prettier.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
# Must match the pin in script/bootstrap.
|
|
||||||
NODE_VERSION="22.17.0"
|
|
||||||
|
|
||||||
# True when the node on PATH has the pinned major version: the same
|
|
||||||
# function as in script/bootstrap, which uses it to decide where it
|
|
||||||
# installs yarn.
|
|
||||||
node_is_pinned_major() {
|
|
||||||
if ! command -v node >/dev/null 2>&1; then return 1; fi
|
|
||||||
installed="$(node --version)"
|
|
||||||
installed="${installed#v}"
|
|
||||||
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
|
|
||||||
}
|
|
||||||
|
|
||||||
# Run the yarn script/bootstrap installed: the yarn on PATH when the
|
|
||||||
# node on PATH has the pinned major version, and otherwise yarn under
|
|
||||||
# the pinned node in nvm, which bootstrap leaves off the PATH of the
|
|
||||||
# shell that called it. nvm is a bash script, hence the subshell.
|
|
||||||
run_yarn() {
|
|
||||||
if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then
|
|
||||||
exec yarn "$@"
|
|
||||||
fi
|
|
||||||
if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
|
||||||
echo "fmt: no yarn; run script/bootstrap first" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
|
|
||||||
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
go fmt ./...
|
go fmt ./...
|
||||||
run_yarn run prettier --write '**/*.md' --tab-width 4 --prose-wrap always
|
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
@@ -5,35 +5,6 @@ set -eu
|
|||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
# Must match the pin in script/bootstrap.
|
|
||||||
NODE_VERSION="22.17.0"
|
|
||||||
|
|
||||||
# True when the node on PATH has the pinned major version: the same
|
|
||||||
# function as in script/bootstrap, which uses it to decide where it
|
|
||||||
# installs yarn.
|
|
||||||
node_is_pinned_major() {
|
|
||||||
if ! command -v node >/dev/null 2>&1; then return 1; fi
|
|
||||||
installed="$(node --version)"
|
|
||||||
installed="${installed#v}"
|
|
||||||
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
|
|
||||||
}
|
|
||||||
|
|
||||||
# Run the yarn script/bootstrap installed: the yarn on PATH when the
|
|
||||||
# node on PATH has the pinned major version, and otherwise yarn under
|
|
||||||
# the pinned node in nvm, which bootstrap leaves off the PATH of the
|
|
||||||
# shell that called it. nvm is a bash script, hence the subshell.
|
|
||||||
run_yarn() {
|
|
||||||
if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then
|
|
||||||
exec yarn "$@"
|
|
||||||
fi
|
|
||||||
if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
|
||||||
echo "fmt-check: no yarn; run script/bootstrap first" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
|
|
||||||
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
unformatted="$(gofmt -l .)"
|
unformatted="$(gofmt -l .)"
|
||||||
@@ -42,7 +13,6 @@ main() {
|
|||||||
echo "$unformatted" >&2
|
echo "$unformatted" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
run_yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always
|
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
+5
-10
@@ -5,14 +5,9 @@
|
|||||||
# make its answer untrustworthy.
|
# make its answer untrustworthy.
|
||||||
#
|
#
|
||||||
# The phase is not the last stage in the file, so it is built only when
|
# The phase is not the last stage in the file, so it is built only when
|
||||||
# --target names it. A new CACHEBUST value reruns every RUN step after
|
# --target names it. --no-cache because a cached lint layer is a lint
|
||||||
# the ARG CACHEBUST line, since a cached lint layer is a lint that did
|
# that did not run. The tag makes each build replace the previous image
|
||||||
# not run; unlike --no-cache, it keeps the layers above that line and
|
# instead of leaving a dangling one behind.
|
||||||
# the cache mounts. This relies on every stage of the Dockerfile
|
|
||||||
# declaring ARG CACHEBUST before every step that lints, tests or
|
|
||||||
# compiles; a stage without that line is served from cache on an
|
|
||||||
# unchanged tree, with no warning.
|
|
||||||
# --output type=cacheonly writes no image, since nothing uses one.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -20,9 +15,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
docker build --no-cache \
|
||||||
--target lint \
|
--target lint \
|
||||||
--output type=cacheonly .
|
-t "$("$SCRIPT_DIR/projectname")-lint" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
+4
-9
@@ -2,13 +2,8 @@
|
|||||||
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
||||||
# and this builds that phase alone, on the same terms as script/lint:
|
# and this builds that phase alone, on the same terms as script/lint:
|
||||||
# --target because a phase that is not the last stage is built only when
|
# --target because a phase that is not the last stage is built only when
|
||||||
# named, and a new CACHEBUST value because a cached test layer is a test
|
# named, --no-cache because a cached test layer is a test that did not
|
||||||
# that did not run; unlike --no-cache, it keeps the layers above the ARG
|
# run, and a tag so each build replaces the previous image.
|
||||||
# CACHEBUST line and the cache mounts. This relies on every stage of the
|
|
||||||
# Dockerfile declaring ARG CACHEBUST before every step that lints, tests
|
|
||||||
# or compiles; a stage without that line is served from cache on an
|
|
||||||
# unchanged tree, with no warning. --output type=cacheonly writes no
|
|
||||||
# image, since nothing uses one.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -16,9 +11,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
docker build --no-cache \
|
||||||
--target test \
|
--target test \
|
||||||
--output type=cacheonly .
|
-t "$("$SCRIPT_DIR/projectname")-test" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
@@ -1,8 +0,0 @@
|
|||||||
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
|
|
||||||
# yarn lockfile v1
|
|
||||||
|
|
||||||
|
|
||||||
prettier@3.8.1:
|
|
||||||
version "3.8.1"
|
|
||||||
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
|
|
||||||
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==
|
|
||||||
Reference in new issue
Block a user