Keep the AS list when the asupdate generator fails (closes #173)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
make asupdate piped the generator into gzip, so the target saw gzip's exit status. A failed download replaced pkg/asinfo/asdata.json.gz with an empty file and the target still exited 0. The target now runs the new script/asupdate. It writes the output of cmd/asinfo-gen to a temporary file. Only when the generator succeeds does it compress the list and move it into place. Otherwise it leaves the file alone and exits non-zero. The generator now fails when as.csv holds no AS. The generator reads as.csv from ipverse/as-metadata, the source's new name. The list it builds there is unchanged, so no new list is committed. Model: opus-5-5
This commit was merged in pull request #176.
This commit is contained in:
@@ -52,7 +52,6 @@ clean:
|
|||||||
run: build
|
run: build
|
||||||
DEBUG=routewatch ./bin/routewatch 2>&1 | tee log.txt
|
DEBUG=routewatch ./bin/routewatch 2>&1 | tee log.txt
|
||||||
|
|
||||||
|
# Rebuild the AS list built into RouteWatch from as.csv (needs the network).
|
||||||
asupdate:
|
asupdate:
|
||||||
@echo "Updating AS info data..."
|
@script/asupdate
|
||||||
@go run cmd/asinfo-gen/main.go | gzip > pkg/asinfo/asdata.json.gz.tmp && \
|
|
||||||
mv pkg/asinfo/asdata.json.gz.tmp pkg/asinfo/asdata.json.gz
|
|
||||||
|
|||||||
@@ -110,9 +110,10 @@ Every route answers `HEAD` as it answers `GET`, without the body.
|
|||||||
`handle` and `description`
|
`handle` and `description`
|
||||||
|
|
||||||
The search reads the AS list built into RouteWatch, not the database. The
|
The search reads the AS list built into RouteWatch, not the database. The
|
||||||
list comes from [ipverse/asn-info](https://github.com/ipverse/asn-info). A
|
list comes from
|
||||||
match can therefore be an AS that the database has no record of. A `q` that
|
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). A match can
|
||||||
is missing, empty or only spaces answers 400.
|
therefore be an AS that the database has no record of. A `q` that is
|
||||||
|
missing, empty or only spaces answers 400.
|
||||||
|
|
||||||
A JSON answer is `{"status": "ok", "data": ...}`, or for an error
|
A JSON answer is `{"status": "ok", "data": ...}`, or for an error
|
||||||
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
|
`{"status": "error", "error": {"msg": ..., "code": ...}}`, and carries an
|
||||||
@@ -485,6 +486,11 @@ provide:
|
|||||||
or `go.sum`) and `go fmt` (writes), then `script/check`
|
or `go.sum`) and `go fmt` (writes), then `script/check`
|
||||||
- `script/install-precommit` — install the git pre-commit hook that runs
|
- `script/install-precommit` — install the git pre-commit hook that runs
|
||||||
`script/precommit`
|
`script/precommit`
|
||||||
|
- `script/asupdate` — rebuild the AS list built into RouteWatch,
|
||||||
|
`pkg/asinfo/asdata.json.gz`, from the `as.csv` file of
|
||||||
|
[ipverse/as-metadata](https://github.com/ipverse/as-metadata). It needs the
|
||||||
|
network. It runs `cmd/asinfo-gen`, which fails when it finds no AS. When
|
||||||
|
`cmd/asinfo-gen` fails, the old list stays and the script exits non-zero.
|
||||||
|
|
||||||
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
|
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
|
||||||
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
|
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
|
||||||
|
|||||||
@@ -33,6 +33,11 @@ tested.
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-27: `make asupdate` runs the new `script/asupdate`. It replaces the AS
|
||||||
|
list built into RouteWatch only when the generator succeeds and finds at least
|
||||||
|
one AS. A failed download used to replace the list with an empty file. The
|
||||||
|
generator reads `as.csv` from `ipverse/as-metadata`, the source's new name
|
||||||
|
(closes #173)
|
||||||
- 2026-10-27: the WHOIS details of an AS at RIPE, APNIC or AFRINIC come only
|
- 2026-10-27: the WHOIS details of an AS at RIPE, APNIC or AFRINIC come only
|
||||||
from its `aut-num` object and the organisation object it names. RIPE and APNIC
|
from its `aut-num` object and the organisation object it names. RIPE and APNIC
|
||||||
start their answer with the `as-block` object of the AS's block. Its dates
|
start their answer with the `as-block` object of the AS's block. Its dates
|
||||||
|
|||||||
+10
-3
@@ -15,7 +15,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
const asCsvURL = "https://github.com/ipverse/asn-info/raw/refs/heads/master/as.csv"
|
const asCsvURL = "https://github.com/ipverse/as-metadata/raw/refs/heads/master/as.csv"
|
||||||
|
|
||||||
// The first columns of as.csv, the ones the generator reads. It reads past
|
// The first columns of as.csv, the ones the generator reads. It reads past
|
||||||
// the columns after them, such as the country code.
|
// the columns after them, such as the country code.
|
||||||
@@ -26,7 +26,10 @@ const (
|
|||||||
columnsRead = 3
|
columnsRead = 3
|
||||||
)
|
)
|
||||||
|
|
||||||
var errUnexpectedHeader = errors.New("unexpected CSV header")
|
var (
|
||||||
|
errUnexpectedHeader = errors.New("unexpected CSV header")
|
||||||
|
errNoASes = errors.New("no AS in the CSV")
|
||||||
|
)
|
||||||
|
|
||||||
// ASInfo represents information about an Autonomous System
|
// ASInfo represents information about an Autonomous System
|
||||||
type ASInfo struct {
|
type ASInfo struct {
|
||||||
@@ -82,7 +85,7 @@ func main() {
|
|||||||
// parseCSV reads as.csv from r and returns its ASes, with the handle and the
|
// parseCSV reads as.csv from r and returns its ASes, with the handle and the
|
||||||
// description as the file spells them. A header whose first columns are not
|
// description as the file spells them. A header whose first columns are not
|
||||||
// the AS number, the handle and the description is an error. A record that
|
// the AS number, the handle and the description is an error. A record that
|
||||||
// does not parse is logged and skipped.
|
// does not parse is logged and skipped. A file left with no AS is an error.
|
||||||
func parseCSV(r io.Reader) ([]ASInfo, error) {
|
func parseCSV(r io.Reader) ([]ASInfo, error) {
|
||||||
reader := csv.NewReader(r)
|
reader := csv.NewReader(r)
|
||||||
|
|
||||||
@@ -126,5 +129,9 @@ func parseCSV(r io.Reader) ([]ASInfo, error) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if len(asInfos) == 0 {
|
||||||
|
return nil, errNoASes
|
||||||
|
}
|
||||||
|
|
||||||
return asInfos, nil
|
return asInfos, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"errors"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -53,3 +54,20 @@ func TestParseCSVRejectsAnotherHeader(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestParseCSVRejectsAFileWithNoASes parses files that hold no AS, one with
|
||||||
|
// only the header and one whose only record has no valid AS number. It
|
||||||
|
// expects errNoASes, so that make asupdate keeps the old list.
|
||||||
|
func TestParseCSVRejectsAFileWithNoASes(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, asCSV := range []string{
|
||||||
|
"asn,handle,description,country-code\n",
|
||||||
|
"asn,handle,description,country-code\nAS1,LVLT-1,Level 3,US\n",
|
||||||
|
} {
|
||||||
|
_, err := parseCSV(strings.NewReader(asCSV))
|
||||||
|
if !errors.Is(err, errNoASes) {
|
||||||
|
t.Errorf("parseCSV(%q) = %v, want %v", asCSV, err, errNoASes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+1
-1
@@ -2,7 +2,7 @@
|
|||||||
Package asinfo provides information about Autonomous Systems (AS).
|
Package asinfo provides information about Autonomous Systems (AS).
|
||||||
|
|
||||||
The package embeds a comprehensive database of AS numbers, handles, and descriptions
|
The package embeds a comprehensive database of AS numbers, handles, and descriptions
|
||||||
sourced from https://github.com/ipverse/asn-info. The data is embedded at compile
|
sourced from https://github.com/ipverse/as-metadata. The data is embedded at compile
|
||||||
time and provides fast, offline lookups.
|
time and provides fast, offline lookups.
|
||||||
|
|
||||||
Basic usage:
|
Basic usage:
|
||||||
|
|||||||
Executable
+34
@@ -0,0 +1,34 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# script/asupdate: rebuild the AS list built into RouteWatch from
|
||||||
|
# as.csv with cmd/asinfo-gen, which fails when it finds no AS. The new
|
||||||
|
# list replaces the old one only when the generator succeeds. Otherwise
|
||||||
|
# the old list stays as it was and the script exits non-zero.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
|
AS_DATA="pkg/asinfo/asdata.json.gz"
|
||||||
|
# Next to AS_DATA, so the final mv is a rename on one file system.
|
||||||
|
AS_DATA_TMP="$AS_DATA.tmp"
|
||||||
|
|
||||||
|
main() {
|
||||||
|
cd "$ROOT"
|
||||||
|
|
||||||
|
json="$(mktemp)"
|
||||||
|
trap 'rm -f "$json" "$AS_DATA_TMP"' EXIT
|
||||||
|
# A signal exits through the EXIT trap too.
|
||||||
|
trap 'exit 1' HUP INT TERM
|
||||||
|
|
||||||
|
echo "Updating AS info data..."
|
||||||
|
if ! go run ./cmd/asinfo-gen >"$json"; then
|
||||||
|
echo "asupdate: the generator failed; $AS_DATA is unchanged" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# -n leaves the time out of the file, so an unchanged list gives the
|
||||||
|
# same bytes.
|
||||||
|
gzip -n -c <"$json" >"$AS_DATA_TMP"
|
||||||
|
mv "$AS_DATA_TMP" "$AS_DATA"
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
Reference in New Issue
Block a user