vendor blogs.json into the repo and embed it with go:embed instead of fetching at runtime #1

New Issue

clawbot · 2026-09-05T05:02:33+02:00

clawbot commented

2026-09-05 05:02:33 +02:00

Requested by sneak in chat, 2026-09-05.

FetchBlogs does an HTTP GET of BlogsURL (https://raw.githubusercontent.com/surprisetalk/blogs.hn/main/blogs.json) on first use, so the library needs network access at runtime and its results change under the caller.

Definition of done:

blogs.json is fetched once and committed to the repo, and loaded with go:embed; no HTTP at runtime.
The public API (FetchBlogs, GetBlogs, RandomBlog, RandomBlogs, Blog) keeps working; FetchBlogs becomes a parse of the embedded data (memoized as today) and cannot fail on network.
A make target (e.g. make update-data) refreshes the vendored file from BlogsURL, so the dataset can be bumped by a commit.
Tests do not touch the network.
README/doc comments say the data is embedded and how to refresh it.

Model: fable-5-1

Requested by sneak in chat, 2026-09-05. `FetchBlogs` does an HTTP GET of `BlogsURL` (https://raw.githubusercontent.com/surprisetalk/blogs.hn/main/blogs.json) on first use, so the library needs network access at runtime and its results change under the caller. Definition of done: - `blogs.json` is fetched once and committed to the repo, and loaded with `go:embed`; no HTTP at runtime. - The public API (`FetchBlogs`, `GetBlogs`, `RandomBlog`, `RandomBlogs`, `Blog`) keeps working; `FetchBlogs` becomes a parse of the embedded data (memoized as today) and cannot fail on network. - A `make` target (e.g. `make update-data`) refreshes the vendored file from `BlogsURL`, so the dataset can be bumped by a commit. - Tests do not touch the network. - `README`/doc comments say the data is embedded and how to refresh it. Model: fable-5-1

clawbot referenced this issue

2026-09-05 05:09:44 +02:00

embed blogs.json instead of fetching it at runtime #2

clawbot commented

2026-09-05 05:10:13 +02:00

Done in #2

blogs.json is vendored and loaded with go:embed; net/http is gone from the package and a test fails on any net/net/* import returning to non-test code. FetchBlogs keeps its signature and sync.Once memoization, decoding the embedded bytes instead of fetching, and can now only error on a malformed committed dataset. make update-data refreshes the file, reading the URL from the BlogsURL constant so it has one definition, replacing blogs.json only on a complete download, then running the tests. Added a README.md, which the repo did not have.

make test, make lint and make docker pass; the Docker test stage was also run with --network=none to confirm the suite needs no network.

Two things needing your call: the dataset is committed verbatim (~8 MB, mostly per-blog post history the Blog struct ignores) rather than trimmed to the six fields it reads (~1.5 MB), because the issue asked for a refresh from BlogsURL rather than a transform of it. And the PR could not be labelled needs-review -- the repo defines no labels and clawbot has pull-only access, so it is opened from a fork.

Model: opus-5

Done in https://git.eeqj.de/sneak/hnblogs/pulls/2 `blogs.json` is vendored and loaded with `go:embed`; `net/http` is gone from the package and a test fails on any `net`/`net/*` import returning to non-test code. `FetchBlogs` keeps its signature and `sync.Once` memoization, decoding the embedded bytes instead of fetching, and can now only error on a malformed committed dataset. `make update-data` refreshes the file, reading the URL from the `BlogsURL` constant so it has one definition, replacing `blogs.json` only on a complete download, then running the tests. Added a `README.md`, which the repo did not have. `make test`, `make lint` and `make docker` pass; the Docker test stage was also run with `--network=none` to confirm the suite needs no network. Two things needing your call: the dataset is committed verbatim (~8 MB, mostly per-blog post history the `Blog` struct ignores) rather than trimmed to the six fields it reads (~1.5 MB), because the issue asked for a refresh from `BlogsURL` rather than a transform of it. And the PR could not be labelled `needs-review` -- the repo defines no labels and `clawbot` has pull-only access, so it is opened from a fork. Model: opus-5

clawbot referenced this issue

2026-09-05 05:20:26 +02:00

make on a fresh clone fails: the default target builds cmd/example, which does not exist #3

clawbot referenced this issue

2026-09-05 05:25:50 +02:00

embed blogs.json instead of fetching it at runtime #2

clawbot referenced this issue

2026-09-05 06:09:25 +02:00

embed blogs.json instead of fetching it at runtime #2

sneak referenced this issue from a commit

2026-09-05 06:41:58 +02:00

embed blogs.json instead of fetching it at runtime (closes #1)