sneak/hnblogs
1
0
Fork 1
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

vendor blogs.json into the repo and embed it with go:embed instead of fetching at runtime #1

New Issue
Closed
opened 2026-09-05 05:02:33 +02:00 by clawbot · 1 comment
clawbot commented 2026-09-05 05:02:33 +02:00
Collaborator
Copy Link
Copy Source

Requested by sneak in chat, 2026-09-05.

FetchBlogs does an HTTP GET of BlogsURL (https://raw.githubusercontent.com/surprisetalk/blogs.hn/main/blogs.json) on first use, so the library needs network access at runtime and its results change under the caller.

Definition of done:

  • blogs.json is fetched once and committed to the repo, and loaded with go:embed; no HTTP at runtime.
  • The public API (FetchBlogs, GetBlogs, RandomBlog, RandomBlogs, Blog) keeps working; FetchBlogs becomes a parse of the embedded data (memoized as today) and cannot fail on network.
  • A make target (e.g. make update-data) refreshes the vendored file from BlogsURL, so the dataset can be bumped by a commit.
  • Tests do not touch the network.
  • README/doc comments say the data is embedded and how to refresh it.

Model: fable-5-1

Requested by sneak in chat, 2026-09-05. `FetchBlogs` does an HTTP GET of `BlogsURL` (https://raw.githubusercontent.com/surprisetalk/blogs.hn/main/blogs.json) on first use, so the library needs network access at runtime and its results change under the caller. Definition of done: - `blogs.json` is fetched once and committed to the repo, and loaded with `go:embed`; no HTTP at runtime. - The public API (`FetchBlogs`, `GetBlogs`, `RandomBlog`, `RandomBlogs`, `Blog`) keeps working; `FetchBlogs` becomes a parse of the embedded data (memoized as today) and cannot fail on network. - A `make` target (e.g. `make update-data`) refreshes the vendored file from `BlogsURL`, so the dataset can be bumped by a commit. - Tests do not touch the network. - `README`/doc comments say the data is embedded and how to refresh it. Model: fable-5-1
clawbot commented 2026-09-05 05:10:13 +02:00
Author
Collaborator
Copy Link
Copy Source

Done in #2

blogs.json is vendored and loaded with go:embed; net/http is gone from the package and a test fails on any net/net/* import returning to non-test code. FetchBlogs keeps its signature and sync.Once memoization, decoding the embedded bytes instead of fetching, and can now only error on a malformed committed dataset. make update-data refreshes the file, reading the URL from the BlogsURL constant so it has one definition, replacing blogs.json only on a complete download, then running the tests. Added a README.md, which the repo did not have.

make test, make lint and make docker pass; the Docker test stage was also run with --network=none to confirm the suite needs no network.

Two things needing your call: the dataset is committed verbatim (~8 MB, mostly per-blog post history the Blog struct ignores) rather than trimmed to the six fields it reads (~1.5 MB), because the issue asked for a refresh from BlogsURL rather than a transform of it. And the PR could not be labelled needs-review -- the repo defines no labels and clawbot has pull-only access, so it is opened from a fork.

Model: opus-5

Done in https://git.eeqj.de/sneak/hnblogs/pulls/2 `blogs.json` is vendored and loaded with `go:embed`; `net/http` is gone from the package and a test fails on any `net`/`net/*` import returning to non-test code. `FetchBlogs` keeps its signature and `sync.Once` memoization, decoding the embedded bytes instead of fetching, and can now only error on a malformed committed dataset. `make update-data` refreshes the file, reading the URL from the `BlogsURL` constant so it has one definition, replacing `blogs.json` only on a complete download, then running the tests. Added a `README.md`, which the repo did not have. `make test`, `make lint` and `make docker` pass; the Docker test stage was also run with `--network=none` to confirm the suite needs no network. Two things needing your call: the dataset is committed verbatim (~8 MB, mostly per-blog post history the `Blog` struct ignores) rather than trimmed to the six fields it reads (~1.5 MB), because the issue asked for a refresh from `BlogsURL` rather than a transform of it. And the PR could not be labelled `needs-review` -- the repo defines no labels and `clawbot` has pull-only access, so it is opened from a fork. Model: opus-5
sneak closed this issue 2026-09-05 06:41:58 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/hnblogs#1