sneak/hnblogs
1
0
Fork 1
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: sneak:d5ed473e315df7699308f0319ff5e7e7c16c25cd
Branches Tags
sneak:main
..
compare: sneak:main
Branches Tags
sneak:main

2 Commits
d5ed473e31 ... main

Author SHA1 Message Date
Jeffrey Paul
477ff660c2 Merge pull request 'embed blogs.json instead of fetching it at runtime' (#2) from clawbot/hnblogs:issue-1-embed-blogs-json into main 
Reviewed-on: #2
 2026-09-05 06:41:55 +02:00
sneak
7e1b010c4f embed blogs.json instead of fetching it at runtime (closes #1) 
The dataset is vendored as blogs.json and compiled in with go:embed, so
the package does no network I/O on any code path.

FetchBlogs is removed rather than kept as a no-op wrapper, since nothing
is fetched any more: GetBlogs decodes the embedded bytes on first call
and memoizes the result, and RandomBlog, RandomBlogs and NthBlog go
through it. Callers of FetchBlogs have to switch to GetBlogs.

make update-data refreshes the vendored copy from the BlogsURL constant,
downloading to a temporary file and replacing blogs.json only once that
file parses as a non-empty JSON array of blog entries, so a
complete-but-wrong response cannot overwrite the good dataset. A test
walks the dependency graph of the non-test build and fails if any net/*
package is reachable.

Model: opus-5
 2026-09-05 04:37:58 +00:00
4 changed files with 14 additions and 32 deletions
Expand all files Collapse all files

2
Makefile
View File

@@ -31,7 +31,7 @@ update-data:
@command -v jq >/dev/null || { echo "update-data requires jq" >&2; exit 1; }
@url=$$(sed -n 's/^const BlogsURL = "\(.*\)"$$/\1/p' hnblogs.go); \
test -n "$$url" || { echo "could not parse BlogsURL from hnblogs.go" >&2; exit 1; }; \
echo "fetching $$url"; \
echo "downloading $$url"; \
curl -fsSL "$$url" -o blogs.json.tmp
@jq -e 'type == "array" and length > 0 and all(.[]; type == "object" and has("url"))' \
blogs.json.tmp >/dev/null 2>&1 \

17
README.md
View File

@@ -16,25 +16,14 @@ the package with `go:embed`. The library performs no network I/O: importing it
does not reach out to anything, results do not change under a caller between
runs of the same build, and `go test` works offline.
`FetchBlogs` keeps its name and its `sync.Once` memoization, but on first call
it decodes the embedded bytes rather than issuing an HTTP request. Its error
return is now only reachable if the committed `blogs.json` is malformed.
`GetBlogs` decodes the embedded bytes on first call and memoizes the result;
every other accessor goes through it. Its error return is only reachable if the
committed `blogs.json` is malformed.
The trade-off is that the dataset is a build-time artifact: it is roughly 8 MB
of JSON, it lands in every binary that links the package, and it is only as
fresh as the last commit that refreshed it.
### Where blogs.json came from
`blogs.json` is not this project's work. It is an unmodified copy of
<https://raw.githubusercontent.com/surprisetalk/blogs.hn/main/blogs.json>
from the [surprisetalk/blogs.hn](https://github.com/surprisetalk/blogs.hn)
repository, which publishes no licence. This repository's `LICENSE` covers the
code here and does not extend to `blogs.json`, and a binary that links this
package redistributes that file too.
## Refreshing the dataset
```sh

23
hnblogs.go
View File

@@ -14,13 +14,11 @@ import (
"sync"
)
// BlogsURL is the upstream source of blogs.json. It is not fetched at runtime;
// it documents where "make update-data" pulls the vendored copy from.
// BlogsURL is the upstream source of blogs.json. Nothing reads it at runtime;
// it documents where "make update-data" downloads the vendored copy from.
const BlogsURL = "https://raw.githubusercontent.com/surprisetalk/blogs.hn/main/blogs.json"
// blogsJSON is the vendored dataset, refreshed by "make update-data". It is an
// unmodified copy of a third party's file, not this project's work; see "Where
// blogs.json came from" in README.md.
// blogsJSON is the vendored dataset, refreshed by "make update-data".
//
//go:embed blogs.json
var blogsJSON []byte
@@ -41,13 +39,13 @@ type Blog struct {
Desc string `json:"desc"`
}
// FetchBlogs returns the embedded list of blogs, decoding it on first call and
// GetBlogs returns the embedded list of blogs, decoding it on first call and
// memoizing the result for subsequent calls.
//
// Despite the name it performs no I/O: the data is compiled into the binary, so
// the only error it can return is a malformed embedded blogs.json, which would
// mean the committed dataset is broken.
func FetchBlogs() ([]Blog, error) {
// It performs no I/O: the data is compiled into the binary, so the only error
// it can return is a malformed embedded blogs.json, which would mean the
// committed dataset is broken.
func GetBlogs() ([]Blog, error) {
once.Do(func() {
var decoded []Blog
if err := json.Unmarshal(blogsJSON, &decoded); err != nil {
@@ -61,11 +59,6 @@ func FetchBlogs() ([]Blog, error) {
return blogs, loadError
}
// GetBlogs returns the memoized list of blogs.
func GetBlogs() ([]Blog, error) {
return FetchBlogs()
}
// RandomBlog returns a random blog from the list of blogs.
func RandomBlog() (Blog, error) {
blogs, err := GetBlogs()

4
hnblogs_test.go
View File

@@ -8,8 +8,8 @@ import (
"testing"
)
func TestFetchBlogs(t *testing.T) {
blogs, err := FetchBlogs()
func TestGetBlogs(t *testing.T) {
blogs, err := GetBlogs()
if err != nil {
t.Fatalf("Expected no error, got %v", err)
}