sneak/hnblogs
1
0
Fork 1
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: sneak:d5ed473e315df7699308f0319ff5e7e7c16c25cd
Branches Tags
sneak:main
..
compare: sneak:main
Branches Tags
sneak:main

2 Commits
d5ed473e31 ... main

Author SHA1 Message Date
Jeffrey Paul
477ff660c2 Merge pull request 'embed blogs.json instead of fetching it at runtime' (#2) from clawbot/hnblogs:issue-1-embed-blogs-json into main 
Reviewed-on: #2
 2026-09-05 06:41:55 +02:00
sneak
7e1b010c4f embed blogs.json instead of fetching it at runtime (closes #1) 
The dataset is vendored as blogs.json and compiled in with go:embed, so
the package does no network I/O on any code path.

FetchBlogs is removed rather than kept as a no-op wrapper, since nothing
is fetched any more: GetBlogs decodes the embedded bytes on first call
and memoizes the result, and RandomBlog, RandomBlogs and NthBlog go
through it. Callers of FetchBlogs have to switch to GetBlogs.

make update-data refreshes the vendored copy from the BlogsURL constant,
downloading to a temporary file and replacing blogs.json only once that
file parses as a non-empty JSON array of blog entries, so a
complete-but-wrong response cannot overwrite the good dataset. A test
walks the dependency graph of the non-test build and fails if any net/*
package is reachable.

Model: opus-5
 2026-09-05 04:37:58 +00:00
4 changed files with 14 additions and 32 deletions
Expand all files Collapse all files

2
Makefile
View File

@@ -31,7 +31,7 @@ update-data:
@command -v jq >/dev/null || { echo "update-data requires jq" >&2; exit 1; } @command -v jq >/dev/null || { echo "update-data requires jq" >&2; exit 1; }
@url=$$(sed -n 's/^const BlogsURL = "\(.*\)"$$/\1/p' hnblogs.go); \ @url=$$(sed -n 's/^const BlogsURL = "\(.*\)"$$/\1/p' hnblogs.go); \
test -n "$$url" || { echo "could not parse BlogsURL from hnblogs.go" >&2; exit 1; }; \ test -n "$$url" || { echo "could not parse BlogsURL from hnblogs.go" >&2; exit 1; }; \
echo "fetching $$url"; \ echo "downloading $$url"; \
curl -fsSL "$$url" -o blogs.json.tmp curl -fsSL "$$url" -o blogs.json.tmp
@jq -e 'type == "array" and length > 0 and all(.[]; type == "object" and has("url"))' \ @jq -e 'type == "array" and length > 0 and all(.[]; type == "object" and has("url"))' \
blogs.json.tmp >/dev/null 2>&1 \ blogs.json.tmp >/dev/null 2>&1 \

17
README.md
View File

@@ -16,25 +16,14 @@ the package with `go:embed`. The library performs no network I/O: importing it
does not reach out to anything, results do not change under a caller between does not reach out to anything, results do not change under a caller between
runs of the same build, and `go test` works offline. runs of the same build, and `go test` works offline.
`FetchBlogs` keeps its name and its `sync.Once` memoization, but on first call `GetBlogs` decodes the embedded bytes on first call and memoizes the result;
it decodes the embedded bytes rather than issuing an HTTP request. Its error every other accessor goes through it. Its error return is only reachable if the
return is now only reachable if the committed `blogs.json` is malformed. committed `blogs.json` is malformed.
The trade-off is that the dataset is a build-time artifact: it is roughly 8 MB The trade-off is that the dataset is a build-time artifact: it is roughly 8 MB
of JSON, it lands in every binary that links the package, and it is only as of JSON, it lands in every binary that links the package, and it is only as
fresh as the last commit that refreshed it. fresh as the last commit that refreshed it.
### Where blogs.json came from
`blogs.json` is not this project's work. It is an unmodified copy of
<https://raw.githubusercontent.com/surprisetalk/blogs.hn/main/blogs.json>
from the [surprisetalk/blogs.hn](https://github.com/surprisetalk/blogs.hn)
repository, which publishes no licence. This repository's `LICENSE` covers the
code here and does not extend to `blogs.json`, and a binary that links this
package redistributes that file too.
## Refreshing the dataset ## Refreshing the dataset
```sh ```sh

23
hnblogs.go
View File

@@ -14,13 +14,11 @@ import (
"sync" "sync"
) )
// BlogsURL is the upstream source of blogs.json. It is not fetched at runtime; // BlogsURL is the upstream source of blogs.json. Nothing reads it at runtime;
// it documents where "make update-data" pulls the vendored copy from. // it documents where "make update-data" downloads the vendored copy from.
const BlogsURL = "https://raw.githubusercontent.com/surprisetalk/blogs.hn/main/blogs.json" const BlogsURL = "https://raw.githubusercontent.com/surprisetalk/blogs.hn/main/blogs.json"
// blogsJSON is the vendored dataset, refreshed by "make update-data". It is an // blogsJSON is the vendored dataset, refreshed by "make update-data".
// unmodified copy of a third party's file, not this project's work; see "Where
// blogs.json came from" in README.md.
// //
//go:embed blogs.json //go:embed blogs.json
var blogsJSON []byte var blogsJSON []byte
@@ -41,13 +39,13 @@ type Blog struct {
Desc string `json:"desc"` Desc string `json:"desc"`
} }
// FetchBlogs returns the embedded list of blogs, decoding it on first call and // GetBlogs returns the embedded list of blogs, decoding it on first call and
// memoizing the result for subsequent calls. // memoizing the result for subsequent calls.
// //
// Despite the name it performs no I/O: the data is compiled into the binary, so // It performs no I/O: the data is compiled into the binary, so the only error
// the only error it can return is a malformed embedded blogs.json, which would // it can return is a malformed embedded blogs.json, which would mean the
// mean the committed dataset is broken. // committed dataset is broken.
func FetchBlogs() ([]Blog, error) { func GetBlogs() ([]Blog, error) {
once.Do(func() { once.Do(func() {
var decoded []Blog var decoded []Blog
if err := json.Unmarshal(blogsJSON, &decoded); err != nil { if err := json.Unmarshal(blogsJSON, &decoded); err != nil {
@@ -61,11 +59,6 @@ func FetchBlogs() ([]Blog, error) {
return blogs, loadError return blogs, loadError
} }
// GetBlogs returns the memoized list of blogs.
func GetBlogs() ([]Blog, error) {
return FetchBlogs()
}
// RandomBlog returns a random blog from the list of blogs. // RandomBlog returns a random blog from the list of blogs.
func RandomBlog() (Blog, error) { func RandomBlog() (Blog, error) {
blogs, err := GetBlogs() blogs, err := GetBlogs()

4
hnblogs_test.go
View File

@@ -8,8 +8,8 @@ import (
"testing" "testing"
) )
func TestFetchBlogs(t *testing.T) { func TestGetBlogs(t *testing.T) {
blogs, err := FetchBlogs() blogs, err := GetBlogs()
if err != nil { if err != nil {
t.Fatalf("Expected no error, got %v", err) t.Fatalf("Expected no error, got %v", err)
} }