embed blogs.json instead of fetching it at runtime (closes #1)
The library needed network access on first use and its results changed under the caller between runs. blogs.json is now vendored and compiled in with go:embed, so the dataset is fixed for a given build. FetchBlogs keeps its name, signature and sync.Once memoization but now decodes the embedded bytes; its error return is only reachable if the committed blogs.json is malformed. net/http is gone from the package, and a test asserts that no net/* package appears anywhere in the dependency graph of the non-test build, not only in its direct imports. make update-data refreshes the vendored file, reading the upstream location from the BlogsURL constant so the URL has one definition. It downloads to a temporary file and replaces blogs.json only once that file parses as a non-empty JSON array of blog entries, so neither a truncated transfer nor a complete-but-wrong response such as an error page can overwrite the good dataset. It then runs the test suite against the new data. That target now needs jq as well as curl. blogs.json is an unmodified copy of a third party's file, redistributed here and in every binary that links the package, and the upstream repository publishes no licence. The README and the embed doc comment now record where it came from and that this repository's LICENSE does not extend to it. Whether that arrangement is acceptable is the owner's call. The dataset is committed verbatim as upstream serves it, which is ~8 MB of JSON in the repo and in every linking binary; most of that is per-blog post history that the Blog struct does not expose. Model: opus-5
This commit is contained in:
60
README.md
Normal file
60
README.md
Normal file
@@ -0,0 +1,60 @@
|
||||
# hnblogs
|
||||
|
||||
A Go library for the [blogs.hn](https://blogs.hn) dataset: a list of personal
|
||||
blogs collected from Hacker News.
|
||||
|
||||
```go
|
||||
import "sneak.berlin/go/hnblogs"
|
||||
|
||||
blog, err := hnblogs.RandomBlog()
|
||||
```
|
||||
|
||||
## Embedded data
|
||||
|
||||
The dataset is vendored into this repository as `blogs.json` and compiled into
|
||||
the package with `go:embed`. The library performs no network I/O: importing it
|
||||
does not reach out to anything, results do not change under a caller between
|
||||
runs of the same build, and `go test` works offline.
|
||||
|
||||
`FetchBlogs` keeps its name and its `sync.Once` memoization, but on first call
|
||||
it decodes the embedded bytes rather than issuing an HTTP request. Its error
|
||||
return is now only reachable if the committed `blogs.json` is malformed.
|
||||
|
||||
The trade-off is that the dataset is a build-time artifact: it is roughly 8 MB
|
||||
of JSON, it lands in every binary that links the package, and it is only as
|
||||
fresh as the last commit that refreshed it.
|
||||
|
||||
### Where blogs.json came from
|
||||
|
||||
`blogs.json` is not this project's work. It is an unmodified copy of
|
||||
|
||||
<https://raw.githubusercontent.com/surprisetalk/blogs.hn/main/blogs.json>
|
||||
|
||||
from the [surprisetalk/blogs.hn](https://github.com/surprisetalk/blogs.hn)
|
||||
repository, which publishes no licence. This repository's `LICENSE` covers the
|
||||
code here and does not extend to `blogs.json`, and a binary that links this
|
||||
package redistributes that file too.
|
||||
|
||||
## Refreshing the dataset
|
||||
|
||||
```sh
|
||||
make update-data
|
||||
```
|
||||
|
||||
That target reads the upstream location from the `BlogsURL` constant in
|
||||
`hnblogs.go` — the single source of truth — and downloads it to a temporary
|
||||
file. It replaces `blogs.json` only once that file parses as a non-empty JSON
|
||||
array of blog entries, so a response that arrives complete but is not the
|
||||
dataset leaves the vendored copy untouched. It then runs the test suite
|
||||
against the new data. Requires `curl` and `jq`. Commit the resulting
|
||||
`blogs.json` to publish the update.
|
||||
|
||||
## Development
|
||||
|
||||
```sh
|
||||
make test
|
||||
make lint
|
||||
make docker
|
||||
```
|
||||
|
||||
`make docker` runs lint and tests in containers.
|
||||
Reference in New Issue
Block a user