Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9b0186c481 | ||
|
|
8125d4aebf | ||
|
|
47e48a2529 | ||
|
|
17b6b84322 |
@@ -0,0 +1,2 @@
|
|||||||
|
node_modules/
|
||||||
|
yarn.lock
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
{
|
||||||
|
"tabWidth": 4,
|
||||||
|
"proseWrap": "always"
|
||||||
|
}
|
||||||
@@ -1,28 +1,26 @@
|
|||||||
# bsdaily
|
# bsdaily
|
||||||
|
|
||||||
[bsdaily](https://git.eeqj.de/sneak/bsdaily) is a command-line utility
|
[bsdaily](https://git.eeqj.de/sneak/bsdaily) is a command-line utility written
|
||||||
written in [Go](https://golang.org) that carves a single day (or a range of
|
in [Go](https://golang.org) that carves a single day (or a range of days) of
|
||||||
days) of [Bluesky](https://bsky.app) firehose data out of a large,
|
[Bluesky](https://bsky.app) firehose data out of a large, continuously-growing
|
||||||
continuously-growing SQLite database and writes it out as a self-contained,
|
SQLite database and writes it out as a self-contained,
|
||||||
[zstd](https://facebook.github.io/zstd/)-compressed SQL dump. The dumps are
|
[zstd](https://facebook.github.io/zstd/)-compressed SQL dump. The dumps are
|
||||||
named by date (e.g. `2026-06-27.sql.zst`), organized into per-month
|
named by date (e.g. `2026-06-27.sql.zst`), organized into per-month directories,
|
||||||
directories, and are designed to be published, archived, mirrored, and later
|
and are designed to be published, archived, mirrored, and later re-merged back
|
||||||
re-merged back into a single database.
|
into a single database.
|
||||||
|
|
||||||
The source database is read from a read-only [ZFS](https://openzfs.org)
|
The source database is read from a read-only [ZFS](https://openzfs.org)
|
||||||
snapshot, so extraction never contends with the live firehose ingester that
|
snapshot, so extraction never contends with the live firehose ingester that is
|
||||||
is writing to the original database. The tool is operationally
|
writing to the original database. The tool is operationally conservative: it
|
||||||
conservative: it checks free disk space before starting, copies the snapshot
|
checks free disk space before starting, copies the snapshot to fast scratch
|
||||||
to fast scratch storage, processes one day at a time to avoid SQLite lock
|
storage, processes one day at a time to avoid SQLite lock contention, verifies
|
||||||
contention, verifies every compressed output before publishing it, and
|
every compressed output before publishing it, and writes output atomically via a
|
||||||
writes output atomically via a temp-file-and-rename so a partial run never
|
temp-file-and-rename so a partial run never leaves a corrupt `.sql.zst` behind.
|
||||||
leaves a corrupt `.sql.zst` behind.
|
|
||||||
|
|
||||||
This project was written by [@sneak](https://sneak.berlin) to produce a
|
This project was written by [@sneak](https://sneak.berlin) to produce a daily,
|
||||||
daily, mergeable, publicly-mirrorable archive of the Bluesky firehose. It is
|
mergeable, publicly-mirrorable archive of the Bluesky firehose. It is currently
|
||||||
currently a one-person effort. The current version is pre-1.0 and there has
|
a one-person effort. The current version is pre-1.0 and there has not yet been a
|
||||||
not yet been a versioned release; [SemVer](https://semver.org) will be used
|
versioned release; [SemVer](https://semver.org) will be used for releases.
|
||||||
for releases.
|
|
||||||
|
|
||||||
# Build Status
|
# Build Status
|
||||||
|
|
||||||
@@ -33,14 +31,14 @@ branch must always be green.
|
|||||||
|
|
||||||
Primary development happens on a privately-run Gitea instance at
|
Primary development happens on a privately-run Gitea instance at
|
||||||
[https://git.eeqj.de/sneak/bsdaily](https://git.eeqj.de/sneak/bsdaily) and
|
[https://git.eeqj.de/sneak/bsdaily](https://git.eeqj.de/sneak/bsdaily) and
|
||||||
issues are [tracked
|
issues are [tracked there](https://git.eeqj.de/sneak/bsdaily/issues).
|
||||||
there](https://git.eeqj.de/sneak/bsdaily/issues).
|
|
||||||
|
|
||||||
Changes must always be formatted with a standard `go fmt`, syntactically
|
Changes must always be formatted with `make fmt` (`go fmt` for Go, prettier for
|
||||||
valid, and must pass the linting defined in the repository (presently the
|
Markdown), syntactically valid, and must pass the linting defined in the
|
||||||
`golangci-lint` defaults), which can be run with a `make lint`. The `main`
|
repository (presently the `golangci-lint` defaults), which can be run with a
|
||||||
branch is protected and all changes must be made via [pull
|
`make lint`. The `main` branch is protected and all changes must be made via
|
||||||
requests](https://git.eeqj.de/sneak/bsdaily/pulls) and pass CI to be merged.
|
[pull requests](https://git.eeqj.de/sneak/bsdaily/pulls) and pass CI to be
|
||||||
|
merged.
|
||||||
|
|
||||||
See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
|
See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
|
||||||
tooling requirements, and workflow conventions.
|
tooling requirements, and workflow conventions.
|
||||||
@@ -50,55 +48,55 @@ tooling requirements, and workflow conventions.
|
|||||||
This repository adheres to the
|
This repository adheres to the
|
||||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||||
standard: normalized scripts in `script/` are the entrypoints for the
|
standard: normalized scripts in `script/` are the entrypoints for the
|
||||||
development workflow, and the Makefile targets are thin shims that call
|
development workflow, and the Makefile targets are thin shims that call them. We
|
||||||
them. We provide:
|
provide:
|
||||||
|
|
||||||
- `script/bootstrap` — install all development dependencies (go, Go
|
- `script/bootstrap` — install all development dependencies (go, Go module
|
||||||
module download); the linter is not installed on the host
|
download, node and yarn at pinned versions when absent, and the prettier
|
||||||
|
pinned in `package.json` and `yarn.lock`); a node or yarn already installed is
|
||||||
|
used whatever its version; the linter is not installed on the host
|
||||||
- `script/setup` — make a fresh clone ready for development: runs
|
- `script/setup` — make a fresh clone ready for development: runs
|
||||||
`script/bootstrap`, then `script/install-precommit`
|
`script/bootstrap`, then `script/install-precommit`
|
||||||
- `script/projectname` — print the project name (used for the Docker
|
- `script/projectname` — print the project name (used for the Docker image tags)
|
||||||
image tags)
|
- `script/test` — build the Dockerfile's `test` phase, which runs the test suite
|
||||||
- `script/test` — build the Dockerfile's `test` phase, which runs the
|
with `-race` (verbose rerun on failure)
|
||||||
test suite with `-race` (verbose rerun on failure)
|
|
||||||
- `script/lint` — build the Dockerfile's `lint` phase, which runs
|
- `script/lint` — build the Dockerfile's `lint` phase, which runs
|
||||||
`golangci-lint run ./...`
|
`golangci-lint run ./...`
|
||||||
- `script/fmt` — format the Go code with `go fmt` (writes)
|
- `script/fmt` — format the Go code with `go fmt` and every Markdown file with
|
||||||
- `script/fmt-check` — check the Go formatting with `gofmt` (read-only)
|
prettier (writes)
|
||||||
- `script/check` — run `script/test`, `script/lint`, and
|
- `script/fmt-check` — check the Go formatting with `gofmt` and the Markdown
|
||||||
`script/fmt-check`
|
formatting with prettier (read-only)
|
||||||
- `script/docker` — build the Docker image tagged via
|
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
|
||||||
`script/projectname`; the build runs the `lint` and `test` phases
|
- `script/docker` — build the Docker image tagged via `script/projectname`; the
|
||||||
first
|
build runs the `lint` and `test` phases first
|
||||||
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and
|
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
|
||||||
`script/check`, then builds the Docker image tagged via
|
then builds the Docker image tagged via `script/projectname`
|
||||||
`script/projectname`
|
- `script/precommit` — pre-commit gate: `go mod tidy` (must not change `go.mod`
|
||||||
- `script/precommit` — pre-commit gate: `go mod tidy` (must not change
|
or `go.sum`) and `go fmt`, then `script/check`
|
||||||
`go.mod` or `go.sum`) and `go fmt`, then `script/check`
|
- `script/install-precommit` — install the git pre-commit hook that runs
|
||||||
- `script/install-precommit` — install the git pre-commit hook that
|
`script/precommit`
|
||||||
runs `script/precommit`
|
|
||||||
|
|
||||||
Every Docker build in `script/` is uncached, so the `lint` and `test`
|
Every Docker build in `script/` is uncached, so the `lint` and `test` phases
|
||||||
phases always run rather than being served from the build cache.
|
always run rather than being served from the build cache.
|
||||||
|
|
||||||
# Problem Statement
|
# Problem Statement
|
||||||
|
|
||||||
A Bluesky firehose ingester writes every observed post (and associated
|
A Bluesky firehose ingester writes every observed post (and associated users,
|
||||||
users, hashtags, URLs, and media references) into a single ever-growing
|
hashtags, URLs, and media references) into a single ever-growing SQLite
|
||||||
SQLite database, `firehose.db`. This database has several properties that
|
database, `firehose.db`. This database has several properties that make it
|
||||||
make it awkward to publish or archive directly:
|
awkward to publish or archive directly:
|
||||||
|
|
||||||
- It is **large and always growing**, so re-publishing the whole thing every
|
- It is **large and always growing**, so re-publishing the whole thing every day
|
||||||
day is wasteful.
|
is wasteful.
|
||||||
- It is **continuously written**, so reading from it directly risks lock
|
- It is **continuously written**, so reading from it directly risks lock
|
||||||
contention with the live ingester and inconsistent reads.
|
contention with the live ingester and inconsistent reads.
|
||||||
- It is **monolithic**, so there is no natural unit at which to mirror,
|
- It is **monolithic**, so there is no natural unit at which to mirror, share,
|
||||||
share, or distribute "just yesterday's posts".
|
or distribute "just yesterday's posts".
|
||||||
|
|
||||||
What is wanted instead is a stable, immutable, per-day artifact: a small
|
What is wanted instead is a stable, immutable, per-day artifact: a small file
|
||||||
file containing exactly one calendar day of firehose data, cheap to publish,
|
containing exactly one calendar day of firehose data, cheap to publish, cheap to
|
||||||
cheap to mirror, and trivially re-mergeable into a full database by anyone
|
mirror, and trivially re-mergeable into a full database by anyone who collects a
|
||||||
who collects a set of them.
|
set of them.
|
||||||
|
|
||||||
# Proposed Solution
|
# Proposed Solution
|
||||||
|
|
||||||
@@ -108,63 +106,63 @@ A tool, `bsdaily`, that:
|
|||||||
filesystem, so it reads from a consistent point-in-time copy that the live
|
filesystem, so it reads from a consistent point-in-time copy that the live
|
||||||
ingester cannot be writing to;
|
ingester cannot be writing to;
|
||||||
- copies the snapshot's database files to fast scratch storage;
|
- copies the snapshot's database files to fast scratch storage;
|
||||||
- **extracts** a single day's `posts` (and all rows reachable from them) into
|
- **extracts** a single day's `posts` (and all rows reachable from them) into a
|
||||||
a fresh, minimal per-day SQLite database;
|
fresh, minimal per-day SQLite database;
|
||||||
- **dumps** that per-day database to SQL and pipes it through multithreaded
|
- **dumps** that per-day database to SQL and pipes it through multithreaded zstd
|
||||||
zstd compression;
|
compression;
|
||||||
- **verifies** the compressed output (zstd integrity check plus a sanity
|
- **verifies** the compressed output (zstd integrity check plus a sanity check
|
||||||
check that the decompressed stream actually looks like SQL);
|
that the decompressed stream actually looks like SQL);
|
||||||
- **publishes** the result atomically as
|
- **publishes** the result atomically as
|
||||||
`DailiesBase/YYYY-MM/YYYY-MM-DD.sql.zst`.
|
`DailiesBase/YYYY-MM/YYYY-MM-DD.sql.zst`.
|
||||||
|
|
||||||
Each daily dump is emitted with `INSERT` statements over the full schema
|
Each daily dump is emitted with `INSERT` statements over the full schema
|
||||||
(including the deduplicated `users`, `hashtags`, and `urls` lookup tables),
|
(including the deduplicated `users`, `hashtags`, and `urls` lookup tables), so
|
||||||
so any collection of daily dumps can be merged into a single database by
|
any collection of daily dumps can be merged into a single database by rewriting
|
||||||
rewriting `INSERT INTO` to `INSERT OR IGNORE INTO` and replaying them in
|
`INSERT INTO` to `INSERT OR IGNORE INTO` and replaying them in sequence. Two
|
||||||
sequence. Two helper scripts ([`merge_daily_dumps.sh`](merge_daily_dumps.sh)
|
helper scripts ([`merge_daily_dumps.sh`](merge_daily_dumps.sh) and
|
||||||
and [`regenerate_auxiliary_tables.sql`](regenerate_auxiliary_tables.sql)) are
|
[`regenerate_auxiliary_tables.sql`](regenerate_auxiliary_tables.sql)) are
|
||||||
included to do exactly this and to rebuild the aggregate statistics
|
included to do exactly this and to rebuild the aggregate statistics
|
||||||
(`use_count`, `first_seen`, user `resolved_at`/`updated_at`) afterward.
|
(`use_count`, `first_seen`, user `resolved_at`/`updated_at`) afterward.
|
||||||
|
|
||||||
# Design Goals
|
# Design Goals
|
||||||
|
|
||||||
- **Never disturb the live ingester.** All reads come from a ZFS snapshot,
|
- **Never disturb the live ingester.** All reads come from a ZFS snapshot, never
|
||||||
never the live database.
|
the live database.
|
||||||
- **Crash-safe, idempotent runs.** Output is written to a temp file and
|
- **Crash-safe, idempotent runs.** Output is written to a temp file and
|
||||||
atomically renamed; a day whose final output already exists is skipped, so
|
atomically renamed; a day whose final output already exists is skipped, so
|
||||||
re-running a range is safe and resumable.
|
re-running a range is safe and resumable.
|
||||||
- **Mergeable output.** Daily dumps re-combine losslessly into a full
|
- **Mergeable output.** Daily dumps re-combine losslessly into a full database
|
||||||
database via `INSERT OR IGNORE`.
|
via `INSERT OR IGNORE`.
|
||||||
- **Operationally cautious.** Free-space preflight checks on both scratch and
|
- **Operationally cautious.** Free-space preflight checks on both scratch and
|
||||||
output filesystems; explicit verification of every artifact before it is
|
output filesystems; explicit verification of every artifact before it is
|
||||||
published.
|
published.
|
||||||
- **Fast where it's free.** Large snapshot copies use a 256MiB buffer,
|
- **Fast where it's free.** Large snapshot copies use a 256MiB buffer,
|
||||||
pre-allocate the destination, and (on Linux) issue `posix_fadvise`
|
pre-allocate the destination, and (on Linux) issue `posix_fadvise`
|
||||||
sequential/willneed hints; extraction uses aggressive,
|
sequential/willneed hints; extraction uses aggressive, crash-unsafe-by-design
|
||||||
crash-unsafe-by-design SQLite pragmas because the working data lives only
|
SQLite pragmas because the working data lives only in disposable scratch
|
||||||
in disposable scratch space.
|
space.
|
||||||
|
|
||||||
# Non-Goals
|
# Non-Goals
|
||||||
|
|
||||||
- **Real-time export.** `bsdaily` operates on daily snapshots; the freshest
|
- **Real-time export.** `bsdaily` operates on daily snapshots; the freshest day
|
||||||
day it can produce is the snapshot date minus one.
|
it can produce is the snapshot date minus one.
|
||||||
- **Schema ownership.** The schema is defined by the upstream firehose
|
- **Schema ownership.** The schema is defined by the upstream firehose ingester;
|
||||||
ingester; [`schema.sql`](schema.sql) is included for reference only.
|
[`schema.sql`](schema.sql) is included for reference only. `bsdaily` copies
|
||||||
`bsdaily` copies whatever table and index DDL it finds in the source.
|
whatever table and index DDL it finds in the source.
|
||||||
- **Cross-platform deployment.** It is built and run on Linux (the
|
- **Cross-platform deployment.** It is built and run on Linux (the free-space
|
||||||
free-space check and fadvise hints use `golang.org/x/sys/unix`; a non-Linux
|
check and fadvise hints use `golang.org/x/sys/unix`; a non-Linux build
|
||||||
build compiles but is a no-op for the fadvise hints). The hard-coded paths
|
compiles but is a no-op for the fadvise hints). The hard-coded paths assume
|
||||||
assume the production host's ZFS layout.
|
the production host's ZFS layout.
|
||||||
|
|
||||||
# How It Works
|
# How It Works
|
||||||
|
|
||||||
A single run proceeds as follows:
|
A single run proceeds as follows:
|
||||||
|
|
||||||
1. **Find the snapshot.** Scan `SnapshotBase` for directories matching
|
1. **Find the snapshot.** Scan `SnapshotBase` for directories matching
|
||||||
`zfs-auto-snap_daily-YYYY-MM-DD-NNNN`, pick the most recent, and confirm
|
`zfs-auto-snap_daily-YYYY-MM-DD-NNNN`, pick the most recent, and confirm it
|
||||||
it contains `firehose.db`.
|
contains `firehose.db`.
|
||||||
2. **Determine target days.** Default to the snapshot date minus one day;
|
2. **Determine target days.** Default to the snapshot date minus one day; or use
|
||||||
or use `--date`, or every day in the inclusive `--from`/`--to` range.
|
`--date`, or every day in the inclusive `--from`/`--to` range.
|
||||||
3. **Preflight disk space.** Require at least 500GiB free on the scratch
|
3. **Preflight disk space.** Require at least 500GiB free on the scratch
|
||||||
filesystem and 20GiB free on the output filesystem.
|
filesystem and 20GiB free on the output filesystem.
|
||||||
4. **Copy the database to scratch.** Copy `firehose.db`, its `-wal`, and (if
|
4. **Copy the database to scratch.** Copy `firehose.db`, its `-wal`, and (if
|
||||||
@@ -173,11 +171,11 @@ A single run proceeds as follows:
|
|||||||
5. **Per day**, processed strictly one at a time to avoid SQLite contention:
|
5. **Per day**, processed strictly one at a time to avoid SQLite contention:
|
||||||
- skip the day if its final output file already exists;
|
- skip the day if its final output file already exists;
|
||||||
- `ATTACH` the copied source DB to a new empty per-day DB, recreate the
|
- `ATTACH` the copied source DB to a new empty per-day DB, recreate the
|
||||||
table DDL, and `INSERT ... SELECT` the target day's `posts` plus all
|
table DDL, and `INSERT ... SELECT` the target day's `posts` plus all rows
|
||||||
rows reachable from them (`posts_hashtags`, `posts_urls`, `hashtags`,
|
reachable from them (`posts_hashtags`, `posts_urls`, `hashtags`, `urls`,
|
||||||
`urls`, `users`, and `media` if that table exists);
|
`users`, and `media` if that table exists);
|
||||||
- abort the day cleanly if there are zero posts (`ErrNoPosts`), rather
|
- abort the day cleanly if there are zero posts (`ErrNoPosts`), rather than
|
||||||
than emitting an empty dump;
|
emitting an empty dump;
|
||||||
- recreate indexes, detach the source, and verify the inserted row count;
|
- recreate indexes, detach the source, and verify the inserted row count;
|
||||||
- `sqlite3 .dump | zstdmt` into a hidden temp file;
|
- `sqlite3 .dump | zstdmt` into a hidden temp file;
|
||||||
- run a zstd integrity check and confirm the decompressed head looks like
|
- run a zstd integrity check and confirm the decompressed head looks like
|
||||||
@@ -219,9 +217,9 @@ sqlite3 merged.db < regenerate_auxiliary_tables.sql
|
|||||||
- **Go** (see [`go.mod`](go.mod) for the toolchain version) to build.
|
- **Go** (see [`go.mod`](go.mod) for the toolchain version) to build.
|
||||||
- **Linux** for production use (ZFS snapshots, `statfs` free-space checks,
|
- **Linux** for production use (ZFS snapshots, `statfs` free-space checks,
|
||||||
`posix_fadvise` hints).
|
`posix_fadvise` hints).
|
||||||
- The **`sqlite3`** and **`zstdmt`** (multithreaded zstd) binaries on
|
- The **`sqlite3`** and **`zstdmt`** (multithreaded zstd) binaries on `PATH`;
|
||||||
`PATH`; `zstdcat` is used for verification. SQLite reads/writes during
|
`zstdcat` is used for verification. SQLite reads/writes during extraction use
|
||||||
extraction use the pure-Go [`modernc.org/sqlite`](https://pkg.go.dev/modernc.org/sqlite)
|
the pure-Go [`modernc.org/sqlite`](https://pkg.go.dev/modernc.org/sqlite)
|
||||||
driver, so no cgo is required for that part.
|
driver, so no cgo is required for that part.
|
||||||
|
|
||||||
# Configuration
|
# Configuration
|
||||||
@@ -241,21 +239,21 @@ elsewhere.
|
|||||||
|
|
||||||
# Data Model
|
# Data Model
|
||||||
|
|
||||||
The firehose schema (reference copy in [`schema.sql`](schema.sql)) centers
|
The firehose schema (reference copy in [`schema.sql`](schema.sql)) centers on a
|
||||||
on a `posts` table, with `users` keyed by DID and many-to-many junction
|
`posts` table, with `users` keyed by DID and many-to-many junction tables
|
||||||
tables linking posts to deduplicated `hashtags` and `urls`. An optional
|
linking posts to deduplicated `hashtags` and `urls`. An optional `media` table
|
||||||
`media` table tracks downloaded blobs by content hash. `bsdaily` does not
|
tracks downloaded blobs by content hash. `bsdaily` does not own this schema; it
|
||||||
own this schema; it reflects whatever DDL exists in the source snapshot and
|
reflects whatever DDL exists in the source snapshot and selects forward from
|
||||||
selects forward from `posts` along the foreign-key relationships to produce a
|
`posts` along the foreign-key relationships to produce a referentially-complete
|
||||||
referentially-complete per-day slice.
|
per-day slice.
|
||||||
|
|
||||||
# Use Cases
|
# Use Cases
|
||||||
|
|
||||||
## Daily public archive
|
## Daily public archive
|
||||||
|
|
||||||
Publish one small, immutable file per day to static HTTP (or IPFS, or a
|
Publish one small, immutable file per day to static HTTP (or IPFS, or a mirror
|
||||||
mirror network) so that anyone can fetch exactly the day(s) they want and
|
network) so that anyone can fetch exactly the day(s) they want and re-merge them
|
||||||
re-merge them locally.
|
locally.
|
||||||
|
|
||||||
## Backfilling a range
|
## Backfilling a range
|
||||||
|
|
||||||
@@ -265,16 +263,17 @@ safe to re-run.
|
|||||||
|
|
||||||
## Reconstituting a full database
|
## Reconstituting a full database
|
||||||
|
|
||||||
Collect any set of daily dumps and merge them with `INSERT OR IGNORE` to
|
Collect any set of daily dumps and merge them with `INSERT OR IGNORE` to rebuild
|
||||||
rebuild a complete, queryable SQLite database, then regenerate the aggregate
|
a complete, queryable SQLite database, then regenerate the aggregate statistics
|
||||||
statistics tables.
|
tables.
|
||||||
|
|
||||||
# See Also
|
# See Also
|
||||||
|
|
||||||
## Links
|
## Links
|
||||||
|
|
||||||
- Repo: [https://git.eeqj.de/sneak/bsdaily](https://git.eeqj.de/sneak/bsdaily)
|
- Repo: [https://git.eeqj.de/sneak/bsdaily](https://git.eeqj.de/sneak/bsdaily)
|
||||||
- Issues: [https://git.eeqj.de/sneak/bsdaily/issues](https://git.eeqj.de/sneak/bsdaily/issues)
|
- Issues:
|
||||||
|
[https://git.eeqj.de/sneak/bsdaily/issues](https://git.eeqj.de/sneak/bsdaily/issues)
|
||||||
- Bluesky: [https://bsky.app](https://bsky.app)
|
- Bluesky: [https://bsky.app](https://bsky.app)
|
||||||
- zstd: [https://facebook.github.io/zstd/](https://facebook.github.io/zstd/)
|
- zstd: [https://facebook.github.io/zstd/](https://facebook.github.io/zstd/)
|
||||||
|
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
# Workflow
|
# Workflow
|
||||||
|
|
||||||
* branch (from `main`)
|
- branch (from `main`)
|
||||||
* do the work in Next Step
|
- do the work in Next Step
|
||||||
* move Next Step to the top of Completed Steps
|
- move Next Step to the top of Completed Steps
|
||||||
* move the top item of Future Steps into Next Step
|
- move the top item of Future Steps into Next Step
|
||||||
* commit (`TODO.md` changes in the same commit as the work)
|
- commit (`TODO.md` changes in the same commit as the work)
|
||||||
* merge to `main` if the branch is not protected, otherwise open a PR
|
- merge to `main` if the branch is not protected, otherwise open a PR
|
||||||
* push
|
- push
|
||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
@@ -14,35 +14,38 @@ pre-1.0
|
|||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
Add the canonical `.golangci.yml`, move the lint phase to golangci-lint
|
Add the canonical `.golangci.yml`, move the lint phase to golangci-lint v2.14.0
|
||||||
v2.14.0 in the same commit, and fix the findings it surfaces
|
in the same commit, and fix the findings it surfaces
|
||||||
(https://git.eeqj.de/sneak/bsdaily/issues/6).
|
(https://git.eeqj.de/sneak/bsdaily/issues/6).
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-06: Formatted Markdown with prettier: `script/fmt` writes and
|
||||||
|
`script/fmt-check` checks every Markdown file; prettier pinned in
|
||||||
|
`package.json` and `yarn.lock`, installed by `script/bootstrap`, which
|
||||||
|
installs node and yarn at pinned versions when absent and otherwise uses the
|
||||||
|
ones already installed; existing Markdown reformatted.
|
||||||
- 2026-10-05: Brought the repo up to the standard layout: canonical
|
- 2026-10-05: Brought the repo up to the standard layout: canonical
|
||||||
`.gitignore`, `.dockerignore` and `.editorconfig`; `lint` and `test`
|
`.gitignore`, `.dockerignore` and `.editorconfig`; `lint` and `test` phases in
|
||||||
phases in the `Dockerfile`, built by `script/lint` and `script/test`;
|
the `Dockerfile`, built by `script/lint` and `script/test`; canonical
|
||||||
canonical `script/cibuild`, `script/docker` and CI workflow; no linter
|
`script/cibuild`, `script/docker` and CI workflow; no linter installed on the
|
||||||
installed on the host; re-vendored `REPO_POLICIES.md`.
|
host; re-vendored `REPO_POLICIES.md`.
|
||||||
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
|
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
||||||
Makefile shims, README Entrypoints section
|
shims, README Entrypoints section
|
||||||
- 2026-06-28: Fixed errcheck lint failures; added compilation smoke test;
|
- 2026-06-28: Fixed errcheck lint failures; added compilation smoke test; tidied
|
||||||
tidied go.mod.
|
go.mod.
|
||||||
- 2026-06-28: Added repo scaffolding: README, LICENSE, Makefile,
|
- 2026-06-28: Added repo scaffolding: README, LICENSE, Makefile, Dockerfile,
|
||||||
Dockerfile, REPO_POLICIES.md, and Gitea CI.
|
REPO_POLICIES.md, and Gitea CI.
|
||||||
- 2026-02-12: Fixed SQLite database locking by removing parallel
|
- 2026-02-12: Fixed SQLite database locking by removing parallel processing;
|
||||||
processing; fixed Linux build via golang.org/x/sys/unix Fadvise.
|
fixed Linux build via golang.org/x/sys/unix Fadvise.
|
||||||
- 2026-02-12: Optimized file copy for large databases; moved temp
|
- 2026-02-12: Optimized file copy for large databases; moved temp directory to
|
||||||
directory to NVMe scratch storage.
|
NVMe scratch storage.
|
||||||
- 2026-02-11: Added date range support.
|
- 2026-02-11: Added date range support.
|
||||||
- 2026-02-09: Initial implementation: single-day extraction, specific-date
|
- 2026-02-09: Initial implementation: single-day extraction, specific-date
|
||||||
targeting, faster pruning of throwaway database copies.
|
targeting, faster pruning of throwaway database copies.
|
||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
- Format Markdown with prettier in `script/fmt` and `script/fmt-check`
|
- Expand tests beyond the compilation smoke test: unit tests for the extraction,
|
||||||
(https://git.eeqj.de/sneak/bsdaily/issues/7).
|
verification, and atomic-publish paths.
|
||||||
- Expand tests beyond the compilation smoke test: unit tests for the
|
|
||||||
extraction, verification, and atomic-publish paths.
|
|
||||||
- Cut a first SemVer release once compliance and test coverage land.
|
- Cut a first SemVer release once compliance and test coverage land.
|
||||||
|
|||||||
@@ -0,0 +1,5 @@
|
|||||||
|
{
|
||||||
|
"devDependencies": {
|
||||||
|
"prettier": "3.8.1"
|
||||||
|
}
|
||||||
|
}
|
||||||
+79
-1
@@ -3,11 +3,20 @@
|
|||||||
# this repo. Idempotent: every install is guarded by a check so already
|
# this repo. Idempotent: every install is guarded by a check so already
|
||||||
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
||||||
# or apk (detected in that order); assumes NOTHING is present (not git,
|
# or apk (detected in that order); assumes NOTHING is present (not git,
|
||||||
# make, or go).
|
# make, go, or node). Node is used directly if installed; otherwise it
|
||||||
|
# is installed at a pinned version via nvm (installing nvm itself first,
|
||||||
|
# from a hash-verified release archive, never curl | sh).
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
|
# Pinned versions, 2026-07-06
|
||||||
|
NODE_VERSION="22.17.0"
|
||||||
|
NVM_VERSION="0.40.3"
|
||||||
|
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
|
||||||
|
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
|
||||||
|
YARN_VERSION="1.22.22"
|
||||||
|
|
||||||
PKGMGR=""
|
PKGMGR=""
|
||||||
SUDO=""
|
SUDO=""
|
||||||
APT_UPDATED=""
|
APT_UPDATED=""
|
||||||
@@ -56,6 +65,69 @@ missing() {
|
|||||||
! command -v "$1" >/dev/null 2>&1
|
! command -v "$1" >/dev/null 2>&1
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# verify_sha256 <file> <expected-hash>
|
||||||
|
verify_sha256() {
|
||||||
|
if command -v sha256sum >/dev/null 2>&1; then
|
||||||
|
actual="$(sha256sum "$1" | cut -d' ' -f1)"
|
||||||
|
else
|
||||||
|
actual="$(shasum -a 256 "$1" | cut -d' ' -f1)"
|
||||||
|
fi
|
||||||
|
if [ "$actual" != "$2" ]; then
|
||||||
|
echo "bootstrap: sha256 mismatch for $1" >&2
|
||||||
|
echo " expected: $2" >&2
|
||||||
|
echo " actual: $actual" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# nvm is a bash script; run a command in a bash with nvm loaded
|
||||||
|
nvm_sh() {
|
||||||
|
bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*"
|
||||||
|
}
|
||||||
|
|
||||||
|
ensure_nvm() {
|
||||||
|
[ -s "$HOME/.nvm/nvm.sh" ] && return 0
|
||||||
|
# nvm prerequisites; nvm itself requires bash
|
||||||
|
if missing bash; then pkg_install bash bash bash bash; fi
|
||||||
|
if missing curl; then pkg_install curl curl curl curl; fi
|
||||||
|
if missing git; then pkg_install git git git git; fi
|
||||||
|
tmp="$(mktemp -d)"
|
||||||
|
curl -fsSL -o "$tmp/nvm.tar.gz" \
|
||||||
|
"https://github.com/nvm-sh/nvm/archive/refs/tags/v${NVM_VERSION}.tar.gz"
|
||||||
|
verify_sha256 "$tmp/nvm.tar.gz" "$NVM_SHA256"
|
||||||
|
mkdir -p "$HOME/.nvm"
|
||||||
|
tar -xzf "$tmp/nvm.tar.gz" -C "$HOME/.nvm" --strip-components=1
|
||||||
|
rm -rf "$tmp"
|
||||||
|
}
|
||||||
|
|
||||||
|
ensure_node() {
|
||||||
|
if ! missing node; then return 0; fi
|
||||||
|
ensure_nvm
|
||||||
|
nvm_sh "nvm install $NODE_VERSION"
|
||||||
|
}
|
||||||
|
|
||||||
|
ensure_yarn() {
|
||||||
|
if ! missing yarn; then return 0; fi
|
||||||
|
if ! missing corepack; then
|
||||||
|
corepack enable
|
||||||
|
corepack prepare "yarn@$YARN_VERSION" --activate
|
||||||
|
elif [ -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
|
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
|
||||||
|
corepack prepare yarn@$YARN_VERSION --activate"
|
||||||
|
else
|
||||||
|
npm install -g "yarn@$YARN_VERSION"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
install_js_deps() {
|
||||||
|
if missing yarn && [ -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
|
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
|
||||||
|
yarn install --frozen-lockfile"
|
||||||
|
else
|
||||||
|
yarn install --frozen-lockfile
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
|
|
||||||
@@ -68,6 +140,12 @@ main() {
|
|||||||
|
|
||||||
go mod download
|
go mod download
|
||||||
|
|
||||||
|
# Node and yarn, then the prettier pinned in package.json and
|
||||||
|
# yarn.lock, which script/fmt and script/fmt-check run
|
||||||
|
ensure_node
|
||||||
|
ensure_yarn
|
||||||
|
install_js_deps
|
||||||
|
|
||||||
echo "bootstrap complete"
|
echo "bootstrap complete"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+23
-1
@@ -1,12 +1,34 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/fmt: format all files (writes).
|
# script/fmt: format all files (writes): the Go code with go fmt and
|
||||||
|
# every Markdown file with prettier.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
|
# Must match the pin in script/bootstrap.
|
||||||
|
NODE_VERSION="22.17.0"
|
||||||
|
|
||||||
|
# script/bootstrap installs node and yarn under nvm and leaves neither
|
||||||
|
# on the PATH of the shell that called it, so resolve the pinned
|
||||||
|
# toolchain here the way bootstrap's own install step does. nvm is a
|
||||||
|
# bash script, hence the subshell.
|
||||||
|
run_yarn() {
|
||||||
|
if command -v yarn >/dev/null 2>&1; then
|
||||||
|
exec yarn "$@"
|
||||||
|
fi
|
||||||
|
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
|
echo "fmt: no yarn; run script/bootstrap first" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
|
||||||
|
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
|
||||||
|
}
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
go fmt ./...
|
go fmt ./...
|
||||||
|
# run_yarn replaces this shell, so it stays the last step.
|
||||||
|
run_yarn run prettier --write '**/*.md' --tab-width 4 --prose-wrap always
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+23
-1
@@ -1,10 +1,30 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/fmt-check: check formatting (read-only). Same scope as
|
# script/fmt-check: check formatting (read-only). Same scope as
|
||||||
# script/fmt, but fails instead of writing.
|
# script/fmt, the Go code and every Markdown file, but fails instead of
|
||||||
|
# writing.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
|
# Must match the pin in script/bootstrap.
|
||||||
|
NODE_VERSION="22.17.0"
|
||||||
|
|
||||||
|
# script/bootstrap installs node and yarn under nvm and leaves neither
|
||||||
|
# on the PATH of the shell that called it, so resolve the pinned
|
||||||
|
# toolchain here the way bootstrap's own install step does. nvm is a
|
||||||
|
# bash script, hence the subshell.
|
||||||
|
run_yarn() {
|
||||||
|
if command -v yarn >/dev/null 2>&1; then
|
||||||
|
exec yarn "$@"
|
||||||
|
fi
|
||||||
|
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
|
echo "fmt-check: no yarn; run script/bootstrap first" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
|
||||||
|
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
|
||||||
|
}
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
unformatted="$(gofmt -l .)"
|
unformatted="$(gofmt -l .)"
|
||||||
@@ -13,6 +33,8 @@ main() {
|
|||||||
echo "$unformatted" >&2
|
echo "$unformatted" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
# run_yarn replaces this shell, so it stays the last step.
|
||||||
|
run_yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
|
||||||
|
# yarn lockfile v1
|
||||||
|
|
||||||
|
|
||||||
|
prettier@3.8.1:
|
||||||
|
version "3.8.1"
|
||||||
|
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
|
||||||
|
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==
|
||||||
Reference in New Issue
Block a user