Refresh vendored REPO_POLICIES.md

Handle every error return flagged by golangci-lint's errcheck rather
than discarding it:

- run.go: add a cleanup() helper that logs a warning (and ignores
  ErrNotExist) when removing a temp file fails, so leaked scratch files
  are surfaced; use it for all best-effort removals.
- copy.go / extract.go: log a warning on deferred Close() failures for
  the source file, destination DB, and result-set rows.
- extract.go: on the rollback path, ignore the benign sql.ErrTxDone
  (already committed) and log any other rollback failure.
- verify.go: add killCat() which ignores os.ErrProcessDone (zstdcat
  already exited via SIGPIPE) and logs any unexpected kill failure.

make check is clean (0 lint issues, tests pass).

.gitea/workflows/check.yml

@@ -0,0 +1,14 @@
+name: check
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      # actions/checkout v4, 2024-09-16
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
+      - name: Build and check
+        run: script/cibuild

Dockerfile

@@ -0,0 +1,57 @@
+# Lint stage
+# golangci/golangci-lint:v2.12.2-alpine
+FROM golangci/golangci-lint:v2.12.2-alpine@sha256:91b27804074a0bacea298707f016911e60cf0cdbc6c7bf5ccacb5f0606d18d60 AS lint
+
+RUN apk add --no-cache make build-base
+
+WORKDIR /src
+
+# Copy go mod files first for better layer caching
+COPY go.mod go.sum ./
+RUN go mod download
+
+# Copy source code
+COPY . .
+
+# Run formatting check and linter
+RUN make fmt-check
+RUN make lint
+
+# Build stage
+# golang:1.26.4-alpine
+FROM golang:1.26.4-alpine@sha256:3ad57304ad93bbec8548a0437ad9e06a455660655d9af011d58b993f6f615648 AS builder
+
+# Depend on lint stage passing
+COPY --from=lint /src/go.sum /dev/null
+
+ARG VERSION=dev
+
+# Install build deps plus the sqlite3 and zstd CLIs the tests/tool shell out to
+RUN apk add --no-cache make build-base sqlite zstd
+
+WORKDIR /src
+
+# Copy go mod files first for better layer caching
+COPY go.mod go.sum ./
+RUN go mod download
+
+# Copy source code
+COPY . .
+
+# Run tests
+RUN make test
+
+# Build (pure Go, no CGO required since we use modernc.org/sqlite)
+RUN CGO_ENABLED=0 go build -o /bsdaily ./cmd/bsdaily
+
+# Runtime stage
+# alpine:3.21
+FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
+
+# bsdaily shells out to sqlite3, zstdmt, zstdcat at runtime
+RUN apk add --no-cache ca-certificates sqlite zstd
+
+# Copy binary from builder
+COPY --from=builder /bsdaily /usr/local/bin/bsdaily
+
+ENTRYPOINT ["/usr/local/bin/bsdaily"]

LICENSE

@@ -0,0 +1,13 @@
+            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
+                    Version 2, December 2004
+
+ Copyright (C) 2004 Sam Hocevar <sam@hocevar.net>
+
+ Everyone is permitted to copy and distribute verbatim or modified
+ copies of this license document, and changing it is allowed as long
+ as the name is changed.
+
+            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. You just DO WHAT THE FUCK YOU WANT TO.

Makefile

@@ -0,0 +1,77 @@
+.PHONY: all bootstrap setup check test lint fmt fmt-check build clean deps test-coverage test-integration install release release-snapshot docker hooks
+
+# Version number
+VERSION := 0.1.0-dev
+
+# Default target
+all: bsdaily
+
+# Install all development dependencies.
+bootstrap:
+	@script/bootstrap
+
+# Prepare a fresh clone: bootstrap plus pre-commit hook.
+setup:
+	@script/setup
+
+# Combined pre-commit/CI gate: tests, lint, format check.
+check:
+	@script/check
+
+# Run tests only.
+test:
+	@script/test
+
+# Check if code is formatted (read-only).
+fmt-check:
+	@script/fmt-check
+
+# Format code.
+fmt:
+	@script/fmt
+
+# Run linter only.
+lint:
+	@script/lint
+
+# Build binary (pure Go; no CGO required since we use modernc.org/sqlite).
+bsdaily: internal/*/*.go cmd/bsdaily/*.go
+	CGO_ENABLED=0 go build -o $@ ./cmd/bsdaily
+
+# Clean build artifacts.
+clean:
+	rm -f bsdaily
+	go clean
+
+# Install dependencies.
+deps:
+	go mod download
+	go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
+
+# Run tests with coverage.
+test-coverage:
+	go test -v -coverprofile=coverage.out ./...
+	go tool cover -html=coverage.out -o coverage.html
+
+# Run integration tests.
+test-integration:
+	go test -v -tags=integration ./...
+
+install: bsdaily
+	cp ./bsdaily $(HOME)/bin/
+
+# Build and publish release artifacts via goreleaser.
+release:
+	goreleaser release --clean
+
+# Dry-run a release build without publishing or tagging.
+release-snapshot:
+	goreleaser release --clean --snapshot
+
+# Build Docker image.
+docker:
+	@script/docker
+
+# Install pre-commit hook.
+hooks:
+	@script/install-precommit

README.md

@@ -0,0 +1,280 @@
+# bsdaily
+
+[bsdaily](https://git.eeqj.de/sneak/bsdaily) is a command-line utility
+written in [Go](https://golang.org) that carves a single day (or a range of
+days) of [Bluesky](https://bsky.app) firehose data out of a large,
+continuously-growing SQLite database and writes it out as a self-contained,
+[zstd](https://facebook.github.io/zstd/)-compressed SQL dump. The dumps are
+named by date (e.g. `2026-06-27.sql.zst`), organized into per-month
+directories, and are designed to be published, archived, mirrored, and later
+re-merged back into a single database.
+
+The source database is read from a read-only [ZFS](https://openzfs.org)
+snapshot, so extraction never contends with the live firehose ingester that
+is writing to the original database. The tool is operationally
+conservative: it checks free disk space before starting, copies the snapshot
+to fast scratch storage, processes one day at a time to avoid SQLite lock
+contention, verifies every compressed output before publishing it, and
+writes output atomically via a temp-file-and-rename so a partial run never
+leaves a corrupt `.sql.zst` behind.
+
+This project was written by [@sneak](https://sneak.berlin) to produce a
+daily, mergeable, publicly-mirrorable archive of the Bluesky firehose. It is
+currently a one-person effort. The current version is pre-1.0 and there has
+not yet been a versioned release; [SemVer](https://semver.org) will be used
+for releases.
+
+# Build Status
+
+CI runs the standard `make check` (formatting, linting, tests). The `main`
+branch must always be green.
+
+# Participation
+
+Primary development happens on a privately-run Gitea instance at
+[https://git.eeqj.de/sneak/bsdaily](https://git.eeqj.de/sneak/bsdaily) and
+issues are [tracked
+there](https://git.eeqj.de/sneak/bsdaily/issues).
+
+Changes must always be formatted with a standard `go fmt`, syntactically
+valid, and must pass the linting defined in the repository (presently the
+`golangci-lint` defaults), which can be run with a `make lint`. The `main`
+branch is protected and all changes must be made via [pull
+requests](https://git.eeqj.de/sneak/bsdaily/pulls) and pass CI to be merged.
+
+See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
+tooling requirements, and workflow conventions.
+
+# Entrypoints
+
+This repository adheres to the
+[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
+standard: normalized scripts in `script/` are the entrypoints for the
+development workflow, and the Makefile targets are thin shims that call
+them. We provide:
+
+- `script/bootstrap` — install all development dependencies (go,
+  golangci-lint, Go module download)
+- `script/setup` — make a fresh clone ready for development: runs
+  `script/bootstrap`, then `script/install-precommit`
+- `script/projectname` — print the project name (used for the Docker
+  image tag)
+- `script/test` — run the test suite (verbose rerun on failure)
+- `script/lint` — run `golangci-lint run ./...`
+- `script/fmt` — format all code (writes)
+- `script/fmt-check` — check formatting (read-only)
+- `script/check` — run `script/test`, `script/lint`, and
+  `script/fmt-check`
+- `script/docker` — build the Docker image tagged via
+  `script/projectname`
+- `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile
+  runs the checks)
+- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
+  not change files), then `script/check`
+- `script/install-precommit` — install the git pre-commit hook that
+  runs `script/precommit`
+
+# Problem Statement
+
+A Bluesky firehose ingester writes every observed post (and associated
+users, hashtags, URLs, and media references) into a single ever-growing
+SQLite database, `firehose.db`. This database has several properties that
+make it awkward to publish or archive directly:
+
+- It is **large and always growing**, so re-publishing the whole thing every
+  day is wasteful.
+- It is **continuously written**, so reading from it directly risks lock
+  contention with the live ingester and inconsistent reads.
+- It is **monolithic**, so there is no natural unit at which to mirror,
+  share, or distribute "just yesterday's posts".
+
+What is wanted instead is a stable, immutable, per-day artifact: a small
+file containing exactly one calendar day of firehose data, cheap to publish,
+cheap to mirror, and trivially re-mergeable into a full database by anyone
+who collects a set of them.
+
+# Proposed Solution
+
+A tool, `bsdaily`, that:
+
+- locates the most recent read-only **ZFS daily snapshot** of the firehose
+  filesystem, so it reads from a consistent point-in-time copy that the live
+  ingester cannot be writing to;
+- copies the snapshot's database files to fast scratch storage;
+- **extracts** a single day's `posts` (and all rows reachable from them) into
+  a fresh, minimal per-day SQLite database;
+- **dumps** that per-day database to SQL and pipes it through multithreaded
+  zstd compression;
+- **verifies** the compressed output (zstd integrity check plus a sanity
+  check that the decompressed stream actually looks like SQL);
+- **publishes** the result atomically as
+  `DailiesBase/YYYY-MM/YYYY-MM-DD.sql.zst`.
+
+Each daily dump is emitted with `INSERT` statements over the full schema
+(including the deduplicated `users`, `hashtags`, and `urls` lookup tables),
+so any collection of daily dumps can be merged into a single database by
+rewriting `INSERT INTO` to `INSERT OR IGNORE INTO` and replaying them in
+sequence. Two helper scripts ([`merge_daily_dumps.sh`](merge_daily_dumps.sh)
+and [`regenerate_auxiliary_tables.sql`](regenerate_auxiliary_tables.sql)) are
+included to do exactly this and to rebuild the aggregate statistics
+(`use_count`, `first_seen`, user `resolved_at`/`updated_at`) afterward.
+
+# Design Goals
+
+- **Never disturb the live ingester.** All reads come from a ZFS snapshot,
+  never the live database.
+- **Crash-safe, idempotent runs.** Output is written to a temp file and
+  atomically renamed; a day whose final output already exists is skipped, so
+  re-running a range is safe and resumable.
+- **Mergeable output.** Daily dumps re-combine losslessly into a full
+  database via `INSERT OR IGNORE`.
+- **Operationally cautious.** Free-space preflight checks on both scratch and
+  output filesystems; explicit verification of every artifact before it is
+  published.
+- **Fast where it's free.** Large snapshot copies use a 256MiB buffer,
+  pre-allocate the destination, and (on Linux) issue `posix_fadvise`
+  sequential/willneed hints; extraction uses aggressive,
+  crash-unsafe-by-design SQLite pragmas because the working data lives only
+  in disposable scratch space.
+
+# Non-Goals
+
+- **Real-time export.** `bsdaily` operates on daily snapshots; the freshest
+  day it can produce is the snapshot date minus one.
+- **Schema ownership.** The schema is defined by the upstream firehose
+  ingester; [`schema.sql`](schema.sql) is included for reference only.
+  `bsdaily` copies whatever table and index DDL it finds in the source.
+- **Cross-platform deployment.** It is built and run on Linux (the
+  free-space check and fadvise hints use `golang.org/x/sys/unix`; a non-Linux
+  build compiles but is a no-op for the fadvise hints). The hard-coded paths
+  assume the production host's ZFS layout.
+
+# How It Works
+
+A single run proceeds as follows:
+
+1. **Find the snapshot.** Scan `SnapshotBase` for directories matching
+   `zfs-auto-snap_daily-YYYY-MM-DD-NNNN`, pick the most recent, and confirm
+   it contains `firehose.db`.
+2. **Determine target days.** Default to the snapshot date minus one day;
+   or use `--date`, or every day in the inclusive `--from`/`--to` range.
+3. **Preflight disk space.** Require at least 500GiB free on the scratch
+   filesystem and 20GiB free on the output filesystem.
+4. **Copy the database to scratch.** Copy `firehose.db`, its `-wal`, and (if
+   present) its `-shm` from the snapshot into a fresh temp directory under
+   `TmpBase`.
+5. **Per day**, processed strictly one at a time to avoid SQLite contention:
+   - skip the day if its final output file already exists;
+   - `ATTACH` the copied source DB to a new empty per-day DB, recreate the
+     table DDL, and `INSERT ... SELECT` the target day's `posts` plus all
+     rows reachable from them (`posts_hashtags`, `posts_urls`, `hashtags`,
+     `urls`, `users`, and `media` if that table exists);
+   - abort the day cleanly if there are zero posts (`ErrNoPosts`), rather
+     than emitting an empty dump;
+   - recreate indexes, detach the source, and verify the inserted row count;
+   - `sqlite3 .dump | zstdmt` into a hidden temp file;
+   - run a zstd integrity check and confirm the decompressed head looks like
+     SQL;
+   - atomically rename into place and delete the per-day scratch DB.
+6. **Clean up** the temp directory and log a processed/skipped/total summary.
+
+# Usage
+
+```
+bsdaily                          # extract the snapshot date minus one day
+bsdaily --date 2026-06-27        # extract a single specific day
+bsdaily --from 2026-06-01 --to 2026-06-27   # extract an inclusive range
+```
+
+Flags:
+
+- `-d`, `--date YYYY-MM-DD` — extract a single day. Mutually exclusive with
+  `--from`/`--to`.
+- `--from YYYY-MM-DD` — start of an inclusive range (requires `--to`).
+- `--to YYYY-MM-DD` — end of an inclusive range (requires `--from`).
+
+With no flags, the tool extracts the day before the latest snapshot. All
+progress is logged as structured `slog` text to stderr.
+
+## Merging dumps back into a database
+
+```
+# Using the helper script:
+./merge_daily_dumps.sh merged.db daily_dumps/*.sql.zst
+
+# Or by hand:
+zstdcat *.sql.zst | sed 's/INSERT INTO/INSERT OR IGNORE INTO/g' | sqlite3 merged.db
+sqlite3 merged.db < regenerate_auxiliary_tables.sql
+```
+
+# Requirements
+
+- **Go** (see [`go.mod`](go.mod) for the toolchain version) to build.
+- **Linux** for production use (ZFS snapshots, `statfs` free-space checks,
+  `posix_fadvise` hints).
+- The **`sqlite3`** and **`zstdmt`** (multithreaded zstd) binaries on
+  `PATH`; `zstdcat` is used for verification. SQLite reads/writes during
+  extraction use the pure-Go [`modernc.org/sqlite`](https://pkg.go.dev/modernc.org/sqlite)
+  driver, so no cgo is required for that part.
+
+# Configuration
+
+Operational parameters are compile-time constants in
+[`internal/bsdaily/config.go`](internal/bsdaily/config.go):
+
+- `SnapshotBase` — `/srv/berlin.sneak.fs.blueskyarchive/.zfs/snapshot`
+- `TmpBase` — `/srv/storage/tmp` (fast scratch space for copies + per-day DBs)
+- `DailiesBase` — `/srv/berlin.sneak.fs.bluesky-dailies` (output root)
+- `MinTmpFreeGB` / `MinDailiesFreeGB` — `500` / `20`
+- `zstdCompressionLevel` — `15`
+- `sqliteCacheSizeKB` — `200000` (≈200MB)
+
+These are tuned for one specific production host; adjust and rebuild to run
+elsewhere.
+
+# Data Model
+
+The firehose schema (reference copy in [`schema.sql`](schema.sql)) centers
+on a `posts` table, with `users` keyed by DID and many-to-many junction
+tables linking posts to deduplicated `hashtags` and `urls`. An optional
+`media` table tracks downloaded blobs by content hash. `bsdaily` does not
+own this schema; it reflects whatever DDL exists in the source snapshot and
+selects forward from `posts` along the foreign-key relationships to produce a
+referentially-complete per-day slice.
+
+# Use Cases
+
+## Daily public archive
+
+Publish one small, immutable file per day to static HTTP (or IPFS, or a
+mirror network) so that anyone can fetch exactly the day(s) they want and
+re-merge them locally.
+
+## Backfilling a range
+
+Run `--from`/`--to` over a span of dates to (re)generate any missing daily
+dumps; already-published days are skipped, so the operation is resumable and
+safe to re-run.
+
+## Reconstituting a full database
+
+Collect any set of daily dumps and merge them with `INSERT OR IGNORE` to
+rebuild a complete, queryable SQLite database, then regenerate the aggregate
+statistics tables.
+
+# See Also
+
+## Links
+
+- Repo: [https://git.eeqj.de/sneak/bsdaily](https://git.eeqj.de/sneak/bsdaily)
+- Issues: [https://git.eeqj.de/sneak/bsdaily/issues](https://git.eeqj.de/sneak/bsdaily/issues)
+- Bluesky: [https://bsky.app](https://bsky.app)
+- zstd: [https://facebook.github.io/zstd/](https://facebook.github.io/zstd/)
+
+# Authors
+
+- [@sneak &lt;sneak@sneak.berlin&gt;](mailto:sneak@sneak.berlin)
+
+# License
+
+- [WTFPL](https://wtfpl.net)

REPO_POLICIES.md

@@ -1,6 +1,6 @@
 ---
 title: Repository Policies
-last_modified: 2026-03-18
+last_modified: 2026-07-06
 ---
 
 This document covers repository structure, tooling, and workflow standards. Code
@@ -34,10 +34,46 @@ style conventions are in separate documents:
   every file before committing. There are zero exceptions to this rule.
 
 - Every repo with software must have a root `Makefile` with these targets:
-  `make test`, `make lint`, `make fmt` (writes), `make fmt-check` (read-only),
-  `make check` (prereqs: `test`, `lint`, `fmt-check`), `make docker`, and
-  `make hooks` (installs pre-commit hook). A model Makefile is at
-  `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`.
+  `make bootstrap`, `make setup`, `make test`, `make lint`, `make fmt` (writes),
+  `make fmt-check` (read-only), `make check` (runs `test`, `lint`, `fmt-check`),
+  `make docker`, and `make hooks` (installs pre-commit hook). A model Makefile
+  is at `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`.
+
+- Repos follow the
+  [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
+  pattern: the implementation of each Makefile target lives in an executable
+  script in `script/` (`script/bootstrap`, `script/setup`, `script/test`,
+  `script/lint`, `script/fmt`, `script/fmt-check`, `script/check`,
+  `script/docker`), and the Makefile targets are thin shims that call them. The
+  scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they run in
+  minimal containers (e.g. alpine images have no bash); locate the repo root
+  with `$(cd "$(dirname "$0")/.." && pwd -P)` and `cd` there before acting. From
+  the standard's canonical set we use `bootstrap`, `setup` (make the repo ready
+  for development after a fresh clone: runs `bootstrap`, then
+  `install-precommit`, plus any repo-specific initialization), `test`, and
+  `cibuild`. `script/bootstrap` installs all dependencies idempotently and
+  assumes nothing is present: base tools come from nix, apt, brew, or apk
+  (detected in that order; apt runs noninteractive). For node it uses the
+  installed node if present; otherwise it installs a PINNED node version via
+  nvm, first installing nvm itself if missing — from a hash-verified GitHub
+  release archive (never `curl | sh`), with bash installed as an explicit
+  prerequisite since nvm requires bash. yarn is then pinned via
+  `corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
+  always exact versions. `script/cibuild` runs the CI build: it changes to the
+  repo root and runs `docker build .`; the Gitea workflow calls it. Four further
+  scripts are our own extensions to the standard: `script/check` runs
+  `script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
+  what the git pre-commit hook runs, and it calls `script/check`;
+  `script/install-precommit` installs the git pre-commit hook (the `make hooks`
+  target shims to it); and `script/projectname` (literally that filename) simply
+  outputs the project's name. Scripts that need the name call
+  `script/projectname` — e.g. `script/docker` assembles its image tag from it —
+  so those scripts stay byte-identical across all repos. Repo-type-specific
+  pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
+  `script/precommit`, not in the hook itself. Model scripts are at
+  `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
+  must document the provided scripts in an **Entrypoints** section (see the
+  README requirements below).
 
 - Always use Makefile targets (`make fmt`, `make test`, `make lint`, etc.)
   instead of invoking the underlying tools directly. The Makefile is the single
@@ -57,7 +93,11 @@ style conventions are in separate documents:
   as a build step so the build fails if the branch is not green. For non-server
   repos, the Dockerfile should bring up a development environment and run
   `make check`. For server repos, `make check` should run as an early build
-  stage before the final image is assembled.
+  stage before the final image is assembled. Dockerfiles install development
+  prerequisites by running `script/bootstrap` rather than duplicating installs
+  inline; COPY `script/` and the dependency manifests (`package.json` +
+  `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
+  layer stays cached until dependencies change.
 
 - **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
   repos use a multistage build where linting runs in an independent stage based
@@ -127,8 +167,9 @@ style conventions are in separate documents:
       artifacts or heavier dependencies.
 
 - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
-  runs `docker build .` on push. Since the Dockerfile already runs `make check`,
-  a successful build implies all checks pass.
+  runs `script/cibuild` (which runs `docker build .`) on push. Since the
+  Dockerfile already runs `make check`, a successful build implies all checks
+  pass.
 
 - Use platform-standard formatters: `black` for Python, `prettier` for
   JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -136,9 +177,11 @@ style conventions are in separate documents:
   Markdown (hard-wrap at 80 columns). Documentation and writing repos (Markdown,
   HTML, CSS) should also have `.prettierrc` and `.prettierignore`.
 
-- Pre-commit hook: `make check` if local testing is possible, otherwise
-  `make lint && make fmt-check`. The Makefile should provide a `make hooks`
-  target to install the pre-commit hook.
+- Pre-commit hook: runs `script/precommit`, which calls `script/check`. If local
+  testing is not possible in the repo, `script/precommit` may skip `script/test`
+  and run only `script/lint` and `script/fmt-check`. The hook is installed by
+  `script/install-precommit`; the Makefile must provide a `make hooks` target
+  that shims to it.
 
 - All repos with software must have tests that run via the platform-standard
   test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful
@@ -297,6 +340,10 @@ style conventions are in separate documents:
       "µPaaS is an MIT-licensed Go web application by @sneak that receives
       git-frontend webhooks and deploys applications via Docker in realtime."
     - **Getting Started**: Copy-pasteable install/usage code block.
+    - **Entrypoints**: Opens by stating that the repo adheres to the
+      [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
+      standard (with that link), then documents each provided `script/`
+      entrypoint and its purpose.
     - **Rationale**: Why does this exist?
     - **Design**: How is the program structured?
     - **TODO**: Update meticulously, even between commits. When planning, put
@@ -351,6 +398,9 @@ style conventions are in separate documents:
     - `README.md`, `.git`, `.gitignore`, `.editorconfig`
     - `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo)
     - `Makefile`
+    - `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`,
+      `lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
+      `install-precommit`)
     - `Dockerfile`, `.dockerignore`
     - `.gitea/workflows/check.yml`
     - Go: `go.mod`, `go.sum`, `.golangci.yml`

TODO.md

@@ -0,0 +1,43 @@
+# Workflow
+
+* branch (from `main`)
+* do the work in Next Step
+* move Next Step to the top of Completed Steps
+* move the top item of Future Steps into Next Step
+* commit (`TODO.md` changes in the same commit as the work)
+* merge to `main` if the branch is not protected, otherwise open a PR
+* push
+
+# Status
+
+pre-1.0
+
+# Next Step
+
+Bring the repo into policy compliance in one commit: add .gitignore,
+.dockerignore, .editorconfig, and .golangci.yml. Verify `make check` stays
+green with the new lint config.
+
+# Completed Steps
+
+- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
+  Makefile shims, README Entrypoints section
+- 2026-06-28: Fixed errcheck lint failures; added compilation smoke test;
+  tidied go.mod.
+- 2026-06-28: Added repo scaffolding: README, LICENSE, Makefile,
+  Dockerfile, REPO_POLICIES.md, and Gitea CI.
+- 2026-02-12: Fixed SQLite database locking by removing parallel
+  processing; fixed Linux build via golang.org/x/sys/unix Fadvise.
+- 2026-02-12: Optimized file copy for large databases; moved temp
+  directory to NVMe scratch storage.
+- 2026-02-11: Added date range support.
+- 2026-02-09: Initial implementation: single-day extraction, specific-date
+  targeting, faster pruning of throwaway database copies.
+
+# Future Steps
+
+- Add .gitignore, .dockerignore, .editorconfig, .golangci.yml (the Next
+  Step).
+- Expand tests beyond the compilation smoke test: unit tests for the
+  extraction, verification, and atomic-publish paths.
+- Cut a first SemVer release once compliance and test coverage land.

cmd/bsdaily/main.go

@@ -21,8 +21,8 @@ func main() {
 	var toFlag string
 
 	rootCmd := &cobra.Command{
-		Use:   "bsdaily",
-		Short: "Extract a single day's data from the latest daily snapshot",
+		Use:          "bsdaily",
+		Short:        "Extract a single day's data from the latest daily snapshot",
 		SilenceUsage: true,
 		RunE: func(cmd *cobra.Command, args []string) error {
 			hasDate := dateFlag != ""

go.mod

@@ -1,8 +1,9 @@
 module git.eeqj.de/sneak/bsdaily
 
-go 1.25.5
+go 1.26.4
 
 require (
+	github.com/spf13/cobra v1.10.2
 	golang.org/x/sys v0.41.0
 	modernc.org/sqlite v1.44.3
 )
@@ -14,7 +15,6 @@ require (
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/ncruces/go-strftime v1.0.0 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
-	github.com/spf13/cobra v1.10.2 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
 	golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect
 	modernc.org/libc v1.67.6 // indirect

internal/bsdaily/bsdaily_test.go

@@ -0,0 +1,21 @@
+package bsdaily
+
+import "testing"
+
+// TestCompiles is a minimal smoke test that references the package's exported
+// surface so that `go test` fails if the package stops compiling. It does not
+// touch the filesystem or any of the hard-coded production paths.
+func TestCompiles(t *testing.T) {
+	if DBFilename == "" || WALFilename == "" || SHMFilename == "" {
+		t.Fatal("expected database filename constants to be set")
+	}
+	if SnapshotBase == "" || TmpBase == "" || DailiesBase == "" {
+		t.Fatal("expected base path constants to be set")
+	}
+	if MinTmpFreeBytes == 0 || MinDailiesFreeBytes == 0 {
+		t.Fatal("expected free-space thresholds to be set")
+	}
+	if ErrNoPosts == nil {
+		t.Fatal("expected ErrNoPosts sentinel to be set")
+	}
+}

internal/bsdaily/copy.go

@@ -21,7 +21,11 @@ func CopyFile(src, dst string) (err error) {
 	if err != nil {
 		return fmt.Errorf("opening source %s: %w", src, err)
 	}
-	defer srcFile.Close()
+	defer func() {
+		if cerr := srcFile.Close(); cerr != nil {
+			slog.Warn("failed to close source file", "src", src, "error", cerr)
+		}
+	}()
 
 	srcInfo, err := srcFile.Stat()
 	if err != nil {

internal/bsdaily/copy_linux.go

@@ -3,8 +3,8 @@
 package bsdaily
 
 import (
-	"os"
 	"golang.org/x/sys/unix"
+	"os"
 )
 
 func applyFileAdvice(file *os.File, size int64) {

internal/bsdaily/extract.go

@@ -29,7 +29,11 @@ func ExtractDay(srcDBPath, dstDBPath string, targetDay time.Time) error {
 	if err != nil {
 		return fmt.Errorf("opening destination database: %w", err)
 	}
-	defer db.Close()
+	defer func() {
+		if cerr := db.Close(); cerr != nil {
+			slog.Warn("failed to close destination database", "path", dstDBPath, "error", cerr)
+		}
+	}()
 
 	// Attach source database
 	if _, err := db.Exec("ATTACH DATABASE ? AS src", srcDBPath); err != nil {
@@ -42,7 +46,11 @@ func ExtractDay(srcDBPath, dstDBPath string, targetDay time.Time) error {
 	if err != nil {
 		return fmt.Errorf("reading source schema: %w", err)
 	}
-	defer rows.Close()
+	defer func() {
+		if cerr := rows.Close(); cerr != nil {
+			slog.Warn("failed to close schema rows", "error", cerr)
+		}
+	}()
 
 	var ddlStatements []string
 	for rows.Next() {
@@ -69,7 +77,9 @@ func ExtractDay(srcDBPath, dstDBPath string, targetDay time.Time) error {
 	}
 	defer func() {
 		if err != nil {
-			tx.Rollback()
+			if rerr := tx.Rollback(); rerr != nil && !errors.Is(rerr, sql.ErrTxDone) {
+				slog.Warn("failed to roll back transaction", "error", rerr)
+			}
 		}
 	}()
 
@@ -132,7 +142,11 @@ func ExtractDay(srcDBPath, dstDBPath string, targetDay time.Time) error {
 	if err != nil {
 		return fmt.Errorf("reading source indexes: %w", err)
 	}
-	defer idxRows.Close()
+	defer func() {
+		if cerr := idxRows.Close(); cerr != nil {
+			slog.Warn("failed to close index rows", "error", cerr)
+		}
+	}()
 
 	var idxStatements []string
 	for idxRows.Next() {

internal/bsdaily/run.go

@@ -9,6 +9,15 @@ import (
 	"time"
 )
 
+// cleanup removes a temporary file, logging a warning if removal fails so
+// that leaked scratch files are surfaced rather than silently ignored. A
+// missing file is not an error.
+func cleanup(path string) {
+	if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
+		slog.Warn("failed to remove temporary file", "path", path, "error", err)
+	}
+}
+
 func Run(targetDates []time.Time) error {
 	snapshotDir, snapshotDate, err := FindLatestDailySnapshot()
 	if err != nil {
@@ -100,7 +109,7 @@ func Run(targetDates []time.Time) error {
 		if err := ExtractDay(dstDB, extractedDB, targetDay); err != nil {
 			if errors.Is(err, ErrNoPosts) {
 				slog.Warn("no posts found, skipping day", "date", dayStr)
-				os.Remove(extractedDB)
+				cleanup(extractedDB)
 				skipped++
 				continue
 			}
@@ -109,7 +118,7 @@ func Run(targetDates []time.Time) error {
 
 		// Dump to SQL and compress
 		if err := os.MkdirAll(outputDir, 0755); err != nil {
-			os.Remove(extractedDB)
+			cleanup(extractedDB)
 			return fmt.Errorf("creating output directory %s: %w", outputDir, err)
 		}
 
@@ -117,35 +126,35 @@ func Run(targetDates []time.Time) error {
 
 		slog.Info("dumping and compressing", "tmp_output", outputTmp)
 		if err := DumpAndCompress(extractedDB, outputTmp); err != nil {
-			os.Remove(outputTmp)
-			os.Remove(extractedDB)
+			cleanup(outputTmp)
+			cleanup(extractedDB)
 			return fmt.Errorf("dump and compress for %s: %w", dayStr, err)
 		}
 
 		slog.Info("verifying compressed output")
 		if err := VerifyOutput(outputTmp); err != nil {
-			os.Remove(outputTmp)
-			os.Remove(extractedDB)
+			cleanup(outputTmp)
+			cleanup(extractedDB)
 			return fmt.Errorf("verification failed for %s: %w", dayStr, err)
 		}
 
 		// Atomic rename to final path
 		slog.Info("renaming to final output", "from", outputTmp, "to", outputFinal)
 		if err := os.Rename(outputTmp, outputFinal); err != nil {
-			os.Remove(outputTmp)
-			os.Remove(extractedDB)
+			cleanup(outputTmp)
+			cleanup(extractedDB)
 			return fmt.Errorf("atomic rename for %s: %w", dayStr, err)
 		}
 
 		info, err := os.Stat(outputFinal)
 		if err != nil {
-			os.Remove(extractedDB)
+			cleanup(extractedDB)
 			return fmt.Errorf("stat final output: %w", err)
 		}
 		slog.Info("day completed", "date", dayStr, "path", outputFinal, "size_bytes", info.Size())
 
 		// Remove extracted DB to reclaim space immediately
-		os.Remove(extractedDB)
+		cleanup(extractedDB)
 		processed++
 	}
 

internal/bsdaily/verify.go

@@ -1,12 +1,26 @@
 package bsdaily
 
 import (
+	"errors"
 	"fmt"
 	"log/slog"
+	"os"
 	"os/exec"
 	"strings"
 )
 
+// killCat terminates the zstdcat process, ignoring the benign case where it
+// has already exited (e.g. after receiving SIGPIPE when head closed the pipe)
+// and logging any other failure.
+func killCat(cmd *exec.Cmd) {
+	if cmd.Process == nil {
+		return
+	}
+	if err := cmd.Process.Kill(); err != nil && !errors.Is(err, os.ErrProcessDone) {
+		slog.Warn("failed to kill zstdcat process", "error", err)
+	}
+}
+
 func VerifyOutput(path string) error {
 	slog.Info("running zstdmt integrity check")
 	testCmd := exec.Command("zstdmt", "--test", path)
@@ -34,18 +48,18 @@ func VerifyOutput(path string) error {
 		return fmt.Errorf("starting zstdcat: %w", err)
 	}
 	if err := headCmd.Start(); err != nil {
-		catCmd.Process.Kill() // Clean up if head fails to start
+		killCat(catCmd) // Clean up if head fails to start
 		return fmt.Errorf("starting head: %w", err)
 	}
 
 	// Wait for head first (it will exit when it has enough lines)
 	if err := headCmd.Wait(); err != nil {
-		catCmd.Process.Kill()
+		killCat(catCmd)
 		return fmt.Errorf("head command failed: %w", err)
 	}
 
 	// Kill zstdcat since head closed the pipe (expected SIGPIPE)
-	catCmd.Process.Kill()
+	killCat(catCmd)
 	_ = catCmd.Wait() // Reap the process
 
 	content := headOut.String()

script/bootstrap

@@ -0,0 +1,73 @@
+#!/bin/sh
+# script/bootstrap: install all dependencies needed to build and develop
+# this repo. Idempotent: every install is guarded by a check so already
+# installed tools are skipped. Base tooling comes from nix, apt, brew,
+# or apk (detected in that order); assumes NOTHING is present (not git,
+# make, or go).
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+PKGMGR=""
+SUDO=""
+
+detect_pkgmgr() {
+    [ -n "$PKGMGR" ] && return 0
+    if command -v nix-env >/dev/null 2>&1; then
+        PKGMGR="nix"
+    elif command -v apt-get >/dev/null 2>&1; then
+        PKGMGR="apt"
+    elif command -v brew >/dev/null 2>&1; then
+        PKGMGR="brew"
+    elif command -v apk >/dev/null 2>&1; then
+        PKGMGR="apk"
+    else
+        echo "bootstrap: no supported package manager (nix, apt, brew, apk)" >&2
+        exit 1
+    fi
+    if [ "$PKGMGR" = "apt" ]; then
+        export DEBIAN_FRONTEND=noninteractive
+        if [ "$(id -u)" != "0" ]; then
+            SUDO="sudo"
+        fi
+    fi
+}
+
+# pkg_install <nix-attr> <apt-pkg> <brew-formula> <apk-pkg>
+pkg_install() {
+    detect_pkgmgr
+    case "$PKGMGR" in
+        nix) nix-env -iA "nixpkgs.$1" ;;
+        apt) $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2" ;;
+        brew) brew install "$3" ;;
+        apk) apk add --no-cache "$4" ;;
+    esac
+}
+
+missing() {
+    ! command -v "$1" >/dev/null 2>&1
+}
+
+main() {
+    cd "$ROOT"
+
+    # Base tooling (every repo)
+    if missing git; then pkg_install git git git git; fi
+    if missing make; then pkg_install gnumake make make make; fi
+
+    # Go toolchain
+    if missing go; then pkg_install go golang go go; fi
+
+    # golangci-lint: packaged in nix, brew, and apk. There is no apt
+    # package; on apt systems install it manually from a hash-verified
+    # GitHub release archive (never curl | sh).
+    if missing golangci-lint; then
+        pkg_install golangci-lint golangci-lint golangci-lint golangci-lint
+    fi
+
+    go mod download
+
+    echo "bootstrap complete"
+}
+
+main "$@"

script/check

@@ -0,0 +1,15 @@
+#!/bin/sh
+# script/check: run all checks (test, lint, fmt-check). Our own
+# extension to scripts-to-rule-them-all. Must not modify any files.
+# Generic: usually needs no adaptation.
+set -eu
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+
+main() {
+    "$SCRIPT_DIR/test"
+    "$SCRIPT_DIR/lint"
+    "$SCRIPT_DIR/fmt-check"
+}
+
+main "$@"

script/cibuild

@@ -0,0 +1,14 @@
+#!/bin/sh
+# script/cibuild: run the CI build. The Dockerfile runs script/check
+# (via make check), so a successful build implies all checks pass.
+# Generic: needs no adaptation. The Gitea workflow runs this on push.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    docker build .
+}
+
+main "$@"

script/docker

@@ -0,0 +1,15 @@
+#!/bin/sh
+# script/docker: build the Docker image tagged with the project name.
+# Identical in all repos; the tag comes from script/projectname.
+# Generic: needs no adaptation.
+set -eu
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    docker build -t "$("$SCRIPT_DIR/projectname")" .
+}
+
+main "$@"

script/fmt

@@ -0,0 +1,12 @@
+#!/bin/sh
+# script/fmt: format all files (writes).
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    go fmt ./...
+}
+
+main "$@"

script/fmt-check

@@ -0,0 +1,18 @@
+#!/bin/sh
+# script/fmt-check: check formatting (read-only). Same scope as
+# script/fmt, but fails instead of writing.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    unformatted="$(gofmt -l .)"
+    if [ -n "$unformatted" ]; then
+        echo "Files not formatted:" >&2
+        echo "$unformatted" >&2
+        exit 1
+    fi
+}
+
+main "$@"

script/install-precommit

@@ -0,0 +1,17 @@
+#!/bin/sh
+# script/install-precommit: install the git pre-commit hook that runs
+# script/precommit. Our own extension to scripts-to-rule-them-all.
+# Generic: needs no adaptation.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    hook=".git/hooks/pre-commit"
+    printf '#!/bin/sh\nset -e\nscript/precommit\n' > .git/hooks/pre-commit
+    chmod +x .git/hooks/pre-commit
+    echo "pre-commit hook installed: runs script/precommit"
+}
+
+main "$@"

script/lint

@@ -0,0 +1,12 @@
+#!/bin/sh
+# script/lint: run the linter.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    golangci-lint run ./...
+}
+
+main "$@"

script/precommit

@@ -0,0 +1,21 @@
+#!/bin/sh
+# script/precommit: run by the git pre-commit hook; fails the commit if
+# checks fail. Our own extension to scripts-to-rule-them-all. Go repo
+# extras: go mod tidy and go fmt must leave the tree unchanged.
+set -eu
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    go mod tidy
+    go fmt ./...
+    git diff --exit-code -- go.mod go.sum || {
+        echo "go mod tidy changed files; please stage and retry"
+        exit 1
+    }
+    "$SCRIPT_DIR/check"
+}
+
+main "$@"

script/projectname

@@ -0,0 +1,12 @@
+#!/bin/sh
+# script/projectname: output the name of this project. Our own
+# extension to scripts-to-rule-them-all. Other scripts that need the
+# name (e.g. script/docker) call this, so they can stay identical
+# across all repos.
+set -eu
+
+main() {
+    echo "bsdaily"
+}
+
+main "$@"

script/setup

@@ -0,0 +1,14 @@
+#!/bin/sh
+# script/setup: set up the repo for development after a fresh clone:
+# installs dependencies (script/bootstrap) and the git pre-commit hook.
+# Add any repo-specific initialization (db init, .env template) here.
+set -eu
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+
+main() {
+    "$SCRIPT_DIR/bootstrap"
+    "$SCRIPT_DIR/install-precommit"
+}
+
+main "$@"

script/test

@@ -0,0 +1,18 @@
+#!/bin/sh
+# script/test: run the test suite. Quiet on success; on failure, rerun
+# verbosely for full diagnostic output (the exit 1 ensures the rerun
+# never turns a failure into a pass).
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    go test -race -timeout 30s ./... || {
+        echo "--- Rerunning with -v for details ---"
+        go test -race -timeout 30s -v ./...
+        exit 1
+    }
+}
+
+main "$@"