Author SHA1 Message Date
sneak 9b0186c481 Say script/bootstrap pins node and yarn only when absent (refs #7)
check / check (push) Waiting to run
The README Entrypoints line and the TODO.md Completed Steps entry said
script/bootstrap installs a pinned node and yarn. It installs them at the
pinned versions only when they are missing and uses a node or yarn already
installed whatever its version, as its header comment says.

Model: opus-5-5
2026-10-06 08:25:55 +00:00
clawbot 8125d4aebf Reformat the existing Markdown with prettier (closes #7)
check / check (push) Waiting to run
The output of make fmt after the previous commit: README.md and TODO.md
are rewrapped at 80 columns, and the TODO.md Workflow list uses -
markers. No wording changes. REPO_POLICIES.md was already formatted.

Model: opus-5-5
2026-10-06 06:31:14 +00:00
clawbot 47e48a2529 Format Markdown with prettier in script/fmt and script/fmt-check (refs #7)
check / check (push) Waiting to run
script/fmt now also runs prettier over every Markdown file, and
script/fmt-check checks them without writing. prettier is pinned in
package.json and yarn.lock. script/bootstrap keeps its Go and apt
handling and gains the canonical pinned node (nvm from a hash-checked
archive) and yarn (corepack) install, then installs the locked
packages. Both fmt scripts find yarn the canonical way, sourcing nvm
when yarn is not on PATH. The new files and the yarn lookup come from
sneak/prompts at cc440118c876. The vendored REPO_POLICIES.md already
passes prettier, so it is not ignored. The Markdown is reformatted in
the next commit.

Deviation: package.json drops the canonical "license": "MIT" line; this
repo is WTFPL.

Model: opus-5-5
2026-10-06 06:30:59 +00:00
clawbot 17b6b84322 Bring the repo up to the standard layout (closes #1)
check / check (push) Waiting to run
Vendors .gitignore, .dockerignore, .editorconfig, REPO_POLICIES.md,
script/cibuild, script/docker, script/lint, script/test and the CI
workflow byte-identical from sneak/prompts commit cc440118c876; the
two ignore files then add this repo's own build output, root-anchored
so cmd/bsdaily/ stays in. The Dockerfile gains a lint phase on the
golangci-lint image already pinned and a test phase on the Debian Go
image with sqlite3 and zstd; the build stage copies a file from each,
so a plain docker build cannot skip them. Nothing installs
golangci-lint on the host. On apt, script/bootstrap refreshes the
package lists once before its first install.

.golangci.yml and the lint cleanup: #6

Model: opus-5-5
2026-10-06 08:25:03 +02:00
20 changed files with 940 additions and 288 deletions
+79
View File
@@ -0,0 +1,79 @@
# .dockerignore does NOT use .gitignore semantics. Docker matches with
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
# `/` and an unprefixed pattern is anchored at the context root. Every
# depth-independent pattern therefore needs `**/`, or `config/.env` and
# `certs/server.key` still ship while this file reads as solved. Only
# genuinely root-anchored entries go unprefixed. Never transplant these
# into .gitignore, where `**/` is wrong.
#
# Matching is case-sensitive, so secrets use character ranges rather
# than an ALL-CAPS twin, which would still miss `Server.Key`.
#
# Extend with this repo's own host-built artifacts, written anchored:
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
# Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
# KNOWN GAP: a repo running agents in subdirectories still ships
# `services/api/.claude/` and must add its own anchored entry.
.claude
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Re-include a committed template with a negation if the
# build needs one: `!docs/example.env`.
**/*.[eE][nN][vV]
**/.[eE][nN][vV].*
**/.[eE][nN][vV][rR][cC]
# Private keys and the bundles carrying them. Public certificates
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
**/*.[pP][eE][mM]
**/*.[kK][eE][yY]
**/*.[pP]12
**/*.[pP][fF][xX]
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in.
**/node_modules
# OS metadata.
**/.DS_Store
**/Thumbs.db
# Editor state: never a build input, and it churns COPY.
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea
**/.vscode
**/*.sublime-*
# This repo's host-built artifacts: `make`, `go test -c` and
# `make test-coverage`.
/bsdaily
/*.test
/coverage.out
/coverage.html
+18
View File
@@ -0,0 +1,18 @@
root = true
[*]
indent_style = space
indent_size = 4
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
[Makefile]
indent_style = tab
[*.go]
indent_style = tab
# This repository's own sections, such as one for another language it
# uses, go below this comment, and a re-vendor keeps them.
+7 -12
View File
@@ -1,14 +1,9 @@
name: check name: check
on: on: [push]
push:
branches: [main]
pull_request:
branches: [main]
jobs: jobs:
check: check:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
# actions/checkout v4, 2024-09-16 # actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- name: Build and check - run: script/cibuild
run: script/cibuild
+58
View File
@@ -0,0 +1,58 @@
# OS
.DS_Store
Thumbs.db
# Editors
*.swp
*.swo
*~
*.bak
.idea/
.vscode/
*.sublime-*
# Agent scratch (worktrees of this repo, created and destroyed by
# in-flight tooling). Unanchored: .gitignore patterns already match at
# every depth, so no prefix is wanted here. This is not a .dockerignore
# entry and must not be given a `**/` prefix on the way into one.
.claude/
# Node
node_modules/
# Secrets. Unanchored like every entry above, so each matches at every
# depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation at the end of this file, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, such as its build outputs, go below
# this comment, and a re-vendor keeps them. Anchor a binary built at the
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
# Go: logs, test binaries, coverage output, and the binary `make` builds.
*.log
*.test
*.out
/coverage.html
/bsdaily
+2
View File
@@ -0,0 +1,2 @@
node_modules/
yarn.lock
+4
View File
@@ -0,0 +1,4 @@
{
"tabWidth": 4,
"proseWrap": "always"
}
+34 -17
View File
@@ -1,9 +1,9 @@
# Lint stage # Lint phase. The linter is invoked directly rather than through `make
# golangci/golangci-lint:v2.12.2-alpine # lint` or `script/lint`, which are themselves a docker build and would
# recurse into a daemon that does not exist in a build step.
# golangci/golangci-lint:v2.12.2-alpine, 2026-06-28
FROM golangci/golangci-lint:v2.12.2-alpine@sha256:91b27804074a0bacea298707f016911e60cf0cdbc6c7bf5ccacb5f0606d18d60 AS lint FROM golangci/golangci-lint:v2.12.2-alpine@sha256:91b27804074a0bacea298707f016911e60cf0cdbc6c7bf5ccacb5f0606d18d60 AS lint
RUN apk add --no-cache make build-base
WORKDIR /src WORKDIR /src
# Copy go mod files first for better layer caching # Copy go mod files first for better layer caching
@@ -13,22 +13,42 @@ RUN go mod download
# Copy source code # Copy source code
COPY . . COPY . .
# Run formatting check and linter RUN golangci-lint run ./...
RUN make fmt-check
RUN make lint
# Build stage # Test phase, invoking `go test` directly for the same reason. -race
# golang:1.26.4-alpine # needs cgo and so a C compiler, which the Debian Go image ships and the
# alpine one does not. The tool shells out to sqlite3, zstdmt and
# zstdcat, so tests that run it need them installed.
# golang:1.26.4-trixie, 2026-10-05
FROM golang:1.26.4-trixie@sha256:68b7145ec43d1820b9a56704554b53d1520aa2a15cb5233e374188a31b2a1bce AS test
RUN apt-get update \
&& apt-get install -y --no-install-recommends sqlite3 zstd \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed. A plain `docker build .` builds only the
# last stage and what it depends on, so the runtime stage stays last.
# golang:1.26.4-alpine, 2026-06-28
FROM golang:1.26.4-alpine@sha256:3ad57304ad93bbec8548a0437ad9e06a455660655d9af011d58b993f6f615648 AS builder FROM golang:1.26.4-alpine@sha256:3ad57304ad93bbec8548a0437ad9e06a455660655d9af011d58b993f6f615648 AS builder
# Depend on lint stage passing
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
ARG VERSION=dev ARG VERSION=dev
# Install build deps plus the sqlite3 and zstd CLIs the tests/tool shell out to
RUN apk add --no-cache make build-base sqlite zstd
WORKDIR /src WORKDIR /src
# Copy go mod files first for better layer caching # Copy go mod files first for better layer caching
@@ -38,14 +58,11 @@ RUN go mod download
# Copy source code # Copy source code
COPY . . COPY . .
# Run tests
RUN make test
# Build (pure Go, no CGO required since we use modernc.org/sqlite) # Build (pure Go, no CGO required since we use modernc.org/sqlite)
RUN CGO_ENABLED=0 go build -o /bsdaily ./cmd/bsdaily RUN CGO_ENABLED=0 go build -o /bsdaily ./cmd/bsdaily
# Runtime stage # Runtime stage
# alpine:3.21 # alpine:3.21, 2026-06-28
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
# bsdaily shells out to sqlite3, zstdmt, zstdcat at runtime # bsdaily shells out to sqlite3, zstdmt, zstdcat at runtime
-1
View File
@@ -46,7 +46,6 @@ clean:
# Install dependencies. # Install dependencies.
deps: deps:
go mod download go mod download
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
# Run tests with coverage. # Run tests with coverage.
test-coverage: test-coverage:
+124 -118
View File
@@ -1,28 +1,26 @@
# bsdaily # bsdaily
[bsdaily](https://git.eeqj.de/sneak/bsdaily) is a command-line utility [bsdaily](https://git.eeqj.de/sneak/bsdaily) is a command-line utility written
written in [Go](https://golang.org) that carves a single day (or a range of in [Go](https://golang.org) that carves a single day (or a range of days) of
days) of [Bluesky](https://bsky.app) firehose data out of a large, [Bluesky](https://bsky.app) firehose data out of a large, continuously-growing
continuously-growing SQLite database and writes it out as a self-contained, SQLite database and writes it out as a self-contained,
[zstd](https://facebook.github.io/zstd/)-compressed SQL dump. The dumps are [zstd](https://facebook.github.io/zstd/)-compressed SQL dump. The dumps are
named by date (e.g. `2026-06-27.sql.zst`), organized into per-month named by date (e.g. `2026-06-27.sql.zst`), organized into per-month directories,
directories, and are designed to be published, archived, mirrored, and later and are designed to be published, archived, mirrored, and later re-merged back
re-merged back into a single database. into a single database.
The source database is read from a read-only [ZFS](https://openzfs.org) The source database is read from a read-only [ZFS](https://openzfs.org)
snapshot, so extraction never contends with the live firehose ingester that snapshot, so extraction never contends with the live firehose ingester that is
is writing to the original database. The tool is operationally writing to the original database. The tool is operationally conservative: it
conservative: it checks free disk space before starting, copies the snapshot checks free disk space before starting, copies the snapshot to fast scratch
to fast scratch storage, processes one day at a time to avoid SQLite lock storage, processes one day at a time to avoid SQLite lock contention, verifies
contention, verifies every compressed output before publishing it, and every compressed output before publishing it, and writes output atomically via a
writes output atomically via a temp-file-and-rename so a partial run never temp-file-and-rename so a partial run never leaves a corrupt `.sql.zst` behind.
leaves a corrupt `.sql.zst` behind.
This project was written by [@sneak](https://sneak.berlin) to produce a This project was written by [@sneak](https://sneak.berlin) to produce a daily,
daily, mergeable, publicly-mirrorable archive of the Bluesky firehose. It is mergeable, publicly-mirrorable archive of the Bluesky firehose. It is currently
currently a one-person effort. The current version is pre-1.0 and there has a one-person effort. The current version is pre-1.0 and there has not yet been a
not yet been a versioned release; [SemVer](https://semver.org) will be used versioned release; [SemVer](https://semver.org) will be used for releases.
for releases.
# Build Status # Build Status
@@ -33,14 +31,14 @@ branch must always be green.
Primary development happens on a privately-run Gitea instance at Primary development happens on a privately-run Gitea instance at
[https://git.eeqj.de/sneak/bsdaily](https://git.eeqj.de/sneak/bsdaily) and [https://git.eeqj.de/sneak/bsdaily](https://git.eeqj.de/sneak/bsdaily) and
issues are [tracked issues are [tracked there](https://git.eeqj.de/sneak/bsdaily/issues).
there](https://git.eeqj.de/sneak/bsdaily/issues).
Changes must always be formatted with a standard `go fmt`, syntactically Changes must always be formatted with `make fmt` (`go fmt` for Go, prettier for
valid, and must pass the linting defined in the repository (presently the Markdown), syntactically valid, and must pass the linting defined in the
`golangci-lint` defaults), which can be run with a `make lint`. The `main` repository (presently the `golangci-lint` defaults), which can be run with a
branch is protected and all changes must be made via [pull `make lint`. The `main` branch is protected and all changes must be made via
requests](https://git.eeqj.de/sneak/bsdaily/pulls) and pass CI to be merged. [pull requests](https://git.eeqj.de/sneak/bsdaily/pulls) and pass CI to be
merged.
See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards, See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
tooling requirements, and workflow conventions. tooling requirements, and workflow conventions.
@@ -50,48 +48,55 @@ tooling requirements, and workflow conventions.
This repository adheres to the This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call development workflow, and the Makefile targets are thin shims that call them. We
them. We provide: provide:
- `script/bootstrap` — install all development dependencies (go, - `script/bootstrap` — install all development dependencies (go, Go module
golangci-lint, Go module download) download, node and yarn at pinned versions when absent, and the prettier
pinned in `package.json` and `yarn.lock`); a node or yarn already installed is
used whatever its version; the linter is not installed on the host
- `script/setup` — make a fresh clone ready for development: runs - `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit` `script/bootstrap`, then `script/install-precommit`
- `script/projectname` — print the project name (used for the Docker - `script/projectname` — print the project name (used for the Docker image tags)
image tag) - `script/test` — build the Dockerfile's `test` phase, which runs the test suite
- `script/test` — run the test suite (verbose rerun on failure) with `-race` (verbose rerun on failure)
- `script/lint` — run `golangci-lint run ./...` - `script/lint` — build the Dockerfile's `lint` phase, which runs
- `script/fmt` — format all code (writes) `golangci-lint run ./...`
- `script/fmt-check` — check formatting (read-only) - `script/fmt` — format the Go code with `go fmt` and every Markdown file with
- `script/check` — run `script/test`, `script/lint`, and prettier (writes)
`script/fmt-check` - `script/fmt-check` — check the Go formatting with `gofmt` and the Markdown
- `script/docker` — build the Docker image tagged via formatting with prettier (read-only)
`script/projectname` - `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
- `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile - `script/docker` — build the Docker image tagged via `script/projectname`; the
runs the checks) build runs the `lint` and `test` phases first
- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must - `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
not change files), then `script/check` then builds the Docker image tagged via `script/projectname`
- `script/install-precommit` — install the git pre-commit hook that - `script/precommit` — pre-commit gate: `go mod tidy` (must not change `go.mod`
runs `script/precommit` or `go.sum`) and `go fmt`, then `script/check`
- `script/install-precommit` — install the git pre-commit hook that runs
`script/precommit`
Every Docker build in `script/` is uncached, so the `lint` and `test` phases
always run rather than being served from the build cache.
# Problem Statement # Problem Statement
A Bluesky firehose ingester writes every observed post (and associated A Bluesky firehose ingester writes every observed post (and associated users,
users, hashtags, URLs, and media references) into a single ever-growing hashtags, URLs, and media references) into a single ever-growing SQLite
SQLite database, `firehose.db`. This database has several properties that database, `firehose.db`. This database has several properties that make it
make it awkward to publish or archive directly: awkward to publish or archive directly:
- It is **large and always growing**, so re-publishing the whole thing every - It is **large and always growing**, so re-publishing the whole thing every day
day is wasteful. is wasteful.
- It is **continuously written**, so reading from it directly risks lock - It is **continuously written**, so reading from it directly risks lock
contention with the live ingester and inconsistent reads. contention with the live ingester and inconsistent reads.
- It is **monolithic**, so there is no natural unit at which to mirror, - It is **monolithic**, so there is no natural unit at which to mirror, share,
share, or distribute "just yesterday's posts". or distribute "just yesterday's posts".
What is wanted instead is a stable, immutable, per-day artifact: a small What is wanted instead is a stable, immutable, per-day artifact: a small file
file containing exactly one calendar day of firehose data, cheap to publish, containing exactly one calendar day of firehose data, cheap to publish, cheap to
cheap to mirror, and trivially re-mergeable into a full database by anyone mirror, and trivially re-mergeable into a full database by anyone who collects a
who collects a set of them. set of them.
# Proposed Solution # Proposed Solution
@@ -101,81 +106,81 @@ A tool, `bsdaily`, that:
filesystem, so it reads from a consistent point-in-time copy that the live filesystem, so it reads from a consistent point-in-time copy that the live
ingester cannot be writing to; ingester cannot be writing to;
- copies the snapshot's database files to fast scratch storage; - copies the snapshot's database files to fast scratch storage;
- **extracts** a single day's `posts` (and all rows reachable from them) into - **extracts** a single day's `posts` (and all rows reachable from them) into a
a fresh, minimal per-day SQLite database; fresh, minimal per-day SQLite database;
- **dumps** that per-day database to SQL and pipes it through multithreaded - **dumps** that per-day database to SQL and pipes it through multithreaded zstd
zstd compression; compression;
- **verifies** the compressed output (zstd integrity check plus a sanity - **verifies** the compressed output (zstd integrity check plus a sanity check
check that the decompressed stream actually looks like SQL); that the decompressed stream actually looks like SQL);
- **publishes** the result atomically as - **publishes** the result atomically as
`DailiesBase/YYYY-MM/YYYY-MM-DD.sql.zst`. `DailiesBase/YYYY-MM/YYYY-MM-DD.sql.zst`.
Each daily dump is emitted with `INSERT` statements over the full schema Each daily dump is emitted with `INSERT` statements over the full schema
(including the deduplicated `users`, `hashtags`, and `urls` lookup tables), (including the deduplicated `users`, `hashtags`, and `urls` lookup tables), so
so any collection of daily dumps can be merged into a single database by any collection of daily dumps can be merged into a single database by rewriting
rewriting `INSERT INTO` to `INSERT OR IGNORE INTO` and replaying them in `INSERT INTO` to `INSERT OR IGNORE INTO` and replaying them in sequence. Two
sequence. Two helper scripts ([`merge_daily_dumps.sh`](merge_daily_dumps.sh) helper scripts ([`merge_daily_dumps.sh`](merge_daily_dumps.sh) and
and [`regenerate_auxiliary_tables.sql`](regenerate_auxiliary_tables.sql)) are [`regenerate_auxiliary_tables.sql`](regenerate_auxiliary_tables.sql)) are
included to do exactly this and to rebuild the aggregate statistics included to do exactly this and to rebuild the aggregate statistics
(`use_count`, `first_seen`, user `resolved_at`/`updated_at`) afterward. (`use_count`, `first_seen`, user `resolved_at`/`updated_at`) afterward.
# Design Goals # Design Goals
- **Never disturb the live ingester.** All reads come from a ZFS snapshot, - **Never disturb the live ingester.** All reads come from a ZFS snapshot, never
never the live database. the live database.
- **Crash-safe, idempotent runs.** Output is written to a temp file and - **Crash-safe, idempotent runs.** Output is written to a temp file and
atomically renamed; a day whose final output already exists is skipped, so atomically renamed; a day whose final output already exists is skipped, so
re-running a range is safe and resumable. re-running a range is safe and resumable.
- **Mergeable output.** Daily dumps re-combine losslessly into a full - **Mergeable output.** Daily dumps re-combine losslessly into a full database
database via `INSERT OR IGNORE`. via `INSERT OR IGNORE`.
- **Operationally cautious.** Free-space preflight checks on both scratch and - **Operationally cautious.** Free-space preflight checks on both scratch and
output filesystems; explicit verification of every artifact before it is output filesystems; explicit verification of every artifact before it is
published. published.
- **Fast where it's free.** Large snapshot copies use a 256MiB buffer, - **Fast where it's free.** Large snapshot copies use a 256MiB buffer,
pre-allocate the destination, and (on Linux) issue `posix_fadvise` pre-allocate the destination, and (on Linux) issue `posix_fadvise`
sequential/willneed hints; extraction uses aggressive, sequential/willneed hints; extraction uses aggressive, crash-unsafe-by-design
crash-unsafe-by-design SQLite pragmas because the working data lives only SQLite pragmas because the working data lives only in disposable scratch
in disposable scratch space. space.
# Non-Goals # Non-Goals
- **Real-time export.** `bsdaily` operates on daily snapshots; the freshest - **Real-time export.** `bsdaily` operates on daily snapshots; the freshest day
day it can produce is the snapshot date minus one. it can produce is the snapshot date minus one.
- **Schema ownership.** The schema is defined by the upstream firehose - **Schema ownership.** The schema is defined by the upstream firehose ingester;
ingester; [`schema.sql`](schema.sql) is included for reference only. [`schema.sql`](schema.sql) is included for reference only. `bsdaily` copies
`bsdaily` copies whatever table and index DDL it finds in the source. whatever table and index DDL it finds in the source.
- **Cross-platform deployment.** It is built and run on Linux (the - **Cross-platform deployment.** It is built and run on Linux (the free-space
free-space check and fadvise hints use `golang.org/x/sys/unix`; a non-Linux check and fadvise hints use `golang.org/x/sys/unix`; a non-Linux build
build compiles but is a no-op for the fadvise hints). The hard-coded paths compiles but is a no-op for the fadvise hints). The hard-coded paths assume
assume the production host's ZFS layout. the production host's ZFS layout.
# How It Works # How It Works
A single run proceeds as follows: A single run proceeds as follows:
1. **Find the snapshot.** Scan `SnapshotBase` for directories matching 1. **Find the snapshot.** Scan `SnapshotBase` for directories matching
`zfs-auto-snap_daily-YYYY-MM-DD-NNNN`, pick the most recent, and confirm `zfs-auto-snap_daily-YYYY-MM-DD-NNNN`, pick the most recent, and confirm it
it contains `firehose.db`. contains `firehose.db`.
2. **Determine target days.** Default to the snapshot date minus one day; 2. **Determine target days.** Default to the snapshot date minus one day; or use
or use `--date`, or every day in the inclusive `--from`/`--to` range. `--date`, or every day in the inclusive `--from`/`--to` range.
3. **Preflight disk space.** Require at least 500GiB free on the scratch 3. **Preflight disk space.** Require at least 500GiB free on the scratch
filesystem and 20GiB free on the output filesystem. filesystem and 20GiB free on the output filesystem.
4. **Copy the database to scratch.** Copy `firehose.db`, its `-wal`, and (if 4. **Copy the database to scratch.** Copy `firehose.db`, its `-wal`, and (if
present) its `-shm` from the snapshot into a fresh temp directory under present) its `-shm` from the snapshot into a fresh temp directory under
`TmpBase`. `TmpBase`.
5. **Per day**, processed strictly one at a time to avoid SQLite contention: 5. **Per day**, processed strictly one at a time to avoid SQLite contention:
- skip the day if its final output file already exists; - skip the day if its final output file already exists;
- `ATTACH` the copied source DB to a new empty per-day DB, recreate the - `ATTACH` the copied source DB to a new empty per-day DB, recreate the
table DDL, and `INSERT ... SELECT` the target day's `posts` plus all table DDL, and `INSERT ... SELECT` the target day's `posts` plus all rows
rows reachable from them (`posts_hashtags`, `posts_urls`, `hashtags`, reachable from them (`posts_hashtags`, `posts_urls`, `hashtags`, `urls`,
`urls`, `users`, and `media` if that table exists); `users`, and `media` if that table exists);
- abort the day cleanly if there are zero posts (`ErrNoPosts`), rather - abort the day cleanly if there are zero posts (`ErrNoPosts`), rather than
than emitting an empty dump; emitting an empty dump;
- recreate indexes, detach the source, and verify the inserted row count; - recreate indexes, detach the source, and verify the inserted row count;
- `sqlite3 .dump | zstdmt` into a hidden temp file; - `sqlite3 .dump | zstdmt` into a hidden temp file;
- run a zstd integrity check and confirm the decompressed head looks like - run a zstd integrity check and confirm the decompressed head looks like
SQL; SQL;
- atomically rename into place and delete the per-day scratch DB. - atomically rename into place and delete the per-day scratch DB.
6. **Clean up** the temp directory and log a processed/skipped/total summary. 6. **Clean up** the temp directory and log a processed/skipped/total summary.
# Usage # Usage
@@ -212,9 +217,9 @@ sqlite3 merged.db < regenerate_auxiliary_tables.sql
- **Go** (see [`go.mod`](go.mod) for the toolchain version) to build. - **Go** (see [`go.mod`](go.mod) for the toolchain version) to build.
- **Linux** for production use (ZFS snapshots, `statfs` free-space checks, - **Linux** for production use (ZFS snapshots, `statfs` free-space checks,
`posix_fadvise` hints). `posix_fadvise` hints).
- The **`sqlite3`** and **`zstdmt`** (multithreaded zstd) binaries on - The **`sqlite3`** and **`zstdmt`** (multithreaded zstd) binaries on `PATH`;
`PATH`; `zstdcat` is used for verification. SQLite reads/writes during `zstdcat` is used for verification. SQLite reads/writes during extraction use
extraction use the pure-Go [`modernc.org/sqlite`](https://pkg.go.dev/modernc.org/sqlite) the pure-Go [`modernc.org/sqlite`](https://pkg.go.dev/modernc.org/sqlite)
driver, so no cgo is required for that part. driver, so no cgo is required for that part.
# Configuration # Configuration
@@ -234,21 +239,21 @@ elsewhere.
# Data Model # Data Model
The firehose schema (reference copy in [`schema.sql`](schema.sql)) centers The firehose schema (reference copy in [`schema.sql`](schema.sql)) centers on a
on a `posts` table, with `users` keyed by DID and many-to-many junction `posts` table, with `users` keyed by DID and many-to-many junction tables
tables linking posts to deduplicated `hashtags` and `urls`. An optional linking posts to deduplicated `hashtags` and `urls`. An optional `media` table
`media` table tracks downloaded blobs by content hash. `bsdaily` does not tracks downloaded blobs by content hash. `bsdaily` does not own this schema; it
own this schema; it reflects whatever DDL exists in the source snapshot and reflects whatever DDL exists in the source snapshot and selects forward from
selects forward from `posts` along the foreign-key relationships to produce a `posts` along the foreign-key relationships to produce a referentially-complete
referentially-complete per-day slice. per-day slice.
# Use Cases # Use Cases
## Daily public archive ## Daily public archive
Publish one small, immutable file per day to static HTTP (or IPFS, or a Publish one small, immutable file per day to static HTTP (or IPFS, or a mirror
mirror network) so that anyone can fetch exactly the day(s) they want and network) so that anyone can fetch exactly the day(s) they want and re-merge them
re-merge them locally. locally.
## Backfilling a range ## Backfilling a range
@@ -258,16 +263,17 @@ safe to re-run.
## Reconstituting a full database ## Reconstituting a full database
Collect any set of daily dumps and merge them with `INSERT OR IGNORE` to Collect any set of daily dumps and merge them with `INSERT OR IGNORE` to rebuild
rebuild a complete, queryable SQLite database, then regenerate the aggregate a complete, queryable SQLite database, then regenerate the aggregate statistics
statistics tables. tables.
# See Also # See Also
## Links ## Links
- Repo: [https://git.eeqj.de/sneak/bsdaily](https://git.eeqj.de/sneak/bsdaily) - Repo: [https://git.eeqj.de/sneak/bsdaily](https://git.eeqj.de/sneak/bsdaily)
- Issues: [https://git.eeqj.de/sneak/bsdaily/issues](https://git.eeqj.de/sneak/bsdaily/issues) - Issues:
[https://git.eeqj.de/sneak/bsdaily/issues](https://git.eeqj.de/sneak/bsdaily/issues)
- Bluesky: [https://bsky.app](https://bsky.app) - Bluesky: [https://bsky.app](https://bsky.app)
- zstd: [https://facebook.github.io/zstd/](https://facebook.github.io/zstd/) - zstd: [https://facebook.github.io/zstd/](https://facebook.github.io/zstd/)
+369 -86
View File
@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-07-06 last_modified: 2026-10-06
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -60,17 +60,28 @@ style conventions are in separate documents:
prerequisite since nvm requires bash. yarn is then pinned via prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts"; `corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root and runs `docker build .`; the Gitea workflow calls it. Four further repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
scripts are our own extensions to the standard: `script/check` runs with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is `script/bootstrap` first**, because the workflow checks out the repo and runs
what the git pre-commit hook runs, and it calls `script/check`; nothing else, while `script/fmt-check` runs the formatter on the host: on a
`script/install-precommit` installs the git pre-commit hook (the `make hooks` pristine checkout with nothing installed the run dies there, after the
target shims to it); and `script/projectname` (literally that filename) simply containerised gates have passed. **The bootstrap alone is not enough**:
outputs the project's name. Scripts that need the name call `script/bootstrap` installs node and yarn under nvm and leaves neither on the
`script/projectname` — e.g. `script/docker` assembles its image tag from it — `PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
so those scripts stay byte-identical across all repos. Repo-type-specific entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in source nvm for the pinned node version before invoking it, exactly as
`script/precommit`, not in the hook itself. Model scripts are at `script/bootstrap`'s own install step does. A runner carrying nothing but
docker and git then gets through `script/check`. Four further scripts are our
own extensions to the standard: `script/check` runs `script/test`,
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/projectname` (literally that filename) simply outputs the project's
name. Scripts that need the name call `script/projectname` — e.g.
`script/docker` assembles its image tag from it — so those scripts stay
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the must document the provided scripts in an **Entrypoints** section (see the
README requirements below). README requirements below).
@@ -89,87 +100,201 @@ style conventions are in separate documents:
contributor should be able to understand the entire development workflow by contributor should be able to understand the entire development workflow by
reading the Makefile. reading the Makefile.
- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check` - Every repo should have a `Dockerfile`, and it carries the repo's gates: a
as a build step so the build fails if the branch is not green. For non-server `lint` phase and a `test` phase, with the final stage depending on both so the
repos, the Dockerfile should bring up a development environment and run image cannot be built unless they pass. For non-server repos the final stage
`make check`. For server repos, `make check` should run as an early build brings up a development environment; for server repos it is the runtime image.
stage before the final image is assembled. Dockerfiles install development The gate phases and the build stage start from their pinned base images and
prerequisites by running `script/bootstrap` rather than duplicating installs install what those images lack either inline, as the canonical Go `Dockerfile`
inline; COPY `script/` and the dependency manifests (`package.json` + below does for `git`, or by running `script/bootstrap`, as the `prompts`
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap repo's own `Dockerfile` does for its yarn packages. The development
layer stays cached until dependencies change. environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go - **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
repos use a multistage build where linting runs in an independent stage based no separate lint file. `script/lint` and `script/test` each build one phase
on the `golangci/golangci-lint` image (pinned by hash). This stage runs and nothing else:
`make fmt-check` and `make lint` before the full build begins. The build stage
then declares an explicit dependency on the lint stage via
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
linting before proceeding to compilation and tests. This ensures lint failures
surface in seconds rather than minutes, without blocking on dependency
download or compilation in the build stage.
The standard pattern for a Go repo Dockerfile is: ```sh
tag="$(script/projectname)"
docker build --no-cache --target lint -t "$tag-lint" .
docker build --no-cache --target test -t "$tag-test" .
```
**A stage that is not the last one in the file is built only when the final
stage's chain depends on it, or when `--target` names it.** That is why the
two gates are always invoked by name here, and why the final stage carries a
`COPY --from=` of a harmless file from each of them: without that edge a
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image. Each script assigns the
tag on its own line before the build, so `set -e` stops it where
`script/projectname` fails.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
themselves a `docker build` and would recurse into a daemon that does not
exist in a build step. Formatting is the exception and stays on the host:
`script/fmt` writes the working tree, and `script/fmt-check` is its
read-only twin.
**No lint verdict may come from a host invocation of the linter.** On a
shared host golangci-lint reads a result cache keyed on file content rather
than location, so a second checkout of the same content is served the first
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`:
```dockerfile ```dockerfile
# Lint stage — fast feedback on formatting and lint issues # Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD # golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
RUN make fmt-check RUN golangci-lint run --config .golangci.yml ./...
RUN make lint
# Build stage # Test phase. -race needs cgo and so a C compiler, which the Debian Go
# golang:1.x-alpine, YYYY-MM-DD # image ships and the alpine one does not.
FROM golang@sha256:... AS builder # golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
WORKDIR /src WORKDIR /src
# Force BuildKit to run the lint stage before proceeding
COPY --from=lint /src/go.sum /dev/null
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
RUN make test RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
ARG VERSION=dev # Build stage. Nothing is wanted from either phase above; the copies
RUN CGO_ENABLED=0 go build -trimpath \ # are what make BuildKit build them first, so this stage cannot run
-ldflags="-s -w -X main.Version=${VERSION}" \ # unless lint and test passed.
-o /app ./cmd/app/ # golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Runtime stage # The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage, and the last one
FROM alpine@sha256:... FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"] ENTRYPOINT ["app"]
``` ```
Key points: Key points:
- The lint stage uses the `golangci/golangci-lint` image directly (it - The lint phase uses the `golangci/golangci-lint` image directly (it has
includes both Go and the linter), so there is no need to install the both Go and the linter), so nothing needs installing.
linter separately. - `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates purpose is the ordering edge. BuildKit runs stages in parallel by default,
a stage dependency. BuildKit runs stages in parallel by default; without and a stage nothing depends on is not built at all, so without these two
this line, the build stage would not wait for lint to finish and a lint lines a red gate would not fail the build.
failure might not fail the overall build. - Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- If the project uses `//go:embed` directives that reference build artifacts - If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint stage must (e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example: create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`. `RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
The lint stage should not depend on the actual build output — it exists to - If the project requires CGO or system libraries for linting, install them
fail fast. in the lint phase. The `golangci/golangci-lint` image is Debian-based and
- If the project requires CGO or system libraries for linting (e.g. has no `apk`, so install with `apt-get` under the Debian package name
`vips-dev`), install them in the lint stage with `apk add`. (`libvips-dev`, where alpine says `vips-dev`), and delete the package
- The build stage runs `make test` after compilation setup. Tests run in the lists in the same `RUN`, so the layer does not keep them:
build stage, not the lint stage, because they may require compiled
artifacts or heavier dependencies. ```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` (which runs `docker build .`) on push. Since the runs `script/cibuild` on push, and checks out the repo as its only other step.
Dockerfile already runs `make check`, a successful build implies all checks That script bootstraps, runs the gate phases, and then builds the image, so a
pass. successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for - Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -189,14 +314,21 @@ style conventions are in separate documents:
module under test to verify it compiles/parses. There is no excuse for module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op. `make test` to be a no-op.
- `make test` must complete in under 20 seconds. Add a 30-second timeout in the - `make test` must complete in under 60 seconds. That is the hard cap, and a
Makefile. suite that exceeds it fails. Under 20 seconds is the target. A suite between
20 and 60 seconds is still green, but the overage must be filed as an
improvement bug against that repo. Add a 90-second timeout to the test
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
hard cap so that it catches a genuinely hung test rather than a merely slow
one.
- **`make test` should use the conditional verbose rerun pattern.** Run tests - **The test command should use the conditional verbose rerun pattern.** Run
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to tests without `-v` (verbose) first. If tests fail, automatically rerun with
show full output. This keeps CI logs and `docker build` output clean on `-v` to show full output. This keeps CI logs and `docker build` output clean
success (just package/suite summaries) while providing full diagnostic detail on success (just package/suite summaries) while providing full diagnostic
on failure (every test case, every assertion). The general shell pattern: detail on failure (every test case, every assertion). The command lives in the
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
Makefile form below is the same pattern for any repo-local invocation:
```makefile ```makefile
test: test:
@@ -209,11 +341,26 @@ style conventions are in separate documents:
```makefile ```makefile
test: test:
@go test -timeout 30s -race -cover ./... || \ @go test -count=1 -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \ { echo "--- Rerunning with -v for details ---"; \
go test -timeout 30s -race -v ./...; exit 1; } go test -count=1 -timeout 90s -race -v ./...; exit 1; }
``` ```
`-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite
and no recompilation.
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. That matters on a developer's machine, where this
target runs and the directory lasts from one run to the next. The `test`
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example: Python example:
```makefile ```makefile
@@ -239,10 +386,89 @@ style conventions are in separate documents:
must be in `.gitignore`. No exceptions. must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`), - `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`. editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
Fetch the standard `.gitignore` from `node_modules/`, and the repo's own build outputs. Fetch the standard
`.gitignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo. a new repo. A repo's `.gitignore` is the standard file followed by the repo's
own entries, such as its binaries; a re-vendor replaces the standard part and
keeps those entries. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
across unmodified leaves secrets in the build context.** Docker matches with
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
therefore excludes only the copies at the repository root, while `config/.env`
and `certs/server.key` still reach the context and can land in an image layer
— which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
also catches something the build needs, re-include it with a negation
(`!docs/example.env`); deleting the pattern reopens the exposure for every
other file it covers. Fetch the standard `.dockerignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
it with the repo's own artifacts.
- **In-repo agent scratch belongs in both files, written to each file's own
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
additional checkout of the repo — so under `COPY . .` the build context
inflates by a multiple of the repo and another session's unreviewed work can
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
prefix, because the prefixed form would also delete any nested directory of
that name from the build. Anchoring carries a known gap that the canonical
`.dockerignore` states in its own comment, since consuming repos receive the
file and not the tracker: the directory is created in the agent's working
directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there.
- **A plain `docker build .` of a clone stamps the version that
`git describe --tags --always` gives**, derived from the `.git` in the build
context as the canonical `Dockerfile` above shows. Without its failure check,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
and the build would still exit 0. `script/docker` and `script/cibuild` pass
the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh
# The version and the tag each get their own line: a failing command
# substitution inside an argument does not trip `set -e`, so the inline
# form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
tag="$(script/projectname)"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$tag" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
than failing, and the `[ -n "$version" ]` line is the single place the
fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
a probe image that does `COPY . .`, and list what actually landed
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
size is not a substitute: a nested secret is a few bytes, and BuildKit
transfers only the delta from the previous build.
- **No build artifacts in version control.** Code-derived data (compiled - **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the bundles, minified output, generated assets) must never be committed to the
@@ -258,9 +484,56 @@ style conventions are in separate documents:
- Make all changes on a feature branch. You can do whatever you want on a - Make all changes on a feature branch. You can do whatever you want on a
feature branch. feature branch.
- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only - `.golangci.yml` is standardized. The vendored copy in a consuming repo must
manually by the user. Fetch from _NEVER_ be modified by an agent: fetch it from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`. `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
byte-identical, so that no repo can quietly loosen its own linting. Linter
configuration changes are made to the canonical copy in the `prompts` repo and
reach consuming repos by re-vendoring; an agent may open a PR against
canonical, which only the user merges. One list is exempt from byte-identity,
because it cannot be written once for every repo: the `deny` list of the
`test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
directive must not name a newer Go minor version than the one golangci-lint
was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
`PATH` only, so on an already-provisioned machine the pin is inert and a
version bump is a silent no-op — while the Dockerfile, installing into a clean
image, gets the pinned version, so a local `make check` and `make docker` can
disagree about what the tool even is. The canonical form:
- compares the installed version against the pin over the **whole** version
token; a parser that stops at the first `-` reports `2.12.2` for a host
running `2.12.2-rc1` and skips the install;
- treats absent, non-zero, empty or unrecognised `--version` output as a
mismatch, so the failure direction is a redundant install and never a
skipped one;
- after installing, re-resolves the binary the way callers do — `hash -r`,
then through `PATH`, not through the directory the installer wrote to —
and fails naming the resolved path, since an install that a shadowing
binary hides succeeds while changing nothing any caller sees;
- is actually called, and prints the version on both success paths: a
function defined and never invoked has the same exit status and the same
empty output as one that worked.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference - When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD). with the version and date (YYYY-MM-DD).
@@ -371,15 +644,21 @@ style conventions are in separate documents:
Never edit existing migrations after release. Never edit existing migrations after release.
- All repos should have an `.editorconfig` enforcing the project's indentation - All repos should have an `.editorconfig` enforcing the project's indentation
settings. settings: the standard file from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, which sets
tabs for `Makefile` and Go files, followed by the repo's own sections, such as
one for another language it uses. A re-vendor replaces the standard part and
keeps those sections.
- Avoid putting files in the repo root unless necessary. Root should contain - Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `Makefile`, `Dockerfile`, only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and `Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
language-specific config). Everything else goes in a subdirectory. Canonical and language-specific config). Everything else goes in a subdirectory.
subdirectory names: Canonical subdirectory names:
- `bin/` — executable scripts and tools - `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints - `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in
`cmd/`
- `configs/` — configuration templates and examples - `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform) - `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root) - `docs/` — documentation and markdown (README.md stays in root)
@@ -406,3 +685,7 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml` - Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore` - JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml` - Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
+32 -24
View File
@@ -1,12 +1,12 @@
# Workflow # Workflow
* branch (from `main`) - branch (from `main`)
* do the work in Next Step - do the work in Next Step
* move Next Step to the top of Completed Steps - move Next Step to the top of Completed Steps
* move the top item of Future Steps into Next Step - move the top item of Future Steps into Next Step
* commit (`TODO.md` changes in the same commit as the work) - commit (`TODO.md` changes in the same commit as the work)
* merge to `main` if the branch is not protected, otherwise open a PR - merge to `main` if the branch is not protected, otherwise open a PR
* push - push
# Status # Status
@@ -14,30 +14,38 @@ pre-1.0
# Next Step # Next Step
Bring the repo into policy compliance in one commit: add .gitignore, Add the canonical `.golangci.yml`, move the lint phase to golangci-lint v2.14.0
.dockerignore, .editorconfig, and .golangci.yml. Verify `make check` stays in the same commit, and fix the findings it surfaces
green with the new lint config. (https://git.eeqj.de/sneak/bsdaily/issues/6).
# Completed Steps # Completed Steps
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, - 2026-10-06: Formatted Markdown with prettier: `script/fmt` writes and
Makefile shims, README Entrypoints section `script/fmt-check` checks every Markdown file; prettier pinned in
- 2026-06-28: Fixed errcheck lint failures; added compilation smoke test; `package.json` and `yarn.lock`, installed by `script/bootstrap`, which
tidied go.mod. installs node and yarn at pinned versions when absent and otherwise uses the
- 2026-06-28: Added repo scaffolding: README, LICENSE, Makefile, ones already installed; existing Markdown reformatted.
Dockerfile, REPO_POLICIES.md, and Gitea CI. - 2026-10-05: Brought the repo up to the standard layout: canonical
- 2026-02-12: Fixed SQLite database locking by removing parallel `.gitignore`, `.dockerignore` and `.editorconfig`; `lint` and `test` phases in
processing; fixed Linux build via golang.org/x/sys/unix Fadvise. the `Dockerfile`, built by `script/lint` and `script/test`; canonical
- 2026-02-12: Optimized file copy for large databases; moved temp `script/cibuild`, `script/docker` and CI workflow; no linter installed on the
directory to NVMe scratch storage. host; re-vendored `REPO_POLICIES.md`.
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
shims, README Entrypoints section
- 2026-06-28: Fixed errcheck lint failures; added compilation smoke test; tidied
go.mod.
- 2026-06-28: Added repo scaffolding: README, LICENSE, Makefile, Dockerfile,
REPO_POLICIES.md, and Gitea CI.
- 2026-02-12: Fixed SQLite database locking by removing parallel processing;
fixed Linux build via golang.org/x/sys/unix Fadvise.
- 2026-02-12: Optimized file copy for large databases; moved temp directory to
NVMe scratch storage.
- 2026-02-11: Added date range support. - 2026-02-11: Added date range support.
- 2026-02-09: Initial implementation: single-day extraction, specific-date - 2026-02-09: Initial implementation: single-day extraction, specific-date
targeting, faster pruning of throwaway database copies. targeting, faster pruning of throwaway database copies.
# Future Steps # Future Steps
- Add .gitignore, .dockerignore, .editorconfig, .golangci.yml (the Next - Expand tests beyond the compilation smoke test: unit tests for the extraction,
Step). verification, and atomic-publish paths.
- Expand tests beyond the compilation smoke test: unit tests for the
extraction, verification, and atomic-publish paths.
- Cut a first SemVer release once compliance and test coverage land. - Cut a first SemVer release once compliance and test coverage land.
+5
View File
@@ -0,0 +1,5 @@
{
"devDependencies": {
"prettier": "3.8.1"
}
}
+88 -9
View File
@@ -3,13 +3,23 @@
# this repo. Idempotent: every install is guarded by a check so already # this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew, # installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes NOTHING is present (not git, # or apk (detected in that order); assumes NOTHING is present (not git,
# make, or go). # make, go, or node). Node is used directly if installed; otherwise it
# is installed at a pinned version via nvm (installing nvm itself first,
# from a hash-verified release archive, never curl | sh).
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-07-06
NODE_VERSION="22.17.0"
NVM_VERSION="0.40.3"
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
YARN_VERSION="1.22.22"
PKGMGR="" PKGMGR=""
SUDO="" SUDO=""
APT_UPDATED=""
detect_pkgmgr() { detect_pkgmgr() {
[ -n "$PKGMGR" ] && return 0 [ -n "$PKGMGR" ] && return 0
@@ -38,7 +48,14 @@ pkg_install() {
detect_pkgmgr detect_pkgmgr
case "$PKGMGR" in case "$PKGMGR" in
nix) nix-env -iA "nixpkgs.$1" ;; nix) nix-env -iA "nixpkgs.$1" ;;
apt) $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2" ;; apt)
# Package lists may be empty (fresh images); refresh once per run.
if [ -z "$APT_UPDATED" ]; then
$SUDO env DEBIAN_FRONTEND=noninteractive apt-get update
APT_UPDATED=1
fi
$SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2"
;;
brew) brew install "$3" ;; brew) brew install "$3" ;;
apk) apk add --no-cache "$4" ;; apk) apk add --no-cache "$4" ;;
esac esac
@@ -48,6 +65,69 @@ missing() {
! command -v "$1" >/dev/null 2>&1 ! command -v "$1" >/dev/null 2>&1
} }
# verify_sha256 <file> <expected-hash>
verify_sha256() {
if command -v sha256sum >/dev/null 2>&1; then
actual="$(sha256sum "$1" | cut -d' ' -f1)"
else
actual="$(shasum -a 256 "$1" | cut -d' ' -f1)"
fi
if [ "$actual" != "$2" ]; then
echo "bootstrap: sha256 mismatch for $1" >&2
echo " expected: $2" >&2
echo " actual: $actual" >&2
exit 1
fi
}
# nvm is a bash script; run a command in a bash with nvm loaded
nvm_sh() {
bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*"
}
ensure_nvm() {
[ -s "$HOME/.nvm/nvm.sh" ] && return 0
# nvm prerequisites; nvm itself requires bash
if missing bash; then pkg_install bash bash bash bash; fi
if missing curl; then pkg_install curl curl curl curl; fi
if missing git; then pkg_install git git git git; fi
tmp="$(mktemp -d)"
curl -fsSL -o "$tmp/nvm.tar.gz" \
"https://github.com/nvm-sh/nvm/archive/refs/tags/v${NVM_VERSION}.tar.gz"
verify_sha256 "$tmp/nvm.tar.gz" "$NVM_SHA256"
mkdir -p "$HOME/.nvm"
tar -xzf "$tmp/nvm.tar.gz" -C "$HOME/.nvm" --strip-components=1
rm -rf "$tmp"
}
ensure_node() {
if ! missing node; then return 0; fi
ensure_nvm
nvm_sh "nvm install $NODE_VERSION"
}
ensure_yarn() {
if ! missing yarn; then return 0; fi
if ! missing corepack; then
corepack enable
corepack prepare "yarn@$YARN_VERSION" --activate
elif [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
corepack prepare yarn@$YARN_VERSION --activate"
else
npm install -g "yarn@$YARN_VERSION"
fi
}
install_js_deps() {
if missing yarn && [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
yarn install --frozen-lockfile"
else
yarn install --frozen-lockfile
fi
}
main() { main() {
cd "$ROOT" cd "$ROOT"
@@ -58,15 +138,14 @@ main() {
# Go toolchain # Go toolchain
if missing go; then pkg_install go golang go go; fi if missing go; then pkg_install go golang go go; fi
# golangci-lint: packaged in nix, brew, and apk. There is no apt
# package; on apt systems install it manually from a hash-verified
# GitHub release archive (never curl | sh).
if missing golangci-lint; then
pkg_install golangci-lint golangci-lint golangci-lint golangci-lint
fi
go mod download go mod download
# Node and yarn, then the prettier pinned in package.json and
# yarn.lock, which script/fmt and script/fmt-check run
ensure_node
ensure_yarn
install_js_deps
echo "bootstrap complete" echo "bootstrap complete"
} }
+21 -5
View File
@@ -1,14 +1,30 @@
#!/bin/sh #!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs script/check # script/cibuild: run the CI build. It bootstraps first: a CI runner
# (via make check), so a successful build implies all checks pass. # checks out and runs this and nothing else, and script/fmt-check runs
# Generic: needs no adaptation. The Gitea workflow runs this on push. # the formatter on the host, which a pristine checkout cannot do.
# --no-cache for the same reason as script/docker: the gate phases the
# final stage depends on are RUN steps, and a cached one is a check that
# did not run.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build . "$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check"
# The version and the tag each get their own line: a failing
# command substitution inside an argument does not trip `set -e`,
# so the inline form degrades silently to an empty constant. The
# VERSION build argument takes precedence over the version a build
# stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
tag="$("$SCRIPT_DIR/projectname")"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$tag" .
} }
main "$@" main "$@"
+13 -2
View File
@@ -1,7 +1,8 @@
#!/bin/sh #!/bin/sh
# script/docker: build the Docker image tagged with the project name. # script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname. # Identical in all repos; the tag comes from script/projectname.
# Generic: needs no adaptation. # --no-cache because the gate phases the final stage depends on are RUN
# steps, and a cached one is a check that did not run.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -9,7 +10,17 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build -t "$("$SCRIPT_DIR/projectname")" . # The version and the tag each get their own line: a failing
# command substitution inside an argument does not trip `set -e`,
# so the inline form degrades silently to an empty constant. The
# VERSION build argument takes precedence over the version a build
# stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
tag="$("$SCRIPT_DIR/projectname")"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$tag" .
} }
main "$@" main "$@"
+23 -1
View File
@@ -1,12 +1,34 @@
#!/bin/sh #!/bin/sh
# script/fmt: format all files (writes). # script/fmt: format all files (writes): the Go code with go fmt and
# every Markdown file with prettier.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt: no yarn; run script/bootstrap first" >&2
exit 1
fi
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
}
main() { main() {
cd "$ROOT" cd "$ROOT"
go fmt ./... go fmt ./...
# run_yarn replaces this shell, so it stays the last step.
run_yarn run prettier --write '**/*.md' --tab-width 4 --prose-wrap always
} }
main "$@" main "$@"
+23 -1
View File
@@ -1,10 +1,30 @@
#!/bin/sh #!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as # script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing. # script/fmt, the Go code and every Markdown file, but fails instead of
# writing.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt-check: no yarn; run script/bootstrap first" >&2
exit 1
fi
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
}
main() { main() {
cd "$ROOT" cd "$ROOT"
unformatted="$(gofmt -l .)" unformatted="$(gofmt -l .)"
@@ -13,6 +33,8 @@ main() {
echo "$unformatted" >&2 echo "$unformatted" >&2
exit 1 exit 1
fi fi
# run_yarn replaces this shell, so it stays the last step.
run_yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always
} }
main "$@" main "$@"
+18 -3
View File
@@ -1,12 +1,27 @@
#!/bin/sh #!/bin/sh
# script/lint: run the linter. # script/lint: run the linter. Linting is a phase of the Dockerfile and
# this builds that phase alone; the linter is never installed or run on
# a developer host, where a shared result cache and a host-global lock
# make its answer untrustworthy.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. The tag makes each build replace the previous image
# instead of leaving a dangling one behind.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
golangci-lint run ./... # The tag gets its own line: a failing command substitution inside
# an argument does not trip `set -e`, so the inline form degrades
# silently to an empty constant.
tag="$("$SCRIPT_DIR/projectname")"
docker build --no-cache \
--target lint \
-t "$tag-lint" .
} }
main "$@" main "$@"
+14 -9
View File
@@ -1,18 +1,23 @@
#!/bin/sh #!/bin/sh
# script/test: run the test suite. Quiet on success; on failure, rerun # script/test: run the test suite. Testing is a phase of the Dockerfile
# verbosely for full diagnostic output (the exit 1 ensures the rerun # and this builds that phase alone, on the same terms as script/lint:
# never turns a failure into a pass). # --target because a phase that is not the last stage is built only when
# named, --no-cache because a cached test layer is a test that did not
# run, and a tag so each build replaces the previous image.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
go test -race -timeout 30s ./... || { # The tag gets its own line: a failing command substitution inside
echo "--- Rerunning with -v for details ---" # an argument does not trip `set -e`, so the inline form degrades
go test -race -timeout 30s -v ./... # silently to an empty constant.
exit 1 tag="$("$SCRIPT_DIR/projectname")"
} docker build --no-cache \
--target test \
-t "$tag-test" .
} }
main "$@" main "$@"
+8
View File
@@ -0,0 +1,8 @@
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
prettier@3.8.1:
version "3.8.1"
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==