Compare commits

...

3 Commits
v0.1.0 ... main

Author SHA1 Message Date
291f85f3ed scripts-to-rule-them-all (#174)
Reviewed-on: sneak/upaas#174
Co-authored-by: sneak <sneak@sneak.berlin>
Co-committed-by: sneak <sneak@sneak.berlin>
2026-07-07 02:15:20 +02:00
bee933ce8e TODO (#173)
Reviewed-on: sneak/upaas#173
2026-07-06 21:36:27 +02:00
d5c3d4d55f Move schema_migrations table creation into 000.sql with INTEGER version column (#172)
Closes #171

Refactors the migration system to follow the [pixa pattern](sneak/pixa#36):

## Changes

### New file: `internal/database/migrations/000.sql`
- Bootstrap migration that creates `schema_migrations` with `version INTEGER PRIMARY KEY`
- Self-contained: includes both `CREATE TABLE IF NOT EXISTS` and `INSERT OR IGNORE INTO schema_migrations (version) VALUES (0)`

### Refactored: `internal/database/migrations.go`
- **Go code no longer creates the migrations table inline** — it reads and executes `000.sql` as a bootstrap step before the normal migration loop
- **`ParseMigrationVersion(filename)`** — exported function that extracts integer version from filenames like `001_initial.sql` → `1`
- **`ApplyMigrations(ctx, db, log)`** — exported function for tests to apply schema without full fx lifecycle
- **`bootstrapMigrationsTable`** — checks if table exists; if missing, runs `000.sql`; if present, checks for legacy format
- **`convertLegacyMigrations`** — one-time conversion for existing databases that stored versions as TEXT filenames (e.g. `"001_initial.sql"`) to INTEGER versions (e.g. `1`)
- **Transaction-wrapped migration application** — each migration runs in a transaction for atomicity
- Sentinel error `ErrInvalidMigrationFilename` for static error wrapping

### New file: `internal/database/migrations_test.go`
- `TestParseMigrationVersion` — valid/invalid filename parsing
- `TestApplyMigrationsFreshDatabase` — verifies bootstrap creates table, all migrations apply, application tables exist
- `TestApplyMigrationsIdempotent` — double-apply is a no-op
- `TestApplyMigrationsLegacyConversion` — simulates old TEXT-based table with filename entries, verifies conversion to integer format

## Backward Compatibility

Existing databases with the old TEXT-based `schema_migrations` table are automatically converted on first run. The conversion:
1. Detects legacy entries (versions containing `_`)
2. Parses integer version from each legacy filename
3. Drops the old table
4. Creates the new INTEGER-based table via `000.sql`
5. Re-inserts all parsed integer versions

All existing migrations (001-007) continue to work unchanged.

Co-authored-by: user <user@Mac.lan guest wan>
Co-authored-by: clawbot <clawbot@sneak.berlin>
Reviewed-on: sneak/upaas#172
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-03-30 21:40:16 +02:00
20 changed files with 900 additions and 90 deletions

View File

@@ -13,4 +13,4 @@ jobs:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4, 2024-10-13
- name: Build (runs make check inside Dockerfile)
run: docker build .
run: script/cibuild

View File

@@ -1,4 +1,4 @@
.PHONY: all build lint fmt fmt-check test check clean docker hooks
.PHONY: all bootstrap setup build lint fmt fmt-check test check clean docker hooks
BINARY := upaasd
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev")
@@ -7,37 +7,37 @@ LDFLAGS := -X main.Version=$(VERSION) -X main.Buildarch=$(BUILDARCH)
all: check build
bootstrap:
@script/bootstrap
setup:
@script/setup
build:
go build -ldflags "$(LDFLAGS)" -o bin/$(BINARY) ./cmd/upaasd
lint:
golangci-lint run --config .golangci.yml ./...
@script/lint
fmt:
gofmt -s -w .
goimports -w .
npx prettier --write --tab-width 4 static/js/*.js
@script/fmt
fmt-check:
@test -z "$$(gofmt -l .)" || (echo "Files not formatted:" && gofmt -l . && exit 1)
@script/fmt-check
test:
go test -v -race -cover -timeout 30s ./...
@script/test
# Check runs all validation without making changes
# Used by CI and Docker build - fails if anything is wrong
check: fmt-check lint test
@echo "==> All checks passed!"
check:
@script/check
docker:
docker build .
@script/docker
hooks:
@echo "Installing pre-commit hook..."
@mkdir -p .git/hooks
@printf '#!/bin/sh\nmake check\n' > .git/hooks/pre-commit
@chmod +x .git/hooks/pre-commit
@echo "Pre-commit hook installed."
@script/install-precommit
clean:
rm -rf bin/

View File

@@ -100,6 +100,31 @@ chi Router ──► Middleware Stack ──► Handler
- **Async deployments**: Webhook triggers deploy via goroutine with `context.WithoutCancel()`
- **Embedded assets**: Templates and static files embedded via `//go:embed`
## Entrypoints
This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call
them. We provide:
- `script/bootstrap` — install all dependencies (idempotent)
- `script/setup` — make a fresh clone ready for development
(bootstrap, then install-precommit)
- `script/projectname` — output the project name ("upaas")
- `script/test` — run the test suite
- `script/lint` — run golangci-lint
- `script/fmt` — format all code (writes)
- `script/fmt-check` — check formatting (read-only)
- `script/check` — run test, lint, and fmt-check
- `script/docker` — build the Docker image tagged via `script/projectname`
- `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile
runs the checks, so a green build implies a green repo)
- `script/precommit` — pre-commit checks (`go mod tidy` guard, then
`script/check`)
- `script/install-precommit` — install the git pre-commit hook that
runs `script/precommit`
## Development
### Prerequisites
@@ -111,14 +136,16 @@ chi Router ──► Middleware Stack ──► Handler
### Commands
```bash
make bootstrap # Install all dependencies (idempotent)
make setup # Bootstrap + install git pre-commit hook
make fmt # Format code
make fmt-check # Check formatting (read-only, fails if unformatted)
make lint # Run comprehensive linting
make test # Run tests with race detection (30s timeout)
make check # Verify everything passes (fmt-check, lint, test)
make check # Verify everything passes (test, lint, fmt-check)
make build # Build binary
make docker # Build Docker image
make hooks # Install pre-commit hook (runs make check)
make hooks # Install pre-commit hook (runs script/precommit)
```
### Commit Requirements

View File

@@ -1,6 +1,6 @@
---
title: Repository Policies
last_modified: 2026-02-22
last_modified: 2026-07-06
---
This document covers repository structure, tooling, and workflow standards. Code
@@ -34,10 +34,46 @@ style conventions are in separate documents:
every file before committing. There are zero exceptions to this rule.
- Every repo with software must have a root `Makefile` with these targets:
`make test`, `make lint`, `make fmt` (writes), `make fmt-check` (read-only),
`make check` (prereqs: `test`, `lint`, `fmt-check`), `make docker`, and
`make hooks` (installs pre-commit hook). A model Makefile is at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`.
`make bootstrap`, `make setup`, `make test`, `make lint`, `make fmt` (writes),
`make fmt-check` (read-only), `make check` (runs `test`, `lint`, `fmt-check`),
`make docker`, and `make hooks` (installs pre-commit hook). A model Makefile
is at `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`.
- Repos follow the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
pattern: the implementation of each Makefile target lives in an executable
script in `script/` (`script/bootstrap`, `script/setup`, `script/test`,
`script/lint`, `script/fmt`, `script/fmt-check`, `script/check`,
`script/docker`), and the Makefile targets are thin shims that call them. The
scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they run in
minimal containers (e.g. alpine images have no bash); locate the repo root
with `$(cd "$(dirname "$0")/.." && pwd -P)` and `cd` there before acting. From
the standard's canonical set we use `bootstrap`, `setup` (make the repo ready
for development after a fresh clone: runs `bootstrap`, then
`install-precommit`, plus any repo-specific initialization), `test`, and
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
assumes nothing is present: base tools come from nix, apt, brew, or apk
(detected in that order; apt runs noninteractive). For node it uses the
installed node if present; otherwise it installs a PINNED node version via
nvm, first installing nvm itself if missing — from a hash-verified GitHub
release archive (never `curl | sh`), with bash installed as an explicit
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root and runs `docker build .`; the Gitea workflow calls it. Four further
scripts are our own extensions to the standard: `script/check` runs
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
what the git pre-commit hook runs, and it calls `script/check`;
`script/install-precommit` installs the git pre-commit hook (the `make hooks`
target shims to it); and `script/projectname` (literally that filename) simply
outputs the project's name. Scripts that need the name call
`script/projectname` — e.g. `script/docker` assembles its image tag from it —
so those scripts stay byte-identical across all repos. Repo-type-specific
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/precommit`, not in the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
- Always use Makefile targets (`make fmt`, `make test`, `make lint`, etc.)
instead of invoking the underlying tools directly. The Makefile is the single
@@ -57,11 +93,83 @@ style conventions are in separate documents:
as a build step so the build fails if the branch is not green. For non-server
repos, the Dockerfile should bring up a development environment and run
`make check`. For server repos, `make check` should run as an early build
stage before the final image is assembled.
stage before the final image is assembled. Dockerfiles install development
prerequisites by running `script/bootstrap` rather than duplicating installs
inline; COPY `script/` and the dependency manifests (`package.json` +
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
layer stays cached until dependencies change.
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
repos use a multistage build where linting runs in an independent stage based
on the `golangci/golangci-lint` image (pinned by hash). This stage runs
`make fmt-check` and `make lint` before the full build begins. The build stage
then declares an explicit dependency on the lint stage via
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
linting before proceeding to compilation and tests. This ensures lint failures
surface in seconds rather than minutes, without blocking on dependency
download or compilation in the build stage.
The standard pattern for a Go repo Dockerfile is:
```dockerfile
# Lint stage — fast feedback on formatting and lint issues
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make fmt-check
RUN make lint
# Build stage
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
WORKDIR /src
# Force BuildKit to run the lint stage before proceeding
COPY --from=lint /src/go.sum /dev/null
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make test
ARG VERSION=dev
RUN CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage
FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
Key points:
- The lint stage uses the `golangci/golangci-lint` image directly (it
includes both Go and the linter), so there is no need to install the
linter separately.
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
a stage dependency. BuildKit runs stages in parallel by default; without
this line, the build stage would not wait for lint to finish and a lint
failure might not fail the overall build.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint stage must
create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
The lint stage should not depend on the actual build output — it exists to
fail fast.
- If the project requires CGO or system libraries for linting (e.g.
`vips-dev`), install them in the lint stage with `apk add`.
- The build stage runs `make test` after compilation setup. Tests run in the
build stage, not the lint stage, because they may require compiled
artifacts or heavier dependencies.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `docker build .` on push. Since the Dockerfile already runs `make check`,
a successful build implies all checks pass.
runs `script/cibuild` (which runs `docker build .`) on push. Since the
Dockerfile already runs `make check`, a successful build implies all checks
pass.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -69,9 +177,11 @@ style conventions are in separate documents:
Markdown (hard-wrap at 80 columns). Documentation and writing repos (Markdown,
HTML, CSS) should also have `.prettierrc` and `.prettierignore`.
- Pre-commit hook: `make check` if local testing is possible, otherwise
`make lint && make fmt-check`. The Makefile should provide a `make hooks`
target to install the pre-commit hook.
- Pre-commit hook: runs `script/precommit`, which calls `script/check`. If local
testing is not possible in the repo, `script/precommit` may skip `script/test`
and run only `script/lint` and `script/fmt-check`. The hook is installed by
`script/install-precommit`; the Makefile must provide a `make hooks` target
that shims to it.
- All repos with software must have tests that run via the platform-standard
test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful
@@ -82,6 +192,42 @@ style conventions are in separate documents:
- `make test` must complete in under 20 seconds. Add a 30-second timeout in the
Makefile.
- **`make test` should use the conditional verbose rerun pattern.** Run tests
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
show full output. This keeps CI logs and `docker build` output clean on
success (just package/suite summaries) while providing full diagnostic detail
on failure (every test case, every assertion). The general shell pattern:
```makefile
test:
@<test-command> || \
{ echo "--- Rerunning with -v for details ---"; \
<test-command-with-v>; exit 1; }
```
Go example:
```makefile
test:
@go test -timeout 30s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 30s -race -v ./...; exit 1; }
```
Python example:
```makefile
test:
@python -m pytest || \
{ echo "--- Rerunning with -v for details ---"; \
python -m pytest -v; exit 1; }
```
The `exit 1` ensures the target always fails after a rerun — the first run
already proved the tests are broken, so the build must not pass even if a
flaky test happens to succeed on the second attempt. The rerun exists solely
for diagnostic output.
- Docker builds must complete in under 5 minutes.
- `make check` must not modify any files in the repo. Tests may use temporary
@@ -98,6 +244,13 @@ style conventions are in separate documents:
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo.
- **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the
repository if it can be avoided. The build process (e.g. Dockerfile, Makefile)
should generate these at build time. Notable exception: Go protobuf generated
files (`.pb.go`) ARE committed because repos need to work with `go get`, which
downloads code but does not execute code generation.
- Never use `git add -A` or `git add .`. Always stage files explicitly by name.
- Never force-push to `main`.
@@ -121,12 +274,76 @@ style conventions are in separate documents:
- Dockerized web services listen on port 8080 by default, overridable with
`PORT`.
- **HTTP/web services must be hardened for production internet exposure before
tagging 1.0.** This means full compliance with security best practices
including, without limitation, all of the following:
- **Security headers** on every response:
- `Strict-Transport-Security` (HSTS) with `max-age` of at least one year
and `includeSubDomains`.
- `Content-Security-Policy` (CSP) with a restrictive default policy
(`default-src 'self'` as a baseline, tightened per-resource as
needed). Never use `unsafe-inline` or `unsafe-eval` unless
unavoidable, and document the reason.
- `X-Frame-Options: DENY` (or `SAMEORIGIN` if framing is required).
Prefer the `frame-ancestors` CSP directive as the primary control.
- `X-Content-Type-Options: nosniff`.
- `Referrer-Policy: strict-origin-when-cross-origin` (or stricter).
- `Permissions-Policy` restricting access to browser features the
application does not use (camera, microphone, geolocation, etc.).
- **Request and response limits:**
- Maximum request body size enforced on all endpoints (e.g. Go
`http.MaxBytesReader`). Choose a sane default per-route; never accept
unbounded input.
- Maximum response body size where applicable (e.g. paginated APIs).
- `ReadTimeout` and `ReadHeaderTimeout` on the `http.Server` to defend
against slowloris attacks.
- `WriteTimeout` on the `http.Server`.
- `IdleTimeout` on the `http.Server`.
- Per-handler execution time limits via `context.WithTimeout` or
chi/stdlib `middleware.Timeout`.
- **Authentication and session security:**
- Rate limiting on password-based authentication endpoints. API keys are
high-entropy and not susceptible to brute force, so they are exempt.
- CSRF tokens on all state-mutating HTML forms. API endpoints
authenticated via `Authorization` header (Bearer token, API key) are
exempt because the browser does not attach these automatically.
- Passwords stored using bcrypt, scrypt, or argon2 — never plain-text,
MD5, or SHA.
- Session cookies set with `HttpOnly`, `Secure`, and `SameSite=Lax` (or
`Strict`) attributes.
- **Reverse proxy awareness:**
- True client IP detection when behind a reverse proxy
(`X-Forwarded-For`, `X-Real-IP`). The application must accept
forwarded headers only from a configured set of trusted proxy
addresses — never trust `X-Forwarded-For` unconditionally.
- **CORS:**
- Authenticated endpoints must restrict `Access-Control-Allow-Origin` to
an explicit allowlist of known origins. Wildcard (`*`) is acceptable
only for public, unauthenticated read-only APIs.
- **Error handling:**
- Internal errors must never leak stack traces, SQL queries, file paths,
or other implementation details to the client. Return generic error
messages in production; detailed errors only when `DEBUG` is enabled.
- **TLS:**
- Services never terminate TLS directly. They are always deployed behind
a TLS-terminating reverse proxy. The service itself listens on plain
HTTP. However, HSTS headers and `Secure` cookie flags must still be
set by the application so that the browser enforces HTTPS end-to-end.
This list is non-exhaustive. Apply defense-in-depth: if a standard security
hardening measure exists for HTTP services and is not listed here, it is
still expected. When in doubt, harden.
- `README.md` is the primary documentation. Required sections:
- **Description**: First line must include the project name, purpose,
category (web server, SPA, CLI tool, etc.), license, and author. Example:
"µPaaS is an MIT-licensed Go web application by @sneak that receives
git-frontend webhooks and deploys applications via Docker in realtime."
- **Getting Started**: Copy-pasteable install/usage code block.
- **Entrypoints**: Opens by stating that the repo adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard (with that link), then documents each provided `script/`
entrypoint and its purpose.
- **Rationale**: Why does this exist?
- **Design**: How is the program structured?
- **TODO**: Update meticulously, even between commits. When planning, put
@@ -145,13 +362,13 @@ style conventions are in separate documents:
- Database migrations live in `internal/db/migrations/` and must be embedded in
the binary.
- `000_migration.sql` — contains ONLY the creation of the migrations tracking
table itself. Nothing else.
- `001_schema.sql` — the full application schema.
- **Pre-1.0.0:** never add additional migration files (002, 003, etc.). There
is no installed base to migrate. Edit `001_schema.sql` directly.
- **Post-1.0.0:** add new numbered migration files for each schema change.
Never edit existing migrations after release.
- `000_migration.sql` — contains ONLY the creation of the migrations
tracking table itself. Nothing else.
- `001_schema.sql` — the full application schema.
- **Pre-1.0.0:** never add additional migration files (002, 003, etc.).
There is no installed base to migrate. Edit `001_schema.sql` directly.
- **Post-1.0.0:** add new numbered migration files for each schema change.
Never edit existing migrations after release.
- All repos should have an `.editorconfig` enforcing the project's indentation
settings.
@@ -181,6 +398,9 @@ style conventions are in separate documents:
- `README.md`, `.git`, `.gitignore`, `.editorconfig`
- `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo)
- `Makefile`
- `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`,
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`install-precommit`)
- `Dockerfile`, `.dockerignore`
- `.gitea/workflows/check.yml`
- Go: `go.mod`, `go.sum`, `.golangci.yml`

55
TODO.md Normal file
View File

@@ -0,0 +1,55 @@
# Workflow
* branch (from `main`)
* do the work in Next Step
* move Next Step to the top of Completed Steps
* move the top item of Future Steps into Next Step
* commit (`TODO.md` changes in the same commit as the work)
* merge to `main` if the branch is not protected, otherwise open a PR
* push
# Status
1.0+. Tagged 1.0.0 on 2026-02-26; 8 commits on main since. Policy
violation: main currently fails make check (91 lint issues), so the tree
is out of compliance until fixed.
# Next Step
Fix the 47 noctx lint findings (HTTP requests without context) in one
commit and confirm the count drops under make check. This is the largest
of the three lint classes blocking a green main.
# Completed Steps
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section
- 2026-03-11: Monolithic env var editing with bulk save (#158).
- 2026-03-10: Webhook event history UI page (#164); added missing
Makefile docker and hooks targets plus test timeout (#159);
notification settings passed from create form (#160).
- 2026-03-03: REPO_POLICIES compliance file set added (#155).
- 2026-03-01: Module path changed to sneak.berlin/go/upaas (#143);
Dockerfile split into lint and build stages with forced lint
execution (#152, #154).
- 2026-02-26: 1.0.0 tagged; dashboard CSRFField crash fixed (#146).
- 1.0 audit bug fixes (#120-#125): deferred rollback on commit error,
deployment log size cap, error path rendering, docker-compose bind
mount, domain type refactor.
- CI simplified to docker build only (#130).
- 2025-12-29 onward: core PaaS built out: deploys with real-time build
log streaming, container start/stop/restart and logs, TCP/UDP port
mapping, Alpine.js UI, Slack notifications, ULID app IDs, session
handling.
# Future Steps
- Get main green (compliance, ordered):
- Fix 47 noctx findings (Next Step).
- Fix 23 gosec findings.
- Fix 21 goconst findings.
- Run make check clean on main and keep it green; main must always
pass.
- Confirm .gitea/workflows/check.yml gates merges on make check so main
cannot regress.
- Resume feature work only after main is green.

View File

@@ -2,36 +2,74 @@ package database
import (
"context"
"database/sql"
"embed"
"errors"
"fmt"
"io/fs"
"log/slog"
"sort"
"strconv"
"strings"
)
//go:embed migrations/*.sql
var migrationsFS embed.FS
func (d *Database) migrate(ctx context.Context) error {
// Create migrations table if not exists
_, err := d.database.ExecContext(ctx, `
CREATE TABLE IF NOT EXISTS schema_migrations (
version TEXT PRIMARY KEY,
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
)
`)
if err != nil {
return fmt.Errorf("failed to create migrations table: %w", err)
// bootstrapVersion is the migration that creates the schema_migrations
// table itself. It is applied before the normal migration loop.
const bootstrapVersion = 0
// ErrInvalidMigrationFilename indicates a migration filename does not follow
// the expected "<version>.sql" or "<version>_<description>.sql" pattern.
var ErrInvalidMigrationFilename = errors.New("invalid migration filename")
// ParseMigrationVersion extracts the numeric version prefix from a migration
// filename. Filenames must follow the pattern "<version>.sql" or
// "<version>_<description>.sql", where version is a zero-padded numeric
// string (e.g. "001", "002"). Returns the version as an integer and an
// error if the filename does not match the expected pattern.
func ParseMigrationVersion(filename string) (int, error) {
name := strings.TrimSuffix(filename, ".sql")
if name == "" || name == filename {
return 0, fmt.Errorf("%w: %q has no .sql extension or is empty", ErrInvalidMigrationFilename, filename)
}
// Get list of migration files
// Split on underscore to separate version from description.
// If there's no underscore, the entire stem is the version.
versionStr, _, _ := strings.Cut(name, "_")
if versionStr == "" {
return 0, fmt.Errorf("%w: %q has empty version prefix", ErrInvalidMigrationFilename, filename)
}
// Validate the version is purely numeric.
for _, ch := range versionStr {
if ch < '0' || ch > '9' {
return 0, fmt.Errorf(
"%w: %q version %q contains non-numeric character %q",
ErrInvalidMigrationFilename, filename, versionStr, string(ch),
)
}
}
version, err := strconv.Atoi(versionStr)
if err != nil {
return 0, fmt.Errorf("%w: %q: %w", ErrInvalidMigrationFilename, filename, err)
}
return version, nil
}
// collectMigrations reads the embedded migrations directory and returns
// migration filenames sorted lexicographically.
func collectMigrations() ([]string, error) {
entries, err := fs.ReadDir(migrationsFS, "migrations")
if err != nil {
return fmt.Errorf("failed to read migrations directory: %w", err)
return nil, fmt.Errorf("failed to read migrations directory: %w", err)
}
// Sort migrations by name
migrations := make([]string, 0, len(entries))
var migrations []string
for _, entry := range entries {
if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".sql") {
@@ -41,54 +79,113 @@ func (d *Database) migrate(ctx context.Context) error {
sort.Strings(migrations)
// Apply each migration
for _, migration := range migrations {
applied, err := d.isMigrationApplied(ctx, migration)
if err != nil {
return fmt.Errorf("failed to check migration %s: %w", migration, err)
}
return migrations, nil
}
if applied {
d.log.Debug("migration already applied", "migration", migration)
// bootstrapMigrationsTable ensures the schema_migrations table exists by
// applying 000_migration.sql if the table is missing.
func bootstrapMigrationsTable(ctx context.Context, db *sql.DB, log *slog.Logger) error {
var tableExists int
continue
}
err := db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='schema_migrations'",
).Scan(&tableExists)
if err != nil {
return fmt.Errorf("failed to check for migrations table: %w", err)
}
err = d.applyMigration(ctx, migration)
if err != nil {
return fmt.Errorf("failed to apply migration %s: %w", migration, err)
}
d.log.Info("migration applied", "migration", migration)
if tableExists == 0 {
return applyBootstrapMigration(ctx, db, log)
}
return nil
}
func (d *Database) isMigrationApplied(ctx context.Context, version string) (bool, error) {
var count int
err := d.database.QueryRowContext(
ctx,
"SELECT COUNT(*) FROM schema_migrations WHERE version = ?",
version,
).Scan(&count)
// applyBootstrapMigration reads and executes 000_migration.sql to create the
// schema_migrations table on a fresh database.
func applyBootstrapMigration(ctx context.Context, db *sql.DB, log *slog.Logger) error {
content, err := migrationsFS.ReadFile("migrations/000_migration.sql")
if err != nil {
return false, fmt.Errorf("failed to query migration status: %w", err)
return fmt.Errorf("failed to read bootstrap migration 000_migration.sql: %w", err)
}
return count > 0, nil
if log != nil {
log.Info("applying bootstrap migration", "version", bootstrapVersion)
}
_, err = db.ExecContext(ctx, string(content))
if err != nil {
return fmt.Errorf("failed to apply bootstrap migration: %w", err)
}
return nil
}
func (d *Database) applyMigration(ctx context.Context, filename string) error {
content, err := migrationsFS.ReadFile("migrations/" + filename)
if err != nil {
return fmt.Errorf("failed to read migration file: %w", err)
// ApplyMigrations applies all pending migrations to db. An optional logger
// may be provided for informational output; pass nil for silent operation.
// This is exported so tests can apply the real schema without the full fx
// lifecycle.
func ApplyMigrations(ctx context.Context, db *sql.DB, log *slog.Logger) error {
bootstrapErr := bootstrapMigrationsTable(ctx, db, log)
if bootstrapErr != nil {
return bootstrapErr
}
transaction, err := d.database.BeginTx(ctx, nil)
migrations, err := collectMigrations()
if err != nil {
return fmt.Errorf("failed to begin transaction: %w", err)
return err
}
for _, migration := range migrations {
version, parseErr := ParseMigrationVersion(migration)
if parseErr != nil {
return parseErr
}
// Check if already applied.
var count int
err = db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM schema_migrations WHERE version = ?",
version,
).Scan(&count)
if err != nil {
return fmt.Errorf("failed to check migration %s: %w", migration, err)
}
if count > 0 {
if log != nil {
log.Debug("migration already applied", "version", version)
}
continue
}
// Apply migration in a transaction.
applyErr := applyMigrationTx(ctx, db, migration, version)
if applyErr != nil {
return applyErr
}
if log != nil {
log.Info("migration applied", "version", version)
}
}
return nil
}
// applyMigrationTx reads and executes a migration file within a transaction,
// recording the version in schema_migrations on success.
func applyMigrationTx(ctx context.Context, db *sql.DB, filename string, version int) error {
content, err := migrationsFS.ReadFile("migrations/" + filename)
if err != nil {
return fmt.Errorf("failed to read migration %s: %w", filename, err)
}
transaction, err := db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("failed to begin transaction for migration %s: %w", filename, err)
}
defer func() {
@@ -97,26 +194,27 @@ func (d *Database) applyMigration(ctx context.Context, filename string) error {
}
}()
// Execute migration
_, err = transaction.ExecContext(ctx, string(content))
if err != nil {
return fmt.Errorf("failed to execute migration: %w", err)
return fmt.Errorf("failed to execute migration %s: %w", filename, err)
}
// Record migration
_, err = transaction.ExecContext(
ctx,
_, err = transaction.ExecContext(ctx,
"INSERT INTO schema_migrations (version) VALUES (?)",
filename,
version,
)
if err != nil {
return fmt.Errorf("failed to record migration: %w", err)
return fmt.Errorf("failed to record migration %s: %w", filename, err)
}
err = transaction.Commit()
if err != nil {
return fmt.Errorf("failed to commit migration: %w", err)
return fmt.Errorf("failed to commit migration %s: %w", filename, err)
}
return nil
}
func (d *Database) migrate(ctx context.Context) error {
return ApplyMigrations(ctx, d.database, d.log)
}

View File

@@ -0,0 +1,9 @@
-- Migration 000: Schema migrations tracking table
-- Applied as a bootstrap step before the normal migration loop.
CREATE TABLE IF NOT EXISTS schema_migrations (
version INTEGER PRIMARY KEY,
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
INSERT OR IGNORE INTO schema_migrations (version) VALUES (0);

View File

@@ -0,0 +1,117 @@
package database_test
import (
"context"
"database/sql"
"testing"
_ "github.com/mattn/go-sqlite3"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/upaas/internal/database"
)
func TestParseMigrationVersion(t *testing.T) {
t.Parallel()
tests := []struct {
filename string
wantVersion int
wantErr bool
}{
{filename: "000_migration.sql", wantVersion: 0},
{filename: "001_initial.sql", wantVersion: 1},
{filename: "002_remove_container_id.sql", wantVersion: 2},
{filename: "007_add_resource_limits.sql", wantVersion: 7},
{filename: "100_large_version.sql", wantVersion: 100},
{filename: ".sql", wantErr: true},
{filename: "_foo.sql", wantErr: true},
{filename: "abc_foo.sql", wantErr: true},
{filename: "1a2_bad.sql", wantErr: true},
}
for _, tt := range tests {
t.Run(tt.filename, func(t *testing.T) {
t.Parallel()
version, err := database.ParseMigrationVersion(tt.filename)
if tt.wantErr {
assert.Error(t, err)
return
}
require.NoError(t, err)
assert.Equal(t, tt.wantVersion, version)
})
}
}
func TestApplyMigrationsFreshDatabase(t *testing.T) {
t.Parallel()
db, err := sql.Open("sqlite3", ":memory:?_foreign_keys=on")
require.NoError(t, err)
defer func() { _ = db.Close() }()
ctx := context.Background()
err = database.ApplyMigrations(ctx, db, nil)
require.NoError(t, err)
// Verify schema_migrations table exists with INTEGER version column.
var version int
err = db.QueryRowContext(ctx,
"SELECT version FROM schema_migrations WHERE version = 0",
).Scan(&version)
require.NoError(t, err)
assert.Equal(t, 0, version)
// Verify that all migrations were recorded.
var count int
err = db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM schema_migrations",
).Scan(&count)
require.NoError(t, err)
// 000 bootstrap + 001 through 007 = 8 entries.
assert.Equal(t, 8, count)
// Verify application tables were created by the migrations.
var tableCount int
err = db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='users'",
).Scan(&tableCount)
require.NoError(t, err)
assert.Equal(t, 1, tableCount)
}
func TestApplyMigrationsIdempotent(t *testing.T) {
t.Parallel()
db, err := sql.Open("sqlite3", ":memory:?_foreign_keys=on")
require.NoError(t, err)
defer func() { _ = db.Close() }()
ctx := context.Background()
// Apply twice — second run should be a no-op.
err = database.ApplyMigrations(ctx, db, nil)
require.NoError(t, err)
err = database.ApplyMigrations(ctx, db, nil)
require.NoError(t, err)
var count int
err = db.QueryRowContext(ctx,
"SELECT COUNT(*) FROM schema_migrations",
).Scan(&count)
require.NoError(t, err)
assert.Equal(t, 8, count)
}

121
script/bootstrap Executable file
View File

@@ -0,0 +1,121 @@
#!/bin/sh
# script/bootstrap: install all dependencies needed to build and develop
# this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes NOTHING is present (not git,
# make, or go). golangci-lint is packaged in nix, brew, and apk; on apt
# it is installed from a hash-verified GitHub release archive (never
# curl | sh).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-07-07. Never "latest"; exact versions only.
GOLANGCI_LINT_VERSION="2.10.1"
# sha256 of golangci-lint-2.10.1-linux-<arch>.tar.gz release archives
GOLANGCI_LINT_SHA256_AMD64="dfa775874cf0561b404a02a8f4481fc69b28091da95aa697259820d429b09c99"
GOLANGCI_LINT_SHA256_ARM64="6652b42ae02915eb2f9cb2a2e0cac99514c8eded8388d88ae3e06e1a52c00de8"
PKGMGR=""
SUDO=""
detect_pkgmgr() {
[ -n "$PKGMGR" ] && return 0
if command -v nix-env >/dev/null 2>&1; then
PKGMGR="nix"
elif command -v apt-get >/dev/null 2>&1; then
PKGMGR="apt"
elif command -v brew >/dev/null 2>&1; then
PKGMGR="brew"
elif command -v apk >/dev/null 2>&1; then
PKGMGR="apk"
else
echo "bootstrap: no supported package manager (nix, apt, brew, apk)" >&2
exit 1
fi
if [ "$PKGMGR" = "apt" ]; then
export DEBIAN_FRONTEND=noninteractive
if [ "$(id -u)" != "0" ]; then
SUDO="sudo"
fi
fi
}
# pkg_install <nix-attr> <apt-pkg> <brew-formula> <apk-pkg>
pkg_install() {
detect_pkgmgr
case "$PKGMGR" in
nix) nix-env -iA "nixpkgs.$1" ;;
apt) $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2" ;;
brew) brew install "$3" ;;
apk) apk add --no-cache "$4" ;;
esac
}
missing() {
! command -v "$1" >/dev/null 2>&1
}
# verify_sha256 <file> <expected-hash>
verify_sha256() {
if command -v sha256sum >/dev/null 2>&1; then
actual="$(sha256sum "$1" | cut -d' ' -f1)"
else
actual="$(shasum -a 256 "$1" | cut -d' ' -f1)"
fi
if [ "$actual" != "$2" ]; then
echo "bootstrap: sha256 mismatch for $1" >&2
echo " expected: $2" >&2
echo " actual: $actual" >&2
exit 1
fi
}
# apt has no golangci-lint package: install a pinned release archive
# from GitHub, verified by hardcoded sha256 (never curl | sh).
install_golangci_lint_release() {
case "$(uname -m)" in
x86_64) goarch="amd64"; sha="$GOLANGCI_LINT_SHA256_AMD64" ;;
aarch64|arm64) goarch="arm64"; sha="$GOLANGCI_LINT_SHA256_ARM64" ;;
*)
echo "bootstrap: unsupported architecture $(uname -m)" >&2
exit 1
;;
esac
if missing curl; then pkg_install curl curl curl curl; fi
name="golangci-lint-${GOLANGCI_LINT_VERSION}-linux-${goarch}"
tmp="$(mktemp -d)"
curl -fsSL -o "$tmp/$name.tar.gz" \
"https://github.com/golangci/golangci-lint/releases/download/v${GOLANGCI_LINT_VERSION}/${name}.tar.gz"
verify_sha256 "$tmp/$name.tar.gz" "$sha"
tar -xzf "$tmp/$name.tar.gz" -C "$tmp"
$SUDO install -m 0755 "$tmp/$name/golangci-lint" /usr/local/bin/golangci-lint
rm -rf "$tmp"
}
ensure_golangci_lint() {
if ! missing golangci-lint; then return 0; fi
detect_pkgmgr
case "$PKGMGR" in
apt) install_golangci_lint_release ;;
*) pkg_install golangci-lint golangci-lint golangci-lint golangci-lint ;;
esac
}
main() {
cd "$ROOT"
# Base tooling
if missing git; then pkg_install git git git git; fi
if missing make; then pkg_install gnumake make make make; fi
# Go toolchain and linter
if missing go; then pkg_install go golang go go; fi
ensure_golangci_lint
go mod download
echo "bootstrap complete"
}
main "$@"

15
script/check Executable file
View File

@@ -0,0 +1,15 @@
#!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. Must not modify any files.
# Generic: usually needs no adaptation.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/test"
"$SCRIPT_DIR/lint"
"$SCRIPT_DIR/fmt-check"
}
main "$@"

15
script/cibuild Executable file
View File

@@ -0,0 +1,15 @@
#!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs the checks
# (make fmt-check, lint, test), so a successful build implies a green
# repo. Generic: needs no adaptation. The Gitea workflow runs this on
# push.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
docker build .
}
main "$@"

15
script/docker Executable file
View File

@@ -0,0 +1,15 @@
#!/bin/sh
# script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname.
# Generic: needs no adaptation.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build -t "$("$SCRIPT_DIR/projectname")" .
}
main "$@"

14
script/fmt Executable file
View File

@@ -0,0 +1,14 @@
#!/bin/sh
# script/fmt: format all files (writes).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
gofmt -s -w .
goimports -w .
npx prettier --write --tab-width 4 static/js/*.js
}
main "$@"

17
script/fmt-check Executable file
View File

@@ -0,0 +1,17 @@
#!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
if [ -n "$(gofmt -l .)" ]; then
echo "Files not formatted:"
gofmt -l .
exit 1
fi
}
main "$@"

16
script/install-precommit Executable file
View File

@@ -0,0 +1,16 @@
#!/bin/sh
# script/install-precommit: install the git pre-commit hook that runs
# script/precommit. Our own extension to scripts-to-rule-them-all.
# Generic: needs no adaptation.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
printf '#!/bin/sh\nset -e\nscript/precommit\n' > .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo "pre-commit hook installed: runs script/precommit"
}
main "$@"

12
script/lint Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/sh
# script/lint: run the linter.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
golangci-lint run --config .golangci.yml ./...
}
main "$@"

21
script/precommit Executable file
View File

@@ -0,0 +1,21 @@
#!/bin/sh
# script/precommit: run by the git pre-commit hook; fails the commit if
# checks fail. Our own extension to scripts-to-rule-them-all. Go repo
# extras: go mod tidy must not change go.mod/go.sum.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
go mod tidy
git diff --exit-code -- go.mod go.sum || {
echo "precommit: go mod tidy changed go.mod/go.sum;" \
"stage the changes and retry" >&2
exit 1
}
"$SCRIPT_DIR/check"
}
main "$@"

12
script/projectname Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/sh
# script/projectname: output the name of this project. Our own
# extension to scripts-to-rule-them-all. Other scripts that need the
# name (e.g. script/docker) call this, so they can stay identical
# across all repos.
set -eu
main() {
echo "upaas"
}
main "$@"

14
script/setup Executable file
View File

@@ -0,0 +1,14 @@
#!/bin/sh
# script/setup: set up the repo for development after a fresh clone:
# installs dependencies (script/bootstrap) and the git pre-commit hook.
# Add any repo-specific initialization (db init, .env template) here.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/install-precommit"
}
main "$@"

12
script/test Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/sh
# script/test: run the test suite.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
go test -v -race -cover -timeout 30s ./...
}
main "$@"