Compare commits
27
Commits
4de726f109
...
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
15d95436a5 | ||
|
|
01823d27db | ||
|
|
a941a80bf9 | ||
|
|
55cf7f4fac | ||
|
|
c434581a54 | ||
|
|
f77faf13de | ||
|
|
ae7c3f226d | ||
|
|
2beba15ae7 | ||
|
|
23ec4026f6 | ||
|
|
ef828f71a5 | ||
|
|
f3231a3c5a | ||
|
|
cca2e3f926 | ||
|
|
708a9bec20 | ||
|
|
8314099abd | ||
|
|
8568c17d1b | ||
|
|
625fd42ace | ||
|
|
66e71b4207 | ||
|
|
847ad5b428 | ||
|
|
233a9c05ad | ||
|
|
842372250f | ||
|
|
58d601ea48 | ||
|
|
48f21d4ecf | ||
|
|
f8c437b83f | ||
|
|
04093f53ad | ||
|
|
be6c715b36 | ||
|
|
604b51eea6 | ||
|
|
ba5a716223 |
+70
-9
@@ -1,12 +1,73 @@
|
|||||||
# .git is sent without its config. Without a VERSION build argument the
|
# .dockerignore does NOT use .gitignore semantics. Docker matches with
|
||||||
# stage that compiles runs `git describe --tags --always` on .git, which
|
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
|
||||||
# does not need .git/config; that file can hold a credential, such as a
|
# `/` and an unprefixed pattern is anchored at the context root. Every
|
||||||
|
# depth-independent pattern therefore needs `**/`, or `config/.env` and
|
||||||
|
# `certs/server.key` still ship while this file reads as solved. Only
|
||||||
|
# genuinely root-anchored entries go unprefixed. Never transplant these
|
||||||
|
# into .gitignore, where `**/` is wrong.
|
||||||
|
#
|
||||||
|
# Matching is case-sensitive, so secrets use character ranges rather
|
||||||
|
# than an ALL-CAPS twin, which would still miss `Server.Key`.
|
||||||
|
#
|
||||||
|
# Extend with this repo's own host-built artifacts, written anchored:
|
||||||
|
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
|
||||||
|
# deletes the package directory from the context.
|
||||||
|
|
||||||
|
# Unlike the standard file, which leaves out all of .git, pixa sends
|
||||||
|
# .git without its config. Without a VERSION build argument the stage
|
||||||
|
# that compiles runs `git describe --tags --always` on .git, which does
|
||||||
|
# not need .git/config; that file can hold a credential, such as a
|
||||||
# password in a remote URL or the token the CI checkout step stores there.
|
# password in a remote URL or the token the CI checkout step stores there.
|
||||||
.git/config
|
.git/config
|
||||||
.gitignore
|
|
||||||
.DS_Store
|
# Agent scratch: one full checkout of the repo per in-flight agent.
|
||||||
.env*
|
# Anchored because it occurs once where agents run at the repo root.
|
||||||
|
# KNOWN GAP: a repo running agents in subdirectories still ships
|
||||||
|
# `services/api/.claude/` and must add its own anchored entry.
|
||||||
.claude
|
.claude
|
||||||
node_modules
|
|
||||||
bin/
|
# Environment files. `*.env` covers bare `.env` and the `prod.env`
|
||||||
data/
|
# convention. Re-include a committed template with a negation if the
|
||||||
|
# build needs one: `!docs/example.env`.
|
||||||
|
**/*.[eE][nN][vV]
|
||||||
|
**/.[eE][nN][vV].*
|
||||||
|
**/.[eE][nN][vV][rR][cC]
|
||||||
|
|
||||||
|
# Private keys and the bundles carrying them. Public certificates
|
||||||
|
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
|
||||||
|
**/*.[pP][eE][mM]
|
||||||
|
**/*.[kK][eE][yY]
|
||||||
|
**/*.[pP]12
|
||||||
|
**/*.[pP][fF][xX]
|
||||||
|
**/[iI][dD]_[rR][sS][aA]
|
||||||
|
**/[iI][dD]_[dD][sS][aA]
|
||||||
|
**/[iI][dD]_[eE][cC][dD][sS][aA]
|
||||||
|
**/[iI][dD]_[eE][dD]25519
|
||||||
|
|
||||||
|
# Dependencies: restored inside the image, never copied in.
|
||||||
|
**/node_modules
|
||||||
|
|
||||||
|
# OS metadata.
|
||||||
|
**/.DS_Store
|
||||||
|
**/Thumbs.db
|
||||||
|
|
||||||
|
# Editor state: never a build input, and it churns COPY.
|
||||||
|
**/*.swp
|
||||||
|
**/*.swo
|
||||||
|
**/*~
|
||||||
|
**/*.bak
|
||||||
|
**/.idea
|
||||||
|
**/.vscode
|
||||||
|
**/*.sublime-*
|
||||||
|
|
||||||
|
# pixa's own entries. Nothing in the build reads .gitignore. On the
|
||||||
|
# host, `make build` writes bin/pixad, and the example config keeps its
|
||||||
|
# state directory in data/.
|
||||||
|
.gitignore
|
||||||
|
/bin
|
||||||
|
/data
|
||||||
|
|
||||||
|
# Local config files, kept out of git because they can hold the signing key.
|
||||||
|
**/[cC][oO][nN][fF][iI][gG].[yY][mM][lL]
|
||||||
|
**/[cC][oO][nN][fF][iI][gG].[yY][aA][mM][lL]
|
||||||
|
**/[cC][oO][nN][fF][iI][gG].[dD][eE][vV].[yY][mM][lL]
|
||||||
|
|||||||
@@ -0,0 +1,4 @@
|
|||||||
|
# Every PR adds an entry at the top of TODO.md's Completed Steps; union keeps
|
||||||
|
# both sides instead of conflicting. Git never reports a conflict here: read
|
||||||
|
# the merged entries after every merge or rebase.
|
||||||
|
TODO.md merge=union
|
||||||
@@ -6,5 +6,10 @@ jobs:
|
|||||||
steps:
|
steps:
|
||||||
# actions/checkout v4.2.2, 2026-02-22
|
# actions/checkout v4.2.2, 2026-02-22
|
||||||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
||||||
|
# The default clone is shallow and has no tags, so the
|
||||||
|
# version the build takes from `git describe` would be a
|
||||||
|
# bare commit; this fetches the whole history with its tags.
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
- run: script/cibuild
|
- run: script/cibuild
|
||||||
- run: script/docker-smoke
|
- run: script/docker-smoke
|
||||||
|
|||||||
@@ -11,6 +11,12 @@ Thumbs.db
|
|||||||
.vscode/
|
.vscode/
|
||||||
*.sublime-*
|
*.sublime-*
|
||||||
|
|
||||||
|
# Agent scratch (worktrees of this repo, created and destroyed by
|
||||||
|
# in-flight tooling). Unanchored: .gitignore patterns already match at
|
||||||
|
# every depth, so no prefix is wanted here. This is not a .dockerignore
|
||||||
|
# entry and must not be given a `**/` prefix on the way into one.
|
||||||
|
.claude/
|
||||||
|
|
||||||
# Environment / secrets
|
# Environment / secrets
|
||||||
.env
|
.env
|
||||||
.env.*
|
.env.*
|
||||||
@@ -31,5 +37,6 @@ node_modules/
|
|||||||
*.sqlite3
|
*.sqlite3
|
||||||
|
|
||||||
# Local dev configs
|
# Local dev configs
|
||||||
|
config.yml
|
||||||
config.yaml
|
config.yaml
|
||||||
config.dev.yml
|
config.dev.yml
|
||||||
|
|||||||
+66
-2
@@ -10,14 +10,20 @@ run:
|
|||||||
|
|
||||||
linters:
|
linters:
|
||||||
default: all
|
default: all
|
||||||
|
enable:
|
||||||
|
# Successor to the deprecated gomodguard. Named explicitly, rather than
|
||||||
|
# left to `default: all`, because it carries the module policy below.
|
||||||
|
- gomodguard_v2
|
||||||
disable:
|
disable:
|
||||||
# Genuinely incompatible with project patterns
|
# Genuinely incompatible with project patterns
|
||||||
- exhaustruct # Requires all struct fields
|
- exhaustruct # Requires all struct fields
|
||||||
- depguard # Dependency allow/block lists
|
|
||||||
- godot # Requires comments to end with periods
|
- godot # Requires comments to end with periods
|
||||||
- wsl # Deprecated, replaced by wsl_v5
|
|
||||||
- wrapcheck # Too verbose for internal packages
|
- wrapcheck # Too verbose for internal packages
|
||||||
- varnamelen # Short names like db, id are idiomatic Go
|
- varnamelen # Short names like db, id are idiomatic Go
|
||||||
|
# Deprecated: the warning is attached to the old name, so it is
|
||||||
|
# silenced by disabling that name, not by enabling the successor.
|
||||||
|
- wsl # Deprecated, replaced by wsl_v5
|
||||||
|
- gomodguard # Deprecated, replaced by gomodguard_v2
|
||||||
settings:
|
settings:
|
||||||
lll:
|
lll:
|
||||||
line-length: 88
|
line-length: 88
|
||||||
@@ -28,6 +34,64 @@ linters:
|
|||||||
max-complexity: 15
|
max-complexity: 15
|
||||||
dupl:
|
dupl:
|
||||||
threshold: 100
|
threshold: 100
|
||||||
|
depguard:
|
||||||
|
# Test-support code must not be compiled into the shipped binary. A
|
||||||
|
# test-support package exists to hand a test privileges the program
|
||||||
|
# itself must never have, so a file that is not a test must not import
|
||||||
|
# one. Test files, and the files inside a package whose directory name
|
||||||
|
# ends in `test`, are where that code belongs, and are exempt.
|
||||||
|
#
|
||||||
|
# The deny list below is the one part of this file a repository is
|
||||||
|
# expected to extend, and the only part it may. depguard matches an
|
||||||
|
# import path against a list of prefixes, so it cannot be told "any path
|
||||||
|
# whose last segment ends in test"; a repository's own test-support
|
||||||
|
# packages have to be named here one at a time, by full import path,
|
||||||
|
# under a module path that differs from repository to repository. Add
|
||||||
|
# them; change nothing else.
|
||||||
|
rules:
|
||||||
|
test-support:
|
||||||
|
list-mode: lax
|
||||||
|
files:
|
||||||
|
- "$all"
|
||||||
|
- "!$test"
|
||||||
|
- "!**/*test/**"
|
||||||
|
deny:
|
||||||
|
- pkg: net/http/httptest
|
||||||
|
desc: >-
|
||||||
|
Test-support code belongs in test files and in packages whose
|
||||||
|
directory name ends in test, not in the shipped binary.
|
||||||
|
# Only decisions already recorded in the Go package defaults are
|
||||||
|
# listed here. Every entry matches the module path exactly.
|
||||||
|
gomodguard_v2:
|
||||||
|
blocked:
|
||||||
|
- module: github.com/rs/zerolog
|
||||||
|
recommendations:
|
||||||
|
- log/slog
|
||||||
|
reason: "Structured logging is stdlib log/slog."
|
||||||
|
# One entry per pre-fork module path, because the later releases
|
||||||
|
# are separate paths. A prefix match would be shorter but would
|
||||||
|
# also reach github.com/go-redis/redismock, the test double for
|
||||||
|
# the successor these entries recommend.
|
||||||
|
- module: github.com/go-redis/redis
|
||||||
|
recommendations:
|
||||||
|
- github.com/redis/go-redis/v9
|
||||||
|
reason: "Pre-fork module; use the maintained go-redis v9."
|
||||||
|
- module: github.com/go-redis/redis/v7
|
||||||
|
recommendations:
|
||||||
|
- github.com/redis/go-redis/v9
|
||||||
|
reason: "Pre-fork module; use the maintained go-redis v9."
|
||||||
|
- module: github.com/go-redis/redis/v8
|
||||||
|
recommendations:
|
||||||
|
- github.com/redis/go-redis/v9
|
||||||
|
reason: "Pre-fork module; use the maintained go-redis v9."
|
||||||
|
- module: github.com/sergi/go-diff
|
||||||
|
recommendations:
|
||||||
|
- github.com/aymanbagabas/go-udiff
|
||||||
|
reason: "No unified diff output; use go-udiff."
|
||||||
|
- module: github.com/hexops/gotextdiff
|
||||||
|
recommendations:
|
||||||
|
- github.com/aymanbagabas/go-udiff
|
||||||
|
reason: "Unmaintained fork; use go-udiff."
|
||||||
|
|
||||||
issues:
|
issues:
|
||||||
max-issues-per-linter: 0
|
max-issues-per-linter: 0
|
||||||
|
|||||||
@@ -0,0 +1,7 @@
|
|||||||
|
node_modules/
|
||||||
|
yarn.lock
|
||||||
|
|
||||||
|
# A byte-for-byte copy of the one in sneak/prompts.
|
||||||
|
REPO_POLICIES.md
|
||||||
|
|
||||||
|
vendor/
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
{
|
||||||
|
"tabWidth": 4,
|
||||||
|
"proseWrap": "always"
|
||||||
|
}
|
||||||
@@ -4,73 +4,68 @@ Last Updated 2026-01-08
|
|||||||
|
|
||||||
These rules MUST be followed at all times, it is very important.
|
These rules MUST be followed at all times, it is very important.
|
||||||
|
|
||||||
* Never use `git add -A` - add specific changes to a deliberate commit. A
|
- Never use `git add -A` - add specific changes to a deliberate commit. A commit
|
||||||
commit should contain one change. After each change, make a commit with a
|
should contain one change. After each change, make a commit with a good
|
||||||
good one-line summary.
|
one-line summary.
|
||||||
|
|
||||||
* NEVER modify the linter config without asking first.
|
- NEVER modify the linter config without asking first.
|
||||||
|
|
||||||
* NEVER modify tests to exclude special cases or otherwise get them to pass
|
- NEVER modify tests to exclude special cases or otherwise get them to pass
|
||||||
without asking first. In almost all cases, the code should be changed,
|
without asking first. In almost all cases, the code should be changed, NOT the
|
||||||
NOT the tests. If you think the test needs to be changed, make your case
|
tests. If you think the test needs to be changed, make your case for that and
|
||||||
for that and ask for permission to proceed, then stop. You need explicit
|
ask for permission to proceed, then stop. You need explicit user approval to
|
||||||
user approval to modify existing tests. (You do not need user approval
|
modify existing tests. (You do not need user approval for writing NEW tests.)
|
||||||
for writing NEW tests.)
|
|
||||||
|
|
||||||
* When linting, assume the linter config is CORRECT, and that each item
|
- When linting, assume the linter config is CORRECT, and that each item output
|
||||||
output by the linter is something that legitimately needs fixing in the
|
by the linter is something that legitimately needs fixing in the code.
|
||||||
code.
|
|
||||||
|
|
||||||
* When running tests, use `make test`.
|
- When running tests, use `make test`.
|
||||||
|
|
||||||
* Before commits, run `make check`. This runs `make lint` and `make test`
|
- Before commits, run `make check`. This runs `make lint` and `make test` and
|
||||||
and `make check-fmt`. Any issues discovered MUST be resolved before
|
`make check-fmt`. Any issues discovered MUST be resolved before committing
|
||||||
committing unless explicitly told otherwise.
|
unless explicitly told otherwise.
|
||||||
|
|
||||||
* When fixing a bug, write a failing test for the bug FIRST. Add
|
- When fixing a bug, write a failing test for the bug FIRST. Add appropriate
|
||||||
appropriate logging to the test to ensure it is written correctly. Commit
|
logging to the test to ensure it is written correctly. Commit that. Then go
|
||||||
that. Then go about fixing the bug until the test passes (without
|
about fixing the bug until the test passes (without modifying the test
|
||||||
modifying the test further). Then commit that.
|
further). Then commit that.
|
||||||
|
|
||||||
* When adding a new feature, do the same - implement a test first (TDD). It
|
- When adding a new feature, do the same - implement a test first (TDD). It
|
||||||
doesn't have to be super complex. Commit the test, then commit the
|
doesn't have to be super complex. Commit the test, then commit the feature.
|
||||||
feature.
|
|
||||||
|
|
||||||
* When adding a new feature, use a feature branch. When the feature is
|
- When adding a new feature, use a feature branch. When the feature is
|
||||||
completely finished and the code is up to standards (passes `make check`)
|
completely finished and the code is up to standards (passes `make check`) then
|
||||||
then and only then can the feature branch be merged into `main` and the
|
and only then can the feature branch be merged into `main` and the branch
|
||||||
branch deleted.
|
deleted.
|
||||||
|
|
||||||
* Write godoc documentation comments for all exported types and functions as
|
- Write godoc documentation comments for all exported types and functions as you
|
||||||
you go along.
|
go along.
|
||||||
|
|
||||||
* ALWAYS be consistent in naming. If you name something one thing in one
|
- ALWAYS be consistent in naming. If you name something one thing in one place,
|
||||||
place, name it the EXACT SAME THING in another place.
|
name it the EXACT SAME THING in another place.
|
||||||
|
|
||||||
* Be descriptive and specific in naming. `wl` is bad;
|
- Be descriptive and specific in naming. `wl` is bad; `SourceHostWhitelist` is
|
||||||
`SourceHostWhitelist` is good. `ConnsPerHost` is bad;
|
good. `ConnsPerHost` is bad; `MaxConnectionsPerHost` is good.
|
||||||
`MaxConnectionsPerHost` is good.
|
|
||||||
|
|
||||||
* This is not prototype or teaching code - this is designed for production.
|
- This is not prototype or teaching code - this is designed for production. Any
|
||||||
Any security issues (such as denial of service) or other web
|
security issues (such as denial of service) or other web vulnerabilities are
|
||||||
vulnerabilities are P1 bugs and must be added to TODO.md at the top.
|
P1 bugs and must be added to TODO.md at the top.
|
||||||
|
|
||||||
* As this is production code, no stubbing of implementations unless
|
- As this is production code, no stubbing of implementations unless specifically
|
||||||
specifically instructed. We need working implementations.
|
instructed. We need working implementations.
|
||||||
|
|
||||||
* NEVER silently fall back to a different setting when a user's parameter
|
- NEVER silently fall back to a different setting when a user's parameter
|
||||||
explicitly specifies a value. If a user requests format=webp and WebP
|
explicitly specifies a value. If a user requests format=webp and WebP encoding
|
||||||
encoding is not supported, return an error - do NOT silently output PNG
|
is not supported, return an error - do NOT silently output PNG instead. If a
|
||||||
instead. If a user specifies fit=invalid and that fit mode doesn't exist,
|
user specifies fit=invalid and that fit mode doesn't exist, return an error -
|
||||||
return an error - do NOT silently default to "cover". Silent fallbacks
|
do NOT silently default to "cover". Silent fallbacks violate the principle of
|
||||||
violate the principle of least surprise and mask bugs. The only acceptable
|
least surprise and mask bugs. The only acceptable defaults are for OMITTED
|
||||||
defaults are for OMITTED parameters, never for INVALID explicit values.
|
parameters, never for INVALID explicit values.
|
||||||
|
|
||||||
* Avoid vendoring deps unless specifically instructed to. NEVER commit
|
- Avoid vendoring deps unless specifically instructed to. NEVER commit the
|
||||||
the vendor directory, NEVER commit compiled binaries. If these
|
vendor directory, NEVER commit compiled binaries. If these directories or
|
||||||
directories or files exist, add them to .gitignore (and commit the
|
files exist, add them to .gitignore (and commit the .gitignore) if they are
|
||||||
.gitignore) if they are not already in there. Keep the entire git
|
not already in there. Keep the entire git repository (with history) small -
|
||||||
repository (with history) small - under 20MiB, unless you specifically
|
under 20MiB, unless you specifically must commit larger files (e.g. test
|
||||||
must commit larger files (e.g. test fixture example media files). Only
|
fixture example media files). Only OUR source code and immediately supporting
|
||||||
OUR source code and immediately supporting files (such as test examples)
|
files (such as test examples) goes into the repo/history.
|
||||||
goes into the repo/history.
|
|
||||||
|
|||||||
-1259
File diff suppressed because it is too large
Load Diff
+38
-30
@@ -1,55 +1,62 @@
|
|||||||
# Lint stage
|
# Lint phase. script/lint builds it alone. The linter is run directly:
|
||||||
# Same image as Dockerfile.lint: change both pins together.
|
# `make lint` and script/lint are themselves a docker build.
|
||||||
# golangci/golangci-lint:v2.12.2-alpine, 2026-08-07
|
# golangci/golangci-lint:v2.12.2, 2026-10-04
|
||||||
FROM golangci/golangci-lint:v2.12.2-alpine@sha256:91b27804074a0bacea298707f016911e60cf0cdbc6c7bf5ccacb5f0606d18d60 AS lint
|
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
|
||||||
|
|
||||||
|
# The linter compiles every package, and govips needs the libvips
|
||||||
|
# headers for that. REPO_POLICIES.md has the lint phase install them
|
||||||
|
# itself; this image is Debian, so with apt-get rather than apk.
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends libvips-dev \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /src
|
||||||
|
COPY go.mod go.sum ./
|
||||||
|
RUN go mod download
|
||||||
|
COPY . .
|
||||||
|
RUN golangci-lint run --config .golangci.yml ./...
|
||||||
|
|
||||||
|
# Test phase. script/test builds it alone.
|
||||||
|
# golang:1.25.4-alpine, 2026-02-25
|
||||||
|
FROM golang:1.25.4-alpine@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS test
|
||||||
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
# script/bootstrap installs the build dependencies and downloads the Go
|
# script/bootstrap --cgo installs the build dependencies (a C compiler
|
||||||
# modules. Only script/, go.mod and go.sum are copied first, so this
|
# and the libvips and libheif headers) and downloads the Go modules.
|
||||||
# layer is reused until one of them changes.
|
|
||||||
COPY script/ ./script/
|
COPY script/ ./script/
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN script/bootstrap
|
RUN script/bootstrap --cgo
|
||||||
|
|
||||||
# Copy source code
|
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# Tells script/lint it is inside a container, so it runs the linter.
|
# Without -v first; on a failure, again with -v for the details, and
|
||||||
ENV container=docker
|
# the step fails even if the second run passes.
|
||||||
|
RUN go test -count=1 -timeout 90s -race -cover ./... || \
|
||||||
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
|
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
|
||||||
|
|
||||||
# Run formatting check and linter. script/cibuild and script/docker pass
|
# Build stage. Nothing is wanted from the two phases above: these copies
|
||||||
# a new CHECK_EPOCH on every run, and each check step names it in its
|
# make BuildKit build them first, so this stage runs only when lint and
|
||||||
# command, so a new value reruns the step instead of reusing a cached
|
# test passed.
|
||||||
# success that checked nothing. A plain `docker build .` leaves it empty
|
|
||||||
# and reuses the check steps only for an identical build context.
|
|
||||||
ARG CHECK_EPOCH
|
|
||||||
RUN echo "check epoch: ${CHECK_EPOCH}" && make fmt-check
|
|
||||||
RUN echo "check epoch: ${CHECK_EPOCH}" && make lint
|
|
||||||
|
|
||||||
# Build stage
|
|
||||||
# golang:1.25.4-alpine, 2026-02-25
|
# golang:1.25.4-alpine, 2026-02-25
|
||||||
FROM golang:1.25.4-alpine@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
|
FROM golang:1.25.4-alpine@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
|
||||||
|
|
||||||
# Depend on lint stage passing
|
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
|
COPY --from=test /src/go.sum /dev/null
|
||||||
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
# Build dependencies and Go modules, as in the lint stage
|
# Build dependencies and Go modules, as in the test phase
|
||||||
COPY script/ ./script/
|
COPY script/ ./script/
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN script/bootstrap
|
RUN script/bootstrap --cgo
|
||||||
|
|
||||||
# Copy source code
|
# Copy source code
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# Run tests; a new CHECK_EPOCH reruns them, as in the lint stage.
|
|
||||||
ARG CHECK_EPOCH
|
|
||||||
RUN echo "check epoch: ${CHECK_EPOCH}" && make test
|
|
||||||
|
|
||||||
# VERSION is declared here, not earlier: a new value reruns only the
|
# VERSION is declared here, not earlier: a new value reruns only the
|
||||||
# build, not script/bootstrap or the tests. Given none, the version is
|
# build, not script/bootstrap. Given none, the version is
|
||||||
# `git describe --tags --always` of the .git in the build context (git
|
# `git describe --tags --always` of the .git in the build context (git
|
||||||
# comes from script/bootstrap): the tag on a tagged commit, tag-N-gHASH
|
# comes from script/bootstrap): the tag on a tagged commit, tag-N-gHASH
|
||||||
# after one, the short commit when no tag is reachable. A context that
|
# after one, the short commit when no tag is reachable. A context that
|
||||||
@@ -68,7 +75,8 @@ RUN version="${VERSION:-$(git describe --tags --always)}"; \
|
|||||||
-ldflags "-s -w -X main.Version=${version}" \
|
-ldflags "-s -w -X main.Version=${version}" \
|
||||||
-o /pixad ./cmd/pixad
|
-o /pixad ./cmd/pixad
|
||||||
|
|
||||||
# Runtime stage
|
# Runtime stage, and the last one: a plain `docker build .` builds this
|
||||||
|
# stage and what it depends on, and nothing else.
|
||||||
# alpine:3.21, 2026-02-25
|
# alpine:3.21, 2026-02-25
|
||||||
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
|
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
|
||||||
|
|
||||||
|
|||||||
@@ -1,34 +0,0 @@
|
|||||||
# Dockerfile.lint: the container script/lint builds to run golangci-lint,
|
|
||||||
# which is never installed on the host. Pinned to the same image as the
|
|
||||||
# Dockerfile lint stage: change both pins together, or the two run
|
|
||||||
# different linter versions.
|
|
||||||
#
|
|
||||||
# golangci/golangci-lint:v2.12.2-alpine, 2026-08-07
|
|
||||||
FROM golangci/golangci-lint:v2.12.2-alpine@sha256:91b27804074a0bacea298707f016911e60cf0cdbc6c7bf5ccacb5f0606d18d60
|
|
||||||
|
|
||||||
WORKDIR /src
|
|
||||||
|
|
||||||
# pixa is CGO/libvips: the type-aware linters compile every package, so
|
|
||||||
# this image needs the same C libraries the build does. script/bootstrap
|
|
||||||
# installs them and downloads the Go modules. Only script/, go.mod and
|
|
||||||
# go.sum are copied first; they settle this layer's result, so it may
|
|
||||||
# safely be reused between runs.
|
|
||||||
COPY script/ ./script/
|
|
||||||
COPY go.mod go.sum ./
|
|
||||||
RUN script/bootstrap
|
|
||||||
|
|
||||||
COPY . .
|
|
||||||
|
|
||||||
# Tells script/lint it is inside a container, so it runs the linter.
|
|
||||||
ENV container=docker
|
|
||||||
|
|
||||||
# script/lint passes a different CACHEBUST on every run, and BuildKit
|
|
||||||
# keys every RUN after this ARG on its value, so the lint step always
|
|
||||||
# runs instead of returning a cached success that linted nothing.
|
|
||||||
#
|
|
||||||
# Go's and golangci-lint's caches (/root/.cache, hundreds of MB) go on a
|
|
||||||
# tmpfs that is discarded after the step. Written into the layer, they
|
|
||||||
# would pile up as build cache on every run, since no later run, with
|
|
||||||
# its new CACHEBUST, can reuse that layer.
|
|
||||||
ARG CACHEBUST
|
|
||||||
RUN --mount=type=tmpfs,target=/root/.cache script/lint
|
|
||||||
@@ -1,10 +1,10 @@
|
|||||||
.PHONY: bootstrap setup check lint test fmt fmt-check build clean docker docker-smoke docker-versioned docker-test devserver devserver-stop hooks
|
.PHONY: bootstrap setup check lint test fmt fmt-check build clean docker docker-smoke docker-versioned docker-test devserver devserver-stop hooks loadtest
|
||||||
|
|
||||||
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev")
|
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev")
|
||||||
LDFLAGS := -X main.Version=$(VERSION)
|
LDFLAGS := -X main.Version=$(VERSION)
|
||||||
|
|
||||||
# Use nix-shell to provide CGO dependencies unless they are already available
|
# Use nix-shell to provide CGO dependencies unless they are already available
|
||||||
# (e.g. inside a Docker build or an existing nix-shell).
|
# (e.g. inside an existing nix-shell).
|
||||||
HAS_PKGCONFIG := $(shell command -v pkg-config 2>/dev/null)
|
HAS_PKGCONFIG := $(shell command -v pkg-config 2>/dev/null)
|
||||||
ifdef HAS_PKGCONFIG
|
ifdef HAS_PKGCONFIG
|
||||||
NIX_RUN_PREFIX =
|
NIX_RUN_PREFIX =
|
||||||
@@ -32,11 +32,11 @@ fmt-check:
|
|||||||
fmt:
|
fmt:
|
||||||
@script/fmt
|
@script/fmt
|
||||||
|
|
||||||
# Run linter
|
# Run linter (the lint phase of the Dockerfile)
|
||||||
lint:
|
lint:
|
||||||
@script/lint
|
@script/lint
|
||||||
|
|
||||||
# Run tests (30-second timeout)
|
# Run tests (the test phase of the Dockerfile)
|
||||||
test:
|
test:
|
||||||
@script/test
|
@script/test
|
||||||
|
|
||||||
@@ -59,20 +59,25 @@ docker:
|
|||||||
docker-smoke:
|
docker-smoke:
|
||||||
@script/docker-smoke
|
@script/docker-smoke
|
||||||
|
|
||||||
# Build Docker image tagged pixad:$(VERSION) and pixad:latest
|
# Measure throughput, latency and peak memory with the default duration and
|
||||||
docker-versioned:
|
# number of clients (needs Docker and Go; a benchmark, not part of check)
|
||||||
docker build --build-arg VERSION=$(VERSION) -t pixad:$(VERSION) -t pixad:latest .
|
loadtest:
|
||||||
|
@script/loadtest
|
||||||
|
|
||||||
# Run tests in Docker (needed for CGO/libvips)
|
# Build Docker image as `make docker` does, and also tag it pixa:$(VERSION)
|
||||||
|
docker-versioned:
|
||||||
|
@script/docker
|
||||||
|
docker tag pixa pixa:$(VERSION)
|
||||||
|
|
||||||
|
# Run tests in Docker, as `make test` does
|
||||||
docker-test:
|
docker-test:
|
||||||
docker build --target builder --build-arg VERSION=$(VERSION) -t pixad-builder .
|
@script/test
|
||||||
docker run --rm pixad-builder sh -c "CGO_ENABLED=1 GOTOOLCHAIN=auto go test -v ./..."
|
|
||||||
|
|
||||||
# Run local dev server in Docker
|
# Run local dev server in Docker
|
||||||
devserver: docker-versioned devserver-stop
|
devserver: docker-versioned devserver-stop
|
||||||
docker run -d --name pixad-dev -p 8080:8080 \
|
docker run -d --name pixad-dev -p 8080:8080 \
|
||||||
-v $(CURDIR)/config.dev.yml:/etc/pixa/config.yml:ro \
|
-v $(CURDIR)/config.dev.yml:/etc/pixa/config.yml:ro \
|
||||||
pixad:latest
|
pixa:latest
|
||||||
@echo "pixad running at http://localhost:8080"
|
@echo "pixad running at http://localhost:8080"
|
||||||
|
|
||||||
# Stop dev server
|
# Stop dev server
|
||||||
|
|||||||
@@ -1,10 +1,9 @@
|
|||||||
# pixa
|
# pixa
|
||||||
|
|
||||||
pixa is a GPL-3.0-licensed Go web server by
|
pixa is a GPL-3.0-licensed Go web server by [@sneak](https://sneak.berlin) that
|
||||||
[@sneak](https://sneak.berlin) that proxies images from upstream
|
proxies images from upstream sources, optionally resizing or transforming them,
|
||||||
sources, optionally resizing or transforming them, and serves the
|
and serves the results. Both source and transformed images are cached to disk so
|
||||||
results. Both source and transformed images are cached to disk so that
|
that subsequent requests are served without origin fetches or additional
|
||||||
subsequent requests are served without origin fetches or additional
|
|
||||||
processing.
|
processing.
|
||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
@@ -18,7 +17,7 @@ make build
|
|||||||
# run with a config file: copy the example and set a real signing key
|
# run with a config file: copy the example and set a real signing key
|
||||||
# (the example placeholder is refused at startup), e.g. with
|
# (the example placeholder is refused at startup), e.g. with
|
||||||
# openssl rand -base64 32
|
# openssl rand -base64 32
|
||||||
cp config.example.yml config.yml
|
cp configs/config.example.yml config.yml
|
||||||
$EDITOR config.yml # replace the signing_key placeholder
|
$EDITOR config.yml # replace the signing_key placeholder
|
||||||
./bin/pixad --config config.yml
|
./bin/pixad --config config.yml
|
||||||
|
|
||||||
@@ -27,44 +26,106 @@ make docker
|
|||||||
docker run -p 8080:8080 -e PIXA_SIGNING_KEY="$(openssl rand -base64 32)" pixa:latest
|
docker run -p 8080:8080 -e PIXA_SIGNING_KEY="$(openssl rand -base64 32)" pixa:latest
|
||||||
```
|
```
|
||||||
|
|
||||||
A container takes its settings from environment variables (see
|
A container takes its settings from environment variables (see Configuration
|
||||||
Configuration below for the list). Only `PIXA_SIGNING_KEY` is required; if
|
below for the list). Only `PIXA_SIGNING_KEY` is required; if it is unset the
|
||||||
it is unset the container exits at startup naming the variable. Everything
|
container exits at startup naming the variable. Everything else has a built-in
|
||||||
else has a built-in default. A config file mounted at `/etc/pixa/config.yml`
|
default. A config file mounted at `/etc/pixa/config.yml` is optional: it is read
|
||||||
is optional: it is read when present, and an environment variable wins over
|
when present, and an environment variable wins over the same setting in it.
|
||||||
the same setting in it.
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
pixa listens on plain HTTP and runs behind a reverse proxy that terminates TLS.
|
||||||
|
[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy, chosen because
|
||||||
|
it is the smallest correct one: Caddy gets the TLS certificate itself and does
|
||||||
|
everything in this list without further settings. The reverse proxy must:
|
||||||
|
|
||||||
|
- terminate TLS, as the login and generator pages work only over HTTPS (see
|
||||||
|
Routes);
|
||||||
|
- pass the `Host`, `Origin` and `Referer` headers on unchanged, as pixa refuses
|
||||||
|
a form from those pages unless `Origin` or `Referer` names the host in `Host`,
|
||||||
|
builds encrypted URLs from `Host`, and checks `Referer` against
|
||||||
|
`referer_blocklist`;
|
||||||
|
- set `X-Forwarded-For` to the client's address, with `trusted_proxies` set to
|
||||||
|
the address pixa sees the proxy's requests come from, so the login limit
|
||||||
|
counts each client by its own address (see `trusted_proxies` under
|
||||||
|
Configuration);
|
||||||
|
- wait for pixa's answer for at least `downstream_timeout` (default `60s`), the
|
||||||
|
longest pixa takes to fetch, convert and send an image.
|
||||||
|
|
||||||
|
It may also refuse `/metrics`, as the example does, so that only a scraper that
|
||||||
|
reaches pixa directly can read it; pixa itself asks for the metrics username and
|
||||||
|
password there.
|
||||||
|
|
||||||
|
pixa does the rest itself: it checks signatures and encrypted URLs, applies the
|
||||||
|
allowlist, refuses upstream hosts with private or local addresses, limits login
|
||||||
|
attempts, upstream response size and image dimensions, and sends the security
|
||||||
|
headers, `Strict-Transport-Security` included, with every response.
|
||||||
|
|
||||||
|
The state directory (`state_dir`, `/var/lib/pixa` in the container) holds the
|
||||||
|
database and the disk cache:
|
||||||
|
|
||||||
|
- It needs a persistent volume: without one, every restart starts with an empty
|
||||||
|
cache. In the container, the startup script gives the directory to the user
|
||||||
|
pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
|
||||||
|
must be able to write the directory.
|
||||||
|
- `cache_max_bytes` limits the source and transformed images together. The
|
||||||
|
database, the metadata files, the `.meta` file beside each transformed image
|
||||||
|
and files still being written come on top, and eviction runs in the
|
||||||
|
background, so the cache can pass the limit for a while: leave room on the
|
||||||
|
volume beyond it.
|
||||||
|
- Set `cache_max_bytes` for a lasting deployment. Its default, worked out each
|
||||||
|
time pixa starts, is 75% of the sum of the space free on the volume and the
|
||||||
|
space the cached images already take, so a restart keeps the limit the cache
|
||||||
|
had, but anything else that fills or frees space on the volume moves it.
|
||||||
|
|
||||||
|
A load balancer's health check can request `/.well-known/healthcheck.json`,
|
||||||
|
which answers 200 whenever pixa is running, in maintenance mode too (see
|
||||||
|
`maintenance_mode`).
|
||||||
|
|
||||||
|
On SIGTERM or SIGINT pixa stops accepting connections, gives the requests in
|
||||||
|
progress and the images being processed 5 seconds to finish, and exits: with 0,
|
||||||
|
or with 1 when images were still being processed after those 5 seconds or
|
||||||
|
another part of pixa failed to stop. A request not finished by then is cut off.
|
||||||
|
`docker stop` waits 10 seconds before it kills the container.
|
||||||
|
|
||||||
|
Outside Docker, pixa needs libvips (the image has 8.15) and libheif to run, as
|
||||||
|
it uses libvips through CGO; building it also needs their development files,
|
||||||
|
`pkg-config` and a C compiler. `script/bootstrap --cgo` installs all of these,
|
||||||
|
as the `Dockerfile` does where it compiles pixa. Plain `script/bootstrap`, which
|
||||||
|
`script/setup` and `script/cibuild` run, installs git, make and Go, and Node,
|
||||||
|
Yarn and the prettier pinned in `yarn.lock` for formatting the markdown, but
|
||||||
|
none of the C libraries: the checks compile pixa in Docker. Docker itself must
|
||||||
|
already be installed.
|
||||||
|
|
||||||
## Running under upaas
|
## Running under upaas
|
||||||
|
|
||||||
What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
|
What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
|
||||||
|
|
||||||
- **Port:** pixa listens on container port `8080`.
|
- **Port:** pixa listens on container port `8080`.
|
||||||
- **Volume:** container path `/var/lib/pixa`, where pixa keeps its
|
- **Volume:** container path `/var/lib/pixa`, where pixa keeps its database and
|
||||||
database and cache. Creating the host directory when it is missing is
|
cache. Creating the host directory when it is missing is upaas's job, tracked
|
||||||
upaas's job, tracked in https://git.eeqj.de/sneak/upaas/issues/235.
|
in https://git.eeqj.de/sneak/upaas/issues/235.
|
||||||
- **Environment variables:**
|
- **Environment variables:**
|
||||||
- `PIXA_SIGNING_KEY` (required): secret for signed and encrypted URLs
|
- `PIXA_SIGNING_KEY` (required): secret for signed and encrypted URLs and
|
||||||
and login, 32+ characters, for example from
|
login, 32+ characters, for example from `openssl rand -base64 32`
|
||||||
`openssl rand -base64 32`
|
|
||||||
- `PIXA_ALLOWLIST_HOSTS`: upstream hosts served without a signature,
|
- `PIXA_ALLOWLIST_HOSTS`: upstream hosts served without a signature,
|
||||||
comma-separated
|
comma-separated
|
||||||
- `PIXA_CACHE_MAX_BYTES`: disk cache limit in bytes; `0` disables it;
|
- `PIXA_CACHE_MAX_BYTES`: disk cache limit in bytes; `0` disables it;
|
||||||
default 75% of free space
|
default 75% of (free space + what the cache holds)
|
||||||
- the rest are in the table under Configuration below
|
- the rest are in the table under Configuration below
|
||||||
- **Health check:** the image's `HEALTHCHECK` requests
|
- **Health check:** the image's `HEALTHCHECK` requests
|
||||||
`/.well-known/healthcheck.json`. upaas reads the container's health 60
|
`/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
|
||||||
seconds after a deploy and marks the deploy failed unless it is
|
after a deploy and marks the deploy failed unless it is `healthy`. The probe
|
||||||
`healthy`. The probe uses the port from `PORT` (default `8080`), so a
|
uses the port from `PORT` (default `8080`), so a port changed only in a
|
||||||
port changed only in a mounted config file is not seen by it: change
|
mounted config file is not seen by it: change the port with `PORT`.
|
||||||
the port with `PORT`.
|
|
||||||
|
|
||||||
## Rationale
|
## Rationale
|
||||||
|
|
||||||
Image-heavy web applications need a fast, caching reverse proxy that
|
Image-heavy web applications need a fast, caching reverse proxy that can resize
|
||||||
can resize and transcode images on the fly. pixa fills that role as a
|
and transcode images on the fly. pixa fills that role as a single,
|
||||||
single, self-contained binary with no external runtime dependencies
|
self-contained binary with no external runtime dependencies beyond libvips. It
|
||||||
beyond libvips. It supports HMAC-SHA256 signed URLs with expiration to
|
supports HMAC-SHA256 signed URLs with expiration to prevent abuse, and
|
||||||
prevent abuse, and allowlisted source hosts for open access.
|
allowlisted source hosts for open access.
|
||||||
|
|
||||||
## Design
|
## Design
|
||||||
|
|
||||||
@@ -73,8 +134,8 @@ prevent abuse, and allowlisted source hosts for open access.
|
|||||||
- **Source content**:
|
- **Source content**:
|
||||||
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of source content>`
|
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of source content>`
|
||||||
- **Source metadata**:
|
- **Source metadata**:
|
||||||
`<state_dir>/cache/metadata/<hostname>/<sha256 of path and query>.json`
|
`<state_dir>/cache/metadata/<hostname>/<sha256 of path and query>.json` (host,
|
||||||
(host, path and query, content hash, upstream status and headers, fetch time)
|
path and query, content hash, upstream status and headers, fetch time)
|
||||||
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
|
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
|
||||||
- **Transformed images**:
|
- **Transformed images**:
|
||||||
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`,
|
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`,
|
||||||
@@ -83,12 +144,13 @@ prevent abuse, and allowlisted source hosts for open access.
|
|||||||
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
|
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
|
||||||
name.
|
name.
|
||||||
|
|
||||||
Multiple source paths may reference the same content blob; the
|
Multiple source paths may reference the same content blob; the database tracks
|
||||||
database tracks references rather than using filesystem refcounting.
|
references rather than using filesystem refcounting.
|
||||||
Toward a target of 1-5k r/s, pixa keeps in memory the content types of
|
|
||||||
the 10,000 transformed images most recently cached or served, so a
|
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
|
||||||
cache hit on one of them reads only the image file from disk and not
|
Test). Toward it, pixa keeps in memory the content types of the 10,000
|
||||||
the metadata file stored beside it.
|
transformed images most recently cached or served, so a cache hit on one of them
|
||||||
|
reads only the image file from disk and not the metadata file stored beside it.
|
||||||
|
|
||||||
### Routes
|
### Routes
|
||||||
|
|
||||||
@@ -98,8 +160,8 @@ answers any method as it answers `GET`. A browser's CORS preflight request
|
|||||||
(`OPTIONS` with `Origin` and `Access-Control-Request-Method` headers) to any
|
(`OPTIONS` with `Origin` and `Access-Control-Request-Method` headers) to any
|
||||||
path under `/v1/` answers 200, in maintenance mode too.
|
path under `/v1/` answers 200, in maintenance mode too.
|
||||||
|
|
||||||
- `GET /` — the login page, or the URL generator page with a login session
|
- `GET /` — the login page, or the URL generator page with a login session (see
|
||||||
(see Encrypted URLs). Needs: nothing. Answers: 200.
|
Encrypted URLs). Needs: nothing. Answers: 200.
|
||||||
- `POST /` — log in with the signing key typed into the login page. Needs: the
|
- `POST /` — log in with the signing key typed into the login page. Needs: the
|
||||||
login page's form (below). Answers: 303 to `/` with a login session cookie
|
login page's form (below). Answers: 303 to `/` with a login session cookie
|
||||||
that lasts 30 days for the right key; 200 with the login page and an error for
|
that lasts 30 days for the right key; 200 with the login page and an error for
|
||||||
@@ -113,27 +175,32 @@ path under `/v1/` answers 200, in maintenance mode too.
|
|||||||
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>` — an image, fetched,
|
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>` — an image, fetched,
|
||||||
resized and converted (below). Needs: a signature, unless the host is
|
resized and converted (below). Needs: a signature, unless the host is
|
||||||
allowlisted (see Source Hosts). Answers: 200; 304 when `If-None-Match` matches
|
allowlisted (see Source Hosts). Answers: 200; 304 when `If-None-Match` matches
|
||||||
the image's `ETag`; 400 for a URL or parameter that is not valid; 401 for a
|
the image's `ETag`; 400 for a URL or parameter that is not valid, or for the
|
||||||
missing or wrong signature, a missing `exp` or an `exp` in the past; 403 when
|
format `auto` an `Accept` header that is not valid; 406 for the format `auto`
|
||||||
the upstream host, or a host it redirects to, is `localhost`, ends in
|
when `Accept` allows none of the formats it chooses from; 401 for a missing or
|
||||||
`.localhost` or `.local`, or has an address in a blocked network (see
|
wrong signature, a missing `exp` or an `exp` in the past; 403 when the
|
||||||
`blocked_networks`); 502 when the upstream answered with an error status, and
|
request's `Referer` names a host in `referer_blocklist`, checked before the
|
||||||
for 5 minutes after that for the same source URL; 503 when pixa is busy or in
|
signature, the cache and the upstream fetch; 403 when the upstream host, or a
|
||||||
maintenance mode; 500 for any other failure.
|
host it redirects to, is `localhost`, ends in `.localhost` or `.local`, or has
|
||||||
|
an address in a blocked network (see `blocked_networks`); 502 when the
|
||||||
|
upstream answered with an error status, and for 5 minutes after that for the
|
||||||
|
same source URL; 503 when pixa is busy or in maintenance mode; 500 for any
|
||||||
|
other failure.
|
||||||
- `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL
|
- `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL
|
||||||
(see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when
|
(see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when
|
||||||
`If-None-Match` matches the image's `ETag`; 400 for a token that does not
|
`If-None-Match` matches the image's `ETag`; 400 for a token that does not
|
||||||
decrypt, or that asks for a size or fit that is not valid; 410 once it has
|
decrypt, or that asks for a size or fit that is not valid; 410 once it has
|
||||||
expired; 504 when the upstream has not sent its response headers within
|
expired; 504 when the upstream has not sent its response headers within
|
||||||
`upstream_fetch_timeout`, but 500 when that time runs out while the image
|
`upstream_fetch_timeout`, but 500 when that time runs out while the image
|
||||||
itself is still arriving; 403, 502, 503 and 500 as for `/v1/image/`.
|
itself is still arriving; 400 for an `Accept` header that is not valid, and
|
||||||
|
406, 403, 502, 503 and 500, as for `/v1/image/`.
|
||||||
- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs:
|
- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs:
|
||||||
nothing. Answers: 200.
|
nothing. Answers: 200.
|
||||||
- `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`,
|
- `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`,
|
||||||
`uptime_seconds`, `uptime_human`, `version`, `appname` and
|
`uptime_seconds`, `uptime_human`, `version`, `appname` and `maintenance_mode`.
|
||||||
`maintenance_mode`. Needs: nothing. Answers: 200, always.
|
Needs: nothing. Answers: 200, always.
|
||||||
- `GET /static/<file>` — the script the login and generator pages load. Needs:
|
- `GET /static/<file>` — the stylesheet and script the login and generator pages
|
||||||
nothing. Answers: 200, or 404 for a file that does not exist.
|
load. Needs: nothing. Answers: 200, or 404 for a file that does not exist.
|
||||||
- `GET /metrics` — Prometheus metrics (see Architecture). Needs: HTTP basic
|
- `GET /metrics` — Prometheus metrics (see Architecture). Needs: HTTP basic
|
||||||
authentication with `metrics.username` and `metrics.password`. Answers: 200;
|
authentication with `metrics.username` and `metrics.password`. Answers: 200;
|
||||||
401 without them; 404 when they are not set, as the route then does not exist.
|
401 without them; 404 when they are not set, as the route then does not exist.
|
||||||
@@ -157,9 +224,9 @@ HTTP is for development on the browser's own machine: the login session cookie
|
|||||||
is always marked `Secure`, and over plain HTTP a browser keeps such a cookie
|
is always marked `Secure`, and over plain HTTP a browser keeps such a cookie
|
||||||
only for its own machine (`localhost`), if at all. A form is also refused with
|
only for its own machine (`localhost`), if at all. A form is also refused with
|
||||||
403 when the page's host is not the `Host` header pixa receives, so a reverse
|
403 when the page's host is not the `Host` header pixa receives, so a reverse
|
||||||
proxy in front of pixa must pass that header on unchanged. A form body over
|
proxy in front of pixa must pass that header on unchanged. A form body over 1
|
||||||
1 MiB is refused with 413. The image routes answer the errors listed for them
|
MiB is refused with 413. The image routes answer the errors listed for them with
|
||||||
with JSON holding `error`, `status` and `timestamp`.
|
JSON holding `error`, `status` and `timestamp`.
|
||||||
|
|
||||||
An image URL has this form:
|
An image URL has this form:
|
||||||
|
|
||||||
@@ -171,48 +238,64 @@ Images are only fetched from origins using TLS with valid certificates, unless
|
|||||||
`allow_http` is set: then pixa fetches every image over plain HTTP, which is for
|
`allow_http` is set: then pixa fetches every image over plain HTTP, which is for
|
||||||
testing only.
|
testing only.
|
||||||
|
|
||||||
A request whose query string cannot be decoded, or gives any parameter more
|
A request whose query string cannot be decoded, or gives any parameter more than
|
||||||
than once, is refused with 400.
|
once, is refused with 400.
|
||||||
|
|
||||||
- `<format>`: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`,
|
- `<format>`: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`,
|
||||||
`avif`, `gif`
|
`avif`, `gif`, or `auto` (below)
|
||||||
- `<size>`: `orig` or `<width>x<height>` (e.g. `800x600`)
|
- `<size>`: `orig` or `<width>x<height>` (e.g. `800x600`)
|
||||||
- `sig` and `exp`: the signature and its expiry, needed unless the host is
|
- `sig` and `exp`: the signature and its expiry, needed unless the host is
|
||||||
allowlisted (see Signature Specification)
|
allowlisted (see Signature Specification)
|
||||||
- `q` and `fit`: the output quality and how the image is fitted to `<size>`,
|
- `q` and `fit`: the output quality and how the image is fitted to `<size>`,
|
||||||
both optional (values under Signature Specification). Both are part of what
|
both optional (values under Signature Specification). Both are part of what is
|
||||||
is cached, so each value of either is a separate cached image.
|
cached, so each value of either is a separate cached image.
|
||||||
|
|
||||||
|
With the format `auto`, pixa chooses the format for each request from its
|
||||||
|
`Accept` header, in this order:
|
||||||
|
|
||||||
|
1. AVIF, when the header names `image/avif`;
|
||||||
|
2. WebP, when it names `image/webp`;
|
||||||
|
3. JPEG, when the first of `image/jpeg`, `image/*` and `*/*` that it names
|
||||||
|
allows it, or when there is no `Accept` header or it is empty.
|
||||||
|
|
||||||
|
An entry with `q=0` refuses its format; other `q` values do not change the
|
||||||
|
order. AVIF and WebP must be named, as clients that cannot show them also send
|
||||||
|
`image/*` and `*/*`. pixa never sends a format the client refused: when the
|
||||||
|
header allows none of the three, the answer is 406, and a header that does not
|
||||||
|
parse, or has a `q` that is not a number from 0 to 1, is refused with 400. The
|
||||||
|
signature, or the token of an encrypted URL, covers `auto` itself, so one URL
|
||||||
|
serves every client. Each format chosen is cached as a separate image, and every
|
||||||
|
answer that depends on `Accept` (the image, a 304, and the 400 and 406 above)
|
||||||
|
carries `Vary: Accept`, so a shared cache keeps the formats apart too.
|
||||||
|
|
||||||
An image is served with `Cache-Control: public, max-age=<seconds>, immutable`.
|
An image is served with `Cache-Control: public, max-age=<seconds>, immutable`.
|
||||||
When the URL has an expiry (an `exp`, or the TTL of an encrypted URL),
|
When the URL has an expiry (an `exp`, or the TTL of an encrypted URL), `max-age`
|
||||||
`max-age` is the whole seconds left until then, at most one year, so no browser
|
is the whole seconds left until then, at most one year, so no browser or proxy
|
||||||
or proxy cache keeps the image after pixa would refuse the URL. A URL with no
|
cache keeps the image after pixa would refuse the URL. A URL with no expiry gets
|
||||||
expiry gets one year. `immutable` only stops a client revalidating while its
|
one year. `immutable` only stops a client revalidating while its copy is fresh.
|
||||||
copy is fresh.
|
|
||||||
|
|
||||||
When several requests for the same image, size, format, quality and fit miss
|
When several requests for the same image, size, format, quality and fit miss the
|
||||||
the cache at once, they share one upstream fetch (or one read of the cached
|
cache at once, they share one upstream fetch (or one read of the cached source)
|
||||||
source) and one transcode: the first request does the work, and the others wait
|
and one transcode: the first request does the work, and the others wait for its
|
||||||
for its image or its error, holding no upstream connection or processing slot
|
image or its error, holding no upstream connection or processing slot of their
|
||||||
of their own. A waiting request stops waiting when its own client goes away.
|
own. A waiting request stops waiting when its own client goes away. The work
|
||||||
The work goes on for the others even if the first request's client goes away,
|
goes on for the others even if the first request's client goes away, until that
|
||||||
until that request's `downstream_timeout` ends. The shared fetch sends the first
|
request's `downstream_timeout` ends. The shared fetch sends the first request's
|
||||||
request's ID upstream, and the lines logged for the fetch and the transcode
|
ID upstream, and the lines logged for the fetch and the transcode carry that ID.
|
||||||
carry that ID.
|
|
||||||
|
|
||||||
The login form (`POST /`) is limited to 5 attempts per minute per client
|
The login form (`POST /`) is limited to 5 attempts per minute per client
|
||||||
address, counting an IPv6 client by its /64; an attempt over the limit is
|
address, counting an IPv6 client by its /64; an attempt over the limit is
|
||||||
refused with 429 and a `Retry-After` header. Behind a reverse proxy the client
|
refused with 429 and a `Retry-After` header. Behind a reverse proxy the client
|
||||||
address comes from `X-Forwarded-For` only when the address pixa sees for
|
address comes from `X-Forwarded-For` only when the address pixa sees for
|
||||||
requests that come through the proxy is in `trusted_proxies`; otherwise all
|
requests that come through the proxy is in `trusted_proxies`; otherwise all
|
||||||
users behind the proxy are counted as one client. That address is not always
|
users behind the proxy are counted as one client. That address is not always the
|
||||||
the proxy's own: a proxy on the Docker host that connects to pixa over
|
proxy's own: a proxy on the Docker host that connects to pixa over `127.0.0.1`
|
||||||
`127.0.0.1` is seen as the gateway of the container's Docker network, such as
|
is seen as the gateway of the container's Docker network, such as `172.17.0.1`
|
||||||
`172.17.0.1` on the default bridge, and one that connects through another of the
|
on the default bridge, and one that connects through another of the host's
|
||||||
host's addresses is seen with that address. To be sure, read it as `remoteIP` in
|
addresses is seen with that address. To be sure, read it as `remoteIP` in pixa's
|
||||||
pixa's request log while it is not in `trusted_proxies` (see `trusted_proxies`
|
request log while it is not in `trusted_proxies` (see `trusted_proxies` under
|
||||||
under Configuration). With the default `trusted_proxies` (the RFC 1918 ranges),
|
Configuration). With the default `trusted_proxies` (the RFC 1918 ranges), a
|
||||||
a client with a private address can choose the address it is counted by through
|
client with a private address can choose the address it is counted by through
|
||||||
its own `X-Forwarded-For`, whether it connects directly or through the proxy,
|
its own `X-Forwarded-For`, whether it connects directly or through the proxy,
|
||||||
because its own address is trusted too. Setting `trusted_proxies` to only the
|
because its own address is trusted too. Setting `trusted_proxies` to only the
|
||||||
address pixa sees for requests that come through the proxy closes this.
|
address pixa sees for requests that come through the proxy closes this.
|
||||||
@@ -235,20 +318,20 @@ nor change what it asks for.
|
|||||||
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
|
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
|
||||||
it expires. `<host>` is the host the page was opened on, and the URL starts
|
it expires. `<host>` is the host the page was opened on, and the URL starts
|
||||||
with `http` instead while `debug` is on. The name after the token is ignored
|
with `http` instead while `debug` is on. The name after the token is ignored
|
||||||
and only gives the URL a file extension, `jpg` for `orig`.
|
and only gives the URL a file extension, `jpg` for `orig` and `auto`.
|
||||||
|
|
||||||
The token holds the source's host, path and query and the size, format,
|
The token holds the source's host, path and query and the size, format, quality,
|
||||||
quality, fit and expiry, encrypted with a key derived from `signing_key`. The
|
fit and expiry, encrypted with a key derived from `signing_key`. The source
|
||||||
source URL's scheme is not kept: the image is fetched like any other (see
|
URL's scheme is not kept: the image is fetched like any other (see Routes), and
|
||||||
Routes), and the blocked networks still apply.
|
the blocked networks still apply.
|
||||||
|
|
||||||
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or
|
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never.
|
||||||
never. The expiry is fixed in the token when the URL is made and cannot be
|
The expiry is fixed in the token when the URL is made and cannot be changed or
|
||||||
changed or revoked afterwards. Until then the image is served with a `max-age`
|
revoked afterwards. Until then the image is served with a `max-age` that ends at
|
||||||
that ends at the expiry (see Routes); after it the URL answers 410
|
the expiry (see Routes); after it the URL answers 410 `URL has expired`. A URL
|
||||||
`URL has expired`. A URL made to last forever stops working only when
|
made to last forever stops working only when `signing_key` changes: changing it
|
||||||
`signing_key` changes: changing it makes every encrypted URL already handed out
|
makes every encrypted URL already handed out answer 400, and ends every login
|
||||||
answer 400, and ends every login session.
|
session.
|
||||||
|
|
||||||
### Image Metadata
|
### Image Metadata
|
||||||
|
|
||||||
@@ -267,16 +350,15 @@ turned off.
|
|||||||
|
|
||||||
### Source Hosts
|
### Source Hosts
|
||||||
|
|
||||||
Source hosts may be allowlisted in the configuration. Non-allowlisted
|
Source hosts may be allowlisted in the configuration. Non-allowlisted hosts
|
||||||
hosts require an HMAC-SHA256 signature.
|
require an HMAC-SHA256 signature.
|
||||||
|
|
||||||
#### Signature Specification
|
#### Signature Specification
|
||||||
|
|
||||||
Signatures use HMAC-SHA256 and include an expiration timestamp to
|
Signatures use HMAC-SHA256 and include an expiration timestamp to prevent replay
|
||||||
prevent replay attacks. Signatures are **exact match only**: every
|
attacks. Signatures are **exact match only**: every component (host, path,
|
||||||
component (host, path, query, dimensions, format, expiration, quality,
|
query, dimensions, format, expiration, quality, fit) must match exactly what was
|
||||||
fit) must match exactly what was signed. No suffix matching, wildcard
|
signed. No suffix matching, wildcard matching, or partial matching is supported.
|
||||||
matching, or partial matching is supported.
|
|
||||||
|
|
||||||
**Signed data format** (colon-separated):
|
**Signed data format** (colon-separated):
|
||||||
|
|
||||||
@@ -292,25 +374,26 @@ Where:
|
|||||||
- `width` — requested width in pixels, `0` for original
|
- `width` — requested width in pixels, `0` for original
|
||||||
- `height` — requested height in pixels, `0` for original
|
- `height` — requested height in pixels, `0` for original
|
||||||
- `format` — output format, one of those listed under Routes, with `original`
|
- `format` — output format, one of those listed under Routes, with `original`
|
||||||
signed as `orig` and `jpg` as `jpeg`
|
signed as `orig` and `jpg` as `jpeg`; `auto` is signed as `auto`, not as the
|
||||||
- `expiration` — the URL's `exp` query parameter, the Unix timestamp when
|
format chosen for the request
|
||||||
the signature expires; a request whose `exp` is not a whole number, an
|
- `expiration` — the URL's `exp` query parameter, the Unix timestamp when the
|
||||||
empty `exp=` included, is refused with 400
|
signature expires; a request whose `exp` is not a whole number, an empty
|
||||||
- `quality` — the URL's `q` query parameter, a whole number from 1 to 100,
|
`exp=` included, is refused with 400
|
||||||
or `85` when the URL has no `q`; a request whose `q` is anything else is
|
- `quality` — the URL's `q` query parameter, a whole number from 1 to 100, or
|
||||||
refused with 400
|
`85` when the URL has no `q`; a request whose `q` is anything else is refused
|
||||||
|
with 400
|
||||||
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
|
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
|
||||||
outside), or `cover` when the URL has no `fit`; a request whose `fit` is
|
outside), or `cover` when the URL has no `fit`; a request whose `fit` is
|
||||||
anything else, an empty `fit=` included, is refused with 400
|
anything else, an empty `fit=` included, is refused with 400
|
||||||
|
|
||||||
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet
|
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of
|
||||||
of RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
|
RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
|
||||||
compares it exactly, so a signature encoded without padding, as Node's
|
compares it exactly, so a signature encoded without padding, as Node's
|
||||||
`base64url` and Go's `base64.RawURLEncoding` do, is refused with 401.
|
`base64url` and Go's `base64.RawURLEncoding` do, is refused with 401.
|
||||||
|
|
||||||
**Example:** with the signing key `example-signing-key-for-documentation`,
|
**Example:** with the signing key `example-signing-key-for-documentation`,
|
||||||
resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP with
|
resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP with expiration
|
||||||
expiration 1704067200, default quality and fit:
|
1704067200, default quality and fit:
|
||||||
|
|
||||||
1. Build input:
|
1. Build input:
|
||||||
`cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200:85:cover`
|
`cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200:85:cover`
|
||||||
@@ -331,33 +414,38 @@ and the URL is
|
|||||||
- **Suffix match**: `.example.com` — matches `cdn.example.com`,
|
- **Suffix match**: `.example.com` — matches `cdn.example.com`,
|
||||||
`images.example.com`, and `example.com`
|
`images.example.com`, and `example.com`
|
||||||
|
|
||||||
|
An IP address is matched exactly; write an IPv6 address without brackets. An
|
||||||
|
entry that is neither a host name (letters, digits, hyphens, underscores and
|
||||||
|
dots, with at most one leading dot) nor an IP address, such as one with a port
|
||||||
|
or a `*.` wildcard, aborts startup.
|
||||||
|
|
||||||
### Configuration
|
### Configuration
|
||||||
|
|
||||||
Every setting can be given as an environment variable, in a YAML config
|
Every setting can be given as an environment variable, in a YAML config file
|
||||||
file (`--config`), or both. A variable present in the environment wins over
|
(`--config`), or both. A variable present in the environment wins over the file,
|
||||||
the file, even when it is empty, and the file wins over the built-in
|
even when it is empty, and the file wins over the built-in default. The one
|
||||||
default. The one exception is a variable named in the file's `env:` section:
|
exception is a variable named in the file's `env:` section: it is set while the
|
||||||
it is set while the file loads, so it overrides both the environment the
|
file loads, so it overrides both the environment the process was started with
|
||||||
process was started with and the file's own key. A variable's value is
|
and the file's own key. A variable's value is parsed as the same text in the
|
||||||
parsed as the same text in the file would be. The three lists take
|
file would be. The three lists take comma-separated entries, with the spaces
|
||||||
comma-separated entries, with the spaces around each trimmed; an empty
|
around each trimmed; an empty variable is an empty list. A value that does not
|
||||||
variable is an empty list. A value that does not parse or is invalid aborts
|
parse or is invalid aborts startup, naming the variable. A variable whose name
|
||||||
startup, naming the variable. A variable whose name starts with `PIXA_` but
|
starts with `PIXA_` but is not in the table below, such as a misspelled one or
|
||||||
is not in the table below, such as a misspelled one or `PIXA_PORT`, aborts
|
`PIXA_PORT`, aborts startup naming it, as an unknown config key does. The one
|
||||||
startup naming it, as an unknown config key does. The one other accepted
|
other accepted name is `PIXA_CONFIG_PATH`, the config file's path (like
|
||||||
name is `PIXA_CONFIG_PATH`, the config file's path (like `--config`). The
|
`--config`). The variables set by the file's `env:` section are checked the same
|
||||||
variables set by the file's `env:` section are checked the same way.
|
way.
|
||||||
|
|
||||||
pixa reads at most one config file: the one given with `--config` (or `-c`),
|
pixa reads at most one config file: the one given with `--config` (or `-c`),
|
||||||
otherwise the one `PIXA_CONFIG_PATH` names, otherwise the first of these that
|
otherwise the one `PIXA_CONFIG_PATH` names, otherwise the first of these that
|
||||||
pixa finds: `/etc/pixa/config.yml`, `/etc/pixa/config.yaml`,
|
pixa finds: `/etc/pixa/config.yml`, `/etc/pixa/config.yaml`,
|
||||||
`~/.config/pixa/config.yml`, `~/.config/pixa/config.yaml`, then `config.yml`
|
`~/.config/pixa/config.yml`, `~/.config/pixa/config.yaml`, then `config.yml` and
|
||||||
and `config.yaml` in the working directory. A named file that does not exist,
|
`config.yaml` in the working directory. A named file that does not exist, cannot
|
||||||
cannot be read or does not parse aborts startup. Of the files pixa looks for on
|
be read or does not parse aborts startup. Of the files pixa looks for on its
|
||||||
its own, one it finds but cannot read or parse aborts startup; one it cannot
|
own, only one that does not exist is passed over, without a message. One that
|
||||||
find, for any reason, is passed over without a message, even when the file is
|
pixa cannot read or parse aborts startup, naming the file. So does one in a
|
||||||
there in a directory pixa may not enter. With no file, pixa uses the environment
|
directory pixa may not enter, whether or not it is there, since pixa cannot
|
||||||
and the defaults.
|
tell. With no file, pixa uses the environment and the defaults.
|
||||||
|
|
||||||
| Variable | Config key | Meaning |
|
| Variable | Config key | Meaning |
|
||||||
| ------------------------------------ | ------------------------------- | ---------------------------------------------------------------------------- |
|
| ------------------------------------ | ------------------------------- | ---------------------------------------------------------------------------- |
|
||||||
@@ -365,8 +453,9 @@ and the defaults.
|
|||||||
| `PORT` | `port` | Port to listen on; default `8080` |
|
| `PORT` | `port` | Port to listen on; default `8080` |
|
||||||
| `PIXA_STATE_DIR` | `state_dir` | Directory for the database and the disk cache; default `/var/lib/pixa` |
|
| `PIXA_STATE_DIR` | `state_dir` | Directory for the database and the disk cache; default `/var/lib/pixa` |
|
||||||
| `PIXA_DB_URL` | `db_url` | SQLite database URL; default `state.sqlite3` in the state directory |
|
| `PIXA_DB_URL` | `db_url` | SQLite database URL; default `state.sqlite3` in the state directory |
|
||||||
| `PIXA_CACHE_MAX_BYTES` | `cache_max_bytes` | Disk cache limit in bytes; `0` disables it; default 75% of free space |
|
| `PIXA_CACHE_MAX_BYTES` | `cache_max_bytes` | Disk cache limit in bytes; `0` disables it; default 75% of (free + cached) |
|
||||||
| `PIXA_ALLOWLIST_HOSTS` | `allowlist_hosts` | Upstream hosts served without a signature |
|
| `PIXA_ALLOWLIST_HOSTS` | `allowlist_hosts` | Upstream hosts served without a signature |
|
||||||
|
| `PIXA_REFERER_BLOCKLIST` | `referer_blocklist` | Hosts whose pages the image routes refuse with 403, by `Referer` |
|
||||||
| `PIXA_BLOCKED_NETWORKS` | `blocked_networks` | CIDR ranges never fetched from, on top of the built-in ones |
|
| `PIXA_BLOCKED_NETWORKS` | `blocked_networks` | CIDR ranges never fetched from, on top of the built-in ones |
|
||||||
| `PIXA_TRUSTED_PROXIES` | `trusted_proxies` | CIDR ranges of proxies whose `X-Forwarded-For` is believed; default RFC 1918 |
|
| `PIXA_TRUSTED_PROXIES` | `trusted_proxies` | CIDR ranges of proxies whose `X-Forwarded-For` is believed; default RFC 1918 |
|
||||||
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
|
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
|
||||||
@@ -389,49 +478,66 @@ Key settings in more detail:
|
|||||||
of the image routes, `/v1/image/` and `/v1/e/`, sent as the CORS
|
of the image routes, `/v1/image/` and `/v1/e/`, sent as the CORS
|
||||||
`Access-Control-Allow-Origin` header; no other route sends it. `*`, the
|
`Access-Control-Allow-Origin` header; no other route sends it. `*`, the
|
||||||
default, is any site; otherwise one `http` or `https` origin such as
|
default, is any site; otherwise one `http` or `https` origin such as
|
||||||
`https://example.com`, whose host is a lowercase host name (letters,
|
`https://example.com`, whose host is a lowercase host name (letters, digits,
|
||||||
digits, hyphens and dots, with a letter in its last part) or an IP address
|
hyphens and dots, with a letter in its last part) or an IP address (IPv6 in
|
||||||
(IPv6 in brackets, in its shortest form), with an optional port 1-65535
|
brackets, in its shortest form), with an optional port 1-65535 that has no
|
||||||
that has no leading zero and is not the scheme's default. Any other value,
|
leading zero and is not the scheme's default. Any other value, including
|
||||||
including another scheme such as a browser extension's, aborts startup
|
another scheme such as a browser extension's, aborts startup
|
||||||
- `allowlist_hosts` — list of allowed upstream hosts
|
- `allowlist_hosts` — list of allowed upstream hosts
|
||||||
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection,
|
- `referer_blocklist` — list of hosts whose pages may not show pixa's images, to
|
||||||
added to the always-enforced built-in ranges (loopback, private,
|
stop other sites hotlinking them. Entries are written and matched as for
|
||||||
link-local, CGNAT, benchmark, NAT64, and the like); an invalid CIDR
|
`allowlist_hosts` (see Allowlist patterns), and an entry that is neither a
|
||||||
aborts startup
|
host name nor an IP address aborts startup. A request to `/v1/image/` or
|
||||||
- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front
|
`/v1/e/` whose `Referer` header names a listed host is refused with 403 before
|
||||||
of pixa. `X-Forwarded-For` is believed only when the direct peer falls
|
its signature or token is checked and before the cache or the upstream host is
|
||||||
inside one of these ranges; the logged and login-recorded client
|
used, so it fetches nothing, and it is refused even when the image is cached.
|
||||||
address is then the rightmost forwarded entry that is not itself a
|
A request with no `Referer`, or one that does not parse as a URL with a host,
|
||||||
trusted proxy. Otherwise the direct peer address is used and the header
|
is served, as many clients send none. So this is easily got around: a site
|
||||||
is ignored, so a client connecting directly from an address outside
|
whose pages send no `Referer` (for example with
|
||||||
these ranges cannot spoof its address.
|
`Referrer-Policy: no-referrer`) is not stopped. It does not apply to the login
|
||||||
An omitted key defaults to the RFC 1918 private ranges (`10.0.0.0/8`,
|
and generator pages. Default: empty
|
||||||
`172.16.0.0/12`, `192.168.0.0/16`), since pixa is deployed behind a
|
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, added
|
||||||
proxy on a private network; an explicitly empty list (`[]`) trusts no
|
to the always-enforced built-in ranges (loopback, private, link-local, CGNAT,
|
||||||
one, and an explicit list replaces the default. An invalid CIDR aborts
|
benchmark, NAT64, and the like); an invalid CIDR aborts startup
|
||||||
startup. Set this to the address pixa sees for requests that come through
|
- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front of
|
||||||
your proxy, such as `172.17.0.1/32`, when the defaults do not cover it, or
|
pixa. `X-Forwarded-For` is believed only when the direct peer falls inside one
|
||||||
to trust nothing else (see the login limit under Routes). For a proxy on
|
of these ranges; the logged and login-recorded client address is then the
|
||||||
the Docker host that connects to pixa over `127.0.0.1`, that address is the
|
rightmost forwarded entry that is not itself a trusted proxy. Otherwise the
|
||||||
gateway of the container's Docker network (`172.17.0.1` on the default
|
direct peer address is used and the header is ignored, so a client connecting
|
||||||
bridge), not the proxy's own address; a proxy that connects through another of
|
directly from an address outside these ranges cannot spoof its address. An
|
||||||
the host's addresses is seen with that address. To be sure which address it
|
omitted key defaults to the RFC 1918 private ranges (`10.0.0.0/8`,
|
||||||
is, set this to `[]` (or `PIXA_TRUSTED_PROXIES` to empty), send a request
|
`172.16.0.0/12`, `192.168.0.0/16`), since pixa is deployed behind a proxy on a
|
||||||
through the proxy, and read `remoteIP` in pixa's request log line for it
|
private network; an explicitly empty list (`[]`) trusts no one, and an
|
||||||
- `upstream_fetch_timeout` — time allowed for one fetch from an upstream
|
explicit list replaces the default. An invalid CIDR aborts startup. Set this
|
||||||
host, as a duration such as `30s` (the default) or `2m`
|
to the address pixa sees for requests that come through your proxy, such as
|
||||||
- `upstream_max_response_size` — largest upstream response accepted, in
|
`172.17.0.1/32`, when the defaults do not cover it, or to trust nothing else
|
||||||
bytes; default `52428800` (50 MiB). It also limits the image data pixa
|
(see the login limit under Routes). For a proxy on the Docker host that
|
||||||
decodes
|
connects to pixa over `127.0.0.1`, that address is the gateway of the
|
||||||
|
container's Docker network (`172.17.0.1` on the default bridge), not the
|
||||||
|
proxy's own address; a proxy that connects through another of the host's
|
||||||
|
addresses is seen with that address. To be sure which address it is, set this
|
||||||
|
to `[]` (or `PIXA_TRUSTED_PROXIES` to empty), send a request through the
|
||||||
|
proxy, and read `remoteIP` in pixa's request log line for it
|
||||||
|
- `upstream_fetch_timeout` — time allowed for one fetch from an upstream host,
|
||||||
|
as a duration such as `30s` (the default) or `2m`
|
||||||
|
- `upstream_max_response_size` — largest upstream response accepted, in bytes;
|
||||||
|
default `52428800` (50 MiB). It also limits the image data pixa decodes
|
||||||
- `downstream_timeout` — time allowed for answering one client request, as a
|
- `downstream_timeout` — time allowed for answering one client request, as a
|
||||||
duration; default `60s`. The upstream fetch counts toward it, and so do the
|
duration; default `60s`. The upstream fetch counts toward it, and so do the
|
||||||
waits for an upstream connection and for a processing slot (up to 10 seconds
|
waits for an upstream connection and for a processing slot (up to 10 seconds
|
||||||
each), so keep it longer than `upstream_fetch_timeout` plus 20 seconds
|
each), so keep it longer than `upstream_fetch_timeout` plus 20 seconds
|
||||||
- `signing_key` — HMAC secret for URL signatures
|
- `signing_key` — HMAC secret for URL signatures
|
||||||
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the
|
- `db_url` — the SQLite database to open; omitted, it is
|
||||||
disk cache entirely; omitted defaults to 75% of the free space on
|
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
|
||||||
the filesystem containing `<state_dir>/cache/` (minimum 500 MiB)
|
database in WAL mode. pixa adds `_pragma=busy_timeout(5000)` to any `db_url`,
|
||||||
|
so a write that finds another in progress waits up to five seconds for it
|
||||||
|
instead of failing. WAL mode comes only from the URL: keep
|
||||||
|
`_pragma=journal_mode(WAL)` in one you set
|
||||||
|
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
|
||||||
|
cache entirely; omitted defaults to 75% of the sum of the free space on the
|
||||||
|
filesystem containing `<state_dir>/cache/` and the bytes of source and
|
||||||
|
transformed images the cache already holds, worked out at startup (minimum 500
|
||||||
|
MiB)
|
||||||
- `upstream_connections` — the most connections to upstream hosts at once, all
|
- `upstream_connections` — the most connections to upstream hosts at once, all
|
||||||
hosts together, on top of `upstream_connections_per_host`; default `64`. A
|
hosts together, on top of `upstream_connections_per_host`; default `64`. A
|
||||||
fetch holds its connection until its image has been processed. A fetch that
|
fetch holds its connection until its image has been processed. A fetch that
|
||||||
@@ -446,12 +552,12 @@ Key settings in more detail:
|
|||||||
- `maintenance_mode` — while `true`, the image routes (`/v1/image/` and
|
- `maintenance_mode` — while `true`, the image routes (`/v1/image/` and
|
||||||
`/v1/e/`) answer every request for an image with 503, a `Retry-After` header
|
`/v1/e/`) answer every request for an image with 503, a `Retry-After` header
|
||||||
and a JSON error body. The health check (`/.well-known/healthcheck.json`)
|
and a JSON error body. The health check (`/.well-known/healthcheck.json`)
|
||||||
still answers 200 and reports `"maintenance_mode": true`. It stays 200
|
still answers 200 and reports `"maintenance_mode": true`. It stays 200 because
|
||||||
because the image's Docker `HEALTHCHECK` requests it: a 503 there would make
|
the image's Docker `HEALTHCHECK` requests it: a 503 there would make the
|
||||||
the container unhealthy, and upaas marks a deploy failed when its container
|
container unhealthy, and upaas marks a deploy failed when its container is
|
||||||
is unhealthy. The login and URL generator pages and `/metrics` keep working
|
unhealthy. The login and URL generator pages and `/metrics` keep working
|
||||||
|
|
||||||
See `config.example.yml` for all options with defaults.
|
See `configs/config.example.yml` for all options with defaults.
|
||||||
|
|
||||||
### Architecture
|
### Architecture
|
||||||
|
|
||||||
@@ -471,28 +577,98 @@ See `config.example.yml` for all options with defaults.
|
|||||||
This repository adheres to the
|
This repository adheres to the
|
||||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||||
standard: normalized scripts in `script/` are the entrypoints for the
|
standard: normalized scripts in `script/` are the entrypoints for the
|
||||||
development workflow, and the Makefile targets are thin shims that call
|
development workflow, and the Makefile targets are thin shims that call them. We
|
||||||
them. We provide:
|
provide:
|
||||||
|
|
||||||
- `script/bootstrap` — install all dependencies (idempotent)
|
- `script/bootstrap` — install git, make, Go, Node, Yarn and prettier and
|
||||||
- `script/setup` — make a fresh clone ready for development
|
download the Go modules (idempotent); with `--cgo`, the C compiler and the
|
||||||
(bootstrap, then install-precommit)
|
libvips and libheif libraries that compiling pixa needs instead of Node, Yarn
|
||||||
|
and prettier
|
||||||
|
- `script/setup` — make a fresh clone ready for development (bootstrap, then
|
||||||
|
install-precommit)
|
||||||
- `script/projectname` — output the project name ("pixa")
|
- `script/projectname` — output the project name ("pixa")
|
||||||
- `script/test` — run the test suite
|
- `script/test` — run the test suite: build the `test` phase of the
|
||||||
- `script/lint` — run golangci-lint, always in a container (builds
|
`Dockerfile`, tagged `pixa-test`
|
||||||
`Dockerfile.lint` when run outside one)
|
- `script/lint` — run golangci-lint: build the `lint` phase of the `Dockerfile`,
|
||||||
- `script/fmt` — format all code (writes)
|
tagged `pixa-lint`; the linter never runs on the host
|
||||||
- `script/fmt-check` — check formatting (read-only)
|
- `script/fmt` — format the Go code with gofmt and the markdown with prettier
|
||||||
|
(writes)
|
||||||
|
- `script/fmt-check` — check the same formatting (read-only), on the host
|
||||||
- `script/check` — run test, lint, and fmt-check
|
- `script/check` — run test, lint, and fmt-check
|
||||||
- `script/docker` — build the Docker image tagged via `script/projectname`
|
- `script/docker` — build the Docker image tagged via `script/projectname`, with
|
||||||
|
the version from `git describe`; the image's build stage depends on the `lint`
|
||||||
|
and `test` phases, so this runs them too
|
||||||
- `script/docker-smoke` — build the image, start it, wait for it to be healthy
|
- `script/docker-smoke` — build the image, start it, wait for it to be healthy
|
||||||
- `script/cibuild` — CI entrypoint: `docker build .` with a new
|
- `script/loadtest` — measure pixad's throughput, latency and peak memory; a
|
||||||
`CHECK_EPOCH` on every run, so the Dockerfile's checks run instead of
|
benchmark, not part of `script/check` (see Load Test)
|
||||||
coming from the build cache, and a green run implies a green repo
|
- `script/cibuild` — CI entrypoint: run `script/bootstrap` (without `--cgo`),
|
||||||
|
then `script/check`, then build the image as `script/docker` does
|
||||||
- `script/precommit` — pre-commit checks (`go mod tidy` guard, then
|
- `script/precommit` — pre-commit checks (`go mod tidy` guard, then
|
||||||
`script/check`)
|
`script/check`)
|
||||||
- `script/install-precommit` — install the git pre-commit hook that
|
- `script/install-precommit` — install the git pre-commit hook that runs
|
||||||
runs `script/precommit`
|
`script/precommit`
|
||||||
|
|
||||||
|
Every `docker build` in these scripts passes `--no-cache`, so the lint and test
|
||||||
|
phases run on every build instead of coming from the build cache.
|
||||||
|
`script/check`, `script/cibuild`, `script/docker`, `script/lint`, `script/test`,
|
||||||
|
`script/setup` and `script/install-precommit` are the standard copies from
|
||||||
|
`sneak/prompts`, kept identical to them. `script/fmt` and `script/fmt-check` are
|
||||||
|
the standard copies with pixa's `gofmt` step kept before prettier. prettier
|
||||||
|
formats the markdown only: not the HTML templates, as it cannot parse a Go
|
||||||
|
template action inside a tag, and not `REPO_POLICIES.md` (see
|
||||||
|
`.prettierignore`), a copy of the one in `sneak/prompts`.
|
||||||
|
|
||||||
|
## Load Test
|
||||||
|
|
||||||
|
`script/loadtest` (or `make loadtest`) measures how fast pixad answers and how
|
||||||
|
much memory it uses. It is a benchmark, not a check: `script/check` does not run
|
||||||
|
it. It needs Docker and Go.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
script/loadtest # 10 seconds per scenario, 4 clients
|
||||||
|
script/loadtest 30s 32 # 30 seconds per scenario, 32 clients
|
||||||
|
```
|
||||||
|
|
||||||
|
It builds the image with `script/docker` and the load tool,
|
||||||
|
[vegeta](https://github.com/tsenart/vegeta), from a pinned commit. Each scenario
|
||||||
|
starts a new pixad container and a new origin container, `cmd/loadtest-origin`:
|
||||||
|
an upstream host that answers every path with the same generated 1600x1200 JPEG.
|
||||||
|
vegeta then sends requests from the given number of clients, each sending its
|
||||||
|
next request as soon as its last one is answered, all for an image resized to
|
||||||
|
400x300 WebP:
|
||||||
|
|
||||||
|
- `hit`: the same image every time, put in the cache first;
|
||||||
|
- `miss`: a new source image every time, so pixad fetches and converts each one;
|
||||||
|
- `herd`: each new source image once per client in a row, so that all clients
|
||||||
|
ask for it at the same time and share one fetch and one conversion (see
|
||||||
|
Routes).
|
||||||
|
|
||||||
|
pixad refuses upstream hosts with private or local addresses, so the containers
|
||||||
|
share a Docker network in `203.0.113.0/24`, a range set aside for documentation.
|
||||||
|
A second run on the same Docker host while one is going fails, as it cannot
|
||||||
|
create that network.
|
||||||
|
|
||||||
|
For each scenario the script prints vegeta's report and two lines of its own:
|
||||||
|
|
||||||
|
- `Requests [total, rate, throughput]`: the requests sent, how many were sent
|
||||||
|
per second, and how many were answered successfully per second; the last is
|
||||||
|
the number to compare with the target under Storage;
|
||||||
|
- `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request
|
||||||
|
to the end of its answer; `50`, `95` and `99` are the 50th, 95th and 99th
|
||||||
|
percentiles;
|
||||||
|
- `Status Codes` and `Error Set`: anything other than `200` means the other
|
||||||
|
numbers are not for the scenario described, such as `503` when pixad was busy;
|
||||||
|
- `Bytes In`: `0`, as vegeta is told not to keep the images it receives;
|
||||||
|
- `pixad peak memory (VmHWM)`: the peak resident memory of pixad's process since
|
||||||
|
its container started, in kB; for `hit` it includes the request that put the
|
||||||
|
image in the cache;
|
||||||
|
- `requests to the origin`: the fetches pixad made: one for `hit`, one per
|
||||||
|
request for `miss`, and one per image for `herd`, that is the requests sent
|
||||||
|
divided by the number of clients.
|
||||||
|
|
||||||
|
The numbers depend on the machine and on whatever else runs on it. The first
|
||||||
|
measurement, made on a shared machine with few clients, is in `TODO.md`; it says
|
||||||
|
nothing about the target.
|
||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
|
|||||||
+270
-75
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Repository Policies
|
title: Repository Policies
|
||||||
last_modified: 2026-07-06
|
last_modified: 2026-09-08
|
||||||
---
|
---
|
||||||
|
|
||||||
This document covers repository structure, tooling, and workflow standards. Code
|
This document covers repository structure, tooling, and workflow standards. Code
|
||||||
@@ -60,17 +60,28 @@ style conventions are in separate documents:
|
|||||||
prerequisite since nvm requires bash. yarn is then pinned via
|
prerequisite since nvm requires bash. yarn is then pinned via
|
||||||
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
|
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
|
||||||
always exact versions. `script/cibuild` runs the CI build: it changes to the
|
always exact versions. `script/cibuild` runs the CI build: it changes to the
|
||||||
repo root and runs `docker build .`; the Gitea workflow calls it. Four further
|
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
|
||||||
scripts are our own extensions to the standard: `script/check` runs
|
with the version; the Gitea workflow calls it. **`script/cibuild` runs
|
||||||
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
|
`script/bootstrap` first**, because the workflow checks out the repo and runs
|
||||||
what the git pre-commit hook runs, and it calls `script/check`;
|
nothing else, while `script/fmt-check` runs the formatter on the host: on a
|
||||||
`script/install-precommit` installs the git pre-commit hook (the `make hooks`
|
pristine checkout with nothing installed the run dies there, after the
|
||||||
target shims to it); and `script/projectname` (literally that filename) simply
|
containerised gates have passed. **The bootstrap alone is not enough**:
|
||||||
outputs the project's name. Scripts that need the name call
|
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
|
||||||
`script/projectname` — e.g. `script/docker` assembles its image tag from it —
|
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
|
||||||
so those scripts stay byte-identical across all repos. Repo-type-specific
|
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
|
||||||
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
|
source nvm for the pinned node version before invoking it, exactly as
|
||||||
`script/precommit`, not in the hook itself. Model scripts are at
|
`script/bootstrap`'s own install step does. A runner carrying nothing but
|
||||||
|
docker and git then gets through `script/check`. Four further scripts are our
|
||||||
|
own extensions to the standard: `script/check` runs `script/test`,
|
||||||
|
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
|
||||||
|
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
|
||||||
|
installs the git pre-commit hook (the `make hooks` target shims to it); and
|
||||||
|
`script/projectname` (literally that filename) simply outputs the project's
|
||||||
|
name. Scripts that need the name call `script/projectname` — e.g.
|
||||||
|
`script/docker` assembles its image tag from it — so those scripts stay
|
||||||
|
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
|
||||||
|
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
|
||||||
|
the hook itself. Model scripts are at
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
||||||
must document the provided scripts in an **Entrypoints** section (see the
|
must document the provided scripts in an **Entrypoints** section (see the
|
||||||
README requirements below).
|
README requirements below).
|
||||||
@@ -89,87 +100,140 @@ style conventions are in separate documents:
|
|||||||
contributor should be able to understand the entire development workflow by
|
contributor should be able to understand the entire development workflow by
|
||||||
reading the Makefile.
|
reading the Makefile.
|
||||||
|
|
||||||
- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check`
|
- Every repo should have a `Dockerfile`, and it carries the repo's gates: a
|
||||||
as a build step so the build fails if the branch is not green. For non-server
|
`lint` phase and a `test` phase, with the final stage depending on both so the
|
||||||
repos, the Dockerfile should bring up a development environment and run
|
image cannot be built unless they pass. For non-server repos the final stage
|
||||||
`make check`. For server repos, `make check` should run as an early build
|
brings up a development environment; for server repos it is the runtime image.
|
||||||
stage before the final image is assembled. Dockerfiles install development
|
Dockerfiles install development prerequisites by running `script/bootstrap`
|
||||||
prerequisites by running `script/bootstrap` rather than duplicating installs
|
rather than duplicating installs inline; COPY `script/` and the dependency
|
||||||
inline; COPY `script/` and the dependency manifests (`package.json` +
|
manifests (`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before
|
||||||
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
|
running it.
|
||||||
layer stays cached until dependencies change.
|
|
||||||
|
|
||||||
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
|
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
|
||||||
repos use a multistage build where linting runs in an independent stage based
|
no separate lint file. `script/lint` and `script/test` each build one phase
|
||||||
on the `golangci/golangci-lint` image (pinned by hash). This stage runs
|
and nothing else:
|
||||||
`make fmt-check` and `make lint` before the full build begins. The build stage
|
|
||||||
then declares an explicit dependency on the lint stage via
|
|
||||||
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
|
|
||||||
linting before proceeding to compilation and tests. This ensures lint failures
|
|
||||||
surface in seconds rather than minutes, without blocking on dependency
|
|
||||||
download or compilation in the build stage.
|
|
||||||
|
|
||||||
The standard pattern for a Go repo Dockerfile is:
|
```sh
|
||||||
|
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
|
||||||
|
docker build --no-cache --target test -t "$(script/projectname)-test" .
|
||||||
|
```
|
||||||
|
|
||||||
|
**A stage that is not the last one in the file is built only when the final
|
||||||
|
stage's chain depends on it, or when `--target` names it.** That is why the
|
||||||
|
two gates are always invoked by name here, and why the final stage carries a
|
||||||
|
`COPY --from=` of a harmless file from each of them: without that edge a
|
||||||
|
plain `docker build .` builds the last stage alone and exits 0 having linted
|
||||||
|
and tested nothing.
|
||||||
|
|
||||||
|
**Every `docker build` in `script/` is tagged**, here and in
|
||||||
|
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
|
||||||
|
image behind on every invocation, on every developer host and every CI
|
||||||
|
runner; a tagged one replaces the previous image.
|
||||||
|
|
||||||
|
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
|
||||||
|
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
|
||||||
|
themselves a `docker build` and would recurse into a daemon that does not
|
||||||
|
exist in a build step. Formatting is the exception and stays on the host:
|
||||||
|
`script/fmt` writes the working tree, and `script/fmt-check` is its
|
||||||
|
read-only twin.
|
||||||
|
|
||||||
|
**No lint verdict may come from a host invocation of the linter.** On a
|
||||||
|
shared host golangci-lint reads a result cache keyed on file content rather
|
||||||
|
than location, so a second checkout of the same content is served the first
|
||||||
|
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
|
||||||
|
exit non-zero with `parallel golangci-lint is running` — a status a caller
|
||||||
|
cannot tell from real findings. Both have produced wrong verdicts in this
|
||||||
|
org, in both directions. A container has its own cache, its own `TMPDIR` and
|
||||||
|
a digest-pinned binary, so neither is reachable.
|
||||||
|
|
||||||
|
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
|
||||||
|
a `COPY` layer only when the copied content changes, so on an unchanged tree
|
||||||
|
the check `RUN` is served from cache, nothing executes, and the build still
|
||||||
|
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
|
||||||
|
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
|
||||||
|
four, and there is no fifth — `script/check` runs the two gate phases and
|
||||||
|
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
|
||||||
|
not evidence that anything ran: a sub-second build reporting success is a
|
||||||
|
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
|
||||||
|
and friends destroy a build cache shared with every other build on the host.
|
||||||
|
|
||||||
|
- **The gate phases are separate stages, and the build stage depends on both.**
|
||||||
|
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
||||||
|
hash), so lint failures surface in seconds rather than after a full compile,
|
||||||
|
and the test phase is based on the Go image. The canonical Go repo
|
||||||
|
`Dockerfile`:
|
||||||
|
|
||||||
```dockerfile
|
```dockerfile
|
||||||
# Lint stage — fast feedback on formatting and lint issues
|
# Lint phase
|
||||||
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
||||||
FROM golangci/golangci-lint@sha256:... AS lint
|
FROM golangci/golangci-lint@sha256:... AS lint
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN make fmt-check
|
RUN golangci-lint run --config .golangci.yml ./...
|
||||||
RUN make lint
|
|
||||||
|
|
||||||
# Build stage
|
# Test phase
|
||||||
# golang:1.x-alpine, YYYY-MM-DD
|
# golang:1.x-alpine, YYYY-MM-DD
|
||||||
|
FROM golang@sha256:... AS test
|
||||||
|
WORKDIR /src
|
||||||
|
COPY go.mod go.sum ./
|
||||||
|
RUN go mod download
|
||||||
|
COPY . .
|
||||||
|
RUN go test -timeout 90s -race -cover ./... || \
|
||||||
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
|
go test -timeout 90s -race -v ./...; exit 1; }
|
||||||
|
|
||||||
|
# Build stage. Nothing is wanted from either phase above; the copies
|
||||||
|
# are what make BuildKit build them first, so this stage cannot run
|
||||||
|
# unless lint and test passed.
|
||||||
|
# golang:1.x-alpine, YYYY-MM-DD
|
||||||
FROM golang@sha256:... AS builder
|
FROM golang@sha256:... AS builder
|
||||||
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
|
COPY --from=test /src/go.sum /dev/null
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
# Force BuildKit to run the lint stage before proceeding
|
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
|
||||||
|
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN make test
|
|
||||||
|
|
||||||
ARG VERSION=dev
|
ARG VERSION=dev
|
||||||
RUN CGO_ENABLED=0 go build -trimpath \
|
RUN CGO_ENABLED=0 go build -trimpath \
|
||||||
-ldflags="-s -w -X main.Version=${VERSION}" \
|
-ldflags="-s -w -X main.Version=${VERSION}" \
|
||||||
-o /app ./cmd/app/
|
-o /app ./cmd/app/
|
||||||
|
|
||||||
# Runtime stage
|
# Runtime stage, and the last one
|
||||||
FROM alpine@sha256:...
|
FROM alpine@sha256:...
|
||||||
COPY --from=builder /app /usr/local/bin/app
|
COPY --from=builder /app /usr/local/bin/app
|
||||||
ENTRYPOINT ["app"]
|
ENTRYPOINT ["app"]
|
||||||
```
|
```
|
||||||
|
|
||||||
Key points:
|
Key points:
|
||||||
- The lint stage uses the `golangci/golangci-lint` image directly (it
|
- The lint phase uses the `golangci/golangci-lint` image directly (it has
|
||||||
includes both Go and the linter), so there is no need to install the
|
both Go and the linter), so nothing needs installing.
|
||||||
linter separately.
|
- `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
|
||||||
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
|
purpose is the ordering edge. BuildKit runs stages in parallel by default,
|
||||||
a stage dependency. BuildKit runs stages in parallel by default; without
|
and a stage nothing depends on is not built at all, so without these two
|
||||||
this line, the build stage would not wait for lint to finish and a lint
|
lines a red gate would not fail the build.
|
||||||
failure might not fail the overall build.
|
- Keep the runtime stage last, and if you add a stage after it, give it the
|
||||||
|
same two copies. A plain `docker build .` builds the last stage's chain
|
||||||
|
and nothing else.
|
||||||
- If the project uses `//go:embed` directives that reference build artifacts
|
- If the project uses `//go:embed` directives that reference build artifacts
|
||||||
(e.g. a web frontend compiled in a separate stage), the lint stage must
|
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
||||||
create placeholder files so the embed directives resolve. Example:
|
create placeholder files so the embed directives resolve. Example:
|
||||||
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
|
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
|
||||||
The lint stage should not depend on the actual build output — it exists to
|
|
||||||
fail fast.
|
|
||||||
- If the project requires CGO or system libraries for linting (e.g.
|
- If the project requires CGO or system libraries for linting (e.g.
|
||||||
`vips-dev`), install them in the lint stage with `apk add`.
|
`vips-dev`), install them in the lint phase with `apk add`.
|
||||||
- The build stage runs `make test` after compilation setup. Tests run in the
|
- `ARG VERSION=dev` is declared in the stage that compiles and supplied by
|
||||||
build stage, not the lint stage, because they may require compiled
|
`script/docker` and `script/cibuild`; no stage may call `git describe`.
|
||||||
artifacts or heavier dependencies.
|
|
||||||
|
|
||||||
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
||||||
runs `script/cibuild` (which runs `docker build .`) on push. Since the
|
runs `script/cibuild` on push, and checks out the repo as its only other step.
|
||||||
Dockerfile already runs `make check`, a successful build implies all checks
|
That script bootstraps, runs the gate phases, and then builds the image, so a
|
||||||
pass.
|
successful run means every check passed; a bare `docker build .` does not
|
||||||
|
carry the same guarantee, because its gate phases may come from the cache. The
|
||||||
|
image build is uncached and so runs the gate phases a second time. That is the
|
||||||
|
price of the rule above, and it is worth paying: the image that ships is built
|
||||||
|
from a run of its own gates rather than from a cache entry.
|
||||||
|
|
||||||
- Use platform-standard formatters: `black` for Python, `prettier` for
|
- Use platform-standard formatters: `black` for Python, `prettier` for
|
||||||
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
||||||
@@ -189,14 +253,21 @@ style conventions are in separate documents:
|
|||||||
module under test to verify it compiles/parses. There is no excuse for
|
module under test to verify it compiles/parses. There is no excuse for
|
||||||
`make test` to be a no-op.
|
`make test` to be a no-op.
|
||||||
|
|
||||||
- `make test` must complete in under 20 seconds. Add a 30-second timeout in the
|
- `make test` must complete in under 60 seconds. That is the hard cap, and a
|
||||||
Makefile.
|
suite that exceeds it fails. Under 20 seconds is the target. A suite between
|
||||||
|
20 and 60 seconds is still green, but the overage must be filed as an
|
||||||
|
improvement bug against that repo. Add a 90-second timeout to the test
|
||||||
|
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
|
||||||
|
hard cap so that it catches a genuinely hung test rather than a merely slow
|
||||||
|
one.
|
||||||
|
|
||||||
- **`make test` should use the conditional verbose rerun pattern.** Run tests
|
- **The test command should use the conditional verbose rerun pattern.** Run
|
||||||
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
|
tests without `-v` (verbose) first. If tests fail, automatically rerun with
|
||||||
show full output. This keeps CI logs and `docker build` output clean on
|
`-v` to show full output. This keeps CI logs and `docker build` output clean
|
||||||
success (just package/suite summaries) while providing full diagnostic detail
|
on success (just package/suite summaries) while providing full diagnostic
|
||||||
on failure (every test case, every assertion). The general shell pattern:
|
detail on failure (every test case, every assertion). The command lives in the
|
||||||
|
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
|
||||||
|
Makefile form below is the same pattern for any repo-local invocation:
|
||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
test:
|
test:
|
||||||
@@ -209,11 +280,24 @@ style conventions are in separate documents:
|
|||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
test:
|
test:
|
||||||
@go test -timeout 30s -race -cover ./... || \
|
@go test -count=1 -timeout 90s -race -cover ./... || \
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
go test -timeout 30s -race -v ./...; exit 1; }
|
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`-count=1` is required on both invocations: it defeats Go's test _result_
|
||||||
|
cache, so the target cannot report a pass it did not earn, and the rerun
|
||||||
|
reproduces a failure instead of replaying it. It leaves the build cache
|
||||||
|
alone, so it costs the runtime of the suite and no recompilation.
|
||||||
|
|
||||||
|
Note that this is a second, independent cache, stacked below the Docker
|
||||||
|
layer cache that [issue #26](https://git.eeqj.de/sneak/prompts/issues/26)
|
||||||
|
addresses. `CHECK_EPOCH` guarantees the `RUN make test` _step_ re-executes;
|
||||||
|
it does not guarantee `go test` inside that step does any work, because the
|
||||||
|
`GOCACHE` baked into earlier image layers survives into the re-executed
|
||||||
|
step. They are two separate defects requiring two separate fixes, and a fix
|
||||||
|
for one must not be recorded as covering the other.
|
||||||
|
|
||||||
Python example:
|
Python example:
|
||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
@@ -239,10 +323,83 @@ style conventions are in separate documents:
|
|||||||
must be in `.gitignore`. No exceptions.
|
must be in `.gitignore`. No exceptions.
|
||||||
|
|
||||||
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
||||||
editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`.
|
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
|
||||||
Fetch the standard `.gitignore` from
|
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
|
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
|
||||||
a new repo.
|
setting up a new repo. These patterns are written to `.gitignore`'s own
|
||||||
|
semantics, in which an unanchored pattern already matches at every depth; they
|
||||||
|
are not a `.dockerignore` and must not be transplanted into one unmodified.
|
||||||
|
|
||||||
|
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
|
||||||
|
across unmodified leaves secrets in the build context.** Docker matches with
|
||||||
|
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
|
||||||
|
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
|
||||||
|
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
|
||||||
|
therefore excludes only the copies at the repository root, while `config/.env`
|
||||||
|
and `certs/server.key` still reach the context and can land in an image layer
|
||||||
|
— which is more dangerous than a short file with no secret patterns at all,
|
||||||
|
because it reads as solved and stops anyone looking. Give every
|
||||||
|
depth-independent pattern the `**/` prefix and leave only genuinely
|
||||||
|
root-anchored entries unprefixed: `.git`, and the repo's own host-built
|
||||||
|
binary, written `/myapp` and never `**/myapp`, which would also match
|
||||||
|
`cmd/myapp/` and delete the package directory from the context. Matching is
|
||||||
|
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
|
||||||
|
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
|
||||||
|
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
|
||||||
|
also catches something the build needs, re-include it with a negation
|
||||||
|
(`!docs/example.env`); deleting the pattern reopens the exposure for every
|
||||||
|
other file it covers. Fetch the standard `.dockerignore` from
|
||||||
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
|
||||||
|
it with the repo's own artifacts.
|
||||||
|
|
||||||
|
- **In-repo agent scratch belongs in both files, written to each file's own
|
||||||
|
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
|
||||||
|
additional checkout of the repo — so under `COPY . .` the build context
|
||||||
|
inflates by a multiple of the repo and another session's unreviewed work can
|
||||||
|
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
|
||||||
|
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
|
||||||
|
prefix, because the prefixed form would also delete any nested directory of
|
||||||
|
that name from the build. Anchoring carries a known gap that the canonical
|
||||||
|
`.dockerignore` states in its own comment, since consuming repos receive the
|
||||||
|
file and not the tracker: the directory is created in the agent's working
|
||||||
|
directory, so a repo running agents in subdirectories still ships
|
||||||
|
`services/api/.claude/` and must add its own anchored entry there.
|
||||||
|
|
||||||
|
- **Excluding `.git` means `git describe` cannot run inside any build stage, and
|
||||||
|
it fails quietly there.** In a build stage there is no repository, so
|
||||||
|
`git describe` writes nothing to stdout, `-X main.Version=` comes out empty,
|
||||||
|
the binary reports no version at all, and the build still exits 0. Compute the
|
||||||
|
version on the host and thread it in as a build arg. `script/docker` and
|
||||||
|
`script/cibuild` do this, byte-identically across repos:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Own line: a failing command substitution inside an argument does not
|
||||||
|
# trip `set -e`, so the inline form degrades to an empty constant.
|
||||||
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
|
[ -n "$version" ] || version="unknown"
|
||||||
|
docker build --no-cache \
|
||||||
|
--build-arg VERSION="$version" \
|
||||||
|
-t "$(script/projectname)" .
|
||||||
|
```
|
||||||
|
|
||||||
|
`--always` makes an untagged repo yield an abbreviated commit hash rather
|
||||||
|
than failing, and the `[ -n "$version" ]` line is the single place the
|
||||||
|
fallback is applied — a live check that fires on a build from an export with
|
||||||
|
no `.git` and on a repository with no commits yet. Do not fold it into the
|
||||||
|
substitution as `|| echo unknown`, which makes the guard unreachable. The
|
||||||
|
Dockerfile's side is `ARG VERSION=dev` in the stage that compiles, declared
|
||||||
|
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
||||||
|
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
||||||
|
the scripts stay byte-identical. One consequence for CI: the standard
|
||||||
|
checkout action clones shallow and fetches no tags, so a repo that embeds a
|
||||||
|
tag-derived version must set `fetch-depth: 0` on its checkout step.
|
||||||
|
|
||||||
|
- **Verify `.dockerignore` by enumerating the image, not by reading the
|
||||||
|
patterns.** Plant files at the root _and_ at least two directories deep, build
|
||||||
|
a probe image that does `COPY . .`, and list what actually landed
|
||||||
|
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
|
||||||
|
size is not a substitute: a nested secret is a few bytes, and BuildKit
|
||||||
|
transfers only the delta from the previous build.
|
||||||
|
|
||||||
- **No build artifacts in version control.** Code-derived data (compiled
|
- **No build artifacts in version control.** Code-derived data (compiled
|
||||||
bundles, minified output, generated assets) must never be committed to the
|
bundles, minified output, generated assets) must never be committed to the
|
||||||
@@ -258,9 +415,45 @@ style conventions are in separate documents:
|
|||||||
- Make all changes on a feature branch. You can do whatever you want on a
|
- Make all changes on a feature branch. You can do whatever you want on a
|
||||||
feature branch.
|
feature branch.
|
||||||
|
|
||||||
- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only
|
- `.golangci.yml` is standardized. The vendored copy in a consuming repo must
|
||||||
manually by the user. Fetch from
|
_NEVER_ be modified by an agent: fetch it from
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`.
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
|
||||||
|
byte-identical, so that no repo can quietly loosen its own linting. Linter
|
||||||
|
configuration changes are made to the canonical copy in the `prompts` repo and
|
||||||
|
reach consuming repos by re-vendoring; an agent may open a PR against
|
||||||
|
canonical, which only the user merges. One list is exempt from byte-identity,
|
||||||
|
because it cannot be written once for every repo: the `deny` list of the
|
||||||
|
`test-support` depguard rule, where a repo names its own test-support packages
|
||||||
|
by full import path. A repo adds entries there and changes nothing else, and a
|
||||||
|
re-vendor carries its entries forward. The canonical golangci-lint version is
|
||||||
|
v2.12.2 (released 2026-05-06), pinned as the digest of the lint phase's base
|
||||||
|
image
|
||||||
|
(`golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240`,
|
||||||
|
which reports `2.12.2 built with go1.26.2 from c0d3ddc9`). That digest is the
|
||||||
|
only pin, since no repo installs golangci-lint on the host: bumping the
|
||||||
|
version means changing it and nothing else.
|
||||||
|
|
||||||
|
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
|
||||||
|
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
|
||||||
|
`PATH` only, so on an already-provisioned machine the pin is inert and a
|
||||||
|
version bump is a silent no-op — while the Dockerfile, installing into a clean
|
||||||
|
image, gets the pinned version, so a local `make check` and `make docker` can
|
||||||
|
disagree about what the tool even is. The canonical form:
|
||||||
|
- compares the installed version against the pin over the **whole** version
|
||||||
|
token; a parser that stops at the first `-` reports `2.12.2` for a host
|
||||||
|
running `2.12.2-rc1` and skips the install;
|
||||||
|
- treats absent, non-zero, empty or unrecognised `--version` output as a
|
||||||
|
mismatch, so the failure direction is a redundant install and never a
|
||||||
|
skipped one;
|
||||||
|
- after installing, re-resolves the binary the way callers do — `hash -r`,
|
||||||
|
then through `PATH`, not through the directory the installer wrote to —
|
||||||
|
and fails naming the resolved path, since an install that a shadowing
|
||||||
|
binary hides succeeds while changing nothing any caller sees;
|
||||||
|
- is actually called, and prints the version on both success paths: a
|
||||||
|
function defined and never invoked has the same exit status and the same
|
||||||
|
empty output as one that worked.
|
||||||
|
|
||||||
|
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
||||||
|
|
||||||
- When pinning images or packages by hash, add a comment above the reference
|
- When pinning images or packages by hash, add a comment above the reference
|
||||||
with the version and date (YYYY-MM-DD).
|
with the version and date (YYYY-MM-DD).
|
||||||
@@ -379,7 +572,9 @@ style conventions are in separate documents:
|
|||||||
language-specific config). Everything else goes in a subdirectory. Canonical
|
language-specific config). Everything else goes in a subdirectory. Canonical
|
||||||
subdirectory names:
|
subdirectory names:
|
||||||
- `bin/` — executable scripts and tools
|
- `bin/` — executable scripts and tools
|
||||||
- `cmd/` — Go command entrypoints
|
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
|
||||||
|
body is a single call into `internal/` or `pkg/`, no project logic in
|
||||||
|
`cmd/`
|
||||||
- `configs/` — configuration templates and examples
|
- `configs/` — configuration templates and examples
|
||||||
- `deploy/` — deployment manifests (k8s, compose, terraform)
|
- `deploy/` — deployment manifests (k8s, compose, terraform)
|
||||||
- `docs/` — documentation and markdown (README.md stays in root)
|
- `docs/` — documentation and markdown (README.md stays in root)
|
||||||
|
|||||||
@@ -1,34 +1,278 @@
|
|||||||
# Workflow
|
# Workflow
|
||||||
|
|
||||||
* branch per issue from `next`
|
- branch per issue from `next`
|
||||||
* do the work in Next Step
|
- do the work in Next Step
|
||||||
* move Next Step to the top of Completed Steps
|
- move Next Step to the top of Completed Steps
|
||||||
* move the top item of Future Steps into Next Step
|
- `TODO.md` merges with git's union merge (`.gitattributes`), which never
|
||||||
* commit (`TODO.md` changes in the same commit as the work)
|
reports a conflict: read the merged entries after every merge or rebase
|
||||||
* open a PR based on `next`
|
- move the top item of Future Steps into Next Step
|
||||||
* an independent reviewer who did not write the change gates it
|
- commit (`TODO.md` changes in the same commit as the work)
|
||||||
* the manager squash-merges the PR into `next` once review passes
|
- open a PR based on `next`
|
||||||
* `next` stays green and mergeable to `main` at any time; only the owner
|
- an independent reviewer who did not write the change gates it
|
||||||
merges `next` into `main`, via the single milestone PR
|
- the manager squash-merges the PR into `next` once review passes
|
||||||
* push
|
- `next` stays green and mergeable to `main` at any time; only the owner merges
|
||||||
|
`next` into `main`, via the single milestone PR
|
||||||
|
- push
|
||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work
|
pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work lands on
|
||||||
lands on `next`, and `main` receives only the milestone PR that the
|
`next`, and `main` receives only the milestone PR that the owner merges. `next`
|
||||||
owner merges. `next` is at the canonical `golangci-lint` v2.12.2 config
|
is at the canonical `golangci-lint` v2.12.2 config and is green. Recent work
|
||||||
and is green. Recent work extracted the internal/magic,
|
extracted the internal/magic, internal/allowlist, internal/httpfetcher, and
|
||||||
internal/allowlist, internal/httpfetcher, and internal/signature
|
internal/signature packages. The gosec findings from the 2026-07-06 survey are
|
||||||
packages. The gosec findings from the 2026-07-06 survey are resolved.
|
resolved. The disk cache is now size-bounded with LRU eviction
|
||||||
The disk cache is now size-bounded with LRU eviction
|
|
||||||
(`cache_max_bytes`), closing the unbounded disk growth DoS vector.
|
(`cache_max_bytes`), closing the unbounded disk growth DoS vector.
|
||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
P2: security: referer blacklist
|
P2: security: per-IP rate limiting on the image routes
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-05 the format `auto` (closes #88): a format in the `/v1/image/` path,
|
||||||
|
an encrypted URL's token and the generator page's format choice, chosen for
|
||||||
|
each request from `Accept` once the signature or token is checked: AVIF when
|
||||||
|
the header names `image/avif`, else WebP when it names `image/webp`, else JPEG
|
||||||
|
when the first of `image/jpeg`, `image/*` and `*/*` that it names allows it,
|
||||||
|
or when it names nothing; `q=0` refuses a format. AVIF and WebP must be named,
|
||||||
|
as clients that cannot show them send the wildcards too. A header that allows
|
||||||
|
none of the three answers 406, one that does not parse 400. The signature and
|
||||||
|
the token cover `auto` itself; the cache key and `ETag` use the format chosen.
|
||||||
|
Answers from the point the format is chosen carry `Vary: Accept`, next to the
|
||||||
|
CORS `Vary: Origin`; fixed-format answers do not.
|
||||||
|
- 2026-10-05 the markdown is formatted with prettier (closes #100): `script/fmt`
|
||||||
|
and `script/fmt-check` run prettier 3.8.1, pinned in `package.json` and
|
||||||
|
`yarn.lock`, on `**/*.md` after `gofmt`, with four-space tabs and
|
||||||
|
`proseWrap: always` as `.prettierrc` says; `.prettierignore` keeps it off
|
||||||
|
`REPO_POLICIES.md`, the copy from `sneak/prompts`, and `vendor/`. Plain
|
||||||
|
`script/bootstrap` installs Node and Yarn as the one in `sneak/prompts` does
|
||||||
|
and then prettier; `script/bootstrap --cgo` does not, as the `Dockerfile`
|
||||||
|
stages that run it format nothing. The HTML templates stay unformatted:
|
||||||
|
prettier cannot parse a Go template action inside a tag. The markdown was
|
||||||
|
reflowed in a commit of its own.
|
||||||
|
- 2026-10-05 lint and tests run as the `lint` and `test` phases of the
|
||||||
|
`Dockerfile`, built with `--no-cache` (closes #202): `script/check`,
|
||||||
|
`script/cibuild`, `script/docker`, `script/lint`, `script/test`,
|
||||||
|
`script/setup` and `script/install-precommit` are now the copies from
|
||||||
|
`sneak/prompts` `main`, unchanged. The `lint` phase runs golangci-lint from
|
||||||
|
the image `REPO_POLICIES.md` names, with `libvips-dev` from `apt-get`; the
|
||||||
|
`test` phase runs the tests with a 90-second timeout; the build stage depends
|
||||||
|
on both. `Dockerfile.lint` and the `CHECK_EPOCH` build argument are gone, the
|
||||||
|
formatting check runs on the host, and `make docker-versioned` and
|
||||||
|
`make docker-test` call the scripts. `script/bootstrap`, `script/fmt`,
|
||||||
|
`script/fmt-check`, `script/precommit` and `script/projectname` stay pixa's
|
||||||
|
own. `script/bootstrap` installs git, make and Go, refreshing apt's package
|
||||||
|
lists before its first apt install; with `--cgo`, which only the `test` phase
|
||||||
|
and the build stage pass, it also installs the C compiler and the libvips and
|
||||||
|
libheif libraries. The stage that compiles still takes the version from
|
||||||
|
`git describe` when no `VERSION` is given, per
|
||||||
|
https://git.eeqj.de/sneak/pixa/issues/166, so the copied scripts' comment that
|
||||||
|
`.dockerignore` leaves out `.git` does not hold for pixa.
|
||||||
|
- 2026-10-04 load test (closes #81): `script/loadtest [duration [clients]]`
|
||||||
|
(`make loadtest`, defaults `10s` and `4`), a benchmark that `script/check`
|
||||||
|
does not run, measures three scenarios, each against a new pixad container and
|
||||||
|
a new upstream host, `cmd/loadtest-origin`: `hit` (one cached image), `miss`
|
||||||
|
(a new source image every request) and `herd` (each new source image asked for
|
||||||
|
by all clients at once). For each it prints vegeta's report (requests per
|
||||||
|
second, latency percentiles, status codes), pixad's peak resident memory and
|
||||||
|
the requests that reached the origin. `README.md` says how to run it and read
|
||||||
|
it, and keeps 1-5k r/s as a target not yet measured. First measurement, with
|
||||||
|
the defaults on a shared 48-CPU machine with other work running: a baseline
|
||||||
|
for later changes, not a test of the target. `hit` 1413 r/s, p50 0.7 ms, p95
|
||||||
|
8.7 ms, p99 44 ms, peak 53 MiB (4 clients that each wait for their answer, so
|
||||||
|
not pixad's limit); `miss` 70 r/s, p50 52 ms, p95 91 ms, p99 122 ms, peak 100
|
||||||
|
MiB, one fetch per request; `herd` 74 r/s, p50 52 ms, p95 69 ms, p99 111 ms,
|
||||||
|
peak 60 MiB, 188 fetches for 749 requests.
|
||||||
|
- 2026-10-04 the CI checkout fetches the tags (closes #208): the checkout step
|
||||||
|
in `.gitea/workflows/check.yml` sets `fetch-depth: 0`, as `REPO_POLICIES.md`
|
||||||
|
asks of a repo that takes its version from the tags, so a CI build of a tagged
|
||||||
|
commit stamps the tag from `git describe` instead of a bare commit.
|
||||||
|
- 2026-10-04 `config.yml` stays out of git and the Docker build context (closes
|
||||||
|
#212): `.gitignore` now ignores `config.yml`, the config file Getting Started
|
||||||
|
creates with the signing key, and `.dockerignore` leaves it out in every
|
||||||
|
directory and in any letter case, as it already did `config.yaml` and
|
||||||
|
`config.dev.yml`.
|
||||||
|
- 2026-10-04 local config files stay out of the Docker build context (closes
|
||||||
|
#211): `.dockerignore` now leaves out `config.yaml` and `config.dev.yml` in
|
||||||
|
every directory and in any letter case, the local config files `.gitignore`
|
||||||
|
keeps out of git because they can hold the signing key.
|
||||||
|
`configs/config.example.yml` is still sent. `config.yml`, which Getting
|
||||||
|
Started creates, is in neither file:
|
||||||
|
https://git.eeqj.de/sneak/pixa/issues/212.
|
||||||
|
- 2026-10-04 `cmd/pixad/main.go` is one call into `internal/` (closes #206):
|
||||||
|
what it did (the command line and its `--config` flag, setting
|
||||||
|
`PIXA_CONFIG_PATH`, ignoring `SIGPIPE`, starting the fx app) is now `Run` in
|
||||||
|
`internal/app`, unchanged, and `main` calls it with `Version`, which the build
|
||||||
|
still sets through `-X main.Version`. That code had no tests to move.
|
||||||
|
- 2026-10-04 `.gitignore` ignores `.claude/` (closes #204): the entry and its
|
||||||
|
comment are copied from the canonical `.gitignore` in `sneak/prompts`,
|
||||||
|
unanchored so it matches at every depth. `.dockerignore` already has
|
||||||
|
`.claude`.
|
||||||
|
- 2026-10-04 `.dockerignore` keeps secrets out at every depth (closes #205): the
|
||||||
|
file is now the standard one from `sneak/prompts`, whose patterns match in
|
||||||
|
every directory and, for environment files and private keys, in any letter
|
||||||
|
case, so a nested `.env` or `server.key` no longer reaches the build context.
|
||||||
|
pixa still sends `.git` without `.git/config` in place of the standard file's
|
||||||
|
`.git` line, and still leaves out `.gitignore`, `/bin` and `/data`.
|
||||||
|
- 2026-10-04 `REPO_POLICIES.md` matches the canonical copy again (closes #196):
|
||||||
|
it is replaced, unchanged, by `prompts/REPO_POLICIES.md` from `sneak/prompts`
|
||||||
|
`main`. The rules it adds that pixa's tree breaks are filed:
|
||||||
|
https://git.eeqj.de/sneak/pixa/issues/202 (lint and tests as `Dockerfile`
|
||||||
|
phases built with `--no-cache`), https://git.eeqj.de/sneak/pixa/issues/203
|
||||||
|
(the workflow's `script/docker-smoke` step),
|
||||||
|
https://git.eeqj.de/sneak/pixa/issues/204 (`.claude/` in `.gitignore`),
|
||||||
|
https://git.eeqj.de/sneak/pixa/issues/205 (`.dockerignore` patterns at every
|
||||||
|
depth), https://git.eeqj.de/sneak/pixa/issues/206 (a thin `cmd/pixad/main.go`)
|
||||||
|
and https://git.eeqj.de/sneak/pixa/issues/208 (`fetch-depth: 0` on the CI
|
||||||
|
checkout, so the build sees the tags). Its rule that no build stage runs
|
||||||
|
`git describe` is not followed: pixa takes the version from the `.git` in the
|
||||||
|
build context, per https://git.eeqj.de/sneak/pixa/issues/166, as the copy on
|
||||||
|
`sneak/prompts` `next` already says.
|
||||||
|
- 2026-10-04 an integration test of the image proxy flow (closes #80):
|
||||||
|
`TestImageProxyFlow` in `internal/server` starts the database, handlers and
|
||||||
|
middleware from the constructors `pixad` uses, with a fresh state directory,
|
||||||
|
and replaces only the upstream origin with a local test server. For a resize
|
||||||
|
with a change to JPEG and for `orig`, the first request goes through the
|
||||||
|
router, the real fetcher, libvips, the disk cache and SQLite and answers 200
|
||||||
|
with the right content type and size and `X-Pixa-Cache: MISS`; the second
|
||||||
|
answers `HIT` with the same image and the upstream has had one request; the
|
||||||
|
source and the converted image are then in `cache/sources` and
|
||||||
|
`cache/variants`, with their rows in `source_content`, `source_metadata` and
|
||||||
|
`variant_content`. Two optional fields make this possible, which `pixad` does
|
||||||
|
not set and the config file and environment cannot:
|
||||||
|
`httpfetcher.Config.DialContext` connects in place of the dialer that refuses
|
||||||
|
internal addresses, the URL and redirect checks still running, and
|
||||||
|
`handlers.Params.Fetcher` replaces the fetcher the handlers build.
|
||||||
|
- 2026-10-04 a URL made on the generator page with a `ttl` is tested to expire
|
||||||
|
(closes #199): a new test in `internal/handlers` makes a URL on the generator
|
||||||
|
page with a `ttl` of one second, checks that `/v1/e/` serves it at once, waits
|
||||||
|
two seconds and checks that it then answers 410. The test waits for real, as
|
||||||
|
pixa reads the clock directly when it makes and checks a URL; it waits two
|
||||||
|
seconds because the time a URL expires is kept in whole seconds. Test only.
|
||||||
|
- 2026-10-04 referer blocklist (closes #90): `referer_blocklist`
|
||||||
|
(`PIXA_REFERER_BLOCKLIST`) lists hosts, written and matched as for
|
||||||
|
`allowlist_hosts` with the same matcher; an entry of either list that is
|
||||||
|
neither a host name (letters, digits, hyphens, underscores and dots, with at
|
||||||
|
most one leading dot) nor an IP address, such as one with a port or a `*.`
|
||||||
|
wildcard, aborts startup naming the setting and the entry. Both image routes
|
||||||
|
refuse a request whose `Referer` names a listed host with 403 and a JSON error
|
||||||
|
before the signature, the cache and the upstream fetch, so it fetches nothing
|
||||||
|
and is refused whether or not the image is cached. A request with no
|
||||||
|
`Referer`, or one that does not parse as a URL with a host, is served, so the
|
||||||
|
list is easily got around; `README.md` and `configs/config.example.yml` say
|
||||||
|
so. It does not apply to the login and generator pages.
|
||||||
|
- 2026-10-04 fewer files in the repository root (closes #97):
|
||||||
|
`config.example.yml` moved unchanged to `configs/config.example.yml`, and
|
||||||
|
`README.md`, the comments in `internal/config/config.go` and the startup error
|
||||||
|
for the placeholder signing key name the new path; `scripts/manual-test.sh`
|
||||||
|
and its directory are deleted, as the handler tests in `internal/handlers`
|
||||||
|
cover every check it made except two: fetching a real image from the internet,
|
||||||
|
and a URL made on the generator page with a `ttl` answering 410 once the `ttl`
|
||||||
|
has passed (https://git.eeqj.de/sneak/pixa/issues/199); `CONVENTIONS.md` is
|
||||||
|
deleted, as `REPO_POLICIES.md` links the canonical Go HTTP server conventions.
|
||||||
|
- 2026-10-04 SQLite writes no longer fail with "database is locked" (closes
|
||||||
|
#198): pixa adds `_pragma=busy_timeout(5000)` to every `db_url`, so a write
|
||||||
|
that finds another in progress on another connection waits up to five seconds
|
||||||
|
for it, and the default `db_url` turns on WAL mode with
|
||||||
|
`_pragma=journal_mode(WAL)`. The old default's `_journal_mode=WAL` is not a
|
||||||
|
parameter the driver reads, so the database was never in WAL mode.
|
||||||
|
- 2026-10-04 `TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup` only
|
||||||
|
passes through a periodic pass (closes #189): it slept for three eviction
|
||||||
|
intervals before writing its file, and a startup pass still running then could
|
||||||
|
adopt the file itself. It now holds the test database's only connection until
|
||||||
|
the startup pass waits for it after walking the empty variant directory,
|
||||||
|
writes the file and lets the connection go, as
|
||||||
|
`TestEvictionRunsOnPeriodicSchedule` does, so only a periodic reconciliation
|
||||||
|
pass can adopt the file. Test only.
|
||||||
|
- 2026-10-04 logging in, logging out, the URL generator and `/v1/e/` have
|
||||||
|
handler tests (closes #77): new tests in `internal/handlers`, with no network,
|
||||||
|
check that `GET /` without a login session shows the login form; a wrong key
|
||||||
|
shows it again with an error and sets no session cookie; the right key answers
|
||||||
|
303 to `/` with a session cookie marked `Secure`, `HttpOnly` and
|
||||||
|
`SameSite=Strict`, with which `GET /` shows the generator page; `GET /logout`
|
||||||
|
answers 303 to `/` with an empty session cookie sent with `Max-Age=0`;
|
||||||
|
`POST /generate` without a login session answers 303 to `/`; `/v1/e/` serves
|
||||||
|
the image for a valid token, answers 410 for an expired one and 400 for one
|
||||||
|
with a character changed, cut short or made with another signing key; and a
|
||||||
|
URL made on the generator page is served by `/v1/e/`. No code changes.
|
||||||
|
- 2026-10-04 `TODO.md` merges with git's union merge (closes #190): a root
|
||||||
|
`.gitattributes`, copied from `sneak/prompts`, marks it `merge=union`, so two
|
||||||
|
branches that each add an entry at the top of Completed Steps merge without a
|
||||||
|
conflict and keep both entries. Git now never reports a conflict in `TODO.md`:
|
||||||
|
a real one keeps both versions of the lines, and two entries that share an
|
||||||
|
identical line can end up one inside the other, which a rebase can do to an
|
||||||
|
entry already on `next`. The Workflow above says to read the merged entries
|
||||||
|
after every merge or rebase.
|
||||||
|
- 2026-10-04 the default `cache_max_bytes` no longer shrinks as the cache fills
|
||||||
|
(closes #184): for an omitted key, the cache works out the limit when it
|
||||||
|
opens, after the database is open, as 75% of the sum of the free space on the
|
||||||
|
filesystem containing `<state_dir>/cache/` and what the cache already holds by
|
||||||
|
its own size accounting, at least 500 MiB, so a cache filled to its limit
|
||||||
|
keeps that limit across a restart. The computation and its tests moved from
|
||||||
|
`internal/config` to `internal/imgcache`; the config only records whether the
|
||||||
|
key was set.
|
||||||
|
- 2026-10-04 `TestEvictionRunsOnPeriodicSchedule` no longer races the evictor
|
||||||
|
(closes #183): it wrote each variant file and then inserted its accounting row
|
||||||
|
by hand, and a reconciliation pass between the two adopted the file first, so
|
||||||
|
the insert failed. It now writes the files only, while holding the test
|
||||||
|
database's only connection so the evictor's startup pass waits after walking
|
||||||
|
the empty variant directory; a periodic reconciliation pass then adopts the
|
||||||
|
files and the eviction pass after it evicts them. No other test in
|
||||||
|
`internal/imgcache` inserts a row by hand after starting the evictor. Test
|
||||||
|
only.
|
||||||
|
- 2026-10-04 a config file pixa cannot read aborts startup (closes #176): of the
|
||||||
|
places pixa looks for its config file on its own, only one where the file does
|
||||||
|
not exist is passed over; any other error, such as a directory on the path
|
||||||
|
that pixa may not enter, aborts startup naming the file, as a file that does
|
||||||
|
not parse already did.
|
||||||
|
- 2026-10-04 `.golangci.yml` re-vendored from the canonical copy (closes #57):
|
||||||
|
the deprecated `gomodguard` is switched off, so lint runs print no deprecation
|
||||||
|
warning; its successor `gomodguard_v2` runs with the shared module block list,
|
||||||
|
and `depguard` keeps `net/http/httptest` out of files that are not tests. The
|
||||||
|
tree needed no code changes.
|
||||||
|
- 2026-10-04 the Content-Security-Policy allows no inline script or style
|
||||||
|
(closes #125): `script-src` and `style-src` are `'self'` only. The generator
|
||||||
|
page's two inline `onclick` handlers moved into
|
||||||
|
`internal/static/generator.js`, attached with `addEventListener`; the bundled
|
||||||
|
Tailwind script, which built styles in the browser, is replaced by a small
|
||||||
|
hand-written `internal/static/style.css` with only the rules the login and
|
||||||
|
generator pages use, the templates carrying a few plain class names in place
|
||||||
|
of Tailwind's. No build step. The pages keep their layout, not every pixel of
|
||||||
|
it.
|
||||||
|
- 2026-10-04 deployment guide and example Caddy config (closes #89):
|
||||||
|
"Deployment" in `README.md` says what the reverse proxy in front of pixa must
|
||||||
|
do (terminate TLS; pass `Host`, `Origin` and `Referer` on unchanged; set
|
||||||
|
`X-Forwarded-For`, with `trusted_proxies` to match; wait at least
|
||||||
|
`downstream_timeout`; optionally refuse `/metrics`) and what pixa does itself,
|
||||||
|
that the state directory needs a persistent volume and what `cache_max_bytes`
|
||||||
|
counts, the health check for a load balancer, what a stop does and its exit
|
||||||
|
codes, and what running outside Docker needs; `configs/Caddyfile` is the
|
||||||
|
example, checked with `caddy validate`.
|
||||||
|
- 2026-10-04 the metrics basic auth, CORS preflight, request logging and metrics
|
||||||
|
recording have tests (closes #79): `MetricsAuth` on its own answers 401 with a
|
||||||
|
challenge without credentials or with a wrong username or password and lets
|
||||||
|
the configured ones through; a preflight request gets `*` for any origin when
|
||||||
|
`access_control_allow_origin` is `*` and no `Access-Control-Allow-Origin` from
|
||||||
|
another origin than the configured one; a `POST /` carrying the signing key
|
||||||
|
leaves no trace of it in the request log line, and the login handler's own log
|
||||||
|
lines leave out the submitted key; the metrics middleware on its own records a
|
||||||
|
request it served, and the router records nothing while no metrics username is
|
||||||
|
set. Not tested: that the router puts the basic auth in front of `/metrics`
|
||||||
|
and records requests when a metrics username is set. Only one test per package
|
||||||
|
can set up `/metrics`, and in `internal/server` that is
|
||||||
|
`TestMaintenanceModeKeepsOtherRoutes`, which needs the owner's approval to
|
||||||
|
change; #180 holds it. Tests only; the basic auth library already compares the
|
||||||
|
password in constant time.
|
||||||
|
- 2026-10-04 the image route's signature check and error answers are tested
|
||||||
|
(closes #76): new tests in `internal/handlers`, with no network, check the
|
||||||
|
status and JSON error body for a missing, wrong, unpadded, upper-case or
|
||||||
|
expired signature on a host not on the allowlist, or a valid one sent for its
|
||||||
|
parent domain, a sibling host, a subdomain or the host with another domain
|
||||||
|
appended (401), an unparseable path (400), `localhost` as the upstream host
|
||||||
|
(403) and an upstream error (502); that an allowlisted host is served without
|
||||||
|
a signature, another host only with a valid one; and the answers of
|
||||||
|
`/robots.txt` and the health check. No code changes.
|
||||||
- 2026-10-04 request IDs returned and passed on, and `/v1/e/` revalidates
|
- 2026-10-04 request IDs returned and passed on, and `/v1/e/` revalidates
|
||||||
(closes #84): pixa's own `RequestID` middleware, in place of chi's, gives each
|
(closes #84): pixa's own `RequestID` middleware, in place of chi's, gives each
|
||||||
request an ID, its own `X-Request-ID` when that is at most 64 letters, digits,
|
request an ID, its own `X-Request-ID` when that is at most 64 letters, digits,
|
||||||
@@ -45,26 +289,26 @@ P2: security: referer blacklist
|
|||||||
`Vary: Accept` is left to #88.
|
`Vary: Accept` is left to #88.
|
||||||
- 2026-10-04 routes, encrypted URLs and config file documented (closes #75):
|
- 2026-10-04 routes, encrypted URLs and config file documented (closes #75):
|
||||||
"Routes" in `README.md` lists every route with its method, purpose, what it
|
"Routes" in `README.md` lists every route with its method, purpose, what it
|
||||||
needs and the status codes it answers with, and says `q` and `fit` are part
|
needs and the status codes it answers with, and says `q` and `fit` are part of
|
||||||
of what is cached; "Encrypted URLs" covers logging in, making one on the
|
what is cached; "Encrypted URLs" covers logging in, making one on the
|
||||||
generator page, how long it lasts and the 410 once it has expired;
|
generator page, how long it lasts and the 410 once it has expired;
|
||||||
"Configuration" gives the order in which pixa looks for its config file;
|
"Configuration" gives the order in which pixa looks for its config file;
|
||||||
`config.example.yml` lists `db_url` and `env` and gives every key's default;
|
`config.example.yml` lists `db_url` and `env` and gives every key's default;
|
||||||
`scripts/manual-test.sh` is left to #97.
|
`scripts/manual-test.sh` is left to #97.
|
||||||
- 2026-10-04 shutdown stops cache eviction in progress (closes #102):
|
- 2026-10-04 shutdown stops cache eviction in progress (closes #102):
|
||||||
`StartEviction` runs the eviction goroutine with its own context, which
|
`StartEviction` runs the eviction goroutine with its own context, which
|
||||||
`StopEviction` cancels, so a pass in progress stops at its next database
|
`StopEviction` cancels, so a pass in progress stops at its next database call,
|
||||||
call, file, row or eviction candidate instead of running to completion, and
|
file, row or eviction candidate instead of running to completion, and no pass
|
||||||
no pass starts after it, so a stop logs at most one warning;
|
starts after it, so a stop logs at most one warning; `StopEviction` takes a
|
||||||
`StopEviction` takes a context and, when that context ends before the
|
context and, when that context ends before the goroutine exits, stops waiting
|
||||||
goroutine exits, stops waiting and returns its error; the handlers' stop hook
|
and returns its error; the handlers' stop hook passes fx's stop context, so an
|
||||||
passes fx's stop context, so an eviction still running when fx's stop
|
eviction still running when fx's stop deadline ends fails the stop and makes
|
||||||
deadline ends fails the stop and makes the exit code 1.
|
the exit code 1.
|
||||||
- 2026-10-04 dead code in `internal/imgcache` is gone (closes #73): `Purge`,
|
- 2026-10-04 dead code in `internal/imgcache` is gone (closes #73): `Purge`,
|
||||||
which only returned an error and which nothing called, is no longer part of
|
which only returned an error and which nothing called, is no longer part of
|
||||||
the `ImageCache` interface or `Service`; the `SignatureValidator`,
|
the `ImageCache` interface or `Service`; the `SignatureValidator`, `Allowlist`
|
||||||
`Allowlist` and `Storage` interfaces, which nothing implemented or used, are
|
and `Storage` interfaces, which nothing implemented or used, are deleted.
|
||||||
deleted. Nothing else changes.
|
Nothing else changes.
|
||||||
- 2026-10-04 upstream host semaphores and variant `.meta` files no longer
|
- 2026-10-04 upstream host semaphores and variant `.meta` files no longer
|
||||||
outlive their use (closes #87): the fetcher counts the fetches holding or
|
outlive their use (closes #87): the fetcher counts the fetches holding or
|
||||||
waiting for a slot of each upstream host's semaphore and removes the host's
|
waiting for a slot of each upstream host's semaphore and removes the host's
|
||||||
@@ -76,21 +320,20 @@ P2: security: referer blacklist
|
|||||||
cache directories pixa uses (`cache/sources`, `cache/metadata`,
|
cache directories pixa uses (`cache/sources`, `cache/metadata`,
|
||||||
`cache/variants`) and how files are named in each, and the comments in
|
`cache/variants`) and how files are named in each, and the comments in
|
||||||
`001_schema.sql` name the same paths; the routes and the signature section
|
`001_schema.sql` name the same paths; the routes and the signature section
|
||||||
list the same output formats, `jpg` and `original` included; the TLS
|
list the same output formats, `jpg` and `original` included; the TLS sentence
|
||||||
sentence names `allow_http` as its exception; "Metrics" says only generic
|
names `allow_http` as its exception; "Metrics" says only generic HTTP and Go
|
||||||
HTTP and Go runtime metrics exist, measured and served only when the metrics
|
runtime metrics exist, measured and served only when the metrics username and
|
||||||
username and password are set.
|
password are set.
|
||||||
- 2026-10-03 shutdown sets the exit code and waits for image processing
|
- 2026-10-03 shutdown sets the exit code and waits for image processing (closes
|
||||||
(closes #86): fx alone handles SIGINT and SIGTERM, and the server's own
|
#86): fx alone handles SIGINT and SIGTERM, and the server's own signal handler
|
||||||
signal handler is gone; fx's `Run` in `cmd/pixad` exits with the shutdown's
|
is gone; fx's `Run` in `cmd/pixad` exits with the shutdown's code: 0 for a
|
||||||
code: 0 for a signal, 1 when the HTTP server cannot listen or the app fails
|
signal, 1 when the HTTP server cannot listen or the app fails to start or to
|
||||||
to start or to stop; the server's stop hook, which fx waits for, stops the
|
stop; the server's stop hook, which fx waits for, stops the HTTP server, waits
|
||||||
HTTP server, waits for the images still being processed, both within 5
|
for the images still being processed, both within 5 seconds, then flushes
|
||||||
seconds, then flushes Sentry; images still being processed after that are
|
Sentry; images still being processed after that are logged with their count
|
||||||
logged with their count and make the exit code 1; a Sentry DSN that cannot be
|
and make the exit code 1; a Sentry DSN that cannot be used fails startup, so
|
||||||
used fails startup, so the stop hooks of what had already started run,
|
the stop hooks of what had already started run, instead of exiting the process
|
||||||
instead of exiting the process from a goroutine; the eviction loop is left to
|
from a goroutine; the eviction loop is left to #102.
|
||||||
#102.
|
|
||||||
- 2026-10-03 every `script/cibuild` and `script/docker` run executes the checks
|
- 2026-10-03 every `script/cibuild` and `script/docker` run executes the checks
|
||||||
(closes #101): the `Dockerfile` declares `CHECK_EPOCH` above `make fmt-check`
|
(closes #101): the `Dockerfile` declares `CHECK_EPOCH` above `make fmt-check`
|
||||||
and `make lint` in the lint stage and above `make test` in the build stage,
|
and `make lint` in the lint stage and above `make test` in the build stage,
|
||||||
@@ -105,11 +348,11 @@ P2: security: referer blacklist
|
|||||||
upstream fetch or cached source read and one transcode through
|
upstream fetch or cached source read and one transcode through
|
||||||
`golang.org/x/sync/singleflight`; the first request's processing ignores its
|
`golang.org/x/sync/singleflight`; the first request's processing ignores its
|
||||||
cancellation but keeps its deadline, and the others wait for its image or
|
cancellation but keeps its deadline, and the others wait for its image or
|
||||||
error holding no upstream connection or processing slot, and stop waiting
|
error holding no upstream connection or processing slot, and stop waiting when
|
||||||
when their own context ends; the request doing the processing waits for it
|
their own context ends; the request doing the processing waits for it even
|
||||||
even then, up to its deadline; a request whose context has already ended
|
then, up to its deadline; a request whose context has already ended starts
|
||||||
starts nothing; each request counts one miss, and the processing counts its
|
nothing; each request counts one miss, and the processing counts its fetch and
|
||||||
fetch and transcode once; a panic while processing is reported to Sentry when
|
transcode once; a panic while processing is reported to Sentry when
|
||||||
`sentry_dsn` is set and becomes an error for every waiting request instead of
|
`sentry_dsn` is set and becomes an error for every waiting request instead of
|
||||||
stopping pixad; documented in `README.md`.
|
stopping pixad; documented in `README.md`.
|
||||||
- 2026-09-29 only the image routes send CORS headers (closes #98): the CORS
|
- 2026-09-29 only the image routes send CORS headers (closes #98): the CORS
|
||||||
@@ -118,20 +361,20 @@ P2: security: referer blacklist
|
|||||||
still answers a preflight `OPTIONS` request; the login and URL generator
|
still answers a preflight `OPTIONS` request; the login and URL generator
|
||||||
pages, `/metrics` and the other routes send no `Access-Control-Allow-Origin`;
|
pages, `/metrics` and the other routes send no `Access-Control-Allow-Origin`;
|
||||||
documented in `README.md` and `config.example.yml`.
|
documented in `README.md` and `config.example.yml`.
|
||||||
- 2026-10-02 a plain `docker build .` stamps the tag or short commit, not
|
- 2026-10-02 a plain `docker build .` stamps the tag or short commit, not `dev`
|
||||||
`dev` (closes #166): `.dockerignore` lets `.git` into the build context,
|
(closes #166): `.dockerignore` lets `.git` into the build context, without
|
||||||
without `.git/config`; with no `VERSION` build argument the `Dockerfile`
|
`.git/config`; with no `VERSION` build argument the `Dockerfile` takes the
|
||||||
takes the version from `git describe --tags --always`, and fails the build if
|
version from `git describe --tags --always`, and fails the build if the
|
||||||
the context carries `.git` and no version comes out; `ARG VERSION` has no
|
context carries `.git` and no version comes out; `ARG VERSION` has no default;
|
||||||
default; pixad logs its version, with its name and architecture, as its first
|
pixad logs its version, with its name and architecture, as its first log line
|
||||||
log line at startup.
|
at startup.
|
||||||
- 2026-09-29 the container makes `/var/lib/pixa` usable by itself (closes
|
- 2026-09-29 the container makes `/var/lib/pixa` usable by itself (closes #159):
|
||||||
#159): `deploy/docker-entrypoint.sh` creates the directory if it is missing,
|
`deploy/docker-entrypoint.sh` creates the directory if it is missing, gives
|
||||||
gives the directory and everything in it to `pixad` when the directory or one
|
the directory and everything in it to `pixad` when the directory or one of its
|
||||||
of its top-level entries belongs to another user or group, sets its mode to
|
top-level entries belongs to another user or group, sets its mode to `750`,
|
||||||
`750`, then runs the server as `pixad`; data left by an earlier run under
|
then runs the server as `pixad`; data left by an earlier run under another uid
|
||||||
another uid is taken over this way; "Running under upaas" in `README.md` no
|
is taken over this way; "Running under upaas" in `README.md` no longer tells
|
||||||
longer tells the operator to create or chown the host directory.
|
the operator to create or chown the host directory.
|
||||||
- 2026-09-29 variant content types kept in memory (closes #70):
|
- 2026-09-29 variant content types kept in memory (closes #70):
|
||||||
`Cache.metaCache` holds the content types of up to 10,000 variants in an LRU
|
`Cache.metaCache` holds the content types of up to 10,000 variants in an LRU
|
||||||
(`github.com/hashicorp/golang-lru/v2`), filled by `StoreVariant` and by
|
(`github.com/hashicorp/golang-lru/v2`), filled by `StoreVariant` and by
|
||||||
@@ -170,8 +413,8 @@ P2: security: referer blacklist
|
|||||||
`ARG VERSION` sits just above the build, so a new version reruns neither
|
`ARG VERSION` sits just above the build, so a new version reruns neither
|
||||||
`script/bootstrap` nor the tests.
|
`script/bootstrap` nor the tests.
|
||||||
- 2026-09-29 migrations at the path `REPO_POLICIES.md` sets (closes #96): the
|
- 2026-09-29 migrations at the path `REPO_POLICIES.md` sets (closes #96): the
|
||||||
migration files moved, contents unchanged, from `internal/database/schema/`
|
migration files moved, contents unchanged, from `internal/database/schema/` to
|
||||||
to `internal/db/migrations/` as `000_migration.sql` and `001_schema.sql`; the
|
`internal/db/migrations/` as `000_migration.sql` and `001_schema.sql`; the
|
||||||
`internal/db/migrations` package embeds them and `internal/database` reads
|
`internal/db/migrations` package embeds them and `internal/database` reads
|
||||||
them through its `FS()`; the `internal/database` package itself stays; the
|
them through its `FS()`; the `internal/database` package itself stays; the
|
||||||
version still comes from the filename prefix, so a database that has recorded
|
version still comes from the filename prefix, so a database that has recorded
|
||||||
@@ -186,8 +429,8 @@ P2: security: referer blacklist
|
|||||||
`=` padding kept, and gives the example's `sig` for a stated signing key.
|
`=` padding kept, and gives the example's `sig` for a stated signing key.
|
||||||
- 2026-09-29 fixed uid and gid for `pixad` (closes #151): the image creates the
|
- 2026-09-29 fixed uid and gid for `pixad` (closes #151): the image creates the
|
||||||
`pixad` group with gid 65532 and the `pixad` user with uid 65532, instead of
|
`pixad` group with gid 65532 and the `pixad` user with uid 65532, instead of
|
||||||
the first free uid 1000, so a bind-mounted `/var/lib/pixa` given to `pixad`
|
the first free uid 1000, so a bind-mounted `/var/lib/pixa` given to `pixad` is
|
||||||
is not owned on the host by a person's login account; the first-run step of
|
not owned on the host by a person's login account; the first-run step of
|
||||||
"Running under upaas" in `README.md` names the uid and gid.
|
"Running under upaas" in `README.md` names the uid and gid.
|
||||||
- 2026-09-29 `max-age` never outlives an expiring URL (closes #63): both image
|
- 2026-09-29 `max-age` never outlives an expiring URL (closes #63): both image
|
||||||
routes build `Cache-Control` from the request's `Expires`, which an encrypted
|
routes build `Cache-Control` from the request's `Expires`, which an encrypted
|
||||||
@@ -196,29 +439,27 @@ P2: security: referer blacklist
|
|||||||
that is sooner, never negative; an allowlisted host's URL that has an `exp`
|
that is sooner, never negative; an allowlisted host's URL that has an `exp`
|
||||||
follows it too; `immutable` stays, as freshness now ends at the expiry;
|
follows it too; `immutable` stays, as freshness now ends at the expiry;
|
||||||
documented in `README.md`.
|
documented in `README.md`.
|
||||||
- 2026-09-28 add the four settings `README.md` documented but pixa did not
|
- 2026-09-28 add the four settings `README.md` documented but pixa did not have,
|
||||||
have, which aborted startup as unknown keys (closes #61):
|
which aborted startup as unknown keys (closes #61):
|
||||||
`access_control_allow_origin` (default `*`, the CORS origin),
|
`access_control_allow_origin` (default `*`, the CORS origin),
|
||||||
`upstream_fetch_timeout` (default `30s`), `upstream_max_response_size`
|
`upstream_fetch_timeout` (default `30s`), `upstream_max_response_size`
|
||||||
(default 50 MiB) and `downstream_timeout` (default `60s`, both the
|
(default 50 MiB) and `downstream_timeout` (default `60s`, both the server's
|
||||||
server's write timeout and the per-request timeout); each has a
|
write timeout and the per-request timeout); each has a `PIXA_` variable;
|
||||||
`PIXA_` variable; durations are positive Go duration strings, the size a
|
durations are positive Go duration strings, the size a whole number of bytes
|
||||||
whole number of bytes up to 1 GiB, the origin `*` or one `http` or
|
up to 1 GiB, the origin `*` or one `http` or `https` origin as `README.md`
|
||||||
`https` origin as `README.md` describes it; an invalid value
|
describes it; an invalid value aborts startup naming the key and the value;
|
||||||
aborts startup naming the key and the value; documented in
|
documented in `config.example.yml` and `README.md`.
|
||||||
`config.example.yml` and `README.md`.
|
- 2026-09-28 cache stats report real numbers (closes #56): `Cache.Stats` counts
|
||||||
- 2026-09-28 cache stats report real numbers (closes #56): `Cache.Stats`
|
the cached source images and processed variants (`source_content` plus
|
||||||
counts the cached source images and processed variants (`source_content`
|
`variant_content`) and takes their size from `Cache.UsageBytes`, instead of
|
||||||
plus `variant_content`) and takes their size from `Cache.UsageBytes`,
|
reading `request_cache` and `output_content`, which nothing writes; those two
|
||||||
instead of reading `request_cache` and `output_content`, which nothing
|
tables are left in the schema; a disabled disk cache reports no items and no
|
||||||
writes; those two tables are left in the schema; a disabled disk cache
|
size. A hit is counted even when the request context has ended. A miss is
|
||||||
reports no items and no size. A hit is counted even when the request
|
counted after it is served or fails, also when the request context has ended
|
||||||
context has ended. A miss is counted after it is served or fails, also
|
by then, with the bytes it read from upstream, so `upstream_fetch_count` and
|
||||||
when the request context has ended by then, with the bytes it read from
|
`upstream_fetch_bytes` move, including for an upstream body that fails partway
|
||||||
upstream, so `upstream_fetch_count` and `upstream_fetch_bytes` move,
|
or a fetched source that then fails the magic byte check; `transform_count`
|
||||||
including for an upstream body that fails partway or a fetched source
|
counts each image the image processor transcodes.
|
||||||
that then fails the magic byte check; `transform_count` counts each image
|
|
||||||
the image processor transcodes.
|
|
||||||
- 2026-09-28 strip metadata from processed images (closes #82): every output is
|
- 2026-09-28 strip metadata from processed images (closes #82): every output is
|
||||||
exported with govips' `StripMetadata`, so it carries no EXIF, XMP, IPTC or ICC
|
exported with govips' `StripMetadata`, so it carries no EXIF, XMP, IPTC or ICC
|
||||||
profile; the image is first turned upright with `AutoRotate` (before sizes are
|
profile; the image is first turned upright with `AutoRotate` (before sizes are
|
||||||
@@ -229,238 +470,213 @@ P2: security: referer blacklist
|
|||||||
attempts per minute per client address, and an attempt over the limit is
|
attempts per minute per client address, and an attempt over the limit is
|
||||||
refused with 429 and a `Retry-After` header; the address is the one
|
refused with 429 and a `Retry-After` header; the address is the one
|
||||||
`internal/clientip` resolves through `trusted_proxies`, an IPv6 client is
|
`internal/clientip` resolves through `trusted_proxies`, an IPv6 client is
|
||||||
counted by its /64, and an IPv4-mapped address as the IPv4 address it
|
counted by its /64, and an IPv4-mapped address as the IPv4 address it carries;
|
||||||
carries; the limit is a `RateLimit` middleware in `internal/middleware` on
|
the limit is a `RateLimit` middleware in `internal/middleware` on
|
||||||
`github.com/go-chi/httprate`, which the image routes can reuse; the library
|
`github.com/go-chi/httprate`, which the image routes can reuse; the library
|
||||||
keeps counts for the current and the previous minute only; documented in
|
keeps counts for the current and the previous minute only; documented in
|
||||||
`README.md`.
|
`README.md`.
|
||||||
- 2026-09-28 refuse an unparseable `exp` on `/v1/image/` and log swallowed
|
- 2026-09-28 refuse an unparseable `exp` on `/v1/image/` and log swallowed cache
|
||||||
cache errors (closes #72): an `exp` in the URL that is not a whole
|
errors (closes #72): an `exp` in the URL that is not a whole number, an empty
|
||||||
number, an empty `exp=` included, is a 400 naming `exp` and the value,
|
`exp=` included, is a 400 naming `exp` and the value, instead of being ignored
|
||||||
instead of being ignored and answered with 401 as if the URL had no
|
and answered with 401 as if the URL had no `exp`; only an `exp` missing from
|
||||||
`exp`; only an `exp` missing from the URL is unchanged; `README.md` says
|
the URL is unchanged; `README.md` says so where it documents `exp`. A failed
|
||||||
so where it documents `exp`. A failed variant `.meta` write, source
|
variant `.meta` write, source metadata JSON write, `Stats` count query, stats
|
||||||
metadata JSON write, `Stats` count query, stats counter update, negative
|
counter update, negative cache write or expired negative cache delete is now
|
||||||
cache write or expired negative cache delete is now logged at `warn`
|
logged at `warn` with the path or key and the error, and stays non-fatal.
|
||||||
with the path or key and the error, and stays non-fatal.
|
- 2026-09-28 refuse an empty `fit` on `/v1/image/` (closes #139): a `fit` in the
|
||||||
- 2026-09-28 refuse an empty `fit` on `/v1/image/` (closes #139): a
|
URL with an empty value (`fit=`) is a 400 naming `fit`, instead of being
|
||||||
`fit` in the URL with an empty value (`fit=`) is a 400 naming `fit`,
|
served as `cover` and verified against a signature made for `cover`; only a
|
||||||
instead of being served as `cover` and verified against a signature
|
`fit` missing from the URL is still `cover`; any other value still goes
|
||||||
made for `cover`; only a `fit` missing from the URL is still `cover`;
|
through the existing fit-mode check; `README.md` says so where it documents
|
||||||
any other value still goes through the existing fit-mode check;
|
`fit`.
|
||||||
`README.md` says so where it documents `fit`.
|
- 2026-09-28 refuse an invalid `q` on `/v1/image/` (closes #134): a `q` that is
|
||||||
- 2026-09-28 refuse an invalid `q` on `/v1/image/` (closes #134): a `q`
|
not a whole number from 1 to 100, an empty `q` included, is a 400 naming `q`
|
||||||
that is not a whole number from 1 to 100, an empty `q` included, is a
|
and the value, instead of being served at the default 85; the route reads `q`
|
||||||
400 naming `q` and the value, instead of being served at the default
|
with the generator's quality check (`parseFormInt` with `minQuality` and
|
||||||
85; the route reads `q` with the generator's quality check
|
`maxQuality`); only a `q` missing from the URL is still 85; a query string
|
||||||
(`parseFormInt` with `minQuality` and `maxQuality`); only a `q` missing
|
that cannot be decoded, such as `q=80%`, is a 400 showing it; any query
|
||||||
from the URL is still 85; a query string that cannot be decoded, such
|
parameter given more than once (`q`, `fit`, `sig`, `exp` alike) is a 400
|
||||||
as `q=80%`, is a 400 showing it; any query parameter given more than
|
naming it, so none is read from its first value only; `README.md` states the
|
||||||
once (`q`, `fit`, `sig`, `exp` alike) is a 400 naming it, so none is
|
range and both query-string rules.
|
||||||
read from its first value only; `README.md` states the range and both
|
- 2026-09-28 unknown `PIXA_` environment variables abort startup (closes #133):
|
||||||
query-string rules.
|
a variable whose name starts with `PIXA_` but is neither a setting's variable
|
||||||
- 2026-09-28 unknown `PIXA_` environment variables abort startup (closes
|
nor `PIXA_CONFIG_PATH` aborts startup naming it, as an unknown config key
|
||||||
#133): a variable whose name starts with `PIXA_` but is neither a
|
does, and `PIXA_PORT` is named with a pointer to `PORT`; the check runs after
|
||||||
setting's variable nor `PIXA_CONFIG_PATH` aborts startup naming it, as
|
the config file loads, so the variables the file's `env:` section sets are
|
||||||
an unknown config key does, and `PIXA_PORT` is named with a pointer to
|
checked too; documented in `README.md`.
|
||||||
`PORT`; the check runs after the config file loads, so the variables
|
- 2026-09-28 start on a fresh upaas volume (closes #129): the image starts as
|
||||||
the file's `env:` section sets are checked too; documented in
|
root only to give `/var/lib/pixa` to `pixad` when `pixad` does not own it
|
||||||
`README.md`.
|
(`deploy/docker-entrypoint.sh`), then runs the server as `pixad` through
|
||||||
- 2026-09-28 start on a fresh upaas volume (closes #129): the image
|
`su-exec`, so a root-owned host directory bind-mounted there no longer stops
|
||||||
starts as root only to give `/var/lib/pixa` to `pixad` when `pixad`
|
the container at startup; `README.md` gains a "Running under upaas" section.
|
||||||
does not own it (`deploy/docker-entrypoint.sh`), then runs the server
|
- 2026-09-28 run all linting in Docker via `Dockerfile.lint` + `script/lint`
|
||||||
as `pixad` through `su-exec`, so a root-owned host directory
|
(closes #104): `make lint` calls `script/lint`, the only way the linter is
|
||||||
bind-mounted there no longer stops the container at startup;
|
run; inside a container (both Dockerfiles set `container=docker`) it runs
|
||||||
`README.md` gains a "Running under upaas" section.
|
`golangci-lint`, anywhere else it builds the hash-pinned `Dockerfile.lint`,
|
||||||
- 2026-09-28 run all linting in Docker via `Dockerfile.lint` +
|
whose last step runs `script/lint` again; the `Dockerfile` lint stage runs
|
||||||
`script/lint` (closes #104): `make lint` calls `script/lint`, the only
|
`make lint`; no host or nix-shell `golangci-lint` path remains
|
||||||
way the linter is run; inside a container (both Dockerfiles set
|
(`script/bootstrap` installs no linter); a per-run `CACHEBUST` build-arg keeps
|
||||||
`container=docker`) it runs `golangci-lint`, anywhere else it builds the
|
the lint step from being served from cache, and a tmpfs mount on that step
|
||||||
hash-pinned `Dockerfile.lint`, whose last step runs `script/lint` again;
|
keeps Go's and golangci-lint's caches out of its layer, so a run leaves no
|
||||||
the `Dockerfile` lint stage runs `make lint`; no host or nix-shell
|
large build cache behind; `golangci-lint config verify` stays out, as it
|
||||||
`golangci-lint` path remains (`script/bootstrap` installs no linter);
|
fetches its schema over an unpinned live HTTPS call
|
||||||
a per-run `CACHEBUST` build-arg keeps the lint step from being served
|
- 2026-09-28 every setting as an environment variable (closes #128, also covers
|
||||||
from cache, and a tmpfs mount on that step keeps Go's and
|
#99): each config key can be set by `PIXA_` plus the key in upper case (`.`
|
||||||
golangci-lint's caches out of its layer, so a run leaves no large build
|
written as `_`), and the port by `PORT`; a variable present in the
|
||||||
cache behind; `golangci-lint config verify` stays out, as it fetches its
|
environment, even empty, wins over the config file, which wins over the
|
||||||
schema over an unpinned live HTTPS call
|
default; the typed getters read the variable first, so every existing check
|
||||||
- 2026-09-28 every setting as an environment variable (closes #128, also
|
applies to it and a bad value aborts startup naming the variable; lists are
|
||||||
covers #99): each config key can be set by `PIXA_` plus the key in upper
|
comma-separated, and an empty variable (or `""` in the file) is an empty list;
|
||||||
case (`.` written as `_`), and the port by `PORT`; a variable present in
|
the Docker image no longer bakes in `config.docker.yml` or passes `--config`,
|
||||||
the environment, even empty, wins over the config file, which wins over
|
and its `HEALTHCHECK` probes `PORT` (default `8080`); the config file is
|
||||||
the default; the typed getters read the variable first, so every existing
|
looked for under `/etc/pixa` and `~/.config/pixa` instead of the daemon name
|
||||||
check applies to it and a bad value aborts startup naming the variable;
|
`pixad`; documented in `README.md` and `config.example.yml`.
|
||||||
lists are comma-separated, and an empty variable (or `""` in the file) is
|
- 2026-09-28 quality and fit in the URL signature (closes #60): the signed data
|
||||||
an empty list; the Docker image no longer bakes in `config.docker.yml` or
|
is now `host:path:query:width:height:format:expiration:quality:fit`, using
|
||||||
passes `--config`, and its `HEALTHCHECK` probes `PORT` (default `8080`);
|
`85` and `cover` when the URL has no `q` or `fit`, so one signed URL can no
|
||||||
the config file is looked for under `/etc/pixa` and `~/.config/pixa`
|
longer be replayed across other quality and fit values to create unauthorized
|
||||||
instead of the daemon name `pixad`; documented in `README.md` and
|
cache entries and transcodes; the known-answer vectors in
|
||||||
`config.example.yml`.
|
`internal/signature/golden_test.go` and the README signature specification
|
||||||
- 2026-09-28 quality and fit in the URL signature (closes #60): the signed
|
describe the new format.
|
||||||
data is now `host:path:query:width:height:format:expiration:quality:fit`,
|
- 2026-09-28 Docker image healthcheck (closes #111): a `HEALTHCHECK` in the
|
||||||
using `85` and `cover` when the URL has no `q` or `fit`, so one signed
|
runtime stage probing `/.well-known/healthcheck.json` with busybox `wget`;
|
||||||
URL can no longer be replayed across other quality and fit values to
|
`script/docker-smoke` (`make docker-smoke`) builds the image, starts it with a
|
||||||
create unauthorized cache entries and transcodes; the known-answer
|
throwaway `PIXA_SIGNING_KEY`, and passes only once Docker reports it healthy
|
||||||
vectors in `internal/signature/golden_test.go` and the README signature
|
within 30 seconds, removing the container on exit; the Gitea workflow runs it
|
||||||
specification describe the new format.
|
after `script/cibuild`.
|
||||||
- 2026-09-28 Docker image healthcheck (closes #111): a `HEALTHCHECK` in
|
|
||||||
the runtime stage probing `/.well-known/healthcheck.json` with busybox
|
|
||||||
`wget`; `script/docker-smoke` (`make docker-smoke`) builds the image,
|
|
||||||
starts it with a throwaway `PIXA_SIGNING_KEY`, and passes only once
|
|
||||||
Docker reports it healthy within 30 seconds, removing the container on
|
|
||||||
exit; the Gitea workflow runs it after `script/cibuild`.
|
|
||||||
- 2026-09-21 trusted-proxy client IP resolution (closes #94): a
|
- 2026-09-21 trusted-proxy client IP resolution (closes #94): a
|
||||||
`trusted_proxies` config key taking a list of CIDRs, parsed by the same
|
`trusted_proxies` config key taking a list of CIDRs, parsed by the same
|
||||||
`net/netip` list parser as `blocked_networks` (an invalid entry aborts
|
`net/netip` list parser as `blocked_networks` (an invalid entry aborts startup
|
||||||
startup naming the key and value; an omitted key defaults to the RFC 1918
|
naming the key and value; an omitted key defaults to the RFC 1918 private
|
||||||
private ranges, an explicitly empty list trusts no one, and an explicit
|
ranges, an explicitly empty list trusts no one, and an explicit list replaces
|
||||||
list replaces the default); a new
|
the default); a new `internal/clientip` package resolves the client address by
|
||||||
`internal/clientip` package resolves the client address by honoring
|
honoring `X-Forwarded-For` only when the direct peer is a trusted proxy,
|
||||||
`X-Forwarded-For` only when the direct peer is a trusted proxy, walking
|
walking the chain right-to-left to the rightmost non-proxy entry, so a client
|
||||||
the chain right-to-left to the rightmost non-proxy entry, so a client
|
connecting directly cannot spoof its address; the resolved address is stored
|
||||||
connecting directly cannot spoof its address; the resolved address is
|
in the request context by a new middleware and used by the request-logging
|
||||||
stored in the request context by a new middleware and used by the
|
middleware and the login-attempt logs in place of the raw peer address;
|
||||||
request-logging middleware and the login-attempt logs in place of the
|
documented in `README.md` and `config.example.yml`.
|
||||||
raw peer address; documented in `README.md` and `config.example.yml`.
|
|
||||||
- 2026-09-21 blocked networks configuration extending SSRF protection: a
|
- 2026-09-21 blocked networks configuration extending SSRF protection: a
|
||||||
`blocked_networks` config key taking a list of CIDRs (parsed with
|
`blocked_networks` config key taking a list of CIDRs (parsed with `net/netip`,
|
||||||
`net/netip`, an invalid entry aborts startup naming the key and value),
|
an invalid entry aborts startup naming the key and value), added to the
|
||||||
added to the built-in blocklist rather than replacing it; the built-in
|
built-in blocklist rather than replacing it; the built-in ranges extended to
|
||||||
ranges extended to CGNAT `100.64.0.0/10`, IETF protocol assignments
|
CGNAT `100.64.0.0/10`, IETF protocol assignments `192.0.0.0/24`, benchmark
|
||||||
`192.0.0.0/24`, benchmark `198.18.0.0/15`, and NAT64 `64:ff9b::/96`
|
`198.18.0.0/15`, and NAT64 `64:ff9b::/96` (IPv4-mapped forms covered);
|
||||||
(IPv4-mapped forms covered); enforcement stays in the dial-time
|
enforcement stays in the dial-time re-resolution so the DNS-rebinding window
|
||||||
re-resolution so the DNS-rebinding window remains closed; documented in
|
remains closed; documented in `README.md` and `config.example.yml`.
|
||||||
`README.md` and `config.example.yml`.
|
- 2026-09-21 validate dimensions and fit mode on the encrypted-URL route and the
|
||||||
- 2026-09-21 validate dimensions and fit mode on the encrypted-URL
|
token generator (closes #62): `imgcache.ValidateDimension` alone holds the
|
||||||
route and the token generator (closes #62): `imgcache.ValidateDimension`
|
`MaxDimension` bound and is used by the path parser, by the new
|
||||||
alone holds the `MaxDimension` bound and is used by the path parser, by
|
`ValidateImageRequest` (which also applies `ValidateFitMode`) and by the
|
||||||
the new `ValidateImageRequest` (which also applies `ValidateFitMode`)
|
generator; both the `/v1/image/` and `/v1/e/` routes call
|
||||||
and by the generator; both the `/v1/image/` and `/v1/e/` routes call
|
`ValidateImageRequest`, so an over-limit size or an unknown fit mode is a 400
|
||||||
`ValidateImageRequest`, so an over-limit size or an unknown fit mode is a
|
rather than an out-of-memory or a 500 from the processor; the URL generator
|
||||||
400 rather than an out-of-memory or a 500 from the processor; the URL
|
answers 400 naming the field for a `width` or `height` that is not a number or
|
||||||
generator answers 400 naming the field for a `width` or `height` that is
|
fails the shared check, a `quality` that is not a number from 1 to 100, a
|
||||||
not a number or fails the shared check, a `quality` that is not a number
|
`ttl` that is not a number from 0 to the largest number of seconds the expiry
|
||||||
from 1 to 100, a `ttl` that is not a number from 0 to the largest number
|
calculation can hold, or an unknown `fit`; an empty `quality` is 85 and an
|
||||||
of seconds the expiry calculation can hold, or an unknown `fit`; an empty
|
empty `ttl` never expires; the form's width and height inputs stop at 8192
|
||||||
`quality` is 85 and
|
- 2026-09-21 http.Server hardening (closes #92): added `HTTPReadHeaderTimeout`
|
||||||
an empty `ttl` never expires; the form's width and height inputs stop at
|
(10s, bounds the slowloris header dribble) and `HTTPIdleTimeout` (120s, bounds
|
||||||
8192
|
keep-alive reuse) alongside the existing timeouts and wired them onto the
|
||||||
- 2026-09-21 http.Server hardening (closes #92): added
|
server; added a `LimitBody` middleware capping the two form POST bodies
|
||||||
`HTTPReadHeaderTimeout` (10s, bounds the slowloris header dribble) and
|
(`POST /`, `POST /generate`) at `MaxFormBytes` (1 MiB) and returning 413,
|
||||||
`HTTPIdleTimeout` (120s, bounds keep-alive reuse) alongside the
|
applied ahead of the CSRF middleware so an oversized body is refused as 413
|
||||||
existing timeouts and wired them onto the server; added a `LimitBody`
|
rather than being read as a missing CSRF token (403); left `WriteTimeout` at
|
||||||
middleware capping the two form POST bodies (`POST /`, `POST /generate`)
|
60s unchanged
|
||||||
at `MaxFormBytes` (1 MiB) and returning 413, applied ahead of the CSRF
|
- 2026-08-07 update golangci-lint to v2.12.2 with the canonical `.golangci.yml`
|
||||||
middleware so an oversized body is refused as 413 rather than being read
|
(v2 schema, `default: all` minus six disabled linters, `lll` 88, tests
|
||||||
as a missing CSRF token (403); left `WriteTimeout` at 60s unchanged
|
included): bumped the pinned `golangci/golangci-lint:v2.12.2-alpine` image in
|
||||||
- 2026-08-07 update golangci-lint to v2.12.2 with the canonical
|
`Dockerfile` and the release-archive sha256 pins in `script/bootstrap`; fixed
|
||||||
`.golangci.yml` (v2 schema, `default: all` minus six disabled
|
the findings the stricter config surfaced (notably `paralleltest`, `wsl_v5`,
|
||||||
linters, `lll` 88, tests included): bumped the pinned
|
|
||||||
`golangci/golangci-lint:v2.12.2-alpine` image in `Dockerfile` and the
|
|
||||||
release-archive sha256 pins in `script/bootstrap`; fixed the findings
|
|
||||||
the stricter config surfaced (notably `paralleltest`, `wsl_v5`,
|
|
||||||
`goconst`, `lll`, `noinlineerr`, `err113`, `errcheck`, `testpackage` —
|
`goconst`, `lll`, `noinlineerr`, `err113`, `errcheck`, `testpackage` —
|
||||||
white-box test files renamed to `*_internal_test.go`), including #55's
|
white-box test files renamed to `*_internal_test.go`), including #55's code
|
||||||
code absorbed after it merged, iterating the pinned linter to
|
absorbed after it merged, iterating the pinned linter to `0 issues.`; no
|
||||||
`0 issues.`; no single finding total is substantiable, since
|
single finding total is substantiable, since golangci-lint's `uniq-by-line`
|
||||||
golangci-lint's `uniq-by-line` reveals new findings on a line as
|
reveals new findings on a line as others there are fixed — the documented
|
||||||
others there are fixed — the documented re-measurements were 81 after
|
re-measurements were 81 after the #53 merge and 149 after the #55 merge; three
|
||||||
the #53 merge and 149 after the #55 merge; three behavior changes, so
|
behavior changes, so not a pure no-op: `Cache.StoreVariant` now takes a
|
||||||
not a pure no-op: `Cache.StoreVariant` now takes a `context.Context`
|
`context.Context` (`noctx`), so a cancelled request skips its best-effort
|
||||||
(`noctx`), so a cancelled request skips its best-effort accounting
|
accounting row; `MetadataStorage.Store`'s cleanup defer was dead on `main` and
|
||||||
row; `MetadataStorage.Store`'s cleanup defer was dead on `main` and
|
leaked `.tmp-*.json` on failure, now fixed with explicit removals; and the
|
||||||
leaked `.tmp-*.json` on failure, now fixed with explicit removals; and
|
`signing_key` validation error text gained `value too short: `; the eviction
|
||||||
the `signing_key` validation error text gained `value too short: `;
|
loop's uncancellable context is deferred to #102 under a
|
||||||
the eviction loop's uncancellable context is deferred to #102 under a
|
`//nolint:contextcheck`; three `//nolint:tagliatelle` directives keep the
|
||||||
`//nolint:contextcheck`; three `//nolint:tagliatelle` directives keep
|
snake_case JSON wire/disk formats unchanged; `make check` green
|
||||||
the snake_case JSON wire/disk formats unchanged; `make check` green
|
- 2026-08-07 implement cache size management and eviction (closes #51): new
|
||||||
- 2026-08-07 implement cache size management and eviction (closes
|
`cache_max_bytes` config key validated by the startup framework (explicit
|
||||||
#51): new `cache_max_bytes` config key validated by the startup
|
values used exactly with no floor, `0` disables the disk cache entirely,
|
||||||
framework (explicit values used exactly with no floor, `0` disables
|
omitted defaults to max(75% of free space on the filesystem containing
|
||||||
the disk cache entirely, omitted defaults to max(75% of free space
|
`<state_dir>/cache/`, 500 MiB), logged at startup); processed variants are now
|
||||||
on the filesystem containing `<state_dir>/cache/`, 500 MiB), logged
|
tracked in the database (a new `variant_content` table and an LRU timestamp on
|
||||||
at startup); processed variants are now tracked in the database (a
|
`source_content`) so total usage is two SUMs, never a directory scan on the
|
||||||
new `variant_content` table and an LRU timestamp on `source_content`)
|
hot path; a background goroutine evicts globally least-recently-used entries
|
||||||
so total usage is two SUMs, never a directory scan on the hot path; a
|
(variants and source blobs merged) to the limit, woken by a periodic ticker
|
||||||
background goroutine evicts globally least-recently-used entries
|
and by write-pressure notifications from stores; a source blob and ALL of its
|
||||||
(variants and source blobs merged) to the limit, woken by a periodic
|
`source_metadata` references are deleted in one transaction before the file is
|
||||||
ticker and by write-pressure notifications from stores; a source
|
unlinked, so multi-referenced blobs are never removed while referenced and
|
||||||
blob and ALL of its `source_metadata` references are deleted in one
|
rows never point at deleted files; a startup and periodic reconciliation pass
|
||||||
transaction before the file is unlinked, so multi-referenced blobs
|
adopts untracked variant files, drops rows for missing files, removes
|
||||||
are never removed while referenced and rows never point at deleted
|
unreachable source blobs, and sweeps stale temp files
|
||||||
files; a startup and periodic reconciliation pass adopts untracked
|
- 2026-08-07 validate configuration on startup, fail fast on bad config (closes
|
||||||
variant files, drops rows for missing files, removes unreachable
|
#52): a config value that is set but unparseable or invalid aborts startup
|
||||||
source blobs, and sweeps stale temp files
|
naming the key and value (defaults apply only to omitted keys), unknown config
|
||||||
- 2026-08-07 validate configuration on startup, fail fast on bad
|
keys abort startup, a malformed config file aborts instead of being skipped,
|
||||||
config (closes #52): a config value that is set but unparseable or
|
and `state_dir` is verified creatable and writable before the listener binds
|
||||||
invalid aborts startup naming the key and value (defaults apply only
|
- 2026-08-07 manual test pass of the auth and encrypted URL flows against a
|
||||||
to omitted keys), unknown config keys abort startup, a malformed
|
locally built and running `pixad` (built from `main` at `6573b9d`, port 18099,
|
||||||
config file aborts instead of being skipped, and `state_dir` is
|
local throwaway config); all six checks passed, plus all nine tests in
|
||||||
verified creatable and writable before the listener binds
|
`scripts/manual-test.sh` (closes #49):
|
||||||
- 2026-08-07 manual test pass of the auth and encrypted URL flows
|
- [x] visit `/` and see the login form: HTTP 200, `Pixa - Login` page with
|
||||||
against a locally built and running `pixad` (built from `main` at
|
`name="key"` password form
|
||||||
`6573b9d`, port 18099, local throwaway config); all six checks
|
- [x] wrong key shows an error: POST `/` with `key=wrong-key` returned HTTP
|
||||||
passed, plus all nine tests in `scripts/manual-test.sh` (closes #49):
|
200 login page containing "Invalid signing key"
|
||||||
- [x] visit `/` and see the login form: HTTP 200, `Pixa - Login`
|
- [x] correct signing key shows the generator form: POST `/` returned HTTP
|
||||||
page with `name="key"` password form
|
303 to `/` with
|
||||||
- [x] wrong key shows an error: POST `/` with `key=wrong-key`
|
`Set-Cookie: pixa_session=...; HttpOnly; Secure; SameSite=Strict`; GET
|
||||||
returned HTTP 200 login page containing "Invalid signing key"
|
`/` with that cookie rendered `Pixa - URL Generator` with the
|
||||||
- [x] correct signing key shows the generator form: POST `/`
|
|
||||||
returned HTTP 303 to `/` with
|
|
||||||
`Set-Cookie: pixa_session=...; HttpOnly; Secure; SameSite=Strict`;
|
|
||||||
GET `/` with that cookie rendered `Pixa - URL Generator` with the
|
|
||||||
`/generate` form and logout link
|
`/generate` form and logout link
|
||||||
- [x] a generated encrypted URL serves the image: POST `/generate`
|
- [x] a generated encrypted URL serves the image: POST `/generate`
|
||||||
(ttl=3600) produced a `/v1/e/<token>/img.jpeg` URL that returned
|
(ttl=3600) produced a `/v1/e/<token>/img.jpeg` URL that returned HTTP
|
||||||
HTTP 200, `Content-Type: image/jpeg`, an 800x600 baseline JPEG of
|
200, `Content-Type: image/jpeg`, an 800x600 baseline JPEG of 61706
|
||||||
61706 bytes
|
bytes
|
||||||
- [x] an expired URL (short TTL) returns 410: a ttl=1 URL fetched
|
- [x] an expired URL (short TTL) returns 410: a ttl=1 URL fetched after 3 s
|
||||||
after 3 s returned HTTP 410 Gone with
|
returned HTTP 410 Gone with
|
||||||
`{"error":"URL has expired","status":410,...}`
|
`{"error":"URL has expired","status":410,...}`
|
||||||
- [x] logout redirects back to login: GET `/logout` returned HTTP
|
- [x] logout redirects back to login: GET `/logout` returned HTTP 303 to `/`
|
||||||
303 to `/` with `Set-Cookie: pixa_session=; Max-Age=0`;
|
with `Set-Cookie: pixa_session=; Max-Age=0`; subsequent GET `/`
|
||||||
subsequent GET `/` rendered the login form again
|
rendered the login form again
|
||||||
- 2026-08-07 fix the two remaining gosec findings (G124 in
|
- 2026-08-07 fix the two remaining gosec findings (G124 in internal/session):
|
||||||
internal/session): session cookies now always carry
|
session cookies now always carry Secure/HttpOnly/SameSite=Strict on both the
|
||||||
Secure/HttpOnly/SameSite=Strict on both the set and clear paths;
|
set and clear paths; `make check` green (closes #47)
|
||||||
`make check` green (closes #47)
|
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
||||||
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
|
shims, README Entrypoints section
|
||||||
Makefile shims, README Entrypoints section
|
|
||||||
- 2026-04-07 extract magic byte detection into internal/magic (#42)
|
- 2026-04-07 extract magic byte detection into internal/magic (#42)
|
||||||
- 2026-03-25 extract allowlist package from internal/imgcache (#41)
|
- 2026-03-25 extract allowlist package from internal/imgcache (#41)
|
||||||
- 2026-03-25 move schema_migrations table creation into 000.sql (#36)
|
- 2026-03-25 move schema_migrations table creation into 000.sql (#36)
|
||||||
- 2026-03-20 enforce and document exact-match-only signature
|
- 2026-03-20 enforce and document exact-match-only signature verification (#40)
|
||||||
verification (#40)
|
- 2026-03-20 bound imageprocessor.Process input read to prevent unbounded memory
|
||||||
- 2026-03-20 bound imageprocessor.Process input read to prevent
|
use (#37); consolidate appname into an internal/globals constant (#34)
|
||||||
unbounded memory use (#37); consolidate appname into an
|
|
||||||
internal/globals constant (#34)
|
|
||||||
- 2026-03-18 parse version prefix from migration filenames (#33)
|
- 2026-03-18 parse version prefix from migration filenames (#33)
|
||||||
- 2026-03-15 QA audit fixes for 1.0/MVP readiness (#25)
|
- 2026-03-15 QA audit fixes for 1.0/MVP readiness (#25)
|
||||||
- 2026-03-02 split Dockerfile with pre-built golangci-lint stage for
|
- 2026-03-02 split Dockerfile with pre-built golangci-lint stage for faster CI
|
||||||
faster CI (#23)
|
(#23)
|
||||||
- 2026-02-25 repo policy compliance: CI workflow, hash-pinned images,
|
- 2026-02-25 repo policy compliance: CI workflow, hash-pinned images,
|
||||||
golangci-lint and gosec fixes of that date (#14); arm64 Docker build
|
golangci-lint and gosec fixes of that date (#14); arm64 Docker build fix (#16)
|
||||||
fix (#16)
|
- 2026-01-08 WebP and AVIF encoding support via govips (both former P0 image
|
||||||
- 2026-01-08 WebP and AVIF encoding support via govips (both former P0
|
processing items, now done)
|
||||||
image processing items, now done)
|
|
||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
- P2: security
|
- P2: security
|
||||||
- per-IP rate limiting on the image routes
|
|
||||||
- per-origin rate limiting
|
- per-origin rate limiting
|
||||||
- P2: HTTP response handling
|
- P2: HTTP response handling
|
||||||
- Last-Modified headers
|
- Last-Modified headers
|
||||||
- Vary header for content negotiation
|
|
||||||
- P2: auto format selection (format=auto based on Accept header)
|
|
||||||
- P2: configuration
|
- P2: configuration
|
||||||
- YAML config file support
|
- YAML config file support
|
||||||
- P2: operational
|
- P2: operational
|
||||||
- optional Sentry error reporting
|
- optional Sentry error reporting
|
||||||
- comprehensive request logging
|
- comprehensive request logging
|
||||||
- Prometheus performance metrics
|
- Prometheus performance metrics
|
||||||
- integration tests for the image proxy flow
|
- measure the 1k to 5k req/s target with `script/loadtest` on a machine not
|
||||||
- load tests to verify the 1k to 5k req/s target
|
shared with other work
|
||||||
- P2: documentation
|
|
||||||
- deployment guide
|
|
||||||
- example nginx or caddy reverse proxy config
|
|
||||||
|
|||||||
@@ -0,0 +1,9 @@
|
|||||||
|
// Command loadtest-origin is the upstream host script/loadtest points pixad
|
||||||
|
// at; internal/loadtestorigin says what it does.
|
||||||
|
package main
|
||||||
|
|
||||||
|
import "sneak.berlin/go/pixa/internal/loadtestorigin"
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
loadtestorigin.Run()
|
||||||
|
}
|
||||||
+2
-61
@@ -1,69 +1,10 @@
|
|||||||
// Package main is the entry point for the pixad image proxy server.
|
// Package main is the entry point for the pixad image proxy server.
|
||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import "sneak.berlin/go/pixa/internal/app"
|
||||||
"fmt"
|
|
||||||
"os"
|
|
||||||
"os/signal"
|
|
||||||
"syscall"
|
|
||||||
|
|
||||||
"github.com/spf13/cobra"
|
|
||||||
"go.uber.org/fx"
|
|
||||||
"sneak.berlin/go/pixa/internal/config"
|
|
||||||
"sneak.berlin/go/pixa/internal/database"
|
|
||||||
"sneak.berlin/go/pixa/internal/globals"
|
|
||||||
"sneak.berlin/go/pixa/internal/handlers"
|
|
||||||
"sneak.berlin/go/pixa/internal/healthcheck"
|
|
||||||
"sneak.berlin/go/pixa/internal/logger"
|
|
||||||
"sneak.berlin/go/pixa/internal/middleware"
|
|
||||||
"sneak.berlin/go/pixa/internal/server"
|
|
||||||
)
|
|
||||||
|
|
||||||
var Version string //nolint:gochecknoglobals // set by ldflags
|
var Version string //nolint:gochecknoglobals // set by ldflags
|
||||||
|
|
||||||
var configPath string //nolint:gochecknoglobals // cobra flag
|
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
rootCmd := &cobra.Command{
|
app.Run(Version)
|
||||||
Use: "pixad",
|
|
||||||
Short: "Pixa image caching proxy server",
|
|
||||||
Run: run,
|
|
||||||
}
|
|
||||||
|
|
||||||
rootCmd.Flags().StringVarP(&configPath, "config", "c", "", "path to config file")
|
|
||||||
|
|
||||||
err := rootCmd.Execute()
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintln(os.Stderr, err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func run(_ *cobra.Command, _ []string) {
|
|
||||||
globals.Version = Version
|
|
||||||
|
|
||||||
// Set config path in environment if specified via flag
|
|
||||||
if configPath != "" {
|
|
||||||
_ = os.Setenv("PIXA_CONFIG_PATH", configPath)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A write to a closed stdout or stderr must not end the process.
|
|
||||||
signal.Ignore(syscall.SIGPIPE)
|
|
||||||
|
|
||||||
fx.New(
|
|
||||||
fx.Provide(
|
|
||||||
config.New,
|
|
||||||
database.New,
|
|
||||||
globals.New,
|
|
||||||
handlers.New,
|
|
||||||
logger.New,
|
|
||||||
server.New,
|
|
||||||
middleware.New,
|
|
||||||
healthcheck.New,
|
|
||||||
),
|
|
||||||
fx.Invoke(
|
|
||||||
func(log *logger.Logger) { log.Identify() },
|
|
||||||
func(*server.Server) {},
|
|
||||||
),
|
|
||||||
).Run()
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# Example Caddy config for running pixa behind Caddy; see "Deployment" in
|
||||||
|
# README.md. Replace images.example.com with pixa's public host name, and
|
||||||
|
# 127.0.0.1:8080 with the address Caddy reaches pixa on.
|
||||||
|
#
|
||||||
|
# Caddy gets and renews the TLS certificate for the host name, passes the
|
||||||
|
# Host, Origin and Referer headers on unchanged, sets X-Forwarded-For to the
|
||||||
|
# client's address, and waits for pixa's answer with no time limit of its
|
||||||
|
# own, so pixa's downstream_timeout is what ends a slow request.
|
||||||
|
|
||||||
|
images.example.com
|
||||||
|
|
||||||
|
# pixa asks for metrics.username and metrics.password on /metrics. This
|
||||||
|
# line also keeps it off the public address, for a scraper that reaches
|
||||||
|
# pixa directly; remove it to read /metrics through Caddy.
|
||||||
|
respond /metrics 404
|
||||||
|
|
||||||
|
reverse_proxy 127.0.0.1:8080
|
||||||
@@ -34,9 +34,10 @@ maintenance_mode: false
|
|||||||
state_dir: ./data
|
state_dir: ./data
|
||||||
|
|
||||||
# SQLite database URL (default:
|
# SQLite database URL (default:
|
||||||
# file:<state_dir>/state.sqlite3?_journal_mode=WAL). An empty value aborts
|
# file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)). pixa adds
|
||||||
# startup; leave the key out to use the default.
|
# _pragma=busy_timeout(5000) to it. An empty value aborts startup; leave the
|
||||||
# db_url: "file:./data/state.sqlite3?_journal_mode=WAL"
|
# key out to use the default.
|
||||||
|
# db_url: "file:./data/state.sqlite3?_pragma=journal_mode(WAL)"
|
||||||
|
|
||||||
# Image proxy settings
|
# Image proxy settings
|
||||||
# HMAC signing key for URL signatures (required, at least 32 characters)
|
# HMAC signing key for URL signatures (required, at least 32 characters)
|
||||||
@@ -45,6 +46,8 @@ signing_key: "CHANGE_ME_generate_with_openssl_rand_base64_32"
|
|||||||
|
|
||||||
# Hosts that don't require signatures (default: none)
|
# Hosts that don't require signatures (default: none)
|
||||||
# Use "." prefix for wildcard subdomain matching (e.g., ".example.com" matches "cdn.example.com")
|
# Use "." prefix for wildcard subdomain matching (e.g., ".example.com" matches "cdn.example.com")
|
||||||
|
# An entry that is neither a host name nor an IP address (IPv6 without
|
||||||
|
# brackets), such as one with a port or a "*." wildcard, aborts startup.
|
||||||
allowlist_hosts:
|
allowlist_hosts:
|
||||||
- s3.sneak.cloud
|
- s3.sneak.cloud
|
||||||
- static.sneak.cloud
|
- static.sneak.cloud
|
||||||
@@ -52,6 +55,16 @@ allowlist_hosts:
|
|||||||
- github.com
|
- github.com
|
||||||
- user-images.githubusercontent.com
|
- user-images.githubusercontent.com
|
||||||
|
|
||||||
|
# Hosts whose pages may not show pixa's images, written as for
|
||||||
|
# allowlist_hosts. A request to /v1/image/ or /v1/e/ whose Referer header
|
||||||
|
# names one of them is answered 403 before anything is fetched, even when
|
||||||
|
# the image is cached. A request with no Referer, or one that does not
|
||||||
|
# parse, is served, so a site whose pages send no Referer is not stopped.
|
||||||
|
# The login and generator pages are not covered. (default: none)
|
||||||
|
# referer_blocklist:
|
||||||
|
# - leech.example
|
||||||
|
# - .hotlinker.example
|
||||||
|
|
||||||
# Additional CIDR ranges to refuse when fetching upstream, extending the
|
# Additional CIDR ranges to refuse when fetching upstream, extending the
|
||||||
# SSRF protection. These are added to the always-enforced built-in ranges
|
# SSRF protection. These are added to the always-enforced built-in ranges
|
||||||
# (loopback, RFC 1918 private, link-local, CGNAT, benchmark, NAT64, and
|
# (loopback, RFC 1918 private, link-local, CGNAT, benchmark, NAT64, and
|
||||||
@@ -128,8 +141,9 @@ access_control_allow_origin: "*"
|
|||||||
|
|
||||||
# Maximum disk cache size in bytes. Explicit values are used exactly as
|
# Maximum disk cache size in bytes. Explicit values are used exactly as
|
||||||
# given; 0 disables the disk cache entirely (every request fetches and
|
# given; 0 disables the disk cache entirely (every request fetches and
|
||||||
# processes uncached). When omitted, the default is 75% of the free
|
# processes uncached). When omitted, the default is 75% of the sum of
|
||||||
# space on the filesystem containing <state_dir>/cache/ at startup,
|
# the free space on the filesystem containing <state_dir>/cache/ and
|
||||||
|
# the bytes of images the cache already holds, worked out at startup,
|
||||||
# with a minimum of 500 MiB.
|
# with a minimum of 500 MiB.
|
||||||
# cache_max_bytes: 10737418240
|
# cache_max_bytes: 10737418240
|
||||||
|
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
// Package app reads the pixad command line and runs the server.
|
||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"os/signal"
|
||||||
|
"syscall"
|
||||||
|
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
"go.uber.org/fx"
|
||||||
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
|
"sneak.berlin/go/pixa/internal/database"
|
||||||
|
"sneak.berlin/go/pixa/internal/globals"
|
||||||
|
"sneak.berlin/go/pixa/internal/handlers"
|
||||||
|
"sneak.berlin/go/pixa/internal/healthcheck"
|
||||||
|
"sneak.berlin/go/pixa/internal/logger"
|
||||||
|
"sneak.berlin/go/pixa/internal/middleware"
|
||||||
|
"sneak.berlin/go/pixa/internal/server"
|
||||||
|
)
|
||||||
|
|
||||||
|
var configPath string //nolint:gochecknoglobals // cobra flag
|
||||||
|
|
||||||
|
// Run reads the command line and runs the server until it stops, with
|
||||||
|
// version as the version pixad logs and reports. It exits the process
|
||||||
|
// with status 1 when the command line is not valid.
|
||||||
|
func Run(version string) {
|
||||||
|
globals.Version = version
|
||||||
|
|
||||||
|
rootCmd := &cobra.Command{
|
||||||
|
Use: "pixad",
|
||||||
|
Short: "Pixa image caching proxy server",
|
||||||
|
Run: run,
|
||||||
|
}
|
||||||
|
|
||||||
|
rootCmd.Flags().StringVarP(&configPath, "config", "c", "", "path to config file")
|
||||||
|
|
||||||
|
err := rootCmd.Execute()
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func run(_ *cobra.Command, _ []string) {
|
||||||
|
// Set config path in environment if specified via flag
|
||||||
|
if configPath != "" {
|
||||||
|
_ = os.Setenv("PIXA_CONFIG_PATH", configPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A write to a closed stdout or stderr must not end the process.
|
||||||
|
signal.Ignore(syscall.SIGPIPE)
|
||||||
|
|
||||||
|
fx.New(
|
||||||
|
fx.Provide(
|
||||||
|
config.New,
|
||||||
|
database.New,
|
||||||
|
globals.New,
|
||||||
|
handlers.New,
|
||||||
|
logger.New,
|
||||||
|
server.New,
|
||||||
|
middleware.New,
|
||||||
|
healthcheck.New,
|
||||||
|
),
|
||||||
|
fx.Invoke(
|
||||||
|
func(log *logger.Logger) { log.Identify() },
|
||||||
|
func(*server.Server) {},
|
||||||
|
),
|
||||||
|
).Run()
|
||||||
|
}
|
||||||
@@ -1,26 +1,10 @@
|
|||||||
package config
|
package config
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"errors"
|
|
||||||
"log/slog"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Static errors returned by the stub free-space probes below.
|
|
||||||
var (
|
|
||||||
errTestStatfsFailed = errors.New("statfs failed")
|
|
||||||
errTestProbeNotExpected = errors.New("probe must not be called")
|
|
||||||
)
|
|
||||||
|
|
||||||
// discardLogger returns a logger that swallows all output, for tests
|
|
||||||
// that exercise code paths which log.
|
|
||||||
func discardLogger() *slog.Logger {
|
|
||||||
return slog.New(slog.DiscardHandler)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCacheMaxBytesExplicitValueUsedWithoutFloor verifies that an
|
// TestCacheMaxBytesExplicitValueUsedWithoutFloor verifies that an
|
||||||
// explicitly configured cache_max_bytes value is used exactly as
|
// explicitly configured cache_max_bytes value is used exactly as
|
||||||
// given: the 500 MiB floor applies only to the computed default, never
|
// given: the 500 MiB floor applies only to the computed default, never
|
||||||
@@ -151,155 +135,30 @@ func TestCacheMaxBytesInvalidValuesAbortStartup(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestComputeDefaultCacheMaxBytesUses75PercentOfFreeSpace verifies the
|
// TestCacheMaxBytesExplicitIsRecorded verifies that an omitted
|
||||||
// computed default is 75% of the probed free space when that exceeds
|
// cache_max_bytes is recorded as not explicit, so the cache works out
|
||||||
// the floor.
|
// the default when it opens, and that an explicit zero is recorded as
|
||||||
func TestComputeDefaultCacheMaxBytesUses75PercentOfFreeSpace(t *testing.T) {
|
// explicit, so it disables the disk cache instead.
|
||||||
|
func TestCacheMaxBytesExplicitIsRecorded(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
// 4 GiB free -> 3 GiB default.
|
signingKeyLine := "signing_key: " + validTestSigningKey + "\n"
|
||||||
probe := func(string) (uint64, error) { return 4294967296, nil }
|
|
||||||
|
|
||||||
got, err := ComputeDefaultCacheMaxBytes(t.TempDir(), probe)
|
omitted, err := configFromYAML(t, signingKeyLine)
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("ComputeDefaultCacheMaxBytes returned error: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got != 3221225472 {
|
|
||||||
t.Errorf("ComputeDefaultCacheMaxBytes = %d, want 3221225472 (75%% of 4 GiB)",
|
|
||||||
got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestComputeDefaultCacheMaxBytesAppliesFloorToComputedDefault
|
|
||||||
// verifies that when 75% of free space is below 500 MiB, the computed
|
|
||||||
// default is floored at DefaultCacheMaxBytesFloor.
|
|
||||||
func TestComputeDefaultCacheMaxBytesAppliesFloorToComputedDefault(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cases := []struct {
|
|
||||||
name string
|
|
||||||
freeBytes uint64
|
|
||||||
}{
|
|
||||||
{name: "100 MiB free", freeBytes: 104857600},
|
|
||||||
{name: "zero free", freeBytes: 0},
|
|
||||||
{name: "just below floor threshold", freeBytes: 699050665},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range cases {
|
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
probe := func(string) (uint64, error) { return tc.freeBytes, nil }
|
|
||||||
|
|
||||||
got, err := ComputeDefaultCacheMaxBytes(t.TempDir(), probe)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("ComputeDefaultCacheMaxBytes returned error: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got != DefaultCacheMaxBytesFloor {
|
|
||||||
t.Errorf("ComputeDefaultCacheMaxBytes = %d, want floor %d",
|
|
||||||
got, DefaultCacheMaxBytesFloor)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestComputeDefaultCacheMaxBytesPropagatesProbeError verifies that a
|
|
||||||
// failing free-space probe produces an error naming the config key,
|
|
||||||
// instead of a silently wrong default.
|
|
||||||
func TestComputeDefaultCacheMaxBytesPropagatesProbeError(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
probe := func(string) (uint64, error) { return 0, errTestStatfsFailed }
|
|
||||||
|
|
||||||
_, err := ComputeDefaultCacheMaxBytes(t.TempDir(), probe)
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("probe failure must produce an error, got nil")
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Logf("got expected error: %v", err)
|
|
||||||
|
|
||||||
if !strings.Contains(err.Error(), keyCacheMaxBytes) {
|
|
||||||
t.Errorf("error %q does not name the config key cache_max_bytes", err.Error())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestResolveCacheMaxBytesComputesDefaultWhenOmitted verifies that an
|
|
||||||
// omitted cache_max_bytes key resolves to the computed default, that
|
|
||||||
// the probe is pointed at <state_dir>/cache/ (which must be created
|
|
||||||
// first so statfs measures the right filesystem), and that the result
|
|
||||||
// lands on the Config.
|
|
||||||
func TestResolveCacheMaxBytesComputesDefaultWhenOmitted(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
c, err := configFromYAML(t, "signing_key: "+validTestSigningKey+"\n")
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("minimal config should be valid, got error: %v", err)
|
t.Fatalf("minimal config should be valid, got error: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
c.StateDir = t.TempDir()
|
if omitted.CacheMaxBytesExplicit {
|
||||||
wantCacheDir := filepath.Join(c.StateDir, "cache")
|
t.Error("omitted cache_max_bytes recorded as explicit")
|
||||||
|
|
||||||
var probedPath string
|
|
||||||
|
|
||||||
// 4 GiB free -> 3 GiB default.
|
|
||||||
probe := func(path string) (uint64, error) {
|
|
||||||
probedPath = path
|
|
||||||
|
|
||||||
return 4294967296, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
err = c.resolveCacheMaxBytes(discardLogger(), probe)
|
zero, err := configFromYAML(t, signingKeyLine+"cache_max_bytes: 0\n")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("resolveCacheMaxBytes returned error: %v", err)
|
t.Fatalf("cache_max_bytes: 0 must be accepted, got error: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if c.CacheMaxBytes != 3221225472 {
|
if !zero.CacheMaxBytesExplicit {
|
||||||
t.Errorf("CacheMaxBytes = %d, want computed default 3221225472",
|
t.Error("cache_max_bytes: 0 not recorded as explicit")
|
||||||
c.CacheMaxBytes)
|
|
||||||
}
|
|
||||||
|
|
||||||
if probedPath != wantCacheDir {
|
|
||||||
t.Errorf("free space probed at %q, want cache directory %q",
|
|
||||||
probedPath, wantCacheDir)
|
|
||||||
}
|
|
||||||
|
|
||||||
info, err := os.Stat(wantCacheDir)
|
|
||||||
if err != nil || !info.IsDir() {
|
|
||||||
t.Errorf("cache directory %q was not created before probing: info=%v err=%v",
|
|
||||||
wantCacheDir, info, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestResolveCacheMaxBytesDoesNotOverrideExplicitValue verifies that
|
|
||||||
// an explicitly configured value survives resolution untouched and
|
|
||||||
// that the free-space probe is never consulted for it.
|
|
||||||
func TestResolveCacheMaxBytesDoesNotOverrideExplicitValue(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
yamlContent := "signing_key: " + validTestSigningKey + "\ncache_max_bytes: 1024\n"
|
|
||||||
|
|
||||||
c, err := configFromYAML(t, yamlContent)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("explicit cache_max_bytes must be accepted, got error: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
c.StateDir = t.TempDir()
|
|
||||||
|
|
||||||
probe := func(string) (uint64, error) {
|
|
||||||
t.Error("free-space probe must not be consulted for explicit values")
|
|
||||||
|
|
||||||
return 0, errTestProbeNotExpected
|
|
||||||
}
|
|
||||||
|
|
||||||
err = c.resolveCacheMaxBytes(discardLogger(), probe)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("resolveCacheMaxBytes returned error: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if c.CacheMaxBytes != 1024 {
|
|
||||||
t.Errorf("CacheMaxBytes = %d, want explicit 1024 (no floor, no recompute)",
|
|
||||||
c.CacheMaxBytes)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,116 +0,0 @@
|
|||||||
package config
|
|
||||||
|
|
||||||
import (
|
|
||||||
"fmt"
|
|
||||||
"log/slog"
|
|
||||||
"math"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"syscall"
|
|
||||||
)
|
|
||||||
|
|
||||||
// DefaultCacheMaxBytesFloor is the minimum computed default for the
|
|
||||||
// cache_max_bytes setting: 500 MiB. The floor applies only to the
|
|
||||||
// computed default (when the key is omitted from the configuration),
|
|
||||||
// never to explicitly configured values.
|
|
||||||
const DefaultCacheMaxBytesFloor int64 = 524288000
|
|
||||||
|
|
||||||
// cacheDirPerms is the permission mode for the cache directory created
|
|
||||||
// before probing free space, matching the state directory permissions.
|
|
||||||
const cacheDirPerms = 0o750
|
|
||||||
|
|
||||||
// freeSpaceFractionNumerator and freeSpaceFractionDenominator express
|
|
||||||
// the 75% share of free space used for the computed default limit as
|
|
||||||
// integer arithmetic (dividing before multiplying avoids overflow).
|
|
||||||
const (
|
|
||||||
freeSpaceFractionNumerator uint64 = 3
|
|
||||||
freeSpaceFractionDenominator uint64 = 4
|
|
||||||
)
|
|
||||||
|
|
||||||
// FreeSpaceProbeFunc reports the number of free bytes available on the
|
|
||||||
// filesystem containing path. It is a function type so tests can
|
|
||||||
// inject a fake probe instead of depending on the host disk.
|
|
||||||
type FreeSpaceProbeFunc func(path string) (uint64, error)
|
|
||||||
|
|
||||||
// defaultFreeSpaceProbe reports free filesystem bytes via statfs on
|
|
||||||
// the given path, as available to unprivileged processes.
|
|
||||||
func defaultFreeSpaceProbe(path string) (uint64, error) {
|
|
||||||
var stat syscall.Statfs_t
|
|
||||||
|
|
||||||
err := syscall.Statfs(path, &stat)
|
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
|
|
||||||
if stat.Bsize < 0 {
|
|
||||||
return 0, fmt.Errorf("%w %d for %q", errNegativeBlockSize, stat.Bsize, path)
|
|
||||||
}
|
|
||||||
|
|
||||||
blockSize := uint64(stat.Bsize)
|
|
||||||
|
|
||||||
return stat.Bavail * blockSize, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// ComputeDefaultCacheMaxBytes returns the default cache size limit for
|
|
||||||
// the filesystem containing cacheDir: 75% of the free bytes reported
|
|
||||||
// by probe, with a floor of DefaultCacheMaxBytesFloor.
|
|
||||||
func ComputeDefaultCacheMaxBytes(
|
|
||||||
cacheDir string, probe FreeSpaceProbeFunc,
|
|
||||||
) (int64, error) {
|
|
||||||
freeBytes, err := probe(cacheDir)
|
|
||||||
if err != nil {
|
|
||||||
return 0, fmt.Errorf("config key %q: cannot determine free space for %q: %w",
|
|
||||||
"cache_max_bytes", cacheDir, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
computed := freeBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
|
|
||||||
computed = min(computed, math.MaxInt64)
|
|
||||||
|
|
||||||
// gosec cannot see that min() above bounds computed, so it reads
|
|
||||||
// this conversion as potentially overflowing. It cannot: computed is
|
|
||||||
// at most math.MaxInt64 on every path here.
|
|
||||||
//nolint:gosec // G115: clamped to MaxInt64 by min above
|
|
||||||
limit := int64(computed)
|
|
||||||
limit = max(limit, DefaultCacheMaxBytesFloor)
|
|
||||||
|
|
||||||
return limit, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// resolveCacheMaxBytes finalizes CacheMaxBytes after state_dir
|
|
||||||
// validation: an explicitly configured value is kept as-is (no floor
|
|
||||||
// applies), while an omitted key receives the computed default based
|
|
||||||
// on free space in <state_dir>/cache/. The cache directory is created
|
|
||||||
// first so statfs measures the filesystem that will actually hold the
|
|
||||||
// cache. The effective limit is logged either way.
|
|
||||||
func (c *Config) resolveCacheMaxBytes(
|
|
||||||
log *slog.Logger, probe FreeSpaceProbeFunc,
|
|
||||||
) error {
|
|
||||||
if !c.cacheMaxBytesExplicit {
|
|
||||||
cacheDir := filepath.Join(c.StateDir, "cache")
|
|
||||||
|
|
||||||
err := os.MkdirAll(cacheDir, cacheDirPerms)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("config key %q: cannot create cache directory %q: %w",
|
|
||||||
keyCacheMaxBytes, cacheDir, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
limit, err := ComputeDefaultCacheMaxBytes(cacheDir, probe)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
c.CacheMaxBytes = limit
|
|
||||||
|
|
||||||
log.Info("computed default cache size limit from free space",
|
|
||||||
"cache_max_bytes", limit,
|
|
||||||
"cache_dir", cacheDir,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
log.Info("effective cache size limit",
|
|
||||||
"cache_max_bytes", c.CacheMaxBytes,
|
|
||||||
"cache_disabled", c.CacheMaxBytes == 0,
|
|
||||||
)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
+119
-56
@@ -4,16 +4,19 @@ package config
|
|||||||
import (
|
import (
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"math"
|
"math"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"net/url"
|
"net/url"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
"runtime"
|
"runtime"
|
||||||
"sort"
|
"sort"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/smartconfig"
|
"git.eeqj.de/sneak/smartconfig"
|
||||||
@@ -46,6 +49,7 @@ const (
|
|||||||
keyMetricsPassword = "metrics.password"
|
keyMetricsPassword = "metrics.password"
|
||||||
keySigningKey = "signing_key"
|
keySigningKey = "signing_key"
|
||||||
keyAllowlistHosts = "allowlist_hosts"
|
keyAllowlistHosts = "allowlist_hosts"
|
||||||
|
keyRefererBlocklist = "referer_blocklist"
|
||||||
keyAllowHTTP = "allow_http"
|
keyAllowHTTP = "allow_http"
|
||||||
keyUpstreamConnectionsPerHost = "upstream_connections_per_host"
|
keyUpstreamConnectionsPerHost = "upstream_connections_per_host"
|
||||||
keyUpstreamConnections = "upstream_connections"
|
keyUpstreamConnections = "upstream_connections"
|
||||||
@@ -60,7 +64,7 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// placeholderSigningKey is the dummy signing_key shipped in
|
// placeholderSigningKey is the dummy signing_key shipped in
|
||||||
// config.example.yml. It is 45 characters, so it passes the length
|
// configs/config.example.yml. It is 45 characters, so it passes the length
|
||||||
// check, but it is public in this repository and must be rejected at
|
// check, but it is public in this repository and must be rejected at
|
||||||
// startup so no deployment ever signs URLs with it.
|
// startup so no deployment ever signs URLs with it.
|
||||||
const placeholderSigningKey = "CHANGE_ME_generate_with_openssl_rand_base64_32"
|
const placeholderSigningKey = "CHANGE_ME_generate_with_openssl_rand_base64_32"
|
||||||
@@ -86,20 +90,17 @@ var (
|
|||||||
errMustBeAtLeastOne = errors.New("must be at least 1")
|
errMustBeAtLeastOne = errors.New("must be at least 1")
|
||||||
errValueTooShort = errors.New("value too short")
|
errValueTooShort = errors.New("value too short")
|
||||||
errPlaceholderKey = errors.New(
|
errPlaceholderKey = errors.New(
|
||||||
"is the placeholder from config.example.yml; " +
|
"is the placeholder from configs/config.example.yml; " +
|
||||||
"generate a real key with: openssl rand -base64 32")
|
"generate a real key with: openssl rand -base64 32")
|
||||||
errMustBeSetTogether = errors.New("must be set together")
|
errMustBeSetTogether = errors.New("must be set together")
|
||||||
errMustNotBeNegative = errors.New("must not be negative")
|
errMustNotBeNegative = errors.New("must not be negative")
|
||||||
errOverflowsInt64 = errors.New("overflows a 64-bit integer")
|
errOverflowsInt64 = errors.New("overflows a 64-bit integer")
|
||||||
errNegativeBlockSize = errors.New(
|
|
||||||
"statfs reported negative block size")
|
|
||||||
errValueNull = errors.New(
|
errValueNull = errors.New(
|
||||||
"value is null; omit the key entirely to use the default")
|
"value is null; omit the key entirely to use the default")
|
||||||
errValuesNull = errors.New(
|
errValuesNull = errors.New(
|
||||||
"value is null; omit a key entirely to use its default")
|
"value is null; omit a key entirely to use its default")
|
||||||
errNotBareHostname = errors.New(
|
errNotAHost = errors.New("must be a host name such as " +
|
||||||
"must be a bare hostname without scheme, path, or whitespace")
|
"cdn.example.com or .example.com, or an IP address")
|
||||||
errNoHostnameLabels = errors.New("contains no hostname labels")
|
|
||||||
errNotADuration = errors.New("not a duration such as 30s or 2m")
|
errNotADuration = errors.New("not a duration such as 30s or 2m")
|
||||||
errMustBePositive = errors.New("must be positive")
|
errMustBePositive = errors.New("must be positive")
|
||||||
errNotAnOrigin = errors.New(
|
errNotAnOrigin = errors.New(
|
||||||
@@ -130,6 +131,10 @@ type Config struct {
|
|||||||
AllowHTTP bool // Allow non-TLS upstream (testing only)
|
AllowHTTP bool // Allow non-TLS upstream (testing only)
|
||||||
UpstreamConnectionsPerHost int // Max concurrent connections per upstream host
|
UpstreamConnectionsPerHost int // Max concurrent connections per upstream host
|
||||||
|
|
||||||
|
// RefererBlocklist holds host patterns, matched as AllowlistHosts is: the
|
||||||
|
// image routes refuse a request whose Referer names a matching host.
|
||||||
|
RefererBlocklist []string
|
||||||
|
|
||||||
// UpstreamConnections is the most concurrent connections to all
|
// UpstreamConnections is the most concurrent connections to all
|
||||||
// upstream hosts together, on top of the per-host limit.
|
// upstream hosts together, on top of the per-host limit.
|
||||||
// MaxConcurrentProcessing is the most images processed at once.
|
// MaxConcurrentProcessing is the most images processed at once.
|
||||||
@@ -169,18 +174,19 @@ type Config struct {
|
|||||||
// address, and an explicit list replaces the default.
|
// address, and an explicit list replaces the default.
|
||||||
TrustedProxies []netip.Prefix
|
TrustedProxies []netip.Prefix
|
||||||
|
|
||||||
// CacheMaxBytes is the disk cache size limit in bytes. Zero
|
// CacheMaxBytes is the disk cache size limit in bytes. Only an
|
||||||
// disables the disk cache entirely. When cache_max_bytes is
|
// explicit zero (CacheMaxBytesExplicit true) disables the disk
|
||||||
// omitted from the configuration, this holds the computed default
|
// cache. Zero with CacheMaxBytesExplicit false means
|
||||||
// (75% of free space on the filesystem containing
|
// cache_max_bytes was omitted, and the cache works out the default
|
||||||
// <state_dir>/cache/, floored at DefaultCacheMaxBytesFloor).
|
// limit when it opens.
|
||||||
CacheMaxBytes int64
|
CacheMaxBytes int64
|
||||||
|
|
||||||
// cacheMaxBytesExplicit records whether cache_max_bytes was
|
// CacheMaxBytesExplicit records whether cache_max_bytes was
|
||||||
// explicitly set, in the environment or the configuration file.
|
// explicitly set, in the environment or the configuration file.
|
||||||
// Explicit values are used exactly as given; only an omitted key
|
// Explicit values are used exactly as given; for an omitted key the
|
||||||
// gets the computed default (and its floor) in resolveCacheMaxBytes.
|
// cache works out the default limit when it opens (see
|
||||||
cacheMaxBytesExplicit bool
|
// imgcache.CacheConfig.UseDefaultMaxBytes).
|
||||||
|
CacheMaxBytesExplicit bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// New creates a new Config instance from the environment and the
|
// New creates a new Config instance from the environment and the
|
||||||
@@ -217,9 +223,13 @@ func New(_ fx.Lifecycle, params Params) (*Config, error) {
|
|||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
err = c.resolveCacheMaxBytes(log, defaultFreeSpaceProbe)
|
// An omitted cache_max_bytes is worked out and logged when the
|
||||||
if err != nil {
|
// cache opens.
|
||||||
return nil, err
|
if c.CacheMaxBytesExplicit {
|
||||||
|
log.Info("effective cache size limit",
|
||||||
|
"cache_max_bytes", c.CacheMaxBytes,
|
||||||
|
"cache_disabled", c.CacheMaxBytes == 0,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
if c.Debug {
|
if c.Debug {
|
||||||
@@ -265,7 +275,6 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
loader := &strictLoader{sc: sc}
|
loader := &strictLoader{sc: sc}
|
||||||
|
|
||||||
c := &Config{
|
c := &Config{
|
||||||
Debug: loader.boolVal(keyDebug, false),
|
Debug: loader.boolVal(keyDebug, false),
|
||||||
MaintenanceMode: loader.boolVal(keyMaintenanceMode, false),
|
MaintenanceMode: loader.boolVal(keyMaintenanceMode, false),
|
||||||
@@ -296,13 +305,14 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
|
|||||||
CacheMaxBytes: loader.int64Val(keyCacheMaxBytes, 0),
|
CacheMaxBytes: loader.int64Val(keyCacheMaxBytes, 0),
|
||||||
BlockedNetworks: blockedNetworks,
|
BlockedNetworks: blockedNetworks,
|
||||||
TrustedProxies: trustedProxies,
|
TrustedProxies: trustedProxies,
|
||||||
|
RefererBlocklist: loader.hostListVal(keyRefererBlocklist),
|
||||||
}
|
}
|
||||||
|
|
||||||
// The computed default for cache_max_bytes needs a validated
|
// The default for an omitted cache_max_bytes is worked out when
|
||||||
// state_dir, so it is resolved later (resolveCacheMaxBytes); here
|
// the cache opens; here we only record whether the operator set
|
||||||
// we only record whether the operator set the key explicitly.
|
// the key explicitly.
|
||||||
if _, present := lookupValue(sc, keyCacheMaxBytes); present {
|
if _, present := lookupValue(sc, keyCacheMaxBytes); present {
|
||||||
c.cacheMaxBytesExplicit = true
|
c.CacheMaxBytesExplicit = true
|
||||||
}
|
}
|
||||||
|
|
||||||
// Build DBURL from StateDir if not explicitly set. The derived URL
|
// Build DBURL from StateDir if not explicitly set. The derived URL
|
||||||
@@ -315,7 +325,8 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
|
|||||||
settingName(keyDBURL), errValueEmpty)
|
settingName(keyDBURL), errValueEmpty)
|
||||||
}
|
}
|
||||||
|
|
||||||
c.DBURL = fmt.Sprintf("file:%s/state.sqlite3?_journal_mode=WAL", c.StateDir)
|
// The driver sets the journal mode only through a _pragma parameter.
|
||||||
|
c.DBURL = fmt.Sprintf("file:%s/state.sqlite3?_pragma=journal_mode(WAL)", c.StateDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
if loader.err != nil {
|
if loader.err != nil {
|
||||||
@@ -414,7 +425,8 @@ func isKnownConfigKey(key string) bool {
|
|||||||
keyUpstreamConnectionsPerHost, keyUpstreamConnections,
|
keyUpstreamConnectionsPerHost, keyUpstreamConnections,
|
||||||
keyMaxConcurrentProcessing, keyCacheMaxBytes, keyBlockedNetworks,
|
keyMaxConcurrentProcessing, keyCacheMaxBytes, keyBlockedNetworks,
|
||||||
keyTrustedProxies, keyAccessControlAllowOrigin, keyUpstreamFetchTimeout,
|
keyTrustedProxies, keyAccessControlAllowOrigin, keyUpstreamFetchTimeout,
|
||||||
keyUpstreamMaxResponseSize, keyDownstreamTimeout, "env":
|
keyUpstreamMaxResponseSize, keyDownstreamTimeout, keyRefererBlocklist,
|
||||||
|
"env":
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -437,6 +449,7 @@ func envVarNames() map[string]string {
|
|||||||
keyMetricsPassword: "PIXA_METRICS_PASSWORD",
|
keyMetricsPassword: "PIXA_METRICS_PASSWORD",
|
||||||
keySigningKey: "PIXA_SIGNING_KEY",
|
keySigningKey: "PIXA_SIGNING_KEY",
|
||||||
keyAllowlistHosts: "PIXA_ALLOWLIST_HOSTS",
|
keyAllowlistHosts: "PIXA_ALLOWLIST_HOSTS",
|
||||||
|
keyRefererBlocklist: "PIXA_REFERER_BLOCKLIST",
|
||||||
keyAllowHTTP: "PIXA_ALLOW_HTTP",
|
keyAllowHTTP: "PIXA_ALLOW_HTTP",
|
||||||
keyUpstreamConnectionsPerHost: "PIXA_UPSTREAM_CONNECTIONS_PER_HOST",
|
keyUpstreamConnectionsPerHost: "PIXA_UPSTREAM_CONNECTIONS_PER_HOST",
|
||||||
keyUpstreamConnections: "PIXA_UPSTREAM_CONNECTIONS",
|
keyUpstreamConnections: "PIXA_UPSTREAM_CONNECTIONS",
|
||||||
@@ -548,8 +561,8 @@ func (c *Config) ensureStateDirWritable() error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// validateSigningKey checks that the signing key is present, long
|
// validateSigningKey checks that the signing key is present, long
|
||||||
// enough, and not the public placeholder from config.example.yml. The
|
// enough, and not the public placeholder from configs/config.example.yml.
|
||||||
// key value itself is never echoed in error messages.
|
// The key value itself is never echoed in error messages.
|
||||||
func (c *Config) validateSigningKey() error {
|
func (c *Config) validateSigningKey() error {
|
||||||
if c.SigningKey == "" {
|
if c.SigningKey == "" {
|
||||||
return fmt.Errorf("%s: %w", settingName(keySigningKey), errValueRequired)
|
return fmt.Errorf("%s: %w", settingName(keySigningKey), errValueRequired)
|
||||||
@@ -606,7 +619,7 @@ func (c *Config) validate() error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
for _, host := range c.AllowlistHosts {
|
for _, host := range c.AllowlistHosts {
|
||||||
err := validateAllowlistHost(host)
|
err := validateHostPattern(keyAllowlistHosts, host)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -729,25 +742,24 @@ func (c *Config) validateConcurrencyLimits() error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// validateAllowlistHost checks that an allowlist_hosts entry is a bare
|
// hostNamePattern matches a host name: letters, digits, hyphens, underscores
|
||||||
// hostname, optionally with a leading dot for suffix matching. URLs,
|
// and dots, optionally after one leading dot.
|
||||||
// paths, and whitespace indicate a misconfigured entry. An entry with
|
var hostNamePattern = regexp.MustCompile(`^\.?[A-Za-z0-9_-][A-Za-z0-9_.-]*$`)
|
||||||
// no hostname labels (such as ".") is rejected: the allowlist matcher
|
|
||||||
// treats a leading dot as a suffix pattern, so a bare "." would match
|
|
||||||
// any upstream host written in FQDN trailing-dot form and effectively
|
|
||||||
// disable URL signing.
|
|
||||||
func validateAllowlistHost(host string) error {
|
|
||||||
if strings.Contains(host, "://") || strings.ContainsAny(host, "/ \t") {
|
|
||||||
return fmt.Errorf("%s: entry %q %w",
|
|
||||||
settingName(keyAllowlistHosts), host, errNotBareHostname)
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Trim(host, ".") == "" {
|
|
||||||
return fmt.Errorf("%s: entry %q %w",
|
|
||||||
settingName(keyAllowlistHosts), host, errNoHostnameLabels)
|
|
||||||
}
|
|
||||||
|
|
||||||
|
// validateHostPattern checks that an entry of the named key, allowlist_hosts
|
||||||
|
// or referer_blocklist, is an IP address or a host name, the host name
|
||||||
|
// optionally with one leading dot for suffix matching. Anything else, such as
|
||||||
|
// a URL, a port or a "*." wildcard, can never match a host name that resolves,
|
||||||
|
// so it is refused.
|
||||||
|
// So is "." alone: the allowlist matcher would match it against any host
|
||||||
|
// written with a trailing dot, which in allowlist_hosts disables URL signing.
|
||||||
|
func validateHostPattern(key, host string) error {
|
||||||
|
_, err := netip.ParseAddr(host)
|
||||||
|
if err == nil || hostNamePattern.MatchString(host) {
|
||||||
return nil
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return fmt.Errorf("%s: entry %q %w", settingName(key), host, errNotAHost)
|
||||||
}
|
}
|
||||||
|
|
||||||
// loadConfigFile loads configuration from the PIXA_CONFIG_PATH env var
|
// loadConfigFile loads configuration from the PIXA_CONFIG_PATH env var
|
||||||
@@ -778,10 +790,19 @@ func loadConfigFile(log *slog.Logger, appName string) (*smartconfig.Config, erro
|
|||||||
for _, path := range configPaths {
|
for _, path := range configPaths {
|
||||||
cleanPath := filepath.Clean(path)
|
cleanPath := filepath.Clean(path)
|
||||||
|
|
||||||
|
// Only a config file that does not exist is skipped, including
|
||||||
|
// one whose path runs through a file, such as under a HOME of
|
||||||
|
// /dev/null. One that cannot be read or does not parse is a
|
||||||
|
// fatal startup error.
|
||||||
_, statErr := os.Stat(cleanPath)
|
_, statErr := os.Stat(cleanPath)
|
||||||
if statErr == nil {
|
if errors.Is(statErr, fs.ErrNotExist) || errors.Is(statErr, syscall.ENOTDIR) {
|
||||||
// A config file that exists but does not parse is a fatal
|
continue
|
||||||
// startup error, never something to skip over.
|
}
|
||||||
|
|
||||||
|
if statErr != nil {
|
||||||
|
return nil, fmt.Errorf("failed to read config file %s: %w", path, statErr)
|
||||||
|
}
|
||||||
|
|
||||||
sc, err := smartconfig.NewFromConfigPath(path)
|
sc, err := smartconfig.NewFromConfigPath(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to parse config file %s: %w", path, err)
|
return nil, fmt.Errorf("failed to parse config file %s: %w", path, err)
|
||||||
@@ -791,7 +812,6 @@ func loadConfigFile(log *slog.Logger, appName string) (*smartconfig.Config, erro
|
|||||||
|
|
||||||
return sc, nil
|
return sc, nil
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|
||||||
return nil, nil //nolint:nilnil // nil config is valid (use defaults)
|
return nil, nil //nolint:nilnil // nil config is valid (use defaults)
|
||||||
}
|
}
|
||||||
@@ -869,6 +889,19 @@ func (l *strictLoader) boolVal(key string, defaultVal bool) bool {
|
|||||||
return val
|
return val
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (l *strictLoader) hostListVal(key string) []string {
|
||||||
|
if l.err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
val, err := parseHostList(l.sc, key)
|
||||||
|
if err != nil {
|
||||||
|
l.err = err
|
||||||
|
}
|
||||||
|
|
||||||
|
return val
|
||||||
|
}
|
||||||
|
|
||||||
// getString returns the string value for key, or defaultVal if the key
|
// getString returns the string value for key, or defaultVal if the key
|
||||||
// is omitted. A present value that is not a string, or is explicitly
|
// is omitted. A present value that is not a string, or is explicitly
|
||||||
// null, is an error.
|
// null, is an error.
|
||||||
@@ -1166,7 +1199,7 @@ func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
|
|||||||
return nil, errNullConfigValue(key)
|
return nil, errNullConfigValue(key)
|
||||||
}
|
}
|
||||||
|
|
||||||
entries, err := cidrListEntries(raw, key)
|
entries, err := listEntries(raw, key)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -1186,11 +1219,41 @@ func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
|
|||||||
return prefixes, nil
|
return prefixes, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// cidrListEntries extracts the raw entries of the named CIDR-list key as
|
// parseHostList parses the value of the named config key into host patterns,
|
||||||
// trimmed, non-empty strings, from either a YAML list of strings or a
|
// or returns nil if the key is omitted. It accepts a YAML list of strings or a
|
||||||
// comma-separated string; an empty string is an empty list, as for
|
// comma-separated string. An explicitly null value, a wrong type, an empty
|
||||||
// allowlist_hosts. Any other shape is a configuration error.
|
// entry, a non-string entry, or an entry validateHostPattern rejects aborts
|
||||||
func cidrListEntries(raw any, key string) ([]string, error) {
|
// startup naming the key and the offending value.
|
||||||
|
func parseHostList(sc *smartconfig.Config, key string) ([]string, error) {
|
||||||
|
raw, ok := lookupValue(sc, key)
|
||||||
|
if !ok {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if raw == nil {
|
||||||
|
return nil, errNullConfigValue(key)
|
||||||
|
}
|
||||||
|
|
||||||
|
entries, err := listEntries(raw, key)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, entry := range entries {
|
||||||
|
err := validateHostPattern(key, entry)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return entries, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// listEntries extracts the raw entries of the named list key as trimmed,
|
||||||
|
// non-empty strings, from either a YAML list of strings or a comma-separated
|
||||||
|
// string; an empty string is an empty list, as for allowlist_hosts. Any other
|
||||||
|
// shape is a configuration error.
|
||||||
|
func listEntries(raw any, key string) ([]string, error) {
|
||||||
switch val := raw.(type) {
|
switch val := raw.(type) {
|
||||||
case []any:
|
case []any:
|
||||||
entries := make([]string, 0, len(val))
|
entries := make([]string, 0, len(val))
|
||||||
|
|||||||
@@ -1,14 +1,18 @@
|
|||||||
package config
|
package config
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"database/sql"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.eeqj.de/sneak/smartconfig"
|
"git.eeqj.de/sneak/smartconfig"
|
||||||
|
|
||||||
|
_ "modernc.org/sqlite" // SQLite driver registration
|
||||||
)
|
)
|
||||||
|
|
||||||
// validTestSigningKey is a 32-character signing key that satisfies the
|
// validTestSigningKey is a 32-character signing key that satisfies the
|
||||||
@@ -94,12 +98,43 @@ func TestOmittedValuesUseDefaults(t *testing.T) {
|
|||||||
t.Errorf("AllowlistHosts = %v, want empty", c.AllowlistHosts)
|
t.Errorf("AllowlistHosts = %v, want empty", c.AllowlistHosts)
|
||||||
}
|
}
|
||||||
|
|
||||||
wantDBURL := "file:" + DefaultStateDir + "/state.sqlite3?_journal_mode=WAL"
|
wantDBURL := "file:" + DefaultStateDir +
|
||||||
|
"/state.sqlite3?_pragma=journal_mode(WAL)"
|
||||||
if c.DBURL != wantDBURL {
|
if c.DBURL != wantDBURL {
|
||||||
t.Errorf("DBURL = %q, want derived default %q", c.DBURL, wantDBURL)
|
t.Errorf("DBURL = %q, want derived default %q", c.DBURL, wantDBURL)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestDefaultDBURLOpensTheDatabaseInWALMode opens the db_url derived from
|
||||||
|
// state_dir with the SQLite driver pixad uses and checks that the database
|
||||||
|
// is in WAL mode: the driver ignores any parameter it does not know.
|
||||||
|
func TestDefaultDBURLOpensTheDatabaseInWALMode(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
c, err := configFromYAML(t, signingKeyLine+"state_dir: "+t.TempDir()+"\n")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("config with only state_dir set should be valid, got: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
db, err := sql.Open("sqlite", c.DBURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to open %q: %v", c.DBURL, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
|
||||||
|
var journalMode string
|
||||||
|
|
||||||
|
err = db.QueryRowContext(t.Context(), "PRAGMA journal_mode").Scan(&journalMode)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to read the journal mode of %q: %v", c.DBURL, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if journalMode != "wal" {
|
||||||
|
t.Errorf("journal mode of %q = %q, want wal", c.DBURL, journalMode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExplicitValidValuesAreUsed(t *testing.T) {
|
func TestExplicitValidValuesAreUsed(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -182,6 +217,25 @@ func TestCommaSeparatedAllowlistStillSupported(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestAllowlistHostsAcceptsUnderscore checks that an upstream host name with
|
||||||
|
// an underscore, which pixa can fetch from, is accepted as an entry.
|
||||||
|
func TestAllowlistHostsAcceptsUnderscore(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
c, err := configFromYAML(t, signingKeyLine+`allowlist_hosts:
|
||||||
|
- my_bucket.example.com
|
||||||
|
- .my_bucket.example.org
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("host names with an underscore should load, got error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []string{"my_bucket.example.com", ".my_bucket.example.org"}
|
||||||
|
if !slices.Equal(c.AllowlistHosts, want) {
|
||||||
|
t.Errorf("AllowlistHosts = %v, want %v", c.AllowlistHosts, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// runAbortCases asserts that each case's config aborts startup with an
|
// runAbortCases asserts that each case's config aborts startup with an
|
||||||
// error message mentioning every expected substring.
|
// error message mentioning every expected substring.
|
||||||
func runAbortCases(t *testing.T, cases []abortCase) {
|
func runAbortCases(t *testing.T, cases []abortCase) {
|
||||||
@@ -284,6 +338,20 @@ func invalidHostAndCredentialCases() []abortCase {
|
|||||||
keyAllowlistHosts, "example.com/images",
|
keyAllowlistHosts, "example.com/images",
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "allowlist host with wildcard",
|
||||||
|
yaml: signingKeyLine + "allowlist_hosts:\n - \"*.example.com\"\n",
|
||||||
|
wantErrSubstrings: []string{
|
||||||
|
keyAllowlistHosts, "*.example.com",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "allowlist host with port",
|
||||||
|
yaml: signingKeyLine + "allowlist_hosts:\n - example.com:8443\n",
|
||||||
|
wantErrSubstrings: []string{
|
||||||
|
keyAllowlistHosts, "example.com:8443",
|
||||||
|
},
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: "allowlist host with whitespace",
|
name: "allowlist host with whitespace",
|
||||||
yaml: signingKeyLine + "allowlist_hosts:\n - \"exa mple.com\"\n",
|
yaml: signingKeyLine + "allowlist_hosts:\n - \"exa mple.com\"\n",
|
||||||
@@ -564,6 +632,116 @@ func TestMalformedConfigFileAbortsStartup(t *testing.T) {
|
|||||||
t.Logf("got expected error: %v", err)
|
t.Logf("got expected error: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestConfigFileInDirectoryPixaMayNotEnterAbortsStartup checks that a
|
||||||
|
// config file pixa cannot read because it may not enter its directory
|
||||||
|
// aborts startup instead of being passed over.
|
||||||
|
func TestConfigFileInDirectoryPixaMayNotEnterAbortsStartup(t *testing.T) {
|
||||||
|
if os.Geteuid() == 0 {
|
||||||
|
t.Skip("root may enter any directory")
|
||||||
|
}
|
||||||
|
|
||||||
|
home := t.TempDir()
|
||||||
|
configDir := filepath.Join(home, ".config", "pixa-test-nonexistent-app")
|
||||||
|
configPath := filepath.Join(configDir, "config.yml")
|
||||||
|
|
||||||
|
err := os.MkdirAll(configDir, 0o700)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create config directory: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = os.WriteFile(configPath, []byte(signingKeyLine), 0o600)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to write config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = os.Chmod(configDir, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to remove the config directory's permissions: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Give the directory back its permissions so t.TempDir can remove it.
|
||||||
|
t.Cleanup(func() {
|
||||||
|
//nolint:gosec // G302: a directory needs its execute bit to be removed
|
||||||
|
_ = os.Chmod(configDir, 0o700)
|
||||||
|
})
|
||||||
|
|
||||||
|
// The ~/.config candidate is the only one that exists: the appname
|
||||||
|
// rules out /etc, and the working directory is empty.
|
||||||
|
t.Setenv("PIXA_CONFIG_PATH", "")
|
||||||
|
t.Setenv("HOME", home)
|
||||||
|
t.Chdir(t.TempDir())
|
||||||
|
|
||||||
|
log := slog.New(slog.DiscardHandler)
|
||||||
|
|
||||||
|
sc, err := loadConfigFile(log, "pixa-test-nonexistent-app")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("config file pixa cannot read must abort startup, got config: %v",
|
||||||
|
sc)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("got expected error: %v", err)
|
||||||
|
|
||||||
|
if !strings.Contains(err.Error(), configPath) {
|
||||||
|
t.Errorf("error %q does not name the config file %s", err.Error(), configPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestConfigFileLinkingToItselfAbortsStartup checks that a config file
|
||||||
|
// pixa cannot read for a reason other than not existing aborts startup,
|
||||||
|
// as root too: a symbolic link to itself fails with "too many levels of
|
||||||
|
// symbolic links".
|
||||||
|
func TestConfigFileLinkingToItselfAbortsStartup(t *testing.T) {
|
||||||
|
workDir := t.TempDir()
|
||||||
|
|
||||||
|
err := os.Symlink("config.yml", filepath.Join(workDir, "config.yml"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create symbolic link: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Only the working directory's config.yml is there: the appname rules
|
||||||
|
// out /etc, and HOME is empty.
|
||||||
|
t.Setenv("PIXA_CONFIG_PATH", "")
|
||||||
|
t.Setenv("HOME", t.TempDir())
|
||||||
|
t.Chdir(workDir)
|
||||||
|
|
||||||
|
log := slog.New(slog.DiscardHandler)
|
||||||
|
|
||||||
|
sc, err := loadConfigFile(log, "pixa-test-nonexistent-app")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("config file pixa cannot read must abort startup, got config: %v",
|
||||||
|
sc)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("got expected error: %v", err)
|
||||||
|
|
||||||
|
if !strings.Contains(err.Error(), "config.yml") {
|
||||||
|
t.Errorf("error %q does not name the config file config.yml", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestConfigPathThroughFileIsPassedOver checks that a config file path
|
||||||
|
// that runs through a file, such as one under a HOME of /dev/null, is
|
||||||
|
// passed over like one that does not exist, since no file can be there.
|
||||||
|
func TestConfigPathThroughFileIsPassedOver(t *testing.T) {
|
||||||
|
// No config file is there: the appname rules out /etc, HOME is
|
||||||
|
// /dev/null, and the working directory is empty.
|
||||||
|
t.Setenv("PIXA_CONFIG_PATH", "")
|
||||||
|
t.Setenv("HOME", os.DevNull)
|
||||||
|
t.Chdir(t.TempDir())
|
||||||
|
|
||||||
|
log := slog.New(slog.DiscardHandler)
|
||||||
|
|
||||||
|
sc, err := loadConfigFile(log, "pixa-test-nonexistent-app")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("a config path through a file must be passed over, got error: %v",
|
||||||
|
err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if sc != nil {
|
||||||
|
t.Errorf("expected no config file, got config: %v", sc)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestEnsureStateDirCreatesDirectory(t *testing.T) {
|
func TestEnsureStateDirCreatesDirectory(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -65,6 +65,7 @@ func TestEnvironmentSetsEveryKey(t *testing.T) {
|
|||||||
t.Setenv("PIXA_METRICS_PASSWORD", "metricspass")
|
t.Setenv("PIXA_METRICS_PASSWORD", "metricspass")
|
||||||
t.Setenv("PIXA_SIGNING_KEY", validTestSigningKey)
|
t.Setenv("PIXA_SIGNING_KEY", validTestSigningKey)
|
||||||
t.Setenv("PIXA_ALLOWLIST_HOSTS", "s3.sneak.cloud,.example.com")
|
t.Setenv("PIXA_ALLOWLIST_HOSTS", "s3.sneak.cloud,.example.com")
|
||||||
|
t.Setenv("PIXA_REFERER_BLOCKLIST", "hotlinker.example,.leech.example")
|
||||||
t.Setenv("PIXA_ALLOW_HTTP", "true")
|
t.Setenv("PIXA_ALLOW_HTTP", "true")
|
||||||
t.Setenv("PIXA_UPSTREAM_CONNECTIONS_PER_HOST", "5")
|
t.Setenv("PIXA_UPSTREAM_CONNECTIONS_PER_HOST", "5")
|
||||||
t.Setenv("PIXA_UPSTREAM_CONNECTIONS", "10")
|
t.Setenv("PIXA_UPSTREAM_CONNECTIONS", "10")
|
||||||
@@ -93,12 +94,13 @@ func TestEnvironmentSetsEveryKey(t *testing.T) {
|
|||||||
MetricsPassword: "metricspass",
|
MetricsPassword: "metricspass",
|
||||||
SigningKey: validTestSigningKey,
|
SigningKey: validTestSigningKey,
|
||||||
AllowlistHosts: []string{testHostS3, ".example.com"},
|
AllowlistHosts: []string{testHostS3, ".example.com"},
|
||||||
|
RefererBlocklist: []string{"hotlinker.example", ".leech.example"},
|
||||||
AllowHTTP: true,
|
AllowHTTP: true,
|
||||||
UpstreamConnectionsPerHost: 5,
|
UpstreamConnectionsPerHost: 5,
|
||||||
UpstreamConnections: 10,
|
UpstreamConnections: 10,
|
||||||
MaxConcurrentProcessing: 3,
|
MaxConcurrentProcessing: 3,
|
||||||
CacheMaxBytes: 1024,
|
CacheMaxBytes: 1024,
|
||||||
cacheMaxBytesExplicit: true,
|
CacheMaxBytesExplicit: true,
|
||||||
BlockedNetworks: []netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")},
|
BlockedNetworks: []netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")},
|
||||||
TrustedProxies: []netip.Prefix{netip.MustParsePrefix("192.0.2.0/24")},
|
TrustedProxies: []netip.Prefix{netip.MustParsePrefix("192.0.2.0/24")},
|
||||||
AccessControlAllowOrigin: "https://app.example.com",
|
AccessControlAllowOrigin: "https://app.example.com",
|
||||||
|
|||||||
@@ -0,0 +1,166 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"slices"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestRefererBlocklistParsed loads a referer_blocklist with a host and a
|
||||||
|
// pattern starting with "." and checks both are kept in order.
|
||||||
|
func TestRefererBlocklistParsed(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
c, err := configFromYAML(t, signingKeyLine+`referer_blocklist:
|
||||||
|
- leech.example
|
||||||
|
- .hotlinker.example
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("valid referer_blocklist should load, got error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []string{"leech.example", ".hotlinker.example"}
|
||||||
|
if !slices.Equal(c.RefererBlocklist, want) {
|
||||||
|
t.Errorf("RefererBlocklist = %v, want %v", c.RefererBlocklist, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRefererBlocklistAcceptsIPAddresses checks that IPv4 and IPv6 addresses,
|
||||||
|
// the IPv6 one written without brackets, are accepted as entries.
|
||||||
|
func TestRefererBlocklistAcceptsIPAddresses(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
c, err := configFromYAML(t, signingKeyLine+`referer_blocklist:
|
||||||
|
- 192.0.2.7
|
||||||
|
- "2001:db8::7"
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("IP address entries should load, got error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []string{"192.0.2.7", "2001:db8::7"}
|
||||||
|
if !slices.Equal(c.RefererBlocklist, want) {
|
||||||
|
t.Errorf("RefererBlocklist = %v, want %v", c.RefererBlocklist, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRefererBlocklistAcceptsUnderscore checks that a host name with an
|
||||||
|
// underscore, which a page can be served from, is accepted as an entry.
|
||||||
|
func TestRefererBlocklistAcceptsUnderscore(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
c, err := configFromYAML(t, signingKeyLine+`referer_blocklist:
|
||||||
|
- my_site.leech.example
|
||||||
|
- .my_site.hotlinker.example
|
||||||
|
`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("host names with an underscore should load, got error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []string{"my_site.leech.example", ".my_site.hotlinker.example"}
|
||||||
|
if !slices.Equal(c.RefererBlocklist, want) {
|
||||||
|
t.Errorf("RefererBlocklist = %v, want %v", c.RefererBlocklist, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRefererBlocklistOmittedIsEmpty checks that an omitted key blocks no
|
||||||
|
// referer.
|
||||||
|
func TestRefererBlocklistOmittedIsEmpty(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
c, err := configFromYAML(t, signingKeyLine)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("minimal config should be valid, got error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(c.RefererBlocklist) != 0 {
|
||||||
|
t.Errorf("RefererBlocklist = %v, want empty", c.RefererBlocklist)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRefererBlocklistInvalidAbortsStartup checks that an entry that is not a
|
||||||
|
// host, or a value that is not a list of them, aborts startup with an error
|
||||||
|
// naming the key and the entry.
|
||||||
|
func TestRefererBlocklistInvalidAbortsStartup(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
runAbortCases(t, []abortCase{
|
||||||
|
{
|
||||||
|
name: "entry with a scheme",
|
||||||
|
yaml: signingKeyLine + "referer_blocklist:\n - https://leech.example\n",
|
||||||
|
wantErrSubstrings: []string{
|
||||||
|
keyRefererBlocklist, "https://leech.example",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "entry with a path",
|
||||||
|
yaml: signingKeyLine + "referer_blocklist:\n - leech.example/page\n",
|
||||||
|
wantErrSubstrings: []string{
|
||||||
|
keyRefererBlocklist, "leech.example/page",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "wildcard entry",
|
||||||
|
yaml: signingKeyLine + "referer_blocklist:\n - \"*.leech.example\"\n",
|
||||||
|
wantErrSubstrings: []string{
|
||||||
|
keyRefererBlocklist, "*.leech.example",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "entry with a port",
|
||||||
|
yaml: signingKeyLine + "referer_blocklist:\n - leech.example:8080\n",
|
||||||
|
wantErrSubstrings: []string{
|
||||||
|
keyRefererBlocklist, "leech.example:8080",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "two leading dots",
|
||||||
|
yaml: signingKeyLine + "referer_blocklist:\n - ..leech.example\n",
|
||||||
|
wantErrSubstrings: []string{
|
||||||
|
keyRefererBlocklist, "..leech.example",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "dot only",
|
||||||
|
yaml: signingKeyLine + "referer_blocklist:\n - \".\"\n",
|
||||||
|
wantErrSubstrings: []string{keyRefererBlocklist, `"."`},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty entry",
|
||||||
|
yaml: signingKeyLine + "referer_blocklist:\n - \"\"\n",
|
||||||
|
wantErrSubstrings: []string{keyRefererBlocklist},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "entry not a string",
|
||||||
|
yaml: signingKeyLine + "referer_blocklist:\n - 42\n",
|
||||||
|
wantErrSubstrings: []string{keyRefererBlocklist, "42"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "null value",
|
||||||
|
yaml: signingKeyLine + "referer_blocklist:\n",
|
||||||
|
wantErrSubstrings: []string{keyRefererBlocklist, nullValueText},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRefererBlocklistFromEnvironment checks that PIXA_REFERER_BLOCKLIST
|
||||||
|
// takes comma-separated entries, and that an entry in it that is not a host
|
||||||
|
// aborts startup naming the variable and the entry.
|
||||||
|
func TestRefererBlocklistFromEnvironment(t *testing.T) {
|
||||||
|
t.Setenv("PIXA_SIGNING_KEY", validTestSigningKey)
|
||||||
|
t.Setenv("PIXA_REFERER_BLOCKLIST", " leech.example , .hotlinker.example ")
|
||||||
|
|
||||||
|
c, err := newFromSmartConfig(nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("valid PIXA_REFERER_BLOCKLIST should load, got error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []string{"leech.example", ".hotlinker.example"}
|
||||||
|
if !slices.Equal(c.RefererBlocklist, want) {
|
||||||
|
t.Errorf("RefererBlocklist = %v, want %v", c.RefererBlocklist, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Setenv("PIXA_REFERER_BLOCKLIST", "leech.example,https://hotlinker.example")
|
||||||
|
|
||||||
|
_, err = newFromSmartConfig(nil)
|
||||||
|
wantStartupError(t, err, "PIXA_REFERER_BLOCKLIST", "https://hotlinker.example")
|
||||||
|
}
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
package database
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"path/filepath"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestConcurrentWritesAllSucceed opens a database the way pixad does and
|
||||||
|
// writes to it from several goroutines at once, so the writes run on
|
||||||
|
// separate connections, as one request's writes and the background eviction
|
||||||
|
// pass do. Every write must succeed, none failing with "database is locked",
|
||||||
|
// whether or not db_url already has parameters, and the parameters it has
|
||||||
|
// must still apply.
|
||||||
|
func TestConcurrentWritesAllSucceed(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
query string
|
||||||
|
wantJournalMode string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "db_url without parameters",
|
||||||
|
query: "",
|
||||||
|
wantJournalMode: "delete",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "db_url with the WAL parameter",
|
||||||
|
query: "?_pragma=journal_mode(WAL)",
|
||||||
|
wantJournalMode: "wal",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dbURL := "file:" + filepath.Join(t.TempDir(), "state.sqlite3") + tt.query
|
||||||
|
|
||||||
|
d := &Database{
|
||||||
|
log: slog.New(slog.DiscardHandler),
|
||||||
|
config: &config.Config{DBURL: dbURL},
|
||||||
|
}
|
||||||
|
|
||||||
|
err := d.connect(t.Context())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to connect to %q: %v", dbURL, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Cleanup(func() { _ = d.db.Close() })
|
||||||
|
|
||||||
|
writeConcurrently(t, d.db)
|
||||||
|
|
||||||
|
var journalMode string
|
||||||
|
|
||||||
|
err = d.db.QueryRowContext(t.Context(), "PRAGMA journal_mode").
|
||||||
|
Scan(&journalMode)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to read the journal mode: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if journalMode != tt.wantJournalMode {
|
||||||
|
t.Errorf("journal mode = %q, want %q", journalMode, tt.wantJournalMode)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeConcurrently runs writeLikeOneRequest from several goroutines at once
|
||||||
|
// and checks that every write was made.
|
||||||
|
func writeConcurrently(t *testing.T, db *sql.DB) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
const (
|
||||||
|
writers = 4
|
||||||
|
requestsEach = 20
|
||||||
|
totalRequests = writers * requestsEach
|
||||||
|
)
|
||||||
|
|
||||||
|
ctx := t.Context()
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
|
||||||
|
for writer := range writers {
|
||||||
|
wg.Go(func() {
|
||||||
|
for request := range requestsEach {
|
||||||
|
key := fmt.Sprintf("%d-%d", writer, request)
|
||||||
|
|
||||||
|
err := writeLikeOneRequest(ctx, db, key)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("writer %d: %v", writer, err)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
var hits, sources int
|
||||||
|
|
||||||
|
err := db.QueryRowContext(ctx, `
|
||||||
|
SELECT hit_count, (SELECT COUNT(*) FROM source_content)
|
||||||
|
FROM cache_stats WHERE id = 1
|
||||||
|
`).Scan(&hits, &sources)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to count the writes: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if hits != totalRequests || sources != totalRequests {
|
||||||
|
t.Errorf("hit_count = %d and %d source_content rows, want %d of each",
|
||||||
|
hits, sources, totalRequests)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeLikeOneRequest makes the writes one request and the eviction pass
|
||||||
|
// make: it counts a cache hit, stores a source, records a transformed image
|
||||||
|
// and deletes that record again.
|
||||||
|
func writeLikeOneRequest(ctx context.Context, db *sql.DB, key string) error {
|
||||||
|
_, err := db.ExecContext(ctx,
|
||||||
|
`UPDATE cache_stats SET hit_count = hit_count + 1 WHERE id = 1`)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("counting a cache hit: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = db.ExecContext(ctx, `INSERT INTO source_content
|
||||||
|
(content_hash, content_type, size_bytes) VALUES (?, 'image/png', 1)`, key)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("storing a source: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = db.ExecContext(ctx, `INSERT INTO variant_content
|
||||||
|
(cache_key, size_bytes, content_type) VALUES (?, 1, 'image/png')`, key)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("recording a transformed image: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = db.ExecContext(ctx,
|
||||||
|
`DELETE FROM variant_content WHERE cache_key = ?`, key)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("evicting a transformed image: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -243,7 +243,17 @@ func (s *Database) DB() *sql.DB {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Database) connect(ctx context.Context) error {
|
func (s *Database) connect(ctx context.Context) error {
|
||||||
dbURL := s.config.DBURL
|
// Requests and the eviction pass write on separate connections. With
|
||||||
|
// a busy timeout, a write that finds another one in progress waits up
|
||||||
|
// to five seconds for it instead of failing at once with "database is
|
||||||
|
// locked". The driver runs each _pragma parameter on every connection
|
||||||
|
// it opens.
|
||||||
|
separator := "?"
|
||||||
|
if strings.Contains(s.config.DBURL, "?") {
|
||||||
|
separator = "&"
|
||||||
|
}
|
||||||
|
|
||||||
|
dbURL := s.config.DBURL + separator + "_pragma=busy_timeout(5000)"
|
||||||
|
|
||||||
s.log.Info("connecting to database", "url", dbURL)
|
s.log.Info("connecting to database", "url", dbURL)
|
||||||
|
|
||||||
|
|||||||
@@ -7,8 +7,8 @@ import (
|
|||||||
|
|
||||||
const appname = "pixad"
|
const appname = "pixad"
|
||||||
|
|
||||||
// Version is populated from main() via ldflags.
|
// Version is set by app.Run to the version main was built with.
|
||||||
var Version string //nolint:gochecknoglobals // set from main
|
var Version string //nolint:gochecknoglobals // set by app.Run
|
||||||
|
|
||||||
// Globals holds application-wide constants.
|
// Globals holds application-wide constants.
|
||||||
type Globals struct {
|
type Globals struct {
|
||||||
|
|||||||
@@ -371,7 +371,7 @@ func (s *Handlers) buildGeneratedURL(r *http.Request, token, format string) stri
|
|||||||
|
|
||||||
// Determine file extension for the trailing filename
|
// Determine file extension for the trailing filename
|
||||||
ext := format
|
ext := format
|
||||||
if ext == "" || ext == "orig" {
|
if ext == "" || ext == "orig" || ext == "auto" {
|
||||||
ext = "jpg" // Default extension
|
ext = "jpg" // Default extension
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,59 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/url"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
|
"sneak.berlin/go/pixa/internal/session"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestLoginLogLeavesOutSubmittedKey verifies that the log lines for a
|
||||||
|
// failed and for a successful login do not contain the submitted key.
|
||||||
|
func TestLoginLogLeavesOutSubmittedKey(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const wrongKey = "wrong-signing-key-fedcba9876543210"
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
sessMgr, err := session.NewManager(testSigningKey)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("session.NewManager() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
h := &Handlers{
|
||||||
|
log: slog.New(slog.NewJSONHandler(&buf, nil)),
|
||||||
|
config: &config.Config{SigningKey: testSigningKey},
|
||||||
|
sessMgr: sessMgr,
|
||||||
|
}
|
||||||
|
|
||||||
|
submittedKeys := []string{wrongKey, testSigningKey}
|
||||||
|
|
||||||
|
for _, key := range submittedKeys {
|
||||||
|
form := url.Values{loginKeyField: {key}}
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodPost, "/",
|
||||||
|
strings.NewReader(form.Encode()))
|
||||||
|
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||||
|
|
||||||
|
h.handleLoginPost(httptest.NewRecorder(), req)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, msg := range []string{"failed login attempt", "successful login"} {
|
||||||
|
if !strings.Contains(buf.String(), msg) {
|
||||||
|
t.Fatalf("log missing %q; got %q", msg, buf.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, key := range submittedKeys {
|
||||||
|
if strings.Contains(buf.String(), key) {
|
||||||
|
t.Errorf("log contains submitted key %q; got %q", key, buf.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,292 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/url"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/imgcache"
|
||||||
|
"sneak.berlin/go/pixa/internal/session"
|
||||||
|
)
|
||||||
|
|
||||||
|
// formatField is the generator form's format field name.
|
||||||
|
const formatField = "format"
|
||||||
|
|
||||||
|
// Markers telling the login page from the generator page.
|
||||||
|
const (
|
||||||
|
loginForm = `action="/"`
|
||||||
|
loginKeyInput = `name="key"`
|
||||||
|
generatorForm = `action="/generate"`
|
||||||
|
)
|
||||||
|
|
||||||
|
// generatedURLPattern extracts the path of the URL the generator page shows.
|
||||||
|
// The test router runs with debug on, so the URL starts with http, and its
|
||||||
|
// host is httptest's default request host.
|
||||||
|
var generatedURLPattern = regexp.MustCompile(
|
||||||
|
`value="http://example\.com(/v1/e/[^"]+)"`)
|
||||||
|
|
||||||
|
// findSessionCookie returns the session cookie rec sets, or nil if it sets
|
||||||
|
// none.
|
||||||
|
func findSessionCookie(rec *httptest.ResponseRecorder) *http.Cookie {
|
||||||
|
for _, c := range rec.Result().Cookies() {
|
||||||
|
if c.Name == session.CookieName {
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleRoot_NoSession_ShowsLoginForm verifies that GET / without a
|
||||||
|
// login session shows the login form.
|
||||||
|
func TestHandleRoot_NoSession_ShowsLoginForm(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, srv := newCSRFTestRouter(t)
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, "/", nil))
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want %d", rec.Code, http.StatusOK)
|
||||||
|
}
|
||||||
|
|
||||||
|
body := rec.Body.String()
|
||||||
|
if !strings.Contains(body, loginForm) || !strings.Contains(body, loginKeyInput) {
|
||||||
|
t.Errorf("page is not the login form: %s", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoginPost_WrongKey_ShowsErrorWithoutSession verifies that a wrong key
|
||||||
|
// shows the login form again with an error, and sets no session cookie.
|
||||||
|
func TestLoginPost_WrongKey_ShowsErrorWithoutSession(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, srv := newCSRFTestRouter(t)
|
||||||
|
|
||||||
|
cookies, token := csrfCredentials(t, srv, nil)
|
||||||
|
|
||||||
|
rec := postForm(srv, "/", cookies, url.Values{
|
||||||
|
loginKeyField: {"wrong-signing-key-fedcba9876543210"},
|
||||||
|
csrfTokenField: {token},
|
||||||
|
})
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want %d", rec.Code, http.StatusOK)
|
||||||
|
}
|
||||||
|
|
||||||
|
body := rec.Body.String()
|
||||||
|
if !strings.Contains(body, loginForm) || !strings.Contains(body, loginKeyInput) {
|
||||||
|
t.Errorf("page is not the login form: %s", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(body, "Invalid signing key") {
|
||||||
|
t.Error("login form does not show the error")
|
||||||
|
}
|
||||||
|
|
||||||
|
if c := findSessionCookie(rec); c != nil {
|
||||||
|
t.Errorf("wrong key set a session cookie: %s", c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoginPost_RightKey_SetsSessionCookie verifies that the right key answers
|
||||||
|
// 303 to / with a session cookie marked Secure, HttpOnly and SameSite=Strict,
|
||||||
|
// and that GET / with that cookie shows the generator page.
|
||||||
|
func TestLoginPost_RightKey_SetsSessionCookie(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, srv := newCSRFTestRouter(t)
|
||||||
|
|
||||||
|
cookies, token := csrfCredentials(t, srv, nil)
|
||||||
|
|
||||||
|
rec := postForm(srv, "/", cookies, url.Values{
|
||||||
|
loginKeyField: {testSigningKey},
|
||||||
|
csrfTokenField: {token},
|
||||||
|
})
|
||||||
|
|
||||||
|
if rec.Code != http.StatusSeeOther || rec.Header().Get("Location") != "/" {
|
||||||
|
t.Fatalf("status = %d, Location = %q, want %d to /",
|
||||||
|
rec.Code, rec.Header().Get("Location"), http.StatusSeeOther)
|
||||||
|
}
|
||||||
|
|
||||||
|
sessionCookie := findSessionCookie(rec)
|
||||||
|
if sessionCookie == nil {
|
||||||
|
t.Fatal("right key set no session cookie")
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("Set-Cookie: %s", sessionCookie)
|
||||||
|
|
||||||
|
if !sessionCookie.Secure {
|
||||||
|
t.Error("session cookie is not Secure")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !sessionCookie.HttpOnly {
|
||||||
|
t.Error("session cookie is not HttpOnly")
|
||||||
|
}
|
||||||
|
|
||||||
|
if sessionCookie.SameSite != http.SameSiteStrictMode {
|
||||||
|
t.Errorf("session cookie SameSite = %v, want Strict", sessionCookie.SameSite)
|
||||||
|
}
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
|
||||||
|
req.AddCookie(sessionCookie)
|
||||||
|
|
||||||
|
rec = httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK ||
|
||||||
|
!strings.Contains(rec.Body.String(), generatorForm) {
|
||||||
|
t.Errorf("GET / with the session cookie: status = %d, "+
|
||||||
|
"want %d and the generator page", rec.Code, http.StatusOK)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleLogout_ClearsSessionCookie verifies that GET /logout answers 303
|
||||||
|
// to / and replaces the session cookie with an empty one sent with
|
||||||
|
// Max-Age=0, which makes the browser delete it.
|
||||||
|
func TestHandleLogout_ClearsSessionCookie(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h, _ := newCSRFTestRouter(t)
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, "/logout", nil)
|
||||||
|
req.AddCookie(newSessionCookie(t, h))
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
h.HandleLogout().ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusSeeOther || rec.Header().Get("Location") != "/" {
|
||||||
|
t.Fatalf("status = %d, Location = %q, want %d to /",
|
||||||
|
rec.Code, rec.Header().Get("Location"), http.StatusSeeOther)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("Set-Cookie: %s", rec.Header().Get("Set-Cookie"))
|
||||||
|
|
||||||
|
sessionCookie := findSessionCookie(rec)
|
||||||
|
if sessionCookie == nil {
|
||||||
|
t.Fatal("logout did not set the session cookie")
|
||||||
|
}
|
||||||
|
|
||||||
|
if sessionCookie.Value != "" {
|
||||||
|
t.Errorf("session cookie value = %q, want empty", sessionCookie.Value)
|
||||||
|
}
|
||||||
|
|
||||||
|
// net/http reads a Max-Age=0 attribute back as MaxAge -1.
|
||||||
|
if sessionCookie.MaxAge != -1 {
|
||||||
|
t.Errorf("session cookie MaxAge = %d, want -1 (Max-Age=0)",
|
||||||
|
sessionCookie.MaxAge)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGeneratePost_NoSession_RedirectsToLogin verifies that POST /generate
|
||||||
|
// with a valid CSRF token but no login session answers 303 to / and makes no
|
||||||
|
// URL.
|
||||||
|
func TestGeneratePost_NoSession_RedirectsToLogin(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, srv := newCSRFTestRouter(t)
|
||||||
|
|
||||||
|
cookies, token := csrfCredentials(t, srv, nil)
|
||||||
|
|
||||||
|
rec := postForm(srv, "/generate", cookies, url.Values{
|
||||||
|
sourceURLField: {testSourceURL},
|
||||||
|
csrfTokenField: {token},
|
||||||
|
})
|
||||||
|
|
||||||
|
if rec.Code != http.StatusSeeOther || rec.Header().Get("Location") != "/" {
|
||||||
|
t.Fatalf("status = %d, Location = %q, want %d to /",
|
||||||
|
rec.Code, rec.Header().Get("Location"), http.StatusSeeOther)
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.Contains(rec.Body.String(), "/v1/e/") {
|
||||||
|
t.Error("a URL was made without a login session")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGeneratePost_URLServesImage verifies that the URL the generator page
|
||||||
|
// makes is served by /v1/e/. The image route runs on handlers of its own,
|
||||||
|
// made with the same signing key.
|
||||||
|
func TestGeneratePost_URLServesImage(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, imageSrv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||||
|
|
||||||
|
rec := generatePost(t, url.Values{
|
||||||
|
sourceURLField: {"https://" + signedHost + photoPath},
|
||||||
|
widthField: {"50"},
|
||||||
|
heightField: {"50"},
|
||||||
|
formatField: {string(imgcache.FormatJPEG)},
|
||||||
|
})
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("POST /generate status = %d, want %d", rec.Code, http.StatusOK)
|
||||||
|
}
|
||||||
|
|
||||||
|
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
|
||||||
|
if match == nil {
|
||||||
|
t.Fatalf("generator page shows no URL: %s", rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("generated URL path: %s", match[1])
|
||||||
|
|
||||||
|
imageRec := httptest.NewRecorder()
|
||||||
|
imageSrv.ServeHTTP(imageRec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, match[1], nil))
|
||||||
|
|
||||||
|
requireServedPhoto(t, imageRec)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGeneratePost_URLWithTTLExpires verifies that a URL the generator page
|
||||||
|
// makes with a ttl of one second is served by /v1/e/ at once and answers 410
|
||||||
|
// once the ttl has passed.
|
||||||
|
func TestGeneratePost_URLWithTTLExpires(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, imageSrv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||||
|
|
||||||
|
rec := generatePost(t, url.Values{
|
||||||
|
sourceURLField: {"https://" + signedHost + photoPath},
|
||||||
|
widthField: {"50"},
|
||||||
|
heightField: {"50"},
|
||||||
|
formatField: {string(imgcache.FormatJPEG)},
|
||||||
|
ttlField: {"1"},
|
||||||
|
})
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("POST /generate status = %d, want %d", rec.Code, http.StatusOK)
|
||||||
|
}
|
||||||
|
|
||||||
|
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
|
||||||
|
if match == nil {
|
||||||
|
t.Fatalf("generator page shows no URL: %s", rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
imageRec := httptest.NewRecorder()
|
||||||
|
imageSrv.ServeHTTP(imageRec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, match[1], nil))
|
||||||
|
|
||||||
|
requireServedPhoto(t, imageRec)
|
||||||
|
|
||||||
|
// The URL keeps the time it expires in whole seconds and is served
|
||||||
|
// through the whole of that second, so a ttl of one second has passed
|
||||||
|
// for certain two seconds after the URL was made.
|
||||||
|
time.Sleep(2 * time.Second)
|
||||||
|
|
||||||
|
imageRec = httptest.NewRecorder()
|
||||||
|
imageSrv.ServeHTTP(imageRec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, match[1], nil))
|
||||||
|
|
||||||
|
t.Logf("GET %s after the ttl: %d %q", match[1], imageRec.Code, imageRec.Body)
|
||||||
|
|
||||||
|
if imageRec.Code != http.StatusGone {
|
||||||
|
t.Errorf("status after the ttl = %d, want %d",
|
||||||
|
imageRec.Code, http.StatusGone)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,147 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"go.uber.org/fx/fxtest"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
|
"sneak.berlin/go/pixa/internal/database"
|
||||||
|
"sneak.berlin/go/pixa/internal/globals"
|
||||||
|
"sneak.berlin/go/pixa/internal/logger"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestNewCacheConfigFromCacheMaxBytes checks the cache configuration
|
||||||
|
// built from cache_max_bytes: omitted, the cache works out the default
|
||||||
|
// limit; 0 turns the disk cache off; a positive value is the limit,
|
||||||
|
// unchanged.
|
||||||
|
func TestNewCacheConfigFromCacheMaxBytes(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const oneGiB = 1 << 30
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
cacheMaxBytes int64
|
||||||
|
cacheMaxBytesExplicit bool
|
||||||
|
wantMaxBytes int64
|
||||||
|
wantUseDefaultMaxBytes bool
|
||||||
|
wantDisableDiskCache bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "cache_max_bytes omitted",
|
||||||
|
cacheMaxBytes: 0,
|
||||||
|
cacheMaxBytesExplicit: false,
|
||||||
|
wantMaxBytes: 0,
|
||||||
|
wantUseDefaultMaxBytes: true,
|
||||||
|
wantDisableDiskCache: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "cache_max_bytes: 0",
|
||||||
|
cacheMaxBytes: 0,
|
||||||
|
cacheMaxBytesExplicit: true,
|
||||||
|
wantMaxBytes: 0,
|
||||||
|
wantUseDefaultMaxBytes: false,
|
||||||
|
wantDisableDiskCache: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "cache_max_bytes: 1 GiB",
|
||||||
|
cacheMaxBytes: oneGiB,
|
||||||
|
cacheMaxBytesExplicit: true,
|
||||||
|
wantMaxBytes: oneGiB,
|
||||||
|
wantUseDefaultMaxBytes: false,
|
||||||
|
wantDisableDiskCache: false,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cfg := &config.Config{
|
||||||
|
CacheMaxBytes: tc.cacheMaxBytes,
|
||||||
|
CacheMaxBytesExplicit: tc.cacheMaxBytesExplicit,
|
||||||
|
}
|
||||||
|
|
||||||
|
got := newCacheConfig(cfg, nil)
|
||||||
|
t.Logf("MaxBytes = %d, UseDefaultMaxBytes = %v, DisableDiskCache = %v",
|
||||||
|
got.MaxBytes, got.UseDefaultMaxBytes, got.DisableDiskCache)
|
||||||
|
|
||||||
|
if got.MaxBytes != tc.wantMaxBytes {
|
||||||
|
t.Errorf("MaxBytes = %d, want %d", got.MaxBytes, tc.wantMaxBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got.UseDefaultMaxBytes != tc.wantUseDefaultMaxBytes {
|
||||||
|
t.Errorf("UseDefaultMaxBytes = %v, want %v",
|
||||||
|
got.UseDefaultMaxBytes, tc.wantUseDefaultMaxBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got.DisableDiskCache != tc.wantDisableDiskCache {
|
||||||
|
t.Errorf("DisableDiskCache = %v, want %v",
|
||||||
|
got.DisableDiskCache, tc.wantDisableDiskCache)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDiskCacheOffOnlyForExplicitZeroCacheMaxBytes starts the handlers
|
||||||
|
// once with cache_max_bytes omitted and once with cache_max_bytes: 0,
|
||||||
|
// and checks by whether the cache directories were created that the
|
||||||
|
// disk cache is on in the first case and off in the second.
|
||||||
|
func TestDiskCacheOffOnlyForExplicitZeroCacheMaxBytes(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
cacheMaxBytesExplicit bool
|
||||||
|
wantDiskCache bool
|
||||||
|
}{
|
||||||
|
{name: "cache_max_bytes omitted", cacheMaxBytesExplicit: false, wantDiskCache: true},
|
||||||
|
{name: "cache_max_bytes: 0", cacheMaxBytesExplicit: true, wantDiskCache: false},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
stateDir := t.TempDir()
|
||||||
|
cfg := &config.Config{
|
||||||
|
SigningKey: testSigningKey,
|
||||||
|
StateDir: stateDir,
|
||||||
|
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
|
||||||
|
CacheMaxBytes: 0,
|
||||||
|
CacheMaxBytesExplicit: tc.cacheMaxBytesExplicit,
|
||||||
|
}
|
||||||
|
|
||||||
|
lc := fxtest.NewLifecycle(t)
|
||||||
|
|
||||||
|
log, err := logger.New(lc, logger.Params{Globals: &globals.Globals{}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("logger.New() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
db, err := database.New(lc, database.Params{Logger: log, Config: cfg})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("database.New() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = New(lc, Params{Logger: log, Database: db, Config: cfg})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
lc.RequireStart()
|
||||||
|
t.Cleanup(lc.RequireStop)
|
||||||
|
|
||||||
|
_, err = os.Stat(filepath.Join(stateDir, "cache", "variants"))
|
||||||
|
gotDiskCache := err == nil
|
||||||
|
|
||||||
|
if gotDiskCache != tc.wantDiskCache {
|
||||||
|
t.Errorf("cache directories created = %v, want %v",
|
||||||
|
gotDiskCache, tc.wantDiskCache)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/netip"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/go-chi/chi/v5"
|
||||||
|
"go.uber.org/fx"
|
||||||
|
"go.uber.org/fx/fxtest"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
|
"sneak.berlin/go/pixa/internal/database"
|
||||||
|
"sneak.berlin/go/pixa/internal/globals"
|
||||||
|
"sneak.berlin/go/pixa/internal/healthcheck"
|
||||||
|
"sneak.berlin/go/pixa/internal/logger"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestHandlersBuildTheirOwnFetcherWhenNoneIsProvided builds the handlers as
|
||||||
|
// pixad does, in an fx app that provides no fetcher, and requests an image
|
||||||
|
// from 192.0.2.10, which is on the allowlist and in blocked_networks. The URL
|
||||||
|
// check accepts that address; only the dialer that refuses internal
|
||||||
|
// addresses checks blocked_networks, so the answer is 403 only if the
|
||||||
|
// fetcher the handlers build from the config connects with that dialer. Any
|
||||||
|
// other dialer would try to connect until the upstream fetch timeout, which
|
||||||
|
// is short so that the test then fails quickly.
|
||||||
|
func TestHandlersBuildTheirOwnFetcherWhenNoneIsProvided(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const host = "192.0.2.10"
|
||||||
|
|
||||||
|
stateDir := t.TempDir()
|
||||||
|
cfg := &config.Config{
|
||||||
|
SigningKey: testSigningKey,
|
||||||
|
StateDir: stateDir,
|
||||||
|
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
|
||||||
|
AllowlistHosts: []string{host},
|
||||||
|
BlockedNetworks: []netip.Prefix{netip.MustParsePrefix("192.0.2.0/24")},
|
||||||
|
UpstreamFetchTimeout: 2 * time.Second,
|
||||||
|
// With no connection slots, the fetch would fail before dialing.
|
||||||
|
UpstreamConnections: config.DefaultUpstreamConnections,
|
||||||
|
}
|
||||||
|
|
||||||
|
var h *Handlers
|
||||||
|
|
||||||
|
app := fxtest.New(t,
|
||||||
|
fx.Supply(cfg),
|
||||||
|
fx.Provide(globals.New, logger.New, database.New, healthcheck.New, New),
|
||||||
|
fx.Populate(&h),
|
||||||
|
)
|
||||||
|
app.RequireStart()
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
r := chi.NewRouter()
|
||||||
|
r.Get("/v1/image/*", h.HandleImage())
|
||||||
|
|
||||||
|
rec := sendGet(t, r, photoURL(host))
|
||||||
|
checkErrorBody(t, rec, http.StatusForbidden, "forbidden")
|
||||||
|
}
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"mime"
|
||||||
|
"net/http"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/imgcache"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Errors for an Accept header that an auto URL cannot be served for.
|
||||||
|
var (
|
||||||
|
errInvalidAccept = errors.New("invalid Accept header")
|
||||||
|
errNotAcceptable = errors.New(
|
||||||
|
"not acceptable: auto serves image/avif, image/webp or image/jpeg")
|
||||||
|
)
|
||||||
|
|
||||||
|
// chooseAutoFormat replaces the format auto in req with the format
|
||||||
|
// formatForAccept chooses from r's Accept header, and adds Vary: Accept to the
|
||||||
|
// response, which then depends on that header. It answers 400 for an Accept
|
||||||
|
// header that is not valid and 406 for one that allows none of the formats,
|
||||||
|
// and reports whether req can be served. Any other format is left as it is.
|
||||||
|
func (s *Handlers) chooseAutoFormat(
|
||||||
|
w http.ResponseWriter, r *http.Request, req *imgcache.ImageRequest,
|
||||||
|
) bool {
|
||||||
|
if req.Format != imgcache.FormatAuto {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
w.Header().Add("Vary", "Accept")
|
||||||
|
|
||||||
|
format, err := formatForAccept(strings.Join(r.Header.Values("Accept"), ","))
|
||||||
|
if errors.Is(err, errNotAcceptable) {
|
||||||
|
s.respondError(w, err.Error(), http.StatusNotAcceptable)
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
s.respondError(w, err.Error(), http.StatusBadRequest)
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
req.Format = format
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// formatForAccept returns the format an auto URL is served in for the Accept
|
||||||
|
// header accept: AVIF when it names image/avif, else WebP when it names
|
||||||
|
// image/webp, else JPEG when its most specific entry of image/jpeg, image/*
|
||||||
|
// and */* allows it, or when it names nothing. A q of 0 refuses a format.
|
||||||
|
// AVIF and WebP must be named, as clients that cannot show them send image/*
|
||||||
|
// and */* too.
|
||||||
|
func formatForAccept(accept string) (imgcache.ImageFormat, error) {
|
||||||
|
qualities, err := parseAccept(accept)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(qualities) == 0 {
|
||||||
|
return imgcache.FormatJPEG, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if qualities["image/avif"] > 0 {
|
||||||
|
return imgcache.FormatAVIF, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if qualities["image/webp"] > 0 {
|
||||||
|
return imgcache.FormatWebP, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// For JPEG, the most specific entry the header has decides
|
||||||
|
quality, named := qualities["image/jpeg"]
|
||||||
|
if !named {
|
||||||
|
quality, named = qualities["image/*"]
|
||||||
|
}
|
||||||
|
|
||||||
|
if !named {
|
||||||
|
quality = qualities["*/*"]
|
||||||
|
}
|
||||||
|
|
||||||
|
if quality > 0 {
|
||||||
|
return imgcache.FormatJPEG, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return "", errNotAcceptable
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseAccept returns the q of each media range the Accept header accept
|
||||||
|
// names, 1 where it gives none. A media range named more than once keeps its
|
||||||
|
// lowest q, so a refusal is never overridden. A media range that does not
|
||||||
|
// parse, or a q that is not a number from 0 to 1, is an error.
|
||||||
|
func parseAccept(accept string) (map[string]float64, error) {
|
||||||
|
qualities := make(map[string]float64)
|
||||||
|
|
||||||
|
for entry := range strings.SplitSeq(accept, ",") {
|
||||||
|
// A header field list may hold empty entries
|
||||||
|
if strings.TrimSpace(entry) == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
mediaRange, params, err := mime.ParseMediaType(entry)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("%w: %q: %w", errInvalidAccept, entry, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
quality := 1.0
|
||||||
|
|
||||||
|
if qParam, given := params["q"]; given {
|
||||||
|
quality, err = strconv.ParseFloat(qParam, 64)
|
||||||
|
inRange := quality >= 0 && quality <= 1
|
||||||
|
|
||||||
|
if err != nil || !inRange {
|
||||||
|
return nil, fmt.Errorf("%w: %q: q is not a number from 0 to 1",
|
||||||
|
errInvalidAccept, entry)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
previous, named := qualities[mediaRange]
|
||||||
|
if !named || quality < previous {
|
||||||
|
qualities[mediaRange] = quality
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return qualities, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,310 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/go-chi/chi/v5"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/encurl"
|
||||||
|
"sneak.berlin/go/pixa/internal/imgcache"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The content types the tests below expect.
|
||||||
|
const (
|
||||||
|
avifType = "image/avif"
|
||||||
|
webpType = "image/webp"
|
||||||
|
jpegType = "image/jpeg"
|
||||||
|
jsonType = "application/json"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestFormatForAccept verifies the format chosen for the format auto from each
|
||||||
|
// Accept header below, and the error for one that allows none of AVIF, WebP
|
||||||
|
// and JPEG or is not valid.
|
||||||
|
func TestFormatForAccept(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
accept string
|
||||||
|
want imgcache.ImageFormat
|
||||||
|
wantErr error
|
||||||
|
}{
|
||||||
|
{"AVIF-capable browser",
|
||||||
|
"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
|
||||||
|
imgcache.FormatAVIF, nil},
|
||||||
|
{"WebP-capable browser",
|
||||||
|
"image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5",
|
||||||
|
imgcache.FormatWebP, nil},
|
||||||
|
{"WebP only", webpType, imgcache.FormatWebP, nil},
|
||||||
|
{"neither", "image/png,image/*;q=0.8,*/*;q=0.5", imgcache.FormatJPEG, nil},
|
||||||
|
{"wildcard only", "*/*", imgcache.FormatJPEG, nil},
|
||||||
|
{"image wildcard only", "image/*", imgcache.FormatJPEG, nil},
|
||||||
|
{"absent", "", imgcache.FormatJPEG, nil},
|
||||||
|
{"q=0 on AVIF", "image/avif;q=0,image/webp,*/*", imgcache.FormatWebP, nil},
|
||||||
|
{"AVIF named twice, once with q=0", "image/avif,image/avif;q=0.0,*/*",
|
||||||
|
imgcache.FormatJPEG, nil},
|
||||||
|
{"upper case and spaces", " Image/AVIF ; Q=0.5 ", imgcache.FormatAVIF, nil},
|
||||||
|
{"q=0 on JPEG", "image/jpeg;q=0,image/*", "", errNotAcceptable},
|
||||||
|
{"q=0 on everything", "*/*;q=0", "", errNotAcceptable},
|
||||||
|
{"PNG only", "image/png", "", errNotAcceptable},
|
||||||
|
{"malformed media range", "image/", "", errInvalidAccept},
|
||||||
|
{"q not a number", "image/avif;q=high", "", errInvalidAccept},
|
||||||
|
{"q above 1", "image/avif;q=2", "", errInvalidAccept},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
got, err := formatForAccept(tt.accept)
|
||||||
|
t.Logf("Accept %q: %q, %v", tt.accept, got, err)
|
||||||
|
|
||||||
|
if got != tt.want || !errors.Is(err, tt.wantErr) {
|
||||||
|
t.Errorf("formatForAccept(%q) = %q, %v, want %q, %v",
|
||||||
|
tt.accept, got, err, tt.want, tt.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// autoPhotoURLs returns a signed /v1/image/ URL and an encrypted /v1/e/ URL,
|
||||||
|
// both valid for a minute, for the JPEG at photoPath on signedHost at 50x50 in
|
||||||
|
// the format auto, made with h's image service and generator.
|
||||||
|
func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
signedURL, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: photoPath,
|
||||||
|
Size: imgcache.Size{Width: 50, Height: 50},
|
||||||
|
Format: imgcache.FormatAuto,
|
||||||
|
}, time.Minute)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
token, err := h.encGen.Generate(&encurl.Payload{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: photoPath,
|
||||||
|
Width: 50,
|
||||||
|
Height: 50,
|
||||||
|
Format: imgcache.FormatAuto,
|
||||||
|
ExpiresAt: time.Now().Add(time.Minute).Unix(),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Generate() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return signedURL, "/v1/e/" + token + "/img.jpg"
|
||||||
|
}
|
||||||
|
|
||||||
|
// requestImage sends method for target to srv with an Accept header line for
|
||||||
|
// each of accept, and returns the response.
|
||||||
|
func requestImage(
|
||||||
|
t *testing.T, srv http.Handler, method, target string, accept ...string,
|
||||||
|
) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), method, target, nil)
|
||||||
|
|
||||||
|
for _, value := range accept {
|
||||||
|
req.Header.Add("Accept", value)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rec, req)
|
||||||
|
t.Logf("%s %s with Accept %q: %d, Content-Type %s, Vary %v, X-Pixa-Cache %s",
|
||||||
|
method, target, accept, rec.Code, rec.Header().Get("Content-Type"),
|
||||||
|
rec.Header().Values("Vary"), rec.Header().Get("X-Pixa-Cache"))
|
||||||
|
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFormatAuto_ChosenFromAccept requests an auto URL on each image route
|
||||||
|
// with each Accept below, and checks the answer and that it carries
|
||||||
|
// Vary: Accept. The signed URL is signed for auto, so it is valid whatever
|
||||||
|
// Accept chooses.
|
||||||
|
func TestFormatAuto_ChosenFromAccept(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||||
|
signedURL, encryptedURL := autoPhotoURLs(t, h)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
accept []string
|
||||||
|
wantStatus int
|
||||||
|
wantType string
|
||||||
|
}{
|
||||||
|
{"AVIF accepted", []string{"image/avif,image/webp,*/*;q=0.8"},
|
||||||
|
http.StatusOK, avifType},
|
||||||
|
{"WebP accepted", []string{"image/webp,*/*;q=0.8"}, http.StatusOK, webpType},
|
||||||
|
{"no Accept", nil, http.StatusOK, jpegType},
|
||||||
|
{"two Accept lines", []string{"image/png", webpType}, http.StatusOK, webpType},
|
||||||
|
{"none of the three", []string{"image/gif"},
|
||||||
|
http.StatusNotAcceptable, jsonType},
|
||||||
|
{"not valid", []string{"image/avif;q=high"}, http.StatusBadRequest, jsonType},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, target := range []string{signedURL, encryptedURL} {
|
||||||
|
rec := requestImage(t, srv, http.MethodGet, target, tt.accept...)
|
||||||
|
gotType := rec.Header().Get("Content-Type")
|
||||||
|
|
||||||
|
if rec.Code != tt.wantStatus || gotType != tt.wantType {
|
||||||
|
t.Errorf("%s: %d %s, want %d %s; body %s", target,
|
||||||
|
rec.Code, gotType, tt.wantStatus, tt.wantType, rec.Body)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !slices.Contains(rec.Header().Values("Vary"), "Accept") {
|
||||||
|
t.Errorf("%s: Vary = %v, want Accept in it",
|
||||||
|
target, rec.Header().Values("Vary"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFormatAuto_SignatureCoversAuto verifies that a /v1/image/ URL with the
|
||||||
|
// format auto is checked against a signature for auto, not for the format
|
||||||
|
// chosen: a URL signed for avif, with auto put in its path, is refused for a
|
||||||
|
// client whose Accept chooses AVIF.
|
||||||
|
func TestFormatAuto_SignatureCoversAuto(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||||
|
|
||||||
|
signedForAVIF, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: photoPath,
|
||||||
|
Size: imgcache.Size{Width: 50, Height: 50},
|
||||||
|
Format: imgcache.FormatAVIF,
|
||||||
|
}, time.Minute)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
target := strings.Replace(signedForAVIF, "/50x50.avif?", "/50x50.auto?", 1)
|
||||||
|
if target == signedForAVIF {
|
||||||
|
t.Fatalf("no /50x50.avif? in %s", signedForAVIF)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := requestImage(t, srv, http.MethodGet, target, avifType)
|
||||||
|
if rec.Code != http.StatusUnauthorized {
|
||||||
|
t.Errorf("status = %d, want %d", rec.Code, http.StatusUnauthorized)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFormatAuto_CachesEachFormatApart requests an auto URL for AVIF, then
|
||||||
|
// JPEG, then both again. Each format is processed once and then served from
|
||||||
|
// the cache, with an ETag of its own, so a client never gets the other format
|
||||||
|
// from the cache.
|
||||||
|
func TestFormatAuto_CachesEachFormatApart(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||||
|
signedURL, _ := autoPhotoURLs(t, h)
|
||||||
|
|
||||||
|
steps := []struct {
|
||||||
|
wantType string
|
||||||
|
wantCache string
|
||||||
|
}{
|
||||||
|
{avifType, "MISS"},
|
||||||
|
{jpegType, "MISS"},
|
||||||
|
{avifType, "HIT"},
|
||||||
|
{jpegType, "HIT"},
|
||||||
|
}
|
||||||
|
|
||||||
|
etags := make(map[string]string)
|
||||||
|
|
||||||
|
for _, step := range steps {
|
||||||
|
rec := requestImage(t, srv, http.MethodGet, signedURL, step.wantType)
|
||||||
|
gotType := rec.Header().Get("Content-Type")
|
||||||
|
gotCache := rec.Header().Get("X-Pixa-Cache")
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK || gotType != step.wantType ||
|
||||||
|
gotCache != step.wantCache {
|
||||||
|
t.Fatalf("Accept %s: %d %s %s, want 200 %s %s", step.wantType,
|
||||||
|
rec.Code, gotType, gotCache, step.wantType, step.wantCache)
|
||||||
|
}
|
||||||
|
|
||||||
|
etag := rec.Header().Get("ETag")
|
||||||
|
if previous, seen := etags[gotType]; seen && previous != etag {
|
||||||
|
t.Errorf("%s ETag changed from %s to %s", gotType, previous, etag)
|
||||||
|
}
|
||||||
|
|
||||||
|
etags[gotType] = etag
|
||||||
|
}
|
||||||
|
|
||||||
|
if etags[avifType] == etags[jpegType] {
|
||||||
|
t.Errorf("AVIF and JPEG have the same ETag %s", etags[avifType])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFormatAuto_Vary verifies that on each image route a HEAD answer and a
|
||||||
|
// 304 for an auto URL carry Vary: Accept, and that the answer for a URL with
|
||||||
|
// a fixed format does not.
|
||||||
|
func TestFormatAuto_Vary(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h, _ := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||||
|
|
||||||
|
srv := chi.NewRouter()
|
||||||
|
srv.Get("/v1/image/*", h.HandleImage())
|
||||||
|
srv.Head("/v1/image/*", h.HandleImage())
|
||||||
|
srv.Get("/v1/e/{token}/*", h.HandleImageEnc())
|
||||||
|
srv.Head("/v1/e/{token}/*", h.HandleImageEnc())
|
||||||
|
|
||||||
|
signedURL, encryptedURL := autoPhotoURLs(t, h)
|
||||||
|
|
||||||
|
for _, urls := range [][2]string{
|
||||||
|
{signedURL, signedPhotoURL(t, h)},
|
||||||
|
{encryptedURL, encPhotoURL(t, h)},
|
||||||
|
} {
|
||||||
|
autoURL, fixedURL := urls[0], urls[1]
|
||||||
|
|
||||||
|
head := requestImage(t, srv, http.MethodHead, autoURL, webpType)
|
||||||
|
checkVaryAccept(t, head, http.StatusOK, true)
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
|
||||||
|
autoURL, nil)
|
||||||
|
req.Header.Set("Accept", webpType)
|
||||||
|
req.Header.Set("If-None-Match", head.Header().Get("ETag"))
|
||||||
|
|
||||||
|
notModified := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(notModified, req)
|
||||||
|
checkVaryAccept(t, notModified, http.StatusNotModified, true)
|
||||||
|
|
||||||
|
fixed := requestImage(t, srv, http.MethodGet, fixedURL)
|
||||||
|
checkVaryAccept(t, fixed, http.StatusOK, false)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkVaryAccept fails the test unless rec answered wantStatus and, as
|
||||||
|
// wantVary says, has or has not Accept in its Vary header.
|
||||||
|
func checkVaryAccept(
|
||||||
|
t *testing.T, rec *httptest.ResponseRecorder, wantStatus int, wantVary bool,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
vary := rec.Header().Values("Vary")
|
||||||
|
t.Logf("status %d, Vary %v", rec.Code, vary)
|
||||||
|
|
||||||
|
if rec.Code != wantStatus {
|
||||||
|
t.Errorf("status = %d, want %d", rec.Code, wantStatus)
|
||||||
|
}
|
||||||
|
|
||||||
|
if slices.Contains(vary, "Accept") != wantVary {
|
||||||
|
t.Errorf("Vary = %v, want Accept in it: %v", vary, wantVary)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -9,6 +9,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
|
"sneak.berlin/go/pixa/internal/allowlist"
|
||||||
"sneak.berlin/go/pixa/internal/config"
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
"sneak.berlin/go/pixa/internal/database"
|
"sneak.berlin/go/pixa/internal/database"
|
||||||
"sneak.berlin/go/pixa/internal/encurl"
|
"sneak.berlin/go/pixa/internal/encurl"
|
||||||
@@ -27,6 +28,11 @@ type Params struct {
|
|||||||
Healthcheck *healthcheck.Healthcheck
|
Healthcheck *healthcheck.Healthcheck
|
||||||
Database *database.Database
|
Database *database.Database
|
||||||
Config *config.Config
|
Config *config.Config
|
||||||
|
|
||||||
|
// Fetcher, when provided, fetches upstream images in place of the
|
||||||
|
// fetcher the handlers build from the config. Only tests provide one;
|
||||||
|
// pixad does not.
|
||||||
|
Fetcher httpfetcher.Fetcher `optional:"true"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Handlers provides HTTP request handlers.
|
// Handlers provides HTTP request handlers.
|
||||||
@@ -35,11 +41,16 @@ type Handlers struct {
|
|||||||
hc *healthcheck.Healthcheck
|
hc *healthcheck.Healthcheck
|
||||||
db *database.Database
|
db *database.Database
|
||||||
config *config.Config
|
config *config.Config
|
||||||
|
fetcher httpfetcher.Fetcher
|
||||||
imgSvc *imgcache.Service
|
imgSvc *imgcache.Service
|
||||||
imgCache *imgcache.Cache
|
imgCache *imgcache.Cache
|
||||||
sessMgr *session.Manager
|
sessMgr *session.Manager
|
||||||
encGen *encurl.Generator
|
encGen *encurl.Generator
|
||||||
csrfProtect func(http.Handler) http.Handler
|
csrfProtect func(http.Handler) http.Handler
|
||||||
|
|
||||||
|
// refererBlocklist matches the hosts of referer_blocklist; its IsAllowed
|
||||||
|
// reports whether a URL's host is on that list.
|
||||||
|
refererBlocklist *allowlist.HostAllowList
|
||||||
}
|
}
|
||||||
|
|
||||||
// New creates a new Handlers instance.
|
// New creates a new Handlers instance.
|
||||||
@@ -54,7 +65,9 @@ func New(lc fx.Lifecycle, params Params) (*Handlers, error) {
|
|||||||
hc: params.Healthcheck,
|
hc: params.Healthcheck,
|
||||||
db: params.Database,
|
db: params.Database,
|
||||||
config: params.Config,
|
config: params.Config,
|
||||||
|
fetcher: params.Fetcher,
|
||||||
csrfProtect: csrfProtect,
|
csrfProtect: csrfProtect,
|
||||||
|
refererBlocklist: allowlist.New(params.Config.RefererBlocklist),
|
||||||
}
|
}
|
||||||
|
|
||||||
lc.Append(fx.Hook{
|
lc.Append(fx.Hook{
|
||||||
@@ -80,18 +93,25 @@ func (s *Handlers) WaitForProcessing(ctx context.Context) int {
|
|||||||
return s.imgSvc.WaitForProcessing(ctx)
|
return s.imgSvc.WaitForProcessing(ctx)
|
||||||
}
|
}
|
||||||
|
|
||||||
// initImageService initializes the image cache and service.
|
// newCacheConfig builds the image cache's configuration from cfg.
|
||||||
func (s *Handlers) initImageService() error {
|
// cache_max_bytes: 0 disables the disk cache entirely; any other value
|
||||||
// Create the cache. cache_max_bytes: 0 disables the disk cache
|
// is the eviction limit in bytes; when it is omitted, the cache works
|
||||||
// entirely; any other value is the eviction limit in bytes.
|
// out the default limit itself.
|
||||||
cache, err := imgcache.NewCache(s.db.DB(), imgcache.CacheConfig{
|
func newCacheConfig(cfg *config.Config, log *slog.Logger) imgcache.CacheConfig {
|
||||||
StateDir: s.config.StateDir,
|
return imgcache.CacheConfig{
|
||||||
|
StateDir: cfg.StateDir,
|
||||||
CacheTTL: imgcache.DefaultCacheTTL,
|
CacheTTL: imgcache.DefaultCacheTTL,
|
||||||
NegativeTTL: imgcache.DefaultNegativeTTL,
|
NegativeTTL: imgcache.DefaultNegativeTTL,
|
||||||
MaxBytes: s.config.CacheMaxBytes,
|
MaxBytes: cfg.CacheMaxBytes,
|
||||||
DisableDiskCache: s.config.CacheMaxBytes == 0,
|
UseDefaultMaxBytes: !cfg.CacheMaxBytesExplicit,
|
||||||
Logger: s.log,
|
DisableDiskCache: cfg.CacheMaxBytesExplicit && cfg.CacheMaxBytes == 0,
|
||||||
})
|
Logger: log,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// initImageService initializes the image cache and service.
|
||||||
|
func (s *Handlers) initImageService() error {
|
||||||
|
cache, err := imgcache.NewCache(s.db.DB(), newCacheConfig(s.config, s.log))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -115,10 +135,12 @@ func (s *Handlers) initImageService() error {
|
|||||||
fetcherCfg.MaxConnections = s.config.UpstreamConnections
|
fetcherCfg.MaxConnections = s.config.UpstreamConnections
|
||||||
fetcherCfg.BlockedNetworks = s.config.BlockedNetworks
|
fetcherCfg.BlockedNetworks = s.config.BlockedNetworks
|
||||||
|
|
||||||
// Create the service
|
// Create the service. With no fetcher provided, it builds its own from
|
||||||
|
// fetcherCfg.
|
||||||
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
|
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
|
||||||
Cache: cache,
|
Cache: cache,
|
||||||
FetcherConfig: fetcherCfg,
|
FetcherConfig: fetcherCfg,
|
||||||
|
Fetcher: s.fetcher,
|
||||||
SigningKey: s.config.SigningKey,
|
SigningKey: s.config.SigningKey,
|
||||||
Allowlist: s.config.AllowlistHosts,
|
Allowlist: s.config.AllowlistHosts,
|
||||||
MaxConcurrentProcessing: s.config.MaxConcurrentProcessing,
|
MaxConcurrentProcessing: s.config.MaxConcurrentProcessing,
|
||||||
|
|||||||
@@ -21,6 +21,10 @@ import (
|
|||||||
// /v1/image/<host>/<path>/<width>x<height>.<format>
|
// /v1/image/<host>/<path>/<width>x<height>.<format>
|
||||||
func (s *Handlers) HandleImage() http.HandlerFunc {
|
func (s *Handlers) HandleImage() http.HandlerFunc {
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if s.refuseBlockedReferer(w, r) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
req, ok := s.parseImageRequest(w, r)
|
req, ok := s.parseImageRequest(w, r)
|
||||||
if !ok {
|
if !ok {
|
||||||
return
|
return
|
||||||
@@ -39,6 +43,11 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The signature covers the format auto, not the format chosen
|
||||||
|
if !s.chooseAutoFormat(w, r, req) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
// Get cache key for logging
|
// Get cache key for logging
|
||||||
cacheKey := imgcache.CacheKey(req)
|
cacheKey := imgcache.CacheKey(req)
|
||||||
|
|
||||||
@@ -248,6 +257,23 @@ func cacheControl(expires time.Time) string {
|
|||||||
return fmt.Sprintf("public, max-age=%d, immutable", int64(maxAge/time.Second))
|
return fmt.Sprintf("public, max-age=%d, immutable", int64(maxAge/time.Second))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// refuseBlockedReferer answers 403 with a JSON error when the request's Referer
|
||||||
|
// names a host on referer_blocklist, and reports whether it answered. A request
|
||||||
|
// with no Referer, or one that does not parse as a URL with a host, is not
|
||||||
|
// refused.
|
||||||
|
func (s *Handlers) refuseBlockedReferer(
|
||||||
|
w http.ResponseWriter, r *http.Request,
|
||||||
|
) bool {
|
||||||
|
referer, err := url.Parse(r.Referer())
|
||||||
|
if err != nil || !s.refererBlocklist.IsAllowed(referer) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
s.respondError(w, "referer blocked", http.StatusForbidden)
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
// notModified sets the ETag header to etag and, when the request's
|
// notModified sets the ETag header to etag and, when the request's
|
||||||
// If-None-Match is that ETag, answers 304 Not Modified. It reports whether it
|
// If-None-Match is that ETag, answers 304 Not Modified. It reports whether it
|
||||||
// answered. An empty etag sets no header and never answers.
|
// answered. An empty etag sets no header and never answers.
|
||||||
|
|||||||
@@ -0,0 +1,267 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"image/color"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/go-chi/chi/v5"
|
||||||
|
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||||
|
"sneak.berlin/go/pixa/internal/imgcache"
|
||||||
|
"sneak.berlin/go/pixa/internal/signature"
|
||||||
|
)
|
||||||
|
|
||||||
|
// allowlistedHost is the only host on the allowlist of the image route
|
||||||
|
// newImageRoute builds.
|
||||||
|
const allowlistedHost = "allowed.example.com"
|
||||||
|
|
||||||
|
// newImageRoute returns the image route of a Handlers whose service fetches
|
||||||
|
// with fetcher and checks signatures with testSigningKey.
|
||||||
|
func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
|
||||||
|
StateDir: t.TempDir(),
|
||||||
|
CacheTTL: time.Hour,
|
||||||
|
NegativeTTL: 5 * time.Minute,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create cache: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
|
||||||
|
Cache: cache,
|
||||||
|
Fetcher: fetcher,
|
||||||
|
SigningKey: testSigningKey,
|
||||||
|
Allowlist: []string{allowlistedHost},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create service: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
h := &Handlers{imgSvc: svc, log: slog.New(slog.DiscardHandler)}
|
||||||
|
|
||||||
|
r := chi.NewRouter()
|
||||||
|
r.Get("/v1/image/*", h.HandleImage())
|
||||||
|
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
|
||||||
|
// newPhotoFetcher returns a mock fetcher that serves a JPEG at photoPath on
|
||||||
|
// each of hosts, and answers any other URL with an upstream error.
|
||||||
|
func newPhotoFetcher(t *testing.T, hosts ...string) *httpfetcher.MockFetcher {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
photo := &fstest.MapFile{
|
||||||
|
Data: generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255}),
|
||||||
|
}
|
||||||
|
|
||||||
|
files := fstest.MapFS{}
|
||||||
|
for _, host := range hosts {
|
||||||
|
files[host+photoPath] = photo
|
||||||
|
}
|
||||||
|
|
||||||
|
return httpfetcher.NewMock(files)
|
||||||
|
}
|
||||||
|
|
||||||
|
// photoURL returns the image route URL of photoPath on host, as a 50x50 JPEG.
|
||||||
|
func photoURL(host string) string {
|
||||||
|
return "/v1/image/" + host + photoPath + "/50x50.jpeg"
|
||||||
|
}
|
||||||
|
|
||||||
|
// photoURLWithSig returns photoURL(host) with sig and expires as its sig and
|
||||||
|
// exp.
|
||||||
|
func photoURLWithSig(host, sig string, expires time.Time) string {
|
||||||
|
return fmt.Sprintf("%s?sig=%s&exp=%d", photoURL(host), sig, expires.Unix())
|
||||||
|
}
|
||||||
|
|
||||||
|
// photoSignature returns the signature of photoURL(host) at the default
|
||||||
|
// quality and fit, made with key and expiring at expires.
|
||||||
|
func photoSignature(key, host string, expires time.Time) string {
|
||||||
|
return signature.New(key).Sign(&signature.Request{
|
||||||
|
SourceHost: host,
|
||||||
|
SourcePath: photoPath,
|
||||||
|
Width: 50,
|
||||||
|
Height: 50,
|
||||||
|
Format: string(imgcache.FormatJPEG),
|
||||||
|
Quality: 85,
|
||||||
|
FitMode: string(imgcache.FitCover),
|
||||||
|
Expires: expires,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// sendGet sends a GET for target to route and returns the response.
|
||||||
|
func sendGet(
|
||||||
|
t *testing.T, route http.Handler, target string,
|
||||||
|
) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil)
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
route.ServeHTTP(rec, req)
|
||||||
|
t.Logf("GET %s: %d", target, rec.Code)
|
||||||
|
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkErrorBody checks that rec has status wantStatus and the JSON error body
|
||||||
|
// the image route sends: wantError, wantStatus and the time in RFC 3339.
|
||||||
|
func checkErrorBody(
|
||||||
|
t *testing.T, rec *httptest.ResponseRecorder, wantStatus int, wantError string,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if rec.Code != wantStatus {
|
||||||
|
t.Errorf("status = %d, want %d", rec.Code, wantStatus)
|
||||||
|
}
|
||||||
|
|
||||||
|
if ct := rec.Header().Get("Content-Type"); ct != "application/json" {
|
||||||
|
t.Errorf("Content-Type = %q, want application/json", ct)
|
||||||
|
}
|
||||||
|
|
||||||
|
var body struct {
|
||||||
|
Error string `json:"error"`
|
||||||
|
Status int `json:"status"`
|
||||||
|
Timestamp string `json:"timestamp"`
|
||||||
|
}
|
||||||
|
|
||||||
|
err := json.NewDecoder(rec.Body).Decode(&body)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decoding response body: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if body.Error != wantError || body.Status != wantStatus {
|
||||||
|
t.Errorf("body error and status = %q %d, want %q %d",
|
||||||
|
body.Error, body.Status, wantError, wantStatus)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = time.Parse(time.RFC3339, body.Timestamp)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("body timestamp: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleImage_ErrorAnswers checks the status and the JSON error body the
|
||||||
|
// image route answers each request below with. The JPEG at photoPath exists on
|
||||||
|
// signedHost and on each host below that differs from it, so a request refused
|
||||||
|
// with 401 would otherwise be served.
|
||||||
|
func TestHandleImage_ErrorAnswers(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// A signature for signedHost must not verify for any of these.
|
||||||
|
parentHost := "example.com"
|
||||||
|
siblingHost := "other.example.com"
|
||||||
|
subdomainHost := "img." + signedHost
|
||||||
|
appendedHost := signedHost + ".example.net"
|
||||||
|
|
||||||
|
photos := newPhotoFetcher(t,
|
||||||
|
signedHost, parentHost, siblingHost, subdomainHost, appendedHost)
|
||||||
|
// The real fetcher refuses localhost before any lookup or connection.
|
||||||
|
realFetcher := httpfetcher.New(httpfetcher.DefaultConfig())
|
||||||
|
|
||||||
|
exp := time.Now().Add(time.Hour)
|
||||||
|
expired := time.Now().Add(-time.Hour)
|
||||||
|
sig := photoSignature(testSigningKey, signedHost, exp)
|
||||||
|
otherKeySig := photoSignature("another-signing-key", signedHost, exp)
|
||||||
|
expiredSig := photoSignature(testSigningKey, signedHost, expired)
|
||||||
|
localhostSig := photoSignature(testSigningKey, "localhost", exp)
|
||||||
|
|
||||||
|
// The error every request refused for its signature gets.
|
||||||
|
const unauthorized = "unauthorized"
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
fetcher httpfetcher.Fetcher
|
||||||
|
target string
|
||||||
|
wantStatus int
|
||||||
|
wantError string
|
||||||
|
}{
|
||||||
|
{"no sig or exp", photos, photoURL(signedHost),
|
||||||
|
http.StatusUnauthorized, unauthorized},
|
||||||
|
{"exp but no sig", photos,
|
||||||
|
fmt.Sprintf("%s?exp=%d", photoURL(signedHost), exp.Unix()),
|
||||||
|
http.StatusUnauthorized, unauthorized},
|
||||||
|
{"sig made with another key", photos,
|
||||||
|
photoURLWithSig(signedHost, otherKeySig, exp),
|
||||||
|
http.StatusUnauthorized, unauthorized},
|
||||||
|
{"sig without its = padding", photos,
|
||||||
|
photoURLWithSig(signedHost, strings.TrimRight(sig, "="), exp),
|
||||||
|
http.StatusUnauthorized, unauthorized},
|
||||||
|
{"sig in upper case", photos,
|
||||||
|
photoURLWithSig(signedHost, strings.ToUpper(sig), exp),
|
||||||
|
http.StatusUnauthorized, unauthorized},
|
||||||
|
{"expired sig", photos, photoURLWithSig(signedHost, expiredSig, expired),
|
||||||
|
http.StatusUnauthorized, unauthorized},
|
||||||
|
{"sig sent for the parent domain", photos,
|
||||||
|
photoURLWithSig(parentHost, sig, exp),
|
||||||
|
http.StatusUnauthorized, unauthorized},
|
||||||
|
{"sig sent for a sibling host", photos,
|
||||||
|
photoURLWithSig(siblingHost, sig, exp),
|
||||||
|
http.StatusUnauthorized, unauthorized},
|
||||||
|
{"sig sent for a subdomain", photos,
|
||||||
|
photoURLWithSig(subdomainHost, sig, exp),
|
||||||
|
http.StatusUnauthorized, unauthorized},
|
||||||
|
{"sig sent with another domain appended", photos,
|
||||||
|
photoURLWithSig(appendedHost, sig, exp),
|
||||||
|
http.StatusUnauthorized, unauthorized},
|
||||||
|
{"unparseable path", photos,
|
||||||
|
"/v1/image/" + allowlistedHost + photoPath + "/big.jpeg",
|
||||||
|
http.StatusBadRequest, "invalid image URL: invalid size format"},
|
||||||
|
{"blocked upstream address", realFetcher,
|
||||||
|
photoURLWithSig("localhost", localhostSig, exp),
|
||||||
|
http.StatusForbidden, "forbidden"},
|
||||||
|
{"upstream error", photos,
|
||||||
|
"/v1/image/" + allowlistedHost + "/images/missing.jpg/50x50.jpeg",
|
||||||
|
http.StatusBadGateway, "upstream error"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
rec := sendGet(t, newImageRoute(t, tt.fetcher), tt.target)
|
||||||
|
checkErrorBody(t, rec, tt.wantStatus, tt.wantError)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleImage_AllowlistOrSignature checks that the image route serves an
|
||||||
|
// image without a signature for a host on the allowlist only, and for another
|
||||||
|
// host only with a valid signature.
|
||||||
|
func TestHandleImage_AllowlistOrSignature(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
photos := newPhotoFetcher(t, allowlistedHost, signedHost)
|
||||||
|
exp := time.Now().Add(time.Hour)
|
||||||
|
sig := photoSignature(testSigningKey, signedHost, exp)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
target string
|
||||||
|
wantStatus int
|
||||||
|
}{
|
||||||
|
{"allowlisted host, no sig", photoURL(allowlistedHost), http.StatusOK},
|
||||||
|
{"other host, no sig", photoURL(signedHost), http.StatusUnauthorized},
|
||||||
|
{"other host, valid sig", photoURLWithSig(signedHost, sig, exp),
|
||||||
|
http.StatusOK},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
rec := sendGet(t, newImageRoute(t, photos), tt.target)
|
||||||
|
if rec.Code != tt.wantStatus {
|
||||||
|
t.Errorf("status = %d, want %d", rec.Code, tt.wantStatus)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -22,11 +22,15 @@ import (
|
|||||||
// browsers identify the content type.
|
// browsers identify the content type.
|
||||||
func (s *Handlers) HandleImageEnc() http.HandlerFunc {
|
func (s *Handlers) HandleImageEnc() http.HandlerFunc {
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if s.refuseBlockedReferer(w, r) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
ctx := r.Context()
|
ctx := r.Context()
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
|
|
||||||
req, ok := s.parseImageEncRequest(w, r)
|
req, ok := s.parseImageEncRequest(w, r)
|
||||||
if !ok {
|
if !ok || !s.chooseAutoFormat(w, r, req) {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,126 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"image/jpeg"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/encurl"
|
||||||
|
)
|
||||||
|
|
||||||
|
// requireServedPhoto requires that rec answers 200 with the JPEG at photoPath
|
||||||
|
// on signedHost at the 50x50 that encPhotoURL and the generator tests ask for.
|
||||||
|
func requireServedPhoto(t *testing.T, rec *httptest.ResponseRecorder) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want %d; body %q",
|
||||||
|
rec.Code, http.StatusOK, rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
contentType := rec.Header().Get("Content-Type")
|
||||||
|
if contentType != "image/jpeg" {
|
||||||
|
t.Errorf("Content-Type = %q, want image/jpeg", contentType)
|
||||||
|
}
|
||||||
|
|
||||||
|
img, err := jpeg.DecodeConfig(rec.Body)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("body is not a JPEG: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if img.Width != 50 || img.Height != 50 {
|
||||||
|
t.Errorf("image is %dx%d, want 50x50", img.Width, img.Height)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleImageEnc_ValidToken_ServesImage verifies that a token made with
|
||||||
|
// the signing key serves the image it asks for.
|
||||||
|
func TestHandleImageEnc_ValidToken_ServesImage(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, encPhotoURL(t, h), nil))
|
||||||
|
|
||||||
|
requireServedPhoto(t, rec)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleImageEnc_RejectedToken verifies that a token that has expired
|
||||||
|
// answers 410, and that a token with one character changed, a token cut
|
||||||
|
// short, and a token made with another signing key answer 400. The server
|
||||||
|
// would serve the photo for a token it accepted.
|
||||||
|
func TestHandleImageEnc_RejectedToken(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||||
|
|
||||||
|
photo := encurl.Payload{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: photoPath,
|
||||||
|
Width: 50,
|
||||||
|
Height: 50,
|
||||||
|
}
|
||||||
|
|
||||||
|
valid, err := h.encGen.Generate(&photo)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Generate() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
expiredPhoto := photo
|
||||||
|
expiredPhoto.ExpiresAt = time.Now().Add(-time.Minute).Unix()
|
||||||
|
|
||||||
|
expired, err := h.encGen.Generate(&expiredPhoto)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Generate() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
otherGen, err := encurl.NewGenerator("another-signing-key-fedcba9876543210")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("encurl.NewGenerator() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
otherKey, err := otherGen.Generate(&photo)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Generate() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Changing a character in the middle always changes the decoded bytes;
|
||||||
|
// the last character of unpadded base64 can carry unused bits.
|
||||||
|
middle := len(valid) / 2
|
||||||
|
|
||||||
|
replacement := "A"
|
||||||
|
if valid[middle] == 'A' {
|
||||||
|
replacement = "B"
|
||||||
|
}
|
||||||
|
|
||||||
|
changed := valid[:middle] + replacement + valid[middle+1:]
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
token string
|
||||||
|
wantStatus int
|
||||||
|
}{
|
||||||
|
{"expired", expired, http.StatusGone},
|
||||||
|
{"one character changed", changed, http.StatusBadRequest},
|
||||||
|
{"cut short", valid[:middle], http.StatusBadRequest},
|
||||||
|
{"another signing key", otherKey, http.StatusBadRequest},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
rec := getEncToken(srv, tt.token)
|
||||||
|
t.Logf("GET /v1/e/%s/img.jpg: %d %s", tt.token, rec.Code, rec.Body)
|
||||||
|
|
||||||
|
if rec.Code != tt.wantStatus {
|
||||||
|
t.Errorf("status = %d, want %d", rec.Code, tt.wantStatus)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,185 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/go-chi/chi/v5"
|
||||||
|
"sneak.berlin/go/pixa/internal/allowlist"
|
||||||
|
"sneak.berlin/go/pixa/internal/encurl"
|
||||||
|
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||||
|
"sneak.berlin/go/pixa/internal/imgcache"
|
||||||
|
)
|
||||||
|
|
||||||
|
// blockedReferer is a page on leech.example, which newRefererRoutes puts on
|
||||||
|
// referer_blocklist.
|
||||||
|
const blockedReferer = "https://leech.example/page.html"
|
||||||
|
|
||||||
|
// countingFetcher passes each fetch on to the fetcher it holds and counts it.
|
||||||
|
type countingFetcher struct {
|
||||||
|
httpfetcher.Fetcher
|
||||||
|
|
||||||
|
fetches atomic.Int32
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fetch counts the fetch and passes it on.
|
||||||
|
func (f *countingFetcher) Fetch(
|
||||||
|
ctx context.Context, url string,
|
||||||
|
) (*httpfetcher.FetchResult, error) {
|
||||||
|
f.fetches.Add(1)
|
||||||
|
|
||||||
|
return f.Fetcher.Fetch(ctx, url)
|
||||||
|
}
|
||||||
|
|
||||||
|
// newRefererRoutes returns both image routes of a Handlers whose
|
||||||
|
// referer_blocklist is "leech.example" and ".hotlinker.example", the
|
||||||
|
// Handlers, and the fetcher the routes fetch through. The JPEG at photoPath
|
||||||
|
// exists on allowlistedHost and on signedHost.
|
||||||
|
func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
fetcher := &countingFetcher{
|
||||||
|
Fetcher: newPhotoFetcher(t, allowlistedHost, signedHost),
|
||||||
|
}
|
||||||
|
|
||||||
|
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
|
||||||
|
StateDir: t.TempDir(),
|
||||||
|
CacheTTL: time.Hour,
|
||||||
|
NegativeTTL: 5 * time.Minute,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("imgcache.NewCache() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
|
||||||
|
Cache: cache,
|
||||||
|
Fetcher: fetcher,
|
||||||
|
SigningKey: testSigningKey,
|
||||||
|
Allowlist: []string{allowlistedHost},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("imgcache.NewService() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
encGen, err := encurl.NewGenerator(testSigningKey)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("encurl.NewGenerator() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
h := &Handlers{
|
||||||
|
log: slog.New(slog.DiscardHandler),
|
||||||
|
imgSvc: svc,
|
||||||
|
encGen: encGen,
|
||||||
|
refererBlocklist: allowlist.New(
|
||||||
|
[]string{"leech.example", ".hotlinker.example"}),
|
||||||
|
}
|
||||||
|
|
||||||
|
r := chi.NewRouter()
|
||||||
|
r.Get("/v1/image/*", h.HandleImage())
|
||||||
|
r.Get("/v1/e/{token}/*", h.HandleImageEnc())
|
||||||
|
|
||||||
|
return r, h, fetcher
|
||||||
|
}
|
||||||
|
|
||||||
|
// getWithReferer sends a GET for target to routes with referer as its
|
||||||
|
// Referer header, or with none when referer is empty, and returns the
|
||||||
|
// response.
|
||||||
|
func getWithReferer(
|
||||||
|
t *testing.T, routes http.Handler, target, referer string,
|
||||||
|
) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil)
|
||||||
|
if referer != "" {
|
||||||
|
req.Header.Set("Referer", referer)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
routes.ServeHTTP(rec, req)
|
||||||
|
t.Logf("GET %s with Referer %q: %d", target, referer, rec.Code)
|
||||||
|
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRefererBlocklist verifies that both image routes refuse a request whose
|
||||||
|
// Referer names a host on referer_blocklist with 403 and the JSON error,
|
||||||
|
// without fetching from the upstream host, and serve a request with no
|
||||||
|
// Referer, one that does not parse, or one naming any other host. Hosts are
|
||||||
|
// matched as allowlist_hosts matches them.
|
||||||
|
func TestRefererBlocklist(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
referer string
|
||||||
|
want int
|
||||||
|
}{
|
||||||
|
{"no referer", "", http.StatusOK},
|
||||||
|
{"unlisted host", "https://unlisted.example/page.html", http.StatusOK},
|
||||||
|
{"unparseable", "%zz", http.StatusOK},
|
||||||
|
{"listed host", blockedReferer, http.StatusForbidden},
|
||||||
|
{"subdomain of listed host", "https://www.leech.example/", http.StatusOK},
|
||||||
|
{"subdomain of dot pattern", "https://www.hotlinker.example/a.html",
|
||||||
|
http.StatusForbidden},
|
||||||
|
{"dot pattern without its dot", "https://hotlinker.example/",
|
||||||
|
http.StatusForbidden},
|
||||||
|
{"host continuing past dot pattern",
|
||||||
|
"https://hotlinker.example.evil.example/", http.StatusOK},
|
||||||
|
}
|
||||||
|
|
||||||
|
// The photo's URL on each image route.
|
||||||
|
photoURLs := map[string]func(t *testing.T, h *Handlers) string{
|
||||||
|
"plain URL": func(t *testing.T, _ *Handlers) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
return photoURL(allowlistedHost)
|
||||||
|
},
|
||||||
|
"encrypted URL": encPhotoURL,
|
||||||
|
}
|
||||||
|
|
||||||
|
for urlName, photoURLFor := range photoURLs {
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(urlName+", "+tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
routes, h, fetcher := newRefererRoutes(t)
|
||||||
|
|
||||||
|
rec := getWithReferer(t, routes, photoURLFor(t, h), tc.referer)
|
||||||
|
|
||||||
|
if tc.want == http.StatusOK {
|
||||||
|
requireServedPhoto(t, rec)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
checkErrorBody(t, rec, http.StatusForbidden, "referer blocked")
|
||||||
|
|
||||||
|
if n := fetcher.fetches.Load(); n != 0 {
|
||||||
|
t.Errorf("upstream fetched %d times, want 0", n)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBlockedRefererRefusedWhenImageIsCached verifies that a request whose
|
||||||
|
// Referer is on referer_blocklist is refused even when the image it asks for
|
||||||
|
// is already cached, so the answer does not depend on the cache.
|
||||||
|
func TestBlockedRefererRefusedWhenImageIsCached(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
routes, h, _ := newRefererRoutes(t)
|
||||||
|
|
||||||
|
for _, target := range []string{photoURL(allowlistedHost), encPhotoURL(t, h)} {
|
||||||
|
requireServedPhoto(t, getWithReferer(t, routes, target, ""))
|
||||||
|
|
||||||
|
rec := getWithReferer(t, routes, target, blockedReferer)
|
||||||
|
checkErrorBody(t, rec, http.StatusForbidden, "referer blocked")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"go.uber.org/fx/fxtest"
|
||||||
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
|
"sneak.berlin/go/pixa/internal/globals"
|
||||||
|
"sneak.berlin/go/pixa/internal/healthcheck"
|
||||||
|
"sneak.berlin/go/pixa/internal/logger"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestHandleRobotsTxt checks that /robots.txt asks every crawler to stay off
|
||||||
|
// the whole site.
|
||||||
|
func TestHandleRobotsTxt(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h := &Handlers{log: slog.New(slog.DiscardHandler)}
|
||||||
|
rec := sendGet(t, h.HandleRobotsTxt(), "/robots.txt")
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Errorf("status = %d, want %d", rec.Code, http.StatusOK)
|
||||||
|
}
|
||||||
|
|
||||||
|
if ct := rec.Header().Get("Content-Type"); ct != "text/plain" {
|
||||||
|
t.Errorf("Content-Type = %q, want text/plain", ct)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := "User-agent: *\nDisallow: /\n"
|
||||||
|
if rec.Body.String() != want {
|
||||||
|
t.Errorf("body = %q, want %q", rec.Body.String(), want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleHealthCheck checks that the health check answers 200 with status
|
||||||
|
// ok, the app's name and version, now, uptime_seconds, uptime_human and
|
||||||
|
// maintenance_mode, which is true here: the health check stays 200 while
|
||||||
|
// maintenance mode is on.
|
||||||
|
func TestHandleHealthCheck(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
lc := fxtest.NewLifecycle(t)
|
||||||
|
|
||||||
|
log, err := logger.New(lc, logger.Params{Globals: &globals.Globals{}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("logger.New() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
hc, err := healthcheck.New(lc, healthcheck.Params{
|
||||||
|
Globals: &globals.Globals{Appname: "pixad", Version: "v1.2.3"},
|
||||||
|
Config: &config.Config{MaintenanceMode: true},
|
||||||
|
Logger: log,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("healthcheck.New() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
h := &Handlers{hc: hc, log: slog.New(slog.DiscardHandler)}
|
||||||
|
rec := sendGet(t, h.HandleHealthCheck(), "/.well-known/healthcheck.json")
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Errorf("status = %d, want %d", rec.Code, http.StatusOK)
|
||||||
|
}
|
||||||
|
|
||||||
|
if ct := rec.Header().Get("Content-Type"); ct != "application/json" {
|
||||||
|
t.Errorf("Content-Type = %q, want application/json", ct)
|
||||||
|
}
|
||||||
|
|
||||||
|
var body map[string]any
|
||||||
|
|
||||||
|
err = json.NewDecoder(rec.Body).Decode(&body)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decoding response body: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if body["status"] != "ok" || body["appname"] != "pixad" ||
|
||||||
|
body["version"] != "v1.2.3" || body["maintenance_mode"] != true {
|
||||||
|
t.Errorf("body = %v, want status ok, appname pixad, version v1.2.3 "+
|
||||||
|
"and maintenance_mode true", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, key := range []string{"now", "uptime_seconds", "uptime_human"} {
|
||||||
|
if _, ok := body[key]; !ok {
|
||||||
|
t.Errorf("body = %v, has no %s", body, key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
package httpfetcher
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"net"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestNewUsesCheckedDialerWithoutDialContext checks that a fetcher built
|
||||||
|
// without DialContext, as pixa builds it, refuses to connect to a local
|
||||||
|
// server.
|
||||||
|
func TestNewUsesCheckedDialerWithoutDialContext(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
srv := startUpstream(t)
|
||||||
|
transport := transportOf(t, New(DefaultConfig()))
|
||||||
|
|
||||||
|
addr := srv.Listener.Addr().String()
|
||||||
|
|
||||||
|
_, err := transport.DialContext(testContext(t), "tcp", addr)
|
||||||
|
if !errors.Is(err, ErrSSRFBlocked) {
|
||||||
|
t.Fatalf("DialContext(%s) error = %v, want ErrSSRFBlocked", addr, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDialContextReplacesOnlyTheDialer checks that a fetcher built with
|
||||||
|
// DialContext connects through it, while the URL check still refuses a
|
||||||
|
// loopback URL and the redirect check a redirect to a link-local address.
|
||||||
|
func TestDialContextReplacesOnlyTheDialer(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
srv := startUpstream(t)
|
||||||
|
dialer := &recordingDialer{target: srv.Listener.Addr().String()}
|
||||||
|
|
||||||
|
cfg := DefaultConfig()
|
||||||
|
cfg.AllowHTTP = true
|
||||||
|
cfg.DialContext = dialer.dialContext
|
||||||
|
f := New(cfg)
|
||||||
|
|
||||||
|
if body := fetchBody(t, f, "/image"); body != imagePayload {
|
||||||
|
t.Errorf("body = %q, want %q", body, imagePayload)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := f.Fetch(testContext(t), "http://127.0.0.1/image")
|
||||||
|
if !errors.Is(err, ErrSSRFBlocked) {
|
||||||
|
t.Errorf("Fetch(loopback URL) error = %v, want ErrSSRFBlocked", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = f.Fetch(testContext(t), upstreamURL("/redirect/private"))
|
||||||
|
if !errors.Is(err, ErrSSRFBlocked) {
|
||||||
|
t.Errorf("Fetch(/redirect/private) error = %v, want ErrSSRFBlocked", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The upstream server is reached through DialContext, and nothing else
|
||||||
|
// is asked of it.
|
||||||
|
dialed := dialer.dialedAddrs()
|
||||||
|
if len(dialed) == 0 {
|
||||||
|
t.Error("DialContext was never called")
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, addr := range dialed {
|
||||||
|
if addr != net.JoinHostPort(testPublicHost, "80") {
|
||||||
|
t.Errorf("DialContext was asked to connect to %s", addr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -137,6 +137,11 @@ type Config struct {
|
|||||||
// BlockedNetworks are operator-supplied CIDR ranges refused by the
|
// BlockedNetworks are operator-supplied CIDR ranges refused by the
|
||||||
// dialer, in addition to the always-enforced built-in ranges.
|
// dialer, in addition to the always-enforced built-in ranges.
|
||||||
BlockedNetworks []netip.Prefix
|
BlockedNetworks []netip.Prefix
|
||||||
|
// DialContext, when set, makes the fetcher's connections in place of
|
||||||
|
// the dialer that refuses internal addresses; the URL and redirect
|
||||||
|
// checks still run. Only tests set it, to reach a local server; the
|
||||||
|
// config file and the environment cannot.
|
||||||
|
DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// DefaultConfig returns a Config with sensible defaults.
|
// DefaultConfig returns a Config with sensible defaults.
|
||||||
@@ -190,13 +195,19 @@ func New(config *Config) *HTTPFetcher {
|
|||||||
config = DefaultConfig()
|
config = DefaultConfig()
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create transport with SSRF-safe dialer. The dialer re-resolves and
|
// Unless config.DialContext replaces it, the transport connects with
|
||||||
// re-checks at connect time (closing the DNS-rebinding window) against
|
// the SSRF-safe dialer, which re-resolves and re-checks at connect time
|
||||||
// both the built-in ranges and the operator-supplied blocklist.
|
// (closing the DNS-rebinding window) against both the built-in ranges
|
||||||
transport := &http.Transport{
|
// and the operator-supplied blocklist.
|
||||||
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
|
dialContext := config.DialContext
|
||||||
|
if dialContext == nil {
|
||||||
|
dialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
|
||||||
return dialSSRFSafe(ctx, network, addr, config.BlockedNetworks)
|
return dialSSRFSafe(ctx, network, addr, config.BlockedNetworks)
|
||||||
},
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
transport := &http.Transport{
|
||||||
|
DialContext: dialContext,
|
||||||
TLSHandshakeTimeout: DefaultTLSTimeout,
|
TLSHandshakeTimeout: DefaultTLSTimeout,
|
||||||
MaxIdleConns: DefaultMaxIdleConns,
|
MaxIdleConns: DefaultMaxIdleConns,
|
||||||
IdleConnTimeout: DefaultIdleConnTimeout,
|
IdleConnTimeout: DefaultIdleConnTimeout,
|
||||||
|
|||||||
@@ -37,11 +37,16 @@ type CacheConfig struct {
|
|||||||
NegativeTTL time.Duration
|
NegativeTTL time.Duration
|
||||||
|
|
||||||
// MaxBytes is the disk cache size limit in bytes that eviction
|
// MaxBytes is the disk cache size limit in bytes that eviction
|
||||||
// enforces. Zero means no limit is enforced (no eviction). The
|
// enforces. Zero means no limit is enforced (no eviction).
|
||||||
// config layer supplies the computed default when the operator
|
|
||||||
// omits cache_max_bytes.
|
|
||||||
MaxBytes int64
|
MaxBytes int64
|
||||||
|
|
||||||
|
// UseDefaultMaxBytes makes NewCache replace MaxBytes with the
|
||||||
|
// default limit: 75% of the sum of the space free on the filesystem
|
||||||
|
// holding the cache and the bytes the cache already holds, at least
|
||||||
|
// DefaultCacheMaxBytesFloor. The config layer sets this when the
|
||||||
|
// operator omits cache_max_bytes.
|
||||||
|
UseDefaultMaxBytes bool
|
||||||
|
|
||||||
// DisableDiskCache turns the disk cache off entirely: no cache
|
// DisableDiskCache turns the disk cache off entirely: no cache
|
||||||
// directories are created, lookups always miss, stores are
|
// directories are created, lookups always miss, stores are
|
||||||
// no-ops, and no eviction machinery runs. The config layer sets
|
// no-ops, and no eviction machinery runs. The config layer sets
|
||||||
@@ -95,6 +100,14 @@ type Cache struct {
|
|||||||
|
|
||||||
// NewCache creates a new cache instance.
|
// NewCache creates a new cache instance.
|
||||||
func NewCache(db *sql.DB, config CacheConfig) (*Cache, error) {
|
func NewCache(db *sql.DB, config CacheConfig) (*Cache, error) {
|
||||||
|
return newCache(db, config, defaultFreeSpaceProbe)
|
||||||
|
}
|
||||||
|
|
||||||
|
// newCache is NewCache with the free-space probe passed in, so tests
|
||||||
|
// can fake the free space the default limit is worked out from.
|
||||||
|
func newCache(
|
||||||
|
db *sql.DB, config CacheConfig, probe FreeSpaceProbeFunc,
|
||||||
|
) (*Cache, error) {
|
||||||
log := config.Logger
|
log := config.Logger
|
||||||
if log == nil {
|
if log == nil {
|
||||||
log = slog.Default()
|
log = slog.Default()
|
||||||
@@ -145,6 +158,20 @@ func NewCache(db *sql.DB, config CacheConfig) (*Cache, error) {
|
|||||||
c.variants = variants
|
c.variants = variants
|
||||||
c.srcMetadata = srcMetadata
|
c.srcMetadata = srcMetadata
|
||||||
|
|
||||||
|
if config.UseDefaultMaxBytes {
|
||||||
|
limit, err := c.computeDefaultMaxBytes(context.Background(), probe)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
c.config.MaxBytes = limit
|
||||||
|
|
||||||
|
log.Info("computed default cache size limit from free space and cache contents",
|
||||||
|
"cache_max_bytes", limit,
|
||||||
|
"cache_dir", filepath.Join(config.StateDir, "cache"),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
return c, nil
|
return c, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,89 @@
|
|||||||
|
package imgcache
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"math"
|
||||||
|
"path/filepath"
|
||||||
|
"syscall"
|
||||||
|
)
|
||||||
|
|
||||||
|
// DefaultCacheMaxBytesFloor is the minimum computed default for the
|
||||||
|
// cache_max_bytes setting: 500 MiB. The floor applies only to the
|
||||||
|
// computed default (when the key is omitted from the configuration),
|
||||||
|
// never to explicitly configured values.
|
||||||
|
const DefaultCacheMaxBytesFloor int64 = 524288000
|
||||||
|
|
||||||
|
// freeSpaceFractionNumerator and freeSpaceFractionDenominator express
|
||||||
|
// the 75% share used for the computed default limit as integer
|
||||||
|
// arithmetic (dividing before multiplying avoids overflow).
|
||||||
|
const (
|
||||||
|
freeSpaceFractionNumerator uint64 = 3
|
||||||
|
freeSpaceFractionDenominator uint64 = 4
|
||||||
|
)
|
||||||
|
|
||||||
|
var errNegativeBlockSize = errors.New("statfs reported negative block size")
|
||||||
|
|
||||||
|
// FreeSpaceProbeFunc reports the number of free bytes available on the
|
||||||
|
// filesystem containing path. It is a function type so tests can
|
||||||
|
// inject a fake probe instead of depending on the host disk.
|
||||||
|
type FreeSpaceProbeFunc func(path string) (uint64, error)
|
||||||
|
|
||||||
|
// defaultFreeSpaceProbe reports free filesystem bytes via statfs on
|
||||||
|
// the given path, as available to unprivileged processes.
|
||||||
|
func defaultFreeSpaceProbe(path string) (uint64, error) {
|
||||||
|
var stat syscall.Statfs_t
|
||||||
|
|
||||||
|
err := syscall.Statfs(path, &stat)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if stat.Bsize < 0 {
|
||||||
|
return 0, fmt.Errorf("%w %d for %q", errNegativeBlockSize, stat.Bsize, path)
|
||||||
|
}
|
||||||
|
|
||||||
|
blockSize := uint64(stat.Bsize)
|
||||||
|
|
||||||
|
return stat.Bavail * blockSize, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// computeDefaultMaxBytes returns the default cache size limit: 75% of
|
||||||
|
// the sum of the free bytes probe reports for <state_dir>/cache/ and
|
||||||
|
// the bytes the cache already holds, with a floor of
|
||||||
|
// DefaultCacheMaxBytesFloor. Counting what the cache holds keeps the
|
||||||
|
// limit from shrinking as the cache fills.
|
||||||
|
func (c *Cache) computeDefaultMaxBytes(
|
||||||
|
ctx context.Context, probe FreeSpaceProbeFunc,
|
||||||
|
) (int64, error) {
|
||||||
|
cacheDir := filepath.Join(c.config.StateDir, "cache")
|
||||||
|
|
||||||
|
freeBytes, err := probe(cacheDir)
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf(
|
||||||
|
"default cache_max_bytes: cannot determine free space for %q: %w",
|
||||||
|
cacheDir, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
usedBytes, err := c.UsageBytes(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Both terms are at most math.MaxInt64, so the sum cannot overflow.
|
||||||
|
//nolint:gosec // G115: UsageBytes sums file sizes, never negative
|
||||||
|
spaceBytes := min(freeBytes, math.MaxInt64) + uint64(usedBytes)
|
||||||
|
|
||||||
|
computed := spaceBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
|
||||||
|
computed = min(computed, math.MaxInt64)
|
||||||
|
|
||||||
|
// gosec cannot see that min() above bounds computed, so it reads
|
||||||
|
// this conversion as potentially overflowing. It cannot: computed is
|
||||||
|
// at most math.MaxInt64 on every path here.
|
||||||
|
//nolint:gosec // G115: clamped to MaxInt64 by min above
|
||||||
|
limit := int64(computed)
|
||||||
|
limit = max(limit, DefaultCacheMaxBytesFloor)
|
||||||
|
|
||||||
|
return limit, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,188 @@
|
|||||||
|
package imgcache
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Static errors returned by the stub free-space probes below.
|
||||||
|
var (
|
||||||
|
errTestStatfsFailed = errors.New("statfs failed")
|
||||||
|
errTestProbeNotExpected = errors.New("probe must not be called")
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestComputeDefaultMaxBytesCountsWhatTheCacheHolds verifies that the
|
||||||
|
// default limit is 75% of the free space plus what the cache already
|
||||||
|
// holds, so a cache filled to its limit keeps that limit across a
|
||||||
|
// restart instead of shrinking to 75% of the space left free.
|
||||||
|
func TestComputeDefaultMaxBytesCountsWhatTheCacheHolds(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||||
|
|
||||||
|
// Empty cache, 4 GiB free -> 3 GiB default.
|
||||||
|
got, err := cache.computeDefaultMaxBytes(t.Context(),
|
||||||
|
func(string) (uint64, error) { return 4294967296, nil })
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("computeDefaultMaxBytes returned error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("default for an empty cache with 4 GiB free: %d", got)
|
||||||
|
|
||||||
|
if got != 3221225472 {
|
||||||
|
t.Errorf("default for an empty cache = %d, want 3221225472 (75%% of 4 GiB)",
|
||||||
|
got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The cache now holds those 3 GiB, which leaves 1 GiB free.
|
||||||
|
_, err = cache.db.ExecContext(t.Context(),
|
||||||
|
`INSERT INTO variant_content (cache_key, size_bytes, content_type)
|
||||||
|
VALUES (?, ?, ?)`,
|
||||||
|
string(testVariantKeyOne), 3221225472, testContentTypeWebP,
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to insert variant accounting row: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err = cache.computeDefaultMaxBytes(t.Context(),
|
||||||
|
func(string) (uint64, error) { return 1073741824, nil })
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("computeDefaultMaxBytes returned error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("default for a cache holding 3 GiB with 1 GiB free: %d", got)
|
||||||
|
|
||||||
|
if got != 3221225472 {
|
||||||
|
t.Errorf("default for a cache holding 3 GiB with 1 GiB free = %d, "+
|
||||||
|
"want 3221225472 (75%% of 1 GiB + 3 GiB)", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestComputeDefaultMaxBytesAppliesFloor verifies that when 75% of the
|
||||||
|
// free space plus what the cache holds is below 500 MiB, the default
|
||||||
|
// is floored at DefaultCacheMaxBytesFloor.
|
||||||
|
func TestComputeDefaultMaxBytesAppliesFloor(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
freeBytes uint64
|
||||||
|
}{
|
||||||
|
{name: "100 MiB free", freeBytes: 104857600},
|
||||||
|
{name: "zero free", freeBytes: 0},
|
||||||
|
{name: "just below floor threshold", freeBytes: 699050665},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||||
|
|
||||||
|
got, err := cache.computeDefaultMaxBytes(t.Context(),
|
||||||
|
func(string) (uint64, error) { return tc.freeBytes, nil })
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("computeDefaultMaxBytes returned error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got != DefaultCacheMaxBytesFloor {
|
||||||
|
t.Errorf("computeDefaultMaxBytes = %d, want floor %d",
|
||||||
|
got, DefaultCacheMaxBytesFloor)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestComputeDefaultMaxBytesPropagatesProbeError verifies that a
|
||||||
|
// failing free-space probe produces an error naming cache_max_bytes,
|
||||||
|
// instead of a silently wrong default.
|
||||||
|
func TestComputeDefaultMaxBytesPropagatesProbeError(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||||
|
|
||||||
|
_, err := cache.computeDefaultMaxBytes(t.Context(),
|
||||||
|
func(string) (uint64, error) { return 0, errTestStatfsFailed })
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("probe failure must produce an error, got nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("got expected error: %v", err)
|
||||||
|
|
||||||
|
if !strings.Contains(err.Error(), "cache_max_bytes") {
|
||||||
|
t.Errorf("error %q does not name cache_max_bytes", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestNewCacheComputesDefaultMaxBytesWhenAsked verifies that with
|
||||||
|
// UseDefaultMaxBytes set, the cache's limit becomes the computed
|
||||||
|
// default, and that the probe is pointed at <state_dir>/cache/, which
|
||||||
|
// must be created first so statfs measures the right filesystem.
|
||||||
|
func TestNewCacheComputesDefaultMaxBytesWhenAsked(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
stateDir := t.TempDir()
|
||||||
|
wantCacheDir := filepath.Join(stateDir, "cache")
|
||||||
|
|
||||||
|
var probedPath string
|
||||||
|
|
||||||
|
// 4 GiB free -> 3 GiB default.
|
||||||
|
probe := func(path string) (uint64, error) {
|
||||||
|
probedPath = path
|
||||||
|
|
||||||
|
info, err := os.Stat(path)
|
||||||
|
if err != nil || !info.IsDir() {
|
||||||
|
t.Errorf("cache directory %q was not created before probing: info=%v err=%v",
|
||||||
|
path, info, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return 4294967296, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
cache, err := newCache(evictionTestDB(t), CacheConfig{
|
||||||
|
StateDir: stateDir,
|
||||||
|
UseDefaultMaxBytes: true,
|
||||||
|
}, probe)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("newCache returned error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cache.config.MaxBytes != 3221225472 {
|
||||||
|
t.Errorf("MaxBytes = %d, want computed default 3221225472",
|
||||||
|
cache.config.MaxBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
if probedPath != wantCacheDir {
|
||||||
|
t.Errorf("free space probed at %q, want cache directory %q",
|
||||||
|
probedPath, wantCacheDir)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestNewCacheKeepsExplicitMaxBytes verifies that without
|
||||||
|
// UseDefaultMaxBytes the cache keeps MaxBytes exactly as given and
|
||||||
|
// never consults the free-space probe.
|
||||||
|
func TestNewCacheKeepsExplicitMaxBytes(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
probe := func(string) (uint64, error) {
|
||||||
|
t.Error("free-space probe must not be consulted for explicit values")
|
||||||
|
|
||||||
|
return 0, errTestProbeNotExpected
|
||||||
|
}
|
||||||
|
|
||||||
|
cache, err := newCache(evictionTestDB(t), CacheConfig{
|
||||||
|
StateDir: t.TempDir(),
|
||||||
|
MaxBytes: 1024,
|
||||||
|
}, probe)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("newCache returned error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cache.config.MaxBytes != 1024 {
|
||||||
|
t.Errorf("MaxBytes = %d, want explicit 1024 (no floor, no recompute)",
|
||||||
|
cache.config.MaxBytes)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -681,6 +681,11 @@ func TestEvictionRunsUnderWritePressure(t *testing.T) {
|
|||||||
assertNoDanglingReferences(t, cache)
|
assertNoDanglingReferences(t, cache)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestEvictionRunsOnPeriodicSchedule writes three variant files straight
|
||||||
|
// to disk, bypassing StoreVariant, so they have no accounting rows and no
|
||||||
|
// write-pressure notification fires. Only a periodic reconciliation pass
|
||||||
|
// can then adopt them, and only the eviction pass that follows it can
|
||||||
|
// evict them.
|
||||||
func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
|
func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -688,13 +693,29 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
|
|||||||
|
|
||||||
cache, _ := newEvictionTestCache(t, limit)
|
cache, _ := newEvictionTestCache(t, limit)
|
||||||
|
|
||||||
// Start the evictor while the cache is empty, then create tracked
|
// Hold the test database's only connection, so the startup pass
|
||||||
// over-limit state WITHOUT going through the store methods, so no
|
// waits for it after walking the still empty variant directory: the
|
||||||
// write-pressure notification fires and only the periodic ticker
|
// files written while it waits are first seen by a periodic pass.
|
||||||
// can trigger eviction.
|
conn, err := cache.db.Conn(t.Context())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to take the database connection: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = conn.Close() }()
|
||||||
|
|
||||||
cache.StartEviction(100 * time.Millisecond)
|
cache.StartEviction(100 * time.Millisecond)
|
||||||
defer func() { _ = cache.StopEviction(t.Context()) }()
|
defer func() { _ = cache.StopEviction(t.Context()) }()
|
||||||
|
|
||||||
|
deadline := time.Now().Add(5 * time.Second)
|
||||||
|
|
||||||
|
for cache.db.Stats().WaitCount == 0 {
|
||||||
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatal("the startup pass never waited for the database")
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
|
||||||
keys := []VariantKey{
|
keys := []VariantKey{
|
||||||
testVariantKeyOne, testVariantKeyTwo, testVariantKeyThree,
|
testVariantKeyOne, testVariantKeyTwo, testVariantKeyThree,
|
||||||
}
|
}
|
||||||
@@ -703,25 +724,43 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
|
|||||||
for i, key := range keys {
|
for i, key := range keys {
|
||||||
content := bytes.Repeat([]byte{fills[i]}, 1000)
|
content := bytes.Repeat([]byte{fills[i]}, 1000)
|
||||||
|
|
||||||
_, err := cache.variants.Store(key, bytes.NewReader(content), "image/webp")
|
_, err = cache.variants.Store(key, bytes.NewReader(content), "image/webp")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("failed to store variant file: %v", err)
|
t.Fatalf("failed to store variant file: %v", err)
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
_, err = cache.db.ExecContext(t.Context(),
|
_ = conn.Close()
|
||||||
`INSERT INTO variant_content (cache_key, size_bytes, content_type)
|
|
||||||
VALUES (?, ?, ?)`,
|
// Only one of the 1000-byte files fits under the limit: wait until
|
||||||
string(key), len(content), "image/webp",
|
// the evictor has removed the other two.
|
||||||
)
|
stored := len(keys)
|
||||||
|
deadline = time.Now().Add(5 * time.Second)
|
||||||
|
|
||||||
|
for stored > 1 && time.Now().Before(deadline) {
|
||||||
|
time.Sleep(25 * time.Millisecond)
|
||||||
|
|
||||||
|
stored = 0
|
||||||
|
|
||||||
|
for _, key := range keys {
|
||||||
|
if cache.variants.Exists(key) {
|
||||||
|
stored++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if stored > 1 {
|
||||||
|
t.Fatalf("periodic schedule did not trigger eviction: %d of %d "+
|
||||||
|
"variant files still on disk, want at most 1", stored, len(keys))
|
||||||
|
}
|
||||||
|
|
||||||
|
usage, err := cache.UsageBytes(t.Context())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("failed to insert variant accounting row: %v", err)
|
t.Fatalf("UsageBytes failed: %v", err)
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
usage := waitForUsageAtOrBelow(t, cache, limit, 5*time.Second)
|
|
||||||
if usage > limit {
|
if usage > limit {
|
||||||
t.Errorf("periodic schedule did not trigger eviction: usage = %d, want <= %d",
|
t.Errorf("usage after eviction = %d, want <= %d", usage, limit)
|
||||||
usage, limit)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
assertNoDanglingReferences(t, cache)
|
assertNoDanglingReferences(t, cache)
|
||||||
@@ -808,15 +847,28 @@ func TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup(t *testing.T) {
|
|||||||
|
|
||||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||||
|
|
||||||
const interval = 100 * time.Millisecond
|
// Hold the test database's only connection, so the startup pass
|
||||||
|
// waits for it after walking the still empty variant directory: the
|
||||||
|
// file written while it waits is first seen by a periodic pass.
|
||||||
|
conn, err := cache.db.Conn(t.Context())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to take the database connection: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
cache.StartEviction(interval)
|
defer func() { _ = conn.Close() }()
|
||||||
|
|
||||||
|
cache.StartEviction(100 * time.Millisecond)
|
||||||
defer func() { _ = cache.StopEviction(t.Context()) }()
|
defer func() { _ = cache.StopEviction(t.Context()) }()
|
||||||
|
|
||||||
// Let startup reconciliation run and settle on an empty cache
|
deadline := time.Now().Add(5 * time.Second)
|
||||||
// before introducing the untracked file, so the adoption we assert
|
|
||||||
// below can only be the work of a later, periodic pass.
|
for cache.db.Stats().WaitCount == 0 {
|
||||||
time.Sleep(3 * interval)
|
if time.Now().After(deadline) {
|
||||||
|
t.Fatal("the startup pass never waited for the database")
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
|
||||||
// Simulate a variant whose accounting insert failed after the
|
// Simulate a variant whose accounting insert failed after the
|
||||||
// process was already running and serving requests: the content
|
// process was already running and serving requests: the content
|
||||||
@@ -825,14 +877,16 @@ func TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup(t *testing.T) {
|
|||||||
// insert had failed and only the file write had succeeded.
|
// insert had failed and only the file write had succeeded.
|
||||||
untracked := bytes.Repeat([]byte{0x41}, 900)
|
untracked := bytes.Repeat([]byte{0x41}, 900)
|
||||||
|
|
||||||
_, err := cache.variants.Store(
|
_, err = cache.variants.Store(
|
||||||
"aabbccdd0099", bytes.NewReader(untracked), "image/webp",
|
"aabbccdd0099", bytes.NewReader(untracked), "image/webp",
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("failed to store untracked variant file: %v", err)
|
t.Fatalf("failed to store untracked variant file: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
deadline := time.Now().Add(5 * time.Second)
|
_ = conn.Close()
|
||||||
|
|
||||||
|
deadline = time.Now().Add(5 * time.Second)
|
||||||
|
|
||||||
var usage int64
|
var usage int64
|
||||||
|
|
||||||
|
|||||||
@@ -22,6 +22,11 @@ const (
|
|||||||
FormatWebP ImageFormat = "webp"
|
FormatWebP ImageFormat = "webp"
|
||||||
FormatAVIF ImageFormat = "avif"
|
FormatAVIF ImageFormat = "avif"
|
||||||
FormatGIF ImageFormat = "gif"
|
FormatGIF ImageFormat = "gif"
|
||||||
|
|
||||||
|
// FormatAuto stands for AVIF, WebP or JPEG, chosen for each request
|
||||||
|
// from its Accept header once the URL's signature or token has been
|
||||||
|
// checked; it is never processed or cached as itself.
|
||||||
|
FormatAuto ImageFormat = "auto"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Size represents requested image dimensions
|
// Size represents requested image dimensions
|
||||||
|
|||||||
@@ -42,7 +42,8 @@ type Service struct {
|
|||||||
type ServiceConfig struct {
|
type ServiceConfig struct {
|
||||||
// Cache is the cache instance
|
// Cache is the cache instance
|
||||||
Cache *Cache
|
Cache *Cache
|
||||||
// FetcherConfig configures the upstream fetcher (ignored if Fetcher is set)
|
// FetcherConfig configures the upstream fetcher built when Fetcher is
|
||||||
|
// not set. Its AllowHTTP and MaxResponseSize are used either way.
|
||||||
FetcherConfig *httpfetcher.Config
|
FetcherConfig *httpfetcher.Config
|
||||||
// Fetcher is an optional custom fetcher (for testing)
|
// Fetcher is an optional custom fetcher (for testing)
|
||||||
Fetcher httpfetcher.Fetcher
|
Fetcher httpfetcher.Fetcher
|
||||||
|
|||||||
@@ -277,6 +277,8 @@ func parseFormat(s string) (ImageFormat, error) {
|
|||||||
return FormatAVIF, nil
|
return FormatAVIF, nil
|
||||||
case "gif":
|
case "gif":
|
||||||
return FormatGIF, nil
|
return FormatGIF, nil
|
||||||
|
case "auto":
|
||||||
|
return FormatAuto, nil
|
||||||
default:
|
default:
|
||||||
return "", fmt.Errorf("%w: %s", ErrInvalidFormat, s)
|
return "", fmt.Errorf("%w: %s", ErrInvalidFormat, s)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -111,6 +111,21 @@ func TestParseImageURL(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestParseImageURL_AutoFormat verifies that the format auto in an image URL
|
||||||
|
// parses as FormatAuto.
|
||||||
|
func TestParseImageURL_AutoFormat(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
got, err := ParseImageURL("/v1/image/example.com/photo.jpg/200x200.auto")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseImageURL() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got.Format != FormatAuto {
|
||||||
|
t.Errorf("Format = %q, want %q", got.Format, FormatAuto)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestParseImageURL_Errors(t *testing.T) {
|
func TestParseImageURL_Errors(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,87 @@
|
|||||||
|
// Package loadtestorigin is the upstream host script/loadtest points pixad
|
||||||
|
// at, run by cmd/loadtest-origin. It answers every request, whatever its path,
|
||||||
|
// with the same generated JPEG, so each new path is a new source image for
|
||||||
|
// pixad to fetch, and it logs one line per request, so its log counts pixad's
|
||||||
|
// fetches.
|
||||||
|
package loadtestorigin
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"image"
|
||||||
|
"image/color"
|
||||||
|
"image/jpeg"
|
||||||
|
"log/slog"
|
||||||
|
"math"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
listenAddress = ":80"
|
||||||
|
readHeaderTimeout = 10 * time.Second
|
||||||
|
imageWidth = 1600
|
||||||
|
imageHeight = 1200
|
||||||
|
jpegQuality = 85
|
||||||
|
)
|
||||||
|
|
||||||
|
// Run makes the image and serves it on port 80 until the server fails, then
|
||||||
|
// exits the process with status 1.
|
||||||
|
func Run() {
|
||||||
|
photo, err := makeJPEG()
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("cannot make the image", "error", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
server := &http.Server{
|
||||||
|
Addr: listenAddress,
|
||||||
|
Handler: newHandler(photo),
|
||||||
|
ReadHeaderTimeout: readHeaderTimeout,
|
||||||
|
}
|
||||||
|
|
||||||
|
err = server.ListenAndServe()
|
||||||
|
slog.Error("server stopped", "error", err)
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
// newHandler answers every request with photo and logs the request's path.
|
||||||
|
func newHandler(photo []byte) http.Handler {
|
||||||
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
slog.Info("request", "path", r.URL.Path)
|
||||||
|
w.Header().Set("Content-Type", "image/jpeg")
|
||||||
|
_, _ = w.Write(photo)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// makeJPEG draws colour gradients crossed with a fine pattern, so the image
|
||||||
|
// has detail to decode and does not compress to almost nothing.
|
||||||
|
func makeJPEG() ([]byte, error) {
|
||||||
|
img := image.NewRGBA(image.Rect(0, 0, imageWidth, imageHeight))
|
||||||
|
|
||||||
|
// red and green count up from 0 to 255 and wrap around, along each row
|
||||||
|
// and down the image.
|
||||||
|
var green uint8
|
||||||
|
|
||||||
|
for y := range imageHeight {
|
||||||
|
var red uint8
|
||||||
|
|
||||||
|
for x := range imageWidth {
|
||||||
|
img.SetRGBA(x, y, color.RGBA{
|
||||||
|
R: red, G: green, B: red ^ green, A: math.MaxUint8,
|
||||||
|
})
|
||||||
|
red++
|
||||||
|
}
|
||||||
|
|
||||||
|
green++
|
||||||
|
}
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: jpegQuality})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return buf.Bytes(), nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
package loadtestorigin
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"image/jpeg"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestEveryPathServesTheSameJPEG checks that the origin answers any path with
|
||||||
|
// 200 and the same JPEG, so every new path script/loadtest asks pixad for is
|
||||||
|
// a valid source image.
|
||||||
|
func TestEveryPathServesTheSameJPEG(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
photo, err := makeJPEG()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("makeJPEG: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
size, err := jpeg.DecodeConfig(bytes.NewReader(photo))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("the image does not decode as a JPEG: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if size.Width != imageWidth || size.Height != imageHeight {
|
||||||
|
t.Errorf("the image is %dx%d, want %dx%d",
|
||||||
|
size.Width, size.Height, imageWidth, imageHeight)
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := newHandler(photo)
|
||||||
|
|
||||||
|
for _, path := range []string{"/", "/miss/1.jpg", "/herd/2.jpg"} {
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
handler.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, path, nil))
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Errorf("%s: status = %d, want %d", path, rec.Code, http.StatusOK)
|
||||||
|
}
|
||||||
|
|
||||||
|
if ct := rec.Header().Get("Content-Type"); ct != "image/jpeg" {
|
||||||
|
t.Errorf("%s: Content-Type = %q, want image/jpeg", path, ct)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !bytes.Equal(rec.Body.Bytes(), photo) {
|
||||||
|
t.Errorf("%s: the body is not the image", path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -35,13 +35,10 @@ const HSTSValue = "max-age=31536000; includeSubDomains"
|
|||||||
|
|
||||||
// ContentSecurityPolicyValue is the Content-Security-Policy header value.
|
// ContentSecurityPolicyValue is the Content-Security-Policy header value.
|
||||||
// default-src 'self' is the baseline and frame-ancestors 'none' is the primary
|
// default-src 'self' is the baseline and frame-ancestors 'none' is the primary
|
||||||
// clickjacking control. 'unsafe-inline' is required in script-src and style-src
|
// clickjacking control.
|
||||||
// because the served templates carry inline onclick handlers (generator page)
|
|
||||||
// and the bundled Tailwind asset injects a runtime <style> element; dropping it
|
|
||||||
// needs template changes outside this issue's scope.
|
|
||||||
const ContentSecurityPolicyValue = "default-src 'self'; " +
|
const ContentSecurityPolicyValue = "default-src 'self'; " +
|
||||||
"script-src 'self' 'unsafe-inline'; " +
|
"script-src 'self'; " +
|
||||||
"style-src 'self' 'unsafe-inline'; " +
|
"style-src 'self'; " +
|
||||||
"object-src 'none'; " +
|
"object-src 'none'; " +
|
||||||
"base-uri 'self'; " +
|
"base-uri 'self'; " +
|
||||||
"form-action 'self'; " +
|
"form-action 'self'; " +
|
||||||
|
|||||||
@@ -1,11 +1,16 @@
|
|||||||
package middleware
|
package middleware
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
|
"net/url"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||||
|
|
||||||
"sneak.berlin/go/pixa/internal/config"
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -56,6 +61,203 @@ func TestCORSAnswersWithConfiguredOrigin(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestCORSAnswersPreflightWithConfiguredOrigin checks that the CORS
|
||||||
|
// middleware answers a preflight request, which the CORS library handles
|
||||||
|
// apart from other requests, the same way: "*" lets any origin read
|
||||||
|
// responses and a single origin lets only that origin read them.
|
||||||
|
func TestCORSAnswersPreflightWithConfiguredOrigin(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const appOrigin = "https://app.example.com"
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
configured string
|
||||||
|
requestOrigin string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"*", "https://any.example.com", "*"},
|
||||||
|
{appOrigin, appOrigin, appOrigin},
|
||||||
|
{appOrigin, "https://other.example.com", ""},
|
||||||
|
}
|
||||||
|
|
||||||
|
testHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
})
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
mw := &Middleware{
|
||||||
|
log: slog.Default(),
|
||||||
|
config: &config.Config{AccessControlAllowOrigin: tc.configured},
|
||||||
|
}
|
||||||
|
|
||||||
|
handler := mw.CORS()(testHandler)
|
||||||
|
|
||||||
|
// An OPTIONS request naming the method it asks about is the
|
||||||
|
// preflight a browser sends before some cross-origin requests.
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodOptions, "/v1/image/example.com/a.jpg/1x1.png", nil)
|
||||||
|
req.Header.Set("Origin", tc.requestOrigin)
|
||||||
|
req.Header.Set("Access-Control-Request-Method", http.MethodGet)
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
handler.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
got := rec.Header().Get("Access-Control-Allow-Origin")
|
||||||
|
if got != tc.want {
|
||||||
|
t.Errorf("configured %q, preflight from %q: "+
|
||||||
|
"Access-Control-Allow-Origin = %q, want %q",
|
||||||
|
tc.configured, tc.requestOrigin, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestMetricsAuthRequiresConfiguredCredentials checks that MetricsAuth on
|
||||||
|
// its own answers 401 with a challenge to a request without credentials or
|
||||||
|
// with a wrong username or password, and lets a request with the configured
|
||||||
|
// username and password through. That the router puts it in front of
|
||||||
|
// /metrics is not tested.
|
||||||
|
func TestMetricsAuthRequiresConfiguredCredentials(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
username = "metricsuser"
|
||||||
|
password = "metricspass"
|
||||||
|
challenge = `Basic realm="metrics"`
|
||||||
|
)
|
||||||
|
|
||||||
|
// An empty username stands for a request sent without credentials.
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
username string
|
||||||
|
password string
|
||||||
|
wantReached bool
|
||||||
|
}{
|
||||||
|
{"no credentials", "", "", false},
|
||||||
|
{"wrong username", "someone", password, false},
|
||||||
|
{"wrong password", username, "wrongpass", false},
|
||||||
|
{"configured credentials", username, password, true},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mw := &Middleware{
|
||||||
|
log: slog.Default(),
|
||||||
|
config: &config.Config{
|
||||||
|
MetricsUsername: username,
|
||||||
|
MetricsPassword: password,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
reached := false
|
||||||
|
handler := mw.MetricsAuth()(http.HandlerFunc(
|
||||||
|
func(http.ResponseWriter, *http.Request) {
|
||||||
|
reached = true
|
||||||
|
}))
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, "/metrics", nil)
|
||||||
|
|
||||||
|
if tc.username != "" {
|
||||||
|
req.SetBasicAuth(tc.username, tc.password)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
handler.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
if reached != tc.wantReached {
|
||||||
|
t.Fatalf("request reached /metrics = %v, want %v",
|
||||||
|
reached, tc.wantReached)
|
||||||
|
}
|
||||||
|
|
||||||
|
if tc.wantReached {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if rec.Code != http.StatusUnauthorized {
|
||||||
|
t.Errorf("status = %d, want %d",
|
||||||
|
rec.Code, http.StatusUnauthorized)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := rec.Header().Get("WWW-Authenticate"); got != challenge {
|
||||||
|
t.Errorf("WWW-Authenticate = %q, want %q", got, challenge)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestMetricsRecordsServedRequest checks that the metrics middleware
|
||||||
|
// records a request it served, so /metrics reports it. It is the only test
|
||||||
|
// in this package that sets up the metrics middleware, which registers with
|
||||||
|
// the process-wide Prometheus registry and can do so only once.
|
||||||
|
func TestMetricsRecordsServedRequest(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// The line /metrics shows once one GET /test has been served.
|
||||||
|
const want = `http_request_duration_seconds_count{` +
|
||||||
|
`code="200",handler="/test",method="GET",service=""} 1`
|
||||||
|
|
||||||
|
mw := &Middleware{log: slog.Default(), config: &config.Config{}}
|
||||||
|
|
||||||
|
handler := mw.Metrics()(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
|
||||||
|
handler.ServeHTTP(httptest.NewRecorder(), httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, "/test", nil))
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
promhttp.Handler().ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, "/metrics", nil))
|
||||||
|
|
||||||
|
if !strings.Contains(rec.Body.String(), want) {
|
||||||
|
t.Errorf("/metrics does not report the GET /test served; "+
|
||||||
|
"want the line %q in:\n%s", want, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoggingLeavesOutSubmittedSigningKey checks that a login, a POST /
|
||||||
|
// whose form carries the signing key, leaves no trace of the key in the
|
||||||
|
// request's log line.
|
||||||
|
func TestLoggingLeavesOutSubmittedSigningKey(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const signingKey = "test-signing-key-0123456789abcdef"
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
mw := newTestMiddleware(t, &buf)
|
||||||
|
|
||||||
|
// The handler reads the key from the form, as the login handler does.
|
||||||
|
handler := mw.Logging()(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if got := r.FormValue("key"); got != signingKey {
|
||||||
|
t.Errorf("key in form = %q, want %q", got, signingKey)
|
||||||
|
}
|
||||||
|
|
||||||
|
w.WriteHeader(http.StatusSeeOther)
|
||||||
|
}))
|
||||||
|
|
||||||
|
form := url.Values{"key": {signingKey}}
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), http.MethodPost, "/",
|
||||||
|
strings.NewReader(form.Encode()))
|
||||||
|
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||||
|
|
||||||
|
handler.ServeHTTP(httptest.NewRecorder(), req)
|
||||||
|
|
||||||
|
if !strings.Contains(buf.String(), `"method":"POST"`) {
|
||||||
|
t.Fatalf("no log line for the request; got %q", buf.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.Contains(buf.String(), signingKey) {
|
||||||
|
t.Errorf("log output contains the signing key; got %q", buf.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestSecurityHeaders(t *testing.T) {
|
func TestSecurityHeaders(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -123,6 +325,13 @@ func TestSecurityHeaders_PolicyHeaders(t *testing.T) {
|
|||||||
|
|
||||||
handler.ServeHTTP(rec, req)
|
handler.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
// The login and generator pages load their script and stylesheet from
|
||||||
|
// /static, so the policy allows no inline script or style.
|
||||||
|
csp := rec.Header().Get("Content-Security-Policy")
|
||||||
|
if strings.Contains(csp, "unsafe-inline") {
|
||||||
|
t.Errorf("Content-Security-Policy allows unsafe-inline: %q", csp)
|
||||||
|
}
|
||||||
|
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
header string
|
header string
|
||||||
want string
|
want string
|
||||||
@@ -131,8 +340,8 @@ func TestSecurityHeaders_PolicyHeaders(t *testing.T) {
|
|||||||
{
|
{
|
||||||
"Content-Security-Policy",
|
"Content-Security-Policy",
|
||||||
"default-src 'self'; " +
|
"default-src 'self'; " +
|
||||||
"script-src 'self' 'unsafe-inline'; " +
|
"script-src 'self'; " +
|
||||||
"style-src 'self' 'unsafe-inline'; " +
|
"style-src 'self'; " +
|
||||||
"object-src 'none'; " +
|
"object-src 'none'; " +
|
||||||
"base-uri 'self'; " +
|
"base-uri 'self'; " +
|
||||||
"form-action 'self'; " +
|
"form-action 'self'; " +
|
||||||
|
|||||||
@@ -0,0 +1,46 @@
|
|||||||
|
package server
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"slices"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestFormatAutoVaryNextToOrigin requests an image URL with the format auto
|
||||||
|
// through the server's routes and verifies that the answer carries
|
||||||
|
// Vary: Accept next to the Vary: Origin the CORS middleware sends.
|
||||||
|
func TestFormatAutoVaryNextToOrigin(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
source := encodeTestPNG(t, 64, 48)
|
||||||
|
|
||||||
|
upstream := httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "image/png")
|
||||||
|
_, _ = w.Write(source)
|
||||||
|
}))
|
||||||
|
t.Cleanup(upstream.Close)
|
||||||
|
|
||||||
|
s, _, _ := startImageProxy(t, upstream)
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
|
||||||
|
"/v1/image/"+upstreamHost+"/photo.png/32x24.auto", nil)
|
||||||
|
req.Header.Set("Origin", "https://app.example.com")
|
||||||
|
req.Header.Set("Accept", "image/webp")
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
vary := rec.Header().Values("Vary")
|
||||||
|
t.Logf("status %d, Content-Type %s, Vary %v",
|
||||||
|
rec.Code, rec.Header().Get("Content-Type"), vary)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want %d", rec.Code, http.StatusOK)
|
||||||
|
}
|
||||||
|
|
||||||
|
if want := []string{"Origin", "Accept"}; !slices.Equal(vary, want) {
|
||||||
|
t.Errorf("Vary = %v, want %v", vary, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,302 @@
|
|||||||
|
package server
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"database/sql"
|
||||||
|
"encoding/hex"
|
||||||
|
"image"
|
||||||
|
"image/color"
|
||||||
|
"image/jpeg"
|
||||||
|
"image/png"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"go.uber.org/fx"
|
||||||
|
"go.uber.org/fx/fxtest"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
|
"sneak.berlin/go/pixa/internal/database"
|
||||||
|
"sneak.berlin/go/pixa/internal/globals"
|
||||||
|
"sneak.berlin/go/pixa/internal/handlers"
|
||||||
|
"sneak.berlin/go/pixa/internal/healthcheck"
|
||||||
|
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||||
|
"sneak.berlin/go/pixa/internal/logger"
|
||||||
|
"sneak.berlin/go/pixa/internal/middleware"
|
||||||
|
)
|
||||||
|
|
||||||
|
// upstreamHost is the upstream host of the image URLs below. It is a
|
||||||
|
// documentation address (RFC 5737), which the fetcher's URL check accepts as
|
||||||
|
// public; the fetcher's dial function connects it to the test upstream server.
|
||||||
|
const upstreamHost = "192.0.2.10"
|
||||||
|
|
||||||
|
// TestImageProxyFlow requests images through pixa's router, handlers,
|
||||||
|
// upstream fetcher, image processor, disk cache and database, with only the
|
||||||
|
// upstream origin replaced by a local test server. The first request for a URL
|
||||||
|
// is fetched and converted; the second is served from the cache without
|
||||||
|
// another upstream request. The source and the converted image are then on
|
||||||
|
// disk, with their rows in the database.
|
||||||
|
func TestImageProxyFlow(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
source := encodeTestPNG(t, 64, 48)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
sizeFormat string // the <size>.<format> part of the image URL
|
||||||
|
contentType string
|
||||||
|
decodeConfig func(io.Reader) (image.Config, error)
|
||||||
|
width, height int
|
||||||
|
}{
|
||||||
|
{"resize and convert to JPEG", "32x24.jpeg", "image/jpeg",
|
||||||
|
jpeg.DecodeConfig, 32, 24},
|
||||||
|
{"orig", "orig.orig", "image/png", png.DecodeConfig, 64, 48},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var upstreamRequests atomic.Int32
|
||||||
|
|
||||||
|
upstream := httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
upstreamRequests.Add(1)
|
||||||
|
w.Header().Set("Content-Type", "image/png")
|
||||||
|
_, _ = w.Write(source)
|
||||||
|
}))
|
||||||
|
t.Cleanup(upstream.Close)
|
||||||
|
|
||||||
|
s, db, stateDir := startImageProxy(t, upstream)
|
||||||
|
target := "/v1/image/" + upstreamHost + "/photo.png/" + tt.sizeFormat
|
||||||
|
|
||||||
|
first := getImage(t, s, target)
|
||||||
|
|
||||||
|
if got := first.Header().Get("X-Pixa-Cache"); got != "MISS" {
|
||||||
|
t.Errorf("first X-Pixa-Cache = %q, want MISS", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := first.Header().Get("Content-Type"); got != tt.contentType {
|
||||||
|
t.Errorf("Content-Type = %q, want %q", got, tt.contentType)
|
||||||
|
}
|
||||||
|
|
||||||
|
decoded, err := tt.decodeConfig(bytes.NewReader(first.Body.Bytes()))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decoding the image: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if decoded.Width != tt.width || decoded.Height != tt.height {
|
||||||
|
t.Errorf("image is %dx%d, want %dx%d",
|
||||||
|
decoded.Width, decoded.Height, tt.width, tt.height)
|
||||||
|
}
|
||||||
|
|
||||||
|
second := getImage(t, s, target)
|
||||||
|
|
||||||
|
if got := second.Header().Get("X-Pixa-Cache"); got != "HIT" {
|
||||||
|
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !bytes.Equal(second.Body.Bytes(), first.Body.Bytes()) {
|
||||||
|
t.Error("the second response is not the image the first served")
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := upstreamRequests.Load(); got != 1 {
|
||||||
|
t.Errorf("upstream received %d requests, want 1", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
checkSourceCached(t, db, stateDir, source)
|
||||||
|
checkVariantCached(t, db, stateDir, first.Body.Bytes(), tt.contentType)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// startImageProxy starts the components pixad's fx app builds, from a config
|
||||||
|
// with a fresh state directory and upstreamHost on the allowlist, and with an
|
||||||
|
// upstream fetcher that connects every upstream address to upstream. It
|
||||||
|
// returns the server with its routes, the database and the state directory.
|
||||||
|
func startImageProxy(
|
||||||
|
t *testing.T, upstream *httptest.Server,
|
||||||
|
) (*Server, *sql.DB, string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
stateDir := t.TempDir()
|
||||||
|
cfg := &config.Config{
|
||||||
|
SigningKey: testSigningKey,
|
||||||
|
StateDir: stateDir,
|
||||||
|
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
|
||||||
|
AllowlistHosts: []string{upstreamHost},
|
||||||
|
// The test upstream server has no TLS.
|
||||||
|
AllowHTTP: true,
|
||||||
|
// A limit of its own, so the cache does not size itself from the
|
||||||
|
// host's free disk space.
|
||||||
|
CacheMaxBytes: 64 << 20,
|
||||||
|
CacheMaxBytesExplicit: true,
|
||||||
|
UpstreamMaxResponseSize: config.DefaultUpstreamMaxResponseSize,
|
||||||
|
DownstreamTimeout: config.DefaultDownstreamTimeout,
|
||||||
|
}
|
||||||
|
|
||||||
|
fetcherCfg := httpfetcher.DefaultConfig()
|
||||||
|
fetcherCfg.AllowHTTP = true
|
||||||
|
fetcherCfg.DialContext = func(
|
||||||
|
ctx context.Context, network, _ string,
|
||||||
|
) (net.Conn, error) {
|
||||||
|
var dialer net.Dialer
|
||||||
|
|
||||||
|
return dialer.DialContext(ctx, network, upstream.Listener.Addr().String())
|
||||||
|
}
|
||||||
|
fetcher := httpfetcher.New(fetcherCfg)
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
mw *middleware.Middleware
|
||||||
|
db *database.Database
|
||||||
|
)
|
||||||
|
|
||||||
|
app := fxtest.New(t,
|
||||||
|
fx.Supply(cfg),
|
||||||
|
fx.Provide(
|
||||||
|
globals.New,
|
||||||
|
logger.New,
|
||||||
|
database.New,
|
||||||
|
healthcheck.New,
|
||||||
|
handlers.New,
|
||||||
|
middleware.New,
|
||||||
|
func() httpfetcher.Fetcher { return fetcher },
|
||||||
|
),
|
||||||
|
fx.Populate(&h, &mw, &db),
|
||||||
|
)
|
||||||
|
app.RequireStart()
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
// Requests go straight to the router, as in newTestServer; the server's
|
||||||
|
// own start hook, which listens on a port, is left out.
|
||||||
|
s := &Server{config: cfg, mw: mw, h: h}
|
||||||
|
s.SetupRoutes()
|
||||||
|
|
||||||
|
return s, db.DB(), stateDir
|
||||||
|
}
|
||||||
|
|
||||||
|
// getImage sends a GET for target to s and fails unless it answers 200.
|
||||||
|
func getImage(t *testing.T, s *Server, target string) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, target, nil))
|
||||||
|
t.Logf("GET %s: %d, X-Pixa-Cache %s",
|
||||||
|
target, rec.Code, rec.Header().Get("X-Pixa-Cache"))
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("GET %s status = %d, want %d; body %s",
|
||||||
|
target, rec.Code, http.StatusOK, rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkSourceCached checks that source is stored under its SHA-256 in
|
||||||
|
// cache/sources, recorded in source_content, and that the source URL's row in
|
||||||
|
// source_metadata points at it.
|
||||||
|
func checkSourceCached(t *testing.T, db *sql.DB, stateDir string, source []byte) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
sum := sha256.Sum256(source)
|
||||||
|
hash := hex.EncodeToString(sum[:])
|
||||||
|
|
||||||
|
checkFile(t, filepath.Join(stateDir, "cache", "sources", hash[0:2], hash[2:4], hash),
|
||||||
|
source)
|
||||||
|
|
||||||
|
var rows int
|
||||||
|
|
||||||
|
err := db.QueryRowContext(t.Context(),
|
||||||
|
"SELECT COUNT(*) FROM source_content WHERE content_hash = ?", hash,
|
||||||
|
).Scan(&rows)
|
||||||
|
if err != nil || rows != 1 {
|
||||||
|
t.Errorf("source_content rows for the source = %d (error %v), want 1",
|
||||||
|
rows, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var metadataHash string
|
||||||
|
|
||||||
|
err = db.QueryRowContext(t.Context(),
|
||||||
|
`SELECT content_hash FROM source_metadata
|
||||||
|
WHERE source_host = ? AND source_path = ?`,
|
||||||
|
upstreamHost, "/photo.png",
|
||||||
|
).Scan(&metadataHash)
|
||||||
|
if err != nil || metadataHash != hash {
|
||||||
|
t.Errorf("source_metadata content_hash = %q (error %v), want %q",
|
||||||
|
metadataHash, err, hash)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkVariantCached checks that the converted image served is recorded in
|
||||||
|
// variant_content with contentType, and stored under its cache key in
|
||||||
|
// cache/variants.
|
||||||
|
func checkVariantCached(
|
||||||
|
t *testing.T, db *sql.DB, stateDir string, served []byte, contentType string,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var cacheKey, storedType string
|
||||||
|
|
||||||
|
err := db.QueryRowContext(t.Context(),
|
||||||
|
"SELECT cache_key, content_type FROM variant_content",
|
||||||
|
).Scan(&cacheKey, &storedType)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("variant_content row: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if storedType != contentType {
|
||||||
|
t.Errorf("variant_content content_type = %q, want %q",
|
||||||
|
storedType, contentType)
|
||||||
|
}
|
||||||
|
|
||||||
|
checkFile(t, filepath.Join(stateDir, "cache", "variants",
|
||||||
|
cacheKey[0:2], cacheKey[2:4], cacheKey), served)
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkFile checks that the file at path holds want.
|
||||||
|
func checkFile(t *testing.T, path string, want []byte) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
//nolint:gosec // G304: a path under the test's state directory
|
||||||
|
got, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("reading %s: %v", path, err)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if !bytes.Equal(got, want) {
|
||||||
|
t.Errorf("%s holds %d bytes that are not the %d expected",
|
||||||
|
path, len(got), len(want))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// encodeTestPNG returns an opaque width x height PNG of one color.
|
||||||
|
func encodeTestPNG(t *testing.T, width, height int) []byte {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
img := image.NewRGBA(image.Rect(0, 0, width, height))
|
||||||
|
for y := range height {
|
||||||
|
for x := range width {
|
||||||
|
img.Set(x, y, color.RGBA{R: 200, G: 40, B: 40, A: 255})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
err := png.Encode(&buf, img)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("encoding the test PNG: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
package server
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestNoMetricsRecordedWithoutMetricsUsername checks that with no metrics
|
||||||
|
// username set the router records nothing about the requests it serves.
|
||||||
|
// /metrics is not served then, so the process-wide Prometheus registry is
|
||||||
|
// read directly. TestMaintenanceModeKeepsOtherRoutes records into the same
|
||||||
|
// registry, but never a GET /robots.txt.
|
||||||
|
func TestNoMetricsRecordedWithoutMetricsUsername(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := newTestServer(t)
|
||||||
|
s.ServeHTTP(httptest.NewRecorder(), httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, "/robots.txt", nil))
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
promhttp.Handler().ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, "/metrics", nil))
|
||||||
|
|
||||||
|
if strings.Contains(rec.Body.String(), `handler="/robots.txt"`) {
|
||||||
|
t.Errorf("with no metrics username GET /robots.txt was recorded:\n%s",
|
||||||
|
rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -53,7 +53,7 @@ func (s *Server) SetupRoutes() {
|
|||||||
// Robots.txt
|
// Robots.txt
|
||||||
s.router.Get("/robots.txt", s.h.HandleRobotsTxt())
|
s.router.Get("/robots.txt", s.h.HandleRobotsTxt())
|
||||||
|
|
||||||
// Static files (Tailwind CSS, etc.)
|
// The login and generator pages' stylesheet and script
|
||||||
s.router.Handle("/static/*", http.StripPrefix("/static/", static.Handler()))
|
s.router.Handle("/static/*", http.StripPrefix("/static/", static.Handler()))
|
||||||
|
|
||||||
// Login/generator UI. The form routes carry CSRF protection; the
|
// Login/generator UI. The form routes carry CSRF protection; the
|
||||||
|
|||||||
@@ -0,0 +1,10 @@
|
|||||||
|
// Generator page: a click on the generated URL selects it, and the Copy
|
||||||
|
// button copies it. Both are on the page only once a URL has been generated.
|
||||||
|
const generatedURL = document.getElementById("generated-url");
|
||||||
|
|
||||||
|
if (generatedURL) {
|
||||||
|
generatedURL.addEventListener("click", () => generatedURL.select());
|
||||||
|
document.getElementById("copy-url").addEventListener("click", () => {
|
||||||
|
navigator.clipboard.writeText(generatedURL.value);
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
)
|
)
|
||||||
|
|
||||||
//go:embed *.js
|
//go:embed *.css *.js
|
||||||
var files embed.FS
|
var files embed.FS
|
||||||
|
|
||||||
// FS returns the embedded filesystem containing static files.
|
// FS returns the embedded filesystem containing static files.
|
||||||
|
|||||||
@@ -0,0 +1,190 @@
|
|||||||
|
/* The login and generator pages. */
|
||||||
|
|
||||||
|
* {
|
||||||
|
box-sizing: border-box;
|
||||||
|
}
|
||||||
|
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
min-height: 100vh;
|
||||||
|
background: #f3f4f6;
|
||||||
|
font-family: system-ui, sans-serif;
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
|
||||||
|
h1 {
|
||||||
|
margin: 0;
|
||||||
|
font-size: 1.5rem;
|
||||||
|
line-height: 2rem;
|
||||||
|
font-weight: 700;
|
||||||
|
color: #1f2937;
|
||||||
|
}
|
||||||
|
|
||||||
|
label {
|
||||||
|
display: block;
|
||||||
|
margin-bottom: 0.25rem;
|
||||||
|
font-size: 0.875rem;
|
||||||
|
font-weight: 500;
|
||||||
|
color: #374151;
|
||||||
|
}
|
||||||
|
|
||||||
|
input,
|
||||||
|
select {
|
||||||
|
width: 100%;
|
||||||
|
padding: 0.5rem 0.75rem;
|
||||||
|
border: 1px solid #d1d5db;
|
||||||
|
border-radius: 0.375rem;
|
||||||
|
box-shadow: 0 1px 2px rgb(0 0 0 / 5%);
|
||||||
|
font: inherit;
|
||||||
|
}
|
||||||
|
|
||||||
|
input:focus,
|
||||||
|
select:focus {
|
||||||
|
outline: none;
|
||||||
|
border-color: #3b82f6;
|
||||||
|
box-shadow: 0 0 0 2px #3b82f6;
|
||||||
|
}
|
||||||
|
|
||||||
|
button {
|
||||||
|
width: 100%;
|
||||||
|
padding: 0.5rem 1rem;
|
||||||
|
border: none;
|
||||||
|
border-radius: 0.375rem;
|
||||||
|
background: #2563eb;
|
||||||
|
color: #fff;
|
||||||
|
font: inherit;
|
||||||
|
cursor: pointer;
|
||||||
|
transition: background-color 0.15s;
|
||||||
|
}
|
||||||
|
|
||||||
|
button:hover {
|
||||||
|
background: #1d4ed8;
|
||||||
|
}
|
||||||
|
|
||||||
|
button:focus {
|
||||||
|
outline: 2px solid #3b82f6;
|
||||||
|
outline-offset: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
form > * + * {
|
||||||
|
margin-top: 1rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card {
|
||||||
|
padding: 1.5rem;
|
||||||
|
border-radius: 0.5rem;
|
||||||
|
background: #fff;
|
||||||
|
box-shadow:
|
||||||
|
0 4px 6px -1px rgb(0 0 0 / 10%),
|
||||||
|
0 2px 4px -2px rgb(0 0 0 / 10%);
|
||||||
|
}
|
||||||
|
|
||||||
|
.error {
|
||||||
|
margin-bottom: 1rem;
|
||||||
|
padding: 0.75rem 1rem;
|
||||||
|
border: 1px solid #f87171;
|
||||||
|
border-radius: 0.25rem;
|
||||||
|
background: #fee2e2;
|
||||||
|
color: #b91c1c;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Login page: the card centred on the screen. */
|
||||||
|
|
||||||
|
.login {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.login .card {
|
||||||
|
width: 100%;
|
||||||
|
max-width: 28rem;
|
||||||
|
padding: 2rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.login h1 {
|
||||||
|
margin-bottom: 1.5rem;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Generator page. */
|
||||||
|
|
||||||
|
.page {
|
||||||
|
max-width: 42rem;
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 2rem 1rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
header {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
align-items: center;
|
||||||
|
margin-bottom: 2rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
header a {
|
||||||
|
font-size: 0.875rem;
|
||||||
|
color: #4b5563;
|
||||||
|
}
|
||||||
|
|
||||||
|
header a:hover {
|
||||||
|
color: #1f2937;
|
||||||
|
}
|
||||||
|
|
||||||
|
.result {
|
||||||
|
margin-bottom: 1.5rem;
|
||||||
|
padding: 1rem;
|
||||||
|
border: 1px solid #bbf7d0;
|
||||||
|
border-radius: 0.5rem;
|
||||||
|
background: #f0fdf4;
|
||||||
|
}
|
||||||
|
|
||||||
|
.result h2 {
|
||||||
|
margin: 0 0 0.5rem;
|
||||||
|
font-size: 0.875rem;
|
||||||
|
font-weight: 500;
|
||||||
|
color: #166534;
|
||||||
|
}
|
||||||
|
|
||||||
|
.result div {
|
||||||
|
display: flex;
|
||||||
|
gap: 0.5rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.result input {
|
||||||
|
flex: 1;
|
||||||
|
border-color: #86efac;
|
||||||
|
box-shadow: none;
|
||||||
|
font-family: ui-monospace, monospace;
|
||||||
|
font-size: 0.875rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.result button {
|
||||||
|
width: auto;
|
||||||
|
padding: 0.5rem 0.75rem;
|
||||||
|
background: #16a34a;
|
||||||
|
font-size: 0.875rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.result button:hover {
|
||||||
|
background: #15803d;
|
||||||
|
}
|
||||||
|
|
||||||
|
.result p {
|
||||||
|
margin: 0.5rem 0 0;
|
||||||
|
font-size: 0.75rem;
|
||||||
|
color: #16a34a;
|
||||||
|
}
|
||||||
|
|
||||||
|
.columns {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||||
|
gap: 1rem;
|
||||||
|
}
|
||||||
|
|
||||||
|
.note {
|
||||||
|
margin-top: 1rem;
|
||||||
|
font-size: 0.75rem;
|
||||||
|
color: #6b7280;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
File diff suppressed because one or more lines are too long
@@ -4,52 +4,47 @@
|
|||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>Pixa - URL Generator</title>
|
<title>Pixa - URL Generator</title>
|
||||||
<script src="/static/tailwind.js"></script>
|
<link rel="stylesheet" href="/static/style.css">
|
||||||
</head>
|
</head>
|
||||||
<body class="bg-gray-100 min-h-screen">
|
<body>
|
||||||
<div class="max-w-2xl mx-auto py-8 px-4">
|
<div class="page">
|
||||||
<div class="flex justify-between items-center mb-8">
|
<header>
|
||||||
<h1 class="text-2xl font-bold text-gray-800">Pixa URL Generator</h1>
|
<h1>Pixa URL Generator</h1>
|
||||||
<a href="/logout" class="text-sm text-gray-600 hover:text-gray-800 underline">
|
<a href="/logout">
|
||||||
Logout
|
Logout
|
||||||
</a>
|
</a>
|
||||||
</div>
|
</header>
|
||||||
|
|
||||||
{{if .GeneratedURL}}
|
{{if .GeneratedURL}}
|
||||||
<div class="bg-green-50 border border-green-200 rounded-lg p-4 mb-6">
|
<div class="result">
|
||||||
<h2 class="text-sm font-medium text-green-800 mb-2">Generated URL</h2>
|
<h2>Generated URL</h2>
|
||||||
<div class="flex gap-2">
|
<div>
|
||||||
<input
|
<input
|
||||||
type="text"
|
type="text"
|
||||||
readonly
|
readonly
|
||||||
value="{{.GeneratedURL}}"
|
value="{{.GeneratedURL}}"
|
||||||
id="generated-url"
|
id="generated-url"
|
||||||
class="flex-1 px-3 py-2 bg-white border border-green-300 rounded-md text-sm font-mono"
|
|
||||||
onclick="this.select()"
|
|
||||||
>
|
|
||||||
<button
|
|
||||||
onclick="navigator.clipboard.writeText(document.getElementById('generated-url').value)"
|
|
||||||
class="px-3 py-2 bg-green-600 text-white rounded-md hover:bg-green-700 text-sm"
|
|
||||||
>
|
>
|
||||||
|
<button id="copy-url">
|
||||||
Copy
|
Copy
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
<p class="text-xs text-green-600 mt-2">
|
<p>
|
||||||
Expires: {{.ExpiresAt}}
|
Expires: {{.ExpiresAt}}
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
{{if .Error}}
|
{{if .Error}}
|
||||||
<div class="bg-red-100 border border-red-400 text-red-700 px-4 py-3 rounded mb-6">
|
<div class="error">
|
||||||
{{.Error}}
|
{{.Error}}
|
||||||
</div>
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
<form method="POST" action="/generate" class="bg-white rounded-lg shadow-md p-6 space-y-4">
|
<form method="POST" action="/generate" class="card">
|
||||||
{{ .CSRFField }}
|
{{ .CSRFField }}
|
||||||
<div>
|
<div>
|
||||||
<label for="url" class="block text-sm font-medium text-gray-700 mb-1">
|
<label for="url">
|
||||||
Source URL
|
Source URL
|
||||||
</label>
|
</label>
|
||||||
<input
|
<input
|
||||||
@@ -59,13 +54,12 @@
|
|||||||
required
|
required
|
||||||
placeholder="https://example.com/image.jpg"
|
placeholder="https://example.com/image.jpg"
|
||||||
value="{{.FormURL}}"
|
value="{{.FormURL}}"
|
||||||
class="w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-blue-500"
|
|
||||||
>
|
>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="grid grid-cols-2 gap-4">
|
<div class="columns">
|
||||||
<div>
|
<div>
|
||||||
<label for="width" class="block text-sm font-medium text-gray-700 mb-1">
|
<label for="width">
|
||||||
Width
|
Width
|
||||||
</label>
|
</label>
|
||||||
<input
|
<input
|
||||||
@@ -76,11 +70,10 @@
|
|||||||
max="8192"
|
max="8192"
|
||||||
value="{{if .FormWidth}}{{.FormWidth}}{{else}}0{{end}}"
|
value="{{if .FormWidth}}{{.FormWidth}}{{else}}0{{end}}"
|
||||||
placeholder="0 = original"
|
placeholder="0 = original"
|
||||||
class="w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-blue-500"
|
|
||||||
>
|
>
|
||||||
</div>
|
</div>
|
||||||
<div>
|
<div>
|
||||||
<label for="height" class="block text-sm font-medium text-gray-700 mb-1">
|
<label for="height">
|
||||||
Height
|
Height
|
||||||
</label>
|
</label>
|
||||||
<input
|
<input
|
||||||
@@ -91,22 +84,18 @@
|
|||||||
max="8192"
|
max="8192"
|
||||||
value="{{if .FormHeight}}{{.FormHeight}}{{else}}0{{end}}"
|
value="{{if .FormHeight}}{{.FormHeight}}{{else}}0{{end}}"
|
||||||
placeholder="0 = original"
|
placeholder="0 = original"
|
||||||
class="w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-blue-500"
|
|
||||||
>
|
>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="grid grid-cols-2 gap-4">
|
<div class="columns">
|
||||||
<div>
|
<div>
|
||||||
<label for="format" class="block text-sm font-medium text-gray-700 mb-1">
|
<label for="format">
|
||||||
Format
|
Format
|
||||||
</label>
|
</label>
|
||||||
<select
|
<select id="format" name="format">
|
||||||
id="format"
|
|
||||||
name="format"
|
|
||||||
class="w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-blue-500"
|
|
||||||
>
|
|
||||||
<option value="orig" {{if eq .FormFormat "orig"}}selected{{end}}>Original</option>
|
<option value="orig" {{if eq .FormFormat "orig"}}selected{{end}}>Original</option>
|
||||||
|
<option value="auto" {{if eq .FormFormat "auto"}}selected{{end}}>Auto (AVIF, WebP or JPEG)</option>
|
||||||
<option value="jpeg" {{if eq .FormFormat "jpeg"}}selected{{end}}>JPEG</option>
|
<option value="jpeg" {{if eq .FormFormat "jpeg"}}selected{{end}}>JPEG</option>
|
||||||
<option value="png" {{if eq .FormFormat "png"}}selected{{end}}>PNG</option>
|
<option value="png" {{if eq .FormFormat "png"}}selected{{end}}>PNG</option>
|
||||||
<option value="webp" {{if eq .FormFormat "webp"}}selected{{end}}>WebP</option>
|
<option value="webp" {{if eq .FormFormat "webp"}}selected{{end}}>WebP</option>
|
||||||
@@ -115,14 +104,10 @@
|
|||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
<div>
|
<div>
|
||||||
<label for="quality" class="block text-sm font-medium text-gray-700 mb-1">
|
<label for="quality">
|
||||||
Quality
|
Quality
|
||||||
</label>
|
</label>
|
||||||
<select
|
<select id="quality" name="quality">
|
||||||
id="quality"
|
|
||||||
name="quality"
|
|
||||||
class="w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-blue-500"
|
|
||||||
>
|
|
||||||
<option value="25" {{if eq .FormQuality "25"}}selected{{end}}>Potato</option>
|
<option value="25" {{if eq .FormQuality "25"}}selected{{end}}>Potato</option>
|
||||||
<option value="50" {{if eq .FormQuality "50"}}selected{{end}}>Low</option>
|
<option value="50" {{if eq .FormQuality "50"}}selected{{end}}>Low</option>
|
||||||
<option value="70" {{if eq .FormQuality "70"}}selected{{end}}>Medium</option>
|
<option value="70" {{if eq .FormQuality "70"}}selected{{end}}>Medium</option>
|
||||||
@@ -132,16 +117,12 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="grid grid-cols-2 gap-4">
|
<div class="columns">
|
||||||
<div>
|
<div>
|
||||||
<label for="fit" class="block text-sm font-medium text-gray-700 mb-1">
|
<label for="fit">
|
||||||
Fit Mode
|
Fit Mode
|
||||||
</label>
|
</label>
|
||||||
<select
|
<select id="fit" name="fit">
|
||||||
id="fit"
|
|
||||||
name="fit"
|
|
||||||
class="w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-blue-500"
|
|
||||||
>
|
|
||||||
<option value="cover" {{if eq .FormFit "cover"}}selected{{end}}>Cover</option>
|
<option value="cover" {{if eq .FormFit "cover"}}selected{{end}}>Cover</option>
|
||||||
<option value="contain" {{if eq .FormFit "contain"}}selected{{end}}>Contain</option>
|
<option value="contain" {{if eq .FormFit "contain"}}selected{{end}}>Contain</option>
|
||||||
<option value="fill" {{if eq .FormFit "fill"}}selected{{end}}>Fill</option>
|
<option value="fill" {{if eq .FormFit "fill"}}selected{{end}}>Fill</option>
|
||||||
@@ -150,14 +131,10 @@
|
|||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
<div>
|
<div>
|
||||||
<label for="ttl" class="block text-sm font-medium text-gray-700 mb-1">
|
<label for="ttl">
|
||||||
Expires In
|
Expires In
|
||||||
</label>
|
</label>
|
||||||
<select
|
<select id="ttl" name="ttl">
|
||||||
id="ttl"
|
|
||||||
name="ttl"
|
|
||||||
class="w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-blue-500"
|
|
||||||
>
|
|
||||||
<option value="0" {{if or (eq .FormTTL "0") (eq .FormTTL "")}}selected{{end}}>Never</option>
|
<option value="0" {{if or (eq .FormTTL "0") (eq .FormTTL "")}}selected{{end}}>Never</option>
|
||||||
<option value="60" {{if eq .FormTTL "60"}}selected{{end}}>1 minute</option>
|
<option value="60" {{if eq .FormTTL "60"}}selected{{end}}>1 minute</option>
|
||||||
<option value="3600" {{if eq .FormTTL "3600"}}selected{{end}}>1 hour</option>
|
<option value="3600" {{if eq .FormTTL "3600"}}selected{{end}}>1 hour</option>
|
||||||
@@ -169,17 +146,15 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<button
|
<button type="submit">
|
||||||
type="submit"
|
|
||||||
class="w-full bg-blue-600 text-white py-2 px-4 rounded-md hover:bg-blue-700 focus:outline-none focus:ring-2 focus:ring-blue-500 focus:ring-offset-2 transition-colors"
|
|
||||||
>
|
|
||||||
Generate Encrypted URL
|
Generate Encrypted URL
|
||||||
</button>
|
</button>
|
||||||
</form>
|
</form>
|
||||||
|
|
||||||
<p class="text-xs text-gray-500 mt-4 text-center">
|
<p class="note">
|
||||||
Generated URLs are encrypted and cannot be modified. They will expire at the specified time.
|
Generated URLs are encrypted and cannot be modified. They will expire at the specified time.
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
<script src="/static/generator.js"></script>
|
||||||
</body>
|
</body>
|
||||||
</html>
|
</html>
|
||||||
|
|||||||
@@ -4,22 +4,22 @@
|
|||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>Pixa - Login</title>
|
<title>Pixa - Login</title>
|
||||||
<script src="/static/tailwind.js"></script>
|
<link rel="stylesheet" href="/static/style.css">
|
||||||
</head>
|
</head>
|
||||||
<body class="bg-gray-100 min-h-screen flex items-center justify-center">
|
<body class="login">
|
||||||
<div class="bg-white p-8 rounded-lg shadow-md w-full max-w-md">
|
<div class="card">
|
||||||
<h1 class="text-2xl font-bold text-gray-800 mb-6 text-center">Pixa Image Proxy</h1>
|
<h1>Pixa Image Proxy</h1>
|
||||||
|
|
||||||
{{if .Error}}
|
{{if .Error}}
|
||||||
<div class="bg-red-100 border border-red-400 text-red-700 px-4 py-3 rounded mb-4">
|
<div class="error">
|
||||||
{{.Error}}
|
{{.Error}}
|
||||||
</div>
|
</div>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|
||||||
<form method="POST" action="/" class="space-y-4">
|
<form method="POST" action="/">
|
||||||
{{ .CSRFField }}
|
{{ .CSRFField }}
|
||||||
<div>
|
<div>
|
||||||
<label for="key" class="block text-sm font-medium text-gray-700 mb-1">
|
<label for="key">
|
||||||
Signing Key
|
Signing Key
|
||||||
</label>
|
</label>
|
||||||
<input
|
<input
|
||||||
@@ -28,15 +28,11 @@
|
|||||||
name="key"
|
name="key"
|
||||||
required
|
required
|
||||||
autocomplete="current-password"
|
autocomplete="current-password"
|
||||||
class="w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-blue-500"
|
|
||||||
placeholder="Enter your signing key"
|
placeholder="Enter your signing key"
|
||||||
>
|
>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<button
|
<button type="submit">
|
||||||
type="submit"
|
|
||||||
class="w-full bg-blue-600 text-white py-2 px-4 rounded-md hover:bg-blue-700 focus:outline-none focus:ring-2 focus:ring-blue-500 focus:ring-offset-2 transition-colors"
|
|
||||||
>
|
|
||||||
Login
|
Login
|
||||||
</button>
|
</button>
|
||||||
</form>
|
</form>
|
||||||
|
|||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"license": "GPL-3.0",
|
||||||
|
"devDependencies": {
|
||||||
|
"prettier": "3.8.1"
|
||||||
|
}
|
||||||
|
}
|
||||||
+107
-7
@@ -3,16 +3,33 @@
|
|||||||
# this repo. Idempotent: every install is guarded by a check so already
|
# this repo. Idempotent: every install is guarded by a check so already
|
||||||
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
||||||
# or apk (detected in that order); assumes NOTHING is present (not git,
|
# or apk (detected in that order); assumes NOTHING is present (not git,
|
||||||
# make, or go). The linter is never installed on the host: golangci-lint
|
# make, or go). Node is used directly if installed; otherwise it is
|
||||||
# runs only inside a container, Dockerfile.lint or the Dockerfile lint
|
# installed at a pinned version via nvm (installing nvm itself first,
|
||||||
# stage (see script/lint). A C compiler and the CGO image libraries
|
# from a hash-verified release archive, never curl | sh). The linter is
|
||||||
# (pkg-config, vips, libheif) are installed for the govips bindings.
|
# never installed on the host: golangci-lint runs only in the lint phase
|
||||||
# Both Dockerfiles run this script too, so their build dependencies are
|
# of the Dockerfile (see script/lint).
|
||||||
# the ones listed here.
|
#
|
||||||
|
# script/bootstrap git, make, Go, and Node, Yarn and the
|
||||||
|
# prettier in yarn.lock for script/fmt and
|
||||||
|
# script/fmt-check: all the host needs, as
|
||||||
|
# the checks compile pixa in Docker
|
||||||
|
# script/bootstrap --cgo git, make, Go, and a C compiler and the
|
||||||
|
# CGO image libraries (pkg-config, vips,
|
||||||
|
# libheif) for the govips bindings instead
|
||||||
|
# of Node: to compile pixa, in the
|
||||||
|
# Dockerfile's test phase and build stage,
|
||||||
|
# which format nothing
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
|
# Pinned versions, 2026-07-06
|
||||||
|
NODE_VERSION="22.17.0"
|
||||||
|
NVM_VERSION="0.40.3"
|
||||||
|
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
|
||||||
|
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
|
||||||
|
YARN_VERSION="1.22.22"
|
||||||
|
|
||||||
PKGMGR=""
|
PKGMGR=""
|
||||||
SUDO=""
|
SUDO=""
|
||||||
|
|
||||||
@@ -35,6 +52,10 @@ detect_pkgmgr() {
|
|||||||
if [ "$(id -u)" != "0" ]; then
|
if [ "$(id -u)" != "0" ]; then
|
||||||
SUDO="sudo"
|
SUDO="sudo"
|
||||||
fi
|
fi
|
||||||
|
# This runs before the first install only. A fresh image, such
|
||||||
|
# as a CI runner's, has no package lists, and apt-get install
|
||||||
|
# finds no package without them.
|
||||||
|
$SUDO apt-get update
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -53,6 +74,69 @@ missing() {
|
|||||||
! command -v "$1" >/dev/null 2>&1
|
! command -v "$1" >/dev/null 2>&1
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# verify_sha256 <file> <expected-hash>
|
||||||
|
verify_sha256() {
|
||||||
|
if command -v sha256sum >/dev/null 2>&1; then
|
||||||
|
actual="$(sha256sum "$1" | cut -d' ' -f1)"
|
||||||
|
else
|
||||||
|
actual="$(shasum -a 256 "$1" | cut -d' ' -f1)"
|
||||||
|
fi
|
||||||
|
if [ "$actual" != "$2" ]; then
|
||||||
|
echo "bootstrap: sha256 mismatch for $1" >&2
|
||||||
|
echo " expected: $2" >&2
|
||||||
|
echo " actual: $actual" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# nvm is a bash script; run a command in a bash with nvm loaded
|
||||||
|
nvm_sh() {
|
||||||
|
bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*"
|
||||||
|
}
|
||||||
|
|
||||||
|
ensure_nvm() {
|
||||||
|
[ -s "$HOME/.nvm/nvm.sh" ] && return 0
|
||||||
|
# nvm prerequisites; nvm itself requires bash
|
||||||
|
if missing bash; then pkg_install bash bash bash bash; fi
|
||||||
|
if missing curl; then pkg_install curl curl curl curl; fi
|
||||||
|
if missing git; then pkg_install git git git git; fi
|
||||||
|
tmp="$(mktemp -d)"
|
||||||
|
curl -fsSL -o "$tmp/nvm.tar.gz" \
|
||||||
|
"https://github.com/nvm-sh/nvm/archive/refs/tags/v${NVM_VERSION}.tar.gz"
|
||||||
|
verify_sha256 "$tmp/nvm.tar.gz" "$NVM_SHA256"
|
||||||
|
mkdir -p "$HOME/.nvm"
|
||||||
|
tar -xzf "$tmp/nvm.tar.gz" -C "$HOME/.nvm" --strip-components=1
|
||||||
|
rm -rf "$tmp"
|
||||||
|
}
|
||||||
|
|
||||||
|
ensure_node() {
|
||||||
|
if ! missing node; then return 0; fi
|
||||||
|
ensure_nvm
|
||||||
|
nvm_sh "nvm install $NODE_VERSION"
|
||||||
|
}
|
||||||
|
|
||||||
|
ensure_yarn() {
|
||||||
|
if ! missing yarn; then return 0; fi
|
||||||
|
if ! missing corepack; then
|
||||||
|
corepack enable
|
||||||
|
corepack prepare "yarn@$YARN_VERSION" --activate
|
||||||
|
elif [ -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
|
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
|
||||||
|
corepack prepare yarn@$YARN_VERSION --activate"
|
||||||
|
else
|
||||||
|
npm install -g "yarn@$YARN_VERSION"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
install_js_deps() {
|
||||||
|
if missing yarn && [ -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
|
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
|
||||||
|
yarn install --frozen-lockfile"
|
||||||
|
else
|
||||||
|
yarn install --frozen-lockfile
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
# CGO dependencies for govips (image processing)
|
# CGO dependencies for govips (image processing)
|
||||||
ensure_cgo_deps() {
|
ensure_cgo_deps() {
|
||||||
# cgo compiles with gcc on Linux; build-base and build-essential
|
# cgo compiles with gcc on Linux; build-base and build-essential
|
||||||
@@ -71,7 +155,16 @@ ensure_cgo_deps() {
|
|||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
echo "usage: script/bootstrap [--cgo]" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
|
case "$*" in
|
||||||
|
"" | --cgo) ;;
|
||||||
|
*) usage ;;
|
||||||
|
esac
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
|
|
||||||
# Base tooling
|
# Base tooling
|
||||||
@@ -81,8 +174,15 @@ main() {
|
|||||||
# Go toolchain
|
# Go toolchain
|
||||||
if missing go; then pkg_install go golang go go; fi
|
if missing go; then pkg_install go golang go go; fi
|
||||||
|
|
||||||
# CGO image libraries
|
# CGO image libraries where pixa is compiled; elsewhere Node, Yarn
|
||||||
|
# and prettier
|
||||||
|
if [ "$*" = "--cgo" ]; then
|
||||||
ensure_cgo_deps
|
ensure_cgo_deps
|
||||||
|
else
|
||||||
|
ensure_node
|
||||||
|
ensure_yarn
|
||||||
|
install_js_deps
|
||||||
|
fi
|
||||||
|
|
||||||
go mod download
|
go mod download
|
||||||
|
|
||||||
|
|||||||
+3
-2
@@ -1,7 +1,8 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/check: run all checks (test, lint, fmt-check). Our own
|
# script/check: run all checks (test, lint, fmt-check). Our own
|
||||||
# extension to scripts-to-rule-them-all. Must not modify any files.
|
# extension to scripts-to-rule-them-all. test and lint are Docker
|
||||||
# Generic: usually needs no adaptation.
|
# phases; fmt-check is native, because a formatter writes the working
|
||||||
|
# tree. Must not modify any files.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
|||||||
+20
-9
@@ -1,18 +1,29 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/cibuild: run the CI build. The Dockerfile runs the checks
|
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
||||||
# (make fmt-check, lint, test) as build steps. This script passes a new
|
# checks out and runs this and nothing else, and script/fmt-check runs
|
||||||
# CHECK_EPOCH on every run, so Docker runs those steps instead of
|
# the formatter on the host, which a pristine checkout cannot do.
|
||||||
# reusing cached results: a successful run means the checks ran and
|
# --no-cache for the same reason as script/docker: the gate phases the
|
||||||
# passed on this tree. Generic: needs no adaptation. The Gitea workflow
|
# final stage depends on are RUN steps, and a cached one is a check that
|
||||||
# runs this on push.
|
# did not run.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
epoch="$(date +%s)$$"
|
"$SCRIPT_DIR/bootstrap"
|
||||||
docker build --build-arg CHECK_EPOCH="$epoch" .
|
"$SCRIPT_DIR/check"
|
||||||
|
# Own line: a failing command substitution inside an argument does
|
||||||
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
|
# empty constant. VERSION is computed here because .dockerignore
|
||||||
|
# excludes .git, so `git describe` in a build stage yields an empty
|
||||||
|
# version without failing.
|
||||||
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
|
[ -n "$version" ] || version="unknown"
|
||||||
|
docker build --no-cache \
|
||||||
|
--build-arg VERSION="$version" \
|
||||||
|
-t "$("$SCRIPT_DIR/projectname")" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+12
-6
@@ -1,9 +1,8 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/docker: build the Docker image tagged with the project name.
|
# script/docker: build the Docker image tagged with the project name.
|
||||||
# Identical in all repos; the tag comes from script/projectname. Like
|
# Identical in all repos; the tag comes from script/projectname.
|
||||||
# script/cibuild, it passes a new CHECK_EPOCH, so the build runs the
|
# --no-cache because the gate phases the final stage depends on are RUN
|
||||||
# checks instead of reusing cached results. Generic: needs no
|
# steps, and a cached one is a check that did not run.
|
||||||
# adaptation.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -11,8 +10,15 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
epoch="$(date +%s)$$"
|
# Own line: a failing command substitution inside an argument does
|
||||||
docker build --build-arg CHECK_EPOCH="$epoch" \
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
|
# empty constant. VERSION is computed here because .dockerignore
|
||||||
|
# excludes .git, so `git describe` in a build stage yields an empty
|
||||||
|
# version without failing.
|
||||||
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
|
[ -n "$version" ] || version="unknown"
|
||||||
|
docker build --no-cache \
|
||||||
|
--build-arg VERSION="$version" \
|
||||||
-t "$("$SCRIPT_DIR/projectname")" .
|
-t "$("$SCRIPT_DIR/projectname")" .
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+20
@@ -4,11 +4,31 @@ set -eu
|
|||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
|
# Must match the pin in script/bootstrap.
|
||||||
|
NODE_VERSION="22.17.0"
|
||||||
|
|
||||||
|
# script/bootstrap installs node and yarn under nvm and leaves neither
|
||||||
|
# on the PATH of the shell that called it, so resolve the pinned
|
||||||
|
# toolchain here the way bootstrap's own install step does. nvm is a
|
||||||
|
# bash script, hence the subshell.
|
||||||
|
run_yarn() {
|
||||||
|
if command -v yarn >/dev/null 2>&1; then
|
||||||
|
exec yarn "$@"
|
||||||
|
fi
|
||||||
|
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
|
echo "fmt: no yarn; run script/bootstrap first" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
|
||||||
|
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
|
||||||
|
}
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
echo "Formatting code..."
|
echo "Formatting code..."
|
||||||
# shellcheck disable=SC2046 # word splitting of file list is wanted
|
# shellcheck disable=SC2046 # word splitting of file list is wanted
|
||||||
gofmt -w $(find . -name '*.go' -not -path './vendor/*')
|
gofmt -w $(find . -name '*.go' -not -path './vendor/*')
|
||||||
|
run_yarn run prettier --write '**/*.md' --tab-width 4 --prose-wrap always
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
@@ -5,6 +5,25 @@ set -eu
|
|||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
|
# Must match the pin in script/bootstrap.
|
||||||
|
NODE_VERSION="22.17.0"
|
||||||
|
|
||||||
|
# script/bootstrap installs node and yarn under nvm and leaves neither
|
||||||
|
# on the PATH of the shell that called it, so resolve the pinned
|
||||||
|
# toolchain here the way bootstrap's own install step does. nvm is a
|
||||||
|
# bash script, hence the subshell.
|
||||||
|
run_yarn() {
|
||||||
|
if command -v yarn >/dev/null 2>&1; then
|
||||||
|
exec yarn "$@"
|
||||||
|
fi
|
||||||
|
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
|
echo "fmt-check: no yarn; run script/bootstrap first" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
|
||||||
|
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
|
||||||
|
}
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
echo "Checking formatting..."
|
echo "Checking formatting..."
|
||||||
@@ -13,6 +32,7 @@ main() {
|
|||||||
gofmt -l . | grep -v '^vendor/'
|
gofmt -l . | grep -v '^vendor/'
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
run_yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
@@ -1,13 +1,13 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/install-precommit: install the git pre-commit hook that runs
|
# script/install-precommit: install the git pre-commit hook that runs
|
||||||
# script/precommit. Our own extension to scripts-to-rule-them-all.
|
# script/precommit. Our own extension to scripts-to-rule-them-all.
|
||||||
# Generic: needs no adaptation.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
|
hook=".git/hooks/pre-commit"
|
||||||
printf '#!/bin/sh\nset -e\nscript/precommit\n' > .git/hooks/pre-commit
|
printf '#!/bin/sh\nset -e\nscript/precommit\n' > .git/hooks/pre-commit
|
||||||
chmod +x .git/hooks/pre-commit
|
chmod +x .git/hooks/pre-commit
|
||||||
echo "pre-commit hook installed: runs script/precommit"
|
echo "pre-commit hook installed: runs script/precommit"
|
||||||
|
|||||||
+13
-27
@@ -1,37 +1,23 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/lint: run golangci-lint over the whole tree. This is the only
|
# script/lint: run the linter. Linting is a phase of the Dockerfile and
|
||||||
# way the linter is run, everywhere; it is never installed on the host.
|
# this builds that phase alone; the linter is never installed or run on
|
||||||
|
# a developer host, where a shared result cache and a host-global lock
|
||||||
|
# make its answer untrustworthy.
|
||||||
#
|
#
|
||||||
# Inside a container it runs the linter. Anywhere else it builds
|
# The phase is not the last stage in the file, so it is built only when
|
||||||
# Dockerfile.lint, whose last step runs this script again inside that
|
# --target names it. --no-cache because a cached lint layer is a lint
|
||||||
# container.
|
# that did not run. The tag makes each build replace the previous image
|
||||||
#
|
# instead of leaving a dangling one behind.
|
||||||
# Dockerfile.lint and the Dockerfile lint stage set container=docker
|
|
||||||
# (the systemd convention for marking a container) to say where we are.
|
|
||||||
# /.dockerenv cannot: it is missing inside build steps, and present on
|
|
||||||
# hosts that are themselves containers.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
if [ "${container:-}" = docker ]; then
|
docker build --no-cache \
|
||||||
# `golangci-lint config verify` is not run: it fetches its JSON
|
--target lint \
|
||||||
# schema over an unpinned live HTTPS call, which REPO_POLICIES.md
|
-t "$("$SCRIPT_DIR/projectname")-lint" .
|
||||||
# forbids.
|
|
||||||
echo "Running linter..."
|
|
||||||
golangci-lint run --config .golangci.yml ./...
|
|
||||||
else
|
|
||||||
# A new CACHEBUST on every run means the lint step is never
|
|
||||||
# served from cache (see Dockerfile.lint). The cacheonly output
|
|
||||||
# leaves no image behind.
|
|
||||||
docker build \
|
|
||||||
--progress=plain \
|
|
||||||
--build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--output=type=cacheonly \
|
|
||||||
-f Dockerfile.lint .
|
|
||||||
fi
|
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
Executable
+177
@@ -0,0 +1,177 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# script/loadtest: measure pixad's throughput, latency and peak memory.
|
||||||
|
#
|
||||||
|
# script/loadtest [duration [clients]] (defaults: 10s and 4)
|
||||||
|
#
|
||||||
|
# A benchmark, not a check: script/check does not run it. It needs Docker
|
||||||
|
# and Go. It builds the image with script/docker and builds vegeta, the
|
||||||
|
# load tool, from a pinned commit. Each scenario then gets a new pixad
|
||||||
|
# container and a new origin container (cmd/loadtest-origin, which answers
|
||||||
|
# every path with the same JPEG), and vegeta sends requests for <duration>
|
||||||
|
# from <clients> clients at once, each asking for an image resized to
|
||||||
|
# 400x300 WebP:
|
||||||
|
#
|
||||||
|
# hit the same image every time, put in the cache first
|
||||||
|
# miss a new source image every time
|
||||||
|
# herd each new source image once per client in a row, so that all
|
||||||
|
# clients ask for it at the same time
|
||||||
|
#
|
||||||
|
# For each, it prints vegeta's report, pixad's peak resident memory and
|
||||||
|
# how many requests reached the origin. README.md says how to read them.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
||||||
|
|
||||||
|
# vegeta v12.13.0, 2026-10-04
|
||||||
|
VEGETA_COMMIT=4b240c3089fa4aa10816542d64a74294d974211f
|
||||||
|
|
||||||
|
# pixad refuses upstream hosts with private or local addresses, so the
|
||||||
|
# containers share a network in 203.0.113.0/24, a range set aside for
|
||||||
|
# documentation (RFC 5737) that pixad does not refuse and that is never
|
||||||
|
# routed on the internet.
|
||||||
|
SUBNET=203.0.113.0/24
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
echo "usage: script/loadtest [duration [clients]]" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
duration="${1:-10s}"
|
||||||
|
clients="${2:-4}"
|
||||||
|
# The duration is a whole number, not zero (vegeta takes 0 to mean no
|
||||||
|
# end), followed by ms, s, m or h.
|
||||||
|
case "$duration" in
|
||||||
|
*ms) number="${duration%ms}" ;;
|
||||||
|
*s | *m | *h) number="${duration%?}" ;;
|
||||||
|
*) usage ;;
|
||||||
|
esac
|
||||||
|
case "$number" in
|
||||||
|
"" | *[!0-9]*) usage ;;
|
||||||
|
esac
|
||||||
|
[ "$number" -gt 0 ] || usage
|
||||||
|
# The number of clients is a whole number that does not start with 0,
|
||||||
|
# which also refuses zero: vegeta reads a leading 0 as octal.
|
||||||
|
case "$clients" in
|
||||||
|
*[!0-9]* | 0*) usage ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
cd "$ROOT"
|
||||||
|
run="pixa-loadtest-$$"
|
||||||
|
tmp="$(mktemp -d)"
|
||||||
|
trap cleanup EXIT
|
||||||
|
trap 'exit 1' HUP INT TERM
|
||||||
|
|
||||||
|
"$SCRIPT_DIR/docker"
|
||||||
|
# The image's ID, so a build elsewhere that moves the tag does not
|
||||||
|
# change what a later scenario starts.
|
||||||
|
image="$(docker image inspect --format '{{.Id}}' \
|
||||||
|
"$("$SCRIPT_DIR/projectname")")"
|
||||||
|
|
||||||
|
GOBIN="$tmp" go install "github.com/tsenart/vegeta/v12@$VEGETA_COMMIT"
|
||||||
|
# The origin runs in a container, so it is built for the Docker host.
|
||||||
|
CGO_ENABLED=0 GOOS=linux \
|
||||||
|
GOARCH="$(docker version --format '{{.Server.Arch}}')" \
|
||||||
|
go build -o "$tmp/loadtest-origin" ./cmd/loadtest-origin
|
||||||
|
|
||||||
|
docker network create --subnet "$SUBNET" "$run" >/dev/null
|
||||||
|
|
||||||
|
start_containers
|
||||||
|
# Put the image the hit scenario asks for in the cache.
|
||||||
|
docker exec "$run-pixad" wget -q -O /dev/null \
|
||||||
|
"http://localhost:8080/v1/image/origin/hit.jpg/400x300.webp"
|
||||||
|
attack hit hit_targets
|
||||||
|
stop_containers
|
||||||
|
|
||||||
|
start_containers
|
||||||
|
attack miss miss_targets
|
||||||
|
stop_containers
|
||||||
|
|
||||||
|
start_containers
|
||||||
|
attack herd herd_targets
|
||||||
|
stop_containers
|
||||||
|
}
|
||||||
|
|
||||||
|
# start_containers starts a new origin and a new pixad, and waits up to 30
|
||||||
|
# seconds for pixad's health check to pass.
|
||||||
|
start_containers() {
|
||||||
|
docker run -d --name "$run-origin" \
|
||||||
|
--network "$run" --network-alias origin \
|
||||||
|
-v "$tmp/loadtest-origin:/usr/local/bin/loadtest-origin:ro" \
|
||||||
|
--entrypoint /usr/local/bin/loadtest-origin "$image" >/dev/null
|
||||||
|
docker run -d --name "$run-pixad" \
|
||||||
|
--network "$run" -p 127.0.0.1::8080 --health-interval=1s \
|
||||||
|
-e PIXA_SIGNING_KEY="$(head -c 32 /dev/urandom | base64)" \
|
||||||
|
-e PIXA_ALLOWLIST_HOSTS=origin -e PIXA_ALLOW_HTTP=true \
|
||||||
|
"$image" >/dev/null
|
||||||
|
|
||||||
|
waited=0
|
||||||
|
until [ "$(docker inspect --format '{{.State.Health.Status}}' \
|
||||||
|
"$run-pixad")" = healthy ]; do
|
||||||
|
if [ "$waited" -ge 30 ]; then
|
||||||
|
echo "loadtest: pixad not healthy after 30 seconds; its log:" >&2
|
||||||
|
docker logs "$run-pixad" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
waited=$((waited + 1))
|
||||||
|
done
|
||||||
|
pixa="http://$(docker port "$run-pixad" 8080/tcp)"
|
||||||
|
}
|
||||||
|
|
||||||
|
stop_containers() {
|
||||||
|
docker rm -f "$run-pixad" "$run-origin" >/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
# attack <scenario> <targets>: send the requests <targets> prints and
|
||||||
|
# report on them.
|
||||||
|
attack() {
|
||||||
|
echo
|
||||||
|
echo "== $1: $clients clients for $duration"
|
||||||
|
"$2" | "$tmp/vegeta" attack -lazy -rate 0 -workers "$clients" \
|
||||||
|
-max-workers "$clients" -duration "$duration" -max-body 0 |
|
||||||
|
"$tmp/vegeta" report
|
||||||
|
# pixad is process 1 in its container: the entrypoint execs it.
|
||||||
|
echo "pixad peak memory (VmHWM):" \
|
||||||
|
"$(docker exec "$run-pixad" awk '/^VmHWM:/ { print $2, $3 }' \
|
||||||
|
/proc/1/status)"
|
||||||
|
echo "requests to the origin:" \
|
||||||
|
"$(docker logs "$run-origin" 2>&1 | grep -c ' request ')"
|
||||||
|
}
|
||||||
|
|
||||||
|
# The targets functions print vegeta targets until vegeta stops reading.
|
||||||
|
|
||||||
|
hit_targets() {
|
||||||
|
while :; do
|
||||||
|
echo "GET $pixa/v1/image/origin/hit.jpg/400x300.webp"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
miss_targets() {
|
||||||
|
i=0
|
||||||
|
while :; do
|
||||||
|
i=$((i + 1))
|
||||||
|
echo "GET $pixa/v1/image/origin/miss/$i.jpg/400x300.webp"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
herd_targets() {
|
||||||
|
i=0
|
||||||
|
while :; do
|
||||||
|
i=$((i + 1))
|
||||||
|
n=0
|
||||||
|
while [ "$n" -lt "$clients" ]; do
|
||||||
|
n=$((n + 1))
|
||||||
|
echo "GET $pixa/v1/image/origin/herd/$i.jpg/400x300.webp"
|
||||||
|
done
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
docker rm -f "$run-pixad" "$run-origin" >/dev/null 2>&1 || :
|
||||||
|
docker network rm "$run" >/dev/null 2>&1 || :
|
||||||
|
rm -rf "$tmp"
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
+1
-2
@@ -1,7 +1,6 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/setup: set up the repo for development after a fresh clone:
|
# script/setup: set up the repo for development after a fresh clone:
|
||||||
# installs dependencies (script/bootstrap) and the git pre-commit hook.
|
# installs dependencies and the git pre-commit hook.
|
||||||
# Add any repo-specific initialization (db init, .env template) here.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
|||||||
+10
-18
@@ -1,27 +1,19 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/test: run the test suite. CGO dependencies (pkg-config, vips,
|
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
||||||
# libheif) come from nix-shell when not already available (e.g. inside
|
# and this builds that phase alone, on the same terms as script/lint:
|
||||||
# a Docker build or an existing nix-shell).
|
# --target because a phase that is not the last stage is built only when
|
||||||
|
# named, --no-cache because a cached test layer is a test that did not
|
||||||
|
# run, and a tag so each build replaces the previous image.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
||||||
run_with_cgo_deps() {
|
|
||||||
if command -v pkg-config >/dev/null 2>&1; then
|
|
||||||
sh -c "$1"
|
|
||||||
else
|
|
||||||
nix-shell -p pkg-config vips libheif git --run "$1"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
echo "Running tests..."
|
docker build --no-cache \
|
||||||
# Run without -v first for clean output on success; on failure rerun
|
--target test \
|
||||||
# with -v for full diagnostics, then exit non-zero (REPO_POLICIES.md
|
-t "$("$SCRIPT_DIR/projectname")-test" .
|
||||||
# conditional-verbose-rerun pattern). The first run already proved the
|
|
||||||
# tests broken, so the build fails even if the rerun happens to pass.
|
|
||||||
run_with_cgo_deps "CGO_ENABLED=1 go test -timeout 30s -race -cover ./... || { echo '--- Rerunning with -v for details ---'; CGO_ENABLED=1 go test -timeout 30s -race -v ./...; exit 1; }"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
@@ -1,147 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
#
|
|
||||||
# Manual test script for pixa server
|
|
||||||
# Requires: server running on localhost:8080
|
|
||||||
#
|
|
||||||
set -e
|
|
||||||
|
|
||||||
BASE_URL="${BASE_URL:-http://localhost:8080}"
|
|
||||||
SIGNING_KEY="${SIGNING_KEY:-test-signing-key-for-development-only}"
|
|
||||||
TEST_IMAGE_URL="https://s3.sneak.cloud/sneak-public/2021/2021-04-18.untitled.a7r4.07723.jpg"
|
|
||||||
COOKIE_JAR=$(mktemp)
|
|
||||||
|
|
||||||
cleanup() {
|
|
||||||
rm -f "$COOKIE_JAR"
|
|
||||||
}
|
|
||||||
trap cleanup EXIT
|
|
||||||
|
|
||||||
pass() {
|
|
||||||
echo "✓ PASS: $1"
|
|
||||||
}
|
|
||||||
|
|
||||||
fail() {
|
|
||||||
echo "✗ FAIL: $1"
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
|
|
||||||
echo "=== Pixa Manual Test Suite ==="
|
|
||||||
echo "Base URL: $BASE_URL"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Test 1: Healthcheck
|
|
||||||
echo "--- Test 1: Healthcheck endpoint ---"
|
|
||||||
HEALTH=$(curl -sf "$BASE_URL/.well-known/healthcheck.json")
|
|
||||||
if echo "$HEALTH" | grep -q '"status"'; then
|
|
||||||
pass "Healthcheck returns status"
|
|
||||||
else
|
|
||||||
fail "Healthcheck did not return expected response"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Test 2: Login page displays
|
|
||||||
echo "--- Test 2: Login page (GET /) ---"
|
|
||||||
LOGIN_PAGE=$(curl -sf "$BASE_URL/")
|
|
||||||
if echo "$LOGIN_PAGE" | grep -qi "password\|login\|sign"; then
|
|
||||||
pass "Login page displays password form"
|
|
||||||
else
|
|
||||||
fail "Login page did not display expected content"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Test 3: Wrong password shows error
|
|
||||||
echo "--- Test 3: Login with wrong password ---"
|
|
||||||
WRONG_LOGIN=$(curl -sf -X POST "$BASE_URL/" -d "key=wrong-key" -c "$COOKIE_JAR")
|
|
||||||
if echo "$WRONG_LOGIN" | grep -qi "invalid\|error\|incorrect\|wrong"; then
|
|
||||||
pass "Wrong password shows error message"
|
|
||||||
else
|
|
||||||
fail "Wrong password did not show error"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Test 4: Correct password redirects to generator
|
|
||||||
echo "--- Test 4: Login with correct signing key ---"
|
|
||||||
curl -sf -X POST "$BASE_URL/" -d "key=$SIGNING_KEY" -c "$COOKIE_JAR" -b "$COOKIE_JAR" -L -o /dev/null
|
|
||||||
GENERATOR_PAGE=$(curl -sf "$BASE_URL/" -b "$COOKIE_JAR")
|
|
||||||
if echo "$GENERATOR_PAGE" | grep -qi "generate\|url\|source\|logout"; then
|
|
||||||
pass "Correct password shows generator page"
|
|
||||||
else
|
|
||||||
fail "Generator page not displayed after login"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Test 5: Generate encrypted URL
|
|
||||||
echo "--- Test 5: Generate encrypted URL ---"
|
|
||||||
GEN_RESULT=$(curl -sf -X POST "$BASE_URL/generate" -b "$COOKIE_JAR" \
|
|
||||||
-d "url=$TEST_IMAGE_URL" \
|
|
||||||
-d "width=800" \
|
|
||||||
-d "height=600" \
|
|
||||||
-d "format=jpeg" \
|
|
||||||
-d "quality=85" \
|
|
||||||
-d "fit=cover" \
|
|
||||||
-d "ttl=3600")
|
|
||||||
if echo "$GEN_RESULT" | grep -q "/v1/e/"; then
|
|
||||||
pass "Encrypted URL generated"
|
|
||||||
# Extract the encrypted URL
|
|
||||||
ENC_URL=$(echo "$GEN_RESULT" | grep -o '/v1/e/[^"<]*' | head -1)
|
|
||||||
echo " Generated URL: $ENC_URL"
|
|
||||||
else
|
|
||||||
fail "Failed to generate encrypted URL"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Test 6: Fetch image via encrypted URL
|
|
||||||
echo "--- Test 6: Fetch image via encrypted URL ---"
|
|
||||||
if [ -n "$ENC_URL" ]; then
|
|
||||||
HTTP_CODE=$(curl -sf -o /dev/null -w "%{http_code}" "$BASE_URL$ENC_URL")
|
|
||||||
if [ "$HTTP_CODE" = "200" ]; then
|
|
||||||
pass "Encrypted URL returns image (HTTP 200)"
|
|
||||||
else
|
|
||||||
fail "Encrypted URL returned HTTP $HTTP_CODE"
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
fail "No encrypted URL to test"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Test 7: Fetch image via allowlisted host (direct proxy)
|
|
||||||
echo "--- Test 7: Fetch image via direct proxy (allowlisted host) ---"
|
|
||||||
# URL format: /v1/image/<host>/<path>/<WxH>.<format>
|
|
||||||
PROXY_PATH="/v1/image/s3.sneak.cloud/sneak-public/2021/2021-04-18.untitled.a7r4.07723.jpg/400x300.jpeg"
|
|
||||||
HTTP_CODE=$(curl -sf -o /dev/null -w "%{http_code}" "$BASE_URL$PROXY_PATH")
|
|
||||||
if [ "$HTTP_CODE" = "200" ]; then
|
|
||||||
pass "Direct proxy returns image (HTTP 200)"
|
|
||||||
else
|
|
||||||
fail "Direct proxy returned HTTP $HTTP_CODE"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Test 8: Logout
|
|
||||||
echo "--- Test 8: Logout ---"
|
|
||||||
curl -sf "$BASE_URL/logout" -b "$COOKIE_JAR" -c "$COOKIE_JAR" -L -o /dev/null
|
|
||||||
AFTER_LOGOUT=$(curl -sf "$BASE_URL/" -b "$COOKIE_JAR")
|
|
||||||
if echo "$AFTER_LOGOUT" | grep -qi "password\|login"; then
|
|
||||||
pass "Logout redirects to login page"
|
|
||||||
else
|
|
||||||
fail "Logout did not redirect to login"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Test 9: Generate short-TTL URL and verify expiration
|
|
||||||
echo "--- Test 9: Expired URL returns 410 ---"
|
|
||||||
# Login again
|
|
||||||
curl -sf -X POST "$BASE_URL/" -d "key=$SIGNING_KEY" -c "$COOKIE_JAR" -b "$COOKIE_JAR" -L -o /dev/null
|
|
||||||
# Generate URL with 1 second TTL
|
|
||||||
GEN_RESULT=$(curl -sf -X POST "$BASE_URL/generate" -b "$COOKIE_JAR" \
|
|
||||||
-d "url=$TEST_IMAGE_URL" \
|
|
||||||
-d "width=100" \
|
|
||||||
-d "height=100" \
|
|
||||||
-d "format=jpeg" \
|
|
||||||
-d "ttl=1")
|
|
||||||
SHORT_URL=$(echo "$GEN_RESULT" | grep -o '/v1/e/[^"<]*' | head -1)
|
|
||||||
if [ -n "$SHORT_URL" ]; then
|
|
||||||
echo " Waiting 2 seconds for URL to expire..."
|
|
||||||
sleep 2
|
|
||||||
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$BASE_URL$SHORT_URL")
|
|
||||||
if [ "$HTTP_CODE" = "410" ]; then
|
|
||||||
pass "Expired URL returns 410 Gone"
|
|
||||||
else
|
|
||||||
fail "Expired URL returned HTTP $HTTP_CODE (expected 410)"
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
fail "Could not generate short-TTL URL"
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "=== All tests passed! ==="
|
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
|
||||||
|
# yarn lockfile v1
|
||||||
|
|
||||||
|
|
||||||
|
prettier@3.8.1:
|
||||||
|
version "3.8.1"
|
||||||
|
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
|
||||||
|
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==
|
||||||
Reference in New Issue
Block a user