Compare commits
3
Commits
next
..
6572b3fe08
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6572b3fe08 | ||
|
|
1ba4c8d39f | ||
|
|
74ce122dc4 |
+9
-18
@@ -13,21 +13,12 @@
|
|||||||
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
|
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
|
||||||
# deletes the package directory from the context.
|
# deletes the package directory from the context.
|
||||||
|
|
||||||
# .git is sent without its config. Without a VERSION build argument the
|
# Unlike the standard file, which leaves out all of .git, pixa sends
|
||||||
# stage that compiles runs `git describe --tags --always` on .git, which
|
# .git without its config. Without a VERSION build argument the stage
|
||||||
# does not need .git/config; that file can hold a credential, such as a
|
# that compiles runs `git describe --tags --always` on .git, which does
|
||||||
|
# not need .git/config; that file can hold a credential, such as a
|
||||||
# password in a remote URL or the token the CI checkout step stores there.
|
# password in a remote URL or the token the CI checkout step stores there.
|
||||||
# Each submodule keeps a config with the same exposure in its git directory
|
.git/config
|
||||||
# under .git/modules/, nested again for a submodule's own submodules, or in
|
|
||||||
# its own .git directory when it keeps one.
|
|
||||||
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
|
|
||||||
# `deploy/config`, `config/lib`) loses its whole git directory, because
|
|
||||||
# `**/.git/modules/**/config` also matches that segment's directory
|
|
||||||
# under .git/modules/. Go's version stamping then fails the build;
|
|
||||||
# nothing leaks. Name such a submodule without that segment:
|
|
||||||
# `git submodule add --name`.
|
|
||||||
**/.git/config
|
|
||||||
**/.git/modules/**/config
|
|
||||||
|
|
||||||
# Agent scratch: one full checkout of the repo per in-flight agent.
|
# Agent scratch: one full checkout of the repo per in-flight agent.
|
||||||
# Anchored because it occurs once where agents run at the repo root.
|
# Anchored because it occurs once where agents run at the repo root.
|
||||||
@@ -51,9 +42,7 @@
|
|||||||
**/[iI][dD]_[rR][sS][aA]
|
**/[iI][dD]_[rR][sS][aA]
|
||||||
**/[iI][dD]_[dD][sS][aA]
|
**/[iI][dD]_[dD][sS][aA]
|
||||||
**/[iI][dD]_[eE][cC][dD][sS][aA]
|
**/[iI][dD]_[eE][cC][dD][sS][aA]
|
||||||
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
|
|
||||||
**/[iI][dD]_[eE][dD]25519
|
**/[iI][dD]_[eE][dD]25519
|
||||||
**/[iI][dD]_[eE][dD]25519_[sS][kK]
|
|
||||||
|
|
||||||
# Dependencies: restored inside the image, never copied in.
|
# Dependencies: restored inside the image, never copied in.
|
||||||
**/node_modules
|
**/node_modules
|
||||||
@@ -71,8 +60,10 @@
|
|||||||
**/.vscode
|
**/.vscode
|
||||||
**/*.sublime-*
|
**/*.sublime-*
|
||||||
|
|
||||||
# pixa's own entries. On the host, `make build` writes bin/pixad, and the
|
# pixa's own entries. Nothing in the build reads .gitignore. On the
|
||||||
# example config keeps its state directory in data/.
|
# host, `make build` writes bin/pixad, and the example config keeps its
|
||||||
|
# state directory in data/.
|
||||||
|
.gitignore
|
||||||
/bin
|
/bin
|
||||||
/data
|
/data
|
||||||
|
|
||||||
|
|||||||
@@ -13,6 +13,3 @@ indent_style = tab
|
|||||||
|
|
||||||
[*.go]
|
[*.go]
|
||||||
indent_style = tab
|
indent_style = tab
|
||||||
|
|
||||||
# This repository's own sections, such as one for another language it
|
|
||||||
# uses, go below this comment, and a re-vendor keeps them.
|
|
||||||
|
|||||||
@@ -1,21 +1,15 @@
|
|||||||
name: check
|
name: check
|
||||||
on: [push]
|
on: [push]
|
||||||
# Free the shared runner: a new push cancels only the same branch's older run.
|
|
||||||
concurrency:
|
|
||||||
group: ${{ github.workflow }}-${{ github.ref }}
|
|
||||||
cancel-in-progress: true
|
|
||||||
jobs:
|
jobs:
|
||||||
check:
|
check:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
# Free the shared runner from a hung build.
|
|
||||||
timeout-minutes: 20
|
|
||||||
steps:
|
steps:
|
||||||
# actions/checkout v4.2.2, 2026-02-22
|
# actions/checkout v4.2.2, 2026-02-22
|
||||||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
||||||
# script/cibuild needs no token, so none is left in .git/config.
|
# The default clone is shallow and has no tags, so the
|
||||||
|
# version the build takes from `git describe` would be a
|
||||||
|
# bare commit; this fetches the whole history with its tags.
|
||||||
with:
|
with:
|
||||||
persist-credentials: false
|
|
||||||
# All history and tags, so git describe finds the version tag.
|
|
||||||
fetch-depth: 0
|
fetch-depth: 0
|
||||||
- run: script/cibuild
|
- run: script/cibuild
|
||||||
- run: script/docker-smoke
|
- run: script/docker-smoke
|
||||||
|
|||||||
+6
-37
@@ -17,46 +17,15 @@ Thumbs.db
|
|||||||
# entry and must not be given a `**/` prefix on the way into one.
|
# entry and must not be given a `**/` prefix on the way into one.
|
||||||
.claude/
|
.claude/
|
||||||
|
|
||||||
# Node
|
# Environment / secrets
|
||||||
node_modules/
|
.env
|
||||||
|
.env.*
|
||||||
# Secrets. Unanchored like every entry above, so each matches at every
|
*.pem
|
||||||
# depth. Matching is case-sensitive on Linux, so names use character
|
*.key
|
||||||
# ranges rather than a lowercase form that misses `Server.Key`.
|
|
||||||
|
|
||||||
# Environment files. `*.env` covers bare `.env` and the `prod.env`
|
|
||||||
# convention. Only the templates `example.env` and `sample.env` are
|
|
||||||
# re-included below. A repository that commits any other template adds
|
|
||||||
# its own negation at the end of this file, for example `!.env.example`.
|
|
||||||
*.[eE][nN][vV]
|
|
||||||
.[eE][nN][vV].*
|
|
||||||
.[eE][nN][vV][rR][cC]
|
|
||||||
!example.env
|
|
||||||
!sample.env
|
|
||||||
|
|
||||||
# Private keys and the bundles carrying them.
|
|
||||||
*.[pP][eE][mM]
|
|
||||||
*.[kK][eE][yY]
|
|
||||||
*.[pP]12
|
|
||||||
*.[pP][fF][xX]
|
|
||||||
[iI][dD]_[rR][sS][aA]
|
|
||||||
[iI][dD]_[dD][sS][aA]
|
|
||||||
[iI][dD]_[eE][cC][dD][sS][aA]
|
|
||||||
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
|
|
||||||
[iI][dD]_[eE][dD]25519
|
|
||||||
[iI][dD]_[eE][dD]25519_[sS][kK]
|
|
||||||
|
|
||||||
# This repository's own entries, such as its build outputs, go below
|
|
||||||
# this comment, and a re-vendor keeps them. Anchor a binary built at the
|
|
||||||
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
|
|
||||||
|
|
||||||
# Go
|
|
||||||
*.log
|
|
||||||
*.out
|
|
||||||
*.test
|
|
||||||
|
|
||||||
# Dependencies
|
# Dependencies
|
||||||
vendor/
|
vendor/
|
||||||
|
node_modules/
|
||||||
|
|
||||||
# Build output
|
# Build output
|
||||||
/bin/
|
/bin/
|
||||||
|
|||||||
@@ -17,7 +17,6 @@ linters:
|
|||||||
disable:
|
disable:
|
||||||
# Genuinely incompatible with project patterns
|
# Genuinely incompatible with project patterns
|
||||||
- exhaustruct # Requires all struct fields
|
- exhaustruct # Requires all struct fields
|
||||||
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
|
|
||||||
- godot # Requires comments to end with periods
|
- godot # Requires comments to end with periods
|
||||||
- wrapcheck # Too verbose for internal packages
|
- wrapcheck # Too verbose for internal packages
|
||||||
- varnamelen # Short names like db, id are idiomatic Go
|
- varnamelen # Short names like db, id are idiomatic Go
|
||||||
@@ -25,8 +24,6 @@ linters:
|
|||||||
# silenced by disabling that name, not by enabling the successor.
|
# silenced by disabling that name, not by enabling the successor.
|
||||||
- wsl # Deprecated, replaced by wsl_v5
|
- wsl # Deprecated, replaced by wsl_v5
|
||||||
- gomodguard # Deprecated, replaced by gomodguard_v2
|
- gomodguard # Deprecated, replaced by gomodguard_v2
|
||||||
# Misses findings at random in v2.14.0; back once a pinned release fixes it
|
|
||||||
- canonicalheader
|
|
||||||
settings:
|
settings:
|
||||||
lll:
|
lll:
|
||||||
line-length: 88
|
line-length: 88
|
||||||
|
|||||||
@@ -1,2 +1,7 @@
|
|||||||
node_modules/
|
node_modules/
|
||||||
yarn.lock
|
yarn.lock
|
||||||
|
|
||||||
|
# A byte-for-byte copy of the one in sneak/prompts.
|
||||||
|
REPO_POLICIES.md
|
||||||
|
|
||||||
|
vendor/
|
||||||
|
|||||||
+30
-48
@@ -1,12 +1,7 @@
|
|||||||
# Every stage declares ARG CACHEBUST first: script/ passes a new value
|
|
||||||
# on every build, which reruns every RUN step after it. The steps that
|
|
||||||
# compile mount Go's build cache, which survives from build to build.
|
|
||||||
|
|
||||||
# Lint phase. script/lint builds it alone. The linter is run directly:
|
# Lint phase. script/lint builds it alone. The linter is run directly:
|
||||||
# `make lint` and script/lint are themselves a docker build.
|
# `make lint` and script/lint are themselves a docker build.
|
||||||
# golangci/golangci-lint:v2.14.0, 2026-10-15
|
# golangci/golangci-lint:v2.12.2, 2026-10-04
|
||||||
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
|
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
|
||||||
ARG CACHEBUST
|
|
||||||
|
|
||||||
# The linter compiles every package, and govips needs the libvips
|
# The linter compiles every package, and govips needs the libvips
|
||||||
# headers for that. REPO_POLICIES.md has the lint phase install them
|
# headers for that. REPO_POLICIES.md has the lint phase install them
|
||||||
@@ -19,20 +14,17 @@ WORKDIR /src
|
|||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN golangci-lint run --config .golangci.yml ./...
|
||||||
golangci-lint run --config .golangci.yml ./...
|
|
||||||
|
|
||||||
# Test phase. script/test builds it alone. -race needs cgo and so a C
|
# Test phase. script/test builds it alone.
|
||||||
# compiler, which the Debian Go image ships and the alpine one does not.
|
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
|
||||||
# golang:1.25.4-trixie, 2026-10-15
|
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS test
|
||||||
FROM golang@sha256:a02d35efc036053fdf0da8c15919276bf777a80cbfda6a35c5e9f087e652adfc AS test
|
|
||||||
ARG CACHEBUST
|
|
||||||
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
# script/bootstrap --cgo installs the build dependencies (the libvips
|
# script/bootstrap --cgo installs the build dependencies (a C compiler,
|
||||||
# and libheif headers, and libvips' JPEG XL support, which the tests
|
# the libvips and libheif headers, and libvips' JPEG XL support, which
|
||||||
# need) and downloads the Go modules.
|
# the tests need) and downloads the Go modules.
|
||||||
COPY script/ ./script/
|
COPY script/ ./script/
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN script/bootstrap --cgo
|
RUN script/bootstrap --cgo
|
||||||
@@ -40,14 +32,11 @@ RUN script/bootstrap --cgo
|
|||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# Without -v first; on a failure, again with -v for the details, and
|
# Without -v first; on a failure, again with -v for the details, and
|
||||||
# the step fails even if the second run passes. -count=1 because the
|
# the step fails even if the second run passes. -parallel 4: by default
|
||||||
# mounted cache also holds test results, which go test would otherwise
|
# a package runs as many of its tests at once as the host has CPUs, and
|
||||||
# replay. -parallel 4: by default a package runs as many of its tests at
|
# on a busy host they then wait so long to be scheduled that a test
|
||||||
# once as the host has CPUs, and on a busy host they then wait so long
|
# that times a request can see it return a second late.
|
||||||
# to be scheduled that a test that times a request can see it return a
|
RUN go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
|
||||||
# second late.
|
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
|
||||||
go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
|
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; }
|
go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; }
|
||||||
|
|
||||||
@@ -56,44 +45,38 @@ RUN --mount=type=cache,target=/root/.cache/go-build \
|
|||||||
# test passed.
|
# test passed.
|
||||||
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
|
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
|
||||||
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
|
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
|
||||||
ARG CACHEBUST
|
|
||||||
|
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
COPY --from=test /src/go.sum /dev/null
|
COPY --from=test /src/go.sum /dev/null
|
||||||
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
# Build dependencies, git among them, and Go modules, as in the test
|
# Build dependencies and Go modules, as in the test phase
|
||||||
# phase
|
|
||||||
COPY script/ ./script/
|
COPY script/ ./script/
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN script/bootstrap --cgo
|
RUN script/bootstrap --cgo
|
||||||
# A tar-stream context keeps the sender's file owners, which git refuses.
|
|
||||||
RUN git config --system --add safe.directory /src
|
|
||||||
|
|
||||||
# Copy source code
|
# Copy source code
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# The VERSION build arg when one is given, otherwise
|
# VERSION is declared here, not earlier: a new value reruns only the
|
||||||
# `git describe --tags --always` on the .git in the build context: the
|
# build, not script/bootstrap. Given none, the version is
|
||||||
# tag on a tagged commit, tag-N-gHASH after one, the short commit when
|
# `git describe --tags --always` of the .git in the build context (git
|
||||||
# no tag is reachable. With .git present, a version that is still empty,
|
# comes from script/bootstrap): the tag on a tagged commit, tag-N-gHASH
|
||||||
# dev or unknown fails the build: git is missing or could not read the
|
# after one, the short commit when no tag is reachable. A context that
|
||||||
# checkout. A context without .git, as from a source tarball, stamps an
|
# carries .git and still yields no version fails the build; one without
|
||||||
# empty version. CGO stays enabled for govips; -trimpath keeps build
|
# .git, as from a source tarball, stamps an empty version. CGO stays
|
||||||
# paths out of the binary, and -s -w leave out the symbol table and
|
# enabled for govips; -trimpath keeps build paths out of the binary, and
|
||||||
# debug information.
|
# -s -w leave out the symbol table and debug information.
|
||||||
ARG VERSION
|
ARG VERSION
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN version="${VERSION:-$(git describe --tags --always)}"; \
|
||||||
VERSION="${VERSION:-$(git describe --tags --always)}"; \
|
if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
|
||||||
if [ -e .git ]; then \
|
[ "$version" = unknown ]; }; then \
|
||||||
case "$VERSION" in ""|dev|unknown) \
|
echo "the build context carries .git but yields no version" >&2; \
|
||||||
echo "version is '$VERSION' although .git is present" >&2; \
|
exit 1; \
|
||||||
exit 1 ;; \
|
|
||||||
esac; \
|
|
||||||
fi; \
|
fi; \
|
||||||
CGO_ENABLED=1 GOTOOLCHAIN=auto go build -trimpath \
|
CGO_ENABLED=1 GOTOOLCHAIN=auto go build -trimpath \
|
||||||
-ldflags="-s -w -X main.Version=${VERSION}" \
|
-ldflags "-s -w -X main.Version=${version}" \
|
||||||
-o /pixad ./cmd/pixad
|
-o /pixad ./cmd/pixad
|
||||||
|
|
||||||
# Runtime stage, and the last one: a plain `docker build .` builds this
|
# Runtime stage, and the last one: a plain `docker build .` builds this
|
||||||
@@ -102,7 +85,6 @@ RUN --mount=type=cache,target=/root/.cache/go-build \
|
|||||||
# the libvips and musl it was built with.
|
# the libvips and musl it was built with.
|
||||||
# alpine:3.22, 2026-10-08
|
# alpine:3.22, 2026-10-08
|
||||||
FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8
|
FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8
|
||||||
ARG CACHEBUST
|
|
||||||
|
|
||||||
# Install runtime dependencies only. vips-jxl is libvips' JPEG XL
|
# Install runtime dependencies only. vips-jxl is libvips' JPEG XL
|
||||||
# support, without which pixad does not start.
|
# support, without which pixad does not start.
|
||||||
|
|||||||
@@ -71,8 +71,7 @@ The state directory (`state_dir`, `/var/lib/pixa` in the container) holds the
|
|||||||
database and the disk cache:
|
database and the disk cache:
|
||||||
|
|
||||||
- It needs a persistent volume: without one, every restart starts with an empty
|
- It needs a persistent volume: without one, every restart starts with an empty
|
||||||
cache. Losing the directory also ends every login session (see Encrypted
|
cache. In the container, the startup script gives the directory to the user
|
||||||
URLs). In the container, the startup script gives the directory to the user
|
|
||||||
pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
|
pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
|
||||||
must be able to write the directory.
|
must be able to write the directory.
|
||||||
- `cache_max_bytes` limits the source and transformed images together. The
|
- `cache_max_bytes` limits the source and transformed images together. The
|
||||||
@@ -124,41 +123,9 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
|
|||||||
comma-separated; default none. Every image on a listed host is served to
|
comma-separated; default none. Every image on a listed host is served to
|
||||||
anyone without a signature, so list only hosts you control, where nobody
|
anyone without a signature, so list only hosts you control, where nobody
|
||||||
else can put images.
|
else can put images.
|
||||||
- `PIXA_CACHE_MAX_BYTES`: disk cache limit in bytes; `0` disables it. Set
|
- `PIXA_CACHE_MAX_BYTES`: disk cache limit in bytes; `0` disables it;
|
||||||
it: the volume is a directory on the host's filesystem, shared with the
|
default 75% of (free space + what the cache holds)
|
||||||
host and every other app. The default is 75% of the sum of that
|
|
||||||
filesystem's free space and what the cache holds, and at least 500 MiB, so
|
|
||||||
the cache can grow to three quarters or more of the space they all have
|
|
||||||
left, and to all of it when that is 500 MiB or less.
|
|
||||||
- the rest are in the table under Configuration below
|
- the rest are in the table under Configuration below
|
||||||
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
|
|
||||||
the images it processes at once, plus `upstream_connections` times
|
|
||||||
`upstream_max_response_size` for the source images it has fetched that wait
|
|
||||||
for processing. 4.2 GiB is the most resident memory was measured to grow while
|
|
||||||
pixa, with pixad's libvips settings, processed one 8192x8192 source, the
|
|
||||||
largest it accepts, saving it at full size to each output format in turn, one
|
|
||||||
at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG,
|
|
||||||
an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point
|
|
||||||
JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as
|
|
||||||
WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64
|
|
||||||
times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
|
|
||||||
GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
|
|
||||||
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
|
|
||||||
use.
|
|
||||||
- **Temporary files:** while pixa processes a large source, libvips writes its
|
|
||||||
decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is
|
|
||||||
written, so a listing of `/tmp` does not show it, but it keeps its space, on
|
|
||||||
disk or, where `/tmp` is a tmpfs, in memory, until the image is done. For a
|
|
||||||
GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory as
|
|
||||||
well as written to that file, and the 4.2 GiB above includes that memory. For
|
|
||||||
an 8192x8192 source the file was measured on pixad at 192 MiB for an 8-bit RGB
|
|
||||||
JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG XL of
|
|
||||||
32-bit floating-point RGBA, the largest measured: up to 1 GiB per image
|
|
||||||
processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs.
|
|
||||||
This `/tmp` is in the container's own filesystem, not on the volume at
|
|
||||||
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
|
|
||||||
does not include these files; where `/tmp` is a tmpfs, they are memory on top
|
|
||||||
of it.
|
|
||||||
- **Health check:** the image's `HEALTHCHECK` requests
|
- **Health check:** the image's `HEALTHCHECK` requests
|
||||||
`/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
|
`/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
|
||||||
after a deploy and marks the deploy failed unless it is `healthy`. The probe
|
after a deploy and marks the deploy failed unless it is `healthy`. The probe
|
||||||
@@ -193,14 +160,6 @@ name.
|
|||||||
Multiple source paths may reference the same content blob; the database tracks
|
Multiple source paths may reference the same content blob; the database tracks
|
||||||
references rather than using filesystem refcounting.
|
references rather than using filesystem refcounting.
|
||||||
|
|
||||||
Source and transformed image files, and the `.meta` file beside each transformed
|
|
||||||
image, are written under a temporary name, synced to disk, and only then renamed
|
|
||||||
into place, so a crash leaves no file under its own name with part of its data.
|
|
||||||
When pixa starts, and every five minutes after, it removes each cached source or
|
|
||||||
transformed image whose file is empty, or not the size the database records for
|
|
||||||
it, with a warning naming the file; the next request for it fetches the source
|
|
||||||
or converts the image again.
|
|
||||||
|
|
||||||
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
|
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
|
||||||
Test). Toward it, pixa keeps in memory the content types of the 10,000
|
Test). Toward it, pixa keeps in memory the content types of the 10,000
|
||||||
transformed images most recently cached or served, so a cache hit on one of them
|
transformed images most recently cached or served, so a cache hit on one of them
|
||||||
@@ -225,11 +184,7 @@ path under `/v1/` answers 200, in maintenance mode too.
|
|||||||
session it answers 303 to `/`. Answers: 200 with the page showing the URL; 400
|
session it answers 303 to `/`. Answers: 200 with the page showing the URL; 400
|
||||||
with the page naming a field that is not valid; 500 when the URL cannot be
|
with the page naming a field that is not valid; 500 when the URL cannot be
|
||||||
made.
|
made.
|
||||||
- `POST /logout` — log out with the generator page's Logout button: end every
|
- `GET /logout` — end the login session. Needs: nothing. Answers: 303 to `/`.
|
||||||
login session, in every browser (see Encrypted URLs), and remove the session
|
|
||||||
cookie. Needs: a login session and the generator page's form (below); without
|
|
||||||
a login session it only removes the cookie. Answers: 303 to `/`; 500 when the
|
|
||||||
logout cannot be saved in the database.
|
|
||||||
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or
|
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or
|
||||||
`/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized
|
`/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized
|
||||||
and converted (below). Needs: a signature, unless the host is allowlisted (see
|
and converted (below). Needs: a signature, unless the host is allowlisted (see
|
||||||
@@ -277,7 +232,7 @@ the request carries the same ID as `request_id`, and so do the lines it logs
|
|||||||
when it fetches, converts and serves an image; the fetch sends it to the
|
when it fetches, converts and serves an image; the fetch sends it to the
|
||||||
upstream host as `X-Request-ID`.
|
upstream host as `X-Request-ID`.
|
||||||
|
|
||||||
The `POST` routes accept only a form that pixa's own page served: the page puts
|
Both `POST` routes accept only a form that pixa's own page served: the page puts
|
||||||
a token in the form and sets a cookie to match, and a request without both is
|
a token in the form and sets a cookie to match, and a request without both is
|
||||||
refused with 403, so another site cannot submit the form from a visitor's
|
refused with 403, so another site cannot submit the form from a visitor's
|
||||||
browser. The login and generator pages are meant to be opened over HTTPS: while
|
browser. The login and generator pages are meant to be opened over HTTPS: while
|
||||||
@@ -316,35 +271,6 @@ once, is refused with 400.
|
|||||||
both optional (values under Signature Specification). Both are part of what is
|
both optional (values under Signature Specification). Both are part of what is
|
||||||
cached, so each value of either is a separate cached image.
|
cached, so each value of either is a separate cached image.
|
||||||
|
|
||||||
A source URL's query follows its path after `%3F`, the encoded `?`:
|
|
||||||
`/v1/image/<host>/<path>%3F<query>/<size>.<format>`. pixa decodes the
|
|
||||||
percent-escapes in the image URL's path once, then reads the host up to the
|
|
||||||
first `/`, the size and format after the last `/`, and between them the source
|
|
||||||
path up to the first `?` and the source query after it. So the `?` before the
|
|
||||||
query, any `?` in the query, and every `%` in the source path or query must be
|
|
||||||
percent-encoded, as `%3F` and `%25`. Any other character may be written as
|
|
||||||
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. Neither
|
|
||||||
the source path nor the source query may hold a `#`, and an image URL that
|
|
||||||
writes one as `%23` is refused with 400. For example,
|
|
||||||
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at
|
|
||||||
800x600 WebP is
|
|
||||||
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp`,
|
|
||||||
signed as shown under Signature Specification.
|
|
||||||
|
|
||||||
The source path is the path with its percent-escapes decoded, and pixa fetches
|
|
||||||
it percent-encoded again: every character other than letters, digits, `-`, `.`,
|
|
||||||
`_`, `~`, `/`, `$`, `&`, `+`, `,`, `:`, `;`, `=` and `@` is sent as a
|
|
||||||
percent-escape. So `my%20cat.jpg` in the image URL is the source path
|
|
||||||
`/my cat.jpg`, fetched as `/my%20cat.jpg`; `my%2520cat.jpg` is the source path
|
|
||||||
`/my%20cat.jpg`, fetched as `/my%2520cat.jpg`; and `100%25.png` is fetched as
|
|
||||||
`/100%25.png`. The source query is the source URL's query as that URL writes it,
|
|
||||||
its percent-escapes included, and is sent as it is, except that the characters a
|
|
||||||
browser percent-encodes in a query (a space, `"`, `'`, `<`, `>`, control
|
|
||||||
characters and characters outside ASCII) and a `%` that does not begin a
|
|
||||||
percent-escape are percent-encoded. pixa signs and caches the query in that
|
|
||||||
form, so `img%3Fname=my%20cat` and `img%3Fname=my%2520cat` in the image URL are
|
|
||||||
the same source query, `name=my%20cat`.
|
|
||||||
|
|
||||||
A size with one side `0` has that side scaled to keep the source's proportions,
|
A size with one side `0` has that side scaled to keep the source's proportions,
|
||||||
and is refused with 400 when that side would come out above 8192, the most
|
and is refused with 400 when that side would come out above 8192, the most
|
||||||
either side may be; so is a size with `fit=outside`, which scales the image to
|
either side may be; so is a size with `fit=outside`, which scales the image to
|
||||||
@@ -393,12 +319,9 @@ fetch sends the first request's ID upstream, and the lines logged for the fetch
|
|||||||
and the transcode carry that ID.
|
and the transcode carry that ID.
|
||||||
|
|
||||||
A transcode whose request's `downstream_timeout` has ended (for a shared one,
|
A transcode whose request's `downstream_timeout` has ended (for a shared one,
|
||||||
the first request's) stops waiting for a processing slot, or, once it has one,
|
the first request's) stops when it gets its processing slot or before it begins
|
||||||
gives it back without beginning to encode. A request waiting for a shared
|
encoding, and gives the slot back. libvips cannot stop an encode once it has
|
||||||
transcode that stops this way is answered 503 with the error
|
begun, so an encode keeps its slot until it finishes, which can be after
|
||||||
`server busy, try again later`, though its own `downstream_timeout` may not have
|
|
||||||
ended. govips gives pixa no way to stop libvips once an encode has begun, so an
|
|
||||||
encode keeps its slot until it finishes, which can be after
|
|
||||||
`downstream_timeout`. The limit of 8192 on each side of the source and of the
|
`downstream_timeout`. The limit of 8192 on each side of the source and of the
|
||||||
output bounds how long that takes. With one libvips thread, as pixad runs it,
|
output bounds how long that takes. With one libvips thread, as pixad runs it,
|
||||||
encoding 8192x8192 random pixels takes up to about 1 second as JPEG, 15 as PNG,
|
encoding 8192x8192 random pixels takes up to about 1 second as JPEG, 15 as PNG,
|
||||||
@@ -444,28 +367,14 @@ nor change what it asks for.
|
|||||||
|
|
||||||
1. Open `/` in a browser over HTTPS (or over plain HTTP while `debug` is on, see
|
1. Open `/` in a browser over HTTPS (or over plain HTTP while `debug` is on, see
|
||||||
Routes) and log in with the signing key (`signing_key`). The login session
|
Routes) and log in with the signing key (`signing_key`). The login session
|
||||||
lasts 30 days, or until someone logs out with the generator page's Logout
|
lasts 30 days, or until `/logout`.
|
||||||
button. Every login is made with the one signing key, so logging out ends
|
|
||||||
every login session, in every browser, a copy of the session cookie included.
|
|
||||||
pixa keeps a random value in its database, which it makes when the database
|
|
||||||
has none and replaces at each logout. A login session carries the value
|
|
||||||
current at its login and is refused once the value has changed, also after a
|
|
||||||
restart (see Deployment). The check for a logout compares no time, so a login
|
|
||||||
after a logout works whatever the host's clock says; the 30 days are still
|
|
||||||
measured by the host's clock.
|
|
||||||
2. On the generator page, give the source image's URL, the width and height, the
|
2. On the generator page, give the source image's URL, the width and height, the
|
||||||
format, quality and fit, and how long the URL lasts, then submit the form
|
format, quality and fit, and how long the URL lasts, then submit the form
|
||||||
(`POST /generate`). The format is JPEG XL unless another is chosen; a form
|
(`POST /generate`). The format is JPEG XL unless another is chosen; a form
|
||||||
sent with an empty format, or none, also makes a JPEG XL URL. Width and
|
sent with an empty format, or none, also makes a JPEG XL URL. Width and
|
||||||
height both empty or `0` keep the original size; if only one of them is empty
|
height both empty or `0` keep the original size; if only one of them is empty
|
||||||
or `0`, that side is scaled to keep the image's proportions. A URL whose
|
or `0`, that side is scaled to keep the image's proportions. A URL whose
|
||||||
scaled side would come out above 8192 answers 400, and so does a source URL
|
scaled side would come out above 8192 answers 400.
|
||||||
whose path holds a `..` segment. So does a source URL that the image URL
|
|
||||||
would fetch written another way, and the page then shows the URL pixa would
|
|
||||||
fetch: one with a user name and password, which pixa does not send, or one
|
|
||||||
whose path holds `%2F`, fetched as `/`, or `(`, fetched as `%28`, as pixa
|
|
||||||
keeps the source path decoded and fetches it percent-encoded again (see
|
|
||||||
Routes).
|
|
||||||
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
|
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
|
||||||
it expires. `<host>` is the host the page was opened on, and the URL starts
|
it expires. `<host>` is the host the page was opened on, and the URL starts
|
||||||
with `http` instead while `debug` is on. The name after the token is ignored
|
with `http` instead while `debug` is on. The name after the token is ignored
|
||||||
@@ -473,11 +382,9 @@ nor change what it asks for.
|
|||||||
`jxl` for a form with no format.
|
`jxl` for a form with no format.
|
||||||
|
|
||||||
The token holds the source's host, path and query and the size, format, quality,
|
The token holds the source's host, path and query and the size, format, quality,
|
||||||
fit and expiry, encrypted with a key derived from `signing_key`. The host is
|
fit and expiry, encrypted with a key derived from `signing_key`. The source
|
||||||
kept in lower case and an empty path as `/`, as for `/v1/image/`, so
|
URL's scheme is not kept: the image is fetched like any other (see Routes), and
|
||||||
`https://CDN.Example.com` and `https://cdn.example.com/` are one source image,
|
the blocked networks still apply.
|
||||||
fetched and cached once. The source URL's scheme is not kept: the image is
|
|
||||||
fetched like any other (see Routes), and the blocked networks still apply.
|
|
||||||
|
|
||||||
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never.
|
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never.
|
||||||
The expiry is fixed in the token when the URL is made and cannot be changed or
|
The expiry is fixed in the token when the URL is made and cannot be changed or
|
||||||
@@ -529,20 +436,13 @@ Where:
|
|||||||
- `host` — source host as the URL writes it up to the first `/`, in lower case,
|
- `host` — source host as the URL writes it up to the first `/`, in lower case,
|
||||||
keeping an IPv6 address's brackets and any `:` and port after it (e.g.
|
keeping an IPv6 address's brackets and any `:` and port after it (e.g.
|
||||||
`cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`)
|
`cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`)
|
||||||
- `path` — source path, with its percent-escapes decoded as under Routes (e.g.
|
- `path` — source path (e.g. `/photos/cat.jpg`), with each `%` written as `%25`
|
||||||
`/photos/cat.jpg`, or `/photos/my cat.jpg` for `/photos/my%20cat.jpg`), with
|
and each `:` as `%3A`, so `/photos/a:b.jpg` is signed as `/photos/a%3Ab.jpg`
|
||||||
each `%` written as `%25` and each `:` as `%3A`, so `/photos/a:b.jpg` is
|
- `query` — source query string, empty string if none, with `%` and `:` written
|
||||||
signed as `/photos/a%3Ab.jpg`
|
as in `path`; neither then holds a `:`, so path `/a:b` with no query and path
|
||||||
- `query` — source query string as under Routes, empty string if none, with `%`
|
`/a` with query `b:` are signed differently
|
||||||
and `:` written as in `path`; neither then holds a `:`, so path `/a:b` with no
|
- `width` — requested width in pixels, `0` for original
|
||||||
query and path `/a` with query `b:` are signed differently
|
- `height` — requested height in pixels, `0` for original
|
||||||
- `width` — requested width in pixels, or `0`: the width is then worked out from
|
|
||||||
the source's proportions and the requested height, rounded down but at least
|
|
||||||
1, or is the source's own width when the height is `0` too (the `orig` size)
|
|
||||||
- `height` — requested height in pixels, or `0`: the height is then worked out
|
|
||||||
from the source's proportions and the requested width, rounded down but at
|
|
||||||
least 1, or is the source's own height when the width is `0` too (the `orig`
|
|
||||||
size)
|
|
||||||
- `format` — output format, one of those listed under Routes, with `original`
|
- `format` — output format, one of those listed under Routes, with `original`
|
||||||
signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no
|
signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no
|
||||||
format has the signature of the same URL ending in `.jxl`; `auto` is signed as
|
format has the signature of the same URL ending in `.jxl`; `auto` is signed as
|
||||||
@@ -555,10 +455,7 @@ Where:
|
|||||||
with 400
|
with 400
|
||||||
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
|
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
|
||||||
outside), or `cover` when the URL has no `fit`; a request whose `fit` is
|
outside), or `cover` when the URL has no `fit`; a request whose `fit` is
|
||||||
anything else, an empty `fit=` included, is refused with 400. `outside` scales
|
anything else, an empty `fit=` included, is refused with 400
|
||||||
the image to cover the requested size, keeping its proportions, so it can come
|
|
||||||
out larger than that size on one side: an 8192x1 source at `100x0`, which is
|
|
||||||
100x1 by its proportions, comes out at 8192x1
|
|
||||||
|
|
||||||
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of
|
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of
|
||||||
RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
|
RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
|
||||||
@@ -582,17 +479,6 @@ For the same image at quality 40 with fit `contain`, the input ends in
|
|||||||
and the URL is
|
and the URL is
|
||||||
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.
|
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.
|
||||||
|
|
||||||
For a source URL with a query,
|
|
||||||
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg`, at
|
|
||||||
800x600 WebP with the same key and expiration and the default quality and fit:
|
|
||||||
|
|
||||||
1. Build input, with the query's `:` written as `%3A` and its `%` as `%25`:
|
|
||||||
`cdn.example.com:/img:url=https%3A//origin.example.com/my%2520cat.jpg:800:600:webp:1704067200:85:cover`
|
|
||||||
2. The signature is `37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=`
|
|
||||||
3. URL, with the `?` before the query written as `%3F` and the query's `%` as
|
|
||||||
`%25` (see Routes):
|
|
||||||
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp?sig=37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=&exp=1704067200`
|
|
||||||
|
|
||||||
**Allowlist patterns:**
|
**Allowlist patterns:**
|
||||||
|
|
||||||
- **Exact match**: `cdn.example.com` — matches only that host
|
- **Exact match**: `cdn.example.com` — matches only that host
|
||||||
@@ -608,11 +494,7 @@ pixa lower-cases the host of a `/v1/image/` URL before anything else uses it, so
|
|||||||
`CDN.Example.com` is matched, fetched, cached and counted against
|
`CDN.Example.com` is matched, fetched, cached and counted against
|
||||||
`upstream_connections_per_host` as `cdn.example.com`. Such a URL whose host has
|
`upstream_connections_per_host` as `cdn.example.com`. Such a URL whose host has
|
||||||
a port, as in `cdn.example.com:8443`, or a `:` with no port, needs a signature
|
a port, as in `cdn.example.com:8443`, or a `:` with no port, needs a signature
|
||||||
even when the host is allowlisted. The host must be a host name or an IPv4
|
even when the host is allowlisted. A host holding `@` is refused with 400.
|
||||||
address (letters, digits, hyphens, underscores and dots) or an IPv6 address in
|
|
||||||
brackets, either one optionally followed by `:` and a port. Any other host, such
|
|
||||||
as one holding `@`, or a `?`, `#`, `%`, `\` or space written percent-encoded, is
|
|
||||||
refused with 400.
|
|
||||||
|
|
||||||
### Configuration
|
### Configuration
|
||||||
|
|
||||||
@@ -656,7 +538,7 @@ tell. With no file, pixa uses the environment and the defaults.
|
|||||||
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
|
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
|
||||||
| `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` |
|
| `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` |
|
||||||
| `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` |
|
| `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` |
|
||||||
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default half the CPUs, at least 1 |
|
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default the number of CPUs |
|
||||||
| `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` |
|
| `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` |
|
||||||
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` |
|
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` |
|
||||||
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB |
|
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB |
|
||||||
@@ -725,14 +607,11 @@ Key settings in more detail:
|
|||||||
- `signing_key` — HMAC secret for URL signatures
|
- `signing_key` — HMAC secret for URL signatures
|
||||||
- `db_url` — the SQLite database to open; omitted, it is
|
- `db_url` — the SQLite database to open; omitted, it is
|
||||||
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
|
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
|
||||||
database in WAL mode. `state_dir` is percent-encoded in it, so a `%`, `?` or
|
database in WAL mode. pixa opens one connection to it, so its own reads and
|
||||||
`#` in the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/`
|
writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to any
|
||||||
or one starting with `//` it starts `file:////`, so SQLite does not read the
|
`db_url`, so a write that finds another program writing to the file waits up
|
||||||
first name in the path as a host. pixa opens one connection to it, so its own
|
to five seconds for it instead of failing. WAL mode comes only from the URL:
|
||||||
reads and writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to
|
keep `_pragma=journal_mode(WAL)` in one you set
|
||||||
any `db_url`, so a write that finds another program writing to the file waits
|
|
||||||
up to five seconds for it instead of failing. WAL mode comes only from the
|
|
||||||
URL: keep `_pragma=journal_mode(WAL)` in one you set
|
|
||||||
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
|
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
|
||||||
cache entirely; omitted defaults to 75% of the sum of the free space on the
|
cache entirely; omitted defaults to 75% of the sum of the free space on the
|
||||||
filesystem containing `<state_dir>/cache/` and the bytes of source and
|
filesystem containing `<state_dir>/cache/` and the bytes of source and
|
||||||
@@ -745,12 +624,10 @@ Key settings in more detail:
|
|||||||
does, and `downstream_timeout` has not ended first, the request is answered
|
does, and `downstream_timeout` has not ended first, the request is answered
|
||||||
503 with the error `server busy, try again later`
|
503 with the error `server busy, try again later`
|
||||||
- `max_concurrent_processing` — the most images decoded and encoded at once;
|
- `max_concurrent_processing` — the most images decoded and encoded at once;
|
||||||
default half the CPUs pixa can use (`GOMAXPROCS`, which follows a container's
|
default the number of CPUs pixa can use (`GOMAXPROCS`), which follows a
|
||||||
CPU limit), rounded down, and at least 1. A value that is not a whole number
|
container's CPU limit. A request that finds all of them in use waits up to 10
|
||||||
of at least 1 aborts startup. A request that finds all of them in use waits up
|
seconds for one to free up; if none does, and `downstream_timeout` has not
|
||||||
to 10 seconds for one to free up; if none does, and `downstream_timeout` has
|
ended first, it is answered 503 the same way
|
||||||
not ended first, it is answered 503 the same way. Each one can take a lot of
|
|
||||||
memory: see "Running under upaas"
|
|
||||||
- `image_requests_per_minute` — how many requests one client may send to the
|
- `image_requests_per_minute` — how many requests one client may send to the
|
||||||
image routes per minute (see the limit on them under Routes); default `600`.
|
image routes per minute (see the limit on them under Routes); default `600`.
|
||||||
`0` turns the limit off, and a negative or non-integer value aborts startup.
|
`0` turns the limit off, and a negative or non-integer value aborts startup.
|
||||||
@@ -777,12 +654,7 @@ See `configs/config.example.yml` for all options with defaults.
|
|||||||
- **Metrics**: Prometheus, at `/metrics`: generic HTTP request metrics
|
- **Metrics**: Prometheus, at `/metrics`: generic HTTP request metrics
|
||||||
(duration, response size, requests in flight) and the Go runtime and process
|
(duration, response size, requests in flight) and the Go runtime and process
|
||||||
metrics; requests are measured and `/metrics` is served only when
|
metrics; requests are measured and `/metrics` is served only when
|
||||||
`metrics.username` and `metrics.password` are set. A request's `handler` label
|
`metrics.username` and `metrics.password` are set
|
||||||
is the pattern of the route it matches, such as `/v1/image/*`, never its path.
|
|
||||||
Requests that match no route share the label `unmatched`: those the router
|
|
||||||
answers 404 or 405, and CORS preflights. A request's `method` label is its
|
|
||||||
method when that is `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `DELETE`,
|
|
||||||
`OPTIONS`, `CONNECT` or `TRACE`, and `other` for any other method
|
|
||||||
- **Logging**: stdlib slog
|
- **Logging**: stdlib slog
|
||||||
|
|
||||||
## Entrypoints
|
## Entrypoints
|
||||||
@@ -801,9 +673,9 @@ provide:
|
|||||||
install-precommit)
|
install-precommit)
|
||||||
- `script/projectname` — output the project name ("pixa")
|
- `script/projectname` — output the project name ("pixa")
|
||||||
- `script/test` — run the test suite: build the `test` phase of the
|
- `script/test` — run the test suite: build the `test` phase of the
|
||||||
`Dockerfile`, writing no image
|
`Dockerfile`, tagged `pixa-test`
|
||||||
- `script/lint` — run golangci-lint: build the `lint` phase of the `Dockerfile`,
|
- `script/lint` — run golangci-lint: build the `lint` phase of the `Dockerfile`,
|
||||||
writing no image; the linter never runs on the host
|
tagged `pixa-lint`; the linter never runs on the host
|
||||||
- `script/fmt` — format the Go code with gofmt and the markdown with prettier
|
- `script/fmt` — format the Go code with gofmt and the markdown with prettier
|
||||||
(writes)
|
(writes)
|
||||||
- `script/fmt-check` — check the same formatting (read-only), on the host
|
- `script/fmt-check` — check the same formatting (read-only), on the host
|
||||||
@@ -821,20 +693,15 @@ provide:
|
|||||||
- `script/install-precommit` — install the git pre-commit hook that runs
|
- `script/install-precommit` — install the git pre-commit hook that runs
|
||||||
`script/precommit`
|
`script/precommit`
|
||||||
|
|
||||||
Every `docker build` in these scripts passes a new value of the `CACHEBUST`
|
Every `docker build` in these scripts passes `--no-cache`, so the lint and test
|
||||||
build argument, which every stage of the `Dockerfile` declares first, so every
|
phases run on every build instead of coming from the build cache.
|
||||||
`RUN` step, the lint and test phases included, runs on every build instead of
|
|
||||||
coming from the build cache. Go's build cache is kept from build to build: the
|
|
||||||
steps that compile mount it, so a package whose inputs are unchanged is not
|
|
||||||
compiled again, and both `go test` runs pass `-count=1` so that no stored test
|
|
||||||
result is replayed. The `test` phase runs on the Debian Go image, with Debian's
|
|
||||||
libvips and libheif; the image pixa ships is built and runs on Alpine.
|
|
||||||
`script/check`, `script/cibuild`, `script/docker`, `script/lint`, `script/test`,
|
`script/check`, `script/cibuild`, `script/docker`, `script/lint`, `script/test`,
|
||||||
`script/setup` and `script/install-precommit` are the standard copies from
|
`script/setup` and `script/install-precommit` are the standard copies from
|
||||||
`sneak/prompts`, kept identical to them. `script/fmt` and `script/fmt-check` are
|
`sneak/prompts`, kept identical to them. `script/fmt` and `script/fmt-check` are
|
||||||
the standard copies with pixa's `gofmt` step kept before prettier. prettier
|
the standard copies with pixa's `gofmt` step kept before prettier. prettier
|
||||||
formats the markdown only, not the HTML templates, as it cannot parse a Go
|
formats the markdown only: not the HTML templates, as it cannot parse a Go
|
||||||
template action inside a tag.
|
template action inside a tag, and not `REPO_POLICIES.md` (see
|
||||||
|
`.prettierignore`), a copy of the one in `sneak/prompts`.
|
||||||
|
|
||||||
## Load Test
|
## Load Test
|
||||||
|
|
||||||
|
|||||||
+105
-258
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Repository Policies
|
title: Repository Policies
|
||||||
last_modified: 2026-10-15
|
last_modified: 2026-09-08
|
||||||
---
|
---
|
||||||
|
|
||||||
This document covers repository structure, tooling, and workflow standards. Code
|
This document covers repository structure, tooling, and workflow standards. Code
|
||||||
@@ -54,39 +54,34 @@ style conventions are in separate documents:
|
|||||||
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
|
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
|
||||||
assumes nothing is present: base tools come from nix, apt, brew, or apk
|
assumes nothing is present: base tools come from nix, apt, brew, or apk
|
||||||
(detected in that order; apt runs noninteractive). For node it uses the
|
(detected in that order; apt runs noninteractive). For node it uses the
|
||||||
installed node only when its major version is the pinned one; otherwise (node
|
installed node if present; otherwise it installs a PINNED node version via
|
||||||
missing, or another major, such as the node 24 the CI runner image ships) it
|
nvm, first installing nvm itself if missing — from a hash-verified GitHub
|
||||||
installs the PINNED node version via nvm and installs yarn and the packages
|
release archive (never `curl | sh`), with bash installed as an explicit
|
||||||
under it. nvm itself is installed first if missing, from a hash-verified
|
prerequisite since nvm requires bash. yarn is then pinned via
|
||||||
GitHub release archive (never `curl | sh`), with bash installed as an explicit
|
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
|
||||||
prerequisite since nvm requires bash. Only the major version is compared
|
always exact versions. `script/cibuild` runs the CI build: it changes to the
|
||||||
because the `Dockerfile` stages start from a node image whose exact version is
|
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
|
||||||
not the pin, and nvm cannot install a prebuilt node on alpine. yarn is pinned
|
with the version; the Gitea workflow calls it. **`script/cibuild` runs
|
||||||
via `corepack prepare yarn@<version> --activate`. Never install "latest" or
|
|
||||||
"lts"; always exact versions. `script/cibuild` runs the CI build: it changes
|
|
||||||
to the repo root, runs `script/bootstrap`, runs `script/check`, and builds the
|
|
||||||
image with the version; the Gitea workflow calls it. **`script/cibuild` runs
|
|
||||||
`script/bootstrap` first**, because the workflow checks out the repo and runs
|
`script/bootstrap` first**, because the workflow checks out the repo and runs
|
||||||
nothing else, while `script/fmt-check` runs the formatter on the host: on a
|
nothing else, while `script/fmt-check` runs the formatter on the host: on a
|
||||||
pristine checkout with nothing installed the run dies there, after the
|
pristine checkout with nothing installed the run dies there, after the
|
||||||
containerised gates have passed. **The bootstrap alone is not enough**: when
|
containerised gates have passed. **The bootstrap alone is not enough**:
|
||||||
`script/bootstrap` installs node and yarn under nvm it leaves neither on the
|
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
|
||||||
`PATH` of the shell that called it, so a bare `yarn` either exits 127 or runs
|
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
|
||||||
under another node. The host entrypoints that need yarn — `script/fmt` and
|
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
|
||||||
`script/fmt-check` — therefore use the same test as `script/bootstrap`: when
|
source nvm for the pinned node version before invoking it, exactly as
|
||||||
the node on `PATH` has the pinned major version they run the `yarn` on `PATH`,
|
`script/bootstrap`'s own install step does. A runner carrying nothing but
|
||||||
and otherwise they load nvm through `$HOME/.nvm/nvm.sh` and run yarn under the
|
docker and git then gets through `script/check`. Four further scripts are our
|
||||||
pinned node. A runner carrying nothing but docker and git then gets through
|
own extensions to the standard: `script/check` runs `script/test`,
|
||||||
`script/check`. Four further scripts are our own extensions to the standard:
|
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
|
||||||
`script/check` runs `script/test`, `script/lint` and `script/fmt-check`;
|
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
|
||||||
`script/precommit` is what the git pre-commit hook runs, and it calls
|
installs the git pre-commit hook (the `make hooks` target shims to it); and
|
||||||
`script/check`; `script/install-precommit` installs the git pre-commit hook
|
`script/projectname` (literally that filename) simply outputs the project's
|
||||||
(the `make hooks` target shims to it); and `script/projectname` (literally
|
name. Scripts that need the name call `script/projectname` — e.g.
|
||||||
that filename) simply outputs the project's name. Scripts that need the name
|
`script/docker` assembles its image tag from it — so those scripts stay
|
||||||
call `script/projectname` — e.g. `script/docker` assembles its image tag from
|
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
|
||||||
it — so those scripts stay byte-identical across all repos. Repo-type-specific
|
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
|
||||||
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
|
the hook itself. Model scripts are at
|
||||||
`script/precommit`, not in the hook itself. Model scripts are at
|
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
||||||
must document the provided scripts in an **Entrypoints** section (see the
|
must document the provided scripts in an **Entrypoints** section (see the
|
||||||
README requirements below).
|
README requirements below).
|
||||||
@@ -109,24 +104,18 @@ style conventions are in separate documents:
|
|||||||
`lint` phase and a `test` phase, with the final stage depending on both so the
|
`lint` phase and a `test` phase, with the final stage depending on both so the
|
||||||
image cannot be built unless they pass. For non-server repos the final stage
|
image cannot be built unless they pass. For non-server repos the final stage
|
||||||
brings up a development environment; for server repos it is the runtime image.
|
brings up a development environment; for server repos it is the runtime image.
|
||||||
The gate phases and the build stage start from their pinned base images and
|
Dockerfiles install development prerequisites by running `script/bootstrap`
|
||||||
install what those images lack either inline, as the canonical Go `Dockerfile`
|
rather than duplicating installs inline; COPY `script/` and the dependency
|
||||||
below does for `git`, or by running `script/bootstrap`, as the `prompts`
|
manifests (`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before
|
||||||
repo's own `Dockerfile` does for its yarn packages. The development
|
running it.
|
||||||
environment stage installs development prerequisites by running
|
|
||||||
`script/bootstrap` rather than duplicating its installs inline. A stage that
|
|
||||||
runs `script/bootstrap` COPYs `script/` and the dependency manifests
|
|
||||||
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
|
|
||||||
|
|
||||||
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
|
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
|
||||||
no separate lint file. `script/lint` and `script/test` each build one phase
|
no separate lint file. `script/lint` and `script/test` each build one phase
|
||||||
and nothing else:
|
and nothing else:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
|
||||||
--target lint --output type=cacheonly .
|
docker build --no-cache --target test -t "$(script/projectname)-test" .
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--target test --output type=cacheonly .
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**A stage that is not the last one in the file is built only when the final
|
**A stage that is not the last one in the file is built only when the final
|
||||||
@@ -136,15 +125,10 @@ style conventions are in separate documents:
|
|||||||
plain `docker build .` builds the last stage alone and exits 0 having linted
|
plain `docker build .` builds the last stage alone and exits 0 having linted
|
||||||
and tested nothing.
|
and tested nothing.
|
||||||
|
|
||||||
**The gate builds write no image.** With `--output type=cacheonly` the phase
|
**Every `docker build` in `script/` is tagged**, here and in
|
||||||
runs and a failing step fails the build, but the result is not exported.
|
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
|
||||||
Nothing uses those images, and writing one out is slow: a Go test phase's
|
image behind on every invocation, on every developer host and every CI
|
||||||
image holds the toolchain. A build given neither `--output` nor `-t` writes
|
runner; a tagged one replaces the previous image.
|
||||||
an untagged image and leaves it dangling, on every developer host and every
|
|
||||||
CI runner. `script/cibuild` and `script/docker` build the image that ships
|
|
||||||
and tag it, so each build replaces the previous image; each assigns the tag
|
|
||||||
on its own line before the build, so `set -e` stops it where
|
|
||||||
`script/projectname` fails.
|
|
||||||
|
|
||||||
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
|
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
|
||||||
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
|
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
|
||||||
@@ -159,107 +143,66 @@ style conventions are in separate documents:
|
|||||||
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
|
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
|
||||||
exit non-zero with `parallel golangci-lint is running` — a status a caller
|
exit non-zero with `parallel golangci-lint is running` — a status a caller
|
||||||
cannot tell from real findings. Both have produced wrong verdicts in this
|
cannot tell from real findings. Both have produced wrong verdicts in this
|
||||||
org, in both directions. A container has its own golangci-lint cache, its
|
org, in both directions. A container has its own cache, its own `TMPDIR` and
|
||||||
own `TMPDIR` and a digest-pinned binary, so neither is reachable.
|
a digest-pinned binary, so neither is reachable.
|
||||||
|
|
||||||
- **Any build that runs checks is given a new `CACHEBUST` value.** Docker
|
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
|
||||||
invalidates a `COPY` layer only when the copied content changes, so on an
|
a `COPY` layer only when the copied content changes, so on an unchanged tree
|
||||||
unchanged tree the check `RUN` is served from cache, nothing executes, and the
|
the check `RUN` is served from cache, nothing executes, and the build still
|
||||||
build still exits 0. Every `docker build` in `script/` therefore passes
|
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
|
||||||
`--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the script's process
|
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
|
||||||
id, so no two builds share a value, not even the three in one
|
four, and there is no fifth — `script/check` runs the two gate phases and
|
||||||
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
|
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
|
||||||
`script/docker` are the four, and there is no fifth — `script/check` runs the
|
not evidence that anything ran: a sub-second build reporting success is a
|
||||||
two gate phases and `script/fmt-check`, and builds no image of its own. Every
|
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
|
||||||
stage of the `Dockerfile` declares `ARG CACHEBUST` as its first line after
|
and friends destroy a build cache shared with every other build on the host.
|
||||||
`FROM`, and a new value reruns every `RUN` step after that line; `COPY` steps
|
|
||||||
are compared by content and need nothing. A stage without the `ARG` line is
|
|
||||||
served from cache on an unchanged tree, so a repository re-vendors the scripts
|
|
||||||
and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`:
|
|
||||||
it reruns every step too, but it also empties every cache mount, and Go's
|
|
||||||
build cache, which the canonical Go `Dockerfile` below mounts so that compiled
|
|
||||||
packages survive between check builds, would start empty every time. A bare
|
|
||||||
`docker build .` is not evidence that anything ran: a sub-second build
|
|
||||||
reporting success is a cache hit, not a result. Never invalidate by pruning —
|
|
||||||
`docker builder prune` and friends destroy a build cache shared with every
|
|
||||||
other build on the host. When a check is added or changed, prove it works by
|
|
||||||
planting a defect it must catch and watching the run fail on it, then revert
|
|
||||||
the defect. A green run alone shows neither that the check ran nor that it
|
|
||||||
covers what it should.
|
|
||||||
|
|
||||||
- **The gate phases are separate stages, and the build stage depends on both.**
|
- **The gate phases are separate stages, and the build stage depends on both.**
|
||||||
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
||||||
hash), so lint failures surface in seconds rather than after a full compile,
|
hash), so lint failures surface in seconds rather than after a full compile,
|
||||||
and the test phase is based on the Debian Go image. The canonical Go repo
|
and the test phase is based on the Go image. The canonical Go repo
|
||||||
`Dockerfile`:
|
`Dockerfile`:
|
||||||
|
|
||||||
```dockerfile
|
```dockerfile
|
||||||
# Every stage declares ARG CACHEBUST first: script/ passes a new value
|
|
||||||
# on every build, which reruns every RUN step after it. The steps that
|
|
||||||
# compile mount Go's build cache, which survives from build to build.
|
|
||||||
|
|
||||||
# Lint phase
|
# Lint phase
|
||||||
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
||||||
FROM golangci/golangci-lint@sha256:... AS lint
|
FROM golangci/golangci-lint@sha256:... AS lint
|
||||||
ARG CACHEBUST
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN golangci-lint run --config .golangci.yml ./...
|
||||||
golangci-lint run --config .golangci.yml ./...
|
|
||||||
|
|
||||||
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
# Test phase
|
||||||
# image ships and the alpine one does not. -count=1 because the mounted
|
# golang:1.x-alpine, YYYY-MM-DD
|
||||||
# cache also holds test results, which go test would otherwise replay.
|
|
||||||
# golang:1.x, YYYY-MM-DD
|
|
||||||
FROM golang@sha256:... AS test
|
FROM golang@sha256:... AS test
|
||||||
ARG CACHEBUST
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN go test -timeout 90s -race -cover ./... || \
|
||||||
go test -count=1 -timeout 90s -race -cover ./... || \
|
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
|
go test -timeout 90s -race -v ./...; exit 1; }
|
||||||
|
|
||||||
# Build stage. Nothing is wanted from either phase above; the copies
|
# Build stage. Nothing is wanted from either phase above; the copies
|
||||||
# are what make BuildKit build them first, so this stage cannot run
|
# are what make BuildKit build them first, so this stage cannot run
|
||||||
# unless lint and test passed.
|
# unless lint and test passed.
|
||||||
# golang:1.x-alpine, YYYY-MM-DD
|
# golang:1.x-alpine, YYYY-MM-DD
|
||||||
FROM golang@sha256:... AS builder
|
FROM golang@sha256:... AS builder
|
||||||
ARG CACHEBUST
|
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
COPY --from=test /src/go.sum /dev/null
|
COPY --from=test /src/go.sum /dev/null
|
||||||
RUN apk add --no-cache git
|
|
||||||
# A tar-stream context keeps the sender's file owners, which git refuses.
|
|
||||||
RUN git config --system --add safe.directory /src
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# The VERSION build arg when one is given, otherwise
|
ARG VERSION=dev
|
||||||
# `git describe --tags --always` on the .git in the build context. With
|
RUN CGO_ENABLED=0 go build -trimpath \
|
||||||
# .git present, a version that is still empty, dev or unknown fails the
|
-ldflags="-s -w -X main.Version=${VERSION}" \
|
||||||
# build: git is missing or could not read the checkout.
|
-o /app ./cmd/app/
|
||||||
ARG VERSION
|
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
|
||||||
VERSION="${VERSION:-$(git describe --tags --always)}"; \
|
|
||||||
if [ -e .git ]; then \
|
|
||||||
case "$VERSION" in ""|dev|unknown) \
|
|
||||||
echo "version is '$VERSION' although .git is present" >&2; \
|
|
||||||
exit 1 ;; \
|
|
||||||
esac; \
|
|
||||||
fi; \
|
|
||||||
CGO_ENABLED=0 go build -trimpath \
|
|
||||||
-ldflags="-s -w -X main.Version=${VERSION}" \
|
|
||||||
-o /app ./cmd/app/
|
|
||||||
|
|
||||||
# Runtime stage, and the last one
|
# Runtime stage, and the last one
|
||||||
FROM alpine@sha256:...
|
FROM alpine@sha256:...
|
||||||
ARG CACHEBUST
|
|
||||||
COPY --from=builder /app /usr/local/bin/app
|
COPY --from=builder /app /usr/local/bin/app
|
||||||
ENTRYPOINT ["app"]
|
ENTRYPOINT ["app"]
|
||||||
```
|
```
|
||||||
@@ -274,88 +217,23 @@ style conventions are in separate documents:
|
|||||||
- Keep the runtime stage last, and if you add a stage after it, give it the
|
- Keep the runtime stage last, and if you add a stage after it, give it the
|
||||||
same two copies. A plain `docker build .` builds the last stage's chain
|
same two copies. A plain `docker build .` builds the last stage's chain
|
||||||
and nothing else.
|
and nothing else.
|
||||||
- Only Go's build cache is mounted (`/root/.cache/go-build`, where Go keeps
|
|
||||||
it for root), and only on the steps that compile: `golangci-lint run`,
|
|
||||||
both `go test` runs and `go build`. It keeps compiled packages between
|
|
||||||
check builds, and Go reuses one only when all its inputs are unchanged, so
|
|
||||||
it saves the compile without skipping a check. Nothing else is kept:
|
|
||||||
golangci-lint's own cache would replay stored findings in place of a lint
|
|
||||||
run, and Go builds from whatever files a kept module cache holds without
|
|
||||||
checking them again (only `go mod verify` does), so a file changed there
|
|
||||||
would go unnoticed. Go's build cache also holds test results, which is why
|
|
||||||
both `go test` runs carry `-count=1`.
|
|
||||||
- If the project uses `//go:embed` directives that reference build artifacts
|
- If the project uses `//go:embed` directives that reference build artifacts
|
||||||
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
||||||
create placeholder files so the embed directives resolve. Example:
|
create placeholder files so the embed directives resolve. Example:
|
||||||
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
|
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
|
||||||
- If the project requires CGO or system libraries for linting, install them
|
- If the project requires CGO or system libraries for linting (e.g.
|
||||||
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
|
`vips-dev`), install them in the lint phase with `apk add`.
|
||||||
has no `apk`, so install with `apt-get` under the Debian package name
|
- `ARG VERSION=dev` is declared in the stage that compiles and supplied by
|
||||||
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
|
`script/docker` and `script/cibuild`; no stage may call `git describe`.
|
||||||
lists in the same `RUN`, so the layer does not keep them:
|
|
||||||
|
|
||||||
```dockerfile
|
|
||||||
RUN apt-get update \
|
|
||||||
&& apt-get install -y --no-install-recommends libvips-dev \
|
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
|
||||||
```
|
|
||||||
|
|
||||||
- `.dockerignore` lets `.git` into the build context. It keeps out every git
|
|
||||||
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
|
|
||||||
repository's own, each submodule's under `.git/modules/`, and that of a
|
|
||||||
submodule keeping its own `.git` directory. `git describe` does not need
|
|
||||||
them, and each can hold a credential: a password in a remote URL, or the
|
|
||||||
token the CI checkout step stores there. A submodule whose name has a
|
|
||||||
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
|
|
||||||
git directory to `**/.git/modules/**/config`, and Go's version stamping
|
|
||||||
then fails the build: give it a name without that segment
|
|
||||||
(`git submodule add --name`). The stage that compiles has `git` (the
|
|
||||||
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
|
|
||||||
takes the version from the `VERSION` build argument when one is given,
|
|
||||||
otherwise from `git describe --tags --always`. That gives the tag on a
|
|
||||||
tagged commit; on a later commit, the tag, the number of commits since it
|
|
||||||
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
|
|
||||||
tag is reachable. The stage that compiles also marks its working directory
|
|
||||||
safe for git (`git config --system --add safe.directory /src`): a context
|
|
||||||
sent as a tar stream keeps the sender's file owners, and git refuses a
|
|
||||||
checkout owned by another user, so the version would come out empty.
|
|
||||||
`ARG VERSION` has no default, and the build fails if the context carries
|
|
||||||
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
|
|
||||||
`docker build .` with no build arguments must succeed; a Dockerfile that
|
|
||||||
refuses an empty build argument drops that refusal and keeps the argument.
|
|
||||||
A checkout whose `.git` is a file (a linked worktree, or a repository
|
|
||||||
checked out as a submodule) is the exception: that file points to a git
|
|
||||||
directory outside the build context, so the build cannot read the version
|
|
||||||
and a plain `docker build .` fails; pass the version with
|
|
||||||
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
|
|
||||||
do.
|
|
||||||
|
|
||||||
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
||||||
runs `script/cibuild` on push, and checks out the repo as its only other step,
|
runs `script/cibuild` on push, and checks out the repo as its only other step.
|
||||||
with `persist-credentials: false`: `script/cibuild` needs no token, and
|
That script bootstraps, runs the gate phases, and then builds the image, so a
|
||||||
without it the checkout leaves the job's token in `.git/config` for every
|
successful run means every check passed; a bare `docker build .` does not
|
||||||
later step. The checkout step also sets `fetch-depth: 0`, which fetches the
|
|
||||||
tags `git describe` needs: by default it clones shallow with no tags, and a
|
|
||||||
tagged repository's CI build would stamp a bare short commit id. The
|
|
||||||
workflow's `concurrency` block groups runs by workflow and branch
|
|
||||||
(`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`,
|
|
||||||
so a new push cancels the older run on the same branch, queued or running, and
|
|
||||||
no other: runs for replaced commits do not hold up the shared runner.
|
|
||||||
`script/cibuild` bootstraps, runs the gate phases, and then builds the image,
|
|
||||||
so a successful run means every check passed; a bare `docker build .` does not
|
|
||||||
carry the same guarantee, because its gate phases may come from the cache. The
|
carry the same guarantee, because its gate phases may come from the cache. The
|
||||||
image build gets a `CACHEBUST` value of its own and so runs the gate phases a
|
image build is uncached and so runs the gate phases a second time. That is the
|
||||||
second time. That is the price of the rule above, and it is worth paying: the
|
price of the rule above, and it is worth paying: the image that ships is built
|
||||||
image that ships is built from a run of its own gates rather than from a cache
|
from a run of its own gates rather than from a cache entry.
|
||||||
entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the
|
|
||||||
shared runner after 20 minutes. That allows for the three Docker builds
|
|
||||||
described above (the test phase, the lint phase, then the image), each held to
|
|
||||||
the 5-minute Docker build limit below, plus the bootstrap. A separate workflow
|
|
||||||
limited to `main` by a `branches` list under `on: push` cannot be checked by
|
|
||||||
review: to try a change to it, add the feature branch to that list and push,
|
|
||||||
then remove the branch from the list again before merging. Keep any job in it
|
|
||||||
that publishes behind `if: github.ref_name == 'main'`, so the run from the
|
|
||||||
feature branch publishes nothing.
|
|
||||||
|
|
||||||
- Use platform-standard formatters: `black` for Python, `prettier` for
|
- Use platform-standard formatters: `black` for Python, `prettier` for
|
||||||
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
||||||
@@ -408,18 +286,17 @@ style conventions are in separate documents:
|
|||||||
```
|
```
|
||||||
|
|
||||||
`-count=1` is required on both invocations: it defeats Go's test _result_
|
`-count=1` is required on both invocations: it defeats Go's test _result_
|
||||||
cache, so neither run can report a stored pass in place of running the
|
cache, so the target cannot report a pass it did not earn, and the rerun
|
||||||
tests. It leaves the build cache alone, so it costs the runtime of the suite
|
reproduces a failure instead of replaying it. It leaves the build cache
|
||||||
and no recompilation.
|
alone, so it costs the runtime of the suite and no recompilation.
|
||||||
|
|
||||||
That cache is Go's own, separate from Docker's layer cache. Go stores a
|
Note that this is a second, independent cache, stacked below the Docker
|
||||||
passing result in its cache directory (`GOCACHE`), and when the same tests
|
layer cache that [issue #26](https://git.eeqj.de/sneak/prompts/issues/26)
|
||||||
run again on unchanged code it prints that result, marked `(cached)`,
|
addresses. `CHECK_EPOCH` guarantees the `RUN make test` _step_ re-executes;
|
||||||
without running them. The directory lasts from one run to the next on a
|
it does not guarantee `go test` inside that step does any work, because the
|
||||||
developer's machine, where this target runs, and in the `test` phase of the
|
`GOCACHE` baked into earlier image layers survives into the re-executed
|
||||||
`Dockerfile`, which mounts it to keep compiled packages between check
|
step. They are two separate defects requiring two separate fixes, and a fix
|
||||||
builds, so the phase's `go test` runs carry `-count=1` too. A new
|
for one must not be recorded as covering the other.
|
||||||
`CACHEBUST` value (above) is what makes that step run on an unchanged tree.
|
|
||||||
|
|
||||||
Python example:
|
Python example:
|
||||||
|
|
||||||
@@ -447,12 +324,9 @@ style conventions are in separate documents:
|
|||||||
|
|
||||||
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
||||||
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
|
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
|
||||||
`node_modules/`, and the repo's own build outputs. Fetch the standard
|
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
|
||||||
`.gitignore` from
|
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
|
setting up a new repo. These patterns are written to `.gitignore`'s own
|
||||||
a new repo. A repo's `.gitignore` is the standard file followed by the repo's
|
|
||||||
own entries, such as its binaries; a re-vendor replaces the standard part and
|
|
||||||
keeps those entries. These patterns are written to `.gitignore`'s own
|
|
||||||
semantics, in which an unanchored pattern already matches at every depth; they
|
semantics, in which an unanchored pattern already matches at every depth; they
|
||||||
are not a `.dockerignore` and must not be transplanted into one unmodified.
|
are not a `.dockerignore` and must not be transplanted into one unmodified.
|
||||||
|
|
||||||
@@ -466,7 +340,7 @@ style conventions are in separate documents:
|
|||||||
— which is more dangerous than a short file with no secret patterns at all,
|
— which is more dangerous than a short file with no secret patterns at all,
|
||||||
because it reads as solved and stops anyone looking. Give every
|
because it reads as solved and stops anyone looking. Give every
|
||||||
depth-independent pattern the `**/` prefix and leave only genuinely
|
depth-independent pattern the `**/` prefix and leave only genuinely
|
||||||
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
|
root-anchored entries unprefixed: `.git`, and the repo's own host-built
|
||||||
binary, written `/myapp` and never `**/myapp`, which would also match
|
binary, written `/myapp` and never `**/myapp`, which would also match
|
||||||
`cmd/myapp/` and delete the package directory from the context. Matching is
|
`cmd/myapp/` and delete the package directory from the context. Matching is
|
||||||
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
|
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
|
||||||
@@ -491,24 +365,21 @@ style conventions are in separate documents:
|
|||||||
directory, so a repo running agents in subdirectories still ships
|
directory, so a repo running agents in subdirectories still ships
|
||||||
`services/api/.claude/` and must add its own anchored entry there.
|
`services/api/.claude/` and must add its own anchored entry there.
|
||||||
|
|
||||||
- **A plain `docker build .` of a clone stamps the version that
|
- **Excluding `.git` means `git describe` cannot run inside any build stage, and
|
||||||
`git describe --tags --always` gives**, derived from the `.git` in the build
|
it fails quietly there.** In a build stage there is no repository, so
|
||||||
context as the canonical `Dockerfile` above shows. Without its failure check,
|
`git describe` writes nothing to stdout, `-X main.Version=` comes out empty,
|
||||||
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
|
the binary reports no version at all, and the build still exits 0. Compute the
|
||||||
and the build would still exit 0. `script/docker` and `script/cibuild` pass
|
version on the host and thread it in as a build arg. `script/docker` and
|
||||||
the version they compute on the host; it takes precedence. They do this
|
`script/cibuild` do this, byte-identically across repos:
|
||||||
byte-identically across repos:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# The version and the tag each get their own line: a failing command
|
# Own line: a failing command substitution inside an argument does not
|
||||||
# substitution inside an argument does not trip `set -e`, so the inline
|
# trip `set -e`, so the inline form degrades to an empty constant.
|
||||||
# form degrades to an empty constant.
|
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
tag="$(script/projectname)"
|
docker build --no-cache \
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$tag" .
|
-t "$(script/projectname)" .
|
||||||
```
|
```
|
||||||
|
|
||||||
`--always` makes an untagged repo yield an abbreviated commit hash rather
|
`--always` makes an untagged repo yield an abbreviated commit hash rather
|
||||||
@@ -516,12 +387,12 @@ style conventions are in separate documents:
|
|||||||
fallback is applied — a live check that fires on a build from an export with
|
fallback is applied — a live check that fires on a build from an export with
|
||||||
no `.git` and on a repository with no commits yet. Do not fold it into the
|
no `.git` and on a repository with no commits yet. Do not fold it into the
|
||||||
substitution as `|| echo unknown`, which makes the guard unreachable. The
|
substitution as `|| echo unknown`, which makes the guard unreachable. The
|
||||||
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
|
Dockerfile's side is `ARG VERSION=dev` in the stage that compiles, declared
|
||||||
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
||||||
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
||||||
the scripts stay byte-identical. One consequence for CI: the standard
|
the scripts stay byte-identical. One consequence for CI: the standard
|
||||||
checkout action clones shallow and fetches no tags, so the canonical
|
checkout action clones shallow and fetches no tags, so a repo that embeds a
|
||||||
`.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step.
|
tag-derived version must set `fetch-depth: 0` on its checkout step.
|
||||||
|
|
||||||
- **Verify `.dockerignore` by enumerating the image, not by reading the
|
- **Verify `.dockerignore` by enumerating the image, not by reading the
|
||||||
patterns.** Plant files at the root _and_ at least two directories deep, build
|
patterns.** Plant files at the root _and_ at least two directories deep, build
|
||||||
@@ -555,18 +426,12 @@ style conventions are in separate documents:
|
|||||||
`test-support` depguard rule, where a repo names its own test-support packages
|
`test-support` depguard rule, where a repo names its own test-support packages
|
||||||
by full import path. A repo adds entries there and changes nothing else, and a
|
by full import path. A repo adds entries there and changes nothing else, and a
|
||||||
re-vendor carries its entries forward. The canonical golangci-lint version is
|
re-vendor carries its entries forward. The canonical golangci-lint version is
|
||||||
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
|
v2.12.2 (released 2026-05-06), pinned as the digest of the lint phase's base
|
||||||
image
|
image
|
||||||
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
|
(`golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240`,
|
||||||
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
|
which reports `2.12.2 built with go1.26.2 from c0d3ddc9`). That digest is the
|
||||||
directive must not name a newer Go minor version than the one golangci-lint
|
only pin, since no repo installs golangci-lint on the host: bumping the
|
||||||
was built with, or golangci-lint refuses to lint it: this release lints
|
version means changing it and nothing else.
|
||||||
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
|
|
||||||
installs golangci-lint on the host. A repo sets the lint phase digest to the
|
|
||||||
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
|
|
||||||
the two prompted the change: the canonical copy can name linters that an older
|
|
||||||
golangci-lint rejects, and a newer golangci-lint can add linters that
|
|
||||||
`default: all` switches on until the canonical copy disables them.
|
|
||||||
|
|
||||||
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
|
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
|
||||||
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
|
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
|
||||||
@@ -588,18 +453,8 @@ style conventions are in separate documents:
|
|||||||
function defined and never invoked has the same exit status and the same
|
function defined and never invoked has the same exit status and the same
|
||||||
empty output as one that worked.
|
empty output as one that worked.
|
||||||
|
|
||||||
Node is the exception to the whole-token comparison: the canonical
|
|
||||||
`script/bootstrap` compares only its major version, because the node 22
|
|
||||||
alpine image the `Dockerfile` stages start from is not the exact pin, and
|
|
||||||
nvm cannot install a prebuilt node on alpine.
|
|
||||||
|
|
||||||
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
||||||
|
|
||||||
A Go tool a repo needs on the host is installed with `go install` pinned to
|
|
||||||
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
|
|
||||||
a `go.mod` tool dependency or through a `tools.go` file, either of which
|
|
||||||
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
|
|
||||||
|
|
||||||
- When pinning images or packages by hash, add a comment above the reference
|
- When pinning images or packages by hash, add a comment above the reference
|
||||||
with the version and date (YYYY-MM-DD).
|
with the version and date (YYYY-MM-DD).
|
||||||
|
|
||||||
@@ -709,17 +564,13 @@ style conventions are in separate documents:
|
|||||||
Never edit existing migrations after release.
|
Never edit existing migrations after release.
|
||||||
|
|
||||||
- All repos should have an `.editorconfig` enforcing the project's indentation
|
- All repos should have an `.editorconfig` enforcing the project's indentation
|
||||||
settings: the standard file from
|
settings.
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, which sets
|
|
||||||
tabs for `Makefile` and Go files, followed by the repo's own sections, such as
|
|
||||||
one for another language it uses. A re-vendor replaces the standard part and
|
|
||||||
keeps those sections.
|
|
||||||
|
|
||||||
- Avoid putting files in the repo root unless necessary. Root should contain
|
- Avoid putting files in the repo root unless necessary. Root should contain
|
||||||
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
|
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
|
||||||
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
|
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
|
||||||
and language-specific config). Everything else goes in a subdirectory.
|
language-specific config). Everything else goes in a subdirectory. Canonical
|
||||||
Canonical subdirectory names:
|
subdirectory names:
|
||||||
- `bin/` — executable scripts and tools
|
- `bin/` — executable scripts and tools
|
||||||
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
|
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
|
||||||
body is a single call into `internal/` or `pkg/`, no project logic in
|
body is a single call into `internal/` or `pkg/`, no project logic in
|
||||||
@@ -750,7 +601,3 @@ style conventions are in separate documents:
|
|||||||
- Go: `go.mod`, `go.sum`, `.golangci.yml`
|
- Go: `go.mod`, `go.sum`, `.golangci.yml`
|
||||||
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
|
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
|
||||||
- Python: `pyproject.toml`
|
- Python: `pyproject.toml`
|
||||||
|
|
||||||
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
|
|
||||||
is never committed under a file or directory named after one agent tool, such
|
|
||||||
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
|
|
||||||
|
|||||||
@@ -18,7 +18,7 @@
|
|||||||
|
|
||||||
pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work lands on
|
pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work lands on
|
||||||
`next`, and `main` receives only the milestone PR that the owner merges. `next`
|
`next`, and `main` receives only the milestone PR that the owner merges. `next`
|
||||||
is at the canonical `golangci-lint` v2.14.0 config and is green. Recent work
|
is at the canonical `golangci-lint` v2.12.2 config and is green. Recent work
|
||||||
extracted the internal/magic, internal/allowlist, internal/httpfetcher, and
|
extracted the internal/magic, internal/allowlist, internal/httpfetcher, and
|
||||||
internal/signature packages. The gosec findings from the 2026-07-06 survey are
|
internal/signature packages. The gosec findings from the 2026-07-06 survey are
|
||||||
resolved. The disk cache is now size-bounded with LRU eviction
|
resolved. The disk cache is now size-bounded with LRU eviction
|
||||||
@@ -30,217 +30,6 @@ P2: security: per-origin rate limiting
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal
|
|
||||||
addresses judges an IPv6 address in the IPv4-compatible form (`::/96`), such
|
|
||||||
as `::127.0.0.1` or `::10.0.0.1`, by the IPv4 address in its last four bytes
|
|
||||||
(closes #297). It read only the mapped form (`::ffff:a.b.c.d`) as IPv4, so a
|
|
||||||
source host such as `[::127.0.0.1]` was checked as a public IPv6 address and
|
|
||||||
fetched, which on a network that routes it reaches the internal IPv4 host; it
|
|
||||||
is now refused with 403. The NAT64 form (`64:ff9b::/96`) was already refused
|
|
||||||
as a whole, as one of the built-in blocked ranges, and still is.
|
|
||||||
- 2026-10-16 a source host that is an IPv6 address in brackets with no port,
|
|
||||||
such as `[2001:db8::a]`, is fetched like one with a port, through `/v1/image/`
|
|
||||||
and through an encrypted URL (closes #295). The fetcher's check of the source
|
|
||||||
URL kept the brackets when no port followed them, could not look the host up,
|
|
||||||
and the request answered 500. It now checks the address without brackets, so a
|
|
||||||
private one is refused with 403, as with a port.
|
|
||||||
- 2026-10-16 a `state_dir` whose name holds `%`, `?` or `#` keeps its database
|
|
||||||
at `state.sqlite3` in that directory (closes #290): the `db_url` derived from
|
|
||||||
it put the name into a `file:` URL as it was, so SQLite read `%41` as `A`, a
|
|
||||||
`?` or `#` ended the path early, and the journal mode parameter was lost. The
|
|
||||||
derived `db_url` is now built with Go's `net/url`, which writes those
|
|
||||||
characters as `%25`, `%3F` and `%23`. For a `state_dir` of `/` or one starting
|
|
||||||
with `//` it starts `file:////`, so SQLite no longer reads the first name in
|
|
||||||
the path as a host.
|
|
||||||
- 2026-10-16 a size with one side `0` keeps the side computed from the source's
|
|
||||||
proportions at 1 or more (closes #283): it was rounded down, so for a thin
|
|
||||||
source it came out at 0, and libvips then made the size square. An 8192x1
|
|
||||||
source at `100x0` came out at 100x100, and at `1300x0` with `fit=cover` made
|
|
||||||
libvips ask for about 39 GB at once; an 8192x2 source at `100x0` answered 500.
|
|
||||||
`fit=contain` and `fit=inside` work both sides out again from the smaller of
|
|
||||||
the two scales and `fit=outside` from the larger, and each keeps both sides at
|
|
||||||
1 or more too, so a 1x8192 source at `0x100` with `fit=contain`, and a 49x8192
|
|
||||||
source at `0x100` or a 49x49 source at `1x1` with `fit=outside`, no longer
|
|
||||||
answer 500. An 8192x1 source at `1300x0` now comes out at 1300x1, and the
|
|
||||||
process running it alone peaked at about 34 MiB resident. With `fit=outside`
|
|
||||||
the image is scaled to cover the size, so that source at `100x0` comes out at
|
|
||||||
8192x1. `README.md` now says this under Signature Specification, and says how
|
|
||||||
a `0` side is worked out where it said `0` meant the original.
|
|
||||||
- 2026-10-16 the generator page lower-cases the source host and writes an empty
|
|
||||||
source path as `/` before it makes the encrypted URL, as `/v1/image/` does
|
|
||||||
(closes #293). A host written in upper and in lower case, or a source URL with
|
|
||||||
no path and with the path `/`, made two cache keys and two upstream requests
|
|
||||||
for one image.
|
|
||||||
- 2026-10-16 the source URL a request fetches is built with `net/url` from the
|
|
||||||
source host, path and query, not joined as text (closes #288). The source
|
|
||||||
path, decoded, is fetched percent-encoded again, so `100%25.png` fetches
|
|
||||||
`/100%25.png` where it answered 401, and `my%20cat.png` and `my%2520cat.png`
|
|
||||||
fetch different paths where both fetched `/my%20cat.png` under two cache keys.
|
|
||||||
The source query stays as the source URL writes it; the characters a browser
|
|
||||||
percent-encodes in a query, such as `'` or a space, which was sent raw and
|
|
||||||
answered 502, are percent-encoded when the image URL is read, so the query is
|
|
||||||
signed and cached as it is sent. The generator page answers 400, showing the
|
|
||||||
URL pixa would fetch, for a source URL its image URL would fetch written
|
|
||||||
another way, such as with `%2F` in the path, whatever else the path holds, or
|
|
||||||
with a user name and password. It also answers 400 for a source path holding a
|
|
||||||
`..` segment, as `/v1/image/` refuses one.
|
|
||||||
- 2026-10-15 a thin source requested with `fit=cover` no longer makes libvips
|
|
||||||
build a huge image (closes #282): libvips' thumbnail scaled the whole source
|
|
||||||
to cover the requested size and copied that into memory before it cut out the
|
|
||||||
centre, so an 8192x64 source at 8192x8192 made it allocate about 24 GiB, and
|
|
||||||
an 8192x1 source answered 500. The image processor now first cuts the centre
|
|
||||||
of the source, widened to whole source pixels, and thumbnail scales and cuts
|
|
||||||
only that, by the same amount on both sides as before. What it copies into
|
|
||||||
memory is then larger than the output by less than one scaled source pixel on
|
|
||||||
each side. With the runtime image's libvips and JPEG output at 8192x8192,
|
|
||||||
resident memory peaked at about 242 MiB for the 8192x64 source and 431 MiB for
|
|
||||||
the 8192x1 one. The output can sit a fraction of an output pixel from where it
|
|
||||||
was.
|
|
||||||
- 2026-10-15 a `/v1/image/` URL whose source path or source query holds a `#`,
|
|
||||||
written as `%23`, is refused with 400 (closes #286): the source URL built from
|
|
||||||
it ended at the `#` when fetched, so each text after it fetched the same
|
|
||||||
upstream image, without a signature for an allowlisted host, and cached it
|
|
||||||
under keys of its own. `README.md` says neither may hold a `#`.
|
|
||||||
- 2026-10-15 the host of a `/v1/image/` URL must be a host name or an IPv4
|
|
||||||
address (letters, digits, hyphens, underscores and dots) or an IPv6 address in
|
|
||||||
brackets, either one optionally followed by `:` and a port, and any other host
|
|
||||||
is refused with 400 (closes #275): only `@` was refused, so a `?` written as
|
|
||||||
`%3F` gave the host `cdn.example.com?x=1`, which Go's URL parser reads as
|
|
||||||
`cdn.example.com` with a query. Such a URL passed the allowlist, and each text
|
|
||||||
after the `?` fetched the host's root path and cached the answer under keys of
|
|
||||||
its own. A `#`, `%`, `\` or space written percent-encoded is refused the same
|
|
||||||
way, and so is an IPv6 address without brackets, which needed a signature
|
|
||||||
before. Underscores are kept because `allowlist_hosts` accepts host names
|
|
||||||
holding them.
|
|
||||||
- 2026-10-15 the README says how a source URL with a query is written into an
|
|
||||||
image URL, and the image route accepts it (closes #255): the query follows the
|
|
||||||
source path after `%3F`. pixa now reads the image URL's path decoded once, so
|
|
||||||
the `?` before the query, any `?` in it and every `%` in the source path or
|
|
||||||
query must be percent-encoded, and any other character may be. The route used
|
|
||||||
to parse chi's wildcard, which holds the path as the client wrote it whenever
|
|
||||||
that is not Go's own encoding of it; a `/`, `;` or `,` in the query written as
|
|
||||||
`%2F`, `%3B` or `%2C`, as `GenerateSignedURL` wrote them, kept the `%3F` from
|
|
||||||
being read as `?`, and the request failed with 401. `GenerateSignedURL` now
|
|
||||||
writes Go's own encoding of the path. The README has an example with a query
|
|
||||||
and its signature. The path traversal check no longer reads the source query,
|
|
||||||
and keeps a `%` that does not start a percent-escape as it is instead of
|
|
||||||
refusing the URL, so a source path or query holding `%25` is accepted, while a
|
|
||||||
double-encoded `..` next to such a `%` is still refused.
|
|
||||||
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
|
|
||||||
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
|
|
||||||
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
|
|
||||||
host pixa processed 48 images at once, each of which can take about 4.2 GiB.
|
|
||||||
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
|
|
||||||
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
|
|
||||||
"Running under upaas" in `README.md` gives the worst-case memory, measured,
|
|
||||||
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
|
|
||||||
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
|
|
||||||
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
|
|
||||||
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
|
|
||||||
`/tmp` is a tmpfs.
|
|
||||||
- 2026-10-15 the canonical files are re-vendored from `sneak/prompts` commit
|
|
||||||
`e3f79c4` (closes #221): `.dockerignore`, `.editorconfig`,
|
|
||||||
`.gitea/workflows/check.yml`, `.gitignore`, `.golangci.yml`,
|
|
||||||
`.prettierignore`, `.prettierrc`, `REPO_POLICIES.md` and the standard scripts
|
|
||||||
are its copies. pixa's own entries follow the standard part: `/bin`, `/data`
|
|
||||||
and the local config files in `.dockerignore`; Go's `*.log`, `*.out` and
|
|
||||||
`*.test`, `vendor/`, the binaries, `/data/`, `*.sqlite3` and the local config
|
|
||||||
files in `.gitignore`. The workflow keeps its `script/docker-smoke` step while
|
|
||||||
https://git.eeqj.de/sneak/pixa/issues/203 is open. The scripts pass a new
|
|
||||||
`CACHEBUST` build argument in place of `--no-cache`, and every `Dockerfile`
|
|
||||||
stage declares it first, so every `RUN` step still runs on every build while
|
|
||||||
Go's build cache, mounted on the steps that compile, is kept; both `go test`
|
|
||||||
runs keep `-count=1`. The lint and test builds write no image. The lint phase
|
|
||||||
runs golangci-lint v2.14.0, the test phase runs on the Debian Go image, and
|
|
||||||
the build stage marks `/src` safe for git. `script/bootstrap`, `script/fmt`
|
|
||||||
and `script/fmt-check` use the node on the `PATH` only when it has the pinned
|
|
||||||
major version. Of golangci-lint v2.14.0's two findings, an unused `nolint`
|
|
||||||
directive is removed, and gosec's new path check is marked as not applying to
|
|
||||||
the fixed config file locations. The agent guidance moves unchanged from
|
|
||||||
`CLAUDE.md` to a root `AGENTS.md`, where `REPO_POLICIES.md` keeps it.
|
|
||||||
- 2026-10-15 the request metrics record any method that is not a standard one as
|
|
||||||
`other` (closes #274): their `method` label was each request's method as sent,
|
|
||||||
and Go's HTTP server accepts any method name, so a client sending made-up
|
|
||||||
methods added series to the Prometheus registry that were never removed. The
|
|
||||||
label is now the request's method only for `GET`, `HEAD`, `POST`, `PUT`,
|
|
||||||
`PATCH`, `DELETE`, `OPTIONS`, `CONNECT` and `TRACE`. The metrics library reads
|
|
||||||
the method from the request it is handed, so the metrics middleware hands it a
|
|
||||||
copy of the request carrying the label; the handlers still see the method the
|
|
||||||
client sent.
|
|
||||||
- 2026-10-15 logging out ends every login session (closes #256): a login session
|
|
||||||
was a cookie with nothing kept on the server, so `/logout` removed it from one
|
|
||||||
browser and a copy of it kept working for its 30 days. Every login is made
|
|
||||||
with the one signing key, so logging out now ends them all: the session
|
|
||||||
manager keeps a random value, made with `crypto/rand` when the new `logout`
|
|
||||||
table has none and replaced by each logout, saves it there so that it outlasts
|
|
||||||
a restart, puts the value at login in each session, and refuses a session
|
|
||||||
carrying another value. The check for a logout compares no time, so a login
|
|
||||||
after a logout works whatever the host's clock does; a session's 30 days are
|
|
||||||
still checked against the clock. Losing the state directory ends every login
|
|
||||||
session, and no logout can make an ended one work again. Logging out is now
|
|
||||||
`POST /logout` from a form on the generator page, behind the CSRF check the
|
|
||||||
other forms have, so a link or image elsewhere on the same site cannot end the
|
|
||||||
sessions. A logout without a valid login session ends none and only removes
|
|
||||||
the cookie, as anyone can take a CSRF token from the login page. A logout that
|
|
||||||
cannot be saved answers 500. `session.NewManager`, which the tests use, keeps
|
|
||||||
the value in memory only; pixad uses `session.NewManagerWithDatabase`.
|
|
||||||
- 2026-10-15 a cached file left empty or partial is no longer served, or fetched
|
|
||||||
again on every request (closes #252). Source and transformed image files, and
|
|
||||||
each transformed image's `.meta` file, are synced to disk before they are
|
|
||||||
renamed into place, and their directory after; the `.meta` file is written
|
|
||||||
before the image, and a failed directory sync removes the renamed file. The
|
|
||||||
reconciliation pass, at startup and every five minutes, removes a cached
|
|
||||||
source or transformed image whose file is empty, or not the size the database
|
|
||||||
records for it, with its rows and a warning naming the file, so the next
|
|
||||||
request fetches the source or converts the image again; requests compare no
|
|
||||||
sizes. Storing a source over a file of another size replaces that file. A
|
|
||||||
source that fails to decode is not removed for that alone: a damaged upstream
|
|
||||||
image would then be fetched on every request.
|
|
||||||
- 2026-10-15 a transcode stops once its request's `downstream_timeout` has
|
|
||||||
ended, unless it has begun encoding (closes #243): the image processor looked
|
|
||||||
at its context only while it waited for a processing slot, so an image was
|
|
||||||
decoded and encoded, holding the slot, however long after its request's
|
|
||||||
deadline that took. `Process` now returns the context's error, and gives the
|
|
||||||
slot back, when the context has ended by the time it has the slot or before it
|
|
||||||
encodes. A request that waited for a shared transcode stopped at the first
|
|
||||||
request's deadline is answered 503 `server busy, try again later`, not 500,
|
|
||||||
though its own `downstream_timeout` may not have ended. govips gives pixa no
|
|
||||||
way to stop libvips once an encode has begun, so that encode keeps its slot
|
|
||||||
until it finishes; the limit of 8192 on each side of the source and of the
|
|
||||||
output bounds how long that takes. With one libvips thread, 8192x8192 random
|
|
||||||
pixels take up to about 1 second to encode as JPEG, 15 as PNG, 16 as GIF, 63
|
|
||||||
as WebP, 110 as AVIF and 96 as JPEG XL, the last three with an alpha channel,
|
|
||||||
plus up to about 11 seconds to decode the source; `README.md` gives these
|
|
||||||
times and no longer says the work stops when `downstream_timeout` ends. The
|
|
||||||
settings of the WebP, AVIF and JPEG XL encoders, which can hold a slot past
|
|
||||||
the default `downstream_timeout` of 60 seconds, are unchanged.
|
|
||||||
- 2026-10-15 "Running under upaas" in `README.md` tells the operator to set
|
|
||||||
`PIXA_CACHE_MAX_BYTES` (closes #251): it listed the variable as optional, but
|
|
||||||
under upaas the volume is a directory on the host's filesystem. The default,
|
|
||||||
75% of the sum of that filesystem's free space and what the cache holds, and
|
|
||||||
at least 500 MiB, let one pixa take three quarters or more of the space the
|
|
||||||
host and every other app had left, and all of it when that was 500 MiB or
|
|
||||||
less. The section now says so. The default is unchanged.
|
|
||||||
- 2026-10-15 the request metrics are labelled by route, not by path (closes
|
|
||||||
#254): their `handler` label was each request's path, so every new path, a 404
|
|
||||||
or an encrypted URL, added series to the Prometheus registry that were never
|
|
||||||
removed, and `/metrics` listed every encrypted URL's token. The label is now
|
|
||||||
the pattern of the chi route the request matches, such as `/v1/image/*`, or
|
|
||||||
`unmatched` for a request that matches none: one the router answers 404 or
|
|
||||||
405, or a CORS preflight. The metrics library takes the label before the
|
|
||||||
request is routed, so the metrics middleware asks the router which route
|
|
||||||
matches.
|
|
||||||
- 2026-10-15 expired negative cache rows are removed on a schedule (closes
|
|
||||||
#244): the row of a failed fetch, which makes pixa answer 502 for that source
|
|
||||||
URL for 5 minutes, was deleted only when the same URL was requested again
|
|
||||||
after that, so a client asking for a new missing path each time added rows
|
|
||||||
that stayed for good. The eviction goroutine now calls `CleanExpired` at
|
|
||||||
startup and on each periodic tick, every 5 minutes in pixad, so a row stays at
|
|
||||||
most about 10 minutes. `CleanExpired` deletes at most 1000 rows per statement,
|
|
||||||
so a request waits behind 1000 deletions at most, however many rows have
|
|
||||||
expired. The goroutine runs with the disk cache disabled
|
|
||||||
(`cache_max_bytes: 0`) too, where it does nothing else, as the negative cache
|
|
||||||
is used either way.
|
|
||||||
- 2026-10-15 a cached variant is served while its source URL is in the negative
|
- 2026-10-15 a cached variant is served while its source URL is in the negative
|
||||||
cache (closes #245): `Get` checked the negative cache before the variant
|
cache (closes #245): `Get` checked the negative cache before the variant
|
||||||
cache, so once a fetch of the source failed, as one does when another variant
|
cache, so once a fetch of the source failed, as one does when another variant
|
||||||
@@ -270,6 +59,21 @@ P2: security: per-origin rate limiting
|
|||||||
and the query now have each `%` written as `%25` and each `:` as `%3A`; a path
|
and the query now have each `%` written as `%25` and each `:` as `%3A`; a path
|
||||||
and query holding neither are signed and keyed as before. Signature
|
and query holding neither are signed and keyed as before. Signature
|
||||||
Specification in `README.md` says so.
|
Specification in `README.md` says so.
|
||||||
|
- 2026-10-15 a transcode stops once its request's `downstream_timeout` has
|
||||||
|
ended, unless it has begun encoding (closes #243): the image processor looked
|
||||||
|
at its context only while it waited for a processing slot, so an image was
|
||||||
|
decoded and encoded, holding the slot, however long after its request's
|
||||||
|
deadline that took. `Process` now returns the context's error, and gives the
|
||||||
|
slot back, when the context has ended by the time it has the slot or before it
|
||||||
|
encodes. libvips cannot stop an encode once it has begun, so that encode keeps
|
||||||
|
its slot until it finishes; the limit of 8192 on each side of the source and
|
||||||
|
of the output bounds how long that takes. With one libvips thread, 8192x8192
|
||||||
|
random pixels take up to about 1 second to encode as JPEG, 15 as PNG, 16 as
|
||||||
|
GIF, 63 as WebP, 110 as AVIF and 96 as JPEG XL, the last three with an alpha
|
||||||
|
channel, plus up to about 11 seconds to decode the source; `README.md` gives
|
||||||
|
these times and no longer says the work stops when `downstream_timeout` ends.
|
||||||
|
The settings of the WebP, AVIF and JPEG XL encoders, which can hold a slot
|
||||||
|
past the default `downstream_timeout` of 60 seconds, are unchanged.
|
||||||
- 2026-10-15 the example config allowlists no host (closes #249): it allowlisted
|
- 2026-10-15 the example config allowlists no host (closes #249): it allowlisted
|
||||||
`github.com` and `user-images.githubusercontent.com`, where anyone can put an
|
`github.com` and `user-images.githubusercontent.com`, where anyone can put an
|
||||||
image, so a deployment set up as Getting Started says served those images to
|
image, so a deployment set up as Getting Started says served those images to
|
||||||
|
|||||||
@@ -108,11 +108,10 @@ upstream_connections_per_host: 20
|
|||||||
# downstream_timeout has ended first.
|
# downstream_timeout has ended first.
|
||||||
upstream_connections: 64
|
upstream_connections: 64
|
||||||
|
|
||||||
# Maximum number of images decoded and encoded at once (default: half the
|
# Maximum number of images decoded and encoded at once (default: the
|
||||||
# number of CPUs pixa can use, at least 1; the number of CPUs follows a
|
# number of CPUs pixa can use, which follows a container's CPU limit). A
|
||||||
# container's CPU limit). A request that finds none free waits up to 10
|
# request that finds none free waits up to 10 seconds for one, and if none
|
||||||
# seconds for one, and if none frees up it is answered 503, unless
|
# frees up it is answered 503, unless downstream_timeout has ended first.
|
||||||
# downstream_timeout has ended first.
|
|
||||||
# max_concurrent_processing: 4
|
# max_concurrent_processing: 4
|
||||||
|
|
||||||
# How many requests to the image routes (/v1/image/ and /v1/e/) one client
|
# How many requests to the image routes (/v1/image/ and /v1/e/) one client
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
package config
|
package config
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
|
||||||
"runtime"
|
"runtime"
|
||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
@@ -13,8 +12,8 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// TestOmittedConcurrencyLimitsUseDefaults checks that an omitted
|
// TestOmittedConcurrencyLimitsUseDefaults checks that an omitted
|
||||||
// max_concurrent_processing is half the CPUs Go uses, and at least 1, and
|
// max_concurrent_processing is the number of CPUs Go uses and an omitted
|
||||||
// an omitted upstream_connections is 64.
|
// upstream_connections is 64.
|
||||||
func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
|
func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -23,9 +22,9 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
|
|||||||
t.Fatalf("minimal config should be valid, got error: %v", err)
|
t.Fatalf("minimal config should be valid, got error: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if c.MaxConcurrentProcessing != max(runtime.GOMAXPROCS(0)/2, 1) {
|
if c.MaxConcurrentProcessing != runtime.GOMAXPROCS(0) {
|
||||||
t.Errorf("MaxConcurrentProcessing = %d, want %d, half the CPUs",
|
t.Errorf("MaxConcurrentProcessing = %d, want %d, one per CPU",
|
||||||
c.MaxConcurrentProcessing, max(runtime.GOMAXPROCS(0)/2, 1))
|
c.MaxConcurrentProcessing, runtime.GOMAXPROCS(0))
|
||||||
}
|
}
|
||||||
|
|
||||||
if c.UpstreamConnections != 64 {
|
if c.UpstreamConnections != 64 {
|
||||||
@@ -33,36 +32,6 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne checks
|
|
||||||
// that an omitted max_concurrent_processing is half the CPUs Go uses,
|
|
||||||
// rounded down, and 1 when Go uses one CPU.
|
|
||||||
//
|
|
||||||
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
|
|
||||||
func TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
|
|
||||||
cpusBefore := runtime.GOMAXPROCS(0)
|
|
||||||
|
|
||||||
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
|
|
||||||
|
|
||||||
for _, tc := range []struct{ cpus, want int }{
|
|
||||||
{1, 1}, {2, 1}, {3, 1}, {8, 4},
|
|
||||||
} {
|
|
||||||
runtime.GOMAXPROCS(tc.cpus)
|
|
||||||
|
|
||||||
c, err := configFromYAML(t, signingKeyLine)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("minimal config should be valid, got error: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Logf("GOMAXPROCS %d: MaxConcurrentProcessing %d",
|
|
||||||
tc.cpus, c.MaxConcurrentProcessing)
|
|
||||||
|
|
||||||
if c.MaxConcurrentProcessing != tc.want {
|
|
||||||
t.Errorf("GOMAXPROCS %d: MaxConcurrentProcessing = %d, want %d",
|
|
||||||
tc.cpus, c.MaxConcurrentProcessing, tc.want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestExplicitConcurrencyLimitsAreUsed checks that valid values for the
|
// TestExplicitConcurrencyLimitsAreUsed checks that valid values for the
|
||||||
// two limits are used as given.
|
// two limits are used as given.
|
||||||
func TestExplicitConcurrencyLimitsAreUsed(t *testing.T) {
|
func TestExplicitConcurrencyLimitsAreUsed(t *testing.T) {
|
||||||
@@ -191,19 +160,3 @@ func TestConcurrencyLimitsFromEnvironment(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup checks
|
|
||||||
// that a PIXA_MAX_CONCURRENT_PROCESSING that is not a whole number of at
|
|
||||||
// least 1, an empty one included, aborts startup naming the variable and
|
|
||||||
// the key instead of taking the default.
|
|
||||||
func TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup(t *testing.T) {
|
|
||||||
for _, value := range []string{"", " ", "0.5", "half", "-1", "4 images"} {
|
|
||||||
t.Run(fmt.Sprintf("value %q", value), func(t *testing.T) {
|
|
||||||
t.Setenv(testMaxConcurrentProcessingVar, value)
|
|
||||||
|
|
||||||
_, err := configFromYAML(t, signingKeyLine)
|
|
||||||
wantStartupError(t, err,
|
|
||||||
testMaxConcurrentProcessingVar, keyMaxConcurrentProcessing)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -37,14 +37,6 @@ const (
|
|||||||
DefaultImageRequestsPerMinute = 600
|
DefaultImageRequestsPerMinute = 600
|
||||||
)
|
)
|
||||||
|
|
||||||
// DefaultMaxConcurrentProcessing returns the most images processed at once
|
|
||||||
// when max_concurrent_processing is omitted: half the CPUs Go uses
|
|
||||||
// (runtime.GOMAXPROCS(0), which follows a container's CPU limit), rounded
|
|
||||||
// down, and at least 1.
|
|
||||||
func DefaultMaxConcurrentProcessing() int {
|
|
||||||
return max(runtime.GOMAXPROCS(0)/2, 1) //nolint:mnd // half the CPUs
|
|
||||||
}
|
|
||||||
|
|
||||||
// Configuration key names.
|
// Configuration key names.
|
||||||
const (
|
const (
|
||||||
keyDebug = "debug"
|
keyDebug = "debug"
|
||||||
@@ -305,8 +297,10 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
|
|||||||
keyUpstreamConnectionsPerHost, DefaultUpstreamConnectionsPerHost),
|
keyUpstreamConnectionsPerHost, DefaultUpstreamConnectionsPerHost),
|
||||||
UpstreamConnections: loader.intVal(
|
UpstreamConnections: loader.intVal(
|
||||||
keyUpstreamConnections, DefaultUpstreamConnections),
|
keyUpstreamConnections, DefaultUpstreamConnections),
|
||||||
|
// Decoding and encoding are CPU-bound, so the default is one image
|
||||||
|
// per CPU Go uses, which follows a container's CPU limit.
|
||||||
MaxConcurrentProcessing: loader.intVal(
|
MaxConcurrentProcessing: loader.intVal(
|
||||||
keyMaxConcurrentProcessing, DefaultMaxConcurrentProcessing()),
|
keyMaxConcurrentProcessing, runtime.GOMAXPROCS(0)),
|
||||||
UpstreamFetchTimeout: loader.durationVal(
|
UpstreamFetchTimeout: loader.durationVal(
|
||||||
keyUpstreamFetchTimeout, DefaultUpstreamFetchTimeout),
|
keyUpstreamFetchTimeout, DefaultUpstreamFetchTimeout),
|
||||||
UpstreamMaxResponseSize: loader.int64Val(
|
UpstreamMaxResponseSize: loader.int64Val(
|
||||||
@@ -338,7 +332,8 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
|
|||||||
settingName(keyDBURL), errValueEmpty)
|
settingName(keyDBURL), errValueEmpty)
|
||||||
}
|
}
|
||||||
|
|
||||||
c.DBURL = defaultDBURL(c.StateDir)
|
// The driver sets the journal mode only through a _pragma parameter.
|
||||||
|
c.DBURL = fmt.Sprintf("file:%s/state.sqlite3?_pragma=journal_mode(WAL)", c.StateDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
if loader.err != nil {
|
if loader.err != nil {
|
||||||
@@ -353,30 +348,6 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
|
|||||||
return c, nil
|
return c, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// defaultDBURL returns the db_url used when db_url is omitted: the file
|
|
||||||
// state.sqlite3 in stateDir, in WAL mode, which the driver sets only
|
|
||||||
// through a _pragma parameter. String() percent-encodes any %, ? or # in
|
|
||||||
// the directory name, which SQLite decodes back, so the URL names exactly
|
|
||||||
// this file. OmitHost writes file:/var/lib/pixa/... instead of
|
|
||||||
// file:///var/lib/pixa/..., and keeps a relative stateDir from being read
|
|
||||||
// as a host name. The path written into the URL is stateDir followed by
|
|
||||||
// /state.sqlite3; when it starts with //, as it does for a stateDir of /
|
|
||||||
// or one starting with //, the empty host is kept, so the URL starts
|
|
||||||
// file://// and SQLite does not read the first name in the path as a
|
|
||||||
// host.
|
|
||||||
func defaultDBURL(stateDir string) string {
|
|
||||||
dbPath := stateDir + "/state.sqlite3"
|
|
||||||
|
|
||||||
dbURL := url.URL{
|
|
||||||
Scheme: "file",
|
|
||||||
OmitHost: !strings.HasPrefix(dbPath, "//"),
|
|
||||||
Path: dbPath,
|
|
||||||
RawQuery: "_pragma=journal_mode(WAL)",
|
|
||||||
}
|
|
||||||
|
|
||||||
return dbURL.String()
|
|
||||||
}
|
|
||||||
|
|
||||||
// validateKnownKeys rejects configuration files containing keys the
|
// validateKnownKeys rejects configuration files containing keys the
|
||||||
// application does not understand, so typos fail at startup instead of
|
// application does not understand, so typos fail at startup instead of
|
||||||
// being silently ignored, and rejects keys that are explicitly set to
|
// being silently ignored, and rejects keys that are explicitly set to
|
||||||
@@ -852,7 +823,6 @@ func loadConfigFile(log *slog.Logger, appName string) (*smartconfig.Config, erro
|
|||||||
// one whose path runs through a file, such as under a HOME of
|
// one whose path runs through a file, such as under a HOME of
|
||||||
// /dev/null. One that cannot be read or does not parse is a
|
// /dev/null. One that cannot be read or does not parse is a
|
||||||
// fatal startup error.
|
// fatal startup error.
|
||||||
//nolint:gosec // G703: a fixed config location; HOME is the operator's
|
|
||||||
_, statErr := os.Stat(cleanPath)
|
_, statErr := os.Stat(cleanPath)
|
||||||
if errors.Is(statErr, fs.ErrNotExist) || errors.Is(statErr, syscall.ENOTDIR) {
|
if errors.Is(statErr, fs.ErrNotExist) || errors.Is(statErr, syscall.ENOTDIR) {
|
||||||
continue
|
continue
|
||||||
|
|||||||
@@ -135,104 +135,6 @@ func TestDefaultDBURLOpensTheDatabaseInWALMode(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestDerivedDBURLOpensStateDirHoldingURLCharacters opens the db_url
|
|
||||||
// derived from a state_dir whose name holds %41, ? and #, and checks that
|
|
||||||
// the database is state.sqlite3 in that directory, in WAL mode. Put into
|
|
||||||
// a file: URL as it is, the name would lose its meaning: %41 is read as
|
|
||||||
// A, and the path ends at the ?, which also loses the journal mode.
|
|
||||||
func TestDerivedDBURLOpensStateDirHoldingURLCharacters(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
stateDir := filepath.Join(t.TempDir(), "a%41b?c#d")
|
|
||||||
|
|
||||||
err := os.Mkdir(stateDir, 0o750)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create %q: %v", stateDir, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Quoted so YAML takes the whole name as it is.
|
|
||||||
c, err := configFromYAML(t, signingKeyLine+"state_dir: '"+stateDir+"'\n")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
|
|
||||||
|
|
||||||
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(stateDir, "state.sqlite3"))
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestDerivedDBURLOpensStateDirStartingWithTwoSlashes opens the db_url
|
|
||||||
// derived from a state_dir that starts with //, and checks that the
|
|
||||||
// database is state.sqlite3 in that directory, in WAL mode. Written right
|
|
||||||
// after file:, such a name gives a URL starting file://, and SQLite reads
|
|
||||||
// the first directory name as a host and refuses to open it.
|
|
||||||
func TestDerivedDBURLOpensStateDirStartingWithTwoSlashes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
stateDir := "/" + dir
|
|
||||||
|
|
||||||
c, err := configFromYAML(t, signingKeyLine+"state_dir: "+stateDir+"\n")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
|
|
||||||
|
|
||||||
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(dir, "state.sqlite3"))
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestDerivedDBURLForRootStateDir checks the db_url derived from a
|
|
||||||
// state_dir of /. The path written into the URL is then
|
|
||||||
// //state.sqlite3, and written right after file: it gives a URL starting
|
|
||||||
// file://, where SQLite reads state.sqlite3 as a host and refuses to open
|
|
||||||
// it. The test only checks the URL, since it cannot write to /.
|
|
||||||
func TestDerivedDBURLForRootStateDir(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
c, err := configFromYAML(t, signingKeyLine+"state_dir: /\n")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("config with state_dir / should be valid, got: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Logf("state_dir / gives db_url %q", c.DBURL)
|
|
||||||
|
|
||||||
wantDBURL := "file:////state.sqlite3?_pragma=journal_mode(WAL)"
|
|
||||||
if c.DBURL != wantDBURL {
|
|
||||||
t.Errorf("DBURL = %q, want %q", c.DBURL, wantDBURL)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// checkDBURLOpensFileInWALMode opens dbURL with the SQLite driver pixad
|
|
||||||
// uses, and checks that the database is in WAL mode and is the file
|
|
||||||
// dbFile.
|
|
||||||
func checkDBURLOpensFileInWALMode(t *testing.T, dbURL, dbFile string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
db, err := sql.Open("sqlite", dbURL)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to open %q: %v", dbURL, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
var journalMode string
|
|
||||||
|
|
||||||
err = db.QueryRowContext(t.Context(), "PRAGMA journal_mode").Scan(&journalMode)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to read the journal mode of %q: %v", dbURL, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if journalMode != "wal" {
|
|
||||||
t.Errorf("journal mode of %q = %q, want wal", dbURL, journalMode)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = os.Stat(dbFile)
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("opening %q did not create %q: %v", dbURL, dbFile, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestExplicitValidValuesAreUsed(t *testing.T) {
|
func TestExplicitValidValuesAreUsed(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -176,11 +176,3 @@ CREATE TABLE IF NOT EXISTS cache_stats (
|
|||||||
last_updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
last_updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||||
);
|
);
|
||||||
INSERT OR IGNORE INTO cache_stats (id) VALUES (1);
|
INSERT OR IGNORE INTO cache_stats (id) VALUES (1);
|
||||||
|
|
||||||
-- A random value that a login session carries from its login. pixa makes it
|
|
||||||
-- when the table has no row, and each logout replaces it; a login session
|
|
||||||
-- carrying another value is refused, so logging out ends every login session.
|
|
||||||
CREATE TABLE IF NOT EXISTS logout (
|
|
||||||
id INTEGER PRIMARY KEY CHECK (id = 1),
|
|
||||||
logout_value TEXT NOT NULL
|
|
||||||
);
|
|
||||||
|
|||||||
@@ -9,9 +9,7 @@ import (
|
|||||||
"math"
|
"math"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/url"
|
"net/url"
|
||||||
"slices"
|
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/pixa/internal/clientip"
|
"sneak.berlin/go/pixa/internal/clientip"
|
||||||
@@ -91,18 +89,10 @@ func (s *Handlers) handleLoginPost(w http.ResponseWriter, r *http.Request) {
|
|||||||
http.Redirect(w, r, "/", http.StatusSeeOther)
|
http.Redirect(w, r, "/", http.StatusSeeOther)
|
||||||
}
|
}
|
||||||
|
|
||||||
// HandleLogout logs out, which ends every login session (see
|
// HandleLogout clears the session and redirects to login.
|
||||||
// session.Manager.Logout), and redirects to login.
|
|
||||||
func (s *Handlers) HandleLogout() http.HandlerFunc {
|
func (s *Handlers) HandleLogout() http.HandlerFunc {
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
err := s.sessMgr.Logout(w, r)
|
s.sessMgr.ClearSession(w)
|
||||||
if err != nil {
|
|
||||||
s.log.Error("failed to log out", "error", err)
|
|
||||||
http.Error(w, "Failed to log out", http.StatusInternalServerError)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
http.Redirect(w, r, "/", http.StatusSeeOther)
|
http.Redirect(w, r, "/", http.StatusSeeOther)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -127,7 +117,7 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
|||||||
sourceURL := r.FormValue("url")
|
sourceURL := r.FormValue("url")
|
||||||
|
|
||||||
// Validate source URL
|
// Validate source URL
|
||||||
parsed, err := parseSourceURL(sourceURL)
|
parsed, err := url.Parse(sourceURL)
|
||||||
if err != nil || parsed.Host == "" {
|
if err != nil || parsed.Host == "" {
|
||||||
s.renderGeneratorWithForm(w, r, "Invalid source URL", r.Form,
|
s.renderGeneratorWithForm(w, r, "Invalid source URL", r.Form,
|
||||||
http.StatusBadRequest)
|
http.StatusBadRequest)
|
||||||
@@ -135,16 +125,6 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// /v1/image/ refuses a ".." segment in the source path, and so does
|
|
||||||
// the generator. The path is decoded, so %2e%2e counts too
|
|
||||||
if slices.Contains(strings.Split(parsed.Path, "/"), "..") {
|
|
||||||
s.renderGeneratorWithForm(w, r,
|
|
||||||
"Source URL path cannot hold a .. segment", r.Form,
|
|
||||||
http.StatusBadRequest)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
payload, expiresAt, ttl, err := buildGeneratePayload(parsed, r.Form)
|
payload, expiresAt, ttl, err := buildGeneratePayload(parsed, r.Form)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.renderGeneratorWithForm(w, r, err.Error(), r.Form,
|
s.renderGeneratorWithForm(w, r, err.Error(), r.Form,
|
||||||
@@ -153,19 +133,6 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// The image URL must fetch the source URL as it was written
|
|
||||||
sourceRequest := payload.ToImageRequest()
|
|
||||||
sourceRequest.AllowHTTP = s.config.AllowHTTP
|
|
||||||
fetchURL := sourceRequest.SourceURL()
|
|
||||||
|
|
||||||
if !fetchedAsWritten(parsed, fetchURL) {
|
|
||||||
s.renderGeneratorWithForm(w, r, "Source URL cannot be fetched as "+
|
|
||||||
"written: pixa would fetch "+fetchURL,
|
|
||||||
r.Form, http.StatusBadRequest)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// Generate encrypted token
|
// Generate encrypted token
|
||||||
token, err := s.encGen.Generate(payload)
|
token, err := s.encGen.Generate(payload)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -198,48 +165,6 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseSourceURL parses the source URL entered on the generator page with its
|
|
||||||
// host lower-cased and an empty path written as "/", as /v1/image/ reads a
|
|
||||||
// source URL, so that each spelling of one source URL makes one cache key and
|
|
||||||
// one upstream request.
|
|
||||||
func parseSourceURL(sourceURL string) (*url.URL, error) {
|
|
||||||
parsed, err := url.Parse(sourceURL)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
parsed.Host = strings.ToLower(parsed.Host)
|
|
||||||
|
|
||||||
if parsed.Path == "" {
|
|
||||||
parsed.Path = "/"
|
|
||||||
}
|
|
||||||
|
|
||||||
return parsed, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// fetchedAsWritten reports whether fetchURL, the URL pixa fetches for an
|
|
||||||
// image URL made from the source URL source, is source as it was written,
|
|
||||||
// apart from the scheme, which the token does not keep. It is not when source
|
|
||||||
// holds a user name and password, which the token does not keep either, or a
|
|
||||||
// "?" with no query after it. The token keeps the source path decoded and
|
|
||||||
// pixa fetches it percent-encoded again, so a path written another way, such
|
|
||||||
// as with %2F, is fetched as another path. url.URL keeps the path as written
|
|
||||||
// in RawPath when it is not the decoded path percent-encoded again.
|
|
||||||
// EscapedPath ignores RawPath when it also holds a character net/url encodes,
|
|
||||||
// such as a space, so RawPath is compared itself.
|
|
||||||
func fetchedAsWritten(source *url.URL, fetchURL string) bool {
|
|
||||||
writtenPath := source.RawPath
|
|
||||||
if writtenPath == "" {
|
|
||||||
writtenPath = source.EscapedPath()
|
|
||||||
}
|
|
||||||
|
|
||||||
fetched, err := url.Parse(fetchURL)
|
|
||||||
|
|
||||||
return err == nil && source.User == nil && !source.ForceQuery &&
|
|
||||||
fetched.EscapedPath() == writtenPath &&
|
|
||||||
fetched.RawQuery == source.RawQuery
|
|
||||||
}
|
|
||||||
|
|
||||||
// buildGeneratePayload parses the numeric form fields and assembles the
|
// buildGeneratePayload parses the numeric form fields and assembles the
|
||||||
// encrypted URL payload. ttl=0 means never expires (ExpiresAt stays 0). A
|
// encrypted URL payload. ttl=0 means never expires (ExpiresAt stays 0). A
|
||||||
// non-numeric or out-of-range width, height, quality or ttl, or an
|
// non-numeric or out-of-range width, height, quality or ttl, or an
|
||||||
|
|||||||
@@ -185,98 +185,6 @@ func TestHandleLogout_ClearsSessionCookie(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestHandleLogout_EndsCopiedSession verifies that after POST /logout a copy
|
|
||||||
// of the session cookie taken before it shows the login page, not the
|
|
||||||
// generator page.
|
|
||||||
func TestHandleLogout_EndsCopiedSession(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
h, srv := newCSRFTestRouter(t)
|
|
||||||
|
|
||||||
copied := newSessionCookie(t, h)
|
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodPost, "/logout", nil)
|
|
||||||
req.AddCookie(copied)
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
h.HandleLogout().ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
if rec.Code != http.StatusSeeOther {
|
|
||||||
t.Fatalf("POST /logout status = %d, want %d",
|
|
||||||
rec.Code, http.StatusSeeOther)
|
|
||||||
}
|
|
||||||
|
|
||||||
req = httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
|
|
||||||
req.AddCookie(copied)
|
|
||||||
|
|
||||||
rec = httptest.NewRecorder()
|
|
||||||
srv.ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
body := rec.Body.String()
|
|
||||||
t.Logf("GET / with the copied cookie: status %d, login page %v",
|
|
||||||
rec.Code, strings.Contains(body, loginForm))
|
|
||||||
|
|
||||||
if !strings.Contains(body, loginForm) || strings.Contains(body, generatorForm) {
|
|
||||||
t.Error("a cookie copied before /logout still opens the generator page")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestHandleLogout_SaveFails_Answers500 verifies that when the logout cannot
|
|
||||||
// be saved in the database, POST /logout answers 500, leaves the session
|
|
||||||
// cookie in place and ends no session.
|
|
||||||
func TestHandleLogout_SaveFails_Answers500(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
h, srv := newCSRFTestRouter(t)
|
|
||||||
|
|
||||||
db := setupTestDB(t)
|
|
||||||
|
|
||||||
sessMgr, err := session.NewManagerWithDatabase(
|
|
||||||
t.Context(), testSigningKey, db)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("session.NewManagerWithDatabase() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
h.sessMgr = sessMgr
|
|
||||||
cookie := newSessionCookie(t, h)
|
|
||||||
|
|
||||||
// Saving the logout fails on a closed database.
|
|
||||||
err = db.Close()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("closing the database: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodPost, "/logout", nil)
|
|
||||||
req.AddCookie(cookie)
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
h.HandleLogout().ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
t.Logf("POST /logout: status %d, Set-Cookie %q",
|
|
||||||
rec.Code, rec.Header().Get("Set-Cookie"))
|
|
||||||
|
|
||||||
if rec.Code != http.StatusInternalServerError {
|
|
||||||
t.Errorf("POST /logout status = %d, want %d",
|
|
||||||
rec.Code, http.StatusInternalServerError)
|
|
||||||
}
|
|
||||||
|
|
||||||
if c := findSessionCookie(rec); c != nil {
|
|
||||||
t.Errorf("a logout that failed set the session cookie: %s", c)
|
|
||||||
}
|
|
||||||
|
|
||||||
req = httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
|
|
||||||
req.AddCookie(cookie)
|
|
||||||
|
|
||||||
rec = httptest.NewRecorder()
|
|
||||||
srv.ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
if !strings.Contains(rec.Body.String(), generatorForm) {
|
|
||||||
t.Error("a logout that failed ended the session")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestGeneratePost_NoSession_RedirectsToLogin verifies that POST /generate
|
// TestGeneratePost_NoSession_RedirectsToLogin verifies that POST /generate
|
||||||
// with a valid CSRF token but no login session answers 303 to / and makes no
|
// with a valid CSRF token but no login session answers 303 to / and makes no
|
||||||
// URL.
|
// URL.
|
||||||
|
|||||||
@@ -124,8 +124,7 @@ func (s *Handlers) initImageService() error {
|
|||||||
s.imgCache = cache
|
s.imgCache = cache
|
||||||
|
|
||||||
// Background eviction: startup reconciliation, then periodic and
|
// Background eviction: startup reconciliation, then periodic and
|
||||||
// write-pressure passes, and the removal of expired negative cache
|
// write-pressure passes. No-op when the disk cache is disabled.
|
||||||
// rows, which is all it does when the disk cache is disabled.
|
|
||||||
cache.StartEviction(imgcache.DefaultEvictionInterval)
|
cache.StartEviction(imgcache.DefaultEvictionInterval)
|
||||||
|
|
||||||
// Writes the counts requests could not write by their deadline
|
// Writes the counts requests could not write by their deadline
|
||||||
@@ -164,10 +163,7 @@ func (s *Handlers) initImageService() error {
|
|||||||
|
|
||||||
// Initialize session manager (signing key is validated at config load
|
// Initialize session manager (signing key is validated at config load
|
||||||
// time). Session cookies are always Secure/HttpOnly/SameSite=Strict.
|
// time). Session cookies are always Secure/HttpOnly/SameSite=Strict.
|
||||||
// The random value that each logout replaces is saved in the database,
|
sessMgr, err := session.NewManager(s.config.SigningKey)
|
||||||
// so it outlasts a restart.
|
|
||||||
sessMgr, err := session.NewManagerWithDatabase(
|
|
||||||
context.Background(), s.config.SigningKey, s.db.DB())
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/go-chi/chi/v5"
|
||||||
"github.com/go-chi/chi/v5/middleware"
|
"github.com/go-chi/chi/v5/middleware"
|
||||||
"sneak.berlin/go/pixa/internal/encurl"
|
"sneak.berlin/go/pixa/internal/encurl"
|
||||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||||
@@ -79,21 +80,20 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseImageRequest parses the image URL's path and query parameters into
|
// parseImageRequest parses the wildcard path and query parameters into
|
||||||
// an ImageRequest. On invalid input it writes an error response and
|
// an ImageRequest. On invalid input it writes an error response and
|
||||||
// returns false.
|
// returns false.
|
||||||
func (s *Handlers) parseImageRequest(
|
func (s *Handlers) parseImageRequest(
|
||||||
w http.ResponseWriter, r *http.Request,
|
w http.ResponseWriter, r *http.Request,
|
||||||
) (*imgcache.ImageRequest, bool) {
|
) (*imgcache.ImageRequest, bool) {
|
||||||
// The path is parsed decoded once, as r.URL.Path holds it, and not from
|
// Get the wildcard path from chi
|
||||||
// chi's wildcard: chi routes on the path as the client wrote it whenever
|
pathParam := chi.URLParam(r, "*")
|
||||||
// that is not Go's own encoding of it, so its wildcard would still hold
|
|
||||||
// the %3F before a source query when, say, a "/" in the query is written
|
// Parse the URL path
|
||||||
// as %2F
|
parsed, err := imgcache.ParseImagePath(pathParam)
|
||||||
parsed, err := imgcache.ParseImageURL(r.URL.Path)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.log.Warn("failed to parse image URL",
|
s.log.Warn("failed to parse image URL",
|
||||||
"path", r.URL.Path,
|
"path", pathParam,
|
||||||
"error", err,
|
"error", err,
|
||||||
)
|
)
|
||||||
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
|
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
|
||||||
@@ -240,8 +240,7 @@ func (s *Handlers) respondImageError(
|
|||||||
}
|
}
|
||||||
|
|
||||||
if errors.Is(err, httpfetcher.ErrTooManyConnections) ||
|
if errors.Is(err, httpfetcher.ErrTooManyConnections) ||
|
||||||
errors.Is(err, imageprocessor.ErrTooManyImages) ||
|
errors.Is(err, imageprocessor.ErrTooManyImages) {
|
||||||
errors.Is(err, imgcache.ErrProcessingTimedOut) {
|
|
||||||
s.respondError(w, "server busy, try again later",
|
s.respondError(w, "server busy, try again later",
|
||||||
http.StatusServiceUnavailable)
|
http.StatusServiceUnavailable)
|
||||||
|
|
||||||
|
|||||||
@@ -264,127 +264,3 @@ func TestHandleImage_AllowlistOrSignature(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestHandleImage_InvalidHost_Returns400 sends image URLs whose host no URL
|
|
||||||
// host can be, and checks that each is refused with 400 naming the host as
|
|
||||||
// decoded: allowlistedHost with a character after it that a URL host cannot
|
|
||||||
// hold, percent-encoded, an IPv6 address without brackets, and brackets
|
|
||||||
// around something that is not an IPv6 address. A "?" or "#" after
|
|
||||||
// allowlistedHost would otherwise end the host when the source URL is read,
|
|
||||||
// so the request would pass the allowlist and be fetched and cached under a
|
|
||||||
// host that is not the one fetched.
|
|
||||||
func TestHandleImage_InvalidHost_Returns400(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
photos := newPhotoFetcher(t, allowlistedHost)
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
writtenHost string
|
|
||||||
decodedHost string
|
|
||||||
}{
|
|
||||||
{"? written as %3F", allowlistedHost + "%3Fx=1", allowlistedHost + "?x=1"},
|
|
||||||
{"# written as %23", allowlistedHost + "%23x", allowlistedHost + "#x"},
|
|
||||||
{"% written as %25", allowlistedHost + "%253F", allowlistedHost + "%3F"},
|
|
||||||
{`\ written as %5C`, allowlistedHost + "%5Cx", allowlistedHost + `\x`},
|
|
||||||
{"space written as %20", allowlistedHost + "%20", allowlistedHost + " "},
|
|
||||||
{"IPv6 address without brackets", "2001:db8::1", "2001:db8::1"},
|
|
||||||
{"IPv4 address in brackets", "[203.0.113.5]", "[203.0.113.5]"},
|
|
||||||
{"no address in brackets", "[abc]", "[abc]"},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
rec := sendGet(t, newImageRoute(t, photos), photoURL(tt.writtenHost))
|
|
||||||
checkErrorBody(t, rec, http.StatusBadRequest,
|
|
||||||
fmt.Sprintf("invalid image URL: invalid source host: %q",
|
|
||||||
tt.decodedHost))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestHandleImage_HostForms_Served checks that the image route serves, with
|
|
||||||
// a valid signature, the image at photoPath on a host name, an IPv4 address,
|
|
||||||
// an IPv6 address in brackets, and a host followed by ":" with or without a
|
|
||||||
// port.
|
|
||||||
func TestHandleImage_HostForms_Served(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
hosts := []string{
|
|
||||||
allowlistedHost,
|
|
||||||
"img_1.example.com",
|
|
||||||
"203.0.113.5",
|
|
||||||
"[2001:db8::1]",
|
|
||||||
allowlistedHost + ":8443",
|
|
||||||
"[2001:db8::1]:8443",
|
|
||||||
allowlistedHost + ":",
|
|
||||||
}
|
|
||||||
|
|
||||||
photos := newPhotoFetcher(t, hosts...)
|
|
||||||
exp := time.Now().Add(time.Hour)
|
|
||||||
|
|
||||||
for _, host := range hosts {
|
|
||||||
t.Run(host, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
sig := photoSignature(testSigningKey, host, exp)
|
|
||||||
|
|
||||||
rec := sendGet(t, newImageRoute(t, photos),
|
|
||||||
photoURLWithSig(host, sig, exp))
|
|
||||||
if rec.Code != http.StatusOK {
|
|
||||||
t.Errorf("status = %d, want %d: %s",
|
|
||||||
rec.Code, http.StatusOK, rec.Body)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestHandleImage_HashInSourcePathOrQuery_Returns400 sends image URLs whose
|
|
||||||
// source path or source query holds a "#", written as %23, and checks that
|
|
||||||
// each is refused with 400, the one signed for that source path included. The
|
|
||||||
// source URL would otherwise end at the "#" when it is fetched, so each text
|
|
||||||
// after it would fetch the image at photoPath and cache it under keys of its
|
|
||||||
// own.
|
|
||||||
func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
photos := newPhotoFetcher(t, allowlistedHost, signedHost)
|
|
||||||
exp := time.Now().Add(time.Hour)
|
|
||||||
|
|
||||||
sig := signature.New(testSigningKey).Sign(&signature.Request{
|
|
||||||
SourceHost: signedHost,
|
|
||||||
SourcePath: photoPath + "#x",
|
|
||||||
Width: 50,
|
|
||||||
Height: 50,
|
|
||||||
Format: string(imgcache.FormatJPEG),
|
|
||||||
Quality: 85,
|
|
||||||
FitMode: string(imgcache.FitCover),
|
|
||||||
Expires: exp,
|
|
||||||
})
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
target string
|
|
||||||
}{
|
|
||||||
{"# in the source path",
|
|
||||||
"/v1/image/" + allowlistedHost + photoPath + "%23x/50x50.jpeg"},
|
|
||||||
{"# in the source query",
|
|
||||||
"/v1/image/" + allowlistedHost + photoPath + "%3Fv=1%23x/50x50.jpeg"},
|
|
||||||
{"# in the source path, signed",
|
|
||||||
fmt.Sprintf("/v1/image/%s%s%%23x/50x50.jpeg?sig=%s&exp=%d",
|
|
||||||
signedHost, photoPath, sig, exp.Unix())},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
rec := sendGet(t, newImageRoute(t, photos), tt.target)
|
|
||||||
checkErrorBody(t, rec, http.StatusBadRequest,
|
|
||||||
`invalid image URL: invalid image path: `+
|
|
||||||
`the source path or query holds a "#"`)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -169,166 +169,3 @@ func TestHandleImage_InvalidExp_Returns400(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies sends URLs built
|
|
||||||
// by the service's signed-URL generator for source URLs with a query to the
|
|
||||||
// route, through a chi router: chi routes on the path as the client wrote it
|
|
||||||
// whenever that is not Go's own encoding of it. A 502 means the signature
|
|
||||||
// passed, so the route read the source path and query that were signed.
|
|
||||||
func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
query string
|
|
||||||
}{
|
|
||||||
{"slash and colon", "url=https://origin.example.com/cat.jpg"},
|
|
||||||
{"semicolon", "a=1;b=2"},
|
|
||||||
{"comma", "crop=0,0,50,50"},
|
|
||||||
{"equals and ampersand", "a=1&b=2"},
|
|
||||||
{"percent-escape", "name=my%20cat"},
|
|
||||||
{"question mark", "next=/a?b=1"},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
fix := setupTestHandler(t)
|
|
||||||
|
|
||||||
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
|
|
||||||
SourceHost: signedHost,
|
|
||||||
SourcePath: photoPath,
|
|
||||||
SourceQuery: tt.query,
|
|
||||||
Size: imgcache.Size{Width: 50, Height: 50},
|
|
||||||
Format: imgcache.FormatJPEG,
|
|
||||||
}, time.Hour)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
status := getImage(t, fix, signedURL)
|
|
||||||
if status != http.StatusBadGateway {
|
|
||||||
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestHandleImage_SourceQueryEncodedAnyWay signs one source URL with a query
|
|
||||||
// and writes it into the image URL in ways the README allows: the "?" before
|
|
||||||
// the query and each "%" in it percent-encoded, and every other character as
|
|
||||||
// itself or percent-encoded. A 502 means the signature passed.
|
|
||||||
func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
expires := time.Now().Add(time.Hour)
|
|
||||||
signer := signature.New("test-signing-key-must-be-32-chars")
|
|
||||||
sig := signer.Sign(&signature.Request{
|
|
||||||
SourceHost: signedHost,
|
|
||||||
SourcePath: photoPath,
|
|
||||||
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
|
|
||||||
Width: 50,
|
|
||||||
Height: 50,
|
|
||||||
Format: string(imgcache.FormatJPEG),
|
|
||||||
Quality: 85,
|
|
||||||
FitMode: string(imgcache.FitCover),
|
|
||||||
Expires: expires,
|
|
||||||
})
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
writtenQuery string
|
|
||||||
}{
|
|
||||||
{"only ? and % encoded",
|
|
||||||
"%3Furl=https://origin.example.com/my%2520cat.jpg"},
|
|
||||||
{"every reserved character encoded",
|
|
||||||
"%3Furl%3Dhttps%3A%2F%2Forigin.example.com%2Fmy%2520cat.jpg"},
|
|
||||||
{"lower-case hex digits",
|
|
||||||
"%3furl=https://origin.example.com/my%2520cat.jpg"},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
target := fmt.Sprintf("/v1/image/%s%s%s/50x50.jpeg?sig=%s&exp=%d",
|
|
||||||
signedHost, photoPath, tt.writtenQuery, sig, expires.Unix())
|
|
||||||
|
|
||||||
status := getImage(t, setupTestHandler(t), target)
|
|
||||||
if status != http.StatusBadGateway {
|
|
||||||
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestHandleImage_SourceURLHoldingPercent25Verifies signs a source path and a
|
|
||||||
// source query that each hold %25, the percent-escape of a literal "%", and
|
|
||||||
// sends each to the route written as the README says, with that "%" written as
|
|
||||||
// %25, and as GenerateSignedURL writes it. A 502 means the signature passed.
|
|
||||||
func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
sourcePath string
|
|
||||||
sourceQuery string
|
|
||||||
// writtenPathAndQuery is the source path and query as the README says
|
|
||||||
// to write them into the image URL
|
|
||||||
writtenPathAndQuery string
|
|
||||||
}{
|
|
||||||
{"path", "/sale/50%25off.jpg", "", "/sale/50%2525off.jpg"},
|
|
||||||
{"query", photoPath, "discount=50%25", photoPath + "%3Fdiscount=50%2525"},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name+" written by the README rule", func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
expires := time.Now().Add(time.Hour)
|
|
||||||
signer := signature.New("test-signing-key-must-be-32-chars")
|
|
||||||
sig := signer.Sign(&signature.Request{
|
|
||||||
SourceHost: signedHost,
|
|
||||||
SourcePath: tt.sourcePath,
|
|
||||||
SourceQuery: tt.sourceQuery,
|
|
||||||
Width: 50,
|
|
||||||
Height: 50,
|
|
||||||
Format: string(imgcache.FormatJPEG),
|
|
||||||
Quality: 85,
|
|
||||||
FitMode: string(imgcache.FitCover),
|
|
||||||
Expires: expires,
|
|
||||||
})
|
|
||||||
|
|
||||||
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
|
|
||||||
signedHost, tt.writtenPathAndQuery, sig, expires.Unix())
|
|
||||||
|
|
||||||
status := getImage(t, setupTestHandler(t), target)
|
|
||||||
if status != http.StatusBadGateway {
|
|
||||||
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
t.Run(tt.name+" written by GenerateSignedURL", func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
fix := setupTestHandler(t)
|
|
||||||
|
|
||||||
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
|
|
||||||
SourceHost: signedHost,
|
|
||||||
SourcePath: tt.sourcePath,
|
|
||||||
SourceQuery: tt.sourceQuery,
|
|
||||||
Size: imgcache.Size{Width: 50, Height: 50},
|
|
||||||
Format: imgcache.FormatJPEG,
|
|
||||||
}, time.Hour)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
status := getImage(t, fix, signedURL)
|
|
||||||
if status != http.StatusBadGateway {
|
|
||||||
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -158,8 +158,7 @@ func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
|
|||||||
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
|
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
|
||||||
s.respondError(w, "upstream timeout", http.StatusGatewayTimeout)
|
s.respondError(w, "upstream timeout", http.StatusGatewayTimeout)
|
||||||
case errors.Is(err, httpfetcher.ErrTooManyConnections),
|
case errors.Is(err, httpfetcher.ErrTooManyConnections),
|
||||||
errors.Is(err, imageprocessor.ErrTooManyImages),
|
errors.Is(err, imageprocessor.ErrTooManyImages):
|
||||||
errors.Is(err, imgcache.ErrProcessingTimedOut):
|
|
||||||
s.respondError(w, "server busy, try again later",
|
s.respondError(w, "server busy, try again later",
|
||||||
http.StatusServiceUnavailable)
|
http.StatusServiceUnavailable)
|
||||||
case errors.Is(err, httpfetcher.ErrInvalidContentType),
|
case errors.Is(err, httpfetcher.ErrInvalidContentType),
|
||||||
|
|||||||
@@ -49,39 +49,38 @@ func TestServerBusyAnswers503(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestProcessingTimedOutAnswers503 checks that respondImageError and
|
// TestProcessingTimedOutAnswers503 checks that both image routes answer 503
|
||||||
// handleImageError, which answer the errors of the two image routes, answer
|
// with the server busy error when the image service reports that processing
|
||||||
// 503 with the server busy error when the image service reports that
|
// stopped at the deadline of the request doing it, wrapped as the service
|
||||||
// processing stopped at the deadline of the request doing it, wrapped as the
|
// wraps it. A request that waited for that processing gets this error while
|
||||||
// service wraps it. A request that waited for that processing gets this error
|
// its own deadline may have time left.
|
||||||
// while its own deadline may have time left. The request's source only goes
|
|
||||||
// into a log line, so it is left empty.
|
|
||||||
func TestProcessingTimedOutAnswers503(t *testing.T) {
|
func TestProcessingTimedOutAnswers503(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
h := &Handlers{log: slog.New(slog.DiscardHandler)}
|
h := &Handlers{log: slog.New(slog.DiscardHandler)}
|
||||||
|
req := &imgcache.ImageRequest{SourceHost: "img.example.com", SourcePath: "/a.jpg"}
|
||||||
err := fmt.Errorf("%w: failed to encode: %w",
|
err := fmt.Errorf("%w: failed to encode: %w",
|
||||||
imgcache.ErrProcessingTimedOut, context.DeadlineExceeded)
|
imgcache.ErrProcessingTimedOut, context.DeadlineExceeded)
|
||||||
|
|
||||||
plain := httptest.NewRecorder()
|
plain := httptest.NewRecorder()
|
||||||
h.respondImageError(plain, &imgcache.ImageRequest{}, err)
|
h.respondImageError(plain, req, err)
|
||||||
|
|
||||||
encrypted := httptest.NewRecorder()
|
encrypted := httptest.NewRecorder()
|
||||||
h.handleImageError(encrypted, err)
|
h.handleImageError(encrypted, err)
|
||||||
|
|
||||||
for name, rec := range map[string]*httptest.ResponseRecorder{
|
for route, rec := range map[string]*httptest.ResponseRecorder{
|
||||||
"respondImageError": plain, "handleImageError": encrypted,
|
"/v1/image/": plain, "/v1/e/": encrypted,
|
||||||
} {
|
} {
|
||||||
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
|
t.Logf("%s: status = %d, body = %q", route, rec.Code, rec.Body.String())
|
||||||
|
|
||||||
if rec.Code != http.StatusServiceUnavailable {
|
if rec.Code != http.StatusServiceUnavailable {
|
||||||
t.Errorf("%s: status = %d, want %d",
|
t.Errorf("%s: status = %d, want %d",
|
||||||
name, rec.Code, http.StatusServiceUnavailable)
|
route, rec.Code, http.StatusServiceUnavailable)
|
||||||
}
|
}
|
||||||
|
|
||||||
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
|
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
|
||||||
t.Errorf("%s: body = %q, want the server busy error",
|
t.Errorf("%s: body = %q, want the server busy error",
|
||||||
name, rec.Body.String())
|
route, rec.Body.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -473,18 +473,18 @@ func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
|
|||||||
return ErrUnsupportedScheme
|
return ErrUnsupportedScheme
|
||||||
}
|
}
|
||||||
|
|
||||||
parsedURL, err := neturl.Parse(rawURL)
|
// Parse to extract host
|
||||||
if err != nil {
|
host := extractHost(rawURL)
|
||||||
return fmt.Errorf("%w: %w", ErrInvalidHost, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Hostname drops any port, and the brackets of an IPv6 address whether
|
|
||||||
// or not a port follows them
|
|
||||||
host := parsedURL.Hostname()
|
|
||||||
if host == "" {
|
if host == "" {
|
||||||
return ErrInvalidHost
|
return ErrInvalidHost
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Remove port if present
|
||||||
|
h, _, err := net.SplitHostPort(host)
|
||||||
|
if err == nil {
|
||||||
|
host = h
|
||||||
|
}
|
||||||
|
|
||||||
// Block obvious localhost patterns
|
// Block obvious localhost patterns
|
||||||
if isLocalhost(host) {
|
if isLocalhost(host) {
|
||||||
return ErrSSRFBlocked
|
return ErrSSRFBlocked
|
||||||
@@ -543,13 +543,6 @@ func isPrivateIP(ip net.IP) bool {
|
|||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// An IPv4-compatible IPv6 address (in ::/96, such as ::127.0.0.1) holds
|
|
||||||
// an IPv4 address in its last four bytes and is judged as that address
|
|
||||||
if addr, ok := netip.AddrFromSlice(ip); ok &&
|
|
||||||
netip.MustParsePrefix("::/96").Contains(addr) {
|
|
||||||
return isPrivateIP(ip[12:])
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check for loopback
|
// Check for loopback
|
||||||
if ip.IsLoopback() {
|
if ip.IsLoopback() {
|
||||||
return true
|
return true
|
||||||
|
|||||||
@@ -186,44 +186,6 @@ func TestIsPrivateIP(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestIsPrivateIPJudgesEmbeddedIPv4Address checks IPv6 addresses that hold an
|
|
||||||
// IPv4 address in their last four bytes: the IPv4-compatible form (::/96) and
|
|
||||||
// the NAT64 form (64:ff9b::/96). One holding a loopback, private or
|
|
||||||
// link-local IPv4 address is refused as that address is, and one holding a
|
|
||||||
// public IPv4 address in the IPv4-compatible form passes.
|
|
||||||
func TestIsPrivateIPJudgesEmbeddedIPv4Address(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
ip string
|
|
||||||
want bool
|
|
||||||
}{
|
|
||||||
{"::127.0.0.1", true}, // loopback
|
|
||||||
{"::10.0.0.1", true}, // private
|
|
||||||
{"::169.254.169.254", true}, // link-local
|
|
||||||
{"64:ff9b::7f00:1", true}, // NAT64 form of 127.0.0.1
|
|
||||||
{"::8.8.8.8", false}, // public
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range tests {
|
|
||||||
t.Run(tc.ip, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
ip := net.ParseIP(tc.ip)
|
|
||||||
if ip == nil {
|
|
||||||
t.Fatalf("failed to parse IP %q", tc.ip)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := isPrivateIP(ip)
|
|
||||||
t.Logf("isPrivateIP(%s) = %v", ip, got)
|
|
||||||
|
|
||||||
if got != tc.want {
|
|
||||||
t.Errorf("isPrivateIP(%q) = %v, want %v", tc.ip, got, tc.want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
|
func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -50,69 +50,3 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestImageProcessor_ComputedSideIsAtLeastOne verifies that, for a size with
|
|
||||||
// one side 0, the side computed from the input's proportions is 1 where it
|
|
||||||
// would round down to 0, and that Process makes that size in every fit mode
|
|
||||||
// but FitOutside, which scales the input to cover the size and so can make
|
|
||||||
// it longer. Each case checks the size targetDimensions gives before it
|
|
||||||
// calls Process: with that side at 0, an 8192x1 input at 1300x0 makes
|
|
||||||
// libvips ask for about 39 GB with FitCover.
|
|
||||||
func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
inputWidth, inputHeight int
|
|
||||||
size Size
|
|
||||||
wantWidth, wantHeight int
|
|
||||||
}{
|
|
||||||
{"8192x1 at 100x0", 8192, 1, Size{Width: 100}, 100, 1},
|
|
||||||
{"8192x1 at 1300x0", 8192, 1, Size{Width: 1300}, 1300, 1},
|
|
||||||
{"8192x2 at 100x0", 8192, 2, Size{Width: 100}, 100, 1},
|
|
||||||
{"1x8192 at 0x100", 1, 8192, Size{Height: 100}, 1, 100},
|
|
||||||
{"1x8192 at 0x1300", 1, 8192, Size{Height: 1300}, 1, 1300},
|
|
||||||
{"2x8192 at 0x100", 2, 8192, Size{Height: 100}, 1, 100},
|
|
||||||
}
|
|
||||||
|
|
||||||
fits := []FitMode{FitCover, FitContain, FitFill, FitInside, FitOutside}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
for _, fit := range fits {
|
|
||||||
t.Run(tt.name+" "+string(fit), func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
width, height, err := targetDimensions(tt.size,
|
|
||||||
tt.inputWidth, tt.inputHeight)
|
|
||||||
if err != nil || width != tt.wantWidth || height != tt.wantHeight {
|
|
||||||
t.Fatalf("targetDimensions() = %dx%d, %v, want %dx%d",
|
|
||||||
width, height, err, tt.wantWidth, tt.wantHeight)
|
|
||||||
}
|
|
||||||
|
|
||||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
|
||||||
|
|
||||||
result, err := New(Params{}).Process(t.Context(),
|
|
||||||
bytes.NewReader(input),
|
|
||||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: fit})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Process() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = result.Content.Close() }()
|
|
||||||
|
|
||||||
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
|
||||||
|
|
||||||
ok := result.Width == tt.wantWidth && result.Height == tt.wantHeight
|
|
||||||
if fit == FitOutside {
|
|
||||||
ok = result.Width >= tt.wantWidth && result.Height >= tt.wantHeight
|
|
||||||
}
|
|
||||||
|
|
||||||
if !ok {
|
|
||||||
t.Errorf("Process() made %dx%d, want %dx%d (at least, for %s)",
|
|
||||||
result.Width, result.Height, tt.wantWidth, tt.wantHeight,
|
|
||||||
FitOutside)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,268 +0,0 @@
|
|||||||
package imageprocessor
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"image"
|
|
||||||
"image/color"
|
|
||||||
"image/png"
|
|
||||||
"math"
|
|
||||||
"testing"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestImageProcessor_FitCoverServesThinSource verifies that FitCover serves
|
|
||||||
// a thin source at the requested size. An 8192x1 source scaled to cover
|
|
||||||
// 8192x8192 is 8192 times as long as the output on its long side. The centre
|
|
||||||
// of an 8192x1 source with the proportions of 1x8192 is 1/8192 of a pixel
|
|
||||||
// wide. Each case is also turned on its side.
|
|
||||||
func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
inputWidth, inputHeight int
|
|
||||||
size Size
|
|
||||||
}{
|
|
||||||
{"8192x1 at 8192x8192", 8192, 1, Size{Width: 8192, Height: 8192}},
|
|
||||||
{"8192x1 at 1x8192", 8192, 1, Size{Width: 1, Height: 8192}},
|
|
||||||
{"1x8192 at 8192x8192", 1, 8192, Size{Width: 8192, Height: 8192}},
|
|
||||||
{"1x8192 at 8192x1", 1, 8192, Size{Width: 8192, Height: 1}},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
|
||||||
|
|
||||||
result, err := New(Params{}).Process(t.Context(),
|
|
||||||
bytes.NewReader(input),
|
|
||||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitCover})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Process() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = result.Content.Close() }()
|
|
||||||
|
|
||||||
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
|
||||||
|
|
||||||
if result.Width != tt.size.Width || result.Height != tt.size.Height {
|
|
||||||
t.Errorf("Process() made %dx%d, want %dx%d",
|
|
||||||
result.Width, result.Height, tt.size.Width, tt.size.Height)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// createBandsPNG creates a PNG of three equal bands, red, green and blue,
|
|
||||||
// side by side when it is wider than tall and top to bottom otherwise.
|
|
||||||
func createBandsPNG(t *testing.T, width, height int) []byte {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
bands := []color.RGBA{
|
|
||||||
{R: 255, A: 255},
|
|
||||||
{G: 255, A: 255},
|
|
||||||
{B: 255, A: 255},
|
|
||||||
}
|
|
||||||
|
|
||||||
img := image.NewRGBA(image.Rect(0, 0, width, height))
|
|
||||||
for y := range height {
|
|
||||||
for x := range width {
|
|
||||||
band := 3 * y / height
|
|
||||||
if width > height {
|
|
||||||
band = 3 * x / width
|
|
||||||
}
|
|
||||||
|
|
||||||
img.Set(x, y, bands[band])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return encodePNG(t, img)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestImageProcessor_FitCoverCropsTheCentre verifies that FitCover keeps the
|
|
||||||
// centre of the source: a 300x100 source of red, green and blue bands side
|
|
||||||
// by side, or a 100x300 one with them top to bottom, comes out at 50x50 as
|
|
||||||
// its green band.
|
|
||||||
func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
inputWidth, inputHeight int
|
|
||||||
}{
|
|
||||||
{"bands side by side", 300, 100},
|
|
||||||
{"bands top to bottom", 100, 300},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
input := createBandsPNG(t, tt.inputWidth, tt.inputHeight)
|
|
||||||
|
|
||||||
result, err := New(Params{}).Process(t.Context(),
|
|
||||||
bytes.NewReader(input),
|
|
||||||
&Request{
|
|
||||||
Size: Size{Width: 50, Height: 50},
|
|
||||||
Format: FormatPNG,
|
|
||||||
FitMode: FitCover,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Process() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = result.Content.Close() }()
|
|
||||||
|
|
||||||
output, err := png.Decode(result.Content)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to decode output: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
bounds := output.Bounds()
|
|
||||||
t.Logf("Process() made %dx%d", bounds.Dx(), bounds.Dy())
|
|
||||||
|
|
||||||
if bounds.Dx() != 50 || bounds.Dy() != 50 {
|
|
||||||
t.Fatalf("Process() made %dx%d, want 50x50",
|
|
||||||
bounds.Dx(), bounds.Dy())
|
|
||||||
}
|
|
||||||
|
|
||||||
// Resampling may blend a little of the bands beside the
|
|
||||||
// green one into the output's edges.
|
|
||||||
for y := bounds.Min.Y; y < bounds.Max.Y; y++ {
|
|
||||||
for x := bounds.Min.X; x < bounds.Max.X; x++ {
|
|
||||||
r, g, b, _ := output.At(x, y).RGBA()
|
|
||||||
if r>>8 > 64 || g>>8 < 192 || b>>8 > 64 {
|
|
||||||
t.Fatalf("output pixel %d,%d is %d,%d,%d, want green",
|
|
||||||
x, y, r>>8, g>>8, b>>8)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// The greys of createSquarePNG, and the grey halfway between them.
|
|
||||||
const (
|
|
||||||
backgroundGrey = 192
|
|
||||||
squareGrey = 64
|
|
||||||
halfwayGrey = (backgroundGrey + squareGrey) / 2
|
|
||||||
)
|
|
||||||
|
|
||||||
// createSquarePNG creates a grey PNG side pixels square with a darker grey
|
|
||||||
// square squareSide pixels wide in its centre.
|
|
||||||
func createSquarePNG(t *testing.T, side, squareSide int) []byte {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
first := (side - squareSide) / 2
|
|
||||||
last := first + squareSide - 1
|
|
||||||
|
|
||||||
img := image.NewGray(image.Rect(0, 0, side, side))
|
|
||||||
for y := range side {
|
|
||||||
for x := range side {
|
|
||||||
grey := uint8(backgroundGrey)
|
|
||||||
if x >= first && x <= last && y >= first && y <= last {
|
|
||||||
grey = squareGrey
|
|
||||||
}
|
|
||||||
|
|
||||||
img.SetGray(x, y, color.Gray{Y: grey})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return encodePNG(t, img)
|
|
||||||
}
|
|
||||||
|
|
||||||
// squareLength returns how long the square is along greys, the greys of a
|
|
||||||
// row or column of the output that crosses it: the distance between the
|
|
||||||
// first and the last point where greys passes halfwayGrey, or 0 when there
|
|
||||||
// is no such point. Resampling blurs the square's edges over several
|
|
||||||
// pixels, and each point is placed between the two pixels on either side
|
|
||||||
// of it in proportion to their greys, so the length is measured to a
|
|
||||||
// fraction of a pixel.
|
|
||||||
func squareLength(greys []float64) float64 {
|
|
||||||
first, last := -1.0, -1.0
|
|
||||||
|
|
||||||
for i := 1; i < len(greys); i++ {
|
|
||||||
before, after := greys[i-1], greys[i]
|
|
||||||
if (before > halfwayGrey) == (after > halfwayGrey) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
point := float64(i-1) + (before-halfwayGrey)/(before-after)
|
|
||||||
if first < 0 {
|
|
||||||
first = point
|
|
||||||
}
|
|
||||||
|
|
||||||
last = point
|
|
||||||
}
|
|
||||||
|
|
||||||
return last - first
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestImageProcessor_FitCoverKeepsProportions verifies that FitCover scales
|
|
||||||
// both sides of the source by the same amount: a square in the centre of
|
|
||||||
// the source comes out square. Its width and height are measured along the
|
|
||||||
// output's middle row and column.
|
|
||||||
func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
side, squareSide int
|
|
||||||
size Size
|
|
||||||
}{
|
|
||||||
{"16x16 at 100x60", 16, 8, Size{Width: 100, Height: 60}},
|
|
||||||
{"16x16 at 60x100", 16, 8, Size{Width: 60, Height: 100}},
|
|
||||||
{"10x10 at 100x35", 10, 2, Size{Width: 100, Height: 35}},
|
|
||||||
{"10x10 at 35x100", 10, 2, Size{Width: 35, Height: 100}},
|
|
||||||
{"64x64 at 300x100", 64, 16, Size{Width: 300, Height: 100}},
|
|
||||||
{"64x64 at 100x300", 64, 16, Size{Width: 100, Height: 300}},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
input := createSquarePNG(t, tt.side, tt.squareSide)
|
|
||||||
|
|
||||||
result, err := New(Params{}).Process(t.Context(),
|
|
||||||
bytes.NewReader(input),
|
|
||||||
&Request{Size: tt.size, Format: FormatPNG, FitMode: FitCover})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Process() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = result.Content.Close() }()
|
|
||||||
|
|
||||||
output, err := png.Decode(result.Content)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to decode output: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
bounds := output.Bounds()
|
|
||||||
if bounds.Dx() != tt.size.Width || bounds.Dy() != tt.size.Height {
|
|
||||||
t.Fatalf("Process() made %dx%d, want %dx%d",
|
|
||||||
bounds.Dx(), bounds.Dy(), tt.size.Width, tt.size.Height)
|
|
||||||
}
|
|
||||||
|
|
||||||
row := make([]float64, bounds.Dx())
|
|
||||||
for x := range row {
|
|
||||||
r, _, _, _ := output.At(x, bounds.Dy()/2).RGBA()
|
|
||||||
row[x] = float64(r >> 8)
|
|
||||||
}
|
|
||||||
|
|
||||||
column := make([]float64, bounds.Dy())
|
|
||||||
for y := range column {
|
|
||||||
r, _, _, _ := output.At(bounds.Dx()/2, y).RGBA()
|
|
||||||
column[y] = float64(r >> 8)
|
|
||||||
}
|
|
||||||
|
|
||||||
width, height := squareLength(row), squareLength(column)
|
|
||||||
t.Logf("square is %.2f wide and %.2f high", width, height)
|
|
||||||
|
|
||||||
if width == 0 || math.Abs(width-height) > 0.5 {
|
|
||||||
t.Errorf("square is %.2f wide and %.2f high, want it square",
|
|
||||||
width, height)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -60,45 +60,3 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestImageProcessor_FitOutsideSideIsAtLeastOne verifies that FitOutside
|
|
||||||
// keeps each side at 1 or more when it works both sides out again from the
|
|
||||||
// larger scale. A side of 49 scaled to 1 is 49 times 1/49, which in floating
|
|
||||||
// point is just under 1 and so rounded down to 0, a width libvips refuses.
|
|
||||||
func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
inputWidth, inputHeight int
|
|
||||||
size Size
|
|
||||||
wantWidth, wantHeight int
|
|
||||||
}{
|
|
||||||
{"49x8192 at 0x100", 49, 8192, Size{Height: 100}, 1, 167},
|
|
||||||
{"49x49 at 1x1", 49, 49, Size{Width: 1, Height: 1}, 1, 1},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
|
||||||
|
|
||||||
result, err := New(Params{}).Process(t.Context(),
|
|
||||||
bytes.NewReader(input),
|
|
||||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitOutside})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Process() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = result.Content.Close() }()
|
|
||||||
|
|
||||||
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
|
||||||
|
|
||||||
if result.Width != tt.wantWidth || result.Height != tt.wantHeight {
|
|
||||||
t.Errorf("Process() made %dx%d, want %dx%d",
|
|
||||||
result.Width, result.Height, tt.wantWidth, tt.wantHeight)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -7,11 +7,11 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"runtime"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/davidbyttow/govips/v2/vips"
|
"github.com/davidbyttow/govips/v2/vips"
|
||||||
"sneak.berlin/go/pixa/internal/config"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// vipsOnce ensures vips is initialized exactly once.
|
// vipsOnce ensures vips is initialized exactly once.
|
||||||
@@ -20,10 +20,11 @@ import (
|
|||||||
var vipsOnce sync.Once
|
var vipsOnce sync.Once
|
||||||
|
|
||||||
// initVips initializes libvips with quiet logging, one worker thread per
|
// initVips initializes libvips with quiet logging, one worker thread per
|
||||||
// image and no operation cache. Each request decodes different source
|
// image and no operation cache. Process already works on one image per CPU
|
||||||
// bytes, so the operation cache would rarely be hit and would hold memory
|
// by default, so more threads per image would only compete for the CPUs.
|
||||||
// outside MaxConcurrentProcessing; repeated requests are served from
|
// Each request decodes different source bytes, so the operation cache
|
||||||
// pixa's disk cache instead.
|
// would rarely be hit and would hold memory outside MaxConcurrentProcessing;
|
||||||
|
// repeated requests are served from pixa's disk cache instead.
|
||||||
func initVips() {
|
func initVips() {
|
||||||
vipsOnce.Do(func() {
|
vipsOnce.Do(func() {
|
||||||
vips.LoggingSettings(nil, vips.LogLevelError)
|
vips.LoggingSettings(nil, vips.LogLevelError)
|
||||||
@@ -178,7 +179,7 @@ type Params struct {
|
|||||||
// If <= 0, DefaultMaxInputBytes is used.
|
// If <= 0, DefaultMaxInputBytes is used.
|
||||||
MaxInputBytes int64
|
MaxInputBytes int64
|
||||||
// MaxConcurrentProcessing is the most images processed at once.
|
// MaxConcurrentProcessing is the most images processed at once.
|
||||||
// If <= 0, config.DefaultMaxConcurrentProcessing() is used.
|
// If <= 0, the number of CPUs Go uses (runtime.GOMAXPROCS(0)) is used.
|
||||||
MaxConcurrentProcessing int
|
MaxConcurrentProcessing int
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -194,7 +195,7 @@ func New(params Params) *ImageProcessor {
|
|||||||
|
|
||||||
maxConcurrentProcessing := params.MaxConcurrentProcessing
|
maxConcurrentProcessing := params.MaxConcurrentProcessing
|
||||||
if maxConcurrentProcessing <= 0 {
|
if maxConcurrentProcessing <= 0 {
|
||||||
maxConcurrentProcessing = config.DefaultMaxConcurrentProcessing()
|
maxConcurrentProcessing = runtime.GOMAXPROCS(0)
|
||||||
}
|
}
|
||||||
|
|
||||||
return &ImageProcessor{
|
return &ImageProcessor{
|
||||||
@@ -208,9 +209,8 @@ func New(params Params) *ImageProcessor {
|
|||||||
// MaxConcurrentProcessing images are already being processed, it waits up
|
// MaxConcurrentProcessing images are already being processed, it waits up
|
||||||
// to ProcessingWaitTimeout for one to finish, then fails with
|
// to ProcessingWaitTimeout for one to finish, then fails with
|
||||||
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
|
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
|
||||||
// encodes, it returns ctx's error and gives the slot back. govips gives pixa
|
// encodes, it returns ctx's error and gives the slot back. libvips cannot stop
|
||||||
// no way to stop libvips once an encode has begun, so an encode keeps its
|
// an encode once it has begun, so an encode keeps its slot until it finishes.
|
||||||
// slot until it finishes.
|
|
||||||
func (p *ImageProcessor) Process(
|
func (p *ImageProcessor) Process(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
input io.Reader,
|
input io.Reader,
|
||||||
@@ -311,10 +311,8 @@ func (p *ImageProcessor) Process(
|
|||||||
|
|
||||||
// targetDimensions calculates the output dimensions for a requested size,
|
// targetDimensions calculates the output dimensions for a requested size,
|
||||||
// scaling proportionally when only one dimension is given and keeping the
|
// scaling proportionally when only one dimension is given and keeping the
|
||||||
// original dimensions when both are zero. A side scaled proportionally is
|
// original dimensions when both are zero. A side scaled proportionally can
|
||||||
// rounded down but kept at 1 or more, as for a thin image it would round
|
// come out above MaxOutputDimension, which is ErrOutputTooLarge.
|
||||||
// down to 0, a side of no pixels. It can come out above MaxOutputDimension,
|
|
||||||
// which is ErrOutputTooLarge.
|
|
||||||
func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
|
func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
|
||||||
width, height := size.Width, size.Height
|
width, height := size.Width, size.Height
|
||||||
|
|
||||||
@@ -324,10 +322,10 @@ func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
|
|||||||
return origWidth, origHeight, nil
|
return origWidth, origHeight, nil
|
||||||
case width == 0:
|
case width == 0:
|
||||||
// Only height specified: calculate width proportionally
|
// Only height specified: calculate width proportionally
|
||||||
width = max(1, origWidth*height/origHeight)
|
width = origWidth * height / origHeight
|
||||||
case height == 0:
|
case height == 0:
|
||||||
// Only width specified: calculate height proportionally
|
// Only width specified: calculate height proportionally
|
||||||
height = max(1, origHeight*width/origWidth)
|
height = origHeight * width / origWidth
|
||||||
}
|
}
|
||||||
|
|
||||||
if width > MaxOutputDimension || height > MaxOutputDimension {
|
if width > MaxOutputDimension || height > MaxOutputDimension {
|
||||||
@@ -477,20 +475,17 @@ func (p *ImageProcessor) resize(
|
|||||||
) error {
|
) error {
|
||||||
switch fit {
|
switch fit {
|
||||||
case FitCover, "":
|
case FitCover, "":
|
||||||
// Crop the centre and resize to fill exact dimensions (default)
|
// Resize and crop to fill exact dimensions (default)
|
||||||
return cropCentreAndResize(img, width, height)
|
return img.Thumbnail(width, height, vips.InterestingCentre)
|
||||||
|
|
||||||
case FitContain:
|
case FitContain:
|
||||||
// Resize to fit within dimensions, maintaining aspect ratio. Scaled
|
// Resize to fit within dimensions, maintaining aspect ratio
|
||||||
// by the smaller scale, the short side of a thin image can round
|
|
||||||
// down to 0, which libvips refuses as a width, so each side is at
|
|
||||||
// least 1.
|
|
||||||
imgW, imgH := img.Width(), img.Height()
|
imgW, imgH := img.Width(), img.Height()
|
||||||
scaleW := float64(width) / float64(imgW)
|
scaleW := float64(width) / float64(imgW)
|
||||||
scaleH := float64(height) / float64(imgH)
|
scaleH := float64(height) / float64(imgH)
|
||||||
scale := min(scaleW, scaleH)
|
scale := min(scaleW, scaleH)
|
||||||
newW := max(1, int(float64(imgW)*scale))
|
newW := int(float64(imgW) * scale)
|
||||||
newH := max(1, int(float64(imgH)*scale))
|
newH := int(float64(imgH) * scale)
|
||||||
|
|
||||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||||
|
|
||||||
@@ -508,22 +503,19 @@ func (p *ImageProcessor) resize(
|
|||||||
scaleW := float64(width) / float64(imgW)
|
scaleW := float64(width) / float64(imgW)
|
||||||
scaleH := float64(height) / float64(imgH)
|
scaleH := float64(height) / float64(imgH)
|
||||||
scale := min(scaleW, scaleH)
|
scale := min(scaleW, scaleH)
|
||||||
newW := max(1, int(float64(imgW)*scale))
|
newW := int(float64(imgW) * scale)
|
||||||
newH := max(1, int(float64(imgH)*scale))
|
newH := int(float64(imgH) * scale)
|
||||||
|
|
||||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||||
|
|
||||||
case FitOutside:
|
case FitOutside:
|
||||||
// Resize so smallest dimension fits, may exceed target on other
|
// Resize so smallest dimension fits, may exceed target on other dimension
|
||||||
// dimension. A side scaled to 1 can come out just under 1 in
|
|
||||||
// floating point, as 49 times 1/49 does, and round down to 0, which
|
|
||||||
// libvips refuses as a width, so each side is at least 1.
|
|
||||||
imgW, imgH := img.Width(), img.Height()
|
imgW, imgH := img.Width(), img.Height()
|
||||||
scaleW := float64(width) / float64(imgW)
|
scaleW := float64(width) / float64(imgW)
|
||||||
scaleH := float64(height) / float64(imgH)
|
scaleH := float64(height) / float64(imgH)
|
||||||
scale := max(scaleW, scaleH)
|
scale := max(scaleW, scaleH)
|
||||||
newW := max(1, int(float64(imgW)*scale))
|
newW := int(float64(imgW) * scale)
|
||||||
newH := max(1, int(float64(imgH)*scale))
|
newH := int(float64(imgH) * scale)
|
||||||
|
|
||||||
// The dimension that exceeds the target is held to the maximum too
|
// The dimension that exceeds the target is held to the maximum too
|
||||||
if newW > MaxOutputDimension || newH > MaxOutputDimension {
|
if newW > MaxOutputDimension || newH > MaxOutputDimension {
|
||||||
@@ -538,43 +530,11 @@ func (p *ImageProcessor) resize(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// cropCentreAndResize scales img by the same amount on both sides, the
|
|
||||||
// least that makes it cover width x height, and cuts the centre width x
|
|
||||||
// height out of it, as libvips' thumbnail does with InterestingCentre.
|
|
||||||
// thumbnail alone scales the whole image and copies the result into memory
|
|
||||||
// before it cuts, and for a thin image that is far larger than the image or
|
|
||||||
// the output: 8192x64 at 8192x8192 is 1048576x8192, about 24 GiB. So the
|
|
||||||
// centre is cut from img first, widened to whole pixels of img, which adds
|
|
||||||
// less than one pixel of img on each side, and thumbnail scales and cuts
|
|
||||||
// only that.
|
|
||||||
func cropCentreAndResize(img *vips.ImageRef, width, height int) error {
|
|
||||||
imgW, imgH := img.Width(), img.Height()
|
|
||||||
|
|
||||||
// An image wider than the requested proportions keeps its height and
|
|
||||||
// loses width down to imgH*width/height, any other keeps its width and
|
|
||||||
// loses height down to imgW*height/width. Half of what is lost comes
|
|
||||||
// off each side, rounded down to whole pixels, so the part kept holds
|
|
||||||
// all of the centre.
|
|
||||||
left, top := 0, 0
|
|
||||||
if imgW*height > imgH*width {
|
|
||||||
left = (imgW*height - imgH*width) / (2 * height) //nolint:mnd // each side
|
|
||||||
} else {
|
|
||||||
top = (imgH*width - imgW*height) / (2 * width) //nolint:mnd // each side
|
|
||||||
}
|
|
||||||
|
|
||||||
err := img.ExtractArea(left, top, imgW-2*left, imgH-2*top)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
return img.Thumbnail(width, height, vips.InterestingCentre)
|
|
||||||
}
|
|
||||||
|
|
||||||
const defaultQuality = 85
|
const defaultQuality = 85
|
||||||
|
|
||||||
// encode encodes an image to the specified format. govips gives pixa no way
|
// encode encodes an image to the specified format. libvips cannot stop an
|
||||||
// to stop libvips once an encode has begun, so when ctx has ended encode
|
// encode once it has begun, so when ctx has ended encode returns ctx's error
|
||||||
// returns ctx's error without beginning one.
|
// without beginning one.
|
||||||
func (p *ImageProcessor) encode(
|
func (p *ImageProcessor) encode(
|
||||||
ctx context.Context, img *vips.ImageRef, format Format, quality int,
|
ctx context.Context, img *vips.ImageRef, format Format, quality int,
|
||||||
) ([]byte, error) {
|
) ([]byte, error) {
|
||||||
@@ -684,11 +644,10 @@ func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
|
|||||||
|
|
||||||
// avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest
|
// avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest
|
||||||
// govips can set. With one thread, as pixad runs libvips, 1 takes about 51
|
// govips can set. With one thread, as pixad runs libvips, 1 takes about 51
|
||||||
// seconds to save an 8192x8192 RGB image of random pixels, and about 110
|
// seconds to save an 8192x8192 image of random pixels, the worst case,
|
||||||
// with an alpha channel, the worst case, against the default
|
// against the default downstream_timeout of 60 seconds. On an image of
|
||||||
// downstream_timeout of 60 seconds. On an image of milder noise, which 1
|
// milder noise, which 1 saves in about 12 seconds, 2 takes nearly a minute
|
||||||
// saves in about 12 seconds, 2 takes nearly a minute and libvips' default,
|
// and libvips' default, 4, takes minutes.
|
||||||
// 4, takes minutes.
|
|
||||||
const avifEffort = 1
|
const avifEffort = 1
|
||||||
|
|
||||||
// avifBitdepth is the AVIF bit depth, 8 bits per sample for every image.
|
// avifBitdepth is the AVIF bit depth, 8 bits per sample for every image.
|
||||||
|
|||||||
@@ -115,14 +115,14 @@ func waitForEntries(t *testing.T, entered <-chan struct{}, count int) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestNewDefaultsMaxConcurrentProcessingToHalfTheCPUs(t *testing.T) {
|
func TestNewDefaultsMaxConcurrentProcessingToCPUs(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
for _, limit := range []int{0, -1} {
|
for _, limit := range []int{0, -1} {
|
||||||
proc := New(Params{MaxConcurrentProcessing: limit})
|
proc := New(Params{MaxConcurrentProcessing: limit})
|
||||||
if got := cap(proc.processingSemaphore); got != max(runtime.GOMAXPROCS(0)/2, 1) {
|
if got := cap(proc.processingSemaphore); got != runtime.GOMAXPROCS(0) {
|
||||||
t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, half the CPUs",
|
t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, one per CPU",
|
||||||
limit, got, max(runtime.GOMAXPROCS(0)/2, 1))
|
limit, got, runtime.GOMAXPROCS(0))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -132,30 +132,6 @@ func TestNewDefaultsMaxConcurrentProcessingToHalfTheCPUs(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestNewDefaultIsHalfTheCPUsAndAtLeastOne checks that New, given no
|
|
||||||
// MaxConcurrentProcessing, makes one slot for every two CPUs Go uses,
|
|
||||||
// rounded down, and one slot when Go uses one CPU.
|
|
||||||
//
|
|
||||||
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
|
|
||||||
func TestNewDefaultIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
|
|
||||||
cpusBefore := runtime.GOMAXPROCS(0)
|
|
||||||
|
|
||||||
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
|
|
||||||
|
|
||||||
for _, tc := range []struct{ cpus, want int }{
|
|
||||||
{1, 1}, {2, 1}, {3, 1}, {8, 4},
|
|
||||||
} {
|
|
||||||
runtime.GOMAXPROCS(tc.cpus)
|
|
||||||
|
|
||||||
got := cap(New(Params{}).processingSemaphore)
|
|
||||||
t.Logf("GOMAXPROCS %d: %d slots", tc.cpus, got)
|
|
||||||
|
|
||||||
if got != tc.want {
|
|
||||||
t.Errorf("GOMAXPROCS %d: %d slots, want %d", tc.cpus, got, tc.want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestProcessNeverExceedsMaxConcurrentProcessing starts more Process calls
|
// TestProcessNeverExceedsMaxConcurrentProcessing starts more Process calls
|
||||||
// than MaxConcurrentProcessing allows and holds the first ones inside
|
// than MaxConcurrentProcessing allows and holds the first ones inside
|
||||||
// Process until the test lets them go. No more than the limit may be
|
// Process until the test lets them go. No more than the limit may be
|
||||||
|
|||||||
+15
-56
@@ -32,12 +32,6 @@ const httpStatusOK = 200
|
|||||||
// variant not among them is served as before, reading its .meta file.
|
// variant not among them is served as before, reading its .meta file.
|
||||||
const metaCacheSize = 10000
|
const metaCacheSize = 10000
|
||||||
|
|
||||||
// cleanExpiredBatchSize is the most expired negative cache rows one
|
|
||||||
// statement of CleanExpired deletes. Each batch is a statement of its
|
|
||||||
// own, so a request waits behind one batch at most, however many rows
|
|
||||||
// have expired.
|
|
||||||
const cleanExpiredBatchSize = 1000
|
|
||||||
|
|
||||||
// CacheConfig holds cache configuration.
|
// CacheConfig holds cache configuration.
|
||||||
type CacheConfig struct {
|
type CacheConfig struct {
|
||||||
StateDir string
|
StateDir string
|
||||||
@@ -56,9 +50,8 @@ type CacheConfig struct {
|
|||||||
|
|
||||||
// DisableDiskCache turns the disk cache off entirely: no cache
|
// DisableDiskCache turns the disk cache off entirely: no cache
|
||||||
// directories are created, lookups always miss, stores are
|
// directories are created, lookups always miss, stores are
|
||||||
// no-ops, and the eviction goroutine only removes the expired
|
// no-ops, and no eviction machinery runs. The config layer sets
|
||||||
// negative cache rows. The config layer sets this when the
|
// this when the operator configures cache_max_bytes: 0.
|
||||||
// operator configures cache_max_bytes: 0.
|
|
||||||
DisableDiskCache bool
|
DisableDiskCache bool
|
||||||
|
|
||||||
// Logger receives accounting and eviction log output. A nil
|
// Logger receives accounting and eviction log output. A nil
|
||||||
@@ -489,27 +482,17 @@ func (c *Cache) GetSourceContent(
|
|||||||
return c.srcContent.LoadWithSize(contentHash)
|
return c.srcContent.LoadWithSize(contentHash)
|
||||||
}
|
}
|
||||||
|
|
||||||
// CleanExpired removes the negative cache rows past their expiry, one
|
// CleanExpired removes expired entries from the cache.
|
||||||
// batch per statement, until none is left. Once ctx ends it starts no
|
|
||||||
// further batch and returns ctx's error. The eviction goroutine calls it
|
|
||||||
// at startup and on every periodic tick, so a row stays about
|
|
||||||
// NegativeTTL plus one eviction interval, whether or not its URL is
|
|
||||||
// requested again.
|
|
||||||
func (c *Cache) CleanExpired(ctx context.Context) error {
|
func (c *Cache) CleanExpired(ctx context.Context) error {
|
||||||
for {
|
// Clean expired negative cache entries
|
||||||
if ctx.Err() != nil {
|
_, err := c.db.ExecContext(ctx, `
|
||||||
return ctx.Err()
|
DELETE FROM negative_cache WHERE expires_at < CURRENT_TIMESTAMP
|
||||||
}
|
`)
|
||||||
|
if err != nil {
|
||||||
deleted, err := c.cleanExpiredBatch(ctx)
|
return fmt.Errorf("failed to clean negative cache: %w", err)
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
if deleted < cleanExpiredBatchSize {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stats returns cache statistics.
|
// Stats returns cache statistics.
|
||||||
@@ -660,10 +643,10 @@ func (c *Cache) readHitAndMissCounts(ctx context.Context, stats *CacheStats) err
|
|||||||
// hold: it reads the content type from the variant's .meta file and
|
// hold: it reads the content type from the variant's .meta file and
|
||||||
// keeps it in metaCache, unless a StoreVariant has put one there
|
// keeps it in metaCache, unless a StoreVariant has put one there
|
||||||
// meanwhile, as the store's is newer. A read that finds no .meta file,
|
// meanwhile, as the store's is newer. A read that finds no .meta file,
|
||||||
// as one can when the store's .meta write failed or an eviction removed
|
// as one can between a store's writing of the variant file and of its
|
||||||
// the file after the read opened the variant, serves
|
// .meta file, serves application/octet-stream and keeps nothing, so
|
||||||
// application/octet-stream and keeps nothing, so metaCache only ever
|
// metaCache only ever holds a type read from a .meta file or passed to
|
||||||
// holds a type read from a .meta file or passed to StoreVariant.
|
// StoreVariant.
|
||||||
func (c *Cache) loadVariantWithMeta(
|
func (c *Cache) loadVariantWithMeta(
|
||||||
cacheKey VariantKey,
|
cacheKey VariantKey,
|
||||||
) (io.ReadCloser, int64, string, error) {
|
) (io.ReadCloser, int64, string, error) {
|
||||||
@@ -773,27 +756,3 @@ func (c *Cache) checkNegativeCache(
|
|||||||
|
|
||||||
return true, nil
|
return true, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// cleanExpiredBatch deletes up to cleanExpiredBatchSize negative cache
|
|
||||||
// rows past their expiry in one statement, and returns how many it
|
|
||||||
// deleted. DELETE ... LIMIT needs a SQLite build option, so the subquery
|
|
||||||
// picks the rows.
|
|
||||||
func (c *Cache) cleanExpiredBatch(ctx context.Context) (int64, error) {
|
|
||||||
result, err := c.db.ExecContext(ctx, `
|
|
||||||
DELETE FROM negative_cache WHERE id IN (
|
|
||||||
SELECT id FROM negative_cache
|
|
||||||
WHERE expires_at < CURRENT_TIMESTAMP
|
|
||||||
LIMIT ?
|
|
||||||
)
|
|
||||||
`, cleanExpiredBatchSize)
|
|
||||||
if err != nil {
|
|
||||||
return 0, fmt.Errorf("failed to clean negative cache: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
deleted, err := result.RowsAffected()
|
|
||||||
if err != nil {
|
|
||||||
return 0, fmt.Errorf("failed to count cleaned negative cache rows: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return deleted, nil
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -78,6 +78,10 @@ func (c *Cache) computeDefaultMaxBytes(
|
|||||||
computed := spaceBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
|
computed := spaceBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
|
||||||
computed = min(computed, math.MaxInt64)
|
computed = min(computed, math.MaxInt64)
|
||||||
|
|
||||||
|
// gosec cannot see that min() above bounds computed, so it reads
|
||||||
|
// this conversion as potentially overflowing. It cannot: computed is
|
||||||
|
// at most math.MaxInt64 on every path here.
|
||||||
|
//nolint:gosec // G115: clamped to MaxInt64 by min above
|
||||||
limit := int64(computed)
|
limit := int64(computed)
|
||||||
limit = max(limit, DefaultCacheMaxBytesFloor)
|
limit = max(limit, DefaultCacheMaxBytesFloor)
|
||||||
|
|
||||||
|
|||||||
+53
-114
@@ -4,7 +4,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"database/sql"
|
"database/sql"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
|
||||||
"fmt"
|
"fmt"
|
||||||
"io/fs"
|
"io/fs"
|
||||||
"os"
|
"os"
|
||||||
@@ -438,16 +437,14 @@ func (c *Cache) notifyWritePressure() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// StartEviction launches the background eviction goroutine, which
|
// StartEviction launches the background eviction goroutine, which
|
||||||
// reconciles the database accounting with the cache directories and
|
// reconciles the database accounting with the cache directories at
|
||||||
// removes the expired negative cache rows at startup and again on every
|
// startup and again on every periodic tick thereafter, and evicts to
|
||||||
// periodic tick thereafter, and evicts to the configured limit on the
|
// the configured limit on the given periodic interval and on
|
||||||
// given periodic interval and on write-pressure notifications. On a
|
// write-pressure notifications. It is a no-op on a disabled cache or
|
||||||
// disabled cache it only removes the expired negative cache rows, as
|
// when already started. The goroutine outlives the caller, so it runs
|
||||||
// the negative cache is used either way. It is a no-op when already
|
// with its own context, which StopEviction cancels.
|
||||||
// started. The goroutine outlives the caller, so it runs with its own
|
|
||||||
// context, which StopEviction cancels.
|
|
||||||
func (c *Cache) StartEviction(interval time.Duration) {
|
func (c *Cache) StartEviction(interval time.Duration) {
|
||||||
if c.evictionCancel != nil {
|
if c.disabled || c.evictionCancel != nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -483,7 +480,6 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
|
|||||||
defer close(c.evictionDone)
|
defer close(c.evictionDone)
|
||||||
|
|
||||||
c.runReconciliationPass(ctx)
|
c.runReconciliationPass(ctx)
|
||||||
c.runCleanExpired(ctx)
|
|
||||||
c.runEvictionPass(ctx)
|
c.runEvictionPass(ctx)
|
||||||
|
|
||||||
ticker := time.NewTicker(interval)
|
ticker := time.NewTicker(interval)
|
||||||
@@ -504,7 +500,6 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
|
|||||||
// unaccounted for to one interval, on a process that is
|
// unaccounted for to one interval, on a process that is
|
||||||
// already running this loop regardless.
|
// already running this loop regardless.
|
||||||
c.runReconciliationPass(ctx)
|
c.runReconciliationPass(ctx)
|
||||||
c.runCleanExpired(ctx)
|
|
||||||
case <-c.evictionPressure:
|
case <-c.evictionPressure:
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -540,20 +535,6 @@ func (c *Cache) runReconciliationPass(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// runCleanExpired runs CleanExpired, logging a failure instead of
|
|
||||||
// propagating it (the loop must keep running). It does nothing once ctx
|
|
||||||
// is cancelled.
|
|
||||||
func (c *Cache) runCleanExpired(ctx context.Context) {
|
|
||||||
if ctx.Err() != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
err := c.CleanExpired(ctx)
|
|
||||||
if err != nil {
|
|
||||||
c.log.Warn("negative cache cleanup failed", "error", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// reconcileAccounting synchronizes the database size accounting with
|
// reconcileAccounting synchronizes the database size accounting with
|
||||||
// the actual contents of the cache directories. It runs at startup and
|
// the actual contents of the cache directories. It runs at startup and
|
||||||
// again on every periodic eviction tick thereafter, off the request
|
// again on every periodic eviction tick thereafter, off the request
|
||||||
@@ -561,11 +542,10 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
|
|||||||
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
|
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
|
||||||
// insert under transient DB contention), drops accounting rows whose
|
// insert under transient DB contention), drops accounting rows whose
|
||||||
// files are missing, removes source blob files the database does not
|
// files are missing, removes source blob files the database does not
|
||||||
// know (and rows whose files are gone), removes source and variant files
|
// know (and rows whose files are gone), sweeps stale temp files left
|
||||||
// that are empty or not the size their rows record, sweeps stale temp
|
// behind by crashed writes, and last checks the total cache usage
|
||||||
// files left behind by crashed writes, and last checks the total cache
|
// against the tables. It reads the tables a page at a time. Running it
|
||||||
// usage against the tables. It reads the tables a page at a time.
|
// periodically, not just once, bounds how long such drift can
|
||||||
// Running it periodically, not just once, bounds how long such drift can
|
|
||||||
// accumulate unaccounted for on a long-running process to one eviction
|
// accumulate unaccounted for on a long-running process to one eviction
|
||||||
// interval. Once ctx is cancelled, it stops at the next file or row and
|
// interval. Once ctx is cancelled, it stops at the next file or row and
|
||||||
// returns ctx's error.
|
// returns ctx's error.
|
||||||
@@ -597,9 +577,8 @@ func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
|||||||
return c.reconcileUsageTotal(ctx)
|
return c.reconcileUsageTotal(ctx)
|
||||||
}
|
}
|
||||||
|
|
||||||
// reconcileVariantFiles walks the variant storage directory, checking
|
// reconcileVariantFiles walks the variant storage directory, adopting
|
||||||
// each variant file against its accounting row (reconcileVariantFile)
|
// files without accounting rows and sweeping stale temp files.
|
||||||
// and sweeping stale temp files.
|
|
||||||
func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
||||||
return filepath.WalkDir(
|
return filepath.WalkDir(
|
||||||
c.variants.baseDir,
|
c.variants.baseDir,
|
||||||
@@ -624,55 +603,38 @@ func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return c.reconcileVariantFile(ctx, path, entry, VariantKey(name))
|
return c.adoptVariantFile(ctx, path, entry, VariantKey(name))
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// reconcileVariantFile checks one variant file against its accounting
|
|
||||||
// row. A file without a row is adopted into the accounting, unless it is
|
|
||||||
// empty. A file that is empty, or not the size its row records, as a
|
|
||||||
// crash or a restored snapshot can leave it, is removed with its row and
|
|
||||||
// a warning naming it, so the variant is converted again.
|
|
||||||
func (c *Cache) reconcileVariantFile(
|
|
||||||
ctx context.Context, path string, entry fs.DirEntry, cacheKey VariantKey,
|
|
||||||
) error {
|
|
||||||
info, err := entry.Info()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to stat variant file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var recordedSize int64
|
|
||||||
|
|
||||||
err = c.db.QueryRowContext(ctx,
|
|
||||||
`SELECT size_bytes FROM variant_content WHERE cache_key = ?`, string(cacheKey),
|
|
||||||
).Scan(&recordedSize)
|
|
||||||
|
|
||||||
switch {
|
|
||||||
case errors.Is(err, sql.ErrNoRows) && info.Size() > 0:
|
|
||||||
return c.adoptVariantFile(ctx, path, info, cacheKey)
|
|
||||||
case err != nil && !errors.Is(err, sql.ErrNoRows):
|
|
||||||
return fmt.Errorf("failed to check variant accounting row: %w", err)
|
|
||||||
case info.Size() > 0 && info.Size() == recordedSize:
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// The file is empty, or not the size its row records
|
|
||||||
c.log.Warn("removing damaged cached variant file", "path", path,
|
|
||||||
"file_bytes", info.Size(), "recorded_bytes", recordedSize)
|
|
||||||
|
|
||||||
return c.evictVariant(ctx, cacheKey)
|
|
||||||
}
|
|
||||||
|
|
||||||
// adoptVariantFile inserts an accounting row for a variant file that
|
// adoptVariantFile inserts an accounting row for a variant file that
|
||||||
// has none, using the file's size and modification time.
|
// has none, using the file's size and modification time.
|
||||||
func (c *Cache) adoptVariantFile(
|
func (c *Cache) adoptVariantFile(
|
||||||
ctx context.Context, path string, info fs.FileInfo, cacheKey VariantKey,
|
ctx context.Context, path string, entry fs.DirEntry, cacheKey VariantKey,
|
||||||
) error {
|
) error {
|
||||||
|
var rowExists int
|
||||||
|
|
||||||
|
err := c.db.QueryRowContext(ctx,
|
||||||
|
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, string(cacheKey),
|
||||||
|
).Scan(&rowExists)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to check variant accounting row: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if rowExists > 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
info, err := entry.Info()
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to stat variant file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
modTime := info.ModTime().UTC().Format(sqliteTimestampLayout)
|
modTime := info.ModTime().UTC().Format(sqliteTimestampLayout)
|
||||||
contentType := c.variantContentTypeFromSidecar(path)
|
contentType := c.variantContentTypeFromSidecar(path)
|
||||||
|
|
||||||
_, err := c.db.ExecContext(ctx, `
|
_, err = c.db.ExecContext(ctx, `
|
||||||
INSERT INTO variant_content
|
INSERT INTO variant_content
|
||||||
(cache_key, size_bytes, content_type, created_at, last_accessed_at)
|
(cache_key, size_bytes, content_type, created_at, last_accessed_at)
|
||||||
VALUES (?, ?, ?, ?, ?)
|
VALUES (?, ?, ?, ?, ?)
|
||||||
@@ -791,9 +753,10 @@ func queryStringColumn[T ~string](
|
|||||||
return values, nil
|
return values, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// reconcileSourceFiles walks the source content directory, checking
|
// reconcileSourceFiles walks the source content directory, removing
|
||||||
// each blob file against its source_content row (reconcileSourceFile)
|
// blob files the database does not track (they are unreachable: source
|
||||||
// and sweeping stale temp files.
|
// lookups always go through source_metadata) and sweeping stale temp
|
||||||
|
// files.
|
||||||
func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
|
func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
|
||||||
return filepath.WalkDir(
|
return filepath.WalkDir(
|
||||||
c.srcContent.baseDir,
|
c.srcContent.baseDir,
|
||||||
@@ -814,55 +777,31 @@ func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return c.reconcileSourceFile(ctx, path, entry, ContentHash(name))
|
return c.removeUntrackedSourceFile(ctx, path, ContentHash(name))
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// reconcileSourceFile checks one source blob file against its
|
|
||||||
// source_content row. A file without a row is removed: it is
|
|
||||||
// unreachable, as source lookups always go through source_metadata. A
|
|
||||||
// file that is empty, or not the size its row records, as a crash or a
|
|
||||||
// restored snapshot can leave it, is removed with its rows and a warning
|
|
||||||
// naming it, so the source is fetched again.
|
|
||||||
func (c *Cache) reconcileSourceFile(
|
|
||||||
ctx context.Context, path string, entry fs.DirEntry, contentHash ContentHash,
|
|
||||||
) error {
|
|
||||||
info, err := entry.Info()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to stat source content file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var recordedSize int64
|
|
||||||
|
|
||||||
err = c.db.QueryRowContext(ctx,
|
|
||||||
`SELECT size_bytes FROM source_content WHERE content_hash = ?`,
|
|
||||||
string(contentHash),
|
|
||||||
).Scan(&recordedSize)
|
|
||||||
|
|
||||||
switch {
|
|
||||||
case errors.Is(err, sql.ErrNoRows):
|
|
||||||
return c.removeUntrackedSourceFile(ctx, path, contentHash)
|
|
||||||
case err != nil:
|
|
||||||
return fmt.Errorf("failed to check source content row: %w", err)
|
|
||||||
case info.Size() > 0 && info.Size() == recordedSize:
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// The file is empty, or not the size its row records
|
|
||||||
c.log.Warn("removing damaged cached source file", "path", path,
|
|
||||||
"file_bytes", info.Size(), "recorded_bytes", recordedSize)
|
|
||||||
|
|
||||||
return c.evictSourceBlob(ctx, contentHash)
|
|
||||||
}
|
|
||||||
|
|
||||||
// removeUntrackedSourceFile deletes a source blob file that has no
|
// removeUntrackedSourceFile deletes a source blob file that has no
|
||||||
// source_content row. Any source_metadata rows referencing the hash
|
// source_content row. Any source_metadata rows referencing the hash
|
||||||
// are removed first so no row ever points at a deleted file.
|
// are removed first so no row ever points at a deleted file.
|
||||||
func (c *Cache) removeUntrackedSourceFile(
|
func (c *Cache) removeUntrackedSourceFile(
|
||||||
ctx context.Context, path string, contentHash ContentHash,
|
ctx context.Context, path string, contentHash ContentHash,
|
||||||
) error {
|
) error {
|
||||||
_, err := c.db.ExecContext(ctx,
|
var rowExists int
|
||||||
|
|
||||||
|
err := c.db.QueryRowContext(ctx,
|
||||||
|
`SELECT COUNT(*) FROM source_content WHERE content_hash = ?`, string(contentHash),
|
||||||
|
).Scan(&rowExists)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to check source content row: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if rowExists > 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = c.db.ExecContext(ctx,
|
||||||
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash))
|
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to delete metadata rows for untracked blob: %w", err)
|
return fmt.Errorf("failed to delete metadata rows for untracked blob: %w", err)
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"net/url"
|
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -87,11 +86,9 @@ func ValidateImageRequest(req *ImageRequest) error {
|
|||||||
type ImageRequest struct {
|
type ImageRequest struct {
|
||||||
// SourceHost is the origin host (e.g., "cdn.example.com")
|
// SourceHost is the origin host (e.g., "cdn.example.com")
|
||||||
SourceHost string
|
SourceHost string
|
||||||
// SourcePath is the path on the origin with its percent-escapes decoded,
|
// SourcePath is the path on the origin (e.g., "/photos/cat.jpg")
|
||||||
// as url.URL's Path holds it (e.g., "/photos/my cat.jpg")
|
|
||||||
SourcePath string
|
SourcePath string
|
||||||
// SourceQuery is the optional query string for the origin URL as the URL
|
// SourceQuery is the optional query string for the origin URL
|
||||||
// writes it, percent-escapes included, as url.URL's RawQuery holds it
|
|
||||||
SourceQuery string
|
SourceQuery string
|
||||||
// Size is the requested output dimensions
|
// Size is the requested output dimensions
|
||||||
Size Size
|
Size Size
|
||||||
@@ -110,25 +107,20 @@ type ImageRequest struct {
|
|||||||
AllowHTTP bool
|
AllowHTTP bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// SourceURL returns the full upstream URL to fetch: the source path
|
// SourceURL returns the full upstream URL to fetch.
|
||||||
// percent-encoded again, as net/url writes a path, so each source path is one
|
// Uses http:// scheme when AllowHTTP is true, otherwise https://.
|
||||||
// upstream path, and the source query as it is, with the characters
|
|
||||||
// escapeQuery encodes percent-encoded. Uses http:// scheme when AllowHTTP is
|
|
||||||
// true, otherwise https://.
|
|
||||||
func (r *ImageRequest) SourceURL() string {
|
func (r *ImageRequest) SourceURL() string {
|
||||||
scheme := "https"
|
scheme := "https"
|
||||||
if r.AllowHTTP {
|
if r.AllowHTTP {
|
||||||
scheme = "http"
|
scheme = "http"
|
||||||
}
|
}
|
||||||
|
|
||||||
sourceURL := url.URL{
|
url := scheme + "://" + r.SourceHost + r.SourcePath
|
||||||
Scheme: scheme,
|
if r.SourceQuery != "" {
|
||||||
Host: r.SourceHost,
|
url += "?" + r.SourceQuery
|
||||||
Path: r.SourcePath,
|
|
||||||
RawQuery: escapeQuery(r.SourceQuery),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return sourceURL.String()
|
return url
|
||||||
}
|
}
|
||||||
|
|
||||||
// ImageResponse represents a processed image ready to serve
|
// ImageResponse represents a processed image ready to serve
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package imgcache
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"strconv"
|
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
@@ -160,242 +159,6 @@ func TestService_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestEvictionLoopRemovesExpiredNegativeCacheRows stores a failed fetch
|
|
||||||
// that expires a second later and starts the eviction loop with a short
|
|
||||||
// interval: a later tick must remove its row although its URL is never
|
|
||||||
// requested again, and must keep a row that has not expired. It runs
|
|
||||||
// with the disk cache enabled and disabled, as the negative cache is
|
|
||||||
// used either way.
|
|
||||||
func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
disableDiskCache bool
|
|
||||||
}{
|
|
||||||
{name: "disk cache enabled", disableDiskCache: false},
|
|
||||||
{name: "disk cache disabled", disableDiskCache: true},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cache, err := NewCache(evictionTestDB(t), CacheConfig{
|
|
||||||
StateDir: t.TempDir(),
|
|
||||||
NegativeTTL: time.Second,
|
|
||||||
DisableDiskCache: tt.disableDiskCache,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create cache: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
expired := &ImageRequest{
|
|
||||||
SourceHost: testHostExample, SourcePath: "/gone.jpg",
|
|
||||||
}
|
|
||||||
|
|
||||||
err = cache.StoreNegative(t.Context(), expired, 404, "not found")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("StoreNegative failed: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
const notExpiredPath = "/later.jpg"
|
|
||||||
|
|
||||||
_, err = cache.db.ExecContext(t.Context(), `
|
|
||||||
INSERT INTO negative_cache
|
|
||||||
(source_host, source_path, status_code, expires_at)
|
|
||||||
VALUES (?, ?, 404, datetime('now', '+1 hour'))
|
|
||||||
`, testHostExample, notExpiredPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to insert a row that has not expired: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
started := time.Now()
|
|
||||||
|
|
||||||
cache.StartEviction(100 * time.Millisecond)
|
|
||||||
defer func() { _ = cache.StopEviction(t.Context()) }()
|
|
||||||
|
|
||||||
for negativeCacheRowCount(t, cache, expired.SourcePath) > 0 {
|
|
||||||
if time.Since(started) > 10*time.Second {
|
|
||||||
t.Fatal("the expired row is still in negative_cache")
|
|
||||||
}
|
|
||||||
|
|
||||||
time.Sleep(50 * time.Millisecond)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Logf("the expired row was removed %s after the loop started",
|
|
||||||
time.Since(started))
|
|
||||||
|
|
||||||
if negativeCacheRowCount(t, cache, notExpiredPath) != 1 {
|
|
||||||
t.Error("the row that has not expired was removed")
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// negativeCacheRowCount returns how many negative_cache rows hold
|
|
||||||
// sourcePath.
|
|
||||||
func negativeCacheRowCount(tb testing.TB, cache *Cache, sourcePath string) int {
|
|
||||||
tb.Helper()
|
|
||||||
|
|
||||||
var count int
|
|
||||||
|
|
||||||
err := cache.db.QueryRowContext(tb.Context(),
|
|
||||||
`SELECT COUNT(*) FROM negative_cache WHERE source_path = ?`,
|
|
||||||
sourcePath,
|
|
||||||
).Scan(&count)
|
|
||||||
if err != nil {
|
|
||||||
tb.Fatalf("failed to count negative_cache rows: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return count
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCleanExpiredDeletesInBatches stores one expired negative cache row
|
|
||||||
// more than one batch holds, and one row that has not expired. One
|
|
||||||
// statement must delete one batch and no more, so a request waits behind
|
|
||||||
// one batch at most; CleanExpired must then remove the expired row left
|
|
||||||
// over and keep the row that has not expired.
|
|
||||||
func TestCleanExpiredDeletesInBatches(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cache, err := NewCache(evictionTestDB(t), CacheConfig{StateDir: t.TempDir()})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create cache: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx := t.Context()
|
|
||||||
|
|
||||||
insertExpiredNegativeCacheRows(t, cache, cleanExpiredBatchSize+1)
|
|
||||||
|
|
||||||
const notExpiredPath = "/later.jpg"
|
|
||||||
|
|
||||||
_, err = cache.db.ExecContext(ctx, `
|
|
||||||
INSERT INTO negative_cache
|
|
||||||
(source_host, source_path, status_code, expires_at)
|
|
||||||
VALUES (?, ?, 404, datetime('now', '+1 hour'))
|
|
||||||
`, testHostExample, notExpiredPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to insert a row that has not expired: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
deleted, err := cache.cleanExpiredBatch(ctx)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("cleanExpiredBatch failed: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Logf("one statement deleted %d of %d expired rows",
|
|
||||||
deleted, cleanExpiredBatchSize+1)
|
|
||||||
|
|
||||||
if deleted != cleanExpiredBatchSize {
|
|
||||||
t.Errorf("one statement deleted %d rows, want %d",
|
|
||||||
deleted, cleanExpiredBatchSize)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = cache.CleanExpired(ctx)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("CleanExpired failed: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if count := negativeCacheTableRowCount(t, cache); count != 1 {
|
|
||||||
t.Errorf("negative_cache holds %d rows after CleanExpired, want 1", count)
|
|
||||||
}
|
|
||||||
|
|
||||||
if negativeCacheRowCount(t, cache, notExpiredPath) != 1 {
|
|
||||||
t.Error("the row that has not expired was removed")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCleanExpiredRemovesMoreThanOneBatch stores one expired negative
|
|
||||||
// cache row more than one batch holds and calls CleanExpired once: it
|
|
||||||
// must remove them all, so it fails if CleanExpired stops after its
|
|
||||||
// first batch.
|
|
||||||
func TestCleanExpiredRemovesMoreThanOneBatch(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cache, err := NewCache(evictionTestDB(t), CacheConfig{StateDir: t.TempDir()})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create cache: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
insertExpiredNegativeCacheRows(t, cache, cleanExpiredBatchSize+1)
|
|
||||||
|
|
||||||
before := negativeCacheTableRowCount(t, cache)
|
|
||||||
|
|
||||||
err = cache.CleanExpired(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("CleanExpired failed: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
after := negativeCacheTableRowCount(t, cache)
|
|
||||||
|
|
||||||
t.Logf("CleanExpired removed %d of %d expired rows, %d per batch",
|
|
||||||
before-after, before, cleanExpiredBatchSize)
|
|
||||||
|
|
||||||
if after != 0 {
|
|
||||||
t.Errorf("negative_cache holds %d rows after CleanExpired, want 0", after)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCleanExpiredStopsOnceContextEnds runs CleanExpired with a context
|
|
||||||
// that has already ended: it must return the context's error and delete
|
|
||||||
// no row.
|
|
||||||
func TestCleanExpiredStopsOnceContextEnds(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cache, err := NewCache(evictionTestDB(t), CacheConfig{StateDir: t.TempDir()})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to create cache: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
insertExpiredNegativeCacheRows(t, cache, 1)
|
|
||||||
|
|
||||||
ctx, cancel := context.WithCancel(t.Context())
|
|
||||||
cancel()
|
|
||||||
|
|
||||||
err = cache.CleanExpired(ctx)
|
|
||||||
if !errors.Is(err, context.Canceled) {
|
|
||||||
t.Errorf("CleanExpired() error = %v, want context.Canceled", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if count := negativeCacheTableRowCount(t, cache); count != 1 {
|
|
||||||
t.Errorf("negative_cache holds %d rows, want the expired row kept", count)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// insertExpiredNegativeCacheRows inserts count negative_cache rows that
|
|
||||||
// expired an hour ago, each for a different source path.
|
|
||||||
func insertExpiredNegativeCacheRows(t *testing.T, cache *Cache, count int) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for i := range count {
|
|
||||||
_, err := cache.db.ExecContext(t.Context(), `
|
|
||||||
INSERT INTO negative_cache
|
|
||||||
(source_host, source_path, status_code, expires_at)
|
|
||||||
VALUES (?, ?, 404, datetime('now', '-1 hour'))
|
|
||||||
`, testHostExample, "/expired/"+strconv.Itoa(i)+".jpg")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to insert an expired row: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// negativeCacheTableRowCount returns how many rows negative_cache holds.
|
|
||||||
func negativeCacheTableRowCount(tb testing.TB, cache *Cache) int {
|
|
||||||
tb.Helper()
|
|
||||||
|
|
||||||
var count int
|
|
||||||
|
|
||||||
err := cache.db.QueryRowContext(tb.Context(),
|
|
||||||
`SELECT COUNT(*) FROM negative_cache`,
|
|
||||||
).Scan(&count)
|
|
||||||
if err != nil {
|
|
||||||
tb.Fatalf("failed to count negative_cache rows: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return count
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestService_Get_ReturnsErrorForNegativeCachedURL(t *testing.T) {
|
func TestService_Get_ReturnsErrorForNegativeCachedURL(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -1,359 +0,0 @@
|
|||||||
package imgcache
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"crypto/sha256"
|
|
||||||
"encoding/hex"
|
|
||||||
"io"
|
|
||||||
"log/slog"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
)
|
|
||||||
|
|
||||||
// getImage calls Get for req and returns the image it answered with and
|
|
||||||
// its cache status.
|
|
||||||
func getImage(t *testing.T, svc *Service, req *ImageRequest) ([]byte, CacheStatus) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
resp, err := svc.Get(t.Context(), req)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Get() of quality %d error = %v", req.Quality, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = resp.Content.Close() }()
|
|
||||||
|
|
||||||
body, err := io.ReadAll(resp.Content)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading the image of quality %d: %v", req.Quality, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return body, resp.CacheStatus
|
|
||||||
}
|
|
||||||
|
|
||||||
// fileSize returns the size of the file at path.
|
|
||||||
func fileSize(t *testing.T, path string) int64 {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
info, err := os.Stat(path)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("stat %s: %v", path, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return info.Size()
|
|
||||||
}
|
|
||||||
|
|
||||||
// truncateFile cuts the file at path down to size bytes.
|
|
||||||
func truncateFile(t *testing.T, path string, size int64) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
t.Logf("cutting %s from %d to %d bytes",
|
|
||||||
filepath.Base(path), fileSize(t, path), size)
|
|
||||||
|
|
||||||
err := os.Truncate(path, size)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("truncating %s: %v", path, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// cachedSourcePath returns the path of the cached source file req uses.
|
|
||||||
func cachedSourcePath(t *testing.T, svc *Service, req *ImageRequest) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
contentHash, _, err := svc.cache.LookupSource(t.Context(), req)
|
|
||||||
if err != nil || contentHash == "" {
|
|
||||||
t.Fatalf("LookupSource() = %q, %v; want the cached source", contentHash, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return svc.cache.srcContent.hashToPath(contentHash)
|
|
||||||
}
|
|
||||||
|
|
||||||
// reconcileAsAtStartup runs the cache's reconciliation pass, as pixa does
|
|
||||||
// when it starts, and returns what the pass logged.
|
|
||||||
func reconcileAsAtStartup(t *testing.T, cache *Cache) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var logBuf bytes.Buffer
|
|
||||||
|
|
||||||
cache.log = slog.New(slog.NewJSONHandler(&logBuf, nil))
|
|
||||||
|
|
||||||
err := cache.reconcileAccounting(t.Context())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reconcileAccounting() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Logf("reconciliation log: %s", logBuf.String())
|
|
||||||
|
|
||||||
return logBuf.String()
|
|
||||||
}
|
|
||||||
|
|
||||||
// assertWarningNames checks that logs hold a warning naming the file at
|
|
||||||
// path.
|
|
||||||
func assertWarningNames(t *testing.T, logs, path string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for line := range strings.Lines(logs) {
|
|
||||||
if strings.Contains(line, `"level":"WARN"`) &&
|
|
||||||
strings.Contains(line, `"path":"`+path+`"`) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Errorf("reconciliation logged no warning naming %s", path)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestService_Get_FetchesAgainForEmptyCachedSource empties the cached source
|
|
||||||
// file, as a crash or a restored snapshot could, and runs the reconciliation
|
|
||||||
// pass, which must remove it with a warning naming it. It then asks for two
|
|
||||||
// new sizes of the source: the first must fetch it again and cache it whole,
|
|
||||||
// so that the second is made from the cache without a third fetch.
|
|
||||||
func TestService_Get_FetchesAgainForEmptyCachedSource(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
svc, fixtures := SetupTestService(t)
|
|
||||||
|
|
||||||
getImage(t, svc, photoVariant(fixtures, 85, FitCover))
|
|
||||||
|
|
||||||
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, FitCover))
|
|
||||||
wholeSize := fileSize(t, path)
|
|
||||||
|
|
||||||
truncateFile(t, path, 0)
|
|
||||||
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
|
|
||||||
|
|
||||||
getImage(t, svc, photoVariant(fixtures, 40, FitCover))
|
|
||||||
getImage(t, svc, photoVariant(fixtures, 30, FitCover))
|
|
||||||
|
|
||||||
fetches := readCacheStatsCounters(t, svc.cache).upstreamFetchCount
|
|
||||||
t.Logf("upstream fetches = %d", fetches)
|
|
||||||
|
|
||||||
if fetches != 2 {
|
|
||||||
t.Errorf("upstream fetches = %d, want 2: the empty cached source "+
|
|
||||||
"was kept and fetched again on every new size", fetches)
|
|
||||||
}
|
|
||||||
|
|
||||||
size := fileSize(t, path)
|
|
||||||
if size != wholeSize {
|
|
||||||
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestService_Get_FetchesAgainForPartialCachedSource cuts the cached source
|
|
||||||
// file to half its size, as a crash or a restored snapshot could, and runs
|
|
||||||
// the reconciliation pass, which must remove it with a warning naming it.
|
|
||||||
// It then asks for two new sizes of the source: the first must fetch it
|
|
||||||
// again rather than fail to decode the partial file, and the second is
|
|
||||||
// made from the cache.
|
|
||||||
func TestService_Get_FetchesAgainForPartialCachedSource(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
svc, fixtures := SetupTestService(t)
|
|
||||||
|
|
||||||
getImage(t, svc, photoVariant(fixtures, 85, FitCover))
|
|
||||||
|
|
||||||
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, FitCover))
|
|
||||||
wholeSize := fileSize(t, path)
|
|
||||||
|
|
||||||
truncateFile(t, path, wholeSize/2)
|
|
||||||
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
|
|
||||||
|
|
||||||
getImage(t, svc, photoVariant(fixtures, 40, FitCover))
|
|
||||||
getImage(t, svc, photoVariant(fixtures, 30, FitCover))
|
|
||||||
|
|
||||||
fetches := readCacheStatsCounters(t, svc.cache).upstreamFetchCount
|
|
||||||
t.Logf("upstream fetches = %d", fetches)
|
|
||||||
|
|
||||||
if fetches != 2 {
|
|
||||||
t.Errorf("upstream fetches = %d, want 2", fetches)
|
|
||||||
}
|
|
||||||
|
|
||||||
size := fileSize(t, path)
|
|
||||||
if size != wholeSize {
|
|
||||||
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// assertConvertedAgain caches a variant of the photo, cuts its file to size
|
|
||||||
// bytes and runs the reconciliation pass, which must remove it with a
|
|
||||||
// warning naming it. It then checks that Get converts the image again
|
|
||||||
// instead of serving the file, and that the next Get serves the whole
|
|
||||||
// variant from the cache.
|
|
||||||
func assertConvertedAgain(t *testing.T, size int64) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
svc, fixtures := SetupTestService(t)
|
|
||||||
req := photoVariant(fixtures, 85, FitCover)
|
|
||||||
|
|
||||||
want, _ := getImage(t, svc, req)
|
|
||||||
|
|
||||||
path := svc.cache.variants.keyToPath(CacheKey(req))
|
|
||||||
|
|
||||||
truncateFile(t, path, size)
|
|
||||||
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
|
|
||||||
|
|
||||||
got, status := getImage(t, svc, req)
|
|
||||||
t.Logf("after the cut: %d bytes, %s", len(got), status)
|
|
||||||
|
|
||||||
if status != CacheMiss || !bytes.Equal(got, want) {
|
|
||||||
t.Errorf("after the cut: Get() = %d bytes, %s; want the %d bytes "+
|
|
||||||
"converted again, %s", len(got), status, len(want), CacheMiss)
|
|
||||||
}
|
|
||||||
|
|
||||||
got, status = getImage(t, svc, req)
|
|
||||||
t.Logf("next request: %d bytes, %s", len(got), status)
|
|
||||||
|
|
||||||
if status != CacheHit || !bytes.Equal(got, want) {
|
|
||||||
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
|
|
||||||
len(got), status, len(want), CacheHit)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestService_Get_ConvertsAgainForEmptyCachedVariant empties a cached
|
|
||||||
// variant file, as a crash or a restored snapshot could, and checks that it
|
|
||||||
// is converted again rather than served empty.
|
|
||||||
func TestService_Get_ConvertsAgainForEmptyCachedVariant(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
assertConvertedAgain(t, 0)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestService_Get_ConvertsAgainForPartialCachedVariant cuts a cached
|
|
||||||
// variant file to ten bytes, as a crash or a restored snapshot could, and
|
|
||||||
// checks that it is converted again rather than served cut short.
|
|
||||||
func TestService_Get_ConvertsAgainForPartialCachedVariant(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
assertConvertedAgain(t, 10)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestReconciliationRemovesEmptyVariantFileWithoutRow checks that the
|
|
||||||
// reconciliation pass removes an empty variant file that has no
|
|
||||||
// accounting row, with a warning naming it, instead of adding a row for
|
|
||||||
// it that would have it served empty.
|
|
||||||
func TestReconciliationRemovesEmptyVariantFileWithoutRow(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
|
||||||
|
|
||||||
_, err := cache.variants.Store(testVariantKeyOne, bytes.NewReader(nil),
|
|
||||||
testContentTypeWebP)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("failed to store an empty variant file: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
logs := reconcileAsAtStartup(t, cache)
|
|
||||||
|
|
||||||
assertWarningNames(t, logs, cache.variants.keyToPath(testVariantKeyOne))
|
|
||||||
|
|
||||||
if cache.variants.Exists(testVariantKeyOne) {
|
|
||||||
t.Error("the empty variant file is still on disk")
|
|
||||||
}
|
|
||||||
|
|
||||||
rows := countRows(t, cache,
|
|
||||||
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`,
|
|
||||||
string(testVariantKeyOne))
|
|
||||||
if rows != 0 {
|
|
||||||
t.Errorf("the empty variant file got %d accounting rows, want 0", rows)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestVariantStorage_StoreReplacesMetaFile stores a variant over one
|
|
||||||
// already stored and checks that Store puts a new .meta file in place of
|
|
||||||
// the old one, as it does the image, rather than writing into the old
|
|
||||||
// file, which a crash could leave empty. The old .meta file is given a
|
|
||||||
// second name before the store, so that it can be told apart from the new
|
|
||||||
// one after it.
|
|
||||||
func TestVariantStorage_StoreReplacesMetaFile(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
storage, key := storeTestVariant(t)
|
|
||||||
metaPath := storage.keyToPath(key) + variantMetaSuffix
|
|
||||||
oldMetaPath := metaPath + ".old"
|
|
||||||
|
|
||||||
err := os.Link(metaPath, oldMetaPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("giving the .meta file a second name: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = storage.Store(key, bytes.NewReader([]byte("new variant data")),
|
|
||||||
testContentTypeJPEG)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Store() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
oldMeta, err := os.Stat(oldMetaPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("stat of the old .meta file: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
newMeta, err := os.Stat(metaPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("stat of the new .meta file: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if os.SameFile(oldMeta, newMeta) {
|
|
||||||
t.Error("Store() wrote into the old .meta file instead of putting " +
|
|
||||||
"a new one in its place")
|
|
||||||
}
|
|
||||||
|
|
||||||
reader, _, contentType, err := storage.LoadWithMeta(key)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("LoadWithMeta() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_ = reader.Close()
|
|
||||||
|
|
||||||
t.Logf("content type after the store: %s", contentType)
|
|
||||||
|
|
||||||
if contentType != testContentTypeJPEG {
|
|
||||||
t.Errorf("content type = %q, want %q", contentType, testContentTypeJPEG)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestContentStorage_StoreHashedReplacesFileOfAnotherSize checks that a
|
|
||||||
// store over a file of another size at the content's path, such as a
|
|
||||||
// partial file a crash left, replaces it with the content.
|
|
||||||
func TestContentStorage_StoreHashedReplacesFileOfAnotherSize(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
storage, err := NewContentStorage(t.TempDir())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewContentStorage() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
content := []byte("the whole source image")
|
|
||||||
sum := sha256.Sum256(content)
|
|
||||||
hash := ContentHash(hex.EncodeToString(sum[:]))
|
|
||||||
path := storage.hashToPath(hash)
|
|
||||||
|
|
||||||
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("creating the content's directory: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = os.WriteFile(path, content[:5], StorageFilePerm)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("writing a partial file: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = storage.StoreHashed(hash, content)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("StoreHashed() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
stored, err := storage.Load(hash)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Load() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = stored.Close() }()
|
|
||||||
|
|
||||||
got, err := io.ReadAll(stored)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("reading the stored content: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !bytes.Equal(got, content) {
|
|
||||||
t.Errorf("stored file = %q, want %q", got, content)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -53,7 +53,7 @@ type ServiceConfig struct {
|
|||||||
// Allowlist is the list of hosts that don't require signatures
|
// Allowlist is the list of hosts that don't require signatures
|
||||||
Allowlist []string
|
Allowlist []string
|
||||||
// MaxConcurrentProcessing is the most images processed at once; zero
|
// MaxConcurrentProcessing is the most images processed at once; zero
|
||||||
// uses the image processor's default, half the CPUs
|
// uses the image processor's default, one per CPU
|
||||||
MaxConcurrentProcessing int
|
MaxConcurrentProcessing int
|
||||||
// Logger for logging
|
// Logger for logging
|
||||||
Logger *slog.Logger
|
Logger *slog.Logger
|
||||||
@@ -555,13 +555,7 @@ func (s *Service) processAndStore(
|
|||||||
FitMode: imageprocessor.FitMode(req.FitMode),
|
FitMode: imageprocessor.FitMode(req.FitMode),
|
||||||
}
|
}
|
||||||
|
|
||||||
// ctx has the deadline of the request doing the processing, which can
|
|
||||||
// end while a request waiting for its image has time left
|
|
||||||
processResult, err := s.processor.Process(ctx, source, processReq)
|
processResult, err := s.processor.Process(ctx, source, processReq)
|
||||||
if errors.Is(err, context.DeadlineExceeded) {
|
|
||||||
return nil, fmt.Errorf("%w: %w", ErrProcessingTimedOut, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("image processing failed: %w", err)
|
return nil, fmt.Errorf("image processing failed: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -80,8 +80,8 @@ func (s *ContentStorage) Store(r io.Reader) (ContentHash, int64, error) {
|
|||||||
// from hash, without recomputing it. Callers that already know the
|
// from hash, without recomputing it. Callers that already know the
|
||||||
// hash before writing (e.g. because they must hold a hash-keyed lock
|
// hash before writing (e.g. because they must hold a hash-keyed lock
|
||||||
// across the whole store operation) use this instead of Store. Like
|
// across the whole store operation) use this instead of Store. Like
|
||||||
// Store, it is idempotent: a file of the content's size already at that
|
// Store, it is idempotent: content already on disk at that path is
|
||||||
// path is left untouched, and a file of another size is replaced.
|
// left untouched.
|
||||||
func (s *ContentStorage) StoreHashed(
|
func (s *ContentStorage) StoreHashed(
|
||||||
hash ContentHash, data []byte,
|
hash ContentHash, data []byte,
|
||||||
) (int64, error) {
|
) (int64, error) {
|
||||||
@@ -152,35 +152,28 @@ func (s *ContentStorage) Exists(hash ContentHash) bool {
|
|||||||
return err == nil
|
return err == nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// writeIfAbsent writes data with writeFileSynced to the path derived
|
// writeIfAbsent writes data to the path derived from hash, unless
|
||||||
// from hash, unless a file of the data's size is already there.
|
// content already exists there, via a temp-file-plus-rename so
|
||||||
|
// concurrent readers never observe a partial file.
|
||||||
func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error {
|
func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error {
|
||||||
// Build path: <basedir>/<ab>/<cd>/<hash>
|
// Build path: <basedir>/<ab>/<cd>/<hash>
|
||||||
path := s.hashToPath(hash)
|
path := s.hashToPath(hash)
|
||||||
|
|
||||||
// A file of another size there is damaged, and is replaced
|
// Check if already exists
|
||||||
info, statErr := os.Stat(path)
|
_, statErr := os.Stat(path)
|
||||||
if statErr == nil && info.Size() == int64(len(data)) {
|
if statErr == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
// Create directory structure
|
||||||
|
dir := filepath.Dir(path)
|
||||||
|
|
||||||
|
err := os.MkdirAll(dir, StorageDirPerm)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to create directory: %w", err)
|
return fmt.Errorf("failed to create directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
return writeFileSynced(path, data)
|
// Write to temp file first, then rename for atomicity
|
||||||
}
|
|
||||||
|
|
||||||
// writeFileSynced writes data to path through a temp file in the same
|
|
||||||
// directory: the temp file is synced to disk, renamed to path, and the
|
|
||||||
// directory synced after, so that neither a concurrent reader nor a
|
|
||||||
// crash finds a partial file under path. When the directory sync fails,
|
|
||||||
// the file is removed from path, so that a failed write leaves no file
|
|
||||||
// there.
|
|
||||||
func writeFileSynced(path string, data []byte) error {
|
|
||||||
dir := filepath.Dir(path)
|
|
||||||
|
|
||||||
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
|
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to create temp file: %w", err)
|
return fmt.Errorf("failed to create temp file: %w", err)
|
||||||
@@ -188,20 +181,18 @@ func writeFileSynced(path string, data []byte) error {
|
|||||||
|
|
||||||
tmpPath := tmpFile.Name()
|
tmpPath := tmpFile.Name()
|
||||||
|
|
||||||
|
// Each failure path below unlinks the temp file explicitly. This
|
||||||
|
// replaces a deferred cleanup that read a named result, which the
|
||||||
|
// canonical config does not permit; the set of paths that remove
|
||||||
|
// tmpPath, and the order relative to Close, is unchanged. This
|
||||||
|
// mirrors how MetadataStorage.Store and VariantStorage.Store below
|
||||||
|
// already express the same cleanup.
|
||||||
_, err = tmpFile.Write(data)
|
_, err = tmpFile.Write(data)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = tmpFile.Close()
|
_ = tmpFile.Close()
|
||||||
_ = os.Remove(tmpPath)
|
_ = os.Remove(tmpPath)
|
||||||
|
|
||||||
return fmt.Errorf("failed to write temp file: %w", err)
|
return fmt.Errorf("failed to write content: %w", err)
|
||||||
}
|
|
||||||
|
|
||||||
err = tmpFile.Sync()
|
|
||||||
if err != nil {
|
|
||||||
_ = tmpFile.Close()
|
|
||||||
_ = os.Remove(tmpPath)
|
|
||||||
|
|
||||||
return fmt.Errorf("failed to sync temp file: %w", err)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
err = tmpFile.Close()
|
err = tmpFile.Close()
|
||||||
@@ -211,6 +202,7 @@ func writeFileSynced(path string, data []byte) error {
|
|||||||
return fmt.Errorf("failed to close temp file: %w", err)
|
return fmt.Errorf("failed to close temp file: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Atomic rename
|
||||||
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
|
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = os.Remove(tmpPath)
|
_ = os.Remove(tmpPath)
|
||||||
@@ -218,31 +210,6 @@ func writeFileSynced(path string, data []byte) error {
|
|||||||
return fmt.Errorf("failed to rename temp file: %w", err)
|
return fmt.Errorf("failed to rename temp file: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
err = syncDirectory(dir)
|
|
||||||
if err != nil {
|
|
||||||
_ = os.Remove(path)
|
|
||||||
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// syncDirectory syncs the directory dir to disk, so that a file just
|
|
||||||
// renamed into it is still there after a crash.
|
|
||||||
func syncDirectory(dir string) error {
|
|
||||||
d, err := os.Open(filepath.Clean(dir))
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to open directory: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = d.Sync()
|
|
||||||
_ = d.Close()
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to sync directory: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -460,9 +427,7 @@ func NewVariantStorage(baseDir string, log *slog.Logger) (*VariantStorage, error
|
|||||||
return &VariantStorage{baseDir: baseDir, log: log}, nil
|
return &VariantStorage{baseDir: baseDir, log: log}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Store writes content and metadata to storage at the given key, each
|
// Store writes content and metadata to storage at the given key.
|
||||||
// with writeFileSynced. The .meta file is written first, so that a crash
|
|
||||||
// cannot leave the content in place without it.
|
|
||||||
func (s *VariantStorage) Store(
|
func (s *VariantStorage) Store(
|
||||||
key VariantKey, r io.Reader, contentType string,
|
key VariantKey, r io.Reader, contentType string,
|
||||||
) (int64, error) {
|
) (int64, error) {
|
||||||
@@ -475,11 +440,46 @@ func (s *VariantStorage) Store(
|
|||||||
path := s.keyToPath(key)
|
path := s.keyToPath(key)
|
||||||
metaPath := path + ".meta"
|
metaPath := path + ".meta"
|
||||||
|
|
||||||
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
// Create directory structure
|
||||||
|
dir := filepath.Dir(path)
|
||||||
|
|
||||||
|
err = os.MkdirAll(dir, StorageDirPerm)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, fmt.Errorf("failed to create directory: %w", err)
|
return 0, fmt.Errorf("failed to create directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Write content to temp file first, then rename for atomicity
|
||||||
|
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf("failed to create temp file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
tmpPath := tmpFile.Name()
|
||||||
|
|
||||||
|
_, err = tmpFile.Write(data)
|
||||||
|
if err != nil {
|
||||||
|
_ = tmpFile.Close()
|
||||||
|
_ = os.Remove(tmpPath)
|
||||||
|
|
||||||
|
return 0, fmt.Errorf("failed to write content: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = tmpFile.Close()
|
||||||
|
if err != nil {
|
||||||
|
_ = os.Remove(tmpPath)
|
||||||
|
|
||||||
|
return 0, fmt.Errorf("failed to close temp file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Atomic rename content
|
||||||
|
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
|
||||||
|
if err != nil {
|
||||||
|
_ = os.Remove(tmpPath)
|
||||||
|
|
||||||
|
return 0, fmt.Errorf("failed to rename temp file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Write metadata
|
||||||
meta := VariantMeta{
|
meta := VariantMeta{
|
||||||
ContentType: contentType,
|
ContentType: contentType,
|
||||||
Size: size,
|
Size: size,
|
||||||
@@ -491,19 +491,13 @@ func (s *VariantStorage) Store(
|
|||||||
return 0, fmt.Errorf("failed to marshal metadata: %w", err)
|
return 0, fmt.Errorf("failed to marshal metadata: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Metadata write failure is non-fatal: the content is then served as
|
// Metadata write failure is non-fatal; content is already stored.
|
||||||
// application/octet-stream.
|
err = os.WriteFile(metaPath, metaData, StorageFilePerm)
|
||||||
err = writeFileSynced(metaPath, metaData)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.log.Warn("failed to write variant metadata sidecar",
|
s.log.Warn("failed to write variant metadata sidecar",
|
||||||
"path", metaPath, "error", err)
|
"path", metaPath, "error", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
err = writeFileSynced(path, data)
|
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
|
|
||||||
return size, nil
|
return size, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package imgcache
|
|||||||
import (
|
import (
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"net/netip"
|
|
||||||
"net/url"
|
"net/url"
|
||||||
"regexp"
|
"regexp"
|
||||||
"strconv"
|
"strconv"
|
||||||
@@ -29,9 +28,7 @@ var (
|
|||||||
const MaxDimension = imageprocessor.MaxOutputDimension
|
const MaxDimension = imageprocessor.MaxOutputDimension
|
||||||
|
|
||||||
// ValidateDimension checks one requested width or height; name ("width" or
|
// ValidateDimension checks one requested width or height; name ("width" or
|
||||||
// "height") appears in the error. 0 is valid: that side is then worked out from
|
// "height") appears in the error. 0 means "original size" and is valid.
|
||||||
// the source's proportions and the other requested side, rounded down but at
|
|
||||||
// least 1, or is the source's own side when the other side is 0 too.
|
|
||||||
func ValidateDimension(name string, value int) error {
|
func ValidateDimension(name string, value int) error {
|
||||||
if value < 0 {
|
if value < 0 {
|
||||||
return fmt.Errorf("%w: %s is negative", ErrInvalidSize, name)
|
return fmt.Errorf("%w: %s is negative", ErrInvalidSize, name)
|
||||||
@@ -50,24 +47,6 @@ func ValidateDimension(name string, value int) error {
|
|||||||
var sizeFormatRegex = regexp.MustCompile(
|
var sizeFormatRegex = regexp.MustCompile(
|
||||||
`^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`)
|
`^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`)
|
||||||
|
|
||||||
// percentEscapeRegex matches one percent-escape, such as "%2e"
|
|
||||||
var percentEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}`)
|
|
||||||
|
|
||||||
// queryEscapeRegex matches a percent-escape, or one character that a browser
|
|
||||||
// percent-encodes in the query of an http or https URL: one outside the
|
|
||||||
// printable ASCII characters "!" to "~", such as a space, a control character
|
|
||||||
// or a character outside ASCII, or one of `"#'<>`. A "%" that does not begin a
|
|
||||||
// percent-escape matches too.
|
|
||||||
var queryEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}|[^!-~]|["#'<>%]`)
|
|
||||||
|
|
||||||
// hostRegex matches the forms the host of an image URL may take: a host name
|
|
||||||
// or an IPv4 address (letters, digits, "-", "_" and "."), or an IPv6 address
|
|
||||||
// in brackets (hex digits, ":" and "."), either one optionally followed by ":"
|
|
||||||
// and a port, which may be empty. Its one group holds the text in brackets,
|
|
||||||
// which parseImageComponents also checks is an IPv6 address.
|
|
||||||
var hostRegex = regexp.MustCompile(
|
|
||||||
`^(?:[A-Za-z0-9_.-]+|\[([0-9A-Fa-f:.]+)\])(?::[0-9]*)?$`)
|
|
||||||
|
|
||||||
// ParsedURL contains the parsed components of an image proxy URL.
|
// ParsedURL contains the parsed components of an image proxy URL.
|
||||||
type ParsedURL struct {
|
type ParsedURL struct {
|
||||||
// Host is the source origin host (e.g., "cdn.example.com")
|
// Host is the source origin host (e.g., "cdn.example.com")
|
||||||
@@ -82,8 +61,9 @@ type ParsedURL struct {
|
|||||||
Format ImageFormat
|
Format ImageFormat
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParseImagePath parses an image URL's path after /v1/image/, decoded once:
|
// ParseImagePath parses the path captured by chi's wildcard:
|
||||||
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL.
|
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL
|
||||||
|
// This is the primary entry point when using chi routing.
|
||||||
// Examples:
|
// Examples:
|
||||||
// - cdn.example.com/photos/cat.jpg/800x600.webp
|
// - cdn.example.com/photos/cat.jpg/800x600.webp
|
||||||
// - cdn.example.com/photos/cat.jpg/0x0.jpeg
|
// - cdn.example.com/photos/cat.jpg/0x0.jpeg
|
||||||
@@ -99,10 +79,9 @@ func ParseImagePath(path string) (*ParsedURL, error) {
|
|||||||
return parseImageComponents(path)
|
return parseImageComponents(path)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParseImageURL parses an image URL's path, decoded once as r.URL.Path holds
|
// ParseImageURL parses a full URL path like /v1/image/<host>/<path>/<size>.<format>,
|
||||||
// it: /v1/image/<host>/<path>/<size>.<format>, or /v1/image/<host>/<path>/<size>
|
// or /v1/image/<host>/<path>/<size> for JPEG XL
|
||||||
// for JPEG XL. A source query follows the source path after a "?", which the
|
// Use ParseImagePath instead when working with chi's wildcard capture.
|
||||||
// URL writes as %3F.
|
|
||||||
func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
||||||
// Remove the /v1/image/ prefix
|
// Remove the /v1/image/ prefix
|
||||||
const prefix = "/v1/image/"
|
const prefix = "/v1/image/"
|
||||||
@@ -121,6 +100,12 @@ func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
|||||||
// parseImageComponents parses <host>/<path>/<size>.<format>, or
|
// parseImageComponents parses <host>/<path>/<size>.<format>, or
|
||||||
// <host>/<path>/<size> for JPEG XL.
|
// <host>/<path>/<size> for JPEG XL.
|
||||||
func parseImageComponents(remainder string) (*ParsedURL, error) {
|
func parseImageComponents(remainder string) (*ParsedURL, error) {
|
||||||
|
// Check for path traversal before any other processing
|
||||||
|
err := checkPathTraversal(remainder)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
// Find the last path segment, which holds "size" or "size.format"
|
// Find the last path segment, which holds "size" or "size.format"
|
||||||
lastSlash := strings.LastIndex(remainder, "/")
|
lastSlash := strings.LastIndex(remainder, "/")
|
||||||
if lastSlash == -1 {
|
if lastSlash == -1 {
|
||||||
@@ -159,20 +144,10 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
|
|||||||
return nil, ErrMissingHost
|
return nil, ErrMissingHost
|
||||||
}
|
}
|
||||||
|
|
||||||
// Only the forms hostRegex describes are accepted. A "?", "#" or "@" in
|
// A URL reads what comes before "@" as a user name and password, and the
|
||||||
// the host would make Go's URL parser, which the allowlist check and the
|
// rest as the host
|
||||||
// fetcher use, read a different host from the one allowlisted and cached
|
if strings.Contains(host, "@") {
|
||||||
hostMatch := hostRegex.FindStringSubmatch(host)
|
return nil, fmt.Errorf("%w: %q holds @", ErrInvalidHost, host)
|
||||||
if hostMatch == nil {
|
|
||||||
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Go's URL parser refuses brackets around anything but an IPv6 address
|
|
||||||
if inBrackets := hostMatch[1]; inBrackets != "" {
|
|
||||||
addr, err := netip.ParseAddr(inBrackets)
|
|
||||||
if err != nil || !addr.Is6() {
|
|
||||||
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Host names do not depend on letter case, so the allowlist, the upstream
|
// Host names do not depend on letter case, so the allowlist, the upstream
|
||||||
@@ -180,30 +155,12 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
|
|||||||
// host
|
// host
|
||||||
host = strings.ToLower(host)
|
host = strings.ToLower(host)
|
||||||
|
|
||||||
// Go's URL parser, which the fetcher uses, reads a "#" as the start of a
|
// Extract query string if present in path
|
||||||
// fragment and drops the rest, so each text after it would be another
|
|
||||||
// spelling, cached under its own keys, of one upstream request
|
|
||||||
if strings.Contains(path, "#") {
|
|
||||||
return nil, fmt.Errorf(
|
|
||||||
`%w: the source path or query holds a "#"`, ErrInvalidPath)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Extract query string if present in path. It is the source URL's query
|
|
||||||
// as that URL writes it. A character a URL's query cannot hold as it is,
|
|
||||||
// such as a space written %20 in the image URL, is percent-encoded here,
|
|
||||||
// so the query is signed and cached as it is sent upstream
|
|
||||||
if qIndex := strings.Index(path, "?"); qIndex != -1 {
|
if qIndex := strings.Index(path, "?"); qIndex != -1 {
|
||||||
query = escapeQuery(path[qIndex+1:])
|
query = path[qIndex+1:]
|
||||||
path = path[:qIndex]
|
path = path[:qIndex]
|
||||||
}
|
}
|
||||||
|
|
||||||
// The source query names no file on the upstream, so only the host and
|
|
||||||
// the source path are checked for path traversal
|
|
||||||
err = checkPathTraversal(host + path)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
// Ensure path starts with /
|
// Ensure path starts with /
|
||||||
if !strings.HasPrefix(path, "/") {
|
if !strings.HasPrefix(path, "/") {
|
||||||
path = "/" + path
|
path = "/" + path
|
||||||
@@ -229,7 +186,12 @@ func checkPathTraversal(path string) error {
|
|||||||
// Decode multiple times to catch double-encoding
|
// Decode multiple times to catch double-encoding
|
||||||
decoded := path
|
decoded := path
|
||||||
for range 3 {
|
for range 3 {
|
||||||
newDecoded := decodePercentEscapes(decoded)
|
newDecoded, err := url.PathUnescape(decoded)
|
||||||
|
if err != nil {
|
||||||
|
// Malformed encoding is suspicious
|
||||||
|
return ErrPathTraversal
|
||||||
|
}
|
||||||
|
|
||||||
if newDecoded == decoded {
|
if newDecoded == decoded {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
@@ -270,32 +232,6 @@ func checkPathTraversal(path string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// decodePercentEscapes decodes every percent-escape in s, such as %2e, and
|
|
||||||
// keeps a "%" that does not start one as it is, where url.PathUnescape would
|
|
||||||
// refuse the whole string. A source path holding %25, a literal "%", has such
|
|
||||||
// a "%" after one more decoding.
|
|
||||||
func decodePercentEscapes(s string) string {
|
|
||||||
return percentEscapeRegex.ReplaceAllStringFunc(s, func(escape string) string {
|
|
||||||
decoded, _ := url.PathUnescape(escape)
|
|
||||||
|
|
||||||
return decoded
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// escapeQuery returns a source query as a URL's query holds it: each
|
|
||||||
// character that queryEscapeRegex names, such as a space, percent-encoded,
|
|
||||||
// and the rest, the query's percent-escapes included, kept as they are. The
|
|
||||||
// result is unchanged by a second escapeQuery.
|
|
||||||
func escapeQuery(query string) string {
|
|
||||||
return queryEscapeRegex.ReplaceAllStringFunc(query, func(match string) string {
|
|
||||||
if percentEscapeRegex.MatchString(match) {
|
|
||||||
return match
|
|
||||||
}
|
|
||||||
|
|
||||||
return url.PathEscape(match)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
|
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
|
||||||
func parseSizeFormat(s string) (Size, ImageFormat, error) {
|
func parseSizeFormat(s string) (Size, ImageFormat, error) {
|
||||||
matches := sizeFormatRegex.FindStringSubmatch(s)
|
matches := sizeFormatRegex.FindStringSubmatch(s)
|
||||||
|
|||||||
@@ -407,71 +407,6 @@ func TestParseImagePath_PathTraversal(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestParseImageURL_SourceURLHoldingPercent25 verifies that a source path and
|
|
||||||
// a source query holding %25, the percent-escape of a literal "%", are
|
|
||||||
// accepted as they are, although decoding them once more leaves a "%" that
|
|
||||||
// does not start a percent-escape.
|
|
||||||
func TestParseImageURL_SourceURLHoldingPercent25(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
input string
|
|
||||||
want *ParsedURL
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
name: "path",
|
|
||||||
input: "/v1/image/cdn.example.com/sale/50%25off.jpg/800x600.webp",
|
|
||||||
want: &ParsedURL{
|
|
||||||
Host: testHostCDN, Path: "/sale/50%25off.jpg",
|
|
||||||
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "query",
|
|
||||||
input: "/v1/image/cdn.example.com/photos/cat.jpg?discount=50%25/800x600.webp",
|
|
||||||
want: &ParsedURL{
|
|
||||||
Host: testHostCDN, Path: testPathCat, Query: "discount=50%25",
|
|
||||||
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
got, err := ParseImageURL(tt.input)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("ParseImageURL() unexpected error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
assertParsedURL(t, got, tt.want)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestParseImageURL_DoubleEncodedTraversalNextToPercent verifies that a
|
|
||||||
// double-encoded ".." is still refused when the path also holds a "%" that
|
|
||||||
// does not start a percent-escape, either as written or after one more
|
|
||||||
// decoding.
|
|
||||||
func TestParseImageURL_DoubleEncodedTraversalNextToPercent(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, input := range []string{
|
|
||||||
"/v1/image/cdn.example.com/photos/%252e%252e/secret/50%25off.jpg/800x600.jpeg",
|
|
||||||
"/v1/image/cdn.example.com/photos/%252e%252e/secret/100%/800x600.jpeg",
|
|
||||||
} {
|
|
||||||
_, err := ParseImageURL(input)
|
|
||||||
t.Logf("ParseImageURL(%q) error = %v", input, err)
|
|
||||||
|
|
||||||
if !errors.Is(err, ErrPathTraversal) {
|
|
||||||
t.Errorf("ParseImageURL(%q) error = %v, want ErrPathTraversal",
|
|
||||||
input, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// errorIs checks if err matches target (handles wrapped errors).
|
// errorIs checks if err matches target (handles wrapped errors).
|
||||||
func errorIs(err, target error) bool {
|
func errorIs(err, target error) bool {
|
||||||
if errors.Is(err, target) {
|
if errors.Is(err, target) {
|
||||||
|
|||||||
@@ -8,15 +8,12 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
"regexp"
|
"regexp"
|
||||||
"strings"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
basicauth "github.com/99designs/basicauth-go"
|
basicauth "github.com/99designs/basicauth-go"
|
||||||
"github.com/go-chi/chi/v5"
|
|
||||||
"github.com/go-chi/chi/v5/middleware"
|
"github.com/go-chi/chi/v5/middleware"
|
||||||
"github.com/go-chi/cors"
|
"github.com/go-chi/cors"
|
||||||
"github.com/go-chi/httprate"
|
"github.com/go-chi/httprate"
|
||||||
"github.com/prometheus/client_golang/prometheus"
|
|
||||||
metrics "github.com/slok/go-http-metrics/metrics/prometheus"
|
metrics "github.com/slok/go-http-metrics/metrics/prometheus"
|
||||||
ghmm "github.com/slok/go-http-metrics/middleware"
|
ghmm "github.com/slok/go-http-metrics/middleware"
|
||||||
"github.com/slok/go-http-metrics/middleware/std"
|
"github.com/slok/go-http-metrics/middleware/std"
|
||||||
@@ -66,10 +63,6 @@ type Middleware struct {
|
|||||||
log *slog.Logger
|
log *slog.Logger
|
||||||
config *config.Config
|
config *config.Config
|
||||||
clientIP *clientip.Resolver
|
clientIP *clientip.Resolver
|
||||||
|
|
||||||
// metricsRegistry is the registry Metrics registers its metrics with.
|
|
||||||
// nil is Prometheus' default registry, the one /metrics serves.
|
|
||||||
metricsRegistry prometheus.Registerer
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// New creates a new Middleware instance.
|
// New creates a new Middleware instance.
|
||||||
@@ -216,91 +209,14 @@ func (s *Middleware) CORS() func(http.Handler) http.Handler {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// Metrics returns a Prometheus metrics middleware. Its handler label is the
|
// Metrics returns a Prometheus metrics middleware.
|
||||||
// pattern of the chi route a request matches, such as /v1/image/*, or
|
|
||||||
// "unmatched" for a request that matches none, so the routes fix how many
|
|
||||||
// values the label has. It is never the path, with which any client could
|
|
||||||
// add series to the registry without limit. Its method label is the
|
|
||||||
// request's method when that is a standard one, such as GET, and "other"
|
|
||||||
// for any other: Go's HTTP server accepts any method name, so a label per
|
|
||||||
// method would let a client add series the same way.
|
|
||||||
func (s *Middleware) Metrics() func(http.Handler) http.Handler {
|
func (s *Middleware) Metrics() func(http.Handler) http.Handler {
|
||||||
mdlw := ghmm.New(ghmm.Config{
|
mdlw := ghmm.New(ghmm.Config{
|
||||||
Recorder: metrics.NewRecorder(metrics.Config{
|
Recorder: metrics.NewRecorder(metrics.Config{}),
|
||||||
Registry: s.metricsRegistry,
|
|
||||||
}),
|
|
||||||
})
|
})
|
||||||
|
|
||||||
return func(next http.Handler) http.Handler {
|
return func(next http.Handler) http.Handler {
|
||||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
return std.Handler("", mdlw, next)
|
||||||
// The metrics library takes the method label from the request
|
|
||||||
// it is handed, so it is handed a shallow copy of r whose
|
|
||||||
// method is the label. next is served r itself, through the
|
|
||||||
// response writer the library wraps to see the status and size.
|
|
||||||
labelled := r.WithContext(r.Context())
|
|
||||||
labelled.Method = methodLabel(r.Method)
|
|
||||||
|
|
||||||
serveNext := http.HandlerFunc(
|
|
||||||
func(wrapped http.ResponseWriter, _ *http.Request) {
|
|
||||||
next.ServeHTTP(wrapped, r)
|
|
||||||
})
|
|
||||||
|
|
||||||
std.Handler(routePattern(r), mdlw, serveNext).ServeHTTP(w, labelled)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// routePattern returns the pattern of the chi route r matches, or
|
|
||||||
// "unmatched" when it matches none or is not served by a chi router. The
|
|
||||||
// metrics library takes the label before the request is routed, so the
|
|
||||||
// router is asked which route matches, with the path it routes on:
|
|
||||||
// r.URL.RawPath when set, else r.URL.Path, else /.
|
|
||||||
func routePattern(r *http.Request) string {
|
|
||||||
pattern := ""
|
|
||||||
|
|
||||||
rctx := chi.RouteContext(r.Context())
|
|
||||||
if rctx != nil {
|
|
||||||
path := r.URL.RawPath
|
|
||||||
if path == "" {
|
|
||||||
path = r.URL.Path
|
|
||||||
}
|
|
||||||
|
|
||||||
if path == "" {
|
|
||||||
path = "/"
|
|
||||||
}
|
|
||||||
|
|
||||||
pattern = rctx.Routes.Find(chi.NewRouteContext(), r.Method, path)
|
|
||||||
|
|
||||||
// For the path a subrouter is mounted at, such as /v1 or /v1/, Find
|
|
||||||
// answers that path without asking the subrouter, which chi hands
|
|
||||||
// the request as /.
|
|
||||||
for _, route := range rctx.Routes.Routes() {
|
|
||||||
if route.SubRoutes != nil &&
|
|
||||||
route.Pattern == strings.TrimSuffix(pattern, "/")+"/*" &&
|
|
||||||
!route.SubRoutes.Match(chi.NewRouteContext(), r.Method, "/") {
|
|
||||||
pattern = ""
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if pattern == "" {
|
|
||||||
return "unmatched"
|
|
||||||
}
|
|
||||||
|
|
||||||
return pattern
|
|
||||||
}
|
|
||||||
|
|
||||||
// methodLabel returns method when it is one of the standard methods net/http
|
|
||||||
// names: GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS, CONNECT and TRACE.
|
|
||||||
// For any other method it returns "other".
|
|
||||||
func methodLabel(method string) string {
|
|
||||||
switch method {
|
|
||||||
case http.MethodGet, http.MethodHead, http.MethodPost, http.MethodPut,
|
|
||||||
http.MethodPatch, http.MethodDelete, http.MethodOptions,
|
|
||||||
http.MethodConnect, http.MethodTrace:
|
|
||||||
return method
|
|
||||||
default:
|
|
||||||
return "other"
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -3,16 +3,12 @@ package middleware
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"maps"
|
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/url"
|
"net/url"
|
||||||
"slices"
|
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/go-chi/chi/v5"
|
|
||||||
"github.com/prometheus/client_golang/prometheus"
|
|
||||||
"github.com/prometheus/client_golang/prometheus/promhttp"
|
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||||
|
|
||||||
"sneak.berlin/go/pixa/internal/config"
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
@@ -202,7 +198,7 @@ func TestMetricsRecordsServedRequest(t *testing.T) {
|
|||||||
|
|
||||||
// The line /metrics shows once one GET /test has been served.
|
// The line /metrics shows once one GET /test has been served.
|
||||||
const want = `http_request_duration_seconds_count{` +
|
const want = `http_request_duration_seconds_count{` +
|
||||||
`code="200",handler="unmatched",method="GET",service=""} 1`
|
`code="200",handler="/test",method="GET",service=""} 1`
|
||||||
|
|
||||||
mw := &Middleware{log: slog.Default(), config: &config.Config{}}
|
mw := &Middleware{log: slog.Default(), config: &config.Config{}}
|
||||||
|
|
||||||
@@ -224,149 +220,6 @@ func TestMetricsRecordsServedRequest(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestMetricsLabelsRequestsByRoute checks that the metrics middleware labels
|
|
||||||
// each request with the pattern of the route that answers it, never with its
|
|
||||||
// path: two paths one route answers share that route's pattern, and requests
|
|
||||||
// no route answers, a 404 or a 405, share "unmatched", among them requests for
|
|
||||||
// the path the subrouter is mounted at, /v1 and /v1/. A request with an empty
|
|
||||||
// path is answered by the / route, as chi routes it as /, and is labelled /.
|
|
||||||
// With a label per path, any client could add series to the registry without
|
|
||||||
// limit. The middleware registers with a registry of its own here, as another
|
|
||||||
// test in this package registers with Prometheus' default registry, which can
|
|
||||||
// be done only once.
|
|
||||||
func TestMetricsLabelsRequestsByRoute(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
registry := prometheus.NewRegistry()
|
|
||||||
mw := &Middleware{
|
|
||||||
log: slog.Default(),
|
|
||||||
config: &config.Config{},
|
|
||||||
metricsRegistry: registry,
|
|
||||||
}
|
|
||||||
|
|
||||||
// The image route is in a subrouter, as in pixa's router.
|
|
||||||
router := chi.NewRouter()
|
|
||||||
router.Use(mw.Metrics())
|
|
||||||
router.Get("/", func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
w.WriteHeader(http.StatusOK)
|
|
||||||
})
|
|
||||||
router.Route("/v1", func(r chi.Router) {
|
|
||||||
r.Get("/image/*", func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
w.WriteHeader(http.StatusOK)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
for _, req := range []struct{ method, path string }{
|
|
||||||
{http.MethodGet, "/v1/image/a.example.com/one.jpg/1x1.png"},
|
|
||||||
{http.MethodGet, "/v1/image/b.example.com/two.jpg/1x1.png"},
|
|
||||||
{http.MethodGet, "/no-such-route"},
|
|
||||||
{http.MethodGet, "/v1/no-such-route"},
|
|
||||||
{http.MethodGet, "/v1"},
|
|
||||||
{http.MethodGet, "/v1/"},
|
|
||||||
{http.MethodPost, "/v1/image/a.example.com/one.jpg/1x1.png"},
|
|
||||||
// The request line GET http://pixa.example HTTP/1.1 leaves both
|
|
||||||
// r.URL.Path and r.URL.RawPath empty.
|
|
||||||
{http.MethodGet, "http://pixa.example"},
|
|
||||||
} {
|
|
||||||
r := httptest.NewRequestWithContext(
|
|
||||||
t.Context(), req.method, req.path, nil)
|
|
||||||
t.Logf("%s %s: r.URL.Path %q, r.URL.RawPath %q",
|
|
||||||
req.method, req.path, r.URL.Path, r.URL.RawPath)
|
|
||||||
router.ServeHTTP(httptest.NewRecorder(), r)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := requestCountsByHandler(t, registry)
|
|
||||||
t.Logf("requests counted by handler label: %v", got)
|
|
||||||
|
|
||||||
want := map[string]uint64{"/": 1, "/v1/image/*": 2, "unmatched": 5}
|
|
||||||
if !maps.Equal(got, want) {
|
|
||||||
t.Errorf("requests counted by handler label = %v, want %v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// requestCountsByHandler returns how many requests the request duration
|
|
||||||
// histogram in registry counted under each value of its handler label.
|
|
||||||
func requestCountsByHandler(
|
|
||||||
t *testing.T, registry *prometheus.Registry,
|
|
||||||
) map[string]uint64 {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
families, err := registry.Gather()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("registry.Gather() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
counts := map[string]uint64{}
|
|
||||||
|
|
||||||
for _, family := range families {
|
|
||||||
if family.GetName() != "http_request_duration_seconds" {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, metric := range family.GetMetric() {
|
|
||||||
for _, label := range metric.GetLabel() {
|
|
||||||
if label.GetName() == "handler" {
|
|
||||||
counts[label.GetValue()] += metric.GetHistogram().GetSampleCount()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return counts
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestMetricsGivesMadeUpMethodsOneLabel checks that the metrics middleware
|
|
||||||
// gives requests with methods that are not standard ones a single method
|
|
||||||
// label, "other", and still serves each request with the method it was sent
|
|
||||||
// with. With a label per method, any client could add series to the
|
|
||||||
// registry without limit by making up a new method for each request.
|
|
||||||
func TestMetricsGivesMadeUpMethodsOneLabel(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// The line /metrics shows once both made-up methods have been served.
|
|
||||||
const want = `http_request_duration_seconds_count{` +
|
|
||||||
`code="200",handler="unmatched",method="other",service=""} 2`
|
|
||||||
|
|
||||||
registry := prometheus.NewRegistry()
|
|
||||||
mw := &Middleware{
|
|
||||||
log: slog.Default(),
|
|
||||||
config: &config.Config{},
|
|
||||||
metricsRegistry: registry,
|
|
||||||
}
|
|
||||||
|
|
||||||
var served []string
|
|
||||||
|
|
||||||
handler := mw.Metrics()(http.HandlerFunc(
|
|
||||||
func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
served = append(served, r.Method)
|
|
||||||
|
|
||||||
w.WriteHeader(http.StatusOK)
|
|
||||||
}))
|
|
||||||
|
|
||||||
methods := []string{"FROBNICATE", "WIBBLE"}
|
|
||||||
for _, method := range methods {
|
|
||||||
handler.ServeHTTP(httptest.NewRecorder(), httptest.NewRequestWithContext(
|
|
||||||
t.Context(), method, "/", nil))
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Logf("methods the handler was served: %v", served)
|
|
||||||
|
|
||||||
if !slices.Equal(served, methods) {
|
|
||||||
t.Errorf("methods the handler was served = %v, want %v",
|
|
||||||
served, methods)
|
|
||||||
}
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
promhttp.HandlerFor(registry, promhttp.HandlerOpts{}).ServeHTTP(rec,
|
|
||||||
httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, "/metrics", nil))
|
|
||||||
|
|
||||||
if !strings.Contains(rec.Body.String(), want) {
|
|
||||||
t.Errorf("/metrics does not count both made-up methods as one; "+
|
|
||||||
"want the line %q in:\n%s", want, rec.Body.String())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLoggingLeavesOutSubmittedSigningKey checks that a login, a POST /
|
// TestLoggingLeavesOutSubmittedSigningKey checks that a login, a POST /
|
||||||
// whose form carries the signing key, leaves no trace of the key in the
|
// whose form carries the signing key, leaves no trace of the key in the
|
||||||
// request's log line.
|
// request's log line.
|
||||||
|
|||||||
@@ -1,269 +0,0 @@
|
|||||||
package server
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"net/url"
|
|
||||||
"path/filepath"
|
|
||||||
"regexp"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"go.uber.org/fx"
|
|
||||||
"go.uber.org/fx/fxtest"
|
|
||||||
|
|
||||||
"sneak.berlin/go/pixa/internal/config"
|
|
||||||
"sneak.berlin/go/pixa/internal/database"
|
|
||||||
"sneak.berlin/go/pixa/internal/globals"
|
|
||||||
"sneak.berlin/go/pixa/internal/handlers"
|
|
||||||
"sneak.berlin/go/pixa/internal/healthcheck"
|
|
||||||
"sneak.berlin/go/pixa/internal/logger"
|
|
||||||
"sneak.berlin/go/pixa/internal/middleware"
|
|
||||||
"sneak.berlin/go/pixa/internal/session"
|
|
||||||
)
|
|
||||||
|
|
||||||
// logoutFormPattern extracts the CSRF token rendered into the generator
|
|
||||||
// page's logout form.
|
|
||||||
var logoutFormPattern = regexp.MustCompile(
|
|
||||||
`(?s)action="/logout".*?name="gorilla\.csrf\.Token" value="([^"]+)"`)
|
|
||||||
|
|
||||||
// startPixa starts the components pixad's fx app builds, from cfg, and
|
|
||||||
// returns the server with its routes, as newTestServer does.
|
|
||||||
func startPixa(t *testing.T, cfg *config.Config) *Server {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var (
|
|
||||||
h *handlers.Handlers
|
|
||||||
mw *middleware.Middleware
|
|
||||||
)
|
|
||||||
|
|
||||||
app := fxtest.New(t,
|
|
||||||
fx.Supply(cfg),
|
|
||||||
fx.Provide(
|
|
||||||
globals.New,
|
|
||||||
logger.New,
|
|
||||||
database.New,
|
|
||||||
healthcheck.New,
|
|
||||||
handlers.New,
|
|
||||||
middleware.New,
|
|
||||||
),
|
|
||||||
fx.Populate(&h, &mw),
|
|
||||||
)
|
|
||||||
app.RequireStart()
|
|
||||||
t.Cleanup(app.RequireStop)
|
|
||||||
|
|
||||||
s := &Server{config: cfg, mw: mw, h: h}
|
|
||||||
s.SetupRoutes()
|
|
||||||
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
// logIn logs in to s with the signing key, as postLogin does, and returns
|
|
||||||
// the session cookie it sets.
|
|
||||||
func logIn(t *testing.T, s *Server) *http.Cookie {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
rec := postLogin(t, s, firstClient, "", testSigningKey)
|
|
||||||
|
|
||||||
for _, c := range rec.Result().Cookies() {
|
|
||||||
if c.Name == session.CookieName {
|
|
||||||
return c
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Fatal("logging in set no session cookie")
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// opensGeneratorPage reports whether GET / with cookie shows s's generator
|
|
||||||
// page, which only a valid login session opens.
|
|
||||||
func opensGeneratorPage(t *testing.T, s *Server, cookie *http.Cookie) bool {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
|
|
||||||
req.AddCookie(cookie)
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
return rec.Code == http.StatusOK &&
|
|
||||||
strings.Contains(rec.Body.String(), `action="/generate"`)
|
|
||||||
}
|
|
||||||
|
|
||||||
// postLogout loads the generator page with GET / and sessionCookie and
|
|
||||||
// submits its logout form with POST /logout, as a browser does.
|
|
||||||
func postLogout(
|
|
||||||
t *testing.T, s *Server, sessionCookie *http.Cookie,
|
|
||||||
) *httptest.ResponseRecorder {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
|
|
||||||
req.AddCookie(sessionCookie)
|
|
||||||
|
|
||||||
page := httptest.NewRecorder()
|
|
||||||
s.ServeHTTP(page, req)
|
|
||||||
|
|
||||||
match := logoutFormPattern.FindStringSubmatch(page.Body.String())
|
|
||||||
if match == nil {
|
|
||||||
t.Fatal("the generator page has no logout form with a CSRF token")
|
|
||||||
}
|
|
||||||
|
|
||||||
form := url.Values{"gorilla.csrf.Token": {match[1]}}
|
|
||||||
req = httptest.NewRequestWithContext(t.Context(), http.MethodPost,
|
|
||||||
"/logout", strings.NewReader(form.Encode()))
|
|
||||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
|
||||||
req.AddCookie(sessionCookie)
|
|
||||||
|
|
||||||
for _, c := range page.Result().Cookies() {
|
|
||||||
req.AddCookie(c)
|
|
||||||
}
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
return rec
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLogoutWithoutCSRFTokenEndsNoSession verifies that a request to /logout
|
|
||||||
// carrying the session cookie but not the logout form's CSRF token ends no
|
|
||||||
// session: GET /logout, as an image on another page of the same site makes a
|
|
||||||
// browser send, answers 405, and POST /logout without the token answers 403.
|
|
||||||
func TestLogoutWithoutCSRFTokenEndsNoSession(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
method string
|
|
||||||
want int
|
|
||||||
}{
|
|
||||||
{http.MethodGet, http.StatusMethodNotAllowed},
|
|
||||||
{http.MethodPost, http.StatusForbidden},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range tests {
|
|
||||||
t.Run(tc.method, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := newTestServer(t)
|
|
||||||
cookie := logIn(t, s)
|
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(
|
|
||||||
t.Context(), tc.method, "/logout", nil)
|
|
||||||
req.AddCookie(cookie)
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.ServeHTTP(rec, req)
|
|
||||||
t.Logf("%s /logout without the CSRF token: status %d",
|
|
||||||
tc.method, rec.Code)
|
|
||||||
|
|
||||||
if rec.Code != tc.want {
|
|
||||||
t.Errorf("%s /logout without the CSRF token status = %d, "+
|
|
||||||
"want %d", tc.method, rec.Code, tc.want)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !opensGeneratorPage(t, s, cookie) {
|
|
||||||
t.Errorf("%s /logout without the CSRF token ended the "+
|
|
||||||
"login session", tc.method)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLoginAfterLogoutOpensGeneratorPage verifies that after a logout from
|
|
||||||
// the generator page a copy of the session cookie taken before it is
|
|
||||||
// refused, and that a login after it opens the generator page.
|
|
||||||
func TestLoginAfterLogoutOpensGeneratorPage(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := newTestServer(t)
|
|
||||||
|
|
||||||
copied := logIn(t, s)
|
|
||||||
if !opensGeneratorPage(t, s, copied) {
|
|
||||||
t.Fatal("logging in did not open the generator page")
|
|
||||||
}
|
|
||||||
|
|
||||||
logout := postLogout(t, s, copied)
|
|
||||||
if logout.Code != http.StatusSeeOther {
|
|
||||||
t.Fatalf("POST /logout status = %d, want %d",
|
|
||||||
logout.Code, http.StatusSeeOther)
|
|
||||||
}
|
|
||||||
|
|
||||||
if opensGeneratorPage(t, s, copied) {
|
|
||||||
t.Error("a cookie copied before the logout still opens the " +
|
|
||||||
"generator page")
|
|
||||||
}
|
|
||||||
|
|
||||||
opens := opensGeneratorPage(t, s, logIn(t, s))
|
|
||||||
t.Logf("a login after the logout opens the generator page: %v", opens)
|
|
||||||
|
|
||||||
if !opens {
|
|
||||||
t.Error("a login after the logout does not open the generator page")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLogoutOutlastsRestart verifies that a copy of the session cookie taken
|
|
||||||
// before a logout is still refused by a pixa started again on the same state
|
|
||||||
// directory, as pixad is after a restart.
|
|
||||||
func TestLogoutOutlastsRestart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := newTestServer(t)
|
|
||||||
|
|
||||||
copied := logIn(t, s)
|
|
||||||
if !opensGeneratorPage(t, s, copied) {
|
|
||||||
t.Fatal("logging in did not open the generator page")
|
|
||||||
}
|
|
||||||
|
|
||||||
logout := postLogout(t, s, copied)
|
|
||||||
if logout.Code != http.StatusSeeOther {
|
|
||||||
t.Fatalf("POST /logout status = %d, want %d",
|
|
||||||
logout.Code, http.StatusSeeOther)
|
|
||||||
}
|
|
||||||
|
|
||||||
restarted := startPixa(t, s.config)
|
|
||||||
|
|
||||||
opens := opensGeneratorPage(t, restarted, copied)
|
|
||||||
t.Logf("after the restart, the copied cookie opens the generator page: %v",
|
|
||||||
opens)
|
|
||||||
|
|
||||||
if opens {
|
|
||||||
t.Error("after a restart, a cookie copied before the logout " +
|
|
||||||
"still opens the generator page")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLostStateDirectoryEndsEveryLoginSession verifies that a pixa started on
|
|
||||||
// an empty state directory, as pixad is after a restart that lost its state
|
|
||||||
// directory, refuses a session cookie from before, and that a login on it
|
|
||||||
// opens the generator page.
|
|
||||||
func TestLostStateDirectoryEndsEveryLoginSession(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s := newTestServer(t)
|
|
||||||
|
|
||||||
before := logIn(t, s)
|
|
||||||
if !opensGeneratorPage(t, s, before) {
|
|
||||||
t.Fatal("logging in did not open the generator page")
|
|
||||||
}
|
|
||||||
|
|
||||||
emptyStateDir := *s.config
|
|
||||||
emptyStateDir.StateDir = t.TempDir()
|
|
||||||
emptyStateDir.DBURL = "file:" +
|
|
||||||
filepath.Join(emptyStateDir.StateDir, "state.sqlite3")
|
|
||||||
|
|
||||||
restarted := startPixa(t, &emptyStateDir)
|
|
||||||
|
|
||||||
opens := opensGeneratorPage(t, restarted, before)
|
|
||||||
t.Logf("on an empty state directory, the cookie from before opens the "+
|
|
||||||
"generator page: %v", opens)
|
|
||||||
|
|
||||||
if opens {
|
|
||||||
t.Error("on an empty state directory, a cookie from before still " +
|
|
||||||
"opens the generator page")
|
|
||||||
}
|
|
||||||
|
|
||||||
if !opensGeneratorPage(t, restarted, logIn(t, restarted)) {
|
|
||||||
t.Error("a login on an empty state directory does not open the " +
|
|
||||||
"generator page")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -69,9 +69,10 @@ func (s *Server) SetupRoutes() {
|
|||||||
r.With(s.mw.RateLimit(LoginAttemptsPerMinute, time.Minute, nil)).
|
r.With(s.mw.RateLimit(LoginAttemptsPerMinute, time.Minute, nil)).
|
||||||
Post("/", s.h.HandleRoot())
|
Post("/", s.h.HandleRoot())
|
||||||
r.Post("/generate", s.h.HandleGenerateURL())
|
r.Post("/generate", s.h.HandleGenerateURL())
|
||||||
r.Post("/logout", s.h.HandleLogout())
|
|
||||||
})
|
})
|
||||||
|
|
||||||
|
s.router.Get("/logout", s.h.HandleLogout())
|
||||||
|
|
||||||
// Image routes, the only ones that send CORS headers, as pages on other
|
// Image routes, the only ones that send CORS headers, as pages on other
|
||||||
// sites read them. They are a subrouter rather than a group: a group's
|
// sites read them. They are a subrouter rather than a group: a group's
|
||||||
// middleware runs only for a request that matches one of its routes,
|
// middleware runs only for a request that matches one of its routes,
|
||||||
|
|||||||
@@ -1,588 +0,0 @@
|
|||||||
package server
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"net/url"
|
|
||||||
"regexp"
|
|
||||||
"slices"
|
|
||||||
"strconv"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"sneak.berlin/go/pixa/internal/encurl"
|
|
||||||
"sneak.berlin/go/pixa/internal/imgcache"
|
|
||||||
"sneak.berlin/go/pixa/internal/session"
|
|
||||||
"sneak.berlin/go/pixa/internal/signature"
|
|
||||||
)
|
|
||||||
|
|
||||||
// percentSignPath is a source path holding a "%", which a URL writes %25.
|
|
||||||
const percentSignPath = "/100%25.png"
|
|
||||||
|
|
||||||
// generateFormPattern extracts the CSRF token rendered into the generator
|
|
||||||
// page's form.
|
|
||||||
var generateFormPattern = regexp.MustCompile(
|
|
||||||
`(?s)action="/generate".*?name="gorilla\.csrf\.Token" value="([^"]+)"`)
|
|
||||||
|
|
||||||
// generatedURLPattern extracts the path of the encrypted URL the generator
|
|
||||||
// page shows.
|
|
||||||
var generatedURLPattern = regexp.MustCompile(`value="[^"]*(/v1/e/[^"]+)"`)
|
|
||||||
|
|
||||||
// recordingUpstream is a test upstream that answers every request with a PNG
|
|
||||||
// and records the request target, the path and query as sent, of each request
|
|
||||||
// it receives.
|
|
||||||
type recordingUpstream struct {
|
|
||||||
*httptest.Server
|
|
||||||
|
|
||||||
mu sync.Mutex
|
|
||||||
received []string
|
|
||||||
}
|
|
||||||
|
|
||||||
// startRecordingUpstream starts a recordingUpstream, which is closed when the
|
|
||||||
// test ends.
|
|
||||||
func startRecordingUpstream(t *testing.T) *recordingUpstream {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
source := encodeTestPNG(t, 64, 48)
|
|
||||||
upstream := &recordingUpstream{}
|
|
||||||
|
|
||||||
upstream.Server = httptest.NewServer(http.HandlerFunc(
|
|
||||||
func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
upstream.mu.Lock()
|
|
||||||
upstream.received = append(upstream.received, r.RequestURI)
|
|
||||||
upstream.mu.Unlock()
|
|
||||||
|
|
||||||
w.Header().Set("Content-Type", "image/png")
|
|
||||||
_, _ = w.Write(source)
|
|
||||||
}))
|
|
||||||
t.Cleanup(upstream.Close)
|
|
||||||
|
|
||||||
return upstream
|
|
||||||
}
|
|
||||||
|
|
||||||
// requestTargets returns the request targets the upstream has received, in
|
|
||||||
// the order it received them.
|
|
||||||
func (u *recordingUpstream) requestTargets() []string {
|
|
||||||
u.mu.Lock()
|
|
||||||
defer u.mu.Unlock()
|
|
||||||
|
|
||||||
return slices.Clone(u.received)
|
|
||||||
}
|
|
||||||
|
|
||||||
// pageOrigin is the origin of the login and generator pages below, opened
|
|
||||||
// over HTTPS; httptest gives every request the host example.com.
|
|
||||||
const pageOrigin = "https://example.com"
|
|
||||||
|
|
||||||
// openRoot sends GET / to s over HTTPS, with cookie when it is not nil, and
|
|
||||||
// returns the page: the login page, or the generator page for a login session.
|
|
||||||
func openRoot(
|
|
||||||
t *testing.T, s *Server, cookie *http.Cookie,
|
|
||||||
) *httptest.ResponseRecorder {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
|
|
||||||
pageOrigin+"/", nil)
|
|
||||||
if cookie != nil {
|
|
||||||
req.AddCookie(cookie)
|
|
||||||
}
|
|
||||||
|
|
||||||
page := httptest.NewRecorder()
|
|
||||||
s.ServeHTTP(page, req)
|
|
||||||
|
|
||||||
return page
|
|
||||||
}
|
|
||||||
|
|
||||||
// submitForm posts form to path on s as a browser submits it from page over
|
|
||||||
// HTTPS: with the CSRF token tokenPattern finds in page, the cookies page set,
|
|
||||||
// cookie when it is not nil, and a Referer naming the page, which pixa needs
|
|
||||||
// while debug is off, as it is in startImageProxy's config.
|
|
||||||
func submitForm(
|
|
||||||
t *testing.T, s *Server, page *httptest.ResponseRecorder,
|
|
||||||
tokenPattern *regexp.Regexp, path string, form url.Values,
|
|
||||||
cookie *http.Cookie,
|
|
||||||
) *httptest.ResponseRecorder {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
match := tokenPattern.FindStringSubmatch(page.Body.String())
|
|
||||||
if match == nil {
|
|
||||||
t.Fatalf("the page has no form for %s with a CSRF token", path)
|
|
||||||
}
|
|
||||||
|
|
||||||
form.Set("gorilla.csrf.Token", match[1])
|
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(t.Context(), http.MethodPost,
|
|
||||||
pageOrigin+path, strings.NewReader(form.Encode()))
|
|
||||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
|
||||||
req.Header.Set("Referer", pageOrigin+"/")
|
|
||||||
|
|
||||||
for _, c := range page.Result().Cookies() {
|
|
||||||
req.AddCookie(c)
|
|
||||||
}
|
|
||||||
|
|
||||||
if cookie != nil {
|
|
||||||
req.AddCookie(cookie)
|
|
||||||
}
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.ServeHTTP(rec, req)
|
|
||||||
|
|
||||||
return rec
|
|
||||||
}
|
|
||||||
|
|
||||||
// postGenerate logs in to s with the signing key, submits the generator page's
|
|
||||||
// form with sourceURL, and returns the response.
|
|
||||||
func postGenerate(
|
|
||||||
t *testing.T, s *Server, sourceURL string,
|
|
||||||
) *httptest.ResponseRecorder {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
login := submitForm(t, s, openRoot(t, s, nil), csrfFieldPattern, "/",
|
|
||||||
url.Values{"key": {testSigningKey}}, nil)
|
|
||||||
|
|
||||||
var sessionCookie *http.Cookie
|
|
||||||
|
|
||||||
for _, c := range login.Result().Cookies() {
|
|
||||||
if c.Name == session.CookieName {
|
|
||||||
sessionCookie = c
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if sessionCookie == nil {
|
|
||||||
t.Fatalf("logging in answered %d and set no session cookie", login.Code)
|
|
||||||
}
|
|
||||||
|
|
||||||
rec := submitForm(t, s, openRoot(t, s, sessionCookie), generateFormPattern,
|
|
||||||
"/generate", url.Values{"url": {sourceURL}}, sessionCookie)
|
|
||||||
t.Logf("POST /generate with url %s: %d", sourceURL, rec.Code)
|
|
||||||
|
|
||||||
return rec
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestImageURLFetchesItsSourcePathAndQuery requests image URLs whose source
|
|
||||||
// query holds a space, written percent-encoded, or a "'", and whose source
|
|
||||||
// path holds a "%", written percent-encoded, and checks the one request the
|
|
||||||
// upstream receives: the source path percent-encoded again, and the source
|
|
||||||
// query with the space written %20 and the "'" written %27, as a browser
|
|
||||||
// writes them in a query.
|
|
||||||
func TestImageURLFetchesItsSourcePathAndQuery(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
// pathAndQuery is the source path and query as the image URL writes
|
|
||||||
// them
|
|
||||||
pathAndQuery string
|
|
||||||
// wantRequest is the path and query the upstream must receive
|
|
||||||
wantRequest string
|
|
||||||
}{
|
|
||||||
{"space in the query", "/photo.png%3Fname=my%20cat",
|
|
||||||
"/photo.png?name=my%20cat"},
|
|
||||||
{"apostrophe in the query", "/photo.png%3Fname=it's",
|
|
||||||
"/photo.png?name=it%27s"},
|
|
||||||
{"percent sign in the path", percentSignPath, percentSignPath},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
upstream := startRecordingUpstream(t)
|
|
||||||
s, _, _ := startImageProxy(t, upstream.Server)
|
|
||||||
|
|
||||||
getImage(t, s, "/v1/image/"+upstreamHost+tt.pathAndQuery+"/orig.png")
|
|
||||||
|
|
||||||
got := upstream.requestTargets()
|
|
||||||
t.Logf("upstream received %q", got)
|
|
||||||
|
|
||||||
if !slices.Equal(got, []string{tt.wantRequest}) {
|
|
||||||
t.Errorf("upstream received %q, want [%q]", got, tt.wantRequest)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart requests twice each
|
|
||||||
// of the source paths /my cat.png and /my%20cat.png, which the image URL
|
|
||||||
// writes my%20cat.png and my%2520cat.png. Each is fetched once, as a request
|
|
||||||
// of its own, and its second request is served from the cache.
|
|
||||||
func TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
upstream := startRecordingUpstream(t)
|
|
||||||
s, _, _ := startImageProxy(t, upstream.Server)
|
|
||||||
|
|
||||||
imageURLs := []string{
|
|
||||||
"/v1/image/" + upstreamHost + "/my%20cat.png/orig.png",
|
|
||||||
"/v1/image/" + upstreamHost + "/my%2520cat.png/orig.png",
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, imageURL := range imageURLs {
|
|
||||||
getImage(t, s, imageURL)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, imageURL := range imageURLs {
|
|
||||||
cacheStatus := getImage(t, s, imageURL).Header().Get("X-Pixa-Cache")
|
|
||||||
if cacheStatus != string(imgcache.CacheHit) {
|
|
||||||
t.Errorf("second GET %s: X-Pixa-Cache = %q, want HIT",
|
|
||||||
imageURL, cacheStatus)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
got := upstream.requestTargets()
|
|
||||||
t.Logf("upstream received %q", got)
|
|
||||||
|
|
||||||
want := []string{"/my%20cat.png", "/my%2520cat.png"}
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("upstream received %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest requests the source query
|
|
||||||
// name=my%20cat written as the README says, name=my%2520cat, and with the
|
|
||||||
// space written %20, name=my%20cat. Both are one source query: the upstream
|
|
||||||
// receives one request, and the second image URL is served from the cache.
|
|
||||||
func TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
upstream := startRecordingUpstream(t)
|
|
||||||
s, _, _ := startImageProxy(t, upstream.Server)
|
|
||||||
|
|
||||||
getImage(t, s, "/v1/image/"+upstreamHost+"/photo.png%3Fname=my%2520cat/orig.png")
|
|
||||||
|
|
||||||
second := getImage(t, s,
|
|
||||||
"/v1/image/"+upstreamHost+"/photo.png%3Fname=my%20cat/orig.png")
|
|
||||||
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
|
|
||||||
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := upstream.requestTargets()
|
|
||||||
t.Logf("upstream received %q", got)
|
|
||||||
|
|
||||||
want := []string{"/photo.png?name=my%20cat"}
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("upstream received %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestGeneratedURLFetchesTheSourceURLAsEntered makes an encrypted URL on the
|
|
||||||
// generator page for source URLs holding %20, %25 and %23 in the path or the
|
|
||||||
// query, requests it, and checks that the upstream receives the path and
|
|
||||||
// query exactly as entered.
|
|
||||||
func TestGeneratedURLFetchesTheSourceURLAsEntered(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// The names hold no "%": each test's state directory is named after it
|
|
||||||
// and goes into the database URL, which decodes percent-escapes
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
pathAndQuery string
|
|
||||||
}{
|
|
||||||
{"space in the path", "/my%20cat.png"},
|
|
||||||
{"percent sign in the path", percentSignPath},
|
|
||||||
{"number sign in the path", "/a%23b.png"},
|
|
||||||
{"all three in the query", "/photo.png?name=my%20cat%25%23"},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
upstream := startRecordingUpstream(t)
|
|
||||||
s, _, _ := startImageProxy(t, upstream.Server)
|
|
||||||
|
|
||||||
rec := postGenerate(t, s, "https://"+upstreamHost+tt.pathAndQuery)
|
|
||||||
|
|
||||||
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
|
|
||||||
if rec.Code != http.StatusOK || match == nil {
|
|
||||||
t.Fatalf("POST /generate: %d with no encrypted URL; body %s",
|
|
||||||
rec.Code, rec.Body.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
getImage(t, s, match[1])
|
|
||||||
|
|
||||||
got := upstream.requestTargets()
|
|
||||||
t.Logf("upstream received %q", got)
|
|
||||||
|
|
||||||
if !slices.Equal(got, []string{tt.pathAndQuery}) {
|
|
||||||
t.Errorf("upstream received %q, want [%q]", got, tt.pathAndQuery)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL enters two
|
|
||||||
// spellings of one source URL on the generator page, requests the encrypted
|
|
||||||
// URL made for each, and checks that the upstream receives one request and
|
|
||||||
// that the second encrypted URL is served from the cache. One pair writes the
|
|
||||||
// host in upper and in lower case, which /v1/image/ lower-cases. The host is
|
|
||||||
// an IPv6 address, whose hex digits have letter case, because the fetcher
|
|
||||||
// would look a host name up in DNS. It has a port, as the fetcher refuses an
|
|
||||||
// IPv6 address written without one. The other pair is a source URL with no
|
|
||||||
// path and with the path "/", both of which fetch "/".
|
|
||||||
func TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
// sourceURLs are the two spellings, after "https://"
|
|
||||||
sourceURLs []string
|
|
||||||
// wantRequest is the path the upstream must receive
|
|
||||||
wantRequest string
|
|
||||||
}{
|
|
||||||
{"host in upper and in lower case",
|
|
||||||
[]string{"[2001:DB8::A]:8443/a.png", "[2001:db8::a]:8443/a.png"},
|
|
||||||
"/a.png"},
|
|
||||||
{"no path and the path written as a slash",
|
|
||||||
[]string{upstreamHost, upstreamHost + "/"}, "/"},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
upstream := startRecordingUpstream(t)
|
|
||||||
s, _, _ := startImageProxy(t, upstream.Server)
|
|
||||||
|
|
||||||
var imageURLs []string
|
|
||||||
|
|
||||||
for _, sourceURL := range tt.sourceURLs {
|
|
||||||
rec := postGenerate(t, s, "https://"+sourceURL)
|
|
||||||
|
|
||||||
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
|
|
||||||
if rec.Code != http.StatusOK || match == nil {
|
|
||||||
t.Fatalf("POST /generate: %d with no encrypted URL; body %s",
|
|
||||||
rec.Code, rec.Body.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
imageURLs = append(imageURLs, match[1])
|
|
||||||
}
|
|
||||||
|
|
||||||
getImage(t, s, imageURLs[0])
|
|
||||||
|
|
||||||
second := getImage(t, s, imageURLs[1])
|
|
||||||
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
|
|
||||||
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := upstream.requestTargets()
|
|
||||||
t.Logf("upstream received %q", got)
|
|
||||||
|
|
||||||
if !slices.Equal(got, []string{tt.wantRequest}) {
|
|
||||||
t.Errorf("upstream received %q, want [%q]", got, tt.wantRequest)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestGeneratorRefusesSourceURLItCannotFetchAsWritten submits source URLs the
|
|
||||||
// generator page must make no encrypted URL for, and checks that it answers
|
|
||||||
// 400 and shows why. pixa keeps the source path decoded and fetches it
|
|
||||||
// percent-encoded again, so a path holding %2F would be fetched with a "/"
|
|
||||||
// there, whatever else the path holds. pixa sends no user name or password,
|
|
||||||
// and no "?" with no query after it. For these the page shows the URL pixa
|
|
||||||
// would fetch. A source path holding a ".." segment is refused, as
|
|
||||||
// /v1/image/ refuses it.
|
|
||||||
func TestGeneratorRefusesSourceURLItCannotFetchAsWritten(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// The test config sets allow_http, so pixa would fetch over plain HTTP
|
|
||||||
const fetchedFrom = "http://" + upstreamHost
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
// sourceURL is the source URL after "https://"
|
|
||||||
sourceURL string
|
|
||||||
// wantShown is what the page must show: the URL pixa would fetch, or
|
|
||||||
// why it makes no encrypted URL
|
|
||||||
wantShown string
|
|
||||||
}{
|
|
||||||
{"encoded slash", upstreamHost + "/a%2Fb.png",
|
|
||||||
fetchedFrom + "/a/b.png"},
|
|
||||||
{"encoded slash and a space", upstreamHost + "/a%2Fb c.png",
|
|
||||||
fetchedFrom + "/a/b%20c.png"},
|
|
||||||
{"encoded slash and a bar", upstreamHost + "/a%2Fb|c.png",
|
|
||||||
fetchedFrom + "/a/b%7Cc.png"},
|
|
||||||
{"encoded slash and a letter outside ASCII", upstreamHost + "/café%2Fx.png",
|
|
||||||
fetchedFrom + "/caf%C3%A9/x.png"},
|
|
||||||
{"user name and password", "user:pass@" + upstreamHost + "/a.png",
|
|
||||||
fetchedFrom + "/a.png"},
|
|
||||||
{"question mark with no query", upstreamHost + "/a.png?",
|
|
||||||
fetchedFrom + "/a.png"},
|
|
||||||
{"two-dot segment", upstreamHost + "/a/../b.png",
|
|
||||||
"cannot hold a .. segment"},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
s, _, _ := startImageProxy(t, startRecordingUpstream(t).Server)
|
|
||||||
|
|
||||||
rec := postGenerate(t, s, "https://"+tt.sourceURL)
|
|
||||||
body := rec.Body.String()
|
|
||||||
|
|
||||||
if rec.Code != http.StatusBadRequest {
|
|
||||||
t.Errorf("status = %d, want %d", rec.Code, http.StatusBadRequest)
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Contains(body, "/v1/e/") {
|
|
||||||
t.Error("the page shows an encrypted URL")
|
|
||||||
}
|
|
||||||
|
|
||||||
if !strings.Contains(body, tt.wantShown) {
|
|
||||||
t.Errorf("the page does not show %s; body %s", tt.wantShown, body)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ipv6SourcePath is the source path of the image URLs signedImageURL and
|
|
||||||
// encryptedImageURL make.
|
|
||||||
const ipv6SourcePath = "/ipv6.png"
|
|
||||||
|
|
||||||
// TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort requests ipv6SourcePath
|
|
||||||
// on an IPv6 source host, written with and without a port, through a
|
|
||||||
// /v1/image/ URL with a signature and through an encrypted URL. The public
|
|
||||||
// address 2001:db8::a, a documentation address (RFC 3849) the fetcher accepts
|
|
||||||
// as public, answers 200, and the upstream receives the request for
|
|
||||||
// ipv6SourcePath. The private address fd00::1 answers 403, and the upstream
|
|
||||||
// receives no request: the test's fetcher connects every address to the test
|
|
||||||
// upstream, so only the fetcher's check of the source URL can refuse it.
|
|
||||||
func TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
sourceHost string
|
|
||||||
wantStatus int
|
|
||||||
// wantRequests are the request targets the upstream must receive
|
|
||||||
wantRequests []string
|
|
||||||
}{
|
|
||||||
{"public address without a port", "[2001:db8::a]",
|
|
||||||
http.StatusOK, []string{ipv6SourcePath}},
|
|
||||||
{"public address with a port", "[2001:db8::a]:8443",
|
|
||||||
http.StatusOK, []string{ipv6SourcePath}},
|
|
||||||
{"private address without a port", "[fd00::1]",
|
|
||||||
http.StatusForbidden, nil},
|
|
||||||
{"private address with a port", "[fd00::1]:8443",
|
|
||||||
http.StatusForbidden, nil},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tt := range tests {
|
|
||||||
imageURLs := map[string]string{
|
|
||||||
"image URL with a signature": signedImageURL(t, tt.sourceHost),
|
|
||||||
"encrypted URL": encryptedImageURL(t, tt.sourceHost),
|
|
||||||
}
|
|
||||||
|
|
||||||
for route, imageURL := range imageURLs {
|
|
||||||
t.Run(tt.name+" "+route, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
upstream := startRecordingUpstream(t)
|
|
||||||
s, _, _ := startImageProxy(t, upstream.Server)
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, imageURL, nil))
|
|
||||||
t.Logf("GET %s: %d", imageURL, rec.Code)
|
|
||||||
|
|
||||||
if rec.Code != tt.wantStatus {
|
|
||||||
t.Errorf("status = %d, want %d; body %s",
|
|
||||||
rec.Code, tt.wantStatus, rec.Body.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
got := upstream.requestTargets()
|
|
||||||
t.Logf("upstream received %q", got)
|
|
||||||
|
|
||||||
if !slices.Equal(got, tt.wantRequests) {
|
|
||||||
t.Errorf("upstream received %q, want %q", got, tt.wantRequests)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// signedImageURL returns the /v1/image/ URL of ipv6SourcePath on sourceHost at
|
|
||||||
// its original size as PNG, with a signature made with the test signing key,
|
|
||||||
// which a host not on the allowlist needs. The URL has no q or fit, so it is
|
|
||||||
// signed with the quality 85 and the fit cover.
|
|
||||||
func signedImageURL(t *testing.T, sourceHost string) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
path, sig, exp := signature.New(testSigningKey).GenerateSignedURL(
|
|
||||||
&signature.Request{
|
|
||||||
SourceHost: sourceHost,
|
|
||||||
SourcePath: ipv6SourcePath,
|
|
||||||
Format: string(imgcache.FormatPNG),
|
|
||||||
Quality: 85,
|
|
||||||
FitMode: string(imgcache.FitCover),
|
|
||||||
}, time.Hour)
|
|
||||||
|
|
||||||
return path + "?sig=" + sig + "&exp=" + strconv.FormatInt(exp, 10)
|
|
||||||
}
|
|
||||||
|
|
||||||
// encryptedImageURL returns the encrypted URL of ipv6SourcePath on sourceHost
|
|
||||||
// at its original size as PNG, made with the test signing key.
|
|
||||||
func encryptedImageURL(t *testing.T, sourceHost string) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
generator, err := encurl.NewGenerator(testSigningKey)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("encurl.NewGenerator: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
token, err := generator.Generate(&encurl.Payload{
|
|
||||||
SourceHost: sourceHost, SourcePath: ipv6SourcePath,
|
|
||||||
Format: imgcache.FormatPNG,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("encurl Generate: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return "/v1/e/" + token + "/img.png"
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address requests
|
|
||||||
// ipv6SourcePath on IPv6 source hosts that hold a loopback or private IPv4
|
|
||||||
// address in their last four bytes, in the IPv4-compatible form (::/96) or the
|
|
||||||
// NAT64 form (64:ff9b::/96), through a /v1/image/ URL with a signature and
|
|
||||||
// through an encrypted URL. Each answers 403, and the upstream receives no
|
|
||||||
// request: the test's fetcher connects every address to the test upstream, so
|
|
||||||
// only the fetcher's check of the source URL can refuse it.
|
|
||||||
func TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
upstream := startRecordingUpstream(t)
|
|
||||||
s, _, _ := startImageProxy(t, upstream.Server)
|
|
||||||
|
|
||||||
sourceHosts := []string{
|
|
||||||
"[::127.0.0.1]", // IPv4-compatible form of a loopback address
|
|
||||||
"[::10.0.0.1]", // IPv4-compatible form of a private address
|
|
||||||
"[64:ff9b::7f00:1]", // NAT64 form of 127.0.0.1
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, sourceHost := range sourceHosts {
|
|
||||||
imageURLs := []string{
|
|
||||||
signedImageURL(t, sourceHost),
|
|
||||||
encryptedImageURL(t, sourceHost),
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, imageURL := range imageURLs {
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
|
||||||
t.Context(), http.MethodGet, imageURL, nil))
|
|
||||||
t.Logf("GET %s (source host %s): %d", imageURL, sourceHost, rec.Code)
|
|
||||||
|
|
||||||
if rec.Code != http.StatusForbidden {
|
|
||||||
t.Errorf("source host %s: status = %d, want %d; body %s",
|
|
||||||
sourceHost, rec.Code, http.StatusForbidden, rec.Body.String())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := upstream.requestTargets(); len(got) != 0 {
|
|
||||||
t.Errorf("upstream received %q, want no request", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+2
-116
@@ -1,16 +1,9 @@
|
|||||||
// Package session provides encrypted session cookie management, and keeps in
|
// Package session provides encrypted session cookie management.
|
||||||
// the database a random value that each logout replaces, so that a logout
|
|
||||||
// ends every session made before it.
|
|
||||||
package session
|
package session
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"crypto/rand"
|
|
||||||
"database/sql"
|
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
|
||||||
"net/http"
|
"net/http"
|
||||||
"sync"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/gorilla/securecookie"
|
"github.com/gorilla/securecookie"
|
||||||
@@ -39,36 +32,14 @@ type Data struct {
|
|||||||
Authenticated bool `json:"auth"`
|
Authenticated bool `json:"auth"`
|
||||||
CreatedAt time.Time `json:"created"`
|
CreatedAt time.Time `json:"created"`
|
||||||
ExpiresAt time.Time `json:"expires"`
|
ExpiresAt time.Time `json:"expires"`
|
||||||
|
|
||||||
// LogoutValue is the Manager's random value, which each logout
|
|
||||||
// replaces, when the session was made.
|
|
||||||
LogoutValue string `json:"logoutValue"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Manager handles session creation and validation using encrypted cookies.
|
// Manager handles session creation and validation using encrypted cookies.
|
||||||
//
|
|
||||||
// Every login is made with the one signing key, so a logout ends every
|
|
||||||
// session made before it, in every browser. The Manager keeps a random
|
|
||||||
// value, which each logout replaces: a session carries the value current at
|
|
||||||
// its login, and a session carrying another value is refused. The check for
|
|
||||||
// a logout compares no time, so a login after a logout works whatever the
|
|
||||||
// host's clock does; the session's expiry is still checked against the
|
|
||||||
// clock. The Manager keeps the value in memory, so checking a session reads
|
|
||||||
// no database; a Manager from NewManagerWithDatabase also saves it in the
|
|
||||||
// database, so that it outlasts a restart.
|
|
||||||
type Manager struct {
|
type Manager struct {
|
||||||
sc *securecookie.SecureCookie
|
sc *securecookie.SecureCookie
|
||||||
db *sql.DB
|
|
||||||
|
|
||||||
// mu guards logoutValue, and is held while a logout is saved, so that
|
|
||||||
// the database and logoutValue agree.
|
|
||||||
mu sync.Mutex
|
|
||||||
logoutValue string
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewManager creates a session manager with keys derived from the signing key.
|
// NewManager creates a session manager with keys derived from the signing key.
|
||||||
// It makes a new random value and keeps it in memory only, so a restart ends
|
|
||||||
// every session; NewManagerWithDatabase keeps the value in the database.
|
|
||||||
//
|
//
|
||||||
// Session cookies always carry the Secure, HttpOnly, and SameSite=Strict
|
// Session cookies always carry the Secure, HttpOnly, and SameSite=Strict
|
||||||
// attributes; this cannot be configured. Browsers treat http://localhost as a
|
// attributes; this cannot be configured. Browsers treat http://localhost as a
|
||||||
@@ -92,44 +63,10 @@ func NewManager(signingKey string) (*Manager, error) {
|
|||||||
sc.MaxAge(int(SessionTTL.Seconds()))
|
sc.MaxAge(int(SessionTTL.Seconds()))
|
||||||
|
|
||||||
return &Manager{
|
return &Manager{
|
||||||
sc: sc,
|
sc: sc,
|
||||||
logoutValue: rand.Text(),
|
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewManagerWithDatabase creates a session manager like NewManager that keeps
|
|
||||||
// its random value in db, which must have pixa's schema: it saves the value
|
|
||||||
// NewManager made when db has none, and otherwise starts from the one saved
|
|
||||||
// there.
|
|
||||||
func NewManagerWithDatabase(
|
|
||||||
ctx context.Context, signingKey string, db *sql.DB,
|
|
||||||
) (*Manager, error) {
|
|
||||||
m, err := NewManager(signingKey)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = db.ExecContext(ctx,
|
|
||||||
"INSERT OR IGNORE INTO logout (id, logout_value) VALUES (1, ?)",
|
|
||||||
m.logoutValue)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf(
|
|
||||||
"failed to save the random value in the logout table: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = db.QueryRowContext(ctx,
|
|
||||||
"SELECT logout_value FROM logout WHERE id = 1",
|
|
||||||
).Scan(&m.logoutValue)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf(
|
|
||||||
"failed to read the random value in the logout table: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
m.db = db
|
|
||||||
|
|
||||||
return m, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// CreateSession creates a new authenticated session and sets the cookie.
|
// CreateSession creates a new authenticated session and sets the cookie.
|
||||||
func (m *Manager) CreateSession(w http.ResponseWriter) error {
|
func (m *Manager) CreateSession(w http.ResponseWriter) error {
|
||||||
now := time.Now()
|
now := time.Now()
|
||||||
@@ -137,7 +74,6 @@ func (m *Manager) CreateSession(w http.ResponseWriter) error {
|
|||||||
Authenticated: true,
|
Authenticated: true,
|
||||||
CreatedAt: now,
|
CreatedAt: now,
|
||||||
ExpiresAt: now.Add(SessionTTL),
|
ExpiresAt: now.Add(SessionTTL),
|
||||||
LogoutValue: m.currentLogoutValue(),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
encoded, err := m.sc.Encode(CookieName, data)
|
encoded, err := m.sc.Encode(CookieName, data)
|
||||||
@@ -186,30 +122,9 @@ func (m *Manager) ValidateSession(r *http.Request) (*Data, error) {
|
|||||||
return nil, ErrInvalidSession
|
return nil, ErrInvalidSession
|
||||||
}
|
}
|
||||||
|
|
||||||
// A logout since the session was made has ended it.
|
|
||||||
if data.LogoutValue != m.currentLogoutValue() {
|
|
||||||
return nil, ErrInvalidSession
|
|
||||||
}
|
|
||||||
|
|
||||||
return &data, nil
|
return &data, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Logout removes the session cookie and, when r has a valid session, ends
|
|
||||||
// every session made until now. A request without one ends no session, so
|
|
||||||
// someone who has not logged in cannot end the sessions of those who have.
|
|
||||||
func (m *Manager) Logout(w http.ResponseWriter, r *http.Request) error {
|
|
||||||
if m.IsAuthenticated(r) {
|
|
||||||
err := m.saveLogout(r.Context())
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
m.ClearSession(w)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// ClearSession removes the session cookie.
|
// ClearSession removes the session cookie.
|
||||||
func (m *Manager) ClearSession(w http.ResponseWriter) {
|
func (m *Manager) ClearSession(w http.ResponseWriter) {
|
||||||
http.SetCookie(w, &http.Cookie{
|
http.SetCookie(w, &http.Cookie{
|
||||||
@@ -230,32 +145,3 @@ func (m *Manager) IsAuthenticated(r *http.Request) bool {
|
|||||||
|
|
||||||
return err == nil && data != nil && data.Authenticated
|
return err == nil && data != nil && data.Authenticated
|
||||||
}
|
}
|
||||||
|
|
||||||
// saveLogout replaces the random value with a new one, saving it in the
|
|
||||||
// database first when the Manager has one.
|
|
||||||
func (m *Manager) saveLogout(ctx context.Context) error {
|
|
||||||
value := rand.Text()
|
|
||||||
|
|
||||||
m.mu.Lock()
|
|
||||||
defer m.mu.Unlock()
|
|
||||||
|
|
||||||
if m.db != nil {
|
|
||||||
_, err := m.db.ExecContext(ctx,
|
|
||||||
"UPDATE logout SET logout_value = ? WHERE id = 1", value)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to save the logout: %w", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
m.logoutValue = value
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// currentLogoutValue returns the Manager's random value.
|
|
||||||
func (m *Manager) currentLogoutValue() string {
|
|
||||||
m.mu.Lock()
|
|
||||||
defer m.mu.Unlock()
|
|
||||||
|
|
||||||
return m.logoutValue
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,148 +0,0 @@
|
|||||||
package session_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"database/sql"
|
|
||||||
"errors"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"sneak.berlin/go/pixa/internal/database"
|
|
||||||
"sneak.berlin/go/pixa/internal/session"
|
|
||||||
)
|
|
||||||
|
|
||||||
const logoutTestSigningKey = "test-signing-key-12345"
|
|
||||||
|
|
||||||
// newTestDatabase opens an empty in-memory database with pixa's schema. It
|
|
||||||
// has one connection, as pixad's has: each connection to ":memory:" would
|
|
||||||
// open a database of its own.
|
|
||||||
func newTestDatabase(t *testing.T) *sql.DB {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
db, err := sql.Open("sqlite", ":memory:")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("sql.Open() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
db.SetMaxOpenConns(1)
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
err = database.ApplyMigrations(t.Context(), db, nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("ApplyMigrations() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return db
|
|
||||||
}
|
|
||||||
|
|
||||||
// createSessionCookie logs in with mgr and returns the session cookie it
|
|
||||||
// sets.
|
|
||||||
func createSessionCookie(t *testing.T, mgr *session.Manager) *http.Cookie {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
rec := httptest.NewRecorder()
|
|
||||||
|
|
||||||
err := mgr.CreateSession(rec)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("CreateSession() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, c := range rec.Result().Cookies() {
|
|
||||||
if c.Name == session.CookieName {
|
|
||||||
return c
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Fatalf("CreateSession() did not set a %q cookie", session.CookieName)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// requestWithCookie returns a GET request carrying cookie, or no cookie when
|
|
||||||
// cookie is nil.
|
|
||||||
func requestWithCookie(t *testing.T, cookie *http.Cookie) *http.Request {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
|
|
||||||
if cookie != nil {
|
|
||||||
req.AddCookie(cookie)
|
|
||||||
}
|
|
||||||
|
|
||||||
return req
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLogout_EndsEverySessionMadeBeforeIt verifies that after a logout a
|
|
||||||
// copy of the session cookie taken before it is refused and a login after it
|
|
||||||
// works, and that both still hold for a new Manager on the same database, as
|
|
||||||
// after a restart.
|
|
||||||
func TestLogout_EndsEverySessionMadeBeforeIt(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := newTestDatabase(t)
|
|
||||||
|
|
||||||
mgr, err := session.NewManagerWithDatabase(
|
|
||||||
t.Context(), logoutTestSigningKey, db)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewManagerWithDatabase() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
copied := createSessionCookie(t, mgr)
|
|
||||||
|
|
||||||
err = mgr.Logout(httptest.NewRecorder(), requestWithCookie(t, copied))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Logout() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = mgr.ValidateSession(requestWithCookie(t, copied))
|
|
||||||
t.Logf("cookie copied before the logout: ValidateSession() error = %v", err)
|
|
||||||
|
|
||||||
if !errors.Is(err, session.ErrInvalidSession) {
|
|
||||||
t.Errorf("cookie copied before the logout: error = %v, want %v",
|
|
||||||
err, session.ErrInvalidSession)
|
|
||||||
}
|
|
||||||
|
|
||||||
loginAfter := createSessionCookie(t, mgr)
|
|
||||||
|
|
||||||
_, err = mgr.ValidateSession(requestWithCookie(t, loginAfter))
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("login after the logout: ValidateSession() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
restarted, err := session.NewManagerWithDatabase(
|
|
||||||
t.Context(), logoutTestSigningKey, db)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewManagerWithDatabase() after restart error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if restarted.IsAuthenticated(requestWithCookie(t, copied)) {
|
|
||||||
t.Error("after a restart, a cookie copied before the logout is accepted")
|
|
||||||
}
|
|
||||||
|
|
||||||
if !restarted.IsAuthenticated(requestWithCookie(t, loginAfter)) {
|
|
||||||
t.Error("after a restart, a login made after the logout is refused")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestLogout_WithoutSession_EndsNoSession verifies that a logout from a
|
|
||||||
// request without a session cookie ends no session, so someone who has not
|
|
||||||
// logged in cannot end the sessions of those who have.
|
|
||||||
func TestLogout_WithoutSession_EndsNoSession(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
mgr, err := session.NewManagerWithDatabase(
|
|
||||||
t.Context(), logoutTestSigningKey, newTestDatabase(t))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewManagerWithDatabase() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
cookie := createSessionCookie(t, mgr)
|
|
||||||
|
|
||||||
err = mgr.Logout(httptest.NewRecorder(), requestWithCookie(t, nil))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Logout() error = %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !mgr.IsAuthenticated(requestWithCookie(t, cookie)) {
|
|
||||||
t.Error("a logout without a session cookie ended a session")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -126,20 +126,31 @@ func (s *Signer) GenerateSignedURL(
|
|||||||
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
|
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Build the path. A source query follows the source path after a "?".
|
// Build the path.
|
||||||
// The image route reads the path decoded once, so the whole path is
|
// When a source query is present, it is embedded as a path segment
|
||||||
// written in Go's own percent-encoding of a URL path, which writes "?"
|
// (e.g. /host/path?query/size.fmt) so that the URL parser can extract
|
||||||
// as %3F and "%" as %25, and leaves "/", ";", ",", "=" and "&" as they
|
// it from the last-slash split. The "?" inside a path segment is
|
||||||
// are.
|
// percent-encoded by clients but chi delivers it decoded, which is
|
||||||
sourcePathAndQuery := req.SourcePath
|
// exactly what the URL parser expects.
|
||||||
|
var path string
|
||||||
if req.SourceQuery != "" {
|
if req.SourceQuery != "" {
|
||||||
sourcePathAndQuery += "?" + req.SourceQuery
|
path = fmt.Sprintf("/v1/image/%s%s%%3F%s/%s.%s",
|
||||||
|
req.SourceHost,
|
||||||
|
req.SourcePath,
|
||||||
|
url.PathEscape(req.SourceQuery),
|
||||||
|
sizeStr,
|
||||||
|
req.Format,
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
path = fmt.Sprintf("/v1/image/%s%s/%s.%s",
|
||||||
|
req.SourceHost,
|
||||||
|
req.SourcePath,
|
||||||
|
sizeStr,
|
||||||
|
req.Format,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
unescapedPath := fmt.Sprintf("/v1/image/%s%s/%s.%s",
|
return path, sig, exp
|
||||||
req.SourceHost, sourcePathAndQuery, sizeStr, req.Format)
|
|
||||||
|
|
||||||
return (&url.URL{Path: unescapedPath}).EscapedPath(), sig, exp
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// buildSignatureData creates the string to be signed.
|
// buildSignatureData creates the string to be signed.
|
||||||
|
|||||||
@@ -475,44 +475,6 @@ func TestGenerateSignedURL_WithoutQueryString(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestGenerateSignedURL_ReadmeQueryExample checks the README's example of a
|
|
||||||
// source URL with a query, the image at
|
|
||||||
// https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg: its
|
|
||||||
// signature with the README's key, and the path GenerateSignedURL writes for
|
|
||||||
// it, with the "?" written as %3F and the "%" as %25.
|
|
||||||
func TestGenerateSignedURL_ReadmeQueryExample(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
signer := signature.New("example-signing-key-for-documentation")
|
|
||||||
|
|
||||||
req := signature.Request{
|
|
||||||
SourceHost: testHost,
|
|
||||||
SourcePath: "/img",
|
|
||||||
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
|
|
||||||
Width: 800,
|
|
||||||
Height: 600,
|
|
||||||
Format: testFormatWebP,
|
|
||||||
Quality: 85,
|
|
||||||
FitMode: testFitCover,
|
|
||||||
Expires: time.Unix(1704067200, 0),
|
|
||||||
}
|
|
||||||
|
|
||||||
const wantSig = "37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA="
|
|
||||||
|
|
||||||
gotSig := signer.Sign(&req)
|
|
||||||
if gotSig != wantSig {
|
|
||||||
t.Errorf("Sign() = %q, want %q", gotSig, wantSig)
|
|
||||||
}
|
|
||||||
|
|
||||||
const wantPath = "/v1/image/cdn.example.com" +
|
|
||||||
"/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp"
|
|
||||||
|
|
||||||
gotPath, _, _ := signer.GenerateSignedURL(&req, time.Hour)
|
|
||||||
if gotPath != wantPath {
|
|
||||||
t.Errorf("GenerateSignedURL() path = %q, want %q", gotPath, wantPath)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestParseParams(t *testing.T) {
|
func TestParseParams(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -122,20 +122,12 @@ header {
|
|||||||
margin-bottom: 2rem;
|
margin-bottom: 2rem;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* The logout button, drawn as a link. */
|
header a {
|
||||||
|
|
||||||
header button {
|
|
||||||
width: auto;
|
|
||||||
margin: 0;
|
|
||||||
padding: 0;
|
|
||||||
background: none;
|
|
||||||
font-size: 0.875rem;
|
font-size: 0.875rem;
|
||||||
color: #4b5563;
|
color: #4b5563;
|
||||||
text-decoration: underline;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
header button:hover {
|
header a:hover {
|
||||||
background: none;
|
|
||||||
color: #1f2937;
|
color: #1f2937;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -10,12 +10,9 @@
|
|||||||
<div class="page">
|
<div class="page">
|
||||||
<header>
|
<header>
|
||||||
<h1>Pixa URL Generator</h1>
|
<h1>Pixa URL Generator</h1>
|
||||||
<form method="POST" action="/logout">
|
<a href="/logout">
|
||||||
{{ .CSRFField }}
|
Logout
|
||||||
<button type="submit">
|
</a>
|
||||||
Logout
|
|
||||||
</button>
|
|
||||||
</form>
|
|
||||||
</header>
|
</header>
|
||||||
|
|
||||||
{{if .GeneratedURL}}
|
{{if .GeneratedURL}}
|
||||||
|
|||||||
+12
-33
@@ -3,13 +3,11 @@
|
|||||||
# this repo. Idempotent: every install is guarded by a check so already
|
# this repo. Idempotent: every install is guarded by a check so already
|
||||||
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
||||||
# or apk (detected in that order); assumes NOTHING is present (not git,
|
# or apk (detected in that order); assumes NOTHING is present (not git,
|
||||||
# make, or go). The installed node is used only when its major version
|
# make, or go). Node is used directly if installed; otherwise it is
|
||||||
# is the pinned one; otherwise (node missing, or another major) the
|
# installed at a pinned version via nvm (installing nvm itself first,
|
||||||
# pinned version is installed via nvm (installing nvm itself first,
|
# from a hash-verified release archive, never curl | sh). The linter is
|
||||||
# from a hash-verified release archive, never curl | sh), and yarn and
|
# never installed on the host: golangci-lint runs only in the lint phase
|
||||||
# the packages are installed under it. The linter is never installed on
|
# of the Dockerfile (see script/lint).
|
||||||
# the host: golangci-lint runs only in the lint phase of the Dockerfile
|
|
||||||
# (see script/lint).
|
|
||||||
#
|
#
|
||||||
# script/bootstrap git, make, Go, and Node, Yarn and the
|
# script/bootstrap git, make, Go, and Node, Yarn and the
|
||||||
# prettier in yarn.lock for script/fmt and
|
# prettier in yarn.lock for script/fmt and
|
||||||
@@ -112,50 +110,31 @@ ensure_nvm() {
|
|||||||
rm -rf "$tmp"
|
rm -rf "$tmp"
|
||||||
}
|
}
|
||||||
|
|
||||||
# True when the node on PATH has the pinned major version. Only the
|
|
||||||
# major is compared, as REPO_POLICIES.md asks. Another major is not
|
|
||||||
# used: the pinned yarn 1 prints a deprecation warning under node 24.
|
|
||||||
# script/fmt and script/fmt-check carry this function unchanged, to run
|
|
||||||
# the yarn installed here.
|
|
||||||
node_is_pinned_major() {
|
|
||||||
if ! command -v node >/dev/null 2>&1; then return 1; fi
|
|
||||||
installed="$(node --version)"
|
|
||||||
installed="${installed#v}"
|
|
||||||
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
|
|
||||||
}
|
|
||||||
|
|
||||||
ensure_node() {
|
ensure_node() {
|
||||||
if node_is_pinned_major; then
|
if ! missing node; then return 0; fi
|
||||||
echo "bootstrap: using node $(node --version)"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
ensure_nvm
|
ensure_nvm
|
||||||
nvm_sh "nvm install $NODE_VERSION"
|
nvm_sh "nvm install $NODE_VERSION"
|
||||||
nvm_sh "nvm use $NODE_VERSION >/dev/null && \
|
|
||||||
echo \"bootstrap: using node \$(node --version)\""
|
|
||||||
}
|
}
|
||||||
|
|
||||||
ensure_yarn() {
|
ensure_yarn() {
|
||||||
if ! node_is_pinned_major; then
|
|
||||||
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
|
|
||||||
corepack prepare yarn@$YARN_VERSION --activate"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
if ! missing yarn; then return 0; fi
|
if ! missing yarn; then return 0; fi
|
||||||
if ! missing corepack; then
|
if ! missing corepack; then
|
||||||
corepack enable
|
corepack enable
|
||||||
corepack prepare "yarn@$YARN_VERSION" --activate
|
corepack prepare "yarn@$YARN_VERSION" --activate
|
||||||
|
elif [ -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
|
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
|
||||||
|
corepack prepare yarn@$YARN_VERSION --activate"
|
||||||
else
|
else
|
||||||
npm install -g "yarn@$YARN_VERSION"
|
npm install -g "yarn@$YARN_VERSION"
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
install_js_deps() {
|
install_js_deps() {
|
||||||
if node_is_pinned_major; then
|
if missing yarn && [ -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
yarn install --frozen-lockfile
|
|
||||||
else
|
|
||||||
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
|
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
|
||||||
yarn install --frozen-lockfile"
|
yarn install --frozen-lockfile"
|
||||||
|
else
|
||||||
|
yarn install --frozen-lockfile
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+10
-13
@@ -2,11 +2,9 @@
|
|||||||
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
||||||
# checks out and runs this and nothing else, and script/fmt-check runs
|
# checks out and runs this and nothing else, and script/fmt-check runs
|
||||||
# the formatter on the host, which a pristine checkout cannot do.
|
# the formatter on the host, which a pristine checkout cannot do.
|
||||||
# A new CACHEBUST value reruns every RUN step, since a cached gate phase
|
# --no-cache for the same reason as script/docker: the gate phases the
|
||||||
# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
|
# final stage depends on are RUN steps, and a cached one is a check that
|
||||||
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
|
# did not run.
|
||||||
# first; a stage without that line is served from cache on an unchanged
|
|
||||||
# tree, with no warning.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -16,17 +14,16 @@ main() {
|
|||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
"$SCRIPT_DIR/bootstrap"
|
"$SCRIPT_DIR/bootstrap"
|
||||||
"$SCRIPT_DIR/check"
|
"$SCRIPT_DIR/check"
|
||||||
# The version and the tag each get their own line: a failing
|
# Own line: a failing command substitution inside an argument does
|
||||||
# command substitution inside an argument does not trip `set -e`,
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
# so the inline form degrades silently to an empty constant. The
|
# empty constant. VERSION is computed here because .dockerignore
|
||||||
# VERSION build argument takes precedence over the version a build
|
# excludes .git, so `git describe` in a build stage yields an empty
|
||||||
# stage derives from the .git in the context.
|
# version without failing.
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
tag="$("$SCRIPT_DIR/projectname")"
|
docker build --no-cache \
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$tag" .
|
-t "$("$SCRIPT_DIR/projectname")" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+9
-13
@@ -1,11 +1,8 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/docker: build the Docker image tagged with the project name.
|
# script/docker: build the Docker image tagged with the project name.
|
||||||
# Identical in all repos; the tag comes from script/projectname.
|
# Identical in all repos; the tag comes from script/projectname.
|
||||||
# A new CACHEBUST value reruns every RUN step, since a cached gate phase
|
# --no-cache because the gate phases the final stage depends on are RUN
|
||||||
# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
|
# steps, and a cached one is a check that did not run.
|
||||||
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
|
|
||||||
# first; a stage without that line is served from cache on an unchanged
|
|
||||||
# tree, with no warning.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -13,17 +10,16 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
# The version and the tag each get their own line: a failing
|
# Own line: a failing command substitution inside an argument does
|
||||||
# command substitution inside an argument does not trip `set -e`,
|
# not trip `set -e`, so the inline form degrades silently to an
|
||||||
# so the inline form degrades silently to an empty constant. The
|
# empty constant. VERSION is computed here because .dockerignore
|
||||||
# VERSION build argument takes precedence over the version a build
|
# excludes .git, so `git describe` in a build stage yields an empty
|
||||||
# stage derives from the .git in the context.
|
# version without failing.
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
tag="$("$SCRIPT_DIR/projectname")"
|
docker build --no-cache \
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$tag" .
|
-t "$("$SCRIPT_DIR/projectname")" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+6
-16
@@ -7,25 +7,15 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|||||||
# Must match the pin in script/bootstrap.
|
# Must match the pin in script/bootstrap.
|
||||||
NODE_VERSION="22.17.0"
|
NODE_VERSION="22.17.0"
|
||||||
|
|
||||||
# True when the node on PATH has the pinned major version: the same
|
# script/bootstrap installs node and yarn under nvm and leaves neither
|
||||||
# function as in script/bootstrap, which uses it to decide where it
|
# on the PATH of the shell that called it, so resolve the pinned
|
||||||
# installs yarn.
|
# toolchain here the way bootstrap's own install step does. nvm is a
|
||||||
node_is_pinned_major() {
|
# bash script, hence the subshell.
|
||||||
if ! command -v node >/dev/null 2>&1; then return 1; fi
|
|
||||||
installed="$(node --version)"
|
|
||||||
installed="${installed#v}"
|
|
||||||
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
|
|
||||||
}
|
|
||||||
|
|
||||||
# Run the yarn script/bootstrap installed: the yarn on PATH when the
|
|
||||||
# node on PATH has the pinned major version, and otherwise yarn under
|
|
||||||
# the pinned node in nvm, which bootstrap leaves off the PATH of the
|
|
||||||
# shell that called it. nvm is a bash script, hence the subshell.
|
|
||||||
run_yarn() {
|
run_yarn() {
|
||||||
if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then
|
if command -v yarn >/dev/null 2>&1; then
|
||||||
exec yarn "$@"
|
exec yarn "$@"
|
||||||
fi
|
fi
|
||||||
if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
echo "fmt: no yarn; run script/bootstrap first" >&2
|
echo "fmt: no yarn; run script/bootstrap first" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|||||||
+6
-16
@@ -8,25 +8,15 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|||||||
# Must match the pin in script/bootstrap.
|
# Must match the pin in script/bootstrap.
|
||||||
NODE_VERSION="22.17.0"
|
NODE_VERSION="22.17.0"
|
||||||
|
|
||||||
# True when the node on PATH has the pinned major version: the same
|
# script/bootstrap installs node and yarn under nvm and leaves neither
|
||||||
# function as in script/bootstrap, which uses it to decide where it
|
# on the PATH of the shell that called it, so resolve the pinned
|
||||||
# installs yarn.
|
# toolchain here the way bootstrap's own install step does. nvm is a
|
||||||
node_is_pinned_major() {
|
# bash script, hence the subshell.
|
||||||
if ! command -v node >/dev/null 2>&1; then return 1; fi
|
|
||||||
installed="$(node --version)"
|
|
||||||
installed="${installed#v}"
|
|
||||||
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
|
|
||||||
}
|
|
||||||
|
|
||||||
# Run the yarn script/bootstrap installed: the yarn on PATH when the
|
|
||||||
# node on PATH has the pinned major version, and otherwise yarn under
|
|
||||||
# the pinned node in nvm, which bootstrap leaves off the PATH of the
|
|
||||||
# shell that called it. nvm is a bash script, hence the subshell.
|
|
||||||
run_yarn() {
|
run_yarn() {
|
||||||
if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then
|
if command -v yarn >/dev/null 2>&1; then
|
||||||
exec yarn "$@"
|
exec yarn "$@"
|
||||||
fi
|
fi
|
||||||
if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
echo "fmt-check: no yarn; run script/bootstrap first" >&2
|
echo "fmt-check: no yarn; run script/bootstrap first" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|||||||
+5
-8
@@ -5,12 +5,9 @@
|
|||||||
# make its answer untrustworthy.
|
# make its answer untrustworthy.
|
||||||
#
|
#
|
||||||
# The phase is not the last stage in the file, so it is built only when
|
# The phase is not the last stage in the file, so it is built only when
|
||||||
# --target names it. A new CACHEBUST value reruns every RUN step, since
|
# --target names it. --no-cache because a cached lint layer is a lint
|
||||||
# a cached lint layer is a lint that did not run; unlike --no-cache, it
|
# that did not run. The tag makes each build replace the previous image
|
||||||
# keeps Go's build cache. This relies on every stage of the Dockerfile
|
# instead of leaving a dangling one behind.
|
||||||
# declaring ARG CACHEBUST first; a stage without that line is served
|
|
||||||
# from cache on an unchanged tree, with no warning.
|
|
||||||
# --output type=cacheonly writes no image, since nothing uses one.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -18,9 +15,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
docker build --no-cache \
|
||||||
--target lint \
|
--target lint \
|
||||||
--output type=cacheonly .
|
-t "$("$SCRIPT_DIR/projectname")-lint" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+4
-8
@@ -2,12 +2,8 @@
|
|||||||
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
||||||
# and this builds that phase alone, on the same terms as script/lint:
|
# and this builds that phase alone, on the same terms as script/lint:
|
||||||
# --target because a phase that is not the last stage is built only when
|
# --target because a phase that is not the last stage is built only when
|
||||||
# named, and a new CACHEBUST value because a cached test layer is a test
|
# named, --no-cache because a cached test layer is a test that did not
|
||||||
# that did not run; unlike --no-cache, it keeps Go's build cache. This
|
# run, and a tag so each build replaces the previous image.
|
||||||
# relies on every stage of the Dockerfile declaring ARG CACHEBUST first;
|
|
||||||
# a stage without that line is served from cache on an unchanged tree,
|
|
||||||
# with no warning. --output type=cacheonly writes no image, since
|
|
||||||
# nothing uses one.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -15,9 +11,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
docker build --no-cache \
|
||||||
--target test \
|
--target test \
|
||||||
--output type=cacheonly .
|
-t "$("$SCRIPT_DIR/projectname")-test" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
Reference in New Issue
Block a user