Compare commits
16
Commits
296a26097d
..
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
faad959bea | ||
|
|
3a6b3bc202 | ||
|
|
08d6c3c459 | ||
|
|
efa431ba18 | ||
|
|
8b4dcda549 | ||
|
|
6d271d4617 | ||
|
|
4a4bc9758f | ||
|
|
9a96740463 | ||
|
|
1fd5cf8bd0 | ||
|
|
62b4f4791e | ||
|
|
22bfc432b6 | ||
|
|
f743930f71 | ||
|
|
4e30dc7467 | ||
|
|
16f9d1dfd1 | ||
|
|
742db9ceb1 | ||
|
|
2ab96155ba |
+18
-9
@@ -13,12 +13,21 @@
|
|||||||
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
|
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
|
||||||
# deletes the package directory from the context.
|
# deletes the package directory from the context.
|
||||||
|
|
||||||
# Unlike the standard file, which leaves out all of .git, pixa sends
|
# .git is sent without its config. Without a VERSION build argument the
|
||||||
# .git without its config. Without a VERSION build argument the stage
|
# stage that compiles runs `git describe --tags --always` on .git, which
|
||||||
# that compiles runs `git describe --tags --always` on .git, which does
|
# does not need .git/config; that file can hold a credential, such as a
|
||||||
# not need .git/config; that file can hold a credential, such as a
|
|
||||||
# password in a remote URL or the token the CI checkout step stores there.
|
# password in a remote URL or the token the CI checkout step stores there.
|
||||||
.git/config
|
# Each submodule keeps a config with the same exposure in its git directory
|
||||||
|
# under .git/modules/, nested again for a submodule's own submodules, or in
|
||||||
|
# its own .git directory when it keeps one.
|
||||||
|
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
|
||||||
|
# `deploy/config`, `config/lib`) loses its whole git directory, because
|
||||||
|
# `**/.git/modules/**/config` also matches that segment's directory
|
||||||
|
# under .git/modules/. Go's version stamping then fails the build;
|
||||||
|
# nothing leaks. Name such a submodule without that segment:
|
||||||
|
# `git submodule add --name`.
|
||||||
|
**/.git/config
|
||||||
|
**/.git/modules/**/config
|
||||||
|
|
||||||
# Agent scratch: one full checkout of the repo per in-flight agent.
|
# Agent scratch: one full checkout of the repo per in-flight agent.
|
||||||
# Anchored because it occurs once where agents run at the repo root.
|
# Anchored because it occurs once where agents run at the repo root.
|
||||||
@@ -42,7 +51,9 @@
|
|||||||
**/[iI][dD]_[rR][sS][aA]
|
**/[iI][dD]_[rR][sS][aA]
|
||||||
**/[iI][dD]_[dD][sS][aA]
|
**/[iI][dD]_[dD][sS][aA]
|
||||||
**/[iI][dD]_[eE][cC][dD][sS][aA]
|
**/[iI][dD]_[eE][cC][dD][sS][aA]
|
||||||
|
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
|
||||||
**/[iI][dD]_[eE][dD]25519
|
**/[iI][dD]_[eE][dD]25519
|
||||||
|
**/[iI][dD]_[eE][dD]25519_[sS][kK]
|
||||||
|
|
||||||
# Dependencies: restored inside the image, never copied in.
|
# Dependencies: restored inside the image, never copied in.
|
||||||
**/node_modules
|
**/node_modules
|
||||||
@@ -60,10 +71,8 @@
|
|||||||
**/.vscode
|
**/.vscode
|
||||||
**/*.sublime-*
|
**/*.sublime-*
|
||||||
|
|
||||||
# pixa's own entries. Nothing in the build reads .gitignore. On the
|
# pixa's own entries. On the host, `make build` writes bin/pixad, and the
|
||||||
# host, `make build` writes bin/pixad, and the example config keeps its
|
# example config keeps its state directory in data/.
|
||||||
# state directory in data/.
|
|
||||||
.gitignore
|
|
||||||
/bin
|
/bin
|
||||||
/data
|
/data
|
||||||
|
|
||||||
|
|||||||
@@ -13,3 +13,6 @@ indent_style = tab
|
|||||||
|
|
||||||
[*.go]
|
[*.go]
|
||||||
indent_style = tab
|
indent_style = tab
|
||||||
|
|
||||||
|
# This repository's own sections, such as one for another language it
|
||||||
|
# uses, go below this comment, and a re-vendor keeps them.
|
||||||
|
|||||||
@@ -1,15 +1,21 @@
|
|||||||
name: check
|
name: check
|
||||||
on: [push]
|
on: [push]
|
||||||
|
# Free the shared runner: a new push cancels only the same branch's older run.
|
||||||
|
concurrency:
|
||||||
|
group: ${{ github.workflow }}-${{ github.ref }}
|
||||||
|
cancel-in-progress: true
|
||||||
jobs:
|
jobs:
|
||||||
check:
|
check:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
|
# Free the shared runner from a hung build.
|
||||||
|
timeout-minutes: 20
|
||||||
steps:
|
steps:
|
||||||
# actions/checkout v4.2.2, 2026-02-22
|
# actions/checkout v4.2.2, 2026-02-22
|
||||||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
||||||
# The default clone is shallow and has no tags, so the
|
# script/cibuild needs no token, so none is left in .git/config.
|
||||||
# version the build takes from `git describe` would be a
|
|
||||||
# bare commit; this fetches the whole history with its tags.
|
|
||||||
with:
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
# All history and tags, so git describe finds the version tag.
|
||||||
fetch-depth: 0
|
fetch-depth: 0
|
||||||
- run: script/cibuild
|
- run: script/cibuild
|
||||||
- run: script/docker-smoke
|
- run: script/docker-smoke
|
||||||
|
|||||||
+37
-6
@@ -17,15 +17,46 @@ Thumbs.db
|
|||||||
# entry and must not be given a `**/` prefix on the way into one.
|
# entry and must not be given a `**/` prefix on the way into one.
|
||||||
.claude/
|
.claude/
|
||||||
|
|
||||||
# Environment / secrets
|
# Node
|
||||||
.env
|
node_modules/
|
||||||
.env.*
|
|
||||||
*.pem
|
# Secrets. Unanchored like every entry above, so each matches at every
|
||||||
*.key
|
# depth. Matching is case-sensitive on Linux, so names use character
|
||||||
|
# ranges rather than a lowercase form that misses `Server.Key`.
|
||||||
|
|
||||||
|
# Environment files. `*.env` covers bare `.env` and the `prod.env`
|
||||||
|
# convention. Only the templates `example.env` and `sample.env` are
|
||||||
|
# re-included below. A repository that commits any other template adds
|
||||||
|
# its own negation at the end of this file, for example `!.env.example`.
|
||||||
|
*.[eE][nN][vV]
|
||||||
|
.[eE][nN][vV].*
|
||||||
|
.[eE][nN][vV][rR][cC]
|
||||||
|
!example.env
|
||||||
|
!sample.env
|
||||||
|
|
||||||
|
# Private keys and the bundles carrying them.
|
||||||
|
*.[pP][eE][mM]
|
||||||
|
*.[kK][eE][yY]
|
||||||
|
*.[pP]12
|
||||||
|
*.[pP][fF][xX]
|
||||||
|
[iI][dD]_[rR][sS][aA]
|
||||||
|
[iI][dD]_[dD][sS][aA]
|
||||||
|
[iI][dD]_[eE][cC][dD][sS][aA]
|
||||||
|
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
|
||||||
|
[iI][dD]_[eE][dD]25519
|
||||||
|
[iI][dD]_[eE][dD]25519_[sS][kK]
|
||||||
|
|
||||||
|
# This repository's own entries, such as its build outputs, go below
|
||||||
|
# this comment, and a re-vendor keeps them. Anchor a binary built at the
|
||||||
|
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
|
||||||
|
|
||||||
|
# Go
|
||||||
|
*.log
|
||||||
|
*.out
|
||||||
|
*.test
|
||||||
|
|
||||||
# Dependencies
|
# Dependencies
|
||||||
vendor/
|
vendor/
|
||||||
node_modules/
|
|
||||||
|
|
||||||
# Build output
|
# Build output
|
||||||
/bin/
|
/bin/
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ linters:
|
|||||||
disable:
|
disable:
|
||||||
# Genuinely incompatible with project patterns
|
# Genuinely incompatible with project patterns
|
||||||
- exhaustruct # Requires all struct fields
|
- exhaustruct # Requires all struct fields
|
||||||
|
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
|
||||||
- godot # Requires comments to end with periods
|
- godot # Requires comments to end with periods
|
||||||
- wrapcheck # Too verbose for internal packages
|
- wrapcheck # Too verbose for internal packages
|
||||||
- varnamelen # Short names like db, id are idiomatic Go
|
- varnamelen # Short names like db, id are idiomatic Go
|
||||||
@@ -24,6 +25,8 @@ linters:
|
|||||||
# silenced by disabling that name, not by enabling the successor.
|
# silenced by disabling that name, not by enabling the successor.
|
||||||
- wsl # Deprecated, replaced by wsl_v5
|
- wsl # Deprecated, replaced by wsl_v5
|
||||||
- gomodguard # Deprecated, replaced by gomodguard_v2
|
- gomodguard # Deprecated, replaced by gomodguard_v2
|
||||||
|
# Misses findings at random in v2.14.0; back once a pinned release fixes it
|
||||||
|
- canonicalheader
|
||||||
settings:
|
settings:
|
||||||
lll:
|
lll:
|
||||||
line-length: 88
|
line-length: 88
|
||||||
|
|||||||
@@ -1,7 +1,2 @@
|
|||||||
node_modules/
|
node_modules/
|
||||||
yarn.lock
|
yarn.lock
|
||||||
|
|
||||||
# A byte-for-byte copy of the one in sneak/prompts.
|
|
||||||
REPO_POLICIES.md
|
|
||||||
|
|
||||||
vendor/
|
|
||||||
|
|||||||
+48
-30
@@ -1,7 +1,12 @@
|
|||||||
|
# Every stage declares ARG CACHEBUST first: script/ passes a new value
|
||||||
|
# on every build, which reruns every RUN step after it. The steps that
|
||||||
|
# compile mount Go's build cache, which survives from build to build.
|
||||||
|
|
||||||
# Lint phase. script/lint builds it alone. The linter is run directly:
|
# Lint phase. script/lint builds it alone. The linter is run directly:
|
||||||
# `make lint` and script/lint are themselves a docker build.
|
# `make lint` and script/lint are themselves a docker build.
|
||||||
# golangci/golangci-lint:v2.12.2, 2026-10-04
|
# golangci/golangci-lint:v2.14.0, 2026-10-15
|
||||||
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
|
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
|
||||||
|
ARG CACHEBUST
|
||||||
|
|
||||||
# The linter compiles every package, and govips needs the libvips
|
# The linter compiles every package, and govips needs the libvips
|
||||||
# headers for that. REPO_POLICIES.md has the lint phase install them
|
# headers for that. REPO_POLICIES.md has the lint phase install them
|
||||||
@@ -14,17 +19,20 @@ WORKDIR /src
|
|||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN golangci-lint run --config .golangci.yml ./...
|
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||||
|
golangci-lint run --config .golangci.yml ./...
|
||||||
|
|
||||||
# Test phase. script/test builds it alone.
|
# Test phase. script/test builds it alone. -race needs cgo and so a C
|
||||||
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
|
# compiler, which the Debian Go image ships and the alpine one does not.
|
||||||
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS test
|
# golang:1.25.4-trixie, 2026-10-15
|
||||||
|
FROM golang@sha256:a02d35efc036053fdf0da8c15919276bf777a80cbfda6a35c5e9f087e652adfc AS test
|
||||||
|
ARG CACHEBUST
|
||||||
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
# script/bootstrap --cgo installs the build dependencies (a C compiler,
|
# script/bootstrap --cgo installs the build dependencies (the libvips
|
||||||
# the libvips and libheif headers, and libvips' JPEG XL support, which
|
# and libheif headers, and libvips' JPEG XL support, which the tests
|
||||||
# the tests need) and downloads the Go modules.
|
# need) and downloads the Go modules.
|
||||||
COPY script/ ./script/
|
COPY script/ ./script/
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN script/bootstrap --cgo
|
RUN script/bootstrap --cgo
|
||||||
@@ -32,11 +40,14 @@ RUN script/bootstrap --cgo
|
|||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# Without -v first; on a failure, again with -v for the details, and
|
# Without -v first; on a failure, again with -v for the details, and
|
||||||
# the step fails even if the second run passes. -parallel 4: by default
|
# the step fails even if the second run passes. -count=1 because the
|
||||||
# a package runs as many of its tests at once as the host has CPUs, and
|
# mounted cache also holds test results, which go test would otherwise
|
||||||
# on a busy host they then wait so long to be scheduled that a test
|
# replay. -parallel 4: by default a package runs as many of its tests at
|
||||||
# that times a request can see it return a second late.
|
# once as the host has CPUs, and on a busy host they then wait so long
|
||||||
RUN go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
|
# to be scheduled that a test that times a request can see it return a
|
||||||
|
# second late.
|
||||||
|
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||||
|
go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; }
|
go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; }
|
||||||
|
|
||||||
@@ -45,38 +56,44 @@ RUN go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
|
|||||||
# test passed.
|
# test passed.
|
||||||
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
|
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
|
||||||
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
|
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
|
||||||
|
ARG CACHEBUST
|
||||||
|
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
COPY --from=test /src/go.sum /dev/null
|
COPY --from=test /src/go.sum /dev/null
|
||||||
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
# Build dependencies and Go modules, as in the test phase
|
# Build dependencies, git among them, and Go modules, as in the test
|
||||||
|
# phase
|
||||||
COPY script/ ./script/
|
COPY script/ ./script/
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN script/bootstrap --cgo
|
RUN script/bootstrap --cgo
|
||||||
|
# A tar-stream context keeps the sender's file owners, which git refuses.
|
||||||
|
RUN git config --system --add safe.directory /src
|
||||||
|
|
||||||
# Copy source code
|
# Copy source code
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# VERSION is declared here, not earlier: a new value reruns only the
|
# The VERSION build arg when one is given, otherwise
|
||||||
# build, not script/bootstrap. Given none, the version is
|
# `git describe --tags --always` on the .git in the build context: the
|
||||||
# `git describe --tags --always` of the .git in the build context (git
|
# tag on a tagged commit, tag-N-gHASH after one, the short commit when
|
||||||
# comes from script/bootstrap): the tag on a tagged commit, tag-N-gHASH
|
# no tag is reachable. With .git present, a version that is still empty,
|
||||||
# after one, the short commit when no tag is reachable. A context that
|
# dev or unknown fails the build: git is missing or could not read the
|
||||||
# carries .git and still yields no version fails the build; one without
|
# checkout. A context without .git, as from a source tarball, stamps an
|
||||||
# .git, as from a source tarball, stamps an empty version. CGO stays
|
# empty version. CGO stays enabled for govips; -trimpath keeps build
|
||||||
# enabled for govips; -trimpath keeps build paths out of the binary, and
|
# paths out of the binary, and -s -w leave out the symbol table and
|
||||||
# -s -w leave out the symbol table and debug information.
|
# debug information.
|
||||||
ARG VERSION
|
ARG VERSION
|
||||||
RUN version="${VERSION:-$(git describe --tags --always)}"; \
|
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||||
if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
|
VERSION="${VERSION:-$(git describe --tags --always)}"; \
|
||||||
[ "$version" = unknown ]; }; then \
|
if [ -e .git ]; then \
|
||||||
echo "the build context carries .git but yields no version" >&2; \
|
case "$VERSION" in ""|dev|unknown) \
|
||||||
exit 1; \
|
echo "version is '$VERSION' although .git is present" >&2; \
|
||||||
|
exit 1 ;; \
|
||||||
|
esac; \
|
||||||
fi; \
|
fi; \
|
||||||
CGO_ENABLED=1 GOTOOLCHAIN=auto go build -trimpath \
|
CGO_ENABLED=1 GOTOOLCHAIN=auto go build -trimpath \
|
||||||
-ldflags "-s -w -X main.Version=${version}" \
|
-ldflags="-s -w -X main.Version=${VERSION}" \
|
||||||
-o /pixad ./cmd/pixad
|
-o /pixad ./cmd/pixad
|
||||||
|
|
||||||
# Runtime stage, and the last one: a plain `docker build .` builds this
|
# Runtime stage, and the last one: a plain `docker build .` builds this
|
||||||
@@ -85,6 +102,7 @@ RUN version="${VERSION:-$(git describe --tags --always)}"; \
|
|||||||
# the libvips and musl it was built with.
|
# the libvips and musl it was built with.
|
||||||
# alpine:3.22, 2026-10-08
|
# alpine:3.22, 2026-10-08
|
||||||
FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8
|
FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8
|
||||||
|
ARG CACHEBUST
|
||||||
|
|
||||||
# Install runtime dependencies only. vips-jxl is libvips' JPEG XL
|
# Install runtime dependencies only. vips-jxl is libvips' JPEG XL
|
||||||
# support, without which pixad does not start.
|
# support, without which pixad does not start.
|
||||||
|
|||||||
@@ -71,7 +71,8 @@ The state directory (`state_dir`, `/var/lib/pixa` in the container) holds the
|
|||||||
database and the disk cache:
|
database and the disk cache:
|
||||||
|
|
||||||
- It needs a persistent volume: without one, every restart starts with an empty
|
- It needs a persistent volume: without one, every restart starts with an empty
|
||||||
cache. In the container, the startup script gives the directory to the user
|
cache. Losing the directory also ends every login session (see Encrypted
|
||||||
|
URLs). In the container, the startup script gives the directory to the user
|
||||||
pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
|
pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
|
||||||
must be able to write the directory.
|
must be able to write the directory.
|
||||||
- `cache_max_bytes` limits the source and transformed images together. The
|
- `cache_max_bytes` limits the source and transformed images together. The
|
||||||
@@ -145,14 +146,16 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
|
|||||||
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
|
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
|
||||||
use.
|
use.
|
||||||
- **Temporary files:** while pixa processes a large source, libvips writes its
|
- **Temporary files:** while pixa processes a large source, libvips writes its
|
||||||
decoded pixels to a file in `/tmp` and deletes the file when the image is
|
decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is
|
||||||
done. For a GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in
|
written, so a listing of `/tmp` does not show it, but it keeps its space, on
|
||||||
memory as well as written to that file, and the 4.2 GiB above includes that
|
disk or, where `/tmp` is a tmpfs, in memory, until the image is done. For a
|
||||||
memory. For an 8192x8192 source the file was measured on pixad at 192 MiB for
|
GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory as
|
||||||
an 8-bit RGB JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a
|
well as written to that file, and the 4.2 GiB above includes that memory. For
|
||||||
JPEG XL of 32-bit floating-point RGBA, the largest measured: up to 1 GiB per
|
an 8192x8192 source the file was measured on pixad at 192 MiB for an 8-bit RGB
|
||||||
image processed at once, 24 GiB at the defaults on a host where pixa sees 48
|
JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG XL of
|
||||||
CPUs. This `/tmp` is in the container's own filesystem, not on the volume at
|
32-bit floating-point RGBA, the largest measured: up to 1 GiB per image
|
||||||
|
processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs.
|
||||||
|
This `/tmp` is in the container's own filesystem, not on the volume at
|
||||||
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
|
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
|
||||||
does not include these files; where `/tmp` is a tmpfs, they are memory on top
|
does not include these files; where `/tmp` is a tmpfs, they are memory on top
|
||||||
of it.
|
of it.
|
||||||
@@ -190,6 +193,14 @@ name.
|
|||||||
Multiple source paths may reference the same content blob; the database tracks
|
Multiple source paths may reference the same content blob; the database tracks
|
||||||
references rather than using filesystem refcounting.
|
references rather than using filesystem refcounting.
|
||||||
|
|
||||||
|
Source and transformed image files, and the `.meta` file beside each transformed
|
||||||
|
image, are written under a temporary name, synced to disk, and only then renamed
|
||||||
|
into place, so a crash leaves no file under its own name with part of its data.
|
||||||
|
When pixa starts, and every five minutes after, it removes each cached source or
|
||||||
|
transformed image whose file is empty, or not the size the database records for
|
||||||
|
it, with a warning naming the file; the next request for it fetches the source
|
||||||
|
or converts the image again.
|
||||||
|
|
||||||
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
|
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
|
||||||
Test). Toward it, pixa keeps in memory the content types of the 10,000
|
Test). Toward it, pixa keeps in memory the content types of the 10,000
|
||||||
transformed images most recently cached or served, so a cache hit on one of them
|
transformed images most recently cached or served, so a cache hit on one of them
|
||||||
@@ -214,7 +225,11 @@ path under `/v1/` answers 200, in maintenance mode too.
|
|||||||
session it answers 303 to `/`. Answers: 200 with the page showing the URL; 400
|
session it answers 303 to `/`. Answers: 200 with the page showing the URL; 400
|
||||||
with the page naming a field that is not valid; 500 when the URL cannot be
|
with the page naming a field that is not valid; 500 when the URL cannot be
|
||||||
made.
|
made.
|
||||||
- `GET /logout` — end the login session. Needs: nothing. Answers: 303 to `/`.
|
- `POST /logout` — log out with the generator page's Logout button: end every
|
||||||
|
login session, in every browser (see Encrypted URLs), and remove the session
|
||||||
|
cookie. Needs: a login session and the generator page's form (below); without
|
||||||
|
a login session it only removes the cookie. Answers: 303 to `/`; 500 when the
|
||||||
|
logout cannot be saved in the database.
|
||||||
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or
|
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or
|
||||||
`/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized
|
`/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized
|
||||||
and converted (below). Needs: a signature, unless the host is allowlisted (see
|
and converted (below). Needs: a signature, unless the host is allowlisted (see
|
||||||
@@ -262,7 +277,7 @@ the request carries the same ID as `request_id`, and so do the lines it logs
|
|||||||
when it fetches, converts and serves an image; the fetch sends it to the
|
when it fetches, converts and serves an image; the fetch sends it to the
|
||||||
upstream host as `X-Request-ID`.
|
upstream host as `X-Request-ID`.
|
||||||
|
|
||||||
Both `POST` routes accept only a form that pixa's own page served: the page puts
|
The `POST` routes accept only a form that pixa's own page served: the page puts
|
||||||
a token in the form and sets a cookie to match, and a request without both is
|
a token in the form and sets a cookie to match, and a request without both is
|
||||||
refused with 403, so another site cannot submit the form from a visitor's
|
refused with 403, so another site cannot submit the form from a visitor's
|
||||||
browser. The login and generator pages are meant to be opened over HTTPS: while
|
browser. The login and generator pages are meant to be opened over HTTPS: while
|
||||||
@@ -301,6 +316,35 @@ once, is refused with 400.
|
|||||||
both optional (values under Signature Specification). Both are part of what is
|
both optional (values under Signature Specification). Both are part of what is
|
||||||
cached, so each value of either is a separate cached image.
|
cached, so each value of either is a separate cached image.
|
||||||
|
|
||||||
|
A source URL's query follows its path after `%3F`, the encoded `?`:
|
||||||
|
`/v1/image/<host>/<path>%3F<query>/<size>.<format>`. pixa decodes the
|
||||||
|
percent-escapes in the image URL's path once, then reads the host up to the
|
||||||
|
first `/`, the size and format after the last `/`, and between them the source
|
||||||
|
path up to the first `?` and the source query after it. So the `?` before the
|
||||||
|
query, any `?` in the query, and every `%` in the source path or query must be
|
||||||
|
percent-encoded, as `%3F` and `%25`. Any other character may be written as
|
||||||
|
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. Neither
|
||||||
|
the source path nor the source query may hold a `#`, and an image URL that
|
||||||
|
writes one as `%23` is refused with 400. For example,
|
||||||
|
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at
|
||||||
|
800x600 WebP is
|
||||||
|
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp`,
|
||||||
|
signed as shown under Signature Specification.
|
||||||
|
|
||||||
|
The source path is the path with its percent-escapes decoded, and pixa fetches
|
||||||
|
it percent-encoded again: every character other than letters, digits, `-`, `.`,
|
||||||
|
`_`, `~`, `/`, `$`, `&`, `+`, `,`, `:`, `;`, `=` and `@` is sent as a
|
||||||
|
percent-escape. So `my%20cat.jpg` in the image URL is the source path
|
||||||
|
`/my cat.jpg`, fetched as `/my%20cat.jpg`; `my%2520cat.jpg` is the source path
|
||||||
|
`/my%20cat.jpg`, fetched as `/my%2520cat.jpg`; and `100%25.png` is fetched as
|
||||||
|
`/100%25.png`. The source query is the source URL's query as that URL writes it,
|
||||||
|
its percent-escapes included, and is sent as it is, except that the characters a
|
||||||
|
browser percent-encodes in a query (a space, `"`, `'`, `<`, `>`, control
|
||||||
|
characters and characters outside ASCII) and a `%` that does not begin a
|
||||||
|
percent-escape are percent-encoded. pixa signs and caches the query in that
|
||||||
|
form, so `img%3Fname=my%20cat` and `img%3Fname=my%2520cat` in the image URL are
|
||||||
|
the same source query, `name=my%20cat`.
|
||||||
|
|
||||||
A size with one side `0` has that side scaled to keep the source's proportions,
|
A size with one side `0` has that side scaled to keep the source's proportions,
|
||||||
and is refused with 400 when that side would come out above 8192, the most
|
and is refused with 400 when that side would come out above 8192, the most
|
||||||
either side may be; so is a size with `fit=outside`, which scales the image to
|
either side may be; so is a size with `fit=outside`, which scales the image to
|
||||||
@@ -400,14 +444,28 @@ nor change what it asks for.
|
|||||||
|
|
||||||
1. Open `/` in a browser over HTTPS (or over plain HTTP while `debug` is on, see
|
1. Open `/` in a browser over HTTPS (or over plain HTTP while `debug` is on, see
|
||||||
Routes) and log in with the signing key (`signing_key`). The login session
|
Routes) and log in with the signing key (`signing_key`). The login session
|
||||||
lasts 30 days, or until `/logout`.
|
lasts 30 days, or until someone logs out with the generator page's Logout
|
||||||
|
button. Every login is made with the one signing key, so logging out ends
|
||||||
|
every login session, in every browser, a copy of the session cookie included.
|
||||||
|
pixa keeps a random value in its database, which it makes when the database
|
||||||
|
has none and replaces at each logout. A login session carries the value
|
||||||
|
current at its login and is refused once the value has changed, also after a
|
||||||
|
restart (see Deployment). The check for a logout compares no time, so a login
|
||||||
|
after a logout works whatever the host's clock says; the 30 days are still
|
||||||
|
measured by the host's clock.
|
||||||
2. On the generator page, give the source image's URL, the width and height, the
|
2. On the generator page, give the source image's URL, the width and height, the
|
||||||
format, quality and fit, and how long the URL lasts, then submit the form
|
format, quality and fit, and how long the URL lasts, then submit the form
|
||||||
(`POST /generate`). The format is JPEG XL unless another is chosen; a form
|
(`POST /generate`). The format is JPEG XL unless another is chosen; a form
|
||||||
sent with an empty format, or none, also makes a JPEG XL URL. Width and
|
sent with an empty format, or none, also makes a JPEG XL URL. Width and
|
||||||
height both empty or `0` keep the original size; if only one of them is empty
|
height both empty or `0` keep the original size; if only one of them is empty
|
||||||
or `0`, that side is scaled to keep the image's proportions. A URL whose
|
or `0`, that side is scaled to keep the image's proportions. A URL whose
|
||||||
scaled side would come out above 8192 answers 400.
|
scaled side would come out above 8192 answers 400, and so does a source URL
|
||||||
|
whose path holds a `..` segment. So does a source URL that the image URL
|
||||||
|
would fetch written another way, and the page then shows the URL pixa would
|
||||||
|
fetch: one with a user name and password, which pixa does not send, or one
|
||||||
|
whose path holds `%2F`, fetched as `/`, or `(`, fetched as `%28`, as pixa
|
||||||
|
keeps the source path decoded and fetches it percent-encoded again (see
|
||||||
|
Routes).
|
||||||
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
|
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
|
||||||
it expires. `<host>` is the host the page was opened on, and the URL starts
|
it expires. `<host>` is the host the page was opened on, and the URL starts
|
||||||
with `http` instead while `debug` is on. The name after the token is ignored
|
with `http` instead while `debug` is on. The name after the token is ignored
|
||||||
@@ -415,9 +473,11 @@ nor change what it asks for.
|
|||||||
`jxl` for a form with no format.
|
`jxl` for a form with no format.
|
||||||
|
|
||||||
The token holds the source's host, path and query and the size, format, quality,
|
The token holds the source's host, path and query and the size, format, quality,
|
||||||
fit and expiry, encrypted with a key derived from `signing_key`. The source
|
fit and expiry, encrypted with a key derived from `signing_key`. The host is
|
||||||
URL's scheme is not kept: the image is fetched like any other (see Routes), and
|
kept in lower case and an empty path as `/`, as for `/v1/image/`, so
|
||||||
the blocked networks still apply.
|
`https://CDN.Example.com` and `https://cdn.example.com/` are one source image,
|
||||||
|
fetched and cached once. The source URL's scheme is not kept: the image is
|
||||||
|
fetched like any other (see Routes), and the blocked networks still apply.
|
||||||
|
|
||||||
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never.
|
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never.
|
||||||
The expiry is fixed in the token when the URL is made and cannot be changed or
|
The expiry is fixed in the token when the URL is made and cannot be changed or
|
||||||
@@ -469,13 +529,20 @@ Where:
|
|||||||
- `host` — source host as the URL writes it up to the first `/`, in lower case,
|
- `host` — source host as the URL writes it up to the first `/`, in lower case,
|
||||||
keeping an IPv6 address's brackets and any `:` and port after it (e.g.
|
keeping an IPv6 address's brackets and any `:` and port after it (e.g.
|
||||||
`cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`)
|
`cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`)
|
||||||
- `path` — source path (e.g. `/photos/cat.jpg`), with each `%` written as `%25`
|
- `path` — source path, with its percent-escapes decoded as under Routes (e.g.
|
||||||
and each `:` as `%3A`, so `/photos/a:b.jpg` is signed as `/photos/a%3Ab.jpg`
|
`/photos/cat.jpg`, or `/photos/my cat.jpg` for `/photos/my%20cat.jpg`), with
|
||||||
- `query` — source query string, empty string if none, with `%` and `:` written
|
each `%` written as `%25` and each `:` as `%3A`, so `/photos/a:b.jpg` is
|
||||||
as in `path`; neither then holds a `:`, so path `/a:b` with no query and path
|
signed as `/photos/a%3Ab.jpg`
|
||||||
`/a` with query `b:` are signed differently
|
- `query` — source query string as under Routes, empty string if none, with `%`
|
||||||
- `width` — requested width in pixels, `0` for original
|
and `:` written as in `path`; neither then holds a `:`, so path `/a:b` with no
|
||||||
- `height` — requested height in pixels, `0` for original
|
query and path `/a` with query `b:` are signed differently
|
||||||
|
- `width` — requested width in pixels, or `0`: the width is then worked out from
|
||||||
|
the source's proportions and the requested height, rounded down but at least
|
||||||
|
1, or is the source's own width when the height is `0` too (the `orig` size)
|
||||||
|
- `height` — requested height in pixels, or `0`: the height is then worked out
|
||||||
|
from the source's proportions and the requested width, rounded down but at
|
||||||
|
least 1, or is the source's own height when the width is `0` too (the `orig`
|
||||||
|
size)
|
||||||
- `format` — output format, one of those listed under Routes, with `original`
|
- `format` — output format, one of those listed under Routes, with `original`
|
||||||
signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no
|
signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no
|
||||||
format has the signature of the same URL ending in `.jxl`; `auto` is signed as
|
format has the signature of the same URL ending in `.jxl`; `auto` is signed as
|
||||||
@@ -488,7 +555,10 @@ Where:
|
|||||||
with 400
|
with 400
|
||||||
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
|
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
|
||||||
outside), or `cover` when the URL has no `fit`; a request whose `fit` is
|
outside), or `cover` when the URL has no `fit`; a request whose `fit` is
|
||||||
anything else, an empty `fit=` included, is refused with 400
|
anything else, an empty `fit=` included, is refused with 400. `outside` scales
|
||||||
|
the image to cover the requested size, keeping its proportions, so it can come
|
||||||
|
out larger than that size on one side: an 8192x1 source at `100x0`, which is
|
||||||
|
100x1 by its proportions, comes out at 8192x1
|
||||||
|
|
||||||
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of
|
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of
|
||||||
RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
|
RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
|
||||||
@@ -512,6 +582,17 @@ For the same image at quality 40 with fit `contain`, the input ends in
|
|||||||
and the URL is
|
and the URL is
|
||||||
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.
|
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.
|
||||||
|
|
||||||
|
For a source URL with a query,
|
||||||
|
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg`, at
|
||||||
|
800x600 WebP with the same key and expiration and the default quality and fit:
|
||||||
|
|
||||||
|
1. Build input, with the query's `:` written as `%3A` and its `%` as `%25`:
|
||||||
|
`cdn.example.com:/img:url=https%3A//origin.example.com/my%2520cat.jpg:800:600:webp:1704067200:85:cover`
|
||||||
|
2. The signature is `37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=`
|
||||||
|
3. URL, with the `?` before the query written as `%3F` and the query's `%` as
|
||||||
|
`%25` (see Routes):
|
||||||
|
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp?sig=37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=&exp=1704067200`
|
||||||
|
|
||||||
**Allowlist patterns:**
|
**Allowlist patterns:**
|
||||||
|
|
||||||
- **Exact match**: `cdn.example.com` — matches only that host
|
- **Exact match**: `cdn.example.com` — matches only that host
|
||||||
@@ -527,7 +608,11 @@ pixa lower-cases the host of a `/v1/image/` URL before anything else uses it, so
|
|||||||
`CDN.Example.com` is matched, fetched, cached and counted against
|
`CDN.Example.com` is matched, fetched, cached and counted against
|
||||||
`upstream_connections_per_host` as `cdn.example.com`. Such a URL whose host has
|
`upstream_connections_per_host` as `cdn.example.com`. Such a URL whose host has
|
||||||
a port, as in `cdn.example.com:8443`, or a `:` with no port, needs a signature
|
a port, as in `cdn.example.com:8443`, or a `:` with no port, needs a signature
|
||||||
even when the host is allowlisted. A host holding `@` is refused with 400.
|
even when the host is allowlisted. The host must be a host name or an IPv4
|
||||||
|
address (letters, digits, hyphens, underscores and dots) or an IPv6 address in
|
||||||
|
brackets, either one optionally followed by `:` and a port. Any other host, such
|
||||||
|
as one holding `@`, or a `?`, `#`, `%`, `\` or space written percent-encoded, is
|
||||||
|
refused with 400.
|
||||||
|
|
||||||
### Configuration
|
### Configuration
|
||||||
|
|
||||||
@@ -609,7 +694,10 @@ Key settings in more detail:
|
|||||||
and generator pages. Default: empty
|
and generator pages. Default: empty
|
||||||
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, added
|
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, added
|
||||||
to the always-enforced built-in ranges (loopback, private, link-local, CGNAT,
|
to the always-enforced built-in ranges (loopback, private, link-local, CGNAT,
|
||||||
benchmark, NAT64, and the like); an invalid CIDR aborts startup
|
benchmark, NAT64, and the like); an invalid CIDR aborts startup. An IPv6
|
||||||
|
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
|
||||||
|
address it holds, so an IPv6 range such as `::/0` refuses neither it nor a
|
||||||
|
plain IPv4 address
|
||||||
- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front of
|
- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front of
|
||||||
pixa. `X-Forwarded-For` is believed only when the direct peer falls inside one
|
pixa. `X-Forwarded-For` is believed only when the direct peer falls inside one
|
||||||
of these ranges; the logged and login-recorded client address is then the
|
of these ranges; the logged and login-recorded client address is then the
|
||||||
@@ -640,11 +728,14 @@ Key settings in more detail:
|
|||||||
- `signing_key` — HMAC secret for URL signatures
|
- `signing_key` — HMAC secret for URL signatures
|
||||||
- `db_url` — the SQLite database to open; omitted, it is
|
- `db_url` — the SQLite database to open; omitted, it is
|
||||||
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
|
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
|
||||||
database in WAL mode. pixa opens one connection to it, so its own reads and
|
database in WAL mode. `state_dir` is percent-encoded in it, so a `%`, `?` or
|
||||||
writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to any
|
`#` in the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/`
|
||||||
`db_url`, so a write that finds another program writing to the file waits up
|
or one starting with `//` it starts `file:////`, so SQLite does not read the
|
||||||
to five seconds for it instead of failing. WAL mode comes only from the URL:
|
first name in the path as a host. pixa opens one connection to it, so its own
|
||||||
keep `_pragma=journal_mode(WAL)` in one you set
|
reads and writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to
|
||||||
|
any `db_url`, so a write that finds another program writing to the file waits
|
||||||
|
up to five seconds for it instead of failing. WAL mode comes only from the
|
||||||
|
URL: keep `_pragma=journal_mode(WAL)` in one you set
|
||||||
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
|
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
|
||||||
cache entirely; omitted defaults to 75% of the sum of the free space on the
|
cache entirely; omitted defaults to 75% of the sum of the free space on the
|
||||||
filesystem containing `<state_dir>/cache/` and the bytes of source and
|
filesystem containing `<state_dir>/cache/` and the bytes of source and
|
||||||
@@ -692,7 +783,9 @@ See `configs/config.example.yml` for all options with defaults.
|
|||||||
`metrics.username` and `metrics.password` are set. A request's `handler` label
|
`metrics.username` and `metrics.password` are set. A request's `handler` label
|
||||||
is the pattern of the route it matches, such as `/v1/image/*`, never its path.
|
is the pattern of the route it matches, such as `/v1/image/*`, never its path.
|
||||||
Requests that match no route share the label `unmatched`: those the router
|
Requests that match no route share the label `unmatched`: those the router
|
||||||
answers 404 or 405, and CORS preflights
|
answers 404 or 405, and CORS preflights. A request's `method` label is its
|
||||||
|
method when that is `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `DELETE`,
|
||||||
|
`OPTIONS`, `CONNECT` or `TRACE`, and `other` for any other method
|
||||||
- **Logging**: stdlib slog
|
- **Logging**: stdlib slog
|
||||||
|
|
||||||
## Entrypoints
|
## Entrypoints
|
||||||
@@ -711,9 +804,9 @@ provide:
|
|||||||
install-precommit)
|
install-precommit)
|
||||||
- `script/projectname` — output the project name ("pixa")
|
- `script/projectname` — output the project name ("pixa")
|
||||||
- `script/test` — run the test suite: build the `test` phase of the
|
- `script/test` — run the test suite: build the `test` phase of the
|
||||||
`Dockerfile`, tagged `pixa-test`
|
`Dockerfile`, writing no image
|
||||||
- `script/lint` — run golangci-lint: build the `lint` phase of the `Dockerfile`,
|
- `script/lint` — run golangci-lint: build the `lint` phase of the `Dockerfile`,
|
||||||
tagged `pixa-lint`; the linter never runs on the host
|
writing no image; the linter never runs on the host
|
||||||
- `script/fmt` — format the Go code with gofmt and the markdown with prettier
|
- `script/fmt` — format the Go code with gofmt and the markdown with prettier
|
||||||
(writes)
|
(writes)
|
||||||
- `script/fmt-check` — check the same formatting (read-only), on the host
|
- `script/fmt-check` — check the same formatting (read-only), on the host
|
||||||
@@ -731,15 +824,20 @@ provide:
|
|||||||
- `script/install-precommit` — install the git pre-commit hook that runs
|
- `script/install-precommit` — install the git pre-commit hook that runs
|
||||||
`script/precommit`
|
`script/precommit`
|
||||||
|
|
||||||
Every `docker build` in these scripts passes `--no-cache`, so the lint and test
|
Every `docker build` in these scripts passes a new value of the `CACHEBUST`
|
||||||
phases run on every build instead of coming from the build cache.
|
build argument, which every stage of the `Dockerfile` declares first, so every
|
||||||
|
`RUN` step, the lint and test phases included, runs on every build instead of
|
||||||
|
coming from the build cache. Go's build cache is kept from build to build: the
|
||||||
|
steps that compile mount it, so a package whose inputs are unchanged is not
|
||||||
|
compiled again, and both `go test` runs pass `-count=1` so that no stored test
|
||||||
|
result is replayed. The `test` phase runs on the Debian Go image, with Debian's
|
||||||
|
libvips and libheif; the image pixa ships is built and runs on Alpine.
|
||||||
`script/check`, `script/cibuild`, `script/docker`, `script/lint`, `script/test`,
|
`script/check`, `script/cibuild`, `script/docker`, `script/lint`, `script/test`,
|
||||||
`script/setup` and `script/install-precommit` are the standard copies from
|
`script/setup` and `script/install-precommit` are the standard copies from
|
||||||
`sneak/prompts`, kept identical to them. `script/fmt` and `script/fmt-check` are
|
`sneak/prompts`, kept identical to them. `script/fmt` and `script/fmt-check` are
|
||||||
the standard copies with pixa's `gofmt` step kept before prettier. prettier
|
the standard copies with pixa's `gofmt` step kept before prettier. prettier
|
||||||
formats the markdown only: not the HTML templates, as it cannot parse a Go
|
formats the markdown only, not the HTML templates, as it cannot parse a Go
|
||||||
template action inside a tag, and not `REPO_POLICIES.md` (see
|
template action inside a tag.
|
||||||
`.prettierignore`), a copy of the one in `sneak/prompts`.
|
|
||||||
|
|
||||||
## Load Test
|
## Load Test
|
||||||
|
|
||||||
|
|||||||
+258
-105
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Repository Policies
|
title: Repository Policies
|
||||||
last_modified: 2026-09-08
|
last_modified: 2026-10-15
|
||||||
---
|
---
|
||||||
|
|
||||||
This document covers repository structure, tooling, and workflow standards. Code
|
This document covers repository structure, tooling, and workflow standards. Code
|
||||||
@@ -54,34 +54,39 @@ style conventions are in separate documents:
|
|||||||
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
|
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
|
||||||
assumes nothing is present: base tools come from nix, apt, brew, or apk
|
assumes nothing is present: base tools come from nix, apt, brew, or apk
|
||||||
(detected in that order; apt runs noninteractive). For node it uses the
|
(detected in that order; apt runs noninteractive). For node it uses the
|
||||||
installed node if present; otherwise it installs a PINNED node version via
|
installed node only when its major version is the pinned one; otherwise (node
|
||||||
nvm, first installing nvm itself if missing — from a hash-verified GitHub
|
missing, or another major, such as the node 24 the CI runner image ships) it
|
||||||
release archive (never `curl | sh`), with bash installed as an explicit
|
installs the PINNED node version via nvm and installs yarn and the packages
|
||||||
prerequisite since nvm requires bash. yarn is then pinned via
|
under it. nvm itself is installed first if missing, from a hash-verified
|
||||||
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
|
GitHub release archive (never `curl | sh`), with bash installed as an explicit
|
||||||
always exact versions. `script/cibuild` runs the CI build: it changes to the
|
prerequisite since nvm requires bash. Only the major version is compared
|
||||||
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
|
because the `Dockerfile` stages start from a node image whose exact version is
|
||||||
with the version; the Gitea workflow calls it. **`script/cibuild` runs
|
not the pin, and nvm cannot install a prebuilt node on alpine. yarn is pinned
|
||||||
|
via `corepack prepare yarn@<version> --activate`. Never install "latest" or
|
||||||
|
"lts"; always exact versions. `script/cibuild` runs the CI build: it changes
|
||||||
|
to the repo root, runs `script/bootstrap`, runs `script/check`, and builds the
|
||||||
|
image with the version; the Gitea workflow calls it. **`script/cibuild` runs
|
||||||
`script/bootstrap` first**, because the workflow checks out the repo and runs
|
`script/bootstrap` first**, because the workflow checks out the repo and runs
|
||||||
nothing else, while `script/fmt-check` runs the formatter on the host: on a
|
nothing else, while `script/fmt-check` runs the formatter on the host: on a
|
||||||
pristine checkout with nothing installed the run dies there, after the
|
pristine checkout with nothing installed the run dies there, after the
|
||||||
containerised gates have passed. **The bootstrap alone is not enough**:
|
containerised gates have passed. **The bootstrap alone is not enough**: when
|
||||||
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
|
`script/bootstrap` installs node and yarn under nvm it leaves neither on the
|
||||||
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
|
`PATH` of the shell that called it, so a bare `yarn` either exits 127 or runs
|
||||||
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
|
under another node. The host entrypoints that need yarn — `script/fmt` and
|
||||||
source nvm for the pinned node version before invoking it, exactly as
|
`script/fmt-check` — therefore use the same test as `script/bootstrap`: when
|
||||||
`script/bootstrap`'s own install step does. A runner carrying nothing but
|
the node on `PATH` has the pinned major version they run the `yarn` on `PATH`,
|
||||||
docker and git then gets through `script/check`. Four further scripts are our
|
and otherwise they load nvm through `$HOME/.nvm/nvm.sh` and run yarn under the
|
||||||
own extensions to the standard: `script/check` runs `script/test`,
|
pinned node. A runner carrying nothing but docker and git then gets through
|
||||||
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
|
`script/check`. Four further scripts are our own extensions to the standard:
|
||||||
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
|
`script/check` runs `script/test`, `script/lint` and `script/fmt-check`;
|
||||||
installs the git pre-commit hook (the `make hooks` target shims to it); and
|
`script/precommit` is what the git pre-commit hook runs, and it calls
|
||||||
`script/projectname` (literally that filename) simply outputs the project's
|
`script/check`; `script/install-precommit` installs the git pre-commit hook
|
||||||
name. Scripts that need the name call `script/projectname` — e.g.
|
(the `make hooks` target shims to it); and `script/projectname` (literally
|
||||||
`script/docker` assembles its image tag from it — so those scripts stay
|
that filename) simply outputs the project's name. Scripts that need the name
|
||||||
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
|
call `script/projectname` — e.g. `script/docker` assembles its image tag from
|
||||||
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
|
it — so those scripts stay byte-identical across all repos. Repo-type-specific
|
||||||
the hook itself. Model scripts are at
|
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
|
||||||
|
`script/precommit`, not in the hook itself. Model scripts are at
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
||||||
must document the provided scripts in an **Entrypoints** section (see the
|
must document the provided scripts in an **Entrypoints** section (see the
|
||||||
README requirements below).
|
README requirements below).
|
||||||
@@ -104,18 +109,24 @@ style conventions are in separate documents:
|
|||||||
`lint` phase and a `test` phase, with the final stage depending on both so the
|
`lint` phase and a `test` phase, with the final stage depending on both so the
|
||||||
image cannot be built unless they pass. For non-server repos the final stage
|
image cannot be built unless they pass. For non-server repos the final stage
|
||||||
brings up a development environment; for server repos it is the runtime image.
|
brings up a development environment; for server repos it is the runtime image.
|
||||||
Dockerfiles install development prerequisites by running `script/bootstrap`
|
The gate phases and the build stage start from their pinned base images and
|
||||||
rather than duplicating installs inline; COPY `script/` and the dependency
|
install what those images lack either inline, as the canonical Go `Dockerfile`
|
||||||
manifests (`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before
|
below does for `git`, or by running `script/bootstrap`, as the `prompts`
|
||||||
running it.
|
repo's own `Dockerfile` does for its yarn packages. The development
|
||||||
|
environment stage installs development prerequisites by running
|
||||||
|
`script/bootstrap` rather than duplicating its installs inline. A stage that
|
||||||
|
runs `script/bootstrap` COPYs `script/` and the dependency manifests
|
||||||
|
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
|
||||||
|
|
||||||
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
|
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
|
||||||
no separate lint file. `script/lint` and `script/test` each build one phase
|
no separate lint file. `script/lint` and `script/test` each build one phase
|
||||||
and nothing else:
|
and nothing else:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
|
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
||||||
docker build --no-cache --target test -t "$(script/projectname)-test" .
|
--target lint --output type=cacheonly .
|
||||||
|
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
||||||
|
--target test --output type=cacheonly .
|
||||||
```
|
```
|
||||||
|
|
||||||
**A stage that is not the last one in the file is built only when the final
|
**A stage that is not the last one in the file is built only when the final
|
||||||
@@ -125,10 +136,15 @@ style conventions are in separate documents:
|
|||||||
plain `docker build .` builds the last stage alone and exits 0 having linted
|
plain `docker build .` builds the last stage alone and exits 0 having linted
|
||||||
and tested nothing.
|
and tested nothing.
|
||||||
|
|
||||||
**Every `docker build` in `script/` is tagged**, here and in
|
**The gate builds write no image.** With `--output type=cacheonly` the phase
|
||||||
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
|
runs and a failing step fails the build, but the result is not exported.
|
||||||
image behind on every invocation, on every developer host and every CI
|
Nothing uses those images, and writing one out is slow: a Go test phase's
|
||||||
runner; a tagged one replaces the previous image.
|
image holds the toolchain. A build given neither `--output` nor `-t` writes
|
||||||
|
an untagged image and leaves it dangling, on every developer host and every
|
||||||
|
CI runner. `script/cibuild` and `script/docker` build the image that ships
|
||||||
|
and tag it, so each build replaces the previous image; each assigns the tag
|
||||||
|
on its own line before the build, so `set -e` stops it where
|
||||||
|
`script/projectname` fails.
|
||||||
|
|
||||||
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
|
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
|
||||||
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
|
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
|
||||||
@@ -143,66 +159,107 @@ style conventions are in separate documents:
|
|||||||
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
|
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
|
||||||
exit non-zero with `parallel golangci-lint is running` — a status a caller
|
exit non-zero with `parallel golangci-lint is running` — a status a caller
|
||||||
cannot tell from real findings. Both have produced wrong verdicts in this
|
cannot tell from real findings. Both have produced wrong verdicts in this
|
||||||
org, in both directions. A container has its own cache, its own `TMPDIR` and
|
org, in both directions. A container has its own golangci-lint cache, its
|
||||||
a digest-pinned binary, so neither is reachable.
|
own `TMPDIR` and a digest-pinned binary, so neither is reachable.
|
||||||
|
|
||||||
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
|
- **Any build that runs checks is given a new `CACHEBUST` value.** Docker
|
||||||
a `COPY` layer only when the copied content changes, so on an unchanged tree
|
invalidates a `COPY` layer only when the copied content changes, so on an
|
||||||
the check `RUN` is served from cache, nothing executes, and the build still
|
unchanged tree the check `RUN` is served from cache, nothing executes, and the
|
||||||
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
|
build still exits 0. Every `docker build` in `script/` therefore passes
|
||||||
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
|
`--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the script's process
|
||||||
four, and there is no fifth — `script/check` runs the two gate phases and
|
id, so no two builds share a value, not even the three in one
|
||||||
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
|
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
|
||||||
not evidence that anything ran: a sub-second build reporting success is a
|
`script/docker` are the four, and there is no fifth — `script/check` runs the
|
||||||
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
|
two gate phases and `script/fmt-check`, and builds no image of its own. Every
|
||||||
and friends destroy a build cache shared with every other build on the host.
|
stage of the `Dockerfile` declares `ARG CACHEBUST` as its first line after
|
||||||
|
`FROM`, and a new value reruns every `RUN` step after that line; `COPY` steps
|
||||||
|
are compared by content and need nothing. A stage without the `ARG` line is
|
||||||
|
served from cache on an unchanged tree, so a repository re-vendors the scripts
|
||||||
|
and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`:
|
||||||
|
it reruns every step too, but it also empties every cache mount, and Go's
|
||||||
|
build cache, which the canonical Go `Dockerfile` below mounts so that compiled
|
||||||
|
packages survive between check builds, would start empty every time. A bare
|
||||||
|
`docker build .` is not evidence that anything ran: a sub-second build
|
||||||
|
reporting success is a cache hit, not a result. Never invalidate by pruning —
|
||||||
|
`docker builder prune` and friends destroy a build cache shared with every
|
||||||
|
other build on the host. When a check is added or changed, prove it works by
|
||||||
|
planting a defect it must catch and watching the run fail on it, then revert
|
||||||
|
the defect. A green run alone shows neither that the check ran nor that it
|
||||||
|
covers what it should.
|
||||||
|
|
||||||
- **The gate phases are separate stages, and the build stage depends on both.**
|
- **The gate phases are separate stages, and the build stage depends on both.**
|
||||||
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
||||||
hash), so lint failures surface in seconds rather than after a full compile,
|
hash), so lint failures surface in seconds rather than after a full compile,
|
||||||
and the test phase is based on the Go image. The canonical Go repo
|
and the test phase is based on the Debian Go image. The canonical Go repo
|
||||||
`Dockerfile`:
|
`Dockerfile`:
|
||||||
|
|
||||||
```dockerfile
|
```dockerfile
|
||||||
|
# Every stage declares ARG CACHEBUST first: script/ passes a new value
|
||||||
|
# on every build, which reruns every RUN step after it. The steps that
|
||||||
|
# compile mount Go's build cache, which survives from build to build.
|
||||||
|
|
||||||
# Lint phase
|
# Lint phase
|
||||||
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
||||||
FROM golangci/golangci-lint@sha256:... AS lint
|
FROM golangci/golangci-lint@sha256:... AS lint
|
||||||
|
ARG CACHEBUST
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN golangci-lint run --config .golangci.yml ./...
|
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||||
|
golangci-lint run --config .golangci.yml ./...
|
||||||
|
|
||||||
# Test phase
|
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
||||||
# golang:1.x-alpine, YYYY-MM-DD
|
# image ships and the alpine one does not. -count=1 because the mounted
|
||||||
|
# cache also holds test results, which go test would otherwise replay.
|
||||||
|
# golang:1.x, YYYY-MM-DD
|
||||||
FROM golang@sha256:... AS test
|
FROM golang@sha256:... AS test
|
||||||
|
ARG CACHEBUST
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN go test -timeout 90s -race -cover ./... || \
|
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||||
|
go test -count=1 -timeout 90s -race -cover ./... || \
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
go test -timeout 90s -race -v ./...; exit 1; }
|
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
|
||||||
|
|
||||||
# Build stage. Nothing is wanted from either phase above; the copies
|
# Build stage. Nothing is wanted from either phase above; the copies
|
||||||
# are what make BuildKit build them first, so this stage cannot run
|
# are what make BuildKit build them first, so this stage cannot run
|
||||||
# unless lint and test passed.
|
# unless lint and test passed.
|
||||||
# golang:1.x-alpine, YYYY-MM-DD
|
# golang:1.x-alpine, YYYY-MM-DD
|
||||||
FROM golang@sha256:... AS builder
|
FROM golang@sha256:... AS builder
|
||||||
|
ARG CACHEBUST
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
COPY --from=test /src/go.sum /dev/null
|
COPY --from=test /src/go.sum /dev/null
|
||||||
|
RUN apk add --no-cache git
|
||||||
|
# A tar-stream context keeps the sender's file owners, which git refuses.
|
||||||
|
RUN git config --system --add safe.directory /src
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
ARG VERSION=dev
|
# The VERSION build arg when one is given, otherwise
|
||||||
RUN CGO_ENABLED=0 go build -trimpath \
|
# `git describe --tags --always` on the .git in the build context. With
|
||||||
-ldflags="-s -w -X main.Version=${VERSION}" \
|
# .git present, a version that is still empty, dev or unknown fails the
|
||||||
-o /app ./cmd/app/
|
# build: git is missing or could not read the checkout.
|
||||||
|
ARG VERSION
|
||||||
|
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||||
|
VERSION="${VERSION:-$(git describe --tags --always)}"; \
|
||||||
|
if [ -e .git ]; then \
|
||||||
|
case "$VERSION" in ""|dev|unknown) \
|
||||||
|
echo "version is '$VERSION' although .git is present" >&2; \
|
||||||
|
exit 1 ;; \
|
||||||
|
esac; \
|
||||||
|
fi; \
|
||||||
|
CGO_ENABLED=0 go build -trimpath \
|
||||||
|
-ldflags="-s -w -X main.Version=${VERSION}" \
|
||||||
|
-o /app ./cmd/app/
|
||||||
|
|
||||||
# Runtime stage, and the last one
|
# Runtime stage, and the last one
|
||||||
FROM alpine@sha256:...
|
FROM alpine@sha256:...
|
||||||
|
ARG CACHEBUST
|
||||||
COPY --from=builder /app /usr/local/bin/app
|
COPY --from=builder /app /usr/local/bin/app
|
||||||
ENTRYPOINT ["app"]
|
ENTRYPOINT ["app"]
|
||||||
```
|
```
|
||||||
@@ -217,23 +274,88 @@ style conventions are in separate documents:
|
|||||||
- Keep the runtime stage last, and if you add a stage after it, give it the
|
- Keep the runtime stage last, and if you add a stage after it, give it the
|
||||||
same two copies. A plain `docker build .` builds the last stage's chain
|
same two copies. A plain `docker build .` builds the last stage's chain
|
||||||
and nothing else.
|
and nothing else.
|
||||||
|
- Only Go's build cache is mounted (`/root/.cache/go-build`, where Go keeps
|
||||||
|
it for root), and only on the steps that compile: `golangci-lint run`,
|
||||||
|
both `go test` runs and `go build`. It keeps compiled packages between
|
||||||
|
check builds, and Go reuses one only when all its inputs are unchanged, so
|
||||||
|
it saves the compile without skipping a check. Nothing else is kept:
|
||||||
|
golangci-lint's own cache would replay stored findings in place of a lint
|
||||||
|
run, and Go builds from whatever files a kept module cache holds without
|
||||||
|
checking them again (only `go mod verify` does), so a file changed there
|
||||||
|
would go unnoticed. Go's build cache also holds test results, which is why
|
||||||
|
both `go test` runs carry `-count=1`.
|
||||||
- If the project uses `//go:embed` directives that reference build artifacts
|
- If the project uses `//go:embed` directives that reference build artifacts
|
||||||
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
||||||
create placeholder files so the embed directives resolve. Example:
|
create placeholder files so the embed directives resolve. Example:
|
||||||
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
|
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
|
||||||
- If the project requires CGO or system libraries for linting (e.g.
|
- If the project requires CGO or system libraries for linting, install them
|
||||||
`vips-dev`), install them in the lint phase with `apk add`.
|
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
|
||||||
- `ARG VERSION=dev` is declared in the stage that compiles and supplied by
|
has no `apk`, so install with `apt-get` under the Debian package name
|
||||||
`script/docker` and `script/cibuild`; no stage may call `git describe`.
|
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
|
||||||
|
lists in the same `RUN`, so the layer does not keep them:
|
||||||
|
|
||||||
|
```dockerfile
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends libvips-dev \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
```
|
||||||
|
|
||||||
|
- `.dockerignore` lets `.git` into the build context. It keeps out every git
|
||||||
|
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
|
||||||
|
repository's own, each submodule's under `.git/modules/`, and that of a
|
||||||
|
submodule keeping its own `.git` directory. `git describe` does not need
|
||||||
|
them, and each can hold a credential: a password in a remote URL, or the
|
||||||
|
token the CI checkout step stores there. A submodule whose name has a
|
||||||
|
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
|
||||||
|
git directory to `**/.git/modules/**/config`, and Go's version stamping
|
||||||
|
then fails the build: give it a name without that segment
|
||||||
|
(`git submodule add --name`). The stage that compiles has `git` (the
|
||||||
|
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
|
||||||
|
takes the version from the `VERSION` build argument when one is given,
|
||||||
|
otherwise from `git describe --tags --always`. That gives the tag on a
|
||||||
|
tagged commit; on a later commit, the tag, the number of commits since it
|
||||||
|
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
|
||||||
|
tag is reachable. The stage that compiles also marks its working directory
|
||||||
|
safe for git (`git config --system --add safe.directory /src`): a context
|
||||||
|
sent as a tar stream keeps the sender's file owners, and git refuses a
|
||||||
|
checkout owned by another user, so the version would come out empty.
|
||||||
|
`ARG VERSION` has no default, and the build fails if the context carries
|
||||||
|
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
|
||||||
|
`docker build .` with no build arguments must succeed; a Dockerfile that
|
||||||
|
refuses an empty build argument drops that refusal and keeps the argument.
|
||||||
|
A checkout whose `.git` is a file (a linked worktree, or a repository
|
||||||
|
checked out as a submodule) is the exception: that file points to a git
|
||||||
|
directory outside the build context, so the build cannot read the version
|
||||||
|
and a plain `docker build .` fails; pass the version with
|
||||||
|
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
|
||||||
|
do.
|
||||||
|
|
||||||
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
||||||
runs `script/cibuild` on push, and checks out the repo as its only other step.
|
runs `script/cibuild` on push, and checks out the repo as its only other step,
|
||||||
That script bootstraps, runs the gate phases, and then builds the image, so a
|
with `persist-credentials: false`: `script/cibuild` needs no token, and
|
||||||
successful run means every check passed; a bare `docker build .` does not
|
without it the checkout leaves the job's token in `.git/config` for every
|
||||||
|
later step. The checkout step also sets `fetch-depth: 0`, which fetches the
|
||||||
|
tags `git describe` needs: by default it clones shallow with no tags, and a
|
||||||
|
tagged repository's CI build would stamp a bare short commit id. The
|
||||||
|
workflow's `concurrency` block groups runs by workflow and branch
|
||||||
|
(`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`,
|
||||||
|
so a new push cancels the older run on the same branch, queued or running, and
|
||||||
|
no other: runs for replaced commits do not hold up the shared runner.
|
||||||
|
`script/cibuild` bootstraps, runs the gate phases, and then builds the image,
|
||||||
|
so a successful run means every check passed; a bare `docker build .` does not
|
||||||
carry the same guarantee, because its gate phases may come from the cache. The
|
carry the same guarantee, because its gate phases may come from the cache. The
|
||||||
image build is uncached and so runs the gate phases a second time. That is the
|
image build gets a `CACHEBUST` value of its own and so runs the gate phases a
|
||||||
price of the rule above, and it is worth paying: the image that ships is built
|
second time. That is the price of the rule above, and it is worth paying: the
|
||||||
from a run of its own gates rather than from a cache entry.
|
image that ships is built from a run of its own gates rather than from a cache
|
||||||
|
entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the
|
||||||
|
shared runner after 20 minutes. That allows for the three Docker builds
|
||||||
|
described above (the test phase, the lint phase, then the image), each held to
|
||||||
|
the 5-minute Docker build limit below, plus the bootstrap. A separate workflow
|
||||||
|
limited to `main` by a `branches` list under `on: push` cannot be checked by
|
||||||
|
review: to try a change to it, add the feature branch to that list and push,
|
||||||
|
then remove the branch from the list again before merging. Keep any job in it
|
||||||
|
that publishes behind `if: github.ref_name == 'main'`, so the run from the
|
||||||
|
feature branch publishes nothing.
|
||||||
|
|
||||||
- Use platform-standard formatters: `black` for Python, `prettier` for
|
- Use platform-standard formatters: `black` for Python, `prettier` for
|
||||||
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
||||||
@@ -286,17 +408,18 @@ style conventions are in separate documents:
|
|||||||
```
|
```
|
||||||
|
|
||||||
`-count=1` is required on both invocations: it defeats Go's test _result_
|
`-count=1` is required on both invocations: it defeats Go's test _result_
|
||||||
cache, so the target cannot report a pass it did not earn, and the rerun
|
cache, so neither run can report a stored pass in place of running the
|
||||||
reproduces a failure instead of replaying it. It leaves the build cache
|
tests. It leaves the build cache alone, so it costs the runtime of the suite
|
||||||
alone, so it costs the runtime of the suite and no recompilation.
|
and no recompilation.
|
||||||
|
|
||||||
Note that this is a second, independent cache, stacked below the Docker
|
That cache is Go's own, separate from Docker's layer cache. Go stores a
|
||||||
layer cache that [issue #26](https://git.eeqj.de/sneak/prompts/issues/26)
|
passing result in its cache directory (`GOCACHE`), and when the same tests
|
||||||
addresses. `CHECK_EPOCH` guarantees the `RUN make test` _step_ re-executes;
|
run again on unchanged code it prints that result, marked `(cached)`,
|
||||||
it does not guarantee `go test` inside that step does any work, because the
|
without running them. The directory lasts from one run to the next on a
|
||||||
`GOCACHE` baked into earlier image layers survives into the re-executed
|
developer's machine, where this target runs, and in the `test` phase of the
|
||||||
step. They are two separate defects requiring two separate fixes, and a fix
|
`Dockerfile`, which mounts it to keep compiled packages between check
|
||||||
for one must not be recorded as covering the other.
|
builds, so the phase's `go test` runs carry `-count=1` too. A new
|
||||||
|
`CACHEBUST` value (above) is what makes that step run on an unchanged tree.
|
||||||
|
|
||||||
Python example:
|
Python example:
|
||||||
|
|
||||||
@@ -324,9 +447,12 @@ style conventions are in separate documents:
|
|||||||
|
|
||||||
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
||||||
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
|
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
|
||||||
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
|
`node_modules/`, and the repo's own build outputs. Fetch the standard
|
||||||
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
|
`.gitignore` from
|
||||||
setting up a new repo. These patterns are written to `.gitignore`'s own
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
|
||||||
|
a new repo. A repo's `.gitignore` is the standard file followed by the repo's
|
||||||
|
own entries, such as its binaries; a re-vendor replaces the standard part and
|
||||||
|
keeps those entries. These patterns are written to `.gitignore`'s own
|
||||||
semantics, in which an unanchored pattern already matches at every depth; they
|
semantics, in which an unanchored pattern already matches at every depth; they
|
||||||
are not a `.dockerignore` and must not be transplanted into one unmodified.
|
are not a `.dockerignore` and must not be transplanted into one unmodified.
|
||||||
|
|
||||||
@@ -340,7 +466,7 @@ style conventions are in separate documents:
|
|||||||
— which is more dangerous than a short file with no secret patterns at all,
|
— which is more dangerous than a short file with no secret patterns at all,
|
||||||
because it reads as solved and stops anyone looking. Give every
|
because it reads as solved and stops anyone looking. Give every
|
||||||
depth-independent pattern the `**/` prefix and leave only genuinely
|
depth-independent pattern the `**/` prefix and leave only genuinely
|
||||||
root-anchored entries unprefixed: `.git`, and the repo's own host-built
|
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
|
||||||
binary, written `/myapp` and never `**/myapp`, which would also match
|
binary, written `/myapp` and never `**/myapp`, which would also match
|
||||||
`cmd/myapp/` and delete the package directory from the context. Matching is
|
`cmd/myapp/` and delete the package directory from the context. Matching is
|
||||||
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
|
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
|
||||||
@@ -365,21 +491,24 @@ style conventions are in separate documents:
|
|||||||
directory, so a repo running agents in subdirectories still ships
|
directory, so a repo running agents in subdirectories still ships
|
||||||
`services/api/.claude/` and must add its own anchored entry there.
|
`services/api/.claude/` and must add its own anchored entry there.
|
||||||
|
|
||||||
- **Excluding `.git` means `git describe` cannot run inside any build stage, and
|
- **A plain `docker build .` of a clone stamps the version that
|
||||||
it fails quietly there.** In a build stage there is no repository, so
|
`git describe --tags --always` gives**, derived from the `.git` in the build
|
||||||
`git describe` writes nothing to stdout, `-X main.Version=` comes out empty,
|
context as the canonical `Dockerfile` above shows. Without its failure check,
|
||||||
the binary reports no version at all, and the build still exits 0. Compute the
|
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
|
||||||
version on the host and thread it in as a build arg. `script/docker` and
|
and the build would still exit 0. `script/docker` and `script/cibuild` pass
|
||||||
`script/cibuild` do this, byte-identically across repos:
|
the version they compute on the host; it takes precedence. They do this
|
||||||
|
byte-identically across repos:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
# Own line: a failing command substitution inside an argument does not
|
# The version and the tag each get their own line: a failing command
|
||||||
# trip `set -e`, so the inline form degrades to an empty constant.
|
# substitution inside an argument does not trip `set -e`, so the inline
|
||||||
|
# form degrades to an empty constant.
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
docker build --no-cache \
|
tag="$(script/projectname)"
|
||||||
|
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$(script/projectname)" .
|
-t "$tag" .
|
||||||
```
|
```
|
||||||
|
|
||||||
`--always` makes an untagged repo yield an abbreviated commit hash rather
|
`--always` makes an untagged repo yield an abbreviated commit hash rather
|
||||||
@@ -387,12 +516,12 @@ style conventions are in separate documents:
|
|||||||
fallback is applied — a live check that fires on a build from an export with
|
fallback is applied — a live check that fires on a build from an export with
|
||||||
no `.git` and on a repository with no commits yet. Do not fold it into the
|
no `.git` and on a repository with no commits yet. Do not fold it into the
|
||||||
substitution as `|| echo unknown`, which makes the guard unreachable. The
|
substitution as `|| echo unknown`, which makes the guard unreachable. The
|
||||||
Dockerfile's side is `ARG VERSION=dev` in the stage that compiles, declared
|
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
|
||||||
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
||||||
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
||||||
the scripts stay byte-identical. One consequence for CI: the standard
|
the scripts stay byte-identical. One consequence for CI: the standard
|
||||||
checkout action clones shallow and fetches no tags, so a repo that embeds a
|
checkout action clones shallow and fetches no tags, so the canonical
|
||||||
tag-derived version must set `fetch-depth: 0` on its checkout step.
|
`.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step.
|
||||||
|
|
||||||
- **Verify `.dockerignore` by enumerating the image, not by reading the
|
- **Verify `.dockerignore` by enumerating the image, not by reading the
|
||||||
patterns.** Plant files at the root _and_ at least two directories deep, build
|
patterns.** Plant files at the root _and_ at least two directories deep, build
|
||||||
@@ -426,12 +555,18 @@ style conventions are in separate documents:
|
|||||||
`test-support` depguard rule, where a repo names its own test-support packages
|
`test-support` depguard rule, where a repo names its own test-support packages
|
||||||
by full import path. A repo adds entries there and changes nothing else, and a
|
by full import path. A repo adds entries there and changes nothing else, and a
|
||||||
re-vendor carries its entries forward. The canonical golangci-lint version is
|
re-vendor carries its entries forward. The canonical golangci-lint version is
|
||||||
v2.12.2 (released 2026-05-06), pinned as the digest of the lint phase's base
|
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
|
||||||
image
|
image
|
||||||
(`golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240`,
|
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
|
||||||
which reports `2.12.2 built with go1.26.2 from c0d3ddc9`). That digest is the
|
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
|
||||||
only pin, since no repo installs golangci-lint on the host: bumping the
|
directive must not name a newer Go minor version than the one golangci-lint
|
||||||
version means changing it and nothing else.
|
was built with, or golangci-lint refuses to lint it: this release lints
|
||||||
|
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
|
||||||
|
installs golangci-lint on the host. A repo sets the lint phase digest to the
|
||||||
|
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
|
||||||
|
the two prompted the change: the canonical copy can name linters that an older
|
||||||
|
golangci-lint rejects, and a newer golangci-lint can add linters that
|
||||||
|
`default: all` switches on until the canonical copy disables them.
|
||||||
|
|
||||||
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
|
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
|
||||||
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
|
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
|
||||||
@@ -453,8 +588,18 @@ style conventions are in separate documents:
|
|||||||
function defined and never invoked has the same exit status and the same
|
function defined and never invoked has the same exit status and the same
|
||||||
empty output as one that worked.
|
empty output as one that worked.
|
||||||
|
|
||||||
|
Node is the exception to the whole-token comparison: the canonical
|
||||||
|
`script/bootstrap` compares only its major version, because the node 22
|
||||||
|
alpine image the `Dockerfile` stages start from is not the exact pin, and
|
||||||
|
nvm cannot install a prebuilt node on alpine.
|
||||||
|
|
||||||
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
||||||
|
|
||||||
|
A Go tool a repo needs on the host is installed with `go install` pinned to
|
||||||
|
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
|
||||||
|
a `go.mod` tool dependency or through a `tools.go` file, either of which
|
||||||
|
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
|
||||||
|
|
||||||
- When pinning images or packages by hash, add a comment above the reference
|
- When pinning images or packages by hash, add a comment above the reference
|
||||||
with the version and date (YYYY-MM-DD).
|
with the version and date (YYYY-MM-DD).
|
||||||
|
|
||||||
@@ -564,13 +709,17 @@ style conventions are in separate documents:
|
|||||||
Never edit existing migrations after release.
|
Never edit existing migrations after release.
|
||||||
|
|
||||||
- All repos should have an `.editorconfig` enforcing the project's indentation
|
- All repos should have an `.editorconfig` enforcing the project's indentation
|
||||||
settings.
|
settings: the standard file from
|
||||||
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, which sets
|
||||||
|
tabs for `Makefile` and Go files, followed by the repo's own sections, such as
|
||||||
|
one for another language it uses. A re-vendor replaces the standard part and
|
||||||
|
keeps those sections.
|
||||||
|
|
||||||
- Avoid putting files in the repo root unless necessary. Root should contain
|
- Avoid putting files in the repo root unless necessary. Root should contain
|
||||||
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
|
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
|
||||||
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
|
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
|
||||||
language-specific config). Everything else goes in a subdirectory. Canonical
|
and language-specific config). Everything else goes in a subdirectory.
|
||||||
subdirectory names:
|
Canonical subdirectory names:
|
||||||
- `bin/` — executable scripts and tools
|
- `bin/` — executable scripts and tools
|
||||||
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
|
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
|
||||||
body is a single call into `internal/` or `pkg/`, no project logic in
|
body is a single call into `internal/` or `pkg/`, no project logic in
|
||||||
@@ -601,3 +750,7 @@ style conventions are in separate documents:
|
|||||||
- Go: `go.mod`, `go.sum`, `.golangci.yml`
|
- Go: `go.mod`, `go.sum`, `.golangci.yml`
|
||||||
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
|
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
|
||||||
- Python: `pyproject.toml`
|
- Python: `pyproject.toml`
|
||||||
|
|
||||||
|
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
|
||||||
|
is never committed under a file or directory named after one agent tool, such
|
||||||
|
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
|
||||||
|
|||||||
@@ -18,7 +18,7 @@
|
|||||||
|
|
||||||
pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work lands on
|
pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work lands on
|
||||||
`next`, and `main` receives only the milestone PR that the owner merges. `next`
|
`next`, and `main` receives only the milestone PR that the owner merges. `next`
|
||||||
is at the canonical `golangci-lint` v2.12.2 config and is green. Recent work
|
is at the canonical `golangci-lint` v2.14.0 config and is green. Recent work
|
||||||
extracted the internal/magic, internal/allowlist, internal/httpfetcher, and
|
extracted the internal/magic, internal/allowlist, internal/httpfetcher, and
|
||||||
internal/signature packages. The gosec findings from the 2026-07-06 survey are
|
internal/signature packages. The gosec findings from the 2026-07-06 survey are
|
||||||
resolved. The disk cache is now size-bounded with LRU eviction
|
resolved. The disk cache is now size-bounded with LRU eviction
|
||||||
@@ -30,6 +30,114 @@ P2: security: per-origin rate limiting
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal
|
||||||
|
addresses refuses the local-use NAT64 range (`64:ff9b:1::/48`, RFC 8215) as it
|
||||||
|
refuses `64:ff9b::/96`, and refuses an IPv6 address in the IPv4-compatible
|
||||||
|
form (`::/96`) when `blocked_networks` contains either the address as written
|
||||||
|
or the IPv4 address it holds (closes #299). A source host such as
|
||||||
|
`[64:ff9b:1::a00:1]` was fetched, which on a network that runs NAT64 on that
|
||||||
|
range reaches the IPv4 address it holds, here `10.0.0.1`; and with
|
||||||
|
`203.0.113.0/24` in `blocked_networks`, `[::ffff:203.0.113.5]` was refused but
|
||||||
|
`[::203.0.113.5]` was fetched. Both now answer 403. The mapped form
|
||||||
|
(`::ffff:a.b.c.d`) is still matched only as the IPv4 address it holds: Go
|
||||||
|
keeps a plain IPv4 address in that form, and an IPv6 entry such as `::/0` must
|
||||||
|
not refuse every IPv4 address.
|
||||||
|
- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal
|
||||||
|
addresses judges an IPv6 address in the IPv4-compatible form (`::/96`), such
|
||||||
|
as `::127.0.0.1` or `::10.0.0.1`, by the IPv4 address in its last four bytes
|
||||||
|
(closes #297). It read only the mapped form (`::ffff:a.b.c.d`) as IPv4, so a
|
||||||
|
source host such as `[::127.0.0.1]` was checked as a public IPv6 address and
|
||||||
|
fetched, which on a network that routes it reaches the internal IPv4 host; it
|
||||||
|
is now refused with 403. The NAT64 form (`64:ff9b::/96`) was already refused
|
||||||
|
as a whole, as one of the built-in blocked ranges, and still is.
|
||||||
|
- 2026-10-16 a source host that is an IPv6 address in brackets with no port,
|
||||||
|
such as `[2001:db8::a]`, is fetched like one with a port, through `/v1/image/`
|
||||||
|
and through an encrypted URL (closes #295). The fetcher's check of the source
|
||||||
|
URL kept the brackets when no port followed them, could not look the host up,
|
||||||
|
and the request answered 500. It now checks the address without brackets, so a
|
||||||
|
private one is refused with 403, as with a port.
|
||||||
|
- 2026-10-16 a `state_dir` whose name holds `%`, `?` or `#` keeps its database
|
||||||
|
at `state.sqlite3` in that directory (closes #290): the `db_url` derived from
|
||||||
|
it put the name into a `file:` URL as it was, so SQLite read `%41` as `A`, a
|
||||||
|
`?` or `#` ended the path early, and the journal mode parameter was lost. The
|
||||||
|
derived `db_url` is now built with Go's `net/url`, which writes those
|
||||||
|
characters as `%25`, `%3F` and `%23`. For a `state_dir` of `/` or one starting
|
||||||
|
with `//` it starts `file:////`, so SQLite no longer reads the first name in
|
||||||
|
the path as a host.
|
||||||
|
- 2026-10-16 a size with one side `0` keeps the side computed from the source's
|
||||||
|
proportions at 1 or more (closes #283): it was rounded down, so for a thin
|
||||||
|
source it came out at 0, and libvips then made the size square. An 8192x1
|
||||||
|
source at `100x0` came out at 100x100, and at `1300x0` with `fit=cover` made
|
||||||
|
libvips ask for about 39 GB at once; an 8192x2 source at `100x0` answered 500.
|
||||||
|
`fit=contain` and `fit=inside` work both sides out again from the smaller of
|
||||||
|
the two scales and `fit=outside` from the larger, and each keeps both sides at
|
||||||
|
1 or more too, so a 1x8192 source at `0x100` with `fit=contain`, and a 49x8192
|
||||||
|
source at `0x100` or a 49x49 source at `1x1` with `fit=outside`, no longer
|
||||||
|
answer 500. An 8192x1 source at `1300x0` now comes out at 1300x1, and the
|
||||||
|
process running it alone peaked at about 34 MiB resident. With `fit=outside`
|
||||||
|
the image is scaled to cover the size, so that source at `100x0` comes out at
|
||||||
|
8192x1. `README.md` now says this under Signature Specification, and says how
|
||||||
|
a `0` side is worked out where it said `0` meant the original.
|
||||||
|
- 2026-10-16 the generator page lower-cases the source host and writes an empty
|
||||||
|
source path as `/` before it makes the encrypted URL, as `/v1/image/` does
|
||||||
|
(closes #293). A host written in upper and in lower case, or a source URL with
|
||||||
|
no path and with the path `/`, made two cache keys and two upstream requests
|
||||||
|
for one image.
|
||||||
|
- 2026-10-16 the source URL a request fetches is built with `net/url` from the
|
||||||
|
source host, path and query, not joined as text (closes #288). The source
|
||||||
|
path, decoded, is fetched percent-encoded again, so `100%25.png` fetches
|
||||||
|
`/100%25.png` where it answered 401, and `my%20cat.png` and `my%2520cat.png`
|
||||||
|
fetch different paths where both fetched `/my%20cat.png` under two cache keys.
|
||||||
|
The source query stays as the source URL writes it; the characters a browser
|
||||||
|
percent-encodes in a query, such as `'` or a space, which was sent raw and
|
||||||
|
answered 502, are percent-encoded when the image URL is read, so the query is
|
||||||
|
signed and cached as it is sent. The generator page answers 400, showing the
|
||||||
|
URL pixa would fetch, for a source URL its image URL would fetch written
|
||||||
|
another way, such as with `%2F` in the path, whatever else the path holds, or
|
||||||
|
with a user name and password. It also answers 400 for a source path holding a
|
||||||
|
`..` segment, as `/v1/image/` refuses one.
|
||||||
|
- 2026-10-15 a thin source requested with `fit=cover` no longer makes libvips
|
||||||
|
build a huge image (closes #282): libvips' thumbnail scaled the whole source
|
||||||
|
to cover the requested size and copied that into memory before it cut out the
|
||||||
|
centre, so an 8192x64 source at 8192x8192 made it allocate about 24 GiB, and
|
||||||
|
an 8192x1 source answered 500. The image processor now first cuts the centre
|
||||||
|
of the source, widened to whole source pixels, and thumbnail scales and cuts
|
||||||
|
only that, by the same amount on both sides as before. What it copies into
|
||||||
|
memory is then larger than the output by less than one scaled source pixel on
|
||||||
|
each side. With the runtime image's libvips and JPEG output at 8192x8192,
|
||||||
|
resident memory peaked at about 242 MiB for the 8192x64 source and 431 MiB for
|
||||||
|
the 8192x1 one. The output can sit a fraction of an output pixel from where it
|
||||||
|
was.
|
||||||
|
- 2026-10-15 a `/v1/image/` URL whose source path or source query holds a `#`,
|
||||||
|
written as `%23`, is refused with 400 (closes #286): the source URL built from
|
||||||
|
it ended at the `#` when fetched, so each text after it fetched the same
|
||||||
|
upstream image, without a signature for an allowlisted host, and cached it
|
||||||
|
under keys of its own. `README.md` says neither may hold a `#`.
|
||||||
|
- 2026-10-15 the host of a `/v1/image/` URL must be a host name or an IPv4
|
||||||
|
address (letters, digits, hyphens, underscores and dots) or an IPv6 address in
|
||||||
|
brackets, either one optionally followed by `:` and a port, and any other host
|
||||||
|
is refused with 400 (closes #275): only `@` was refused, so a `?` written as
|
||||||
|
`%3F` gave the host `cdn.example.com?x=1`, which Go's URL parser reads as
|
||||||
|
`cdn.example.com` with a query. Such a URL passed the allowlist, and each text
|
||||||
|
after the `?` fetched the host's root path and cached the answer under keys of
|
||||||
|
its own. A `#`, `%`, `\` or space written percent-encoded is refused the same
|
||||||
|
way, and so is an IPv6 address without brackets, which needed a signature
|
||||||
|
before. Underscores are kept because `allowlist_hosts` accepts host names
|
||||||
|
holding them.
|
||||||
|
- 2026-10-15 the README says how a source URL with a query is written into an
|
||||||
|
image URL, and the image route accepts it (closes #255): the query follows the
|
||||||
|
source path after `%3F`. pixa now reads the image URL's path decoded once, so
|
||||||
|
the `?` before the query, any `?` in it and every `%` in the source path or
|
||||||
|
query must be percent-encoded, and any other character may be. The route used
|
||||||
|
to parse chi's wildcard, which holds the path as the client wrote it whenever
|
||||||
|
that is not Go's own encoding of it; a `/`, `;` or `,` in the query written as
|
||||||
|
`%2F`, `%3B` or `%2C`, as `GenerateSignedURL` wrote them, kept the `%3F` from
|
||||||
|
being read as `?`, and the request failed with 401. `GenerateSignedURL` now
|
||||||
|
writes Go's own encoding of the path. The README has an example with a query
|
||||||
|
and its signature. The path traversal check no longer reads the source query,
|
||||||
|
and keeps a `%` that does not start a percent-escape as it is instead of
|
||||||
|
refusing the URL, so a source path or query holding `%25` is accepted, while a
|
||||||
|
double-encoded `..` next to such a `%` is still refused.
|
||||||
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
|
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
|
||||||
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
|
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
|
||||||
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
|
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
|
||||||
@@ -42,6 +150,64 @@ P2: security: per-origin rate limiting
|
|||||||
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
|
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
|
||||||
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
|
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
|
||||||
`/tmp` is a tmpfs.
|
`/tmp` is a tmpfs.
|
||||||
|
- 2026-10-15 the canonical files are re-vendored from `sneak/prompts` commit
|
||||||
|
`e3f79c4` (closes #221): `.dockerignore`, `.editorconfig`,
|
||||||
|
`.gitea/workflows/check.yml`, `.gitignore`, `.golangci.yml`,
|
||||||
|
`.prettierignore`, `.prettierrc`, `REPO_POLICIES.md` and the standard scripts
|
||||||
|
are its copies. pixa's own entries follow the standard part: `/bin`, `/data`
|
||||||
|
and the local config files in `.dockerignore`; Go's `*.log`, `*.out` and
|
||||||
|
`*.test`, `vendor/`, the binaries, `/data/`, `*.sqlite3` and the local config
|
||||||
|
files in `.gitignore`. The workflow keeps its `script/docker-smoke` step while
|
||||||
|
https://git.eeqj.de/sneak/pixa/issues/203 is open. The scripts pass a new
|
||||||
|
`CACHEBUST` build argument in place of `--no-cache`, and every `Dockerfile`
|
||||||
|
stage declares it first, so every `RUN` step still runs on every build while
|
||||||
|
Go's build cache, mounted on the steps that compile, is kept; both `go test`
|
||||||
|
runs keep `-count=1`. The lint and test builds write no image. The lint phase
|
||||||
|
runs golangci-lint v2.14.0, the test phase runs on the Debian Go image, and
|
||||||
|
the build stage marks `/src` safe for git. `script/bootstrap`, `script/fmt`
|
||||||
|
and `script/fmt-check` use the node on the `PATH` only when it has the pinned
|
||||||
|
major version. Of golangci-lint v2.14.0's two findings, an unused `nolint`
|
||||||
|
directive is removed, and gosec's new path check is marked as not applying to
|
||||||
|
the fixed config file locations. The agent guidance moves unchanged from
|
||||||
|
`CLAUDE.md` to a root `AGENTS.md`, where `REPO_POLICIES.md` keeps it.
|
||||||
|
- 2026-10-15 the request metrics record any method that is not a standard one as
|
||||||
|
`other` (closes #274): their `method` label was each request's method as sent,
|
||||||
|
and Go's HTTP server accepts any method name, so a client sending made-up
|
||||||
|
methods added series to the Prometheus registry that were never removed. The
|
||||||
|
label is now the request's method only for `GET`, `HEAD`, `POST`, `PUT`,
|
||||||
|
`PATCH`, `DELETE`, `OPTIONS`, `CONNECT` and `TRACE`. The metrics library reads
|
||||||
|
the method from the request it is handed, so the metrics middleware hands it a
|
||||||
|
copy of the request carrying the label; the handlers still see the method the
|
||||||
|
client sent.
|
||||||
|
- 2026-10-15 logging out ends every login session (closes #256): a login session
|
||||||
|
was a cookie with nothing kept on the server, so `/logout` removed it from one
|
||||||
|
browser and a copy of it kept working for its 30 days. Every login is made
|
||||||
|
with the one signing key, so logging out now ends them all: the session
|
||||||
|
manager keeps a random value, made with `crypto/rand` when the new `logout`
|
||||||
|
table has none and replaced by each logout, saves it there so that it outlasts
|
||||||
|
a restart, puts the value at login in each session, and refuses a session
|
||||||
|
carrying another value. The check for a logout compares no time, so a login
|
||||||
|
after a logout works whatever the host's clock does; a session's 30 days are
|
||||||
|
still checked against the clock. Losing the state directory ends every login
|
||||||
|
session, and no logout can make an ended one work again. Logging out is now
|
||||||
|
`POST /logout` from a form on the generator page, behind the CSRF check the
|
||||||
|
other forms have, so a link or image elsewhere on the same site cannot end the
|
||||||
|
sessions. A logout without a valid login session ends none and only removes
|
||||||
|
the cookie, as anyone can take a CSRF token from the login page. A logout that
|
||||||
|
cannot be saved answers 500. `session.NewManager`, which the tests use, keeps
|
||||||
|
the value in memory only; pixad uses `session.NewManagerWithDatabase`.
|
||||||
|
- 2026-10-15 a cached file left empty or partial is no longer served, or fetched
|
||||||
|
again on every request (closes #252). Source and transformed image files, and
|
||||||
|
each transformed image's `.meta` file, are synced to disk before they are
|
||||||
|
renamed into place, and their directory after; the `.meta` file is written
|
||||||
|
before the image, and a failed directory sync removes the renamed file. The
|
||||||
|
reconciliation pass, at startup and every five minutes, removes a cached
|
||||||
|
source or transformed image whose file is empty, or not the size the database
|
||||||
|
records for it, with its rows and a warning naming the file, so the next
|
||||||
|
request fetches the source or converts the image again; requests compare no
|
||||||
|
sizes. Storing a source over a file of another size replaces that file. A
|
||||||
|
source that fails to decode is not removed for that alone: a damaged upstream
|
||||||
|
image would then be fetched on every request.
|
||||||
- 2026-10-15 a transcode stops once its request's `downstream_timeout` has
|
- 2026-10-15 a transcode stops once its request's `downstream_timeout` has
|
||||||
ended, unless it has begun encoding (closes #243): the image processor looked
|
ended, unless it has begun encoding (closes #243): the image processor looked
|
||||||
at its context only while it waited for a processing slot, so an image was
|
at its context only while it waited for a processing slot, so an image was
|
||||||
|
|||||||
@@ -338,8 +338,7 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
|
|||||||
settingName(keyDBURL), errValueEmpty)
|
settingName(keyDBURL), errValueEmpty)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The driver sets the journal mode only through a _pragma parameter.
|
c.DBURL = defaultDBURL(c.StateDir)
|
||||||
c.DBURL = fmt.Sprintf("file:%s/state.sqlite3?_pragma=journal_mode(WAL)", c.StateDir)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if loader.err != nil {
|
if loader.err != nil {
|
||||||
@@ -354,6 +353,30 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
|
|||||||
return c, nil
|
return c, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// defaultDBURL returns the db_url used when db_url is omitted: the file
|
||||||
|
// state.sqlite3 in stateDir, in WAL mode, which the driver sets only
|
||||||
|
// through a _pragma parameter. String() percent-encodes any %, ? or # in
|
||||||
|
// the directory name, which SQLite decodes back, so the URL names exactly
|
||||||
|
// this file. OmitHost writes file:/var/lib/pixa/... instead of
|
||||||
|
// file:///var/lib/pixa/..., and keeps a relative stateDir from being read
|
||||||
|
// as a host name. The path written into the URL is stateDir followed by
|
||||||
|
// /state.sqlite3; when it starts with //, as it does for a stateDir of /
|
||||||
|
// or one starting with //, the empty host is kept, so the URL starts
|
||||||
|
// file://// and SQLite does not read the first name in the path as a
|
||||||
|
// host.
|
||||||
|
func defaultDBURL(stateDir string) string {
|
||||||
|
dbPath := stateDir + "/state.sqlite3"
|
||||||
|
|
||||||
|
dbURL := url.URL{
|
||||||
|
Scheme: "file",
|
||||||
|
OmitHost: !strings.HasPrefix(dbPath, "//"),
|
||||||
|
Path: dbPath,
|
||||||
|
RawQuery: "_pragma=journal_mode(WAL)",
|
||||||
|
}
|
||||||
|
|
||||||
|
return dbURL.String()
|
||||||
|
}
|
||||||
|
|
||||||
// validateKnownKeys rejects configuration files containing keys the
|
// validateKnownKeys rejects configuration files containing keys the
|
||||||
// application does not understand, so typos fail at startup instead of
|
// application does not understand, so typos fail at startup instead of
|
||||||
// being silently ignored, and rejects keys that are explicitly set to
|
// being silently ignored, and rejects keys that are explicitly set to
|
||||||
@@ -829,6 +852,7 @@ func loadConfigFile(log *slog.Logger, appName string) (*smartconfig.Config, erro
|
|||||||
// one whose path runs through a file, such as under a HOME of
|
// one whose path runs through a file, such as under a HOME of
|
||||||
// /dev/null. One that cannot be read or does not parse is a
|
// /dev/null. One that cannot be read or does not parse is a
|
||||||
// fatal startup error.
|
// fatal startup error.
|
||||||
|
//nolint:gosec // G703: a fixed config location; HOME is the operator's
|
||||||
_, statErr := os.Stat(cleanPath)
|
_, statErr := os.Stat(cleanPath)
|
||||||
if errors.Is(statErr, fs.ErrNotExist) || errors.Is(statErr, syscall.ENOTDIR) {
|
if errors.Is(statErr, fs.ErrNotExist) || errors.Is(statErr, syscall.ENOTDIR) {
|
||||||
continue
|
continue
|
||||||
|
|||||||
@@ -135,6 +135,104 @@ func TestDefaultDBURLOpensTheDatabaseInWALMode(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestDerivedDBURLOpensStateDirHoldingURLCharacters opens the db_url
|
||||||
|
// derived from a state_dir whose name holds %41, ? and #, and checks that
|
||||||
|
// the database is state.sqlite3 in that directory, in WAL mode. Put into
|
||||||
|
// a file: URL as it is, the name would lose its meaning: %41 is read as
|
||||||
|
// A, and the path ends at the ?, which also loses the journal mode.
|
||||||
|
func TestDerivedDBURLOpensStateDirHoldingURLCharacters(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
stateDir := filepath.Join(t.TempDir(), "a%41b?c#d")
|
||||||
|
|
||||||
|
err := os.Mkdir(stateDir, 0o750)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to create %q: %v", stateDir, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Quoted so YAML takes the whole name as it is.
|
||||||
|
c, err := configFromYAML(t, signingKeyLine+"state_dir: '"+stateDir+"'\n")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
|
||||||
|
|
||||||
|
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(stateDir, "state.sqlite3"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDerivedDBURLOpensStateDirStartingWithTwoSlashes opens the db_url
|
||||||
|
// derived from a state_dir that starts with //, and checks that the
|
||||||
|
// database is state.sqlite3 in that directory, in WAL mode. Written right
|
||||||
|
// after file:, such a name gives a URL starting file://, and SQLite reads
|
||||||
|
// the first directory name as a host and refuses to open it.
|
||||||
|
func TestDerivedDBURLOpensStateDirStartingWithTwoSlashes(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
stateDir := "/" + dir
|
||||||
|
|
||||||
|
c, err := configFromYAML(t, signingKeyLine+"state_dir: "+stateDir+"\n")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
|
||||||
|
|
||||||
|
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(dir, "state.sqlite3"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDerivedDBURLForRootStateDir checks the db_url derived from a
|
||||||
|
// state_dir of /. The path written into the URL is then
|
||||||
|
// //state.sqlite3, and written right after file: it gives a URL starting
|
||||||
|
// file://, where SQLite reads state.sqlite3 as a host and refuses to open
|
||||||
|
// it. The test only checks the URL, since it cannot write to /.
|
||||||
|
func TestDerivedDBURLForRootStateDir(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
c, err := configFromYAML(t, signingKeyLine+"state_dir: /\n")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("config with state_dir / should be valid, got: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("state_dir / gives db_url %q", c.DBURL)
|
||||||
|
|
||||||
|
wantDBURL := "file:////state.sqlite3?_pragma=journal_mode(WAL)"
|
||||||
|
if c.DBURL != wantDBURL {
|
||||||
|
t.Errorf("DBURL = %q, want %q", c.DBURL, wantDBURL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkDBURLOpensFileInWALMode opens dbURL with the SQLite driver pixad
|
||||||
|
// uses, and checks that the database is in WAL mode and is the file
|
||||||
|
// dbFile.
|
||||||
|
func checkDBURLOpensFileInWALMode(t *testing.T, dbURL, dbFile string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
db, err := sql.Open("sqlite", dbURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to open %q: %v", dbURL, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
|
||||||
|
var journalMode string
|
||||||
|
|
||||||
|
err = db.QueryRowContext(t.Context(), "PRAGMA journal_mode").Scan(&journalMode)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to read the journal mode of %q: %v", dbURL, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if journalMode != "wal" {
|
||||||
|
t.Errorf("journal mode of %q = %q, want wal", dbURL, journalMode)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = os.Stat(dbFile)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("opening %q did not create %q: %v", dbURL, dbFile, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExplicitValidValuesAreUsed(t *testing.T) {
|
func TestExplicitValidValuesAreUsed(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -176,3 +176,11 @@ CREATE TABLE IF NOT EXISTS cache_stats (
|
|||||||
last_updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
last_updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||||
);
|
);
|
||||||
INSERT OR IGNORE INTO cache_stats (id) VALUES (1);
|
INSERT OR IGNORE INTO cache_stats (id) VALUES (1);
|
||||||
|
|
||||||
|
-- A random value that a login session carries from its login. pixa makes it
|
||||||
|
-- when the table has no row, and each logout replaces it; a login session
|
||||||
|
-- carrying another value is refused, so logging out ends every login session.
|
||||||
|
CREATE TABLE IF NOT EXISTS logout (
|
||||||
|
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||||
|
logout_value TEXT NOT NULL
|
||||||
|
);
|
||||||
|
|||||||
@@ -9,7 +9,9 @@ import (
|
|||||||
"math"
|
"math"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/url"
|
"net/url"
|
||||||
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/pixa/internal/clientip"
|
"sneak.berlin/go/pixa/internal/clientip"
|
||||||
@@ -89,10 +91,18 @@ func (s *Handlers) handleLoginPost(w http.ResponseWriter, r *http.Request) {
|
|||||||
http.Redirect(w, r, "/", http.StatusSeeOther)
|
http.Redirect(w, r, "/", http.StatusSeeOther)
|
||||||
}
|
}
|
||||||
|
|
||||||
// HandleLogout clears the session and redirects to login.
|
// HandleLogout logs out, which ends every login session (see
|
||||||
|
// session.Manager.Logout), and redirects to login.
|
||||||
func (s *Handlers) HandleLogout() http.HandlerFunc {
|
func (s *Handlers) HandleLogout() http.HandlerFunc {
|
||||||
return func(w http.ResponseWriter, r *http.Request) {
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
s.sessMgr.ClearSession(w)
|
err := s.sessMgr.Logout(w, r)
|
||||||
|
if err != nil {
|
||||||
|
s.log.Error("failed to log out", "error", err)
|
||||||
|
http.Error(w, "Failed to log out", http.StatusInternalServerError)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
http.Redirect(w, r, "/", http.StatusSeeOther)
|
http.Redirect(w, r, "/", http.StatusSeeOther)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -117,7 +127,7 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
|||||||
sourceURL := r.FormValue("url")
|
sourceURL := r.FormValue("url")
|
||||||
|
|
||||||
// Validate source URL
|
// Validate source URL
|
||||||
parsed, err := url.Parse(sourceURL)
|
parsed, err := parseSourceURL(sourceURL)
|
||||||
if err != nil || parsed.Host == "" {
|
if err != nil || parsed.Host == "" {
|
||||||
s.renderGeneratorWithForm(w, r, "Invalid source URL", r.Form,
|
s.renderGeneratorWithForm(w, r, "Invalid source URL", r.Form,
|
||||||
http.StatusBadRequest)
|
http.StatusBadRequest)
|
||||||
@@ -125,6 +135,16 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// /v1/image/ refuses a ".." segment in the source path, and so does
|
||||||
|
// the generator. The path is decoded, so %2e%2e counts too
|
||||||
|
if slices.Contains(strings.Split(parsed.Path, "/"), "..") {
|
||||||
|
s.renderGeneratorWithForm(w, r,
|
||||||
|
"Source URL path cannot hold a .. segment", r.Form,
|
||||||
|
http.StatusBadRequest)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
payload, expiresAt, ttl, err := buildGeneratePayload(parsed, r.Form)
|
payload, expiresAt, ttl, err := buildGeneratePayload(parsed, r.Form)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.renderGeneratorWithForm(w, r, err.Error(), r.Form,
|
s.renderGeneratorWithForm(w, r, err.Error(), r.Form,
|
||||||
@@ -133,6 +153,19 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The image URL must fetch the source URL as it was written
|
||||||
|
sourceRequest := payload.ToImageRequest()
|
||||||
|
sourceRequest.AllowHTTP = s.config.AllowHTTP
|
||||||
|
fetchURL := sourceRequest.SourceURL()
|
||||||
|
|
||||||
|
if !fetchedAsWritten(parsed, fetchURL) {
|
||||||
|
s.renderGeneratorWithForm(w, r, "Source URL cannot be fetched as "+
|
||||||
|
"written: pixa would fetch "+fetchURL,
|
||||||
|
r.Form, http.StatusBadRequest)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
// Generate encrypted token
|
// Generate encrypted token
|
||||||
token, err := s.encGen.Generate(payload)
|
token, err := s.encGen.Generate(payload)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -165,6 +198,48 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// parseSourceURL parses the source URL entered on the generator page with its
|
||||||
|
// host lower-cased and an empty path written as "/", as /v1/image/ reads a
|
||||||
|
// source URL, so that each spelling of one source URL makes one cache key and
|
||||||
|
// one upstream request.
|
||||||
|
func parseSourceURL(sourceURL string) (*url.URL, error) {
|
||||||
|
parsed, err := url.Parse(sourceURL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
parsed.Host = strings.ToLower(parsed.Host)
|
||||||
|
|
||||||
|
if parsed.Path == "" {
|
||||||
|
parsed.Path = "/"
|
||||||
|
}
|
||||||
|
|
||||||
|
return parsed, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// fetchedAsWritten reports whether fetchURL, the URL pixa fetches for an
|
||||||
|
// image URL made from the source URL source, is source as it was written,
|
||||||
|
// apart from the scheme, which the token does not keep. It is not when source
|
||||||
|
// holds a user name and password, which the token does not keep either, or a
|
||||||
|
// "?" with no query after it. The token keeps the source path decoded and
|
||||||
|
// pixa fetches it percent-encoded again, so a path written another way, such
|
||||||
|
// as with %2F, is fetched as another path. url.URL keeps the path as written
|
||||||
|
// in RawPath when it is not the decoded path percent-encoded again.
|
||||||
|
// EscapedPath ignores RawPath when it also holds a character net/url encodes,
|
||||||
|
// such as a space, so RawPath is compared itself.
|
||||||
|
func fetchedAsWritten(source *url.URL, fetchURL string) bool {
|
||||||
|
writtenPath := source.RawPath
|
||||||
|
if writtenPath == "" {
|
||||||
|
writtenPath = source.EscapedPath()
|
||||||
|
}
|
||||||
|
|
||||||
|
fetched, err := url.Parse(fetchURL)
|
||||||
|
|
||||||
|
return err == nil && source.User == nil && !source.ForceQuery &&
|
||||||
|
fetched.EscapedPath() == writtenPath &&
|
||||||
|
fetched.RawQuery == source.RawQuery
|
||||||
|
}
|
||||||
|
|
||||||
// buildGeneratePayload parses the numeric form fields and assembles the
|
// buildGeneratePayload parses the numeric form fields and assembles the
|
||||||
// encrypted URL payload. ttl=0 means never expires (ExpiresAt stays 0). A
|
// encrypted URL payload. ttl=0 means never expires (ExpiresAt stays 0). A
|
||||||
// non-numeric or out-of-range width, height, quality or ttl, or an
|
// non-numeric or out-of-range width, height, quality or ttl, or an
|
||||||
|
|||||||
@@ -185,6 +185,98 @@ func TestHandleLogout_ClearsSessionCookie(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestHandleLogout_EndsCopiedSession verifies that after POST /logout a copy
|
||||||
|
// of the session cookie taken before it shows the login page, not the
|
||||||
|
// generator page.
|
||||||
|
func TestHandleLogout_EndsCopiedSession(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h, srv := newCSRFTestRouter(t)
|
||||||
|
|
||||||
|
copied := newSessionCookie(t, h)
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodPost, "/logout", nil)
|
||||||
|
req.AddCookie(copied)
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
h.HandleLogout().ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusSeeOther {
|
||||||
|
t.Fatalf("POST /logout status = %d, want %d",
|
||||||
|
rec.Code, http.StatusSeeOther)
|
||||||
|
}
|
||||||
|
|
||||||
|
req = httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
|
||||||
|
req.AddCookie(copied)
|
||||||
|
|
||||||
|
rec = httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
body := rec.Body.String()
|
||||||
|
t.Logf("GET / with the copied cookie: status %d, login page %v",
|
||||||
|
rec.Code, strings.Contains(body, loginForm))
|
||||||
|
|
||||||
|
if !strings.Contains(body, loginForm) || strings.Contains(body, generatorForm) {
|
||||||
|
t.Error("a cookie copied before /logout still opens the generator page")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleLogout_SaveFails_Answers500 verifies that when the logout cannot
|
||||||
|
// be saved in the database, POST /logout answers 500, leaves the session
|
||||||
|
// cookie in place and ends no session.
|
||||||
|
func TestHandleLogout_SaveFails_Answers500(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h, srv := newCSRFTestRouter(t)
|
||||||
|
|
||||||
|
db := setupTestDB(t)
|
||||||
|
|
||||||
|
sessMgr, err := session.NewManagerWithDatabase(
|
||||||
|
t.Context(), testSigningKey, db)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("session.NewManagerWithDatabase() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
h.sessMgr = sessMgr
|
||||||
|
cookie := newSessionCookie(t, h)
|
||||||
|
|
||||||
|
// Saving the logout fails on a closed database.
|
||||||
|
err = db.Close()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("closing the database: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodPost, "/logout", nil)
|
||||||
|
req.AddCookie(cookie)
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
h.HandleLogout().ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
t.Logf("POST /logout: status %d, Set-Cookie %q",
|
||||||
|
rec.Code, rec.Header().Get("Set-Cookie"))
|
||||||
|
|
||||||
|
if rec.Code != http.StatusInternalServerError {
|
||||||
|
t.Errorf("POST /logout status = %d, want %d",
|
||||||
|
rec.Code, http.StatusInternalServerError)
|
||||||
|
}
|
||||||
|
|
||||||
|
if c := findSessionCookie(rec); c != nil {
|
||||||
|
t.Errorf("a logout that failed set the session cookie: %s", c)
|
||||||
|
}
|
||||||
|
|
||||||
|
req = httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
|
||||||
|
req.AddCookie(cookie)
|
||||||
|
|
||||||
|
rec = httptest.NewRecorder()
|
||||||
|
srv.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
if !strings.Contains(rec.Body.String(), generatorForm) {
|
||||||
|
t.Error("a logout that failed ended the session")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestGeneratePost_NoSession_RedirectsToLogin verifies that POST /generate
|
// TestGeneratePost_NoSession_RedirectsToLogin verifies that POST /generate
|
||||||
// with a valid CSRF token but no login session answers 303 to / and makes no
|
// with a valid CSRF token but no login session answers 303 to / and makes no
|
||||||
// URL.
|
// URL.
|
||||||
|
|||||||
@@ -164,7 +164,10 @@ func (s *Handlers) initImageService() error {
|
|||||||
|
|
||||||
// Initialize session manager (signing key is validated at config load
|
// Initialize session manager (signing key is validated at config load
|
||||||
// time). Session cookies are always Secure/HttpOnly/SameSite=Strict.
|
// time). Session cookies are always Secure/HttpOnly/SameSite=Strict.
|
||||||
sessMgr, err := session.NewManager(s.config.SigningKey)
|
// The random value that each logout replaces is saved in the database,
|
||||||
|
// so it outlasts a restart.
|
||||||
|
sessMgr, err := session.NewManagerWithDatabase(
|
||||||
|
context.Background(), s.config.SigningKey, s.db.DB())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,7 +9,6 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/go-chi/chi/v5"
|
|
||||||
"github.com/go-chi/chi/v5/middleware"
|
"github.com/go-chi/chi/v5/middleware"
|
||||||
"sneak.berlin/go/pixa/internal/encurl"
|
"sneak.berlin/go/pixa/internal/encurl"
|
||||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||||
@@ -80,20 +79,21 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseImageRequest parses the wildcard path and query parameters into
|
// parseImageRequest parses the image URL's path and query parameters into
|
||||||
// an ImageRequest. On invalid input it writes an error response and
|
// an ImageRequest. On invalid input it writes an error response and
|
||||||
// returns false.
|
// returns false.
|
||||||
func (s *Handlers) parseImageRequest(
|
func (s *Handlers) parseImageRequest(
|
||||||
w http.ResponseWriter, r *http.Request,
|
w http.ResponseWriter, r *http.Request,
|
||||||
) (*imgcache.ImageRequest, bool) {
|
) (*imgcache.ImageRequest, bool) {
|
||||||
// Get the wildcard path from chi
|
// The path is parsed decoded once, as r.URL.Path holds it, and not from
|
||||||
pathParam := chi.URLParam(r, "*")
|
// chi's wildcard: chi routes on the path as the client wrote it whenever
|
||||||
|
// that is not Go's own encoding of it, so its wildcard would still hold
|
||||||
// Parse the URL path
|
// the %3F before a source query when, say, a "/" in the query is written
|
||||||
parsed, err := imgcache.ParseImagePath(pathParam)
|
// as %2F
|
||||||
|
parsed, err := imgcache.ParseImageURL(r.URL.Path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.log.Warn("failed to parse image URL",
|
s.log.Warn("failed to parse image URL",
|
||||||
"path", pathParam,
|
"path", r.URL.Path,
|
||||||
"error", err,
|
"error", err,
|
||||||
)
|
)
|
||||||
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
|
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
|
||||||
|
|||||||
@@ -264,3 +264,127 @@ func TestHandleImage_AllowlistOrSignature(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestHandleImage_InvalidHost_Returns400 sends image URLs whose host no URL
|
||||||
|
// host can be, and checks that each is refused with 400 naming the host as
|
||||||
|
// decoded: allowlistedHost with a character after it that a URL host cannot
|
||||||
|
// hold, percent-encoded, an IPv6 address without brackets, and brackets
|
||||||
|
// around something that is not an IPv6 address. A "?" or "#" after
|
||||||
|
// allowlistedHost would otherwise end the host when the source URL is read,
|
||||||
|
// so the request would pass the allowlist and be fetched and cached under a
|
||||||
|
// host that is not the one fetched.
|
||||||
|
func TestHandleImage_InvalidHost_Returns400(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
photos := newPhotoFetcher(t, allowlistedHost)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
writtenHost string
|
||||||
|
decodedHost string
|
||||||
|
}{
|
||||||
|
{"? written as %3F", allowlistedHost + "%3Fx=1", allowlistedHost + "?x=1"},
|
||||||
|
{"# written as %23", allowlistedHost + "%23x", allowlistedHost + "#x"},
|
||||||
|
{"% written as %25", allowlistedHost + "%253F", allowlistedHost + "%3F"},
|
||||||
|
{`\ written as %5C`, allowlistedHost + "%5Cx", allowlistedHost + `\x`},
|
||||||
|
{"space written as %20", allowlistedHost + "%20", allowlistedHost + " "},
|
||||||
|
{"IPv6 address without brackets", "2001:db8::1", "2001:db8::1"},
|
||||||
|
{"IPv4 address in brackets", "[203.0.113.5]", "[203.0.113.5]"},
|
||||||
|
{"no address in brackets", "[abc]", "[abc]"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
rec := sendGet(t, newImageRoute(t, photos), photoURL(tt.writtenHost))
|
||||||
|
checkErrorBody(t, rec, http.StatusBadRequest,
|
||||||
|
fmt.Sprintf("invalid image URL: invalid source host: %q",
|
||||||
|
tt.decodedHost))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleImage_HostForms_Served checks that the image route serves, with
|
||||||
|
// a valid signature, the image at photoPath on a host name, an IPv4 address,
|
||||||
|
// an IPv6 address in brackets, and a host followed by ":" with or without a
|
||||||
|
// port.
|
||||||
|
func TestHandleImage_HostForms_Served(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
hosts := []string{
|
||||||
|
allowlistedHost,
|
||||||
|
"img_1.example.com",
|
||||||
|
"203.0.113.5",
|
||||||
|
"[2001:db8::1]",
|
||||||
|
allowlistedHost + ":8443",
|
||||||
|
"[2001:db8::1]:8443",
|
||||||
|
allowlistedHost + ":",
|
||||||
|
}
|
||||||
|
|
||||||
|
photos := newPhotoFetcher(t, hosts...)
|
||||||
|
exp := time.Now().Add(time.Hour)
|
||||||
|
|
||||||
|
for _, host := range hosts {
|
||||||
|
t.Run(host, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
sig := photoSignature(testSigningKey, host, exp)
|
||||||
|
|
||||||
|
rec := sendGet(t, newImageRoute(t, photos),
|
||||||
|
photoURLWithSig(host, sig, exp))
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Errorf("status = %d, want %d: %s",
|
||||||
|
rec.Code, http.StatusOK, rec.Body)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleImage_HashInSourcePathOrQuery_Returns400 sends image URLs whose
|
||||||
|
// source path or source query holds a "#", written as %23, and checks that
|
||||||
|
// each is refused with 400, the one signed for that source path included. The
|
||||||
|
// source URL would otherwise end at the "#" when it is fetched, so each text
|
||||||
|
// after it would fetch the image at photoPath and cache it under keys of its
|
||||||
|
// own.
|
||||||
|
func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
photos := newPhotoFetcher(t, allowlistedHost, signedHost)
|
||||||
|
exp := time.Now().Add(time.Hour)
|
||||||
|
|
||||||
|
sig := signature.New(testSigningKey).Sign(&signature.Request{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: photoPath + "#x",
|
||||||
|
Width: 50,
|
||||||
|
Height: 50,
|
||||||
|
Format: string(imgcache.FormatJPEG),
|
||||||
|
Quality: 85,
|
||||||
|
FitMode: string(imgcache.FitCover),
|
||||||
|
Expires: exp,
|
||||||
|
})
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
target string
|
||||||
|
}{
|
||||||
|
{"# in the source path",
|
||||||
|
"/v1/image/" + allowlistedHost + photoPath + "%23x/50x50.jpeg"},
|
||||||
|
{"# in the source query",
|
||||||
|
"/v1/image/" + allowlistedHost + photoPath + "%3Fv=1%23x/50x50.jpeg"},
|
||||||
|
{"# in the source path, signed",
|
||||||
|
fmt.Sprintf("/v1/image/%s%s%%23x/50x50.jpeg?sig=%s&exp=%d",
|
||||||
|
signedHost, photoPath, sig, exp.Unix())},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
rec := sendGet(t, newImageRoute(t, photos), tt.target)
|
||||||
|
checkErrorBody(t, rec, http.StatusBadRequest,
|
||||||
|
`invalid image URL: invalid image path: `+
|
||||||
|
`the source path or query holds a "#"`)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -169,3 +169,166 @@ func TestHandleImage_InvalidExp_Returns400(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies sends URLs built
|
||||||
|
// by the service's signed-URL generator for source URLs with a query to the
|
||||||
|
// route, through a chi router: chi routes on the path as the client wrote it
|
||||||
|
// whenever that is not Go's own encoding of it. A 502 means the signature
|
||||||
|
// passed, so the route read the source path and query that were signed.
|
||||||
|
func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
query string
|
||||||
|
}{
|
||||||
|
{"slash and colon", "url=https://origin.example.com/cat.jpg"},
|
||||||
|
{"semicolon", "a=1;b=2"},
|
||||||
|
{"comma", "crop=0,0,50,50"},
|
||||||
|
{"equals and ampersand", "a=1&b=2"},
|
||||||
|
{"percent-escape", "name=my%20cat"},
|
||||||
|
{"question mark", "next=/a?b=1"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
fix := setupTestHandler(t)
|
||||||
|
|
||||||
|
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: photoPath,
|
||||||
|
SourceQuery: tt.query,
|
||||||
|
Size: imgcache.Size{Width: 50, Height: 50},
|
||||||
|
Format: imgcache.FormatJPEG,
|
||||||
|
}, time.Hour)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
status := getImage(t, fix, signedURL)
|
||||||
|
if status != http.StatusBadGateway {
|
||||||
|
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleImage_SourceQueryEncodedAnyWay signs one source URL with a query
|
||||||
|
// and writes it into the image URL in ways the README allows: the "?" before
|
||||||
|
// the query and each "%" in it percent-encoded, and every other character as
|
||||||
|
// itself or percent-encoded. A 502 means the signature passed.
|
||||||
|
func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
expires := time.Now().Add(time.Hour)
|
||||||
|
signer := signature.New("test-signing-key-must-be-32-chars")
|
||||||
|
sig := signer.Sign(&signature.Request{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: photoPath,
|
||||||
|
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
|
||||||
|
Width: 50,
|
||||||
|
Height: 50,
|
||||||
|
Format: string(imgcache.FormatJPEG),
|
||||||
|
Quality: 85,
|
||||||
|
FitMode: string(imgcache.FitCover),
|
||||||
|
Expires: expires,
|
||||||
|
})
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
writtenQuery string
|
||||||
|
}{
|
||||||
|
{"only ? and % encoded",
|
||||||
|
"%3Furl=https://origin.example.com/my%2520cat.jpg"},
|
||||||
|
{"every reserved character encoded",
|
||||||
|
"%3Furl%3Dhttps%3A%2F%2Forigin.example.com%2Fmy%2520cat.jpg"},
|
||||||
|
{"lower-case hex digits",
|
||||||
|
"%3furl=https://origin.example.com/my%2520cat.jpg"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
target := fmt.Sprintf("/v1/image/%s%s%s/50x50.jpeg?sig=%s&exp=%d",
|
||||||
|
signedHost, photoPath, tt.writtenQuery, sig, expires.Unix())
|
||||||
|
|
||||||
|
status := getImage(t, setupTestHandler(t), target)
|
||||||
|
if status != http.StatusBadGateway {
|
||||||
|
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleImage_SourceURLHoldingPercent25Verifies signs a source path and a
|
||||||
|
// source query that each hold %25, the percent-escape of a literal "%", and
|
||||||
|
// sends each to the route written as the README says, with that "%" written as
|
||||||
|
// %25, and as GenerateSignedURL writes it. A 502 means the signature passed.
|
||||||
|
func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
sourcePath string
|
||||||
|
sourceQuery string
|
||||||
|
// writtenPathAndQuery is the source path and query as the README says
|
||||||
|
// to write them into the image URL
|
||||||
|
writtenPathAndQuery string
|
||||||
|
}{
|
||||||
|
{"path", "/sale/50%25off.jpg", "", "/sale/50%2525off.jpg"},
|
||||||
|
{"query", photoPath, "discount=50%25", photoPath + "%3Fdiscount=50%2525"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name+" written by the README rule", func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
expires := time.Now().Add(time.Hour)
|
||||||
|
signer := signature.New("test-signing-key-must-be-32-chars")
|
||||||
|
sig := signer.Sign(&signature.Request{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: tt.sourcePath,
|
||||||
|
SourceQuery: tt.sourceQuery,
|
||||||
|
Width: 50,
|
||||||
|
Height: 50,
|
||||||
|
Format: string(imgcache.FormatJPEG),
|
||||||
|
Quality: 85,
|
||||||
|
FitMode: string(imgcache.FitCover),
|
||||||
|
Expires: expires,
|
||||||
|
})
|
||||||
|
|
||||||
|
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
|
||||||
|
signedHost, tt.writtenPathAndQuery, sig, expires.Unix())
|
||||||
|
|
||||||
|
status := getImage(t, setupTestHandler(t), target)
|
||||||
|
if status != http.StatusBadGateway {
|
||||||
|
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run(tt.name+" written by GenerateSignedURL", func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
fix := setupTestHandler(t)
|
||||||
|
|
||||||
|
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: tt.sourcePath,
|
||||||
|
SourceQuery: tt.sourceQuery,
|
||||||
|
Size: imgcache.Size{Width: 50, Height: 50},
|
||||||
|
Format: imgcache.FormatJPEG,
|
||||||
|
}, time.Hour)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
status := getImage(t, fix, signedURL)
|
||||||
|
if status != http.StatusBadGateway {
|
||||||
|
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -59,15 +59,17 @@ const (
|
|||||||
// builtinBlockedPrefixes are internal or special-use ranges that Go's
|
// builtinBlockedPrefixes are internal or special-use ranges that Go's
|
||||||
// net.IP predicates (IsPrivate, IsLinkLocalUnicast, and the like) do not
|
// net.IP predicates (IsPrivate, IsLinkLocalUnicast, and the like) do not
|
||||||
// already cover. They are always blocked, in addition to any
|
// already cover. They are always blocked, in addition to any
|
||||||
// operator-supplied networks. IPv4-mapped IPv6 addresses are unmapped
|
// operator-supplied networks. IPv4-mapped and IPv4-compatible IPv6
|
||||||
// before matching, so these IPv4 ranges are caught in both forms.
|
// addresses are read as the IPv4 address they hold before matching, so
|
||||||
|
// these IPv4 ranges are caught in all three forms.
|
||||||
//
|
//
|
||||||
//nolint:gochecknoglobals // immutable built-in blocklist
|
//nolint:gochecknoglobals // immutable built-in blocklist
|
||||||
var builtinBlockedPrefixes = []netip.Prefix{
|
var builtinBlockedPrefixes = []netip.Prefix{
|
||||||
netip.MustParsePrefix("100.64.0.0/10"), // RFC 6598 CGNAT / carrier-grade NAT
|
netip.MustParsePrefix("100.64.0.0/10"), // RFC 6598 CGNAT / carrier-grade NAT
|
||||||
netip.MustParsePrefix("192.0.0.0/24"), // RFC 6890 IETF protocol assignments
|
netip.MustParsePrefix("192.0.0.0/24"), // RFC 6890 IETF protocol assignments
|
||||||
netip.MustParsePrefix("198.18.0.0/15"), // RFC 2544 benchmarking range
|
netip.MustParsePrefix("198.18.0.0/15"), // RFC 2544 benchmarking range
|
||||||
netip.MustParsePrefix("64:ff9b::/96"), // RFC 6052 NAT64 (maps onto IPv4)
|
netip.MustParsePrefix("64:ff9b::/96"), // RFC 6052 NAT64 (maps onto IPv4)
|
||||||
|
netip.MustParsePrefix("64:ff9b:1::/48"), // RFC 8215 local-use NAT64 (maps onto IPv4)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fetcher errors.
|
// Fetcher errors.
|
||||||
@@ -473,16 +475,16 @@ func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
|
|||||||
return ErrUnsupportedScheme
|
return ErrUnsupportedScheme
|
||||||
}
|
}
|
||||||
|
|
||||||
// Parse to extract host
|
parsedURL, err := neturl.Parse(rawURL)
|
||||||
host := extractHost(rawURL)
|
if err != nil {
|
||||||
if host == "" {
|
return fmt.Errorf("%w: %w", ErrInvalidHost, err)
|
||||||
return ErrInvalidHost
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Remove port if present
|
// Hostname drops any port, and the brackets of an IPv6 address whether
|
||||||
h, _, err := net.SplitHostPort(host)
|
// or not a port follows them
|
||||||
if err == nil {
|
host := parsedURL.Hostname()
|
||||||
host = h
|
if host == "" {
|
||||||
|
return ErrInvalidHost
|
||||||
}
|
}
|
||||||
|
|
||||||
// Block obvious localhost patterns
|
// Block obvious localhost patterns
|
||||||
@@ -543,6 +545,9 @@ func isPrivateIP(ip net.IP) bool {
|
|||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// An IPv4-compatible IPv6 address is judged as the IPv4 address it holds
|
||||||
|
ip = ipv4CompatibleToIPv4(ip)
|
||||||
|
|
||||||
// Check for loopback
|
// Check for loopback
|
||||||
if ip.IsLoopback() {
|
if ip.IsLoopback() {
|
||||||
return true
|
return true
|
||||||
@@ -594,7 +599,12 @@ func isPrivateIP(ip net.IP) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// isBlockedIP reports whether ip is refused, either by the built-in
|
// isBlockedIP reports whether ip is refused, either by the built-in
|
||||||
// internal-range check or by one of the operator-supplied prefixes.
|
// internal-range check or by one of the operator-supplied prefixes. An
|
||||||
|
// IPv4-compatible address (in ::/96) is refused when a prefix contains the
|
||||||
|
// address as written or the IPv4 address it holds. An IPv4-mapped address (in
|
||||||
|
// ::ffff:0:0/96) is matched only as the IPv4 address it holds: Go keeps a
|
||||||
|
// plain IPv4 address in that form too, and an IPv6 prefix such as ::/0 must
|
||||||
|
// not refuse every IPv4 address.
|
||||||
func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
|
func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
|
||||||
if isPrivateIP(ip) {
|
if isPrivateIP(ip) {
|
||||||
return true
|
return true
|
||||||
@@ -607,11 +617,28 @@ func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
|
|||||||
|
|
||||||
addr = addr.Unmap()
|
addr = addr.Unmap()
|
||||||
|
|
||||||
|
// ip read as IPv4 where it is or holds an IPv4 address, otherwise the
|
||||||
|
// zero netip.Addr, which no prefix contains. Matching asIPv4 as well as
|
||||||
|
// addr refuses more only for an IPv4-compatible address.
|
||||||
|
asIPv4, _ := netip.AddrFromSlice(ipv4CompatibleToIPv4(ip).To4())
|
||||||
|
|
||||||
return slices.ContainsFunc(blocked, func(prefix netip.Prefix) bool {
|
return slices.ContainsFunc(blocked, func(prefix netip.Prefix) bool {
|
||||||
return prefix.Contains(addr)
|
return prefix.Contains(addr) || prefix.Contains(asIPv4)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ipv4CompatibleToIPv4 returns the IPv4 address that an IPv4-compatible IPv6
|
||||||
|
// address (in ::/96, such as ::127.0.0.1) holds in its last four bytes, and
|
||||||
|
// any other address unchanged.
|
||||||
|
func ipv4CompatibleToIPv4(ip net.IP) net.IP {
|
||||||
|
addr, ok := netip.AddrFromSlice(ip)
|
||||||
|
if ok && netip.MustParsePrefix("::/96").Contains(addr) {
|
||||||
|
return ip[12:]
|
||||||
|
}
|
||||||
|
|
||||||
|
return ip
|
||||||
|
}
|
||||||
|
|
||||||
// ssrfSafeDialer validates IP addresses against the built-in blocked ranges
|
// ssrfSafeDialer validates IP addresses against the built-in blocked ranges
|
||||||
// before connecting. New wraps dialSSRFSafe with the operator-supplied
|
// before connecting. New wraps dialSSRFSafe with the operator-supplied
|
||||||
// blocklist; this entry point enforces the built-in ranges alone.
|
// blocklist; this entry point enforces the built-in ranges alone.
|
||||||
|
|||||||
@@ -186,6 +186,44 @@ func TestIsPrivateIP(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestIsPrivateIPJudgesEmbeddedIPv4Address checks IPv6 addresses that hold an
|
||||||
|
// IPv4 address in their last four bytes: the IPv4-compatible form (::/96) and
|
||||||
|
// the NAT64 form (64:ff9b::/96). One holding a loopback, private or
|
||||||
|
// link-local IPv4 address is refused as that address is, and one holding a
|
||||||
|
// public IPv4 address in the IPv4-compatible form passes.
|
||||||
|
func TestIsPrivateIPJudgesEmbeddedIPv4Address(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
ip string
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{"::127.0.0.1", true}, // loopback
|
||||||
|
{"::10.0.0.1", true}, // private
|
||||||
|
{"::169.254.169.254", true}, // link-local
|
||||||
|
{"64:ff9b::7f00:1", true}, // NAT64 form of 127.0.0.1
|
||||||
|
{"::8.8.8.8", false}, // public
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.ip, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ip := net.ParseIP(tc.ip)
|
||||||
|
if ip == nil {
|
||||||
|
t.Fatalf("failed to parse IP %q", tc.ip)
|
||||||
|
}
|
||||||
|
|
||||||
|
got := isPrivateIP(ip)
|
||||||
|
t.Logf("isPrivateIP(%s) = %v", ip, got)
|
||||||
|
|
||||||
|
if got != tc.want {
|
||||||
|
t.Errorf("isPrivateIP(%q) = %v, want %v", tc.ip, got, tc.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
|
func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -50,3 +50,69 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestImageProcessor_ComputedSideIsAtLeastOne verifies that, for a size with
|
||||||
|
// one side 0, the side computed from the input's proportions is 1 where it
|
||||||
|
// would round down to 0, and that Process makes that size in every fit mode
|
||||||
|
// but FitOutside, which scales the input to cover the size and so can make
|
||||||
|
// it longer. Each case checks the size targetDimensions gives before it
|
||||||
|
// calls Process: with that side at 0, an 8192x1 input at 1300x0 makes
|
||||||
|
// libvips ask for about 39 GB with FitCover.
|
||||||
|
func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
inputWidth, inputHeight int
|
||||||
|
size Size
|
||||||
|
wantWidth, wantHeight int
|
||||||
|
}{
|
||||||
|
{"8192x1 at 100x0", 8192, 1, Size{Width: 100}, 100, 1},
|
||||||
|
{"8192x1 at 1300x0", 8192, 1, Size{Width: 1300}, 1300, 1},
|
||||||
|
{"8192x2 at 100x0", 8192, 2, Size{Width: 100}, 100, 1},
|
||||||
|
{"1x8192 at 0x100", 1, 8192, Size{Height: 100}, 1, 100},
|
||||||
|
{"1x8192 at 0x1300", 1, 8192, Size{Height: 1300}, 1, 1300},
|
||||||
|
{"2x8192 at 0x100", 2, 8192, Size{Height: 100}, 1, 100},
|
||||||
|
}
|
||||||
|
|
||||||
|
fits := []FitMode{FitCover, FitContain, FitFill, FitInside, FitOutside}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
for _, fit := range fits {
|
||||||
|
t.Run(tt.name+" "+string(fit), func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
width, height, err := targetDimensions(tt.size,
|
||||||
|
tt.inputWidth, tt.inputHeight)
|
||||||
|
if err != nil || width != tt.wantWidth || height != tt.wantHeight {
|
||||||
|
t.Fatalf("targetDimensions() = %dx%d, %v, want %dx%d",
|
||||||
|
width, height, err, tt.wantWidth, tt.wantHeight)
|
||||||
|
}
|
||||||
|
|
||||||
|
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||||
|
|
||||||
|
result, err := New(Params{}).Process(t.Context(),
|
||||||
|
bytes.NewReader(input),
|
||||||
|
&Request{Size: tt.size, Format: FormatJPEG, FitMode: fit})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Process() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = result.Content.Close() }()
|
||||||
|
|
||||||
|
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
||||||
|
|
||||||
|
ok := result.Width == tt.wantWidth && result.Height == tt.wantHeight
|
||||||
|
if fit == FitOutside {
|
||||||
|
ok = result.Width >= tt.wantWidth && result.Height >= tt.wantHeight
|
||||||
|
}
|
||||||
|
|
||||||
|
if !ok {
|
||||||
|
t.Errorf("Process() made %dx%d, want %dx%d (at least, for %s)",
|
||||||
|
result.Width, result.Height, tt.wantWidth, tt.wantHeight,
|
||||||
|
FitOutside)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,268 @@
|
|||||||
|
package imageprocessor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"image"
|
||||||
|
"image/color"
|
||||||
|
"image/png"
|
||||||
|
"math"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestImageProcessor_FitCoverServesThinSource verifies that FitCover serves
|
||||||
|
// a thin source at the requested size. An 8192x1 source scaled to cover
|
||||||
|
// 8192x8192 is 8192 times as long as the output on its long side. The centre
|
||||||
|
// of an 8192x1 source with the proportions of 1x8192 is 1/8192 of a pixel
|
||||||
|
// wide. Each case is also turned on its side.
|
||||||
|
func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
inputWidth, inputHeight int
|
||||||
|
size Size
|
||||||
|
}{
|
||||||
|
{"8192x1 at 8192x8192", 8192, 1, Size{Width: 8192, Height: 8192}},
|
||||||
|
{"8192x1 at 1x8192", 8192, 1, Size{Width: 1, Height: 8192}},
|
||||||
|
{"1x8192 at 8192x8192", 1, 8192, Size{Width: 8192, Height: 8192}},
|
||||||
|
{"1x8192 at 8192x1", 1, 8192, Size{Width: 8192, Height: 1}},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||||
|
|
||||||
|
result, err := New(Params{}).Process(t.Context(),
|
||||||
|
bytes.NewReader(input),
|
||||||
|
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitCover})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Process() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = result.Content.Close() }()
|
||||||
|
|
||||||
|
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
||||||
|
|
||||||
|
if result.Width != tt.size.Width || result.Height != tt.size.Height {
|
||||||
|
t.Errorf("Process() made %dx%d, want %dx%d",
|
||||||
|
result.Width, result.Height, tt.size.Width, tt.size.Height)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// createBandsPNG creates a PNG of three equal bands, red, green and blue,
|
||||||
|
// side by side when it is wider than tall and top to bottom otherwise.
|
||||||
|
func createBandsPNG(t *testing.T, width, height int) []byte {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
bands := []color.RGBA{
|
||||||
|
{R: 255, A: 255},
|
||||||
|
{G: 255, A: 255},
|
||||||
|
{B: 255, A: 255},
|
||||||
|
}
|
||||||
|
|
||||||
|
img := image.NewRGBA(image.Rect(0, 0, width, height))
|
||||||
|
for y := range height {
|
||||||
|
for x := range width {
|
||||||
|
band := 3 * y / height
|
||||||
|
if width > height {
|
||||||
|
band = 3 * x / width
|
||||||
|
}
|
||||||
|
|
||||||
|
img.Set(x, y, bands[band])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return encodePNG(t, img)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestImageProcessor_FitCoverCropsTheCentre verifies that FitCover keeps the
|
||||||
|
// centre of the source: a 300x100 source of red, green and blue bands side
|
||||||
|
// by side, or a 100x300 one with them top to bottom, comes out at 50x50 as
|
||||||
|
// its green band.
|
||||||
|
func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
inputWidth, inputHeight int
|
||||||
|
}{
|
||||||
|
{"bands side by side", 300, 100},
|
||||||
|
{"bands top to bottom", 100, 300},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
input := createBandsPNG(t, tt.inputWidth, tt.inputHeight)
|
||||||
|
|
||||||
|
result, err := New(Params{}).Process(t.Context(),
|
||||||
|
bytes.NewReader(input),
|
||||||
|
&Request{
|
||||||
|
Size: Size{Width: 50, Height: 50},
|
||||||
|
Format: FormatPNG,
|
||||||
|
FitMode: FitCover,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Process() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = result.Content.Close() }()
|
||||||
|
|
||||||
|
output, err := png.Decode(result.Content)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to decode output: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
bounds := output.Bounds()
|
||||||
|
t.Logf("Process() made %dx%d", bounds.Dx(), bounds.Dy())
|
||||||
|
|
||||||
|
if bounds.Dx() != 50 || bounds.Dy() != 50 {
|
||||||
|
t.Fatalf("Process() made %dx%d, want 50x50",
|
||||||
|
bounds.Dx(), bounds.Dy())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Resampling may blend a little of the bands beside the
|
||||||
|
// green one into the output's edges.
|
||||||
|
for y := bounds.Min.Y; y < bounds.Max.Y; y++ {
|
||||||
|
for x := bounds.Min.X; x < bounds.Max.X; x++ {
|
||||||
|
r, g, b, _ := output.At(x, y).RGBA()
|
||||||
|
if r>>8 > 64 || g>>8 < 192 || b>>8 > 64 {
|
||||||
|
t.Fatalf("output pixel %d,%d is %d,%d,%d, want green",
|
||||||
|
x, y, r>>8, g>>8, b>>8)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The greys of createSquarePNG, and the grey halfway between them.
|
||||||
|
const (
|
||||||
|
backgroundGrey = 192
|
||||||
|
squareGrey = 64
|
||||||
|
halfwayGrey = (backgroundGrey + squareGrey) / 2
|
||||||
|
)
|
||||||
|
|
||||||
|
// createSquarePNG creates a grey PNG side pixels square with a darker grey
|
||||||
|
// square squareSide pixels wide in its centre.
|
||||||
|
func createSquarePNG(t *testing.T, side, squareSide int) []byte {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
first := (side - squareSide) / 2
|
||||||
|
last := first + squareSide - 1
|
||||||
|
|
||||||
|
img := image.NewGray(image.Rect(0, 0, side, side))
|
||||||
|
for y := range side {
|
||||||
|
for x := range side {
|
||||||
|
grey := uint8(backgroundGrey)
|
||||||
|
if x >= first && x <= last && y >= first && y <= last {
|
||||||
|
grey = squareGrey
|
||||||
|
}
|
||||||
|
|
||||||
|
img.SetGray(x, y, color.Gray{Y: grey})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return encodePNG(t, img)
|
||||||
|
}
|
||||||
|
|
||||||
|
// squareLength returns how long the square is along greys, the greys of a
|
||||||
|
// row or column of the output that crosses it: the distance between the
|
||||||
|
// first and the last point where greys passes halfwayGrey, or 0 when there
|
||||||
|
// is no such point. Resampling blurs the square's edges over several
|
||||||
|
// pixels, and each point is placed between the two pixels on either side
|
||||||
|
// of it in proportion to their greys, so the length is measured to a
|
||||||
|
// fraction of a pixel.
|
||||||
|
func squareLength(greys []float64) float64 {
|
||||||
|
first, last := -1.0, -1.0
|
||||||
|
|
||||||
|
for i := 1; i < len(greys); i++ {
|
||||||
|
before, after := greys[i-1], greys[i]
|
||||||
|
if (before > halfwayGrey) == (after > halfwayGrey) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
point := float64(i-1) + (before-halfwayGrey)/(before-after)
|
||||||
|
if first < 0 {
|
||||||
|
first = point
|
||||||
|
}
|
||||||
|
|
||||||
|
last = point
|
||||||
|
}
|
||||||
|
|
||||||
|
return last - first
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestImageProcessor_FitCoverKeepsProportions verifies that FitCover scales
|
||||||
|
// both sides of the source by the same amount: a square in the centre of
|
||||||
|
// the source comes out square. Its width and height are measured along the
|
||||||
|
// output's middle row and column.
|
||||||
|
func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
side, squareSide int
|
||||||
|
size Size
|
||||||
|
}{
|
||||||
|
{"16x16 at 100x60", 16, 8, Size{Width: 100, Height: 60}},
|
||||||
|
{"16x16 at 60x100", 16, 8, Size{Width: 60, Height: 100}},
|
||||||
|
{"10x10 at 100x35", 10, 2, Size{Width: 100, Height: 35}},
|
||||||
|
{"10x10 at 35x100", 10, 2, Size{Width: 35, Height: 100}},
|
||||||
|
{"64x64 at 300x100", 64, 16, Size{Width: 300, Height: 100}},
|
||||||
|
{"64x64 at 100x300", 64, 16, Size{Width: 100, Height: 300}},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
input := createSquarePNG(t, tt.side, tt.squareSide)
|
||||||
|
|
||||||
|
result, err := New(Params{}).Process(t.Context(),
|
||||||
|
bytes.NewReader(input),
|
||||||
|
&Request{Size: tt.size, Format: FormatPNG, FitMode: FitCover})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Process() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = result.Content.Close() }()
|
||||||
|
|
||||||
|
output, err := png.Decode(result.Content)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to decode output: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
bounds := output.Bounds()
|
||||||
|
if bounds.Dx() != tt.size.Width || bounds.Dy() != tt.size.Height {
|
||||||
|
t.Fatalf("Process() made %dx%d, want %dx%d",
|
||||||
|
bounds.Dx(), bounds.Dy(), tt.size.Width, tt.size.Height)
|
||||||
|
}
|
||||||
|
|
||||||
|
row := make([]float64, bounds.Dx())
|
||||||
|
for x := range row {
|
||||||
|
r, _, _, _ := output.At(x, bounds.Dy()/2).RGBA()
|
||||||
|
row[x] = float64(r >> 8)
|
||||||
|
}
|
||||||
|
|
||||||
|
column := make([]float64, bounds.Dy())
|
||||||
|
for y := range column {
|
||||||
|
r, _, _, _ := output.At(bounds.Dx()/2, y).RGBA()
|
||||||
|
column[y] = float64(r >> 8)
|
||||||
|
}
|
||||||
|
|
||||||
|
width, height := squareLength(row), squareLength(column)
|
||||||
|
t.Logf("square is %.2f wide and %.2f high", width, height)
|
||||||
|
|
||||||
|
if width == 0 || math.Abs(width-height) > 0.5 {
|
||||||
|
t.Errorf("square is %.2f wide and %.2f high, want it square",
|
||||||
|
width, height)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -60,3 +60,45 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestImageProcessor_FitOutsideSideIsAtLeastOne verifies that FitOutside
|
||||||
|
// keeps each side at 1 or more when it works both sides out again from the
|
||||||
|
// larger scale. A side of 49 scaled to 1 is 49 times 1/49, which in floating
|
||||||
|
// point is just under 1 and so rounded down to 0, a width libvips refuses.
|
||||||
|
func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
inputWidth, inputHeight int
|
||||||
|
size Size
|
||||||
|
wantWidth, wantHeight int
|
||||||
|
}{
|
||||||
|
{"49x8192 at 0x100", 49, 8192, Size{Height: 100}, 1, 167},
|
||||||
|
{"49x49 at 1x1", 49, 49, Size{Width: 1, Height: 1}, 1, 1},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||||
|
|
||||||
|
result, err := New(Params{}).Process(t.Context(),
|
||||||
|
bytes.NewReader(input),
|
||||||
|
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitOutside})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Process() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = result.Content.Close() }()
|
||||||
|
|
||||||
|
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
||||||
|
|
||||||
|
if result.Width != tt.wantWidth || result.Height != tt.wantHeight {
|
||||||
|
t.Errorf("Process() made %dx%d, want %dx%d",
|
||||||
|
result.Width, result.Height, tt.wantWidth, tt.wantHeight)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -311,8 +311,10 @@ func (p *ImageProcessor) Process(
|
|||||||
|
|
||||||
// targetDimensions calculates the output dimensions for a requested size,
|
// targetDimensions calculates the output dimensions for a requested size,
|
||||||
// scaling proportionally when only one dimension is given and keeping the
|
// scaling proportionally when only one dimension is given and keeping the
|
||||||
// original dimensions when both are zero. A side scaled proportionally can
|
// original dimensions when both are zero. A side scaled proportionally is
|
||||||
// come out above MaxOutputDimension, which is ErrOutputTooLarge.
|
// rounded down but kept at 1 or more, as for a thin image it would round
|
||||||
|
// down to 0, a side of no pixels. It can come out above MaxOutputDimension,
|
||||||
|
// which is ErrOutputTooLarge.
|
||||||
func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
|
func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
|
||||||
width, height := size.Width, size.Height
|
width, height := size.Width, size.Height
|
||||||
|
|
||||||
@@ -322,10 +324,10 @@ func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
|
|||||||
return origWidth, origHeight, nil
|
return origWidth, origHeight, nil
|
||||||
case width == 0:
|
case width == 0:
|
||||||
// Only height specified: calculate width proportionally
|
// Only height specified: calculate width proportionally
|
||||||
width = origWidth * height / origHeight
|
width = max(1, origWidth*height/origHeight)
|
||||||
case height == 0:
|
case height == 0:
|
||||||
// Only width specified: calculate height proportionally
|
// Only width specified: calculate height proportionally
|
||||||
height = origHeight * width / origWidth
|
height = max(1, origHeight*width/origWidth)
|
||||||
}
|
}
|
||||||
|
|
||||||
if width > MaxOutputDimension || height > MaxOutputDimension {
|
if width > MaxOutputDimension || height > MaxOutputDimension {
|
||||||
@@ -475,17 +477,20 @@ func (p *ImageProcessor) resize(
|
|||||||
) error {
|
) error {
|
||||||
switch fit {
|
switch fit {
|
||||||
case FitCover, "":
|
case FitCover, "":
|
||||||
// Resize and crop to fill exact dimensions (default)
|
// Crop the centre and resize to fill exact dimensions (default)
|
||||||
return img.Thumbnail(width, height, vips.InterestingCentre)
|
return cropCentreAndResize(img, width, height)
|
||||||
|
|
||||||
case FitContain:
|
case FitContain:
|
||||||
// Resize to fit within dimensions, maintaining aspect ratio
|
// Resize to fit within dimensions, maintaining aspect ratio. Scaled
|
||||||
|
// by the smaller scale, the short side of a thin image can round
|
||||||
|
// down to 0, which libvips refuses as a width, so each side is at
|
||||||
|
// least 1.
|
||||||
imgW, imgH := img.Width(), img.Height()
|
imgW, imgH := img.Width(), img.Height()
|
||||||
scaleW := float64(width) / float64(imgW)
|
scaleW := float64(width) / float64(imgW)
|
||||||
scaleH := float64(height) / float64(imgH)
|
scaleH := float64(height) / float64(imgH)
|
||||||
scale := min(scaleW, scaleH)
|
scale := min(scaleW, scaleH)
|
||||||
newW := int(float64(imgW) * scale)
|
newW := max(1, int(float64(imgW)*scale))
|
||||||
newH := int(float64(imgH) * scale)
|
newH := max(1, int(float64(imgH)*scale))
|
||||||
|
|
||||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||||
|
|
||||||
@@ -503,19 +508,22 @@ func (p *ImageProcessor) resize(
|
|||||||
scaleW := float64(width) / float64(imgW)
|
scaleW := float64(width) / float64(imgW)
|
||||||
scaleH := float64(height) / float64(imgH)
|
scaleH := float64(height) / float64(imgH)
|
||||||
scale := min(scaleW, scaleH)
|
scale := min(scaleW, scaleH)
|
||||||
newW := int(float64(imgW) * scale)
|
newW := max(1, int(float64(imgW)*scale))
|
||||||
newH := int(float64(imgH) * scale)
|
newH := max(1, int(float64(imgH)*scale))
|
||||||
|
|
||||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||||
|
|
||||||
case FitOutside:
|
case FitOutside:
|
||||||
// Resize so smallest dimension fits, may exceed target on other dimension
|
// Resize so smallest dimension fits, may exceed target on other
|
||||||
|
// dimension. A side scaled to 1 can come out just under 1 in
|
||||||
|
// floating point, as 49 times 1/49 does, and round down to 0, which
|
||||||
|
// libvips refuses as a width, so each side is at least 1.
|
||||||
imgW, imgH := img.Width(), img.Height()
|
imgW, imgH := img.Width(), img.Height()
|
||||||
scaleW := float64(width) / float64(imgW)
|
scaleW := float64(width) / float64(imgW)
|
||||||
scaleH := float64(height) / float64(imgH)
|
scaleH := float64(height) / float64(imgH)
|
||||||
scale := max(scaleW, scaleH)
|
scale := max(scaleW, scaleH)
|
||||||
newW := int(float64(imgW) * scale)
|
newW := max(1, int(float64(imgW)*scale))
|
||||||
newH := int(float64(imgH) * scale)
|
newH := max(1, int(float64(imgH)*scale))
|
||||||
|
|
||||||
// The dimension that exceeds the target is held to the maximum too
|
// The dimension that exceeds the target is held to the maximum too
|
||||||
if newW > MaxOutputDimension || newH > MaxOutputDimension {
|
if newW > MaxOutputDimension || newH > MaxOutputDimension {
|
||||||
@@ -530,6 +538,38 @@ func (p *ImageProcessor) resize(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// cropCentreAndResize scales img by the same amount on both sides, the
|
||||||
|
// least that makes it cover width x height, and cuts the centre width x
|
||||||
|
// height out of it, as libvips' thumbnail does with InterestingCentre.
|
||||||
|
// thumbnail alone scales the whole image and copies the result into memory
|
||||||
|
// before it cuts, and for a thin image that is far larger than the image or
|
||||||
|
// the output: 8192x64 at 8192x8192 is 1048576x8192, about 24 GiB. So the
|
||||||
|
// centre is cut from img first, widened to whole pixels of img, which adds
|
||||||
|
// less than one pixel of img on each side, and thumbnail scales and cuts
|
||||||
|
// only that.
|
||||||
|
func cropCentreAndResize(img *vips.ImageRef, width, height int) error {
|
||||||
|
imgW, imgH := img.Width(), img.Height()
|
||||||
|
|
||||||
|
// An image wider than the requested proportions keeps its height and
|
||||||
|
// loses width down to imgH*width/height, any other keeps its width and
|
||||||
|
// loses height down to imgW*height/width. Half of what is lost comes
|
||||||
|
// off each side, rounded down to whole pixels, so the part kept holds
|
||||||
|
// all of the centre.
|
||||||
|
left, top := 0, 0
|
||||||
|
if imgW*height > imgH*width {
|
||||||
|
left = (imgW*height - imgH*width) / (2 * height) //nolint:mnd // each side
|
||||||
|
} else {
|
||||||
|
top = (imgH*width - imgW*height) / (2 * width) //nolint:mnd // each side
|
||||||
|
}
|
||||||
|
|
||||||
|
err := img.ExtractArea(left, top, imgW-2*left, imgH-2*top)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
return img.Thumbnail(width, height, vips.InterestingCentre)
|
||||||
|
}
|
||||||
|
|
||||||
const defaultQuality = 85
|
const defaultQuality = 85
|
||||||
|
|
||||||
// encode encodes an image to the specified format. govips gives pixa no way
|
// encode encodes an image to the specified format. govips gives pixa no way
|
||||||
|
|||||||
@@ -660,10 +660,10 @@ func (c *Cache) readHitAndMissCounts(ctx context.Context, stats *CacheStats) err
|
|||||||
// hold: it reads the content type from the variant's .meta file and
|
// hold: it reads the content type from the variant's .meta file and
|
||||||
// keeps it in metaCache, unless a StoreVariant has put one there
|
// keeps it in metaCache, unless a StoreVariant has put one there
|
||||||
// meanwhile, as the store's is newer. A read that finds no .meta file,
|
// meanwhile, as the store's is newer. A read that finds no .meta file,
|
||||||
// as one can between a store's writing of the variant file and of its
|
// as one can when the store's .meta write failed or an eviction removed
|
||||||
// .meta file, serves application/octet-stream and keeps nothing, so
|
// the file after the read opened the variant, serves
|
||||||
// metaCache only ever holds a type read from a .meta file or passed to
|
// application/octet-stream and keeps nothing, so metaCache only ever
|
||||||
// StoreVariant.
|
// holds a type read from a .meta file or passed to StoreVariant.
|
||||||
func (c *Cache) loadVariantWithMeta(
|
func (c *Cache) loadVariantWithMeta(
|
||||||
cacheKey VariantKey,
|
cacheKey VariantKey,
|
||||||
) (io.ReadCloser, int64, string, error) {
|
) (io.ReadCloser, int64, string, error) {
|
||||||
|
|||||||
@@ -78,10 +78,6 @@ func (c *Cache) computeDefaultMaxBytes(
|
|||||||
computed := spaceBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
|
computed := spaceBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
|
||||||
computed = min(computed, math.MaxInt64)
|
computed = min(computed, math.MaxInt64)
|
||||||
|
|
||||||
// gosec cannot see that min() above bounds computed, so it reads
|
|
||||||
// this conversion as potentially overflowing. It cannot: computed is
|
|
||||||
// at most math.MaxInt64 on every path here.
|
|
||||||
//nolint:gosec // G115: clamped to MaxInt64 by min above
|
|
||||||
limit := int64(computed)
|
limit := int64(computed)
|
||||||
limit = max(limit, DefaultCacheMaxBytesFloor)
|
limit = max(limit, DefaultCacheMaxBytesFloor)
|
||||||
|
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"database/sql"
|
"database/sql"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io/fs"
|
"io/fs"
|
||||||
"os"
|
"os"
|
||||||
@@ -560,10 +561,11 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
|
|||||||
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
|
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
|
||||||
// insert under transient DB contention), drops accounting rows whose
|
// insert under transient DB contention), drops accounting rows whose
|
||||||
// files are missing, removes source blob files the database does not
|
// files are missing, removes source blob files the database does not
|
||||||
// know (and rows whose files are gone), sweeps stale temp files left
|
// know (and rows whose files are gone), removes source and variant files
|
||||||
// behind by crashed writes, and last checks the total cache usage
|
// that are empty or not the size their rows record, sweeps stale temp
|
||||||
// against the tables. It reads the tables a page at a time. Running it
|
// files left behind by crashed writes, and last checks the total cache
|
||||||
// periodically, not just once, bounds how long such drift can
|
// usage against the tables. It reads the tables a page at a time.
|
||||||
|
// Running it periodically, not just once, bounds how long such drift can
|
||||||
// accumulate unaccounted for on a long-running process to one eviction
|
// accumulate unaccounted for on a long-running process to one eviction
|
||||||
// interval. Once ctx is cancelled, it stops at the next file or row and
|
// interval. Once ctx is cancelled, it stops at the next file or row and
|
||||||
// returns ctx's error.
|
// returns ctx's error.
|
||||||
@@ -595,8 +597,9 @@ func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
|||||||
return c.reconcileUsageTotal(ctx)
|
return c.reconcileUsageTotal(ctx)
|
||||||
}
|
}
|
||||||
|
|
||||||
// reconcileVariantFiles walks the variant storage directory, adopting
|
// reconcileVariantFiles walks the variant storage directory, checking
|
||||||
// files without accounting rows and sweeping stale temp files.
|
// each variant file against its accounting row (reconcileVariantFile)
|
||||||
|
// and sweeping stale temp files.
|
||||||
func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
||||||
return filepath.WalkDir(
|
return filepath.WalkDir(
|
||||||
c.variants.baseDir,
|
c.variants.baseDir,
|
||||||
@@ -621,38 +624,55 @@ func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return c.adoptVariantFile(ctx, path, entry, VariantKey(name))
|
return c.reconcileVariantFile(ctx, path, entry, VariantKey(name))
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// adoptVariantFile inserts an accounting row for a variant file that
|
// reconcileVariantFile checks one variant file against its accounting
|
||||||
// has none, using the file's size and modification time.
|
// row. A file without a row is adopted into the accounting, unless it is
|
||||||
func (c *Cache) adoptVariantFile(
|
// empty. A file that is empty, or not the size its row records, as a
|
||||||
|
// crash or a restored snapshot can leave it, is removed with its row and
|
||||||
|
// a warning naming it, so the variant is converted again.
|
||||||
|
func (c *Cache) reconcileVariantFile(
|
||||||
ctx context.Context, path string, entry fs.DirEntry, cacheKey VariantKey,
|
ctx context.Context, path string, entry fs.DirEntry, cacheKey VariantKey,
|
||||||
) error {
|
) error {
|
||||||
var rowExists int
|
|
||||||
|
|
||||||
err := c.db.QueryRowContext(ctx,
|
|
||||||
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, string(cacheKey),
|
|
||||||
).Scan(&rowExists)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to check variant accounting row: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if rowExists > 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
info, err := entry.Info()
|
info, err := entry.Info()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to stat variant file: %w", err)
|
return fmt.Errorf("failed to stat variant file: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
var recordedSize int64
|
||||||
|
|
||||||
|
err = c.db.QueryRowContext(ctx,
|
||||||
|
`SELECT size_bytes FROM variant_content WHERE cache_key = ?`, string(cacheKey),
|
||||||
|
).Scan(&recordedSize)
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, sql.ErrNoRows) && info.Size() > 0:
|
||||||
|
return c.adoptVariantFile(ctx, path, info, cacheKey)
|
||||||
|
case err != nil && !errors.Is(err, sql.ErrNoRows):
|
||||||
|
return fmt.Errorf("failed to check variant accounting row: %w", err)
|
||||||
|
case info.Size() > 0 && info.Size() == recordedSize:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// The file is empty, or not the size its row records
|
||||||
|
c.log.Warn("removing damaged cached variant file", "path", path,
|
||||||
|
"file_bytes", info.Size(), "recorded_bytes", recordedSize)
|
||||||
|
|
||||||
|
return c.evictVariant(ctx, cacheKey)
|
||||||
|
}
|
||||||
|
|
||||||
|
// adoptVariantFile inserts an accounting row for a variant file that
|
||||||
|
// has none, using the file's size and modification time.
|
||||||
|
func (c *Cache) adoptVariantFile(
|
||||||
|
ctx context.Context, path string, info fs.FileInfo, cacheKey VariantKey,
|
||||||
|
) error {
|
||||||
modTime := info.ModTime().UTC().Format(sqliteTimestampLayout)
|
modTime := info.ModTime().UTC().Format(sqliteTimestampLayout)
|
||||||
contentType := c.variantContentTypeFromSidecar(path)
|
contentType := c.variantContentTypeFromSidecar(path)
|
||||||
|
|
||||||
_, err = c.db.ExecContext(ctx, `
|
_, err := c.db.ExecContext(ctx, `
|
||||||
INSERT INTO variant_content
|
INSERT INTO variant_content
|
||||||
(cache_key, size_bytes, content_type, created_at, last_accessed_at)
|
(cache_key, size_bytes, content_type, created_at, last_accessed_at)
|
||||||
VALUES (?, ?, ?, ?, ?)
|
VALUES (?, ?, ?, ?, ?)
|
||||||
@@ -771,10 +791,9 @@ func queryStringColumn[T ~string](
|
|||||||
return values, nil
|
return values, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// reconcileSourceFiles walks the source content directory, removing
|
// reconcileSourceFiles walks the source content directory, checking
|
||||||
// blob files the database does not track (they are unreachable: source
|
// each blob file against its source_content row (reconcileSourceFile)
|
||||||
// lookups always go through source_metadata) and sweeping stale temp
|
// and sweeping stale temp files.
|
||||||
// files.
|
|
||||||
func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
|
func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
|
||||||
return filepath.WalkDir(
|
return filepath.WalkDir(
|
||||||
c.srcContent.baseDir,
|
c.srcContent.baseDir,
|
||||||
@@ -795,31 +814,55 @@ func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return c.removeUntrackedSourceFile(ctx, path, ContentHash(name))
|
return c.reconcileSourceFile(ctx, path, entry, ContentHash(name))
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// reconcileSourceFile checks one source blob file against its
|
||||||
|
// source_content row. A file without a row is removed: it is
|
||||||
|
// unreachable, as source lookups always go through source_metadata. A
|
||||||
|
// file that is empty, or not the size its row records, as a crash or a
|
||||||
|
// restored snapshot can leave it, is removed with its rows and a warning
|
||||||
|
// naming it, so the source is fetched again.
|
||||||
|
func (c *Cache) reconcileSourceFile(
|
||||||
|
ctx context.Context, path string, entry fs.DirEntry, contentHash ContentHash,
|
||||||
|
) error {
|
||||||
|
info, err := entry.Info()
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to stat source content file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var recordedSize int64
|
||||||
|
|
||||||
|
err = c.db.QueryRowContext(ctx,
|
||||||
|
`SELECT size_bytes FROM source_content WHERE content_hash = ?`,
|
||||||
|
string(contentHash),
|
||||||
|
).Scan(&recordedSize)
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, sql.ErrNoRows):
|
||||||
|
return c.removeUntrackedSourceFile(ctx, path, contentHash)
|
||||||
|
case err != nil:
|
||||||
|
return fmt.Errorf("failed to check source content row: %w", err)
|
||||||
|
case info.Size() > 0 && info.Size() == recordedSize:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// The file is empty, or not the size its row records
|
||||||
|
c.log.Warn("removing damaged cached source file", "path", path,
|
||||||
|
"file_bytes", info.Size(), "recorded_bytes", recordedSize)
|
||||||
|
|
||||||
|
return c.evictSourceBlob(ctx, contentHash)
|
||||||
|
}
|
||||||
|
|
||||||
// removeUntrackedSourceFile deletes a source blob file that has no
|
// removeUntrackedSourceFile deletes a source blob file that has no
|
||||||
// source_content row. Any source_metadata rows referencing the hash
|
// source_content row. Any source_metadata rows referencing the hash
|
||||||
// are removed first so no row ever points at a deleted file.
|
// are removed first so no row ever points at a deleted file.
|
||||||
func (c *Cache) removeUntrackedSourceFile(
|
func (c *Cache) removeUntrackedSourceFile(
|
||||||
ctx context.Context, path string, contentHash ContentHash,
|
ctx context.Context, path string, contentHash ContentHash,
|
||||||
) error {
|
) error {
|
||||||
var rowExists int
|
_, err := c.db.ExecContext(ctx,
|
||||||
|
|
||||||
err := c.db.QueryRowContext(ctx,
|
|
||||||
`SELECT COUNT(*) FROM source_content WHERE content_hash = ?`, string(contentHash),
|
|
||||||
).Scan(&rowExists)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to check source content row: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if rowExists > 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = c.db.ExecContext(ctx,
|
|
||||||
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash))
|
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to delete metadata rows for untracked blob: %w", err)
|
return fmt.Errorf("failed to delete metadata rows for untracked blob: %w", err)
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
|
"net/url"
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -86,9 +87,11 @@ func ValidateImageRequest(req *ImageRequest) error {
|
|||||||
type ImageRequest struct {
|
type ImageRequest struct {
|
||||||
// SourceHost is the origin host (e.g., "cdn.example.com")
|
// SourceHost is the origin host (e.g., "cdn.example.com")
|
||||||
SourceHost string
|
SourceHost string
|
||||||
// SourcePath is the path on the origin (e.g., "/photos/cat.jpg")
|
// SourcePath is the path on the origin with its percent-escapes decoded,
|
||||||
|
// as url.URL's Path holds it (e.g., "/photos/my cat.jpg")
|
||||||
SourcePath string
|
SourcePath string
|
||||||
// SourceQuery is the optional query string for the origin URL
|
// SourceQuery is the optional query string for the origin URL as the URL
|
||||||
|
// writes it, percent-escapes included, as url.URL's RawQuery holds it
|
||||||
SourceQuery string
|
SourceQuery string
|
||||||
// Size is the requested output dimensions
|
// Size is the requested output dimensions
|
||||||
Size Size
|
Size Size
|
||||||
@@ -107,20 +110,25 @@ type ImageRequest struct {
|
|||||||
AllowHTTP bool
|
AllowHTTP bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// SourceURL returns the full upstream URL to fetch.
|
// SourceURL returns the full upstream URL to fetch: the source path
|
||||||
// Uses http:// scheme when AllowHTTP is true, otherwise https://.
|
// percent-encoded again, as net/url writes a path, so each source path is one
|
||||||
|
// upstream path, and the source query as it is, with the characters
|
||||||
|
// escapeQuery encodes percent-encoded. Uses http:// scheme when AllowHTTP is
|
||||||
|
// true, otherwise https://.
|
||||||
func (r *ImageRequest) SourceURL() string {
|
func (r *ImageRequest) SourceURL() string {
|
||||||
scheme := "https"
|
scheme := "https"
|
||||||
if r.AllowHTTP {
|
if r.AllowHTTP {
|
||||||
scheme = "http"
|
scheme = "http"
|
||||||
}
|
}
|
||||||
|
|
||||||
url := scheme + "://" + r.SourceHost + r.SourcePath
|
sourceURL := url.URL{
|
||||||
if r.SourceQuery != "" {
|
Scheme: scheme,
|
||||||
url += "?" + r.SourceQuery
|
Host: r.SourceHost,
|
||||||
|
Path: r.SourcePath,
|
||||||
|
RawQuery: escapeQuery(r.SourceQuery),
|
||||||
}
|
}
|
||||||
|
|
||||||
return url
|
return sourceURL.String()
|
||||||
}
|
}
|
||||||
|
|
||||||
// ImageResponse represents a processed image ready to serve
|
// ImageResponse represents a processed image ready to serve
|
||||||
|
|||||||
@@ -0,0 +1,359 @@
|
|||||||
|
package imgcache
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// getImage calls Get for req and returns the image it answered with and
|
||||||
|
// its cache status.
|
||||||
|
func getImage(t *testing.T, svc *Service, req *ImageRequest) ([]byte, CacheStatus) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
resp, err := svc.Get(t.Context(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Get() of quality %d error = %v", req.Quality, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = resp.Content.Close() }()
|
||||||
|
|
||||||
|
body, err := io.ReadAll(resp.Content)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("reading the image of quality %d: %v", req.Quality, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return body, resp.CacheStatus
|
||||||
|
}
|
||||||
|
|
||||||
|
// fileSize returns the size of the file at path.
|
||||||
|
func fileSize(t *testing.T, path string) int64 {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
info, err := os.Stat(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("stat %s: %v", path, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return info.Size()
|
||||||
|
}
|
||||||
|
|
||||||
|
// truncateFile cuts the file at path down to size bytes.
|
||||||
|
func truncateFile(t *testing.T, path string, size int64) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
t.Logf("cutting %s from %d to %d bytes",
|
||||||
|
filepath.Base(path), fileSize(t, path), size)
|
||||||
|
|
||||||
|
err := os.Truncate(path, size)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("truncating %s: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// cachedSourcePath returns the path of the cached source file req uses.
|
||||||
|
func cachedSourcePath(t *testing.T, svc *Service, req *ImageRequest) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
contentHash, _, err := svc.cache.LookupSource(t.Context(), req)
|
||||||
|
if err != nil || contentHash == "" {
|
||||||
|
t.Fatalf("LookupSource() = %q, %v; want the cached source", contentHash, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return svc.cache.srcContent.hashToPath(contentHash)
|
||||||
|
}
|
||||||
|
|
||||||
|
// reconcileAsAtStartup runs the cache's reconciliation pass, as pixa does
|
||||||
|
// when it starts, and returns what the pass logged.
|
||||||
|
func reconcileAsAtStartup(t *testing.T, cache *Cache) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var logBuf bytes.Buffer
|
||||||
|
|
||||||
|
cache.log = slog.New(slog.NewJSONHandler(&logBuf, nil))
|
||||||
|
|
||||||
|
err := cache.reconcileAccounting(t.Context())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("reconcileAccounting() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("reconciliation log: %s", logBuf.String())
|
||||||
|
|
||||||
|
return logBuf.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertWarningNames checks that logs hold a warning naming the file at
|
||||||
|
// path.
|
||||||
|
func assertWarningNames(t *testing.T, logs, path string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
for line := range strings.Lines(logs) {
|
||||||
|
if strings.Contains(line, `"level":"WARN"`) &&
|
||||||
|
strings.Contains(line, `"path":"`+path+`"`) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Errorf("reconciliation logged no warning naming %s", path)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestService_Get_FetchesAgainForEmptyCachedSource empties the cached source
|
||||||
|
// file, as a crash or a restored snapshot could, and runs the reconciliation
|
||||||
|
// pass, which must remove it with a warning naming it. It then asks for two
|
||||||
|
// new sizes of the source: the first must fetch it again and cache it whole,
|
||||||
|
// so that the second is made from the cache without a third fetch.
|
||||||
|
func TestService_Get_FetchesAgainForEmptyCachedSource(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
svc, fixtures := SetupTestService(t)
|
||||||
|
|
||||||
|
getImage(t, svc, photoVariant(fixtures, 85, FitCover))
|
||||||
|
|
||||||
|
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, FitCover))
|
||||||
|
wholeSize := fileSize(t, path)
|
||||||
|
|
||||||
|
truncateFile(t, path, 0)
|
||||||
|
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
|
||||||
|
|
||||||
|
getImage(t, svc, photoVariant(fixtures, 40, FitCover))
|
||||||
|
getImage(t, svc, photoVariant(fixtures, 30, FitCover))
|
||||||
|
|
||||||
|
fetches := readCacheStatsCounters(t, svc.cache).upstreamFetchCount
|
||||||
|
t.Logf("upstream fetches = %d", fetches)
|
||||||
|
|
||||||
|
if fetches != 2 {
|
||||||
|
t.Errorf("upstream fetches = %d, want 2: the empty cached source "+
|
||||||
|
"was kept and fetched again on every new size", fetches)
|
||||||
|
}
|
||||||
|
|
||||||
|
size := fileSize(t, path)
|
||||||
|
if size != wholeSize {
|
||||||
|
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestService_Get_FetchesAgainForPartialCachedSource cuts the cached source
|
||||||
|
// file to half its size, as a crash or a restored snapshot could, and runs
|
||||||
|
// the reconciliation pass, which must remove it with a warning naming it.
|
||||||
|
// It then asks for two new sizes of the source: the first must fetch it
|
||||||
|
// again rather than fail to decode the partial file, and the second is
|
||||||
|
// made from the cache.
|
||||||
|
func TestService_Get_FetchesAgainForPartialCachedSource(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
svc, fixtures := SetupTestService(t)
|
||||||
|
|
||||||
|
getImage(t, svc, photoVariant(fixtures, 85, FitCover))
|
||||||
|
|
||||||
|
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, FitCover))
|
||||||
|
wholeSize := fileSize(t, path)
|
||||||
|
|
||||||
|
truncateFile(t, path, wholeSize/2)
|
||||||
|
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
|
||||||
|
|
||||||
|
getImage(t, svc, photoVariant(fixtures, 40, FitCover))
|
||||||
|
getImage(t, svc, photoVariant(fixtures, 30, FitCover))
|
||||||
|
|
||||||
|
fetches := readCacheStatsCounters(t, svc.cache).upstreamFetchCount
|
||||||
|
t.Logf("upstream fetches = %d", fetches)
|
||||||
|
|
||||||
|
if fetches != 2 {
|
||||||
|
t.Errorf("upstream fetches = %d, want 2", fetches)
|
||||||
|
}
|
||||||
|
|
||||||
|
size := fileSize(t, path)
|
||||||
|
if size != wholeSize {
|
||||||
|
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertConvertedAgain caches a variant of the photo, cuts its file to size
|
||||||
|
// bytes and runs the reconciliation pass, which must remove it with a
|
||||||
|
// warning naming it. It then checks that Get converts the image again
|
||||||
|
// instead of serving the file, and that the next Get serves the whole
|
||||||
|
// variant from the cache.
|
||||||
|
func assertConvertedAgain(t *testing.T, size int64) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
svc, fixtures := SetupTestService(t)
|
||||||
|
req := photoVariant(fixtures, 85, FitCover)
|
||||||
|
|
||||||
|
want, _ := getImage(t, svc, req)
|
||||||
|
|
||||||
|
path := svc.cache.variants.keyToPath(CacheKey(req))
|
||||||
|
|
||||||
|
truncateFile(t, path, size)
|
||||||
|
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
|
||||||
|
|
||||||
|
got, status := getImage(t, svc, req)
|
||||||
|
t.Logf("after the cut: %d bytes, %s", len(got), status)
|
||||||
|
|
||||||
|
if status != CacheMiss || !bytes.Equal(got, want) {
|
||||||
|
t.Errorf("after the cut: Get() = %d bytes, %s; want the %d bytes "+
|
||||||
|
"converted again, %s", len(got), status, len(want), CacheMiss)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, status = getImage(t, svc, req)
|
||||||
|
t.Logf("next request: %d bytes, %s", len(got), status)
|
||||||
|
|
||||||
|
if status != CacheHit || !bytes.Equal(got, want) {
|
||||||
|
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
|
||||||
|
len(got), status, len(want), CacheHit)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestService_Get_ConvertsAgainForEmptyCachedVariant empties a cached
|
||||||
|
// variant file, as a crash or a restored snapshot could, and checks that it
|
||||||
|
// is converted again rather than served empty.
|
||||||
|
func TestService_Get_ConvertsAgainForEmptyCachedVariant(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
assertConvertedAgain(t, 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestService_Get_ConvertsAgainForPartialCachedVariant cuts a cached
|
||||||
|
// variant file to ten bytes, as a crash or a restored snapshot could, and
|
||||||
|
// checks that it is converted again rather than served cut short.
|
||||||
|
func TestService_Get_ConvertsAgainForPartialCachedVariant(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
assertConvertedAgain(t, 10)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReconciliationRemovesEmptyVariantFileWithoutRow checks that the
|
||||||
|
// reconciliation pass removes an empty variant file that has no
|
||||||
|
// accounting row, with a warning naming it, instead of adding a row for
|
||||||
|
// it that would have it served empty.
|
||||||
|
func TestReconciliationRemovesEmptyVariantFileWithoutRow(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||||
|
|
||||||
|
_, err := cache.variants.Store(testVariantKeyOne, bytes.NewReader(nil),
|
||||||
|
testContentTypeWebP)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to store an empty variant file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
logs := reconcileAsAtStartup(t, cache)
|
||||||
|
|
||||||
|
assertWarningNames(t, logs, cache.variants.keyToPath(testVariantKeyOne))
|
||||||
|
|
||||||
|
if cache.variants.Exists(testVariantKeyOne) {
|
||||||
|
t.Error("the empty variant file is still on disk")
|
||||||
|
}
|
||||||
|
|
||||||
|
rows := countRows(t, cache,
|
||||||
|
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`,
|
||||||
|
string(testVariantKeyOne))
|
||||||
|
if rows != 0 {
|
||||||
|
t.Errorf("the empty variant file got %d accounting rows, want 0", rows)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestVariantStorage_StoreReplacesMetaFile stores a variant over one
|
||||||
|
// already stored and checks that Store puts a new .meta file in place of
|
||||||
|
// the old one, as it does the image, rather than writing into the old
|
||||||
|
// file, which a crash could leave empty. The old .meta file is given a
|
||||||
|
// second name before the store, so that it can be told apart from the new
|
||||||
|
// one after it.
|
||||||
|
func TestVariantStorage_StoreReplacesMetaFile(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
storage, key := storeTestVariant(t)
|
||||||
|
metaPath := storage.keyToPath(key) + variantMetaSuffix
|
||||||
|
oldMetaPath := metaPath + ".old"
|
||||||
|
|
||||||
|
err := os.Link(metaPath, oldMetaPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("giving the .meta file a second name: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = storage.Store(key, bytes.NewReader([]byte("new variant data")),
|
||||||
|
testContentTypeJPEG)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Store() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
oldMeta, err := os.Stat(oldMetaPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("stat of the old .meta file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
newMeta, err := os.Stat(metaPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("stat of the new .meta file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if os.SameFile(oldMeta, newMeta) {
|
||||||
|
t.Error("Store() wrote into the old .meta file instead of putting " +
|
||||||
|
"a new one in its place")
|
||||||
|
}
|
||||||
|
|
||||||
|
reader, _, contentType, err := storage.LoadWithMeta(key)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadWithMeta() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_ = reader.Close()
|
||||||
|
|
||||||
|
t.Logf("content type after the store: %s", contentType)
|
||||||
|
|
||||||
|
if contentType != testContentTypeJPEG {
|
||||||
|
t.Errorf("content type = %q, want %q", contentType, testContentTypeJPEG)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestContentStorage_StoreHashedReplacesFileOfAnotherSize checks that a
|
||||||
|
// store over a file of another size at the content's path, such as a
|
||||||
|
// partial file a crash left, replaces it with the content.
|
||||||
|
func TestContentStorage_StoreHashedReplacesFileOfAnotherSize(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
storage, err := NewContentStorage(t.TempDir())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewContentStorage() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
content := []byte("the whole source image")
|
||||||
|
sum := sha256.Sum256(content)
|
||||||
|
hash := ContentHash(hex.EncodeToString(sum[:]))
|
||||||
|
path := storage.hashToPath(hash)
|
||||||
|
|
||||||
|
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("creating the content's directory: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = os.WriteFile(path, content[:5], StorageFilePerm)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("writing a partial file: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = storage.StoreHashed(hash, content)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("StoreHashed() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
stored, err := storage.Load(hash)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = stored.Close() }()
|
||||||
|
|
||||||
|
got, err := io.ReadAll(stored)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("reading the stored content: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !bytes.Equal(got, content) {
|
||||||
|
t.Errorf("stored file = %q, want %q", got, content)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -80,8 +80,8 @@ func (s *ContentStorage) Store(r io.Reader) (ContentHash, int64, error) {
|
|||||||
// from hash, without recomputing it. Callers that already know the
|
// from hash, without recomputing it. Callers that already know the
|
||||||
// hash before writing (e.g. because they must hold a hash-keyed lock
|
// hash before writing (e.g. because they must hold a hash-keyed lock
|
||||||
// across the whole store operation) use this instead of Store. Like
|
// across the whole store operation) use this instead of Store. Like
|
||||||
// Store, it is idempotent: content already on disk at that path is
|
// Store, it is idempotent: a file of the content's size already at that
|
||||||
// left untouched.
|
// path is left untouched, and a file of another size is replaced.
|
||||||
func (s *ContentStorage) StoreHashed(
|
func (s *ContentStorage) StoreHashed(
|
||||||
hash ContentHash, data []byte,
|
hash ContentHash, data []byte,
|
||||||
) (int64, error) {
|
) (int64, error) {
|
||||||
@@ -152,28 +152,35 @@ func (s *ContentStorage) Exists(hash ContentHash) bool {
|
|||||||
return err == nil
|
return err == nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// writeIfAbsent writes data to the path derived from hash, unless
|
// writeIfAbsent writes data with writeFileSynced to the path derived
|
||||||
// content already exists there, via a temp-file-plus-rename so
|
// from hash, unless a file of the data's size is already there.
|
||||||
// concurrent readers never observe a partial file.
|
|
||||||
func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error {
|
func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error {
|
||||||
// Build path: <basedir>/<ab>/<cd>/<hash>
|
// Build path: <basedir>/<ab>/<cd>/<hash>
|
||||||
path := s.hashToPath(hash)
|
path := s.hashToPath(hash)
|
||||||
|
|
||||||
// Check if already exists
|
// A file of another size there is damaged, and is replaced
|
||||||
_, statErr := os.Stat(path)
|
info, statErr := os.Stat(path)
|
||||||
if statErr == nil {
|
if statErr == nil && info.Size() == int64(len(data)) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create directory structure
|
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
||||||
dir := filepath.Dir(path)
|
|
||||||
|
|
||||||
err := os.MkdirAll(dir, StorageDirPerm)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to create directory: %w", err)
|
return fmt.Errorf("failed to create directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Write to temp file first, then rename for atomicity
|
return writeFileSynced(path, data)
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeFileSynced writes data to path through a temp file in the same
|
||||||
|
// directory: the temp file is synced to disk, renamed to path, and the
|
||||||
|
// directory synced after, so that neither a concurrent reader nor a
|
||||||
|
// crash finds a partial file under path. When the directory sync fails,
|
||||||
|
// the file is removed from path, so that a failed write leaves no file
|
||||||
|
// there.
|
||||||
|
func writeFileSynced(path string, data []byte) error {
|
||||||
|
dir := filepath.Dir(path)
|
||||||
|
|
||||||
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
|
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to create temp file: %w", err)
|
return fmt.Errorf("failed to create temp file: %w", err)
|
||||||
@@ -181,18 +188,20 @@ func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error {
|
|||||||
|
|
||||||
tmpPath := tmpFile.Name()
|
tmpPath := tmpFile.Name()
|
||||||
|
|
||||||
// Each failure path below unlinks the temp file explicitly. This
|
|
||||||
// replaces a deferred cleanup that read a named result, which the
|
|
||||||
// canonical config does not permit; the set of paths that remove
|
|
||||||
// tmpPath, and the order relative to Close, is unchanged. This
|
|
||||||
// mirrors how MetadataStorage.Store and VariantStorage.Store below
|
|
||||||
// already express the same cleanup.
|
|
||||||
_, err = tmpFile.Write(data)
|
_, err = tmpFile.Write(data)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = tmpFile.Close()
|
_ = tmpFile.Close()
|
||||||
_ = os.Remove(tmpPath)
|
_ = os.Remove(tmpPath)
|
||||||
|
|
||||||
return fmt.Errorf("failed to write content: %w", err)
|
return fmt.Errorf("failed to write temp file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = tmpFile.Sync()
|
||||||
|
if err != nil {
|
||||||
|
_ = tmpFile.Close()
|
||||||
|
_ = os.Remove(tmpPath)
|
||||||
|
|
||||||
|
return fmt.Errorf("failed to sync temp file: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
err = tmpFile.Close()
|
err = tmpFile.Close()
|
||||||
@@ -202,7 +211,6 @@ func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error {
|
|||||||
return fmt.Errorf("failed to close temp file: %w", err)
|
return fmt.Errorf("failed to close temp file: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Atomic rename
|
|
||||||
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
|
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = os.Remove(tmpPath)
|
_ = os.Remove(tmpPath)
|
||||||
@@ -210,6 +218,31 @@ func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error {
|
|||||||
return fmt.Errorf("failed to rename temp file: %w", err)
|
return fmt.Errorf("failed to rename temp file: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
err = syncDirectory(dir)
|
||||||
|
if err != nil {
|
||||||
|
_ = os.Remove(path)
|
||||||
|
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// syncDirectory syncs the directory dir to disk, so that a file just
|
||||||
|
// renamed into it is still there after a crash.
|
||||||
|
func syncDirectory(dir string) error {
|
||||||
|
d, err := os.Open(filepath.Clean(dir))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to open directory: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = d.Sync()
|
||||||
|
_ = d.Close()
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to sync directory: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -427,7 +460,9 @@ func NewVariantStorage(baseDir string, log *slog.Logger) (*VariantStorage, error
|
|||||||
return &VariantStorage{baseDir: baseDir, log: log}, nil
|
return &VariantStorage{baseDir: baseDir, log: log}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Store writes content and metadata to storage at the given key.
|
// Store writes content and metadata to storage at the given key, each
|
||||||
|
// with writeFileSynced. The .meta file is written first, so that a crash
|
||||||
|
// cannot leave the content in place without it.
|
||||||
func (s *VariantStorage) Store(
|
func (s *VariantStorage) Store(
|
||||||
key VariantKey, r io.Reader, contentType string,
|
key VariantKey, r io.Reader, contentType string,
|
||||||
) (int64, error) {
|
) (int64, error) {
|
||||||
@@ -440,46 +475,11 @@ func (s *VariantStorage) Store(
|
|||||||
path := s.keyToPath(key)
|
path := s.keyToPath(key)
|
||||||
metaPath := path + ".meta"
|
metaPath := path + ".meta"
|
||||||
|
|
||||||
// Create directory structure
|
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
||||||
dir := filepath.Dir(path)
|
|
||||||
|
|
||||||
err = os.MkdirAll(dir, StorageDirPerm)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, fmt.Errorf("failed to create directory: %w", err)
|
return 0, fmt.Errorf("failed to create directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Write content to temp file first, then rename for atomicity
|
|
||||||
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
|
|
||||||
if err != nil {
|
|
||||||
return 0, fmt.Errorf("failed to create temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
tmpPath := tmpFile.Name()
|
|
||||||
|
|
||||||
_, err = tmpFile.Write(data)
|
|
||||||
if err != nil {
|
|
||||||
_ = tmpFile.Close()
|
|
||||||
_ = os.Remove(tmpPath)
|
|
||||||
|
|
||||||
return 0, fmt.Errorf("failed to write content: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = tmpFile.Close()
|
|
||||||
if err != nil {
|
|
||||||
_ = os.Remove(tmpPath)
|
|
||||||
|
|
||||||
return 0, fmt.Errorf("failed to close temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Atomic rename content
|
|
||||||
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
|
|
||||||
if err != nil {
|
|
||||||
_ = os.Remove(tmpPath)
|
|
||||||
|
|
||||||
return 0, fmt.Errorf("failed to rename temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Write metadata
|
|
||||||
meta := VariantMeta{
|
meta := VariantMeta{
|
||||||
ContentType: contentType,
|
ContentType: contentType,
|
||||||
Size: size,
|
Size: size,
|
||||||
@@ -491,13 +491,19 @@ func (s *VariantStorage) Store(
|
|||||||
return 0, fmt.Errorf("failed to marshal metadata: %w", err)
|
return 0, fmt.Errorf("failed to marshal metadata: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Metadata write failure is non-fatal; content is already stored.
|
// Metadata write failure is non-fatal: the content is then served as
|
||||||
err = os.WriteFile(metaPath, metaData, StorageFilePerm)
|
// application/octet-stream.
|
||||||
|
err = writeFileSynced(metaPath, metaData)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.log.Warn("failed to write variant metadata sidecar",
|
s.log.Warn("failed to write variant metadata sidecar",
|
||||||
"path", metaPath, "error", err)
|
"path", metaPath, "error", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
err = writeFileSynced(path, data)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
|
||||||
return size, nil
|
return size, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package imgcache
|
|||||||
import (
|
import (
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net/netip"
|
||||||
"net/url"
|
"net/url"
|
||||||
"regexp"
|
"regexp"
|
||||||
"strconv"
|
"strconv"
|
||||||
@@ -28,7 +29,9 @@ var (
|
|||||||
const MaxDimension = imageprocessor.MaxOutputDimension
|
const MaxDimension = imageprocessor.MaxOutputDimension
|
||||||
|
|
||||||
// ValidateDimension checks one requested width or height; name ("width" or
|
// ValidateDimension checks one requested width or height; name ("width" or
|
||||||
// "height") appears in the error. 0 means "original size" and is valid.
|
// "height") appears in the error. 0 is valid: that side is then worked out from
|
||||||
|
// the source's proportions and the other requested side, rounded down but at
|
||||||
|
// least 1, or is the source's own side when the other side is 0 too.
|
||||||
func ValidateDimension(name string, value int) error {
|
func ValidateDimension(name string, value int) error {
|
||||||
if value < 0 {
|
if value < 0 {
|
||||||
return fmt.Errorf("%w: %s is negative", ErrInvalidSize, name)
|
return fmt.Errorf("%w: %s is negative", ErrInvalidSize, name)
|
||||||
@@ -47,6 +50,24 @@ func ValidateDimension(name string, value int) error {
|
|||||||
var sizeFormatRegex = regexp.MustCompile(
|
var sizeFormatRegex = regexp.MustCompile(
|
||||||
`^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`)
|
`^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`)
|
||||||
|
|
||||||
|
// percentEscapeRegex matches one percent-escape, such as "%2e"
|
||||||
|
var percentEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}`)
|
||||||
|
|
||||||
|
// queryEscapeRegex matches a percent-escape, or one character that a browser
|
||||||
|
// percent-encodes in the query of an http or https URL: one outside the
|
||||||
|
// printable ASCII characters "!" to "~", such as a space, a control character
|
||||||
|
// or a character outside ASCII, or one of `"#'<>`. A "%" that does not begin a
|
||||||
|
// percent-escape matches too.
|
||||||
|
var queryEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}|[^!-~]|["#'<>%]`)
|
||||||
|
|
||||||
|
// hostRegex matches the forms the host of an image URL may take: a host name
|
||||||
|
// or an IPv4 address (letters, digits, "-", "_" and "."), or an IPv6 address
|
||||||
|
// in brackets (hex digits, ":" and "."), either one optionally followed by ":"
|
||||||
|
// and a port, which may be empty. Its one group holds the text in brackets,
|
||||||
|
// which parseImageComponents also checks is an IPv6 address.
|
||||||
|
var hostRegex = regexp.MustCompile(
|
||||||
|
`^(?:[A-Za-z0-9_.-]+|\[([0-9A-Fa-f:.]+)\])(?::[0-9]*)?$`)
|
||||||
|
|
||||||
// ParsedURL contains the parsed components of an image proxy URL.
|
// ParsedURL contains the parsed components of an image proxy URL.
|
||||||
type ParsedURL struct {
|
type ParsedURL struct {
|
||||||
// Host is the source origin host (e.g., "cdn.example.com")
|
// Host is the source origin host (e.g., "cdn.example.com")
|
||||||
@@ -61,9 +82,8 @@ type ParsedURL struct {
|
|||||||
Format ImageFormat
|
Format ImageFormat
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParseImagePath parses the path captured by chi's wildcard:
|
// ParseImagePath parses an image URL's path after /v1/image/, decoded once:
|
||||||
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL
|
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL.
|
||||||
// This is the primary entry point when using chi routing.
|
|
||||||
// Examples:
|
// Examples:
|
||||||
// - cdn.example.com/photos/cat.jpg/800x600.webp
|
// - cdn.example.com/photos/cat.jpg/800x600.webp
|
||||||
// - cdn.example.com/photos/cat.jpg/0x0.jpeg
|
// - cdn.example.com/photos/cat.jpg/0x0.jpeg
|
||||||
@@ -79,9 +99,10 @@ func ParseImagePath(path string) (*ParsedURL, error) {
|
|||||||
return parseImageComponents(path)
|
return parseImageComponents(path)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParseImageURL parses a full URL path like /v1/image/<host>/<path>/<size>.<format>,
|
// ParseImageURL parses an image URL's path, decoded once as r.URL.Path holds
|
||||||
// or /v1/image/<host>/<path>/<size> for JPEG XL
|
// it: /v1/image/<host>/<path>/<size>.<format>, or /v1/image/<host>/<path>/<size>
|
||||||
// Use ParseImagePath instead when working with chi's wildcard capture.
|
// for JPEG XL. A source query follows the source path after a "?", which the
|
||||||
|
// URL writes as %3F.
|
||||||
func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
||||||
// Remove the /v1/image/ prefix
|
// Remove the /v1/image/ prefix
|
||||||
const prefix = "/v1/image/"
|
const prefix = "/v1/image/"
|
||||||
@@ -100,12 +121,6 @@ func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
|||||||
// parseImageComponents parses <host>/<path>/<size>.<format>, or
|
// parseImageComponents parses <host>/<path>/<size>.<format>, or
|
||||||
// <host>/<path>/<size> for JPEG XL.
|
// <host>/<path>/<size> for JPEG XL.
|
||||||
func parseImageComponents(remainder string) (*ParsedURL, error) {
|
func parseImageComponents(remainder string) (*ParsedURL, error) {
|
||||||
// Check for path traversal before any other processing
|
|
||||||
err := checkPathTraversal(remainder)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
// Find the last path segment, which holds "size" or "size.format"
|
// Find the last path segment, which holds "size" or "size.format"
|
||||||
lastSlash := strings.LastIndex(remainder, "/")
|
lastSlash := strings.LastIndex(remainder, "/")
|
||||||
if lastSlash == -1 {
|
if lastSlash == -1 {
|
||||||
@@ -144,10 +159,20 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
|
|||||||
return nil, ErrMissingHost
|
return nil, ErrMissingHost
|
||||||
}
|
}
|
||||||
|
|
||||||
// A URL reads what comes before "@" as a user name and password, and the
|
// Only the forms hostRegex describes are accepted. A "?", "#" or "@" in
|
||||||
// rest as the host
|
// the host would make Go's URL parser, which the allowlist check and the
|
||||||
if strings.Contains(host, "@") {
|
// fetcher use, read a different host from the one allowlisted and cached
|
||||||
return nil, fmt.Errorf("%w: %q holds @", ErrInvalidHost, host)
|
hostMatch := hostRegex.FindStringSubmatch(host)
|
||||||
|
if hostMatch == nil {
|
||||||
|
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Go's URL parser refuses brackets around anything but an IPv6 address
|
||||||
|
if inBrackets := hostMatch[1]; inBrackets != "" {
|
||||||
|
addr, err := netip.ParseAddr(inBrackets)
|
||||||
|
if err != nil || !addr.Is6() {
|
||||||
|
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Host names do not depend on letter case, so the allowlist, the upstream
|
// Host names do not depend on letter case, so the allowlist, the upstream
|
||||||
@@ -155,12 +180,30 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
|
|||||||
// host
|
// host
|
||||||
host = strings.ToLower(host)
|
host = strings.ToLower(host)
|
||||||
|
|
||||||
// Extract query string if present in path
|
// Go's URL parser, which the fetcher uses, reads a "#" as the start of a
|
||||||
|
// fragment and drops the rest, so each text after it would be another
|
||||||
|
// spelling, cached under its own keys, of one upstream request
|
||||||
|
if strings.Contains(path, "#") {
|
||||||
|
return nil, fmt.Errorf(
|
||||||
|
`%w: the source path or query holds a "#"`, ErrInvalidPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract query string if present in path. It is the source URL's query
|
||||||
|
// as that URL writes it. A character a URL's query cannot hold as it is,
|
||||||
|
// such as a space written %20 in the image URL, is percent-encoded here,
|
||||||
|
// so the query is signed and cached as it is sent upstream
|
||||||
if qIndex := strings.Index(path, "?"); qIndex != -1 {
|
if qIndex := strings.Index(path, "?"); qIndex != -1 {
|
||||||
query = path[qIndex+1:]
|
query = escapeQuery(path[qIndex+1:])
|
||||||
path = path[:qIndex]
|
path = path[:qIndex]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The source query names no file on the upstream, so only the host and
|
||||||
|
// the source path are checked for path traversal
|
||||||
|
err = checkPathTraversal(host + path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
// Ensure path starts with /
|
// Ensure path starts with /
|
||||||
if !strings.HasPrefix(path, "/") {
|
if !strings.HasPrefix(path, "/") {
|
||||||
path = "/" + path
|
path = "/" + path
|
||||||
@@ -186,12 +229,7 @@ func checkPathTraversal(path string) error {
|
|||||||
// Decode multiple times to catch double-encoding
|
// Decode multiple times to catch double-encoding
|
||||||
decoded := path
|
decoded := path
|
||||||
for range 3 {
|
for range 3 {
|
||||||
newDecoded, err := url.PathUnescape(decoded)
|
newDecoded := decodePercentEscapes(decoded)
|
||||||
if err != nil {
|
|
||||||
// Malformed encoding is suspicious
|
|
||||||
return ErrPathTraversal
|
|
||||||
}
|
|
||||||
|
|
||||||
if newDecoded == decoded {
|
if newDecoded == decoded {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
@@ -232,6 +270,32 @@ func checkPathTraversal(path string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// decodePercentEscapes decodes every percent-escape in s, such as %2e, and
|
||||||
|
// keeps a "%" that does not start one as it is, where url.PathUnescape would
|
||||||
|
// refuse the whole string. A source path holding %25, a literal "%", has such
|
||||||
|
// a "%" after one more decoding.
|
||||||
|
func decodePercentEscapes(s string) string {
|
||||||
|
return percentEscapeRegex.ReplaceAllStringFunc(s, func(escape string) string {
|
||||||
|
decoded, _ := url.PathUnescape(escape)
|
||||||
|
|
||||||
|
return decoded
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// escapeQuery returns a source query as a URL's query holds it: each
|
||||||
|
// character that queryEscapeRegex names, such as a space, percent-encoded,
|
||||||
|
// and the rest, the query's percent-escapes included, kept as they are. The
|
||||||
|
// result is unchanged by a second escapeQuery.
|
||||||
|
func escapeQuery(query string) string {
|
||||||
|
return queryEscapeRegex.ReplaceAllStringFunc(query, func(match string) string {
|
||||||
|
if percentEscapeRegex.MatchString(match) {
|
||||||
|
return match
|
||||||
|
}
|
||||||
|
|
||||||
|
return url.PathEscape(match)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
|
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
|
||||||
func parseSizeFormat(s string) (Size, ImageFormat, error) {
|
func parseSizeFormat(s string) (Size, ImageFormat, error) {
|
||||||
matches := sizeFormatRegex.FindStringSubmatch(s)
|
matches := sizeFormatRegex.FindStringSubmatch(s)
|
||||||
|
|||||||
@@ -407,6 +407,71 @@ func TestParseImagePath_PathTraversal(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestParseImageURL_SourceURLHoldingPercent25 verifies that a source path and
|
||||||
|
// a source query holding %25, the percent-escape of a literal "%", are
|
||||||
|
// accepted as they are, although decoding them once more leaves a "%" that
|
||||||
|
// does not start a percent-escape.
|
||||||
|
func TestParseImageURL_SourceURLHoldingPercent25(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
input string
|
||||||
|
want *ParsedURL
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "path",
|
||||||
|
input: "/v1/image/cdn.example.com/sale/50%25off.jpg/800x600.webp",
|
||||||
|
want: &ParsedURL{
|
||||||
|
Host: testHostCDN, Path: "/sale/50%25off.jpg",
|
||||||
|
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "query",
|
||||||
|
input: "/v1/image/cdn.example.com/photos/cat.jpg?discount=50%25/800x600.webp",
|
||||||
|
want: &ParsedURL{
|
||||||
|
Host: testHostCDN, Path: testPathCat, Query: "discount=50%25",
|
||||||
|
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
got, err := ParseImageURL(tt.input)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseImageURL() unexpected error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertParsedURL(t, got, tt.want)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestParseImageURL_DoubleEncodedTraversalNextToPercent verifies that a
|
||||||
|
// double-encoded ".." is still refused when the path also holds a "%" that
|
||||||
|
// does not start a percent-escape, either as written or after one more
|
||||||
|
// decoding.
|
||||||
|
func TestParseImageURL_DoubleEncodedTraversalNextToPercent(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, input := range []string{
|
||||||
|
"/v1/image/cdn.example.com/photos/%252e%252e/secret/50%25off.jpg/800x600.jpeg",
|
||||||
|
"/v1/image/cdn.example.com/photos/%252e%252e/secret/100%/800x600.jpeg",
|
||||||
|
} {
|
||||||
|
_, err := ParseImageURL(input)
|
||||||
|
t.Logf("ParseImageURL(%q) error = %v", input, err)
|
||||||
|
|
||||||
|
if !errors.Is(err, ErrPathTraversal) {
|
||||||
|
t.Errorf("ParseImageURL(%q) error = %v, want ErrPathTraversal",
|
||||||
|
input, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// errorIs checks if err matches target (handles wrapped errors).
|
// errorIs checks if err matches target (handles wrapped errors).
|
||||||
func errorIs(err, target error) bool {
|
func errorIs(err, target error) bool {
|
||||||
if errors.Is(err, target) {
|
if errors.Is(err, target) {
|
||||||
|
|||||||
@@ -220,7 +220,10 @@ func (s *Middleware) CORS() func(http.Handler) http.Handler {
|
|||||||
// pattern of the chi route a request matches, such as /v1/image/*, or
|
// pattern of the chi route a request matches, such as /v1/image/*, or
|
||||||
// "unmatched" for a request that matches none, so the routes fix how many
|
// "unmatched" for a request that matches none, so the routes fix how many
|
||||||
// values the label has. It is never the path, with which any client could
|
// values the label has. It is never the path, with which any client could
|
||||||
// add series to the registry without limit.
|
// add series to the registry without limit. Its method label is the
|
||||||
|
// request's method when that is a standard one, such as GET, and "other"
|
||||||
|
// for any other: Go's HTTP server accepts any method name, so a label per
|
||||||
|
// method would let a client add series the same way.
|
||||||
func (s *Middleware) Metrics() func(http.Handler) http.Handler {
|
func (s *Middleware) Metrics() func(http.Handler) http.Handler {
|
||||||
mdlw := ghmm.New(ghmm.Config{
|
mdlw := ghmm.New(ghmm.Config{
|
||||||
Recorder: metrics.NewRecorder(metrics.Config{
|
Recorder: metrics.NewRecorder(metrics.Config{
|
||||||
@@ -230,7 +233,19 @@ func (s *Middleware) Metrics() func(http.Handler) http.Handler {
|
|||||||
|
|
||||||
return func(next http.Handler) http.Handler {
|
return func(next http.Handler) http.Handler {
|
||||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
std.Handler(routePattern(r), mdlw, next).ServeHTTP(w, r)
|
// The metrics library takes the method label from the request
|
||||||
|
// it is handed, so it is handed a shallow copy of r whose
|
||||||
|
// method is the label. next is served r itself, through the
|
||||||
|
// response writer the library wraps to see the status and size.
|
||||||
|
labelled := r.WithContext(r.Context())
|
||||||
|
labelled.Method = methodLabel(r.Method)
|
||||||
|
|
||||||
|
serveNext := http.HandlerFunc(
|
||||||
|
func(wrapped http.ResponseWriter, _ *http.Request) {
|
||||||
|
next.ServeHTTP(wrapped, r)
|
||||||
|
})
|
||||||
|
|
||||||
|
std.Handler(routePattern(r), mdlw, serveNext).ServeHTTP(w, labelled)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -275,6 +290,20 @@ func routePattern(r *http.Request) string {
|
|||||||
return pattern
|
return pattern
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// methodLabel returns method when it is one of the standard methods net/http
|
||||||
|
// names: GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS, CONNECT and TRACE.
|
||||||
|
// For any other method it returns "other".
|
||||||
|
func methodLabel(method string) string {
|
||||||
|
switch method {
|
||||||
|
case http.MethodGet, http.MethodHead, http.MethodPost, http.MethodPut,
|
||||||
|
http.MethodPatch, http.MethodDelete, http.MethodOptions,
|
||||||
|
http.MethodConnect, http.MethodTrace:
|
||||||
|
return method
|
||||||
|
default:
|
||||||
|
return "other"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// MetricsAuth returns a basic auth middleware for the metrics endpoint.
|
// MetricsAuth returns a basic auth middleware for the metrics endpoint.
|
||||||
func (s *Middleware) MetricsAuth() func(http.Handler) http.Handler {
|
func (s *Middleware) MetricsAuth() func(http.Handler) http.Handler {
|
||||||
return basicauth.New(
|
return basicauth.New(
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/url"
|
"net/url"
|
||||||
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
@@ -314,6 +315,58 @@ func requestCountsByHandler(
|
|||||||
return counts
|
return counts
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestMetricsGivesMadeUpMethodsOneLabel checks that the metrics middleware
|
||||||
|
// gives requests with methods that are not standard ones a single method
|
||||||
|
// label, "other", and still serves each request with the method it was sent
|
||||||
|
// with. With a label per method, any client could add series to the
|
||||||
|
// registry without limit by making up a new method for each request.
|
||||||
|
func TestMetricsGivesMadeUpMethodsOneLabel(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// The line /metrics shows once both made-up methods have been served.
|
||||||
|
const want = `http_request_duration_seconds_count{` +
|
||||||
|
`code="200",handler="unmatched",method="other",service=""} 2`
|
||||||
|
|
||||||
|
registry := prometheus.NewRegistry()
|
||||||
|
mw := &Middleware{
|
||||||
|
log: slog.Default(),
|
||||||
|
config: &config.Config{},
|
||||||
|
metricsRegistry: registry,
|
||||||
|
}
|
||||||
|
|
||||||
|
var served []string
|
||||||
|
|
||||||
|
handler := mw.Metrics()(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
served = append(served, r.Method)
|
||||||
|
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
|
||||||
|
methods := []string{"FROBNICATE", "WIBBLE"}
|
||||||
|
for _, method := range methods {
|
||||||
|
handler.ServeHTTP(httptest.NewRecorder(), httptest.NewRequestWithContext(
|
||||||
|
t.Context(), method, "/", nil))
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("methods the handler was served: %v", served)
|
||||||
|
|
||||||
|
if !slices.Equal(served, methods) {
|
||||||
|
t.Errorf("methods the handler was served = %v, want %v",
|
||||||
|
served, methods)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
promhttp.HandlerFor(registry, promhttp.HandlerOpts{}).ServeHTTP(rec,
|
||||||
|
httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, "/metrics", nil))
|
||||||
|
|
||||||
|
if !strings.Contains(rec.Body.String(), want) {
|
||||||
|
t.Errorf("/metrics does not count both made-up methods as one; "+
|
||||||
|
"want the line %q in:\n%s", want, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestLoggingLeavesOutSubmittedSigningKey checks that a login, a POST /
|
// TestLoggingLeavesOutSubmittedSigningKey checks that a login, a POST /
|
||||||
// whose form carries the signing key, leaves no trace of the key in the
|
// whose form carries the signing key, leaves no trace of the key in the
|
||||||
// request's log line.
|
// request's log line.
|
||||||
|
|||||||
@@ -14,10 +14,12 @@ import (
|
|||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
"go.uber.org/fx/fxtest"
|
"go.uber.org/fx/fxtest"
|
||||||
@@ -183,6 +185,57 @@ func startImageProxy(
|
|||||||
return s, db.DB(), stateDir
|
return s, db.DB(), stateDir
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// startImageProxyWithBlockedNetworks starts the components pixad's fx app
|
||||||
|
// builds, from a config with a fresh state directory and blockedNetworks as
|
||||||
|
// blocked_networks. Unlike startImageProxy it provides no upstream fetcher, so
|
||||||
|
// the handlers build the one pixad uses, which checks every address it
|
||||||
|
// connects to against the built-in blocked ranges and blocked_networks. It
|
||||||
|
// returns the server with its routes.
|
||||||
|
func startImageProxyWithBlockedNetworks(
|
||||||
|
t *testing.T, blockedNetworks []netip.Prefix,
|
||||||
|
) *Server {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
stateDir := t.TempDir()
|
||||||
|
cfg := &config.Config{
|
||||||
|
SigningKey: testSigningKey,
|
||||||
|
StateDir: stateDir,
|
||||||
|
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
|
||||||
|
BlockedNetworks: blockedNetworks,
|
||||||
|
// A fetch that gets past the checks and finds nothing at the address
|
||||||
|
// fails within this time at the latest.
|
||||||
|
UpstreamFetchTimeout: 2 * time.Second,
|
||||||
|
// With no connection slots, the fetch would fail before connecting.
|
||||||
|
UpstreamConnections: config.DefaultUpstreamConnections,
|
||||||
|
DownstreamTimeout: config.DefaultDownstreamTimeout,
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
mw *middleware.Middleware
|
||||||
|
)
|
||||||
|
|
||||||
|
app := fxtest.New(t,
|
||||||
|
fx.Supply(cfg),
|
||||||
|
fx.Provide(
|
||||||
|
globals.New,
|
||||||
|
logger.New,
|
||||||
|
database.New,
|
||||||
|
healthcheck.New,
|
||||||
|
handlers.New,
|
||||||
|
middleware.New,
|
||||||
|
),
|
||||||
|
fx.Populate(&h, &mw),
|
||||||
|
)
|
||||||
|
app.RequireStart()
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
s := &Server{config: cfg, mw: mw, h: h}
|
||||||
|
s.SetupRoutes()
|
||||||
|
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
// getImage sends a GET for target to s and fails unless it answers 200.
|
// getImage sends a GET for target to s and fails unless it answers 200.
|
||||||
func getImage(t *testing.T, s *Server, target string) *httptest.ResponseRecorder {
|
func getImage(t *testing.T, s *Server, target string) *httptest.ResponseRecorder {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|||||||
@@ -0,0 +1,269 @@
|
|||||||
|
package server
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/url"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"go.uber.org/fx"
|
||||||
|
"go.uber.org/fx/fxtest"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
|
"sneak.berlin/go/pixa/internal/database"
|
||||||
|
"sneak.berlin/go/pixa/internal/globals"
|
||||||
|
"sneak.berlin/go/pixa/internal/handlers"
|
||||||
|
"sneak.berlin/go/pixa/internal/healthcheck"
|
||||||
|
"sneak.berlin/go/pixa/internal/logger"
|
||||||
|
"sneak.berlin/go/pixa/internal/middleware"
|
||||||
|
"sneak.berlin/go/pixa/internal/session"
|
||||||
|
)
|
||||||
|
|
||||||
|
// logoutFormPattern extracts the CSRF token rendered into the generator
|
||||||
|
// page's logout form.
|
||||||
|
var logoutFormPattern = regexp.MustCompile(
|
||||||
|
`(?s)action="/logout".*?name="gorilla\.csrf\.Token" value="([^"]+)"`)
|
||||||
|
|
||||||
|
// startPixa starts the components pixad's fx app builds, from cfg, and
|
||||||
|
// returns the server with its routes, as newTestServer does.
|
||||||
|
func startPixa(t *testing.T, cfg *config.Config) *Server {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
mw *middleware.Middleware
|
||||||
|
)
|
||||||
|
|
||||||
|
app := fxtest.New(t,
|
||||||
|
fx.Supply(cfg),
|
||||||
|
fx.Provide(
|
||||||
|
globals.New,
|
||||||
|
logger.New,
|
||||||
|
database.New,
|
||||||
|
healthcheck.New,
|
||||||
|
handlers.New,
|
||||||
|
middleware.New,
|
||||||
|
),
|
||||||
|
fx.Populate(&h, &mw),
|
||||||
|
)
|
||||||
|
app.RequireStart()
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
s := &Server{config: cfg, mw: mw, h: h}
|
||||||
|
s.SetupRoutes()
|
||||||
|
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// logIn logs in to s with the signing key, as postLogin does, and returns
|
||||||
|
// the session cookie it sets.
|
||||||
|
func logIn(t *testing.T, s *Server) *http.Cookie {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
rec := postLogin(t, s, firstClient, "", testSigningKey)
|
||||||
|
|
||||||
|
for _, c := range rec.Result().Cookies() {
|
||||||
|
if c.Name == session.CookieName {
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Fatal("logging in set no session cookie")
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// opensGeneratorPage reports whether GET / with cookie shows s's generator
|
||||||
|
// page, which only a valid login session opens.
|
||||||
|
func opensGeneratorPage(t *testing.T, s *Server, cookie *http.Cookie) bool {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
|
||||||
|
req.AddCookie(cookie)
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
return rec.Code == http.StatusOK &&
|
||||||
|
strings.Contains(rec.Body.String(), `action="/generate"`)
|
||||||
|
}
|
||||||
|
|
||||||
|
// postLogout loads the generator page with GET / and sessionCookie and
|
||||||
|
// submits its logout form with POST /logout, as a browser does.
|
||||||
|
func postLogout(
|
||||||
|
t *testing.T, s *Server, sessionCookie *http.Cookie,
|
||||||
|
) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
|
||||||
|
req.AddCookie(sessionCookie)
|
||||||
|
|
||||||
|
page := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(page, req)
|
||||||
|
|
||||||
|
match := logoutFormPattern.FindStringSubmatch(page.Body.String())
|
||||||
|
if match == nil {
|
||||||
|
t.Fatal("the generator page has no logout form with a CSRF token")
|
||||||
|
}
|
||||||
|
|
||||||
|
form := url.Values{"gorilla.csrf.Token": {match[1]}}
|
||||||
|
req = httptest.NewRequestWithContext(t.Context(), http.MethodPost,
|
||||||
|
"/logout", strings.NewReader(form.Encode()))
|
||||||
|
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||||
|
req.AddCookie(sessionCookie)
|
||||||
|
|
||||||
|
for _, c := range page.Result().Cookies() {
|
||||||
|
req.AddCookie(c)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLogoutWithoutCSRFTokenEndsNoSession verifies that a request to /logout
|
||||||
|
// carrying the session cookie but not the logout form's CSRF token ends no
|
||||||
|
// session: GET /logout, as an image on another page of the same site makes a
|
||||||
|
// browser send, answers 405, and POST /logout without the token answers 403.
|
||||||
|
func TestLogoutWithoutCSRFTokenEndsNoSession(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
method string
|
||||||
|
want int
|
||||||
|
}{
|
||||||
|
{http.MethodGet, http.StatusMethodNotAllowed},
|
||||||
|
{http.MethodPost, http.StatusForbidden},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.method, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := newTestServer(t)
|
||||||
|
cookie := logIn(t, s)
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
t.Context(), tc.method, "/logout", nil)
|
||||||
|
req.AddCookie(cookie)
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(rec, req)
|
||||||
|
t.Logf("%s /logout without the CSRF token: status %d",
|
||||||
|
tc.method, rec.Code)
|
||||||
|
|
||||||
|
if rec.Code != tc.want {
|
||||||
|
t.Errorf("%s /logout without the CSRF token status = %d, "+
|
||||||
|
"want %d", tc.method, rec.Code, tc.want)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !opensGeneratorPage(t, s, cookie) {
|
||||||
|
t.Errorf("%s /logout without the CSRF token ended the "+
|
||||||
|
"login session", tc.method)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoginAfterLogoutOpensGeneratorPage verifies that after a logout from
|
||||||
|
// the generator page a copy of the session cookie taken before it is
|
||||||
|
// refused, and that a login after it opens the generator page.
|
||||||
|
func TestLoginAfterLogoutOpensGeneratorPage(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := newTestServer(t)
|
||||||
|
|
||||||
|
copied := logIn(t, s)
|
||||||
|
if !opensGeneratorPage(t, s, copied) {
|
||||||
|
t.Fatal("logging in did not open the generator page")
|
||||||
|
}
|
||||||
|
|
||||||
|
logout := postLogout(t, s, copied)
|
||||||
|
if logout.Code != http.StatusSeeOther {
|
||||||
|
t.Fatalf("POST /logout status = %d, want %d",
|
||||||
|
logout.Code, http.StatusSeeOther)
|
||||||
|
}
|
||||||
|
|
||||||
|
if opensGeneratorPage(t, s, copied) {
|
||||||
|
t.Error("a cookie copied before the logout still opens the " +
|
||||||
|
"generator page")
|
||||||
|
}
|
||||||
|
|
||||||
|
opens := opensGeneratorPage(t, s, logIn(t, s))
|
||||||
|
t.Logf("a login after the logout opens the generator page: %v", opens)
|
||||||
|
|
||||||
|
if !opens {
|
||||||
|
t.Error("a login after the logout does not open the generator page")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLogoutOutlastsRestart verifies that a copy of the session cookie taken
|
||||||
|
// before a logout is still refused by a pixa started again on the same state
|
||||||
|
// directory, as pixad is after a restart.
|
||||||
|
func TestLogoutOutlastsRestart(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := newTestServer(t)
|
||||||
|
|
||||||
|
copied := logIn(t, s)
|
||||||
|
if !opensGeneratorPage(t, s, copied) {
|
||||||
|
t.Fatal("logging in did not open the generator page")
|
||||||
|
}
|
||||||
|
|
||||||
|
logout := postLogout(t, s, copied)
|
||||||
|
if logout.Code != http.StatusSeeOther {
|
||||||
|
t.Fatalf("POST /logout status = %d, want %d",
|
||||||
|
logout.Code, http.StatusSeeOther)
|
||||||
|
}
|
||||||
|
|
||||||
|
restarted := startPixa(t, s.config)
|
||||||
|
|
||||||
|
opens := opensGeneratorPage(t, restarted, copied)
|
||||||
|
t.Logf("after the restart, the copied cookie opens the generator page: %v",
|
||||||
|
opens)
|
||||||
|
|
||||||
|
if opens {
|
||||||
|
t.Error("after a restart, a cookie copied before the logout " +
|
||||||
|
"still opens the generator page")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLostStateDirectoryEndsEveryLoginSession verifies that a pixa started on
|
||||||
|
// an empty state directory, as pixad is after a restart that lost its state
|
||||||
|
// directory, refuses a session cookie from before, and that a login on it
|
||||||
|
// opens the generator page.
|
||||||
|
func TestLostStateDirectoryEndsEveryLoginSession(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := newTestServer(t)
|
||||||
|
|
||||||
|
before := logIn(t, s)
|
||||||
|
if !opensGeneratorPage(t, s, before) {
|
||||||
|
t.Fatal("logging in did not open the generator page")
|
||||||
|
}
|
||||||
|
|
||||||
|
emptyStateDir := *s.config
|
||||||
|
emptyStateDir.StateDir = t.TempDir()
|
||||||
|
emptyStateDir.DBURL = "file:" +
|
||||||
|
filepath.Join(emptyStateDir.StateDir, "state.sqlite3")
|
||||||
|
|
||||||
|
restarted := startPixa(t, &emptyStateDir)
|
||||||
|
|
||||||
|
opens := opensGeneratorPage(t, restarted, before)
|
||||||
|
t.Logf("on an empty state directory, the cookie from before opens the "+
|
||||||
|
"generator page: %v", opens)
|
||||||
|
|
||||||
|
if opens {
|
||||||
|
t.Error("on an empty state directory, a cookie from before still " +
|
||||||
|
"opens the generator page")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !opensGeneratorPage(t, restarted, logIn(t, restarted)) {
|
||||||
|
t.Error("a login on an empty state directory does not open the " +
|
||||||
|
"generator page")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -69,10 +69,9 @@ func (s *Server) SetupRoutes() {
|
|||||||
r.With(s.mw.RateLimit(LoginAttemptsPerMinute, time.Minute, nil)).
|
r.With(s.mw.RateLimit(LoginAttemptsPerMinute, time.Minute, nil)).
|
||||||
Post("/", s.h.HandleRoot())
|
Post("/", s.h.HandleRoot())
|
||||||
r.Post("/generate", s.h.HandleGenerateURL())
|
r.Post("/generate", s.h.HandleGenerateURL())
|
||||||
|
r.Post("/logout", s.h.HandleLogout())
|
||||||
})
|
})
|
||||||
|
|
||||||
s.router.Get("/logout", s.h.HandleLogout())
|
|
||||||
|
|
||||||
// Image routes, the only ones that send CORS headers, as pages on other
|
// Image routes, the only ones that send CORS headers, as pages on other
|
||||||
// sites read them. They are a subrouter rather than a group: a group's
|
// sites read them. They are a subrouter rather than a group: a group's
|
||||||
// middleware runs only for a request that matches one of its routes,
|
// middleware runs only for a request that matches one of its routes,
|
||||||
|
|||||||
@@ -0,0 +1,665 @@
|
|||||||
|
package server
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"net/url"
|
||||||
|
"regexp"
|
||||||
|
"slices"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/encurl"
|
||||||
|
"sneak.berlin/go/pixa/internal/imgcache"
|
||||||
|
"sneak.berlin/go/pixa/internal/session"
|
||||||
|
"sneak.berlin/go/pixa/internal/signature"
|
||||||
|
)
|
||||||
|
|
||||||
|
// percentSignPath is a source path holding a "%", which a URL writes %25.
|
||||||
|
const percentSignPath = "/100%25.png"
|
||||||
|
|
||||||
|
// generateFormPattern extracts the CSRF token rendered into the generator
|
||||||
|
// page's form.
|
||||||
|
var generateFormPattern = regexp.MustCompile(
|
||||||
|
`(?s)action="/generate".*?name="gorilla\.csrf\.Token" value="([^"]+)"`)
|
||||||
|
|
||||||
|
// generatedURLPattern extracts the path of the encrypted URL the generator
|
||||||
|
// page shows.
|
||||||
|
var generatedURLPattern = regexp.MustCompile(`value="[^"]*(/v1/e/[^"]+)"`)
|
||||||
|
|
||||||
|
// recordingUpstream is a test upstream that answers every request with a PNG
|
||||||
|
// and records the request target, the path and query as sent, of each request
|
||||||
|
// it receives.
|
||||||
|
type recordingUpstream struct {
|
||||||
|
*httptest.Server
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
received []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// startRecordingUpstream starts a recordingUpstream, which is closed when the
|
||||||
|
// test ends.
|
||||||
|
func startRecordingUpstream(t *testing.T) *recordingUpstream {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
source := encodeTestPNG(t, 64, 48)
|
||||||
|
upstream := &recordingUpstream{}
|
||||||
|
|
||||||
|
upstream.Server = httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
upstream.mu.Lock()
|
||||||
|
upstream.received = append(upstream.received, r.RequestURI)
|
||||||
|
upstream.mu.Unlock()
|
||||||
|
|
||||||
|
w.Header().Set("Content-Type", "image/png")
|
||||||
|
_, _ = w.Write(source)
|
||||||
|
}))
|
||||||
|
t.Cleanup(upstream.Close)
|
||||||
|
|
||||||
|
return upstream
|
||||||
|
}
|
||||||
|
|
||||||
|
// requestTargets returns the request targets the upstream has received, in
|
||||||
|
// the order it received them.
|
||||||
|
func (u *recordingUpstream) requestTargets() []string {
|
||||||
|
u.mu.Lock()
|
||||||
|
defer u.mu.Unlock()
|
||||||
|
|
||||||
|
return slices.Clone(u.received)
|
||||||
|
}
|
||||||
|
|
||||||
|
// pageOrigin is the origin of the login and generator pages below, opened
|
||||||
|
// over HTTPS; httptest gives every request the host example.com.
|
||||||
|
const pageOrigin = "https://example.com"
|
||||||
|
|
||||||
|
// openRoot sends GET / to s over HTTPS, with cookie when it is not nil, and
|
||||||
|
// returns the page: the login page, or the generator page for a login session.
|
||||||
|
func openRoot(
|
||||||
|
t *testing.T, s *Server, cookie *http.Cookie,
|
||||||
|
) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
|
||||||
|
pageOrigin+"/", nil)
|
||||||
|
if cookie != nil {
|
||||||
|
req.AddCookie(cookie)
|
||||||
|
}
|
||||||
|
|
||||||
|
page := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(page, req)
|
||||||
|
|
||||||
|
return page
|
||||||
|
}
|
||||||
|
|
||||||
|
// submitForm posts form to path on s as a browser submits it from page over
|
||||||
|
// HTTPS: with the CSRF token tokenPattern finds in page, the cookies page set,
|
||||||
|
// cookie when it is not nil, and a Referer naming the page, which pixa needs
|
||||||
|
// while debug is off, as it is in startImageProxy's config.
|
||||||
|
func submitForm(
|
||||||
|
t *testing.T, s *Server, page *httptest.ResponseRecorder,
|
||||||
|
tokenPattern *regexp.Regexp, path string, form url.Values,
|
||||||
|
cookie *http.Cookie,
|
||||||
|
) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
match := tokenPattern.FindStringSubmatch(page.Body.String())
|
||||||
|
if match == nil {
|
||||||
|
t.Fatalf("the page has no form for %s with a CSRF token", path)
|
||||||
|
}
|
||||||
|
|
||||||
|
form.Set("gorilla.csrf.Token", match[1])
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), http.MethodPost,
|
||||||
|
pageOrigin+path, strings.NewReader(form.Encode()))
|
||||||
|
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||||
|
req.Header.Set("Referer", pageOrigin+"/")
|
||||||
|
|
||||||
|
for _, c := range page.Result().Cookies() {
|
||||||
|
req.AddCookie(c)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cookie != nil {
|
||||||
|
req.AddCookie(cookie)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
// postGenerate logs in to s with the signing key, submits the generator page's
|
||||||
|
// form with sourceURL, and returns the response.
|
||||||
|
func postGenerate(
|
||||||
|
t *testing.T, s *Server, sourceURL string,
|
||||||
|
) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
login := submitForm(t, s, openRoot(t, s, nil), csrfFieldPattern, "/",
|
||||||
|
url.Values{"key": {testSigningKey}}, nil)
|
||||||
|
|
||||||
|
var sessionCookie *http.Cookie
|
||||||
|
|
||||||
|
for _, c := range login.Result().Cookies() {
|
||||||
|
if c.Name == session.CookieName {
|
||||||
|
sessionCookie = c
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if sessionCookie == nil {
|
||||||
|
t.Fatalf("logging in answered %d and set no session cookie", login.Code)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := submitForm(t, s, openRoot(t, s, sessionCookie), generateFormPattern,
|
||||||
|
"/generate", url.Values{"url": {sourceURL}}, sessionCookie)
|
||||||
|
t.Logf("POST /generate with url %s: %d", sourceURL, rec.Code)
|
||||||
|
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestImageURLFetchesItsSourcePathAndQuery requests image URLs whose source
|
||||||
|
// query holds a space, written percent-encoded, or a "'", and whose source
|
||||||
|
// path holds a "%", written percent-encoded, and checks the one request the
|
||||||
|
// upstream receives: the source path percent-encoded again, and the source
|
||||||
|
// query with the space written %20 and the "'" written %27, as a browser
|
||||||
|
// writes them in a query.
|
||||||
|
func TestImageURLFetchesItsSourcePathAndQuery(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
// pathAndQuery is the source path and query as the image URL writes
|
||||||
|
// them
|
||||||
|
pathAndQuery string
|
||||||
|
// wantRequest is the path and query the upstream must receive
|
||||||
|
wantRequest string
|
||||||
|
}{
|
||||||
|
{"space in the query", "/photo.png%3Fname=my%20cat",
|
||||||
|
"/photo.png?name=my%20cat"},
|
||||||
|
{"apostrophe in the query", "/photo.png%3Fname=it's",
|
||||||
|
"/photo.png?name=it%27s"},
|
||||||
|
{"percent sign in the path", percentSignPath, percentSignPath},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
upstream := startRecordingUpstream(t)
|
||||||
|
s, _, _ := startImageProxy(t, upstream.Server)
|
||||||
|
|
||||||
|
getImage(t, s, "/v1/image/"+upstreamHost+tt.pathAndQuery+"/orig.png")
|
||||||
|
|
||||||
|
got := upstream.requestTargets()
|
||||||
|
t.Logf("upstream received %q", got)
|
||||||
|
|
||||||
|
if !slices.Equal(got, []string{tt.wantRequest}) {
|
||||||
|
t.Errorf("upstream received %q, want [%q]", got, tt.wantRequest)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart requests twice each
|
||||||
|
// of the source paths /my cat.png and /my%20cat.png, which the image URL
|
||||||
|
// writes my%20cat.png and my%2520cat.png. Each is fetched once, as a request
|
||||||
|
// of its own, and its second request is served from the cache.
|
||||||
|
func TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
upstream := startRecordingUpstream(t)
|
||||||
|
s, _, _ := startImageProxy(t, upstream.Server)
|
||||||
|
|
||||||
|
imageURLs := []string{
|
||||||
|
"/v1/image/" + upstreamHost + "/my%20cat.png/orig.png",
|
||||||
|
"/v1/image/" + upstreamHost + "/my%2520cat.png/orig.png",
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, imageURL := range imageURLs {
|
||||||
|
getImage(t, s, imageURL)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, imageURL := range imageURLs {
|
||||||
|
cacheStatus := getImage(t, s, imageURL).Header().Get("X-Pixa-Cache")
|
||||||
|
if cacheStatus != string(imgcache.CacheHit) {
|
||||||
|
t.Errorf("second GET %s: X-Pixa-Cache = %q, want HIT",
|
||||||
|
imageURL, cacheStatus)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
got := upstream.requestTargets()
|
||||||
|
t.Logf("upstream received %q", got)
|
||||||
|
|
||||||
|
want := []string{"/my%20cat.png", "/my%2520cat.png"}
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("upstream received %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest requests the source query
|
||||||
|
// name=my%20cat written as the README says, name=my%2520cat, and with the
|
||||||
|
// space written %20, name=my%20cat. Both are one source query: the upstream
|
||||||
|
// receives one request, and the second image URL is served from the cache.
|
||||||
|
func TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
upstream := startRecordingUpstream(t)
|
||||||
|
s, _, _ := startImageProxy(t, upstream.Server)
|
||||||
|
|
||||||
|
getImage(t, s, "/v1/image/"+upstreamHost+"/photo.png%3Fname=my%2520cat/orig.png")
|
||||||
|
|
||||||
|
second := getImage(t, s,
|
||||||
|
"/v1/image/"+upstreamHost+"/photo.png%3Fname=my%20cat/orig.png")
|
||||||
|
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
|
||||||
|
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
got := upstream.requestTargets()
|
||||||
|
t.Logf("upstream received %q", got)
|
||||||
|
|
||||||
|
want := []string{"/photo.png?name=my%20cat"}
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("upstream received %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGeneratedURLFetchesTheSourceURLAsEntered makes an encrypted URL on the
|
||||||
|
// generator page for source URLs holding %20, %25 and %23 in the path or the
|
||||||
|
// query, requests it, and checks that the upstream receives the path and
|
||||||
|
// query exactly as entered.
|
||||||
|
func TestGeneratedURLFetchesTheSourceURLAsEntered(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// The names hold no "%": each test's state directory is named after it
|
||||||
|
// and goes into the database URL, which decodes percent-escapes
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
pathAndQuery string
|
||||||
|
}{
|
||||||
|
{"space in the path", "/my%20cat.png"},
|
||||||
|
{"percent sign in the path", percentSignPath},
|
||||||
|
{"number sign in the path", "/a%23b.png"},
|
||||||
|
{"all three in the query", "/photo.png?name=my%20cat%25%23"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
upstream := startRecordingUpstream(t)
|
||||||
|
s, _, _ := startImageProxy(t, upstream.Server)
|
||||||
|
|
||||||
|
rec := postGenerate(t, s, "https://"+upstreamHost+tt.pathAndQuery)
|
||||||
|
|
||||||
|
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
|
||||||
|
if rec.Code != http.StatusOK || match == nil {
|
||||||
|
t.Fatalf("POST /generate: %d with no encrypted URL; body %s",
|
||||||
|
rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
getImage(t, s, match[1])
|
||||||
|
|
||||||
|
got := upstream.requestTargets()
|
||||||
|
t.Logf("upstream received %q", got)
|
||||||
|
|
||||||
|
if !slices.Equal(got, []string{tt.pathAndQuery}) {
|
||||||
|
t.Errorf("upstream received %q, want [%q]", got, tt.pathAndQuery)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL enters two
|
||||||
|
// spellings of one source URL on the generator page, requests the encrypted
|
||||||
|
// URL made for each, and checks that the upstream receives one request and
|
||||||
|
// that the second encrypted URL is served from the cache. One pair writes the
|
||||||
|
// host in upper and in lower case, which /v1/image/ lower-cases. The host is
|
||||||
|
// an IPv6 address, whose hex digits have letter case, because the fetcher
|
||||||
|
// would look a host name up in DNS. It has a port, as the fetcher refuses an
|
||||||
|
// IPv6 address written without one. The other pair is a source URL with no
|
||||||
|
// path and with the path "/", both of which fetch "/".
|
||||||
|
func TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
// sourceURLs are the two spellings, after "https://"
|
||||||
|
sourceURLs []string
|
||||||
|
// wantRequest is the path the upstream must receive
|
||||||
|
wantRequest string
|
||||||
|
}{
|
||||||
|
{"host in upper and in lower case",
|
||||||
|
[]string{"[2001:DB8::A]:8443/a.png", "[2001:db8::a]:8443/a.png"},
|
||||||
|
"/a.png"},
|
||||||
|
{"no path and the path written as a slash",
|
||||||
|
[]string{upstreamHost, upstreamHost + "/"}, "/"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
upstream := startRecordingUpstream(t)
|
||||||
|
s, _, _ := startImageProxy(t, upstream.Server)
|
||||||
|
|
||||||
|
var imageURLs []string
|
||||||
|
|
||||||
|
for _, sourceURL := range tt.sourceURLs {
|
||||||
|
rec := postGenerate(t, s, "https://"+sourceURL)
|
||||||
|
|
||||||
|
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
|
||||||
|
if rec.Code != http.StatusOK || match == nil {
|
||||||
|
t.Fatalf("POST /generate: %d with no encrypted URL; body %s",
|
||||||
|
rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
imageURLs = append(imageURLs, match[1])
|
||||||
|
}
|
||||||
|
|
||||||
|
getImage(t, s, imageURLs[0])
|
||||||
|
|
||||||
|
second := getImage(t, s, imageURLs[1])
|
||||||
|
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
|
||||||
|
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
got := upstream.requestTargets()
|
||||||
|
t.Logf("upstream received %q", got)
|
||||||
|
|
||||||
|
if !slices.Equal(got, []string{tt.wantRequest}) {
|
||||||
|
t.Errorf("upstream received %q, want [%q]", got, tt.wantRequest)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGeneratorRefusesSourceURLItCannotFetchAsWritten submits source URLs the
|
||||||
|
// generator page must make no encrypted URL for, and checks that it answers
|
||||||
|
// 400 and shows why. pixa keeps the source path decoded and fetches it
|
||||||
|
// percent-encoded again, so a path holding %2F would be fetched with a "/"
|
||||||
|
// there, whatever else the path holds. pixa sends no user name or password,
|
||||||
|
// and no "?" with no query after it. For these the page shows the URL pixa
|
||||||
|
// would fetch. A source path holding a ".." segment is refused, as
|
||||||
|
// /v1/image/ refuses it.
|
||||||
|
func TestGeneratorRefusesSourceURLItCannotFetchAsWritten(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// The test config sets allow_http, so pixa would fetch over plain HTTP
|
||||||
|
const fetchedFrom = "http://" + upstreamHost
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
// sourceURL is the source URL after "https://"
|
||||||
|
sourceURL string
|
||||||
|
// wantShown is what the page must show: the URL pixa would fetch, or
|
||||||
|
// why it makes no encrypted URL
|
||||||
|
wantShown string
|
||||||
|
}{
|
||||||
|
{"encoded slash", upstreamHost + "/a%2Fb.png",
|
||||||
|
fetchedFrom + "/a/b.png"},
|
||||||
|
{"encoded slash and a space", upstreamHost + "/a%2Fb c.png",
|
||||||
|
fetchedFrom + "/a/b%20c.png"},
|
||||||
|
{"encoded slash and a bar", upstreamHost + "/a%2Fb|c.png",
|
||||||
|
fetchedFrom + "/a/b%7Cc.png"},
|
||||||
|
{"encoded slash and a letter outside ASCII", upstreamHost + "/café%2Fx.png",
|
||||||
|
fetchedFrom + "/caf%C3%A9/x.png"},
|
||||||
|
{"user name and password", "user:pass@" + upstreamHost + "/a.png",
|
||||||
|
fetchedFrom + "/a.png"},
|
||||||
|
{"question mark with no query", upstreamHost + "/a.png?",
|
||||||
|
fetchedFrom + "/a.png"},
|
||||||
|
{"two-dot segment", upstreamHost + "/a/../b.png",
|
||||||
|
"cannot hold a .. segment"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s, _, _ := startImageProxy(t, startRecordingUpstream(t).Server)
|
||||||
|
|
||||||
|
rec := postGenerate(t, s, "https://"+tt.sourceURL)
|
||||||
|
body := rec.Body.String()
|
||||||
|
|
||||||
|
if rec.Code != http.StatusBadRequest {
|
||||||
|
t.Errorf("status = %d, want %d", rec.Code, http.StatusBadRequest)
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.Contains(body, "/v1/e/") {
|
||||||
|
t.Error("the page shows an encrypted URL")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(body, tt.wantShown) {
|
||||||
|
t.Errorf("the page does not show %s; body %s", tt.wantShown, body)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ipv6SourcePath is the source path of the image URLs signedImageURL and
|
||||||
|
// encryptedImageURL make.
|
||||||
|
const ipv6SourcePath = "/ipv6.png"
|
||||||
|
|
||||||
|
// TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort requests ipv6SourcePath
|
||||||
|
// on an IPv6 source host, written with and without a port, through a
|
||||||
|
// /v1/image/ URL with a signature and through an encrypted URL. The public
|
||||||
|
// address 2001:db8::a, a documentation address (RFC 3849) the fetcher accepts
|
||||||
|
// as public, answers 200, and the upstream receives the request for
|
||||||
|
// ipv6SourcePath. The private address fd00::1 answers 403, and the upstream
|
||||||
|
// receives no request: the test's fetcher connects every address to the test
|
||||||
|
// upstream, so only the fetcher's check of the source URL can refuse it.
|
||||||
|
func TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
sourceHost string
|
||||||
|
wantStatus int
|
||||||
|
// wantRequests are the request targets the upstream must receive
|
||||||
|
wantRequests []string
|
||||||
|
}{
|
||||||
|
{"public address without a port", "[2001:db8::a]",
|
||||||
|
http.StatusOK, []string{ipv6SourcePath}},
|
||||||
|
{"public address with a port", "[2001:db8::a]:8443",
|
||||||
|
http.StatusOK, []string{ipv6SourcePath}},
|
||||||
|
{"private address without a port", "[fd00::1]",
|
||||||
|
http.StatusForbidden, nil},
|
||||||
|
{"private address with a port", "[fd00::1]:8443",
|
||||||
|
http.StatusForbidden, nil},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
imageURLs := map[string]string{
|
||||||
|
"image URL with a signature": signedImageURL(t, tt.sourceHost),
|
||||||
|
"encrypted URL": encryptedImageURL(t, tt.sourceHost),
|
||||||
|
}
|
||||||
|
|
||||||
|
for route, imageURL := range imageURLs {
|
||||||
|
t.Run(tt.name+" "+route, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
upstream := startRecordingUpstream(t)
|
||||||
|
s, _, _ := startImageProxy(t, upstream.Server)
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, imageURL, nil))
|
||||||
|
t.Logf("GET %s: %d", imageURL, rec.Code)
|
||||||
|
|
||||||
|
if rec.Code != tt.wantStatus {
|
||||||
|
t.Errorf("status = %d, want %d; body %s",
|
||||||
|
rec.Code, tt.wantStatus, rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
got := upstream.requestTargets()
|
||||||
|
t.Logf("upstream received %q", got)
|
||||||
|
|
||||||
|
if !slices.Equal(got, tt.wantRequests) {
|
||||||
|
t.Errorf("upstream received %q, want %q", got, tt.wantRequests)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// signedImageURL returns the /v1/image/ URL of ipv6SourcePath on sourceHost at
|
||||||
|
// its original size as PNG, with a signature made with the test signing key,
|
||||||
|
// which a host not on the allowlist needs. The URL has no q or fit, so it is
|
||||||
|
// signed with the quality 85 and the fit cover.
|
||||||
|
func signedImageURL(t *testing.T, sourceHost string) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
path, sig, exp := signature.New(testSigningKey).GenerateSignedURL(
|
||||||
|
&signature.Request{
|
||||||
|
SourceHost: sourceHost,
|
||||||
|
SourcePath: ipv6SourcePath,
|
||||||
|
Format: string(imgcache.FormatPNG),
|
||||||
|
Quality: 85,
|
||||||
|
FitMode: string(imgcache.FitCover),
|
||||||
|
}, time.Hour)
|
||||||
|
|
||||||
|
return path + "?sig=" + sig + "&exp=" + strconv.FormatInt(exp, 10)
|
||||||
|
}
|
||||||
|
|
||||||
|
// encryptedImageURL returns the encrypted URL of ipv6SourcePath on sourceHost
|
||||||
|
// at its original size as PNG, made with the test signing key.
|
||||||
|
func encryptedImageURL(t *testing.T, sourceHost string) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
generator, err := encurl.NewGenerator(testSigningKey)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("encurl.NewGenerator: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
token, err := generator.Generate(&encurl.Payload{
|
||||||
|
SourceHost: sourceHost, SourcePath: ipv6SourcePath,
|
||||||
|
Format: imgcache.FormatPNG,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("encurl Generate: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return "/v1/e/" + token + "/img.png"
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address requests
|
||||||
|
// ipv6SourcePath on IPv6 source hosts that hold a loopback or private IPv4
|
||||||
|
// address in their last four bytes, in the IPv4-compatible form (::/96) or the
|
||||||
|
// NAT64 form (64:ff9b::/96), through a /v1/image/ URL with a signature and
|
||||||
|
// through an encrypted URL. Each answers 403, and the upstream receives no
|
||||||
|
// request: the test's fetcher connects every address to the test upstream, so
|
||||||
|
// only the fetcher's check of the source URL can refuse it.
|
||||||
|
func TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
upstream := startRecordingUpstream(t)
|
||||||
|
s, _, _ := startImageProxy(t, upstream.Server)
|
||||||
|
|
||||||
|
sourceHosts := []string{
|
||||||
|
"[::127.0.0.1]", // IPv4-compatible form of a loopback address
|
||||||
|
"[::10.0.0.1]", // IPv4-compatible form of a private address
|
||||||
|
"[64:ff9b::7f00:1]", // NAT64 form of 127.0.0.1
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, sourceHost := range sourceHosts {
|
||||||
|
imageURLs := []string{
|
||||||
|
signedImageURL(t, sourceHost),
|
||||||
|
encryptedImageURL(t, sourceHost),
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, imageURL := range imageURLs {
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, imageURL, nil))
|
||||||
|
t.Logf("GET %s (source host %s): %d", imageURL, sourceHost, rec.Code)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusForbidden {
|
||||||
|
t.Errorf("source host %s: status = %d, want %d; body %s",
|
||||||
|
sourceHost, rec.Code, http.StatusForbidden, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := upstream.requestTargets(); len(got) != 0 {
|
||||||
|
t.Errorf("upstream received %q, want no request", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestImageRoutesCheckIPv4AddressInIPv6SourceHost requests ipv6SourcePath on
|
||||||
|
// source hosts that hold an IPv4 address, through a /v1/image/ URL with a
|
||||||
|
// signature and through an encrypted URL, with the fetcher pixad uses.
|
||||||
|
// [64:ff9b:1::a00:1], in the local-use NAT64 range (64:ff9b:1::/48, RFC 8215),
|
||||||
|
// answers 403. [::203.0.113.5] holds the public address 203.0.113.5 in the
|
||||||
|
// IPv4-compatible form (::/96) and answers 403 when blocked_networks lists
|
||||||
|
// 203.0.113.0/24 or ::/96. [::ffff:203.0.113.5] holds it in the IPv4-mapped
|
||||||
|
// form (::ffff:0:0/96) and answers 403 when blocked_networks lists
|
||||||
|
// 203.0.113.0/24. Go keeps the plain address 203.0.113.5 in the mapped form
|
||||||
|
// too, and an IPv6 range must not refuse it: with ::/0 listed it is not
|
||||||
|
// refused. Neither is [::203.0.113.5] with blocked_networks empty. Where a
|
||||||
|
// host is not refused, the fetcher tries to connect to it, finds nothing
|
||||||
|
// there, and the request fails with another status.
|
||||||
|
func TestImageRoutesCheckIPv4AddressInIPv6SourceHost(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
ipv4CompatibleHost = "[::203.0.113.5]"
|
||||||
|
ipv4MappedHost = "[::ffff:203.0.113.5]"
|
||||||
|
)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
sourceHost string
|
||||||
|
blockedNetworks []netip.Prefix
|
||||||
|
wantForbidden bool
|
||||||
|
}{
|
||||||
|
{"local-use NAT64 form of 10.0.0.1", "[64:ff9b:1::a00:1]", nil, true},
|
||||||
|
{"IPv4-compatible form of an address in blocked_networks",
|
||||||
|
ipv4CompatibleHost,
|
||||||
|
[]netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")}, true},
|
||||||
|
{"IPv4-compatible form with ::/96 in blocked_networks",
|
||||||
|
ipv4CompatibleHost,
|
||||||
|
[]netip.Prefix{netip.MustParsePrefix("::/96")}, true},
|
||||||
|
{"IPv4-mapped form of an address in blocked_networks",
|
||||||
|
ipv4MappedHost,
|
||||||
|
[]netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")}, true},
|
||||||
|
{"plain IPv4 address with ::/0 in blocked_networks",
|
||||||
|
"203.0.113.5",
|
||||||
|
[]netip.Prefix{netip.MustParsePrefix("::/0")}, false},
|
||||||
|
{"IPv4-compatible form of an address with blocked_networks empty",
|
||||||
|
ipv4CompatibleHost, nil, false},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
imageURLs := map[string]string{
|
||||||
|
"image URL with a signature": signedImageURL(t, tt.sourceHost),
|
||||||
|
"encrypted URL": encryptedImageURL(t, tt.sourceHost),
|
||||||
|
}
|
||||||
|
|
||||||
|
for route, imageURL := range imageURLs {
|
||||||
|
t.Run(tt.name+" "+route, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
s := startImageProxyWithBlockedNetworks(t, tt.blockedNetworks)
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, imageURL, nil))
|
||||||
|
t.Logf("GET %s: %d", imageURL, rec.Code)
|
||||||
|
|
||||||
|
forbidden := rec.Code == http.StatusForbidden
|
||||||
|
if tt.wantForbidden && !forbidden {
|
||||||
|
t.Errorf("status = %d, want %d; body %s",
|
||||||
|
rec.Code, http.StatusForbidden, rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
if !tt.wantForbidden && forbidden {
|
||||||
|
t.Errorf("status = %d, want any other status; body %s",
|
||||||
|
rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
+116
-2
@@ -1,9 +1,16 @@
|
|||||||
// Package session provides encrypted session cookie management.
|
// Package session provides encrypted session cookie management, and keeps in
|
||||||
|
// the database a random value that each logout replaces, so that a logout
|
||||||
|
// ends every session made before it.
|
||||||
package session
|
package session
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"database/sql"
|
||||||
"errors"
|
"errors"
|
||||||
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/gorilla/securecookie"
|
"github.com/gorilla/securecookie"
|
||||||
@@ -32,14 +39,36 @@ type Data struct {
|
|||||||
Authenticated bool `json:"auth"`
|
Authenticated bool `json:"auth"`
|
||||||
CreatedAt time.Time `json:"created"`
|
CreatedAt time.Time `json:"created"`
|
||||||
ExpiresAt time.Time `json:"expires"`
|
ExpiresAt time.Time `json:"expires"`
|
||||||
|
|
||||||
|
// LogoutValue is the Manager's random value, which each logout
|
||||||
|
// replaces, when the session was made.
|
||||||
|
LogoutValue string `json:"logoutValue"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Manager handles session creation and validation using encrypted cookies.
|
// Manager handles session creation and validation using encrypted cookies.
|
||||||
|
//
|
||||||
|
// Every login is made with the one signing key, so a logout ends every
|
||||||
|
// session made before it, in every browser. The Manager keeps a random
|
||||||
|
// value, which each logout replaces: a session carries the value current at
|
||||||
|
// its login, and a session carrying another value is refused. The check for
|
||||||
|
// a logout compares no time, so a login after a logout works whatever the
|
||||||
|
// host's clock does; the session's expiry is still checked against the
|
||||||
|
// clock. The Manager keeps the value in memory, so checking a session reads
|
||||||
|
// no database; a Manager from NewManagerWithDatabase also saves it in the
|
||||||
|
// database, so that it outlasts a restart.
|
||||||
type Manager struct {
|
type Manager struct {
|
||||||
sc *securecookie.SecureCookie
|
sc *securecookie.SecureCookie
|
||||||
|
db *sql.DB
|
||||||
|
|
||||||
|
// mu guards logoutValue, and is held while a logout is saved, so that
|
||||||
|
// the database and logoutValue agree.
|
||||||
|
mu sync.Mutex
|
||||||
|
logoutValue string
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewManager creates a session manager with keys derived from the signing key.
|
// NewManager creates a session manager with keys derived from the signing key.
|
||||||
|
// It makes a new random value and keeps it in memory only, so a restart ends
|
||||||
|
// every session; NewManagerWithDatabase keeps the value in the database.
|
||||||
//
|
//
|
||||||
// Session cookies always carry the Secure, HttpOnly, and SameSite=Strict
|
// Session cookies always carry the Secure, HttpOnly, and SameSite=Strict
|
||||||
// attributes; this cannot be configured. Browsers treat http://localhost as a
|
// attributes; this cannot be configured. Browsers treat http://localhost as a
|
||||||
@@ -63,10 +92,44 @@ func NewManager(signingKey string) (*Manager, error) {
|
|||||||
sc.MaxAge(int(SessionTTL.Seconds()))
|
sc.MaxAge(int(SessionTTL.Seconds()))
|
||||||
|
|
||||||
return &Manager{
|
return &Manager{
|
||||||
sc: sc,
|
sc: sc,
|
||||||
|
logoutValue: rand.Text(),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// NewManagerWithDatabase creates a session manager like NewManager that keeps
|
||||||
|
// its random value in db, which must have pixa's schema: it saves the value
|
||||||
|
// NewManager made when db has none, and otherwise starts from the one saved
|
||||||
|
// there.
|
||||||
|
func NewManagerWithDatabase(
|
||||||
|
ctx context.Context, signingKey string, db *sql.DB,
|
||||||
|
) (*Manager, error) {
|
||||||
|
m, err := NewManager(signingKey)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = db.ExecContext(ctx,
|
||||||
|
"INSERT OR IGNORE INTO logout (id, logout_value) VALUES (1, ?)",
|
||||||
|
m.logoutValue)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf(
|
||||||
|
"failed to save the random value in the logout table: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = db.QueryRowContext(ctx,
|
||||||
|
"SELECT logout_value FROM logout WHERE id = 1",
|
||||||
|
).Scan(&m.logoutValue)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf(
|
||||||
|
"failed to read the random value in the logout table: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
m.db = db
|
||||||
|
|
||||||
|
return m, nil
|
||||||
|
}
|
||||||
|
|
||||||
// CreateSession creates a new authenticated session and sets the cookie.
|
// CreateSession creates a new authenticated session and sets the cookie.
|
||||||
func (m *Manager) CreateSession(w http.ResponseWriter) error {
|
func (m *Manager) CreateSession(w http.ResponseWriter) error {
|
||||||
now := time.Now()
|
now := time.Now()
|
||||||
@@ -74,6 +137,7 @@ func (m *Manager) CreateSession(w http.ResponseWriter) error {
|
|||||||
Authenticated: true,
|
Authenticated: true,
|
||||||
CreatedAt: now,
|
CreatedAt: now,
|
||||||
ExpiresAt: now.Add(SessionTTL),
|
ExpiresAt: now.Add(SessionTTL),
|
||||||
|
LogoutValue: m.currentLogoutValue(),
|
||||||
}
|
}
|
||||||
|
|
||||||
encoded, err := m.sc.Encode(CookieName, data)
|
encoded, err := m.sc.Encode(CookieName, data)
|
||||||
@@ -122,9 +186,30 @@ func (m *Manager) ValidateSession(r *http.Request) (*Data, error) {
|
|||||||
return nil, ErrInvalidSession
|
return nil, ErrInvalidSession
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A logout since the session was made has ended it.
|
||||||
|
if data.LogoutValue != m.currentLogoutValue() {
|
||||||
|
return nil, ErrInvalidSession
|
||||||
|
}
|
||||||
|
|
||||||
return &data, nil
|
return &data, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Logout removes the session cookie and, when r has a valid session, ends
|
||||||
|
// every session made until now. A request without one ends no session, so
|
||||||
|
// someone who has not logged in cannot end the sessions of those who have.
|
||||||
|
func (m *Manager) Logout(w http.ResponseWriter, r *http.Request) error {
|
||||||
|
if m.IsAuthenticated(r) {
|
||||||
|
err := m.saveLogout(r.Context())
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
m.ClearSession(w)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// ClearSession removes the session cookie.
|
// ClearSession removes the session cookie.
|
||||||
func (m *Manager) ClearSession(w http.ResponseWriter) {
|
func (m *Manager) ClearSession(w http.ResponseWriter) {
|
||||||
http.SetCookie(w, &http.Cookie{
|
http.SetCookie(w, &http.Cookie{
|
||||||
@@ -145,3 +230,32 @@ func (m *Manager) IsAuthenticated(r *http.Request) bool {
|
|||||||
|
|
||||||
return err == nil && data != nil && data.Authenticated
|
return err == nil && data != nil && data.Authenticated
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// saveLogout replaces the random value with a new one, saving it in the
|
||||||
|
// database first when the Manager has one.
|
||||||
|
func (m *Manager) saveLogout(ctx context.Context) error {
|
||||||
|
value := rand.Text()
|
||||||
|
|
||||||
|
m.mu.Lock()
|
||||||
|
defer m.mu.Unlock()
|
||||||
|
|
||||||
|
if m.db != nil {
|
||||||
|
_, err := m.db.ExecContext(ctx,
|
||||||
|
"UPDATE logout SET logout_value = ? WHERE id = 1", value)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to save the logout: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
m.logoutValue = value
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// currentLogoutValue returns the Manager's random value.
|
||||||
|
func (m *Manager) currentLogoutValue() string {
|
||||||
|
m.mu.Lock()
|
||||||
|
defer m.mu.Unlock()
|
||||||
|
|
||||||
|
return m.logoutValue
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,148 @@
|
|||||||
|
package session_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"database/sql"
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/database"
|
||||||
|
"sneak.berlin/go/pixa/internal/session"
|
||||||
|
)
|
||||||
|
|
||||||
|
const logoutTestSigningKey = "test-signing-key-12345"
|
||||||
|
|
||||||
|
// newTestDatabase opens an empty in-memory database with pixa's schema. It
|
||||||
|
// has one connection, as pixad's has: each connection to ":memory:" would
|
||||||
|
// open a database of its own.
|
||||||
|
func newTestDatabase(t *testing.T) *sql.DB {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
db, err := sql.Open("sqlite", ":memory:")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("sql.Open() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
db.SetMaxOpenConns(1)
|
||||||
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
|
||||||
|
err = database.ApplyMigrations(t.Context(), db, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ApplyMigrations() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return db
|
||||||
|
}
|
||||||
|
|
||||||
|
// createSessionCookie logs in with mgr and returns the session cookie it
|
||||||
|
// sets.
|
||||||
|
func createSessionCookie(t *testing.T, mgr *session.Manager) *http.Cookie {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
err := mgr.CreateSession(rec)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateSession() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, c := range rec.Result().Cookies() {
|
||||||
|
if c.Name == session.CookieName {
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Fatalf("CreateSession() did not set a %q cookie", session.CookieName)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// requestWithCookie returns a GET request carrying cookie, or no cookie when
|
||||||
|
// cookie is nil.
|
||||||
|
func requestWithCookie(t *testing.T, cookie *http.Cookie) *http.Request {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
|
||||||
|
if cookie != nil {
|
||||||
|
req.AddCookie(cookie)
|
||||||
|
}
|
||||||
|
|
||||||
|
return req
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLogout_EndsEverySessionMadeBeforeIt verifies that after a logout a
|
||||||
|
// copy of the session cookie taken before it is refused and a login after it
|
||||||
|
// works, and that both still hold for a new Manager on the same database, as
|
||||||
|
// after a restart.
|
||||||
|
func TestLogout_EndsEverySessionMadeBeforeIt(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
db := newTestDatabase(t)
|
||||||
|
|
||||||
|
mgr, err := session.NewManagerWithDatabase(
|
||||||
|
t.Context(), logoutTestSigningKey, db)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewManagerWithDatabase() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
copied := createSessionCookie(t, mgr)
|
||||||
|
|
||||||
|
err = mgr.Logout(httptest.NewRecorder(), requestWithCookie(t, copied))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Logout() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = mgr.ValidateSession(requestWithCookie(t, copied))
|
||||||
|
t.Logf("cookie copied before the logout: ValidateSession() error = %v", err)
|
||||||
|
|
||||||
|
if !errors.Is(err, session.ErrInvalidSession) {
|
||||||
|
t.Errorf("cookie copied before the logout: error = %v, want %v",
|
||||||
|
err, session.ErrInvalidSession)
|
||||||
|
}
|
||||||
|
|
||||||
|
loginAfter := createSessionCookie(t, mgr)
|
||||||
|
|
||||||
|
_, err = mgr.ValidateSession(requestWithCookie(t, loginAfter))
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("login after the logout: ValidateSession() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
restarted, err := session.NewManagerWithDatabase(
|
||||||
|
t.Context(), logoutTestSigningKey, db)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewManagerWithDatabase() after restart error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if restarted.IsAuthenticated(requestWithCookie(t, copied)) {
|
||||||
|
t.Error("after a restart, a cookie copied before the logout is accepted")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !restarted.IsAuthenticated(requestWithCookie(t, loginAfter)) {
|
||||||
|
t.Error("after a restart, a login made after the logout is refused")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLogout_WithoutSession_EndsNoSession verifies that a logout from a
|
||||||
|
// request without a session cookie ends no session, so someone who has not
|
||||||
|
// logged in cannot end the sessions of those who have.
|
||||||
|
func TestLogout_WithoutSession_EndsNoSession(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mgr, err := session.NewManagerWithDatabase(
|
||||||
|
t.Context(), logoutTestSigningKey, newTestDatabase(t))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewManagerWithDatabase() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cookie := createSessionCookie(t, mgr)
|
||||||
|
|
||||||
|
err = mgr.Logout(httptest.NewRecorder(), requestWithCookie(t, nil))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Logout() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !mgr.IsAuthenticated(requestWithCookie(t, cookie)) {
|
||||||
|
t.Error("a logout without a session cookie ended a session")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -126,31 +126,20 @@ func (s *Signer) GenerateSignedURL(
|
|||||||
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
|
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Build the path.
|
// Build the path. A source query follows the source path after a "?".
|
||||||
// When a source query is present, it is embedded as a path segment
|
// The image route reads the path decoded once, so the whole path is
|
||||||
// (e.g. /host/path?query/size.fmt) so that the URL parser can extract
|
// written in Go's own percent-encoding of a URL path, which writes "?"
|
||||||
// it from the last-slash split. The "?" inside a path segment is
|
// as %3F and "%" as %25, and leaves "/", ";", ",", "=" and "&" as they
|
||||||
// percent-encoded by clients but chi delivers it decoded, which is
|
// are.
|
||||||
// exactly what the URL parser expects.
|
sourcePathAndQuery := req.SourcePath
|
||||||
var path string
|
|
||||||
if req.SourceQuery != "" {
|
if req.SourceQuery != "" {
|
||||||
path = fmt.Sprintf("/v1/image/%s%s%%3F%s/%s.%s",
|
sourcePathAndQuery += "?" + req.SourceQuery
|
||||||
req.SourceHost,
|
|
||||||
req.SourcePath,
|
|
||||||
url.PathEscape(req.SourceQuery),
|
|
||||||
sizeStr,
|
|
||||||
req.Format,
|
|
||||||
)
|
|
||||||
} else {
|
|
||||||
path = fmt.Sprintf("/v1/image/%s%s/%s.%s",
|
|
||||||
req.SourceHost,
|
|
||||||
req.SourcePath,
|
|
||||||
sizeStr,
|
|
||||||
req.Format,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return path, sig, exp
|
unescapedPath := fmt.Sprintf("/v1/image/%s%s/%s.%s",
|
||||||
|
req.SourceHost, sourcePathAndQuery, sizeStr, req.Format)
|
||||||
|
|
||||||
|
return (&url.URL{Path: unescapedPath}).EscapedPath(), sig, exp
|
||||||
}
|
}
|
||||||
|
|
||||||
// buildSignatureData creates the string to be signed.
|
// buildSignatureData creates the string to be signed.
|
||||||
|
|||||||
@@ -475,6 +475,44 @@ func TestGenerateSignedURL_WithoutQueryString(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestGenerateSignedURL_ReadmeQueryExample checks the README's example of a
|
||||||
|
// source URL with a query, the image at
|
||||||
|
// https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg: its
|
||||||
|
// signature with the README's key, and the path GenerateSignedURL writes for
|
||||||
|
// it, with the "?" written as %3F and the "%" as %25.
|
||||||
|
func TestGenerateSignedURL_ReadmeQueryExample(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
signer := signature.New("example-signing-key-for-documentation")
|
||||||
|
|
||||||
|
req := signature.Request{
|
||||||
|
SourceHost: testHost,
|
||||||
|
SourcePath: "/img",
|
||||||
|
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
|
||||||
|
Width: 800,
|
||||||
|
Height: 600,
|
||||||
|
Format: testFormatWebP,
|
||||||
|
Quality: 85,
|
||||||
|
FitMode: testFitCover,
|
||||||
|
Expires: time.Unix(1704067200, 0),
|
||||||
|
}
|
||||||
|
|
||||||
|
const wantSig = "37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA="
|
||||||
|
|
||||||
|
gotSig := signer.Sign(&req)
|
||||||
|
if gotSig != wantSig {
|
||||||
|
t.Errorf("Sign() = %q, want %q", gotSig, wantSig)
|
||||||
|
}
|
||||||
|
|
||||||
|
const wantPath = "/v1/image/cdn.example.com" +
|
||||||
|
"/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp"
|
||||||
|
|
||||||
|
gotPath, _, _ := signer.GenerateSignedURL(&req, time.Hour)
|
||||||
|
if gotPath != wantPath {
|
||||||
|
t.Errorf("GenerateSignedURL() path = %q, want %q", gotPath, wantPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestParseParams(t *testing.T) {
|
func TestParseParams(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -122,12 +122,20 @@ header {
|
|||||||
margin-bottom: 2rem;
|
margin-bottom: 2rem;
|
||||||
}
|
}
|
||||||
|
|
||||||
header a {
|
/* The logout button, drawn as a link. */
|
||||||
|
|
||||||
|
header button {
|
||||||
|
width: auto;
|
||||||
|
margin: 0;
|
||||||
|
padding: 0;
|
||||||
|
background: none;
|
||||||
font-size: 0.875rem;
|
font-size: 0.875rem;
|
||||||
color: #4b5563;
|
color: #4b5563;
|
||||||
|
text-decoration: underline;
|
||||||
}
|
}
|
||||||
|
|
||||||
header a:hover {
|
header button:hover {
|
||||||
|
background: none;
|
||||||
color: #1f2937;
|
color: #1f2937;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -10,9 +10,12 @@
|
|||||||
<div class="page">
|
<div class="page">
|
||||||
<header>
|
<header>
|
||||||
<h1>Pixa URL Generator</h1>
|
<h1>Pixa URL Generator</h1>
|
||||||
<a href="/logout">
|
<form method="POST" action="/logout">
|
||||||
Logout
|
{{ .CSRFField }}
|
||||||
</a>
|
<button type="submit">
|
||||||
|
Logout
|
||||||
|
</button>
|
||||||
|
</form>
|
||||||
</header>
|
</header>
|
||||||
|
|
||||||
{{if .GeneratedURL}}
|
{{if .GeneratedURL}}
|
||||||
|
|||||||
+33
-12
@@ -3,11 +3,13 @@
|
|||||||
# this repo. Idempotent: every install is guarded by a check so already
|
# this repo. Idempotent: every install is guarded by a check so already
|
||||||
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
||||||
# or apk (detected in that order); assumes NOTHING is present (not git,
|
# or apk (detected in that order); assumes NOTHING is present (not git,
|
||||||
# make, or go). Node is used directly if installed; otherwise it is
|
# make, or go). The installed node is used only when its major version
|
||||||
# installed at a pinned version via nvm (installing nvm itself first,
|
# is the pinned one; otherwise (node missing, or another major) the
|
||||||
# from a hash-verified release archive, never curl | sh). The linter is
|
# pinned version is installed via nvm (installing nvm itself first,
|
||||||
# never installed on the host: golangci-lint runs only in the lint phase
|
# from a hash-verified release archive, never curl | sh), and yarn and
|
||||||
# of the Dockerfile (see script/lint).
|
# the packages are installed under it. The linter is never installed on
|
||||||
|
# the host: golangci-lint runs only in the lint phase of the Dockerfile
|
||||||
|
# (see script/lint).
|
||||||
#
|
#
|
||||||
# script/bootstrap git, make, Go, and Node, Yarn and the
|
# script/bootstrap git, make, Go, and Node, Yarn and the
|
||||||
# prettier in yarn.lock for script/fmt and
|
# prettier in yarn.lock for script/fmt and
|
||||||
@@ -110,31 +112,50 @@ ensure_nvm() {
|
|||||||
rm -rf "$tmp"
|
rm -rf "$tmp"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# True when the node on PATH has the pinned major version. Only the
|
||||||
|
# major is compared, as REPO_POLICIES.md asks. Another major is not
|
||||||
|
# used: the pinned yarn 1 prints a deprecation warning under node 24.
|
||||||
|
# script/fmt and script/fmt-check carry this function unchanged, to run
|
||||||
|
# the yarn installed here.
|
||||||
|
node_is_pinned_major() {
|
||||||
|
if ! command -v node >/dev/null 2>&1; then return 1; fi
|
||||||
|
installed="$(node --version)"
|
||||||
|
installed="${installed#v}"
|
||||||
|
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
|
||||||
|
}
|
||||||
|
|
||||||
ensure_node() {
|
ensure_node() {
|
||||||
if ! missing node; then return 0; fi
|
if node_is_pinned_major; then
|
||||||
|
echo "bootstrap: using node $(node --version)"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
ensure_nvm
|
ensure_nvm
|
||||||
nvm_sh "nvm install $NODE_VERSION"
|
nvm_sh "nvm install $NODE_VERSION"
|
||||||
|
nvm_sh "nvm use $NODE_VERSION >/dev/null && \
|
||||||
|
echo \"bootstrap: using node \$(node --version)\""
|
||||||
}
|
}
|
||||||
|
|
||||||
ensure_yarn() {
|
ensure_yarn() {
|
||||||
|
if ! node_is_pinned_major; then
|
||||||
|
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
|
||||||
|
corepack prepare yarn@$YARN_VERSION --activate"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
if ! missing yarn; then return 0; fi
|
if ! missing yarn; then return 0; fi
|
||||||
if ! missing corepack; then
|
if ! missing corepack; then
|
||||||
corepack enable
|
corepack enable
|
||||||
corepack prepare "yarn@$YARN_VERSION" --activate
|
corepack prepare "yarn@$YARN_VERSION" --activate
|
||||||
elif [ -s "$HOME/.nvm/nvm.sh" ]; then
|
|
||||||
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
|
|
||||||
corepack prepare yarn@$YARN_VERSION --activate"
|
|
||||||
else
|
else
|
||||||
npm install -g "yarn@$YARN_VERSION"
|
npm install -g "yarn@$YARN_VERSION"
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
install_js_deps() {
|
install_js_deps() {
|
||||||
if missing yarn && [ -s "$HOME/.nvm/nvm.sh" ]; then
|
if node_is_pinned_major; then
|
||||||
|
yarn install --frozen-lockfile
|
||||||
|
else
|
||||||
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
|
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
|
||||||
yarn install --frozen-lockfile"
|
yarn install --frozen-lockfile"
|
||||||
else
|
|
||||||
yarn install --frozen-lockfile
|
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+13
-10
@@ -2,9 +2,11 @@
|
|||||||
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
||||||
# checks out and runs this and nothing else, and script/fmt-check runs
|
# checks out and runs this and nothing else, and script/fmt-check runs
|
||||||
# the formatter on the host, which a pristine checkout cannot do.
|
# the formatter on the host, which a pristine checkout cannot do.
|
||||||
# --no-cache for the same reason as script/docker: the gate phases the
|
# A new CACHEBUST value reruns every RUN step, since a cached gate phase
|
||||||
# final stage depends on are RUN steps, and a cached one is a check that
|
# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
|
||||||
# did not run.
|
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
|
||||||
|
# first; a stage without that line is served from cache on an unchanged
|
||||||
|
# tree, with no warning.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -14,16 +16,17 @@ main() {
|
|||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
"$SCRIPT_DIR/bootstrap"
|
"$SCRIPT_DIR/bootstrap"
|
||||||
"$SCRIPT_DIR/check"
|
"$SCRIPT_DIR/check"
|
||||||
# Own line: a failing command substitution inside an argument does
|
# The version and the tag each get their own line: a failing
|
||||||
# not trip `set -e`, so the inline form degrades silently to an
|
# command substitution inside an argument does not trip `set -e`,
|
||||||
# empty constant. VERSION is computed here because .dockerignore
|
# so the inline form degrades silently to an empty constant. The
|
||||||
# excludes .git, so `git describe` in a build stage yields an empty
|
# VERSION build argument takes precedence over the version a build
|
||||||
# version without failing.
|
# stage derives from the .git in the context.
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
docker build --no-cache \
|
tag="$("$SCRIPT_DIR/projectname")"
|
||||||
|
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$("$SCRIPT_DIR/projectname")" .
|
-t "$tag" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+13
-9
@@ -1,8 +1,11 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/docker: build the Docker image tagged with the project name.
|
# script/docker: build the Docker image tagged with the project name.
|
||||||
# Identical in all repos; the tag comes from script/projectname.
|
# Identical in all repos; the tag comes from script/projectname.
|
||||||
# --no-cache because the gate phases the final stage depends on are RUN
|
# A new CACHEBUST value reruns every RUN step, since a cached gate phase
|
||||||
# steps, and a cached one is a check that did not run.
|
# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
|
||||||
|
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
|
||||||
|
# first; a stage without that line is served from cache on an unchanged
|
||||||
|
# tree, with no warning.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -10,16 +13,17 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
# Own line: a failing command substitution inside an argument does
|
# The version and the tag each get their own line: a failing
|
||||||
# not trip `set -e`, so the inline form degrades silently to an
|
# command substitution inside an argument does not trip `set -e`,
|
||||||
# empty constant. VERSION is computed here because .dockerignore
|
# so the inline form degrades silently to an empty constant. The
|
||||||
# excludes .git, so `git describe` in a build stage yields an empty
|
# VERSION build argument takes precedence over the version a build
|
||||||
# version without failing.
|
# stage derives from the .git in the context.
|
||||||
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
[ -n "$version" ] || version="unknown"
|
[ -n "$version" ] || version="unknown"
|
||||||
docker build --no-cache \
|
tag="$("$SCRIPT_DIR/projectname")"
|
||||||
|
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
||||||
--build-arg VERSION="$version" \
|
--build-arg VERSION="$version" \
|
||||||
-t "$("$SCRIPT_DIR/projectname")" .
|
-t "$tag" .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+16
-6
@@ -7,15 +7,25 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|||||||
# Must match the pin in script/bootstrap.
|
# Must match the pin in script/bootstrap.
|
||||||
NODE_VERSION="22.17.0"
|
NODE_VERSION="22.17.0"
|
||||||
|
|
||||||
# script/bootstrap installs node and yarn under nvm and leaves neither
|
# True when the node on PATH has the pinned major version: the same
|
||||||
# on the PATH of the shell that called it, so resolve the pinned
|
# function as in script/bootstrap, which uses it to decide where it
|
||||||
# toolchain here the way bootstrap's own install step does. nvm is a
|
# installs yarn.
|
||||||
# bash script, hence the subshell.
|
node_is_pinned_major() {
|
||||||
|
if ! command -v node >/dev/null 2>&1; then return 1; fi
|
||||||
|
installed="$(node --version)"
|
||||||
|
installed="${installed#v}"
|
||||||
|
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
# Run the yarn script/bootstrap installed: the yarn on PATH when the
|
||||||
|
# node on PATH has the pinned major version, and otherwise yarn under
|
||||||
|
# the pinned node in nvm, which bootstrap leaves off the PATH of the
|
||||||
|
# shell that called it. nvm is a bash script, hence the subshell.
|
||||||
run_yarn() {
|
run_yarn() {
|
||||||
if command -v yarn >/dev/null 2>&1; then
|
if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then
|
||||||
exec yarn "$@"
|
exec yarn "$@"
|
||||||
fi
|
fi
|
||||||
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
echo "fmt: no yarn; run script/bootstrap first" >&2
|
echo "fmt: no yarn; run script/bootstrap first" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|||||||
+16
-6
@@ -8,15 +8,25 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|||||||
# Must match the pin in script/bootstrap.
|
# Must match the pin in script/bootstrap.
|
||||||
NODE_VERSION="22.17.0"
|
NODE_VERSION="22.17.0"
|
||||||
|
|
||||||
# script/bootstrap installs node and yarn under nvm and leaves neither
|
# True when the node on PATH has the pinned major version: the same
|
||||||
# on the PATH of the shell that called it, so resolve the pinned
|
# function as in script/bootstrap, which uses it to decide where it
|
||||||
# toolchain here the way bootstrap's own install step does. nvm is a
|
# installs yarn.
|
||||||
# bash script, hence the subshell.
|
node_is_pinned_major() {
|
||||||
|
if ! command -v node >/dev/null 2>&1; then return 1; fi
|
||||||
|
installed="$(node --version)"
|
||||||
|
installed="${installed#v}"
|
||||||
|
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
|
||||||
|
}
|
||||||
|
|
||||||
|
# Run the yarn script/bootstrap installed: the yarn on PATH when the
|
||||||
|
# node on PATH has the pinned major version, and otherwise yarn under
|
||||||
|
# the pinned node in nvm, which bootstrap leaves off the PATH of the
|
||||||
|
# shell that called it. nvm is a bash script, hence the subshell.
|
||||||
run_yarn() {
|
run_yarn() {
|
||||||
if command -v yarn >/dev/null 2>&1; then
|
if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then
|
||||||
exec yarn "$@"
|
exec yarn "$@"
|
||||||
fi
|
fi
|
||||||
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
|
||||||
echo "fmt-check: no yarn; run script/bootstrap first" >&2
|
echo "fmt-check: no yarn; run script/bootstrap first" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|||||||
+8
-5
@@ -5,9 +5,12 @@
|
|||||||
# make its answer untrustworthy.
|
# make its answer untrustworthy.
|
||||||
#
|
#
|
||||||
# The phase is not the last stage in the file, so it is built only when
|
# The phase is not the last stage in the file, so it is built only when
|
||||||
# --target names it. --no-cache because a cached lint layer is a lint
|
# --target names it. A new CACHEBUST value reruns every RUN step, since
|
||||||
# that did not run. The tag makes each build replace the previous image
|
# a cached lint layer is a lint that did not run; unlike --no-cache, it
|
||||||
# instead of leaving a dangling one behind.
|
# keeps Go's build cache. This relies on every stage of the Dockerfile
|
||||||
|
# declaring ARG CACHEBUST first; a stage without that line is served
|
||||||
|
# from cache on an unchanged tree, with no warning.
|
||||||
|
# --output type=cacheonly writes no image, since nothing uses one.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -15,9 +18,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
docker build --no-cache \
|
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
||||||
--target lint \
|
--target lint \
|
||||||
-t "$("$SCRIPT_DIR/projectname")-lint" .
|
--output type=cacheonly .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+8
-4
@@ -2,8 +2,12 @@
|
|||||||
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
# script/test: run the test suite. Testing is a phase of the Dockerfile
|
||||||
# and this builds that phase alone, on the same terms as script/lint:
|
# and this builds that phase alone, on the same terms as script/lint:
|
||||||
# --target because a phase that is not the last stage is built only when
|
# --target because a phase that is not the last stage is built only when
|
||||||
# named, --no-cache because a cached test layer is a test that did not
|
# named, and a new CACHEBUST value because a cached test layer is a test
|
||||||
# run, and a tag so each build replaces the previous image.
|
# that did not run; unlike --no-cache, it keeps Go's build cache. This
|
||||||
|
# relies on every stage of the Dockerfile declaring ARG CACHEBUST first;
|
||||||
|
# a stage without that line is served from cache on an unchanged tree,
|
||||||
|
# with no warning. --output type=cacheonly writes no image, since
|
||||||
|
# nothing uses one.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
@@ -11,9 +15,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
docker build --no-cache \
|
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
|
||||||
--target test \
|
--target test \
|
||||||
-t "$("$SCRIPT_DIR/projectname")-test" .
|
--output type=cacheonly .
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
Reference in New Issue
Block a user