sneak/pixa
1
0 Fork 0
Code Issues 3 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Compare commits

base: sneak/pixa:v1.0.0-rc1
sneak/pixa:main sneak/pixa:next sneak/pixa:prod
sneak/pixa:v1.0.0-rc3 sneak/pixa:v1.0.0-rc2 sneak/pixa:v1.0.0-rc1
...
compare: sneak/pixa:6defdaaa9dbe3ecd221f03c0ec685e6f1afa138e
sneak/pixa:next sneak/pixa:main sneak/pixa:prod
sneak/pixa:v1.0.0-rc3 sneak/pixa:v1.0.0-rc2 sneak/pixa:v1.0.0-rc1
69 Commits
v1.0.0-rc1 ... 6defdaaa9d
Author SHA1 Message Date
clawbot 6defdaaa9d Answer 502 for an upstream host with no address or a failed fetch (closes #367)
check / check (push) Canceled after 0s
Details 
The image routes answered 500 and logged at error level when the upstream
host had no address, when the fetch failed before the upstream answered,
such as on a refused connection, or when the fetcher refused a redirect.
The fetcher now marks such a failure with ErrUpstreamRequestFailed; a
cancelled fetch is not marked. respondImageError answers it, and
ErrInvalidHost, 502 "upstream error" and logs them at warn level. The TLS
handshake timeout and an address lookup that times out also answered 500;
both are now ErrUpstreamTimeout and answer 504. README Routes lists these
answers.

Model: opus-5-5
 2026-11-11 22:22:58 +00:00
clawbot b32077c380 Name the cause on the page for a refused form (closes #358)
check / check (push) Canceled after 0s
Details 
The CSRF middleware answered a refused login or generator form with the
library's bare text, such as "Forbidden - origin invalid". It now answers
403 with a page that names the cause and what to do, and logs a warning
with the request ID and the library's reason. A page opened over plain
HTTP with debug off and a form from another host both fail the origin
check; the page tells them apart by the host in the Origin or Referer
header.

Model: opus-5-5
 2026-11-11 22:55:39 +01:00
clawbot 9bccaaf85a README Getting Started ends with a signed image request (closes #357)
check / check (push) Canceled after 0s
Details 
Getting Started now runs script/bootstrap --cgo before make build, says
that logging in at http://localhost:8080/ needs debug, and ends with a
signed /v1/image/ request for an image on any public HTTPS server. Its
curl exits non-zero when pixa refuses the request.

Model: opus-5-5
 2026-11-11 22:40:59 +01:00
clawbot 851a374e53 Test that pixad writes only its own errors to stderr (closes #359)
check / check (push) Canceled after 0s
Details 
Both tests run pixad as a child process. They fail against this code:
every start writes the dependency-injection library's event log to
stderr, and a startup error is printed twice inside its frames.

Model: opus-5-5
 2026-11-11 22:26:52 +01:00
clawbot 0de6972c2d Run make devserver from environment variables, not a config file (closes #360)
check / check (push) Canceled after 0s
Details 
The target mounted config.dev.yml, which nothing creates, so Docker made
a directory in its place and pixad exited. It now passes a new random
PIXA_SIGNING_KEY each run, printed so the developer can log in, and
PIXA_DEBUG=true so the login form works over plain HTTP. The unused
.gitignore entry for config.dev.yml is removed.

Model: opus-5-5
 2026-11-11 21:55:58 +01:00
clawbot 092176d917 Delete a negative-cache test that tests nothing (closes #347)
check / check (push) Canceled after 0s
Details 
TestProxy_Get_ReturnsErrorForNegativeCachedURL never called Get; it only
checked ErrNegativeCached against itself. The test above it covers Get
returning ErrNegativeCached. Deletion approved by sneak on the issue.

Model: opus-5-5
 2026-11-11 16:56:21 +01:00
clawbot 7f87aa17ca Set the expiry in TestHandleImageEnc_MaxAge inside each subtest (closes #352)
check / check (push) Canceled after 0s
Details 
The table set the expiry when it was built. The parallel subtests can wait
more than 10 seconds behind other tests on a busy host, so the 60 second
case saw a max-age under 50. The table now holds the TTL, and each subtest
computes the expiry just before generating its token.

Model: opus-5-5
 2026-11-11 16:25:44 +01:00
sneak fa2ec3da6a Export cache, eviction, upstream and transform metrics (closes #85)
check / check (push) Successful in 1m30s
Details 
New pixa_ metrics, none with labels:
- source cache hits and misses, counted once per variant made or checked;
  a source whose cached file cannot be used is a miss
- cache usage and limit gauges; usage is set each time eviction reads it
- eviction count and bytes, from the size_bytes the deleted row returns
- upstream fetch duration histogram and fetch error counter
- transform duration histogram and an in-progress gauge, both including
  the wait for a processing slot

Removing a damaged file during reconciliation is not counted as an
eviction. README lists every metric; the metrics config comment now says
the pixa_ metrics are measured whether or not /metrics is served.

Model: opus-5-5
 2026-11-11 14:32:38 +00:00
sneak a041e73550 Test the eviction, source cache, upstream and transform metrics 
The tests check the metrics on a variant hit, on misses that fetch or use
the cached source, on a failed fetch, and on both kinds of eviction. A
source whose cached file is gone must count as a source cache miss. A
transform waiting for a processing slot must count as in progress. The
tests fail until the metrics exist.

Model: opus-5-5
 2026-11-11 14:31:03 +00:00
clawbot e34463acf8 Check cached sources with upstream after the cache TTL (closes #69) (#350)
check / check (push) Canceled after 0s
Details
2026-11-11 13:37:15 +01:00
clawbot 1379497897 Keep only open work in TODO.md (closes #348) (#349)
check / check (push) Canceled after 0s
Details
2026-11-11 13:07:16 +01:00
clawbot b3b8408d18 Rewrite README.md sentence by sentence (closes #327) (#345)
check / check (push) Canceled after 0s
Details
2026-11-11 11:37:27 +01:00
clawbot 9f1e9f5253 Rewrite AGENTS.md in short plain sentences (#327)
check / check (push) Canceled after 0s
Details 
Every rule keeps its meaning and emphasis. Sentences are split so each
holds one idea, with the main rule before its exception and every
command, file name and identifier in backticks.

Three statements were false against the repo or the standing rulings
on record and are corrected:

- the format check is `make fmt-check`, not `make check-fmt`
- a feature branch reaches `next` through a PR that passes review, is
  merged only when the feature is completely finished and the code is
  up to standards, and is deleted after the merge; only the user
  merges `next` into `main`
- the naming example uses the code's term: `al` is bad,
  `SourceHostAllowlist` is good

The `Last Updated` line takes today's date.

Model: fable-5-1
 2026-10-27 05:06:36 +01:00
clawbot 2c90144e7b Go on past cache entries the reconciliation pass cannot read (closes #343)
check / check (push) Canceled after 0s
Details 
The walk of each storage directory returned the error of a directory it
could not list, or of a source or variant file whose info it could not
read. The pass then stopped before its later checks, every time it ran.
Now reconcileFiles skips a directory it cannot list and a file whose
info it cannot read, temp files included. It logs a warning naming the
path and the error, and the walk goes on. It reads each file's info
once and hands it to sweepStaleTempFile, reconcileVariantFile or
reconcileSourceFile. A file renamed or removed after its directory was
listed is skipped with no warning. A cancelled context still stops the
walk with its error. README.md says so under Storage.

Model: opus-5-5
 2026-10-27 01:13:55 +00:00
clawbot 26b5160aa8 Test that reconciliation goes on past entries it cannot read 
Each storage directory gets a directory the pass cannot list and a file
whose info it cannot read. Their paths are longer than Linux accepts,
even for root. The test wants a warning naming each path and the error.
It also wants the rest of the pass done: a variant file with no row
adopted, the row of a missing variant file dropped, a source file with
no row removed, and a wrong usage total corrected. A second test puts a
temp file whose info cannot be read in each storage directory and wants
the same warning. On next the first test's pass stops at the first such
directory and returns its error. The second test gets no warning.

Model: opus-5-5
 2026-10-27 01:12:34 +00:00
clawbot 33045a24b5 Keep a variant's accounting row when its file cannot be read (closes #340)
check / check (push) Canceled after 0s
Details 
The reconciliation pass dropped a variant's accounting row on any error
reading its file's size, and logged every such file as missing. A file
still on disk then had no row, so eviction no longer counted it against
cache_max_bytes. Now only a missing file (ErrNotFound, which
FileStore.Size returns for it) loses its row, with the same info line.
Any other error keeps the row and logs a warning naming the variant key
and the error, and the pass goes on to the next key. README.md says so
under Storage.

Model: opus-5-5
 2026-10-26 23:56:34 +00:00
clawbot 701ed9b94b Test that reconciliation drops a variant's row only when its file is missing
check / check (push) Canceled after 0s
Details 
reconcileVariantRows is given two tracked variants. The first one's
file is a symbolic link to itself, so reading its size fails with an
error other than a missing file. The test wants its row kept, with a
warning naming its key and the error. The second one has no file, and
the test wants its row dropped with the info line for a missing file.
The test fails on next, which drops both rows and logs both files as
missing. The fix follows in the next commit.

Model: opus-5-5
 2026-10-26 23:54:24 +00:00
clawbot 233765033a Answer 499 to a client that goes away while waiting for an image (closes #339)
check / check (push) Canceled after 0s
Details 
When a request waits for an image another request is making and its
client goes away, the image proxy returns context.Canceled. serveImage
now answers it 499, the status nginx calls "client closed request". It
logs it once at info level with the request_id, host and path. It was
answered 500 and logged at error level, as pixa's own failure. A
cancellation while the request's context is live is still answered 500.
The request making the image is still answered as before. README.md
lists the 499 on both image routes.

Model: opus-5-5
 2026-10-26 23:37:35 +00:00
clawbot d33316caf3 Test that a client leaving while another request makes its image gets 499 
On both image routes, a request waits for an image another request is
making, and its client goes away. The first test wants that request
answered 499 and logged once at info level. It fails on next, which
answers 500 and logs it at error level. The second test pins the 500
that a cancellation keeps while the request's context is live; it
passes on next. The fix follows in the next commit.

Model: opus-5-5
 2026-10-26 23:26:07 +00:00
clawbot 6de69f7349 Name pixa's concepts at the start of the README's Design section (closes #321)
check / check (push) Canceled after 0s
Details 
The Design section starts with a Concepts list. Each concept gets one
sentence, under the name the code gives it. Storage calls the
transformed images variants, and the sentences it changed for that are
shorter. Its paths end in the source key and the variant key, and it
lists the image cache's tables in the database. Architecture names
internal/providers and lists the packages that work with the concepts.
No code or test changes.

Model: opus-5-5
 2026-10-26 23:19:19 +01:00
clawbot 93b13d6638 Load every setting from one table in internal/config (closes #320)
check / check (push) Canceled after 0s
Details 
Config.settings lists each setting once: its key, its environment variable
and the function that loads it. Most rows load through field, whose
arguments give the type, the default and the check. db_url and
cache_max_bytes keep their own load functions. Loading, the unknown-key
and unknown-variable checks and the names in error messages all read the
table. allowlist_hosts now loads through parseHostList.
isKnownConfigKey, envVarNames, strictLoader, validate,
validateAllowlistHostsValue and getStringSlice are gone, with the
getStringSlice tests. The key constants move to the tests, their only
users. README.md lists the settings one item each instead of in a table,
with shorter sentences above the list. A new test checks that this list
and configs/config.example.yml name exactly the keys of the table.

Model: opus-5-5
 2026-10-26 22:32:03 +01:00
clawbot b73bbde46e Detect a fetched image's format from its bytes once (closes #319)
check / check (push) Canceled after 0s
Details 
magic.DetectFormat returns an imagerequest.Format. ValidateMagicBytes
returns the format it checked against the upstream Content-Type. The
cache reads a cached source's format from its file's first bytes, as it
reads a variant's. The proxy hands the format to Process. Process
refuses a format that is not a source format, and input libvips would
load with another loader, such as an SVG passing as AVIF.
Format.MIMEType declares each MIME string once, and the fetcher, the
cache's variant read and the Accept matching use it. magic.MIMEType,
magic.ImageFormat, MIMEToImageFormat, magic's SVG detection,
detectFormat, formatFromString, FormatToMIME and Result.InputFormat are
gone.

Model: opus-5-5
 2026-10-26 21:21:39 +01:00
clawbot ac850823f4 Keep cached sources and variants in one FileStore type (closes #318)
check / check (push) Canceled after 0s
Details 
imgcache.FileStore is a directory of files named by a hex key, at
<ab>/<cd>/<key>. It writes a file under a temp name, syncs it and renames
it into place. It opens, reports the size of and deletes the file for a
key. The cache keeps its sources in one FileStore and its variants in
another, so ContentStorage and VariantStorage are gone. FileStore has no
Exists: the reconciliation pass checks a variant's file with Size, and the
tests look for the file on disk. Cache.Variant reads the content type from
the file's first bytes itself. storage.go is now filestore.go, and the
unused testdata/red.avif is gone.

Model: opus-5-5
 2026-10-26 20:24:33 +01:00
clawbot 17376d885b Split internal/imgcache into the image proxy and the cache (closes #317)
check / check (push) Canceled after 0s
Details 
The new package internal/imageproxy holds Proxy, which was imgcache.Service,
with its Config, ImageResponse, CacheStatus and errors. imageproxy.New makes
and registers the counters of upstream fetches and transforms.
internal/imgcache holds the cache only, with its hit and miss counters, and
imgcache.CacheConfig is imgcache.Config. The cache reads and writes each kind
it keeps with one pair of methods: Variant and PutVariant, Source and
PutSource, Failure and PutFailure. Source replaces LookupSource followed by
GetSourceContent. imgcache.go and module.go are gone. The proxy's tests moved
with it. The cache's tests check that the reconciliation pass removes an empty
or cut file and its row. The proxy's tests remove them by hand, then check
that Get fetches or converts again.

Model: opus-5-5
 2026-10-26 20:00:25 +01:00
clawbot 87cb1ab5f6 Serve both image routes with one function and one error mapping (closes #316)
check / check (push) Canceled after 0s
Details 
HandleImage parses the path and verifies the signature, and HandleImageEnc
decrypts the token. Each then calls serveImage, which chooses the format for
auto, gets the image and serves it. respondImageError answers the image
service's errors on both routes, and handleImageError is gone. /v1/image/ now
answers 504 when the upstream sends no response headers within
upstream_fetch_timeout, as README.md documented for /v1/e/. respondImageError
logs a 500 at error level and the other answers at warn. The image service
alone logs a source refused for its first bytes. A failed write is logged once,
at warn, with the request and the bytes served. README.md lists each image
route's answers as bullets. Tests send the same errors through both routes.

Model: opus-5-5
 2026-10-26 15:24:24 +01:00
clawbot 77d55abb7e Derive every key from the signing key in one place (closes #315)
check / check (push) Canceled after 0s
Details 
seal.DeriveKeys derives the session hash and block keys, the encrypted URL
key and the CSRF key from the signing key, with their salts declared beside
it, and returns them as one seal.Keys. internal/providers calls it once at
startup and provides the value to the fx graph. The session manager, the
encrypted URL generator and the CSRF protection each take their own key.
session.NewManager, encurl.NewGenerator, newCSRFProtect and handlers.New
return no error, as nothing in them can fail now. Each key is derived as
before, with seal.DeriveKey and the same salt. The signing key itself stays
the HMAC key of signed URLs and the login password. README.md says under
signing_key what it protects.

Model: opus-5-5
 2026-10-26 14:10:20 +01:00
clawbot 7b6712538e Take the signature, the expiry and the URL scheme off the image request (closes #314)
check / check (push) Canceled after 0s
Details 
The image request now holds only its source and its transform, exactly what
the variant key covers. The /v1/image/ handler checks the allowlist and
verifies the signature itself with Signer.Verify(req, sig, expires). Both
image handlers keep the URL's expiry as a local value for Cache-Control. The
image service holds no signer, allowlist or AllowHTTP, and ValidateRequest is
gone. The fetcher takes the source and builds its URL with Source.URL from its
own AllowHTTP, so no URL string is built and parsed again. chooseAutoFormat
returns the format it chooses instead of writing into the request. The signed
URL format and every route's answers are unchanged.

Model: opus-5-5
 2026-10-26 13:49:12 +01:00
clawbot 63690a9ec9 Build the image pipeline in pixad's fx graph (closes #313)
check / check (push) Canceled after 0s
Details 
The new package internal/providers builds the cache, the fetcher, the
processor, the image service, the session manager and the encrypted URL
generator from the config. pixad's fx graph provides them with
providers.Constructors(), and handlers.New receives them built. The
server tests build the same graph, with only the fetcher's dial function
replaced. database.New opens the database, so these parts can read it
while fx builds them; the cache's stop hook still runs before the
database's.

Each setting's default is declared in internal/config, and
internal/providers works out the default cache_max_bytes. The cache's
MaxBytes is the one value for the disk cache limit, and 0 turns the disk
cache off. The fetcher and the processor take explicit values.

Model: opus-5-5
 2026-10-26 12:28:23 +01:00
clawbot ac2bae0489 Remove the code nothing in production calls (closes #312)
check / check (push) Canceled after 0s
Details 
Service.Warm, the ImageCache interface, both GenerateSignedURL methods,
signature.ParseParams, imagerequest.ParseImagePath, four symbols of
internal/magic, the image processor's lists of supported formats,
IsEmpty and Count of the allowlist, ssrfSafeDialer, CacheStale,
ErrCacheMiss, ErrNegativeCache, ImageResponse.LastModified and the Load
methods of ContentStorage and VariantStorage had no caller outside
tests. They are gone, with the tests that tested only them.

The handlers tests sign their image URLs with generateSignedURL, a test
helper that builds the URL as the removed generator did. The SSRF dialer
tests call dialSSRFSafe with no operator blocklist, which is all
ssrfSafeDialer did. The tests of ParseImagePath and ContentStorage.Load
call ParseImageURL and LoadWithSize instead. MIMEToImageFormat,
ContentStorage.Store and CleanExpired stay, as pixa calls them.

Model: opus-5-5
 2026-10-26 10:54:14 +01:00
clawbot b49d0068df Count hits, misses, fetches and transforms as Prometheus counters (closes #311)
check / check (push) Canceled after 0s
Details 
The five counts the cache_stats table held are now Prometheus counters in
memory, as fields of the cache. NewCache makes them and registers them on
the registry it is given. pixad gives it Prometheus' default registry,
which /metrics serves; each test gives it a new one. The Increment methods
add to their cache's counters, so counting makes no database call.

cache_stats, Cache.Stats, Service.Stats, CacheStats and the pending counts
are gone, with their tests. The background goroutine now writes only access
times, and its names say so: StartPendingAccessTimeWrites,
StopPendingAccessTimeWrites and pendingaccesstimes.go. The two stop tests
whose write under way was a count now use access times.

Model: opus-5-5
 2026-10-26 10:20:42 +01:00
clawbot 6b19e4c96c Declare only the tables and columns pixa writes and reads (closes #310)
check / check (push) Canceled after 0s
Details 
Removes output_content and request_cache, which nothing writes, and every
column that was written but never read: the host, path, query, content_type
and fetched_at of source_content, the content_type and created_at of
variant_content, and the status_code, error_message and fetched_at of
negative_cache.

StoreSource no longer takes the fetch result and LookupSource returns only
the source key. StoreVariant no longer takes a content type and
StoreNegative no longer takes a status code or message, so
extractStatusCode and the status constants, which only fed status_code,
are gone. Reconciliation adopts a variant file without opening it, as its
row no longer records a content type. Tests that set or checked a removed
column follow.

Model: opus-5-5
 2026-10-26 09:29:26 +01:00
clawbot 5f9d624d74 Serve a cache hit with no database read or write (closes #309)
check / check (push) Canceled after 0s
Details 
A hit opens the variant file and nothing else: its content type comes
from the file's first bytes, and a file in no output format is a miss.
Its access time and count, and the access time of a cached source a
miss uses, go without waiting to one background goroutine that writes
them once a second; each kind of access time waits in a channel of
10,000 that drops the rest. Expired negative cache rows are left to
CleanExpired. The .meta files, metaCache, Lookup and golang-lru go,
with their tests; the two reconciliation walks, identical without the
.meta skip, share one. make bench runs a parallel cache hit benchmark
on the host, outside make check.

Model: opus-5-5
 2026-10-21 18:12:55 +02:00
clawbot a19cca2f34 Store a source the way a variant is stored, one row and one file per source URL (closes #308)
check / check (push) Canceled after 0s
Details 
A source is now one source_content row and one file named by its
SourceKey, the SHA-256 of its host, path and query, made as the variant
key is. A store writes the file, then upserts the row, and removes the
file when the row cannot be written; an eviction deletes the row, then
the file, with no lock. source_metadata, the JSON files under
cache/metadata/, MetadataStorage, the content lock, the eviction
transaction and its test hook are gone, as is the reconciliation pass
that dropped a source row whose file is missing. Two source URLs
serving the same bytes store them twice.

Model: opus-5-5
 2026-10-21 16:31:33 +02:00
clawbot 00d8ba26c5 One image request type instead of five (closes #307)
check / check (push) Canceled after 0s
Details 
The new package internal/imagerequest holds the request type, its two
parts Source and Transform, the Format and Fit types, the dimension and
fit checks, the /v1/image/ path parsing and the variant key; it imports
no other package of pixa. imgcache.ParsedURL, imgcache.ImageRequest,
signature.Request, imageprocessor.Request, the second Size, format and
fit types and the conversions between them are gone. The image
processor takes a Transform only. encurl keeps a private struct with the
token's short CBOR field names. VariantKey moves with the function that
makes it. Tests follow the moved and renamed code; the tests of the
removed conversions go with them.

Model: opus-5-5
 2026-10-21 14:41:07 +02:00
clawbot 385c0ca76f Run only script/cibuild in the Gitea workflow (closes #203)
check / check (push) Canceled after 0s
Details 
The workflow's script/docker-smoke step is removed, so the workflow
checks out the repo and runs script/cibuild and nothing else, as the
canonical REPO_POLICIES.md says. The owner ruled on the issue that
make check covers what needs covering.

CI no longer starts the image and waits for its healthcheck.
script/docker-smoke and make docker-smoke stay, to run by hand, and
the script's header comment no longer says the workflow runs it. The
dated TODO.md entries that say the workflow runs it are history and
stay as written.

Model: opus-5-5
 2026-10-21 13:22:59 +02:00
clawbot 989da78681 Keep WebP, AVIF and JPEG XL encodes at the limits under 40 seconds (closes #272)
check / check (push) Canceled after 0s
Details 
WebP is saved at effort 2 and JPEG XL at effort 4, and an AVIF output of
more than 16,777,216 pixels is refused with 400 before anything is
decoded, as AVIF is already at the lowest effort govips can set. With
one libvips thread, at any quality, 8192x8192 random pixels with an
alpha channel take up to about 23 seconds as WebP and 27 as JPEG XL, and
4096x4096, 8192x2048 or 2048x8192 up to about 34 as AVIF. Resizing with
fit=cover already decodes the input, so the output's dimensions are now
worked out before resizing, in outputDimensions. Reading the input moves
into readInput to keep Process within the linter's limits.
TestImageProcessor_WebPAtDefaultEffort becomes TestImageProcessor_WebPAtEffort2, as #280 (comment) allows.

Model: opus-5-5
 2026-10-20 15:19:07 +02:00
clawbot 4fb6350aa1 Answer 415 for a source whose first bytes do not match its Content-Type (closes #253)
check / check (push) Canceled after 0s
Details 
A body whose first bytes are not those of the format its Content-Type
names answered 500 on both image routes, was logged as pixa's own
error, and was fetched again on every request. Both routes now answer
415 for each error the first-bytes check returns, and the image service
logs the refused source at warning level. The failure also gets a
negative cache entry, as an upstream error status does, so the source
URL is not fetched again for 5 minutes, and meanwhile an image that
would need that fetch answers 502. README.md says so.

The case of TestService_Get_ConcurrentMissesShareAFailure that expected
such a source to be fetched again now expects the negative cache, as
#280 (comment) allows.

Model: opus-5-5
 2026-10-20 14:40:51 +02:00
clawbot cde9c0d10d Let a bug fix change the test that pins the behavior the bug calls wrong (closes #280)
check / check (push) Canceled after 0s
Details 
AGENTS.md required the owner's approval for every change to an existing
test. A bug fix may now change an existing test without asking when that
test pins exactly the behavior the filed bug calls wrong, and the change
moves it to the fixed behavior and nothing else. Any other change to an
existing test still needs approval. The owner approved this on the issue.

Model: opus-5-5
 2026-10-20 14:25:07 +02:00
clawbot 957ec85f97 Re-vendor the canonical files from sneak/prompts at 0653579 (closes #221)
check / check (push) Canceled after 0s
Details 
REPO_POLICIES.md, script/cibuild, script/docker, script/lint and
script/test are their copies at that commit. In every Dockerfile stage
the system package install and the Go module download now come before
ARG CACHEBUST, so check builds keep those layers until the base image,
the install line, go.mod or go.sum change; every step that lints, tests
or compiles still reruns. golangci-lint run also mounts golangci-lint's
cache. The stages install their packages with apt-get and apk directly
instead of through script/bootstrap --cgo. The test phase installs no
recommended packages and names libheif's AVIF encoder,
libheif-plugin-aomenc, which the tests use.

Model: opus-5-5
 2026-10-19 06:11:43 +02:00
clawbot 930e43961e Read a trusted_proxies entry in the IPv4-mapped form as its IPv4 range (closes #303)
check / check (push) Canceled after 0s
Details 
An entry such as ::ffff:10.0.0.0/104 in trusted_proxies was accepted
but trusted nothing, because the client address is matched as the IPv4
address an IPv4-mapped address holds. The conversion that blocked_networks
gained in #301 now lives in
parseCIDRList, which reads both settings: such an entry is read as the
IPv4 range it covers, and one shorter than /96 stops startup with an
error naming the setting being read and quoting the entry as written.
README.md says so for trusted_proxies.

Model: opus-5-5
 2026-10-19 05:29:05 +02:00
clawbot 9eacfa6406 Space reconciliation passes by ten times the last pass's time (closes #250)
check / check (push) Canceled after 0s
Details 
The reconciliation pass reads a row and stats a file for every cached
file, and it ran on every eviction tick, so on a large cache a pass
longer than the tick was followed by the next one at once. It still runs
at startup and is considered on each tick, but it now starts no sooner
than ten times as long after the last pass ended as that pass took, so
it takes at most about a tenth of the time however large the cache.
Eviction and the removal of expired negative cache rows still run on
every tick. The comments that bound the drift, and README.md, give the
new bound. No existing test changed.

Model: opus-5-5
 2026-10-19 05:00:48 +02:00
clawbot 0ab15324f7 Read a blocked_networks entry in the IPv4-mapped form as its IPv4 range (closes #301)
check / check (push) Canceled after 0s
Details 
The fetcher matches an IPv4-mapped address (::ffff:a.b.c.d) as the
IPv4 address it holds, so a blocked_networks entry written in that form
was accepted and matched nothing. Such an entry is now kept as the IPv4
range it covers: ::ffff:10.0.0.0/104 becomes 10.0.0.0/8, and
::ffff:0:0/96 becomes 0.0.0.0/0. One shorter than /96 covers more than
IPv4 addresses and aborts startup naming the setting. README.md says
both. trusted_proxies shares the list parser, has the same fault and is
unchanged here (issue 303).

Model: opus-5-5
 2026-10-19 03:00:43 +02:00
clawbot faad959bea Refuse local-use NAT64 addresses, and IPv4-compatible ones in blocked_networks (closes #299)
check / check (push) Canceled after 0s
Details 
The check that keeps pixa from fetching internal addresses let through
64:ff9b:1::/48, the local-use NAT64 range, and matched an address in the
IPv4-compatible form (::/96) against blocked_networks only as IPv6, so
::203.0.113.5 passed with 203.0.113.0/24 listed. The local-use NAT64
range is now blocked next to 64:ff9b::/96, and an address in ::/96 is
refused when blocked_networks holds it as written or the IPv4 address it
holds. The mapped form stays matched as IPv4 only, as Go keeps plain
IPv4 addresses in that form; README.md says so.

Model: opus-5-5
 2026-10-16 06:52:17 +02:00
clawbot 3a6b3bc202 Refuse an IPv6 address that holds an internal IPv4 address (closes #297)
check / check (push) Canceled after 0s
Details 
The check that keeps pixa from fetching internal addresses read an
IPv6 address as IPv4 only in the mapped form (::ffff:a.b.c.d), so the
IPv4-compatible form, such as ::127.0.0.1, was judged as IPv6 and
passed. An address in ::/96 is now judged by the IPv4 address it holds,
so one holding a private, loopback or link-local address is refused on
both image routes before any upstream request. The NAT64 range
64:ff9b::/96 stays blocked whole, as it was on next.

Model: opus-5-5
 2026-10-16 05:31:55 +02:00
clawbot 08d6c3c459 Fetch an IPv6 source host written without a port (closes #295)
check / check (push) Canceled after 0s
Details 
The fetcher took the host from the source URL and stripped a port with
net.SplitHostPort, which fails when there is no port, so a bracketed
IPv6 host such as [2001:db8::a] kept its brackets, its name lookup
failed, and the request answered 500. validateURL now takes the host
with url.URL.Hostname(), which drops the brackets and any port, so such
a host is fetched like one with a port, through /v1/image/ and through
an encrypted URL, and a private IPv6 address is still refused.

Model: opus-5-5
 2026-10-16 05:00:47 +02:00
clawbot efa431ba18 Escape state_dir in the db_url derived from it (closes #290)
check / check (push) Canceled after 0s
Details 
When db_url is omitted, pixa derives it from state_dir, and it put the
directory name into a file: URL as it was, so a state_dir holding %, ?
or # opened another file and lost the journal mode. The derived URL is
now built with net/url, which percent-encodes the directory name; an
ordinary path gives the same URL as before. When the path written into
the URL starts with //, as for a state_dir of / or one starting with //,
the empty host is kept so SQLite does not read the first name in the
path as a host.

Model: opus-5-5
 2026-10-16 04:37:54 +02:00
clawbot 8b4dcda549 Keep a side computed for a size with one side 0 at 1 or more (closes #283)
check / check (push) Canceled after 0s
Details 
A size with one side 0 has that side worked out from the source's
proportions by integer division, and nothing kept it at 1 or more. When
it rounded down to 0, libvips was asked for a square: an 8192x1 source
at 100x0 came out 100x100, at 1300x0 it asked for one block of about
39 GB, and some shapes answered 500. Each computed side, and each side
contain, inside and outside work out again from the scale, is now at
least 1. README.md now says what fit=outside does and what a 0 side
means; with fit=outside an 8192x1 source at 100x0 comes out 8192x1, as
covering the size requires.

Model: opus-5-5
 2026-10-16 04:17:59 +02:00
clawbot 6d271d4617 Make one image URL per source on the generator page for host case and an empty path (closes #293)
check / check (push) Canceled after 0s
Details 
The generator page passed the source host and path into the encrypted
URL as entered, so a host spelled in another letter case, or a source
with no path against the same source with /, made separate cache keys
and fetched one upstream image twice. The page now lower-cases the
source host, as /v1/image/ does, and writes an empty source path as /,
before it builds the encrypted URL. The checks the page gained for
#288 still hold.

Model: opus-5-5
 2026-10-16 04:04:19 +02:00
clawbot 4a4bc9758f Fetch the source path and query URL-encoded, not joined as text (closes #288)
check / check (push) Canceled after 0s
Details 
The source URL was built by joining the decoded path and query as
text, so a space in the query reached the upstream raw and failed with
502, 100%25.png made a URL Go could not parse and answered 401, and
%20 and %2520 fetched the same path under different cache keys. The
source URL is now built with net/url, so each decoded path and query is
one upstream request matching its signature and cache key; the query
encodes the characters a browser encodes, ' included. The generator page
compares what pixa would fetch with the URL as entered, and answers 400
with the URL to use when they differ, or for a user name and password
or a .. segment.

Model: opus-5-5
 2026-10-16 03:31:15 +02:00
clawbot 9a96740463 Crop the centre before resizing for fit=cover (closes #282)
check / check (push) Canceled after 0s
Details 
For fit=cover, the default fit, libvips' thumbnail scaled the whole
source to cover the requested size and copied that into memory before
cutting out the centre, so an 8192x64 source at 8192x8192 asked for
about 24 GiB, and 8192x1 failed with 500. resize now first cuts the
centre of the source, widened to whole source pixels, and gives only
that to thumbnail, which scales both sides by the same amount as before;
the image it holds exceeds the output by less than one scaled source
pixel on each side. Output size and scale are unchanged; its position
can move by up to one output pixel.

Model: opus-5-5
 2026-10-16 02:23:18 +02:00
clawbot 1fd5cf8bd0 Refuse an image URL whose source path or query holds a # (closes #286)
check / check (push) Canceled after 0s
Details 
A # written as %23 in the source path or query reached the parser
decoded and stayed in the source URL, where Go reads it as the start of
a fragment and drops it before fetching. So each text after the # was
another unsigned spelling of one upstream request, cached under its own
keys. An image URL whose decoded source path or query holds a # is now
refused with 400, signed or not, and README.md's rule for writing a
source path and query says a # cannot be part of either.

Model: opus-5-5
 2026-10-16 01:36:56 +02:00
clawbot 62b4f4791e Refuse an image URL whose host is not a host name or IP address (closes #275)
check / check (push) Canceled after 0s
Details 
A ? written into the host segment as %3F reached the parser decoded,
so cdn.example.com?x=1 passed the allowlist as cdn.example.com, fetched
the host's root path with that query, and was cached under its own keys:
another unsigned spelling of one upstream request. The host segment is
now accepted only as a host name, an IPv4 address or a bracketed IPv6
address, with an optional port, and anything else is refused with 400.
An IPv6 address without brackets is now refused instead of needing a
signature; a host name outside ASCII is refused, its punycode accepted.

Model: opus-5-5
 2026-10-16 01:14:51 +02:00
clawbot 22bfc432b6 Document how a source query goes into an image URL, and accept it (closes #255)
check / check (push) Canceled after 0s
Details 
The README gave no way to put a source URL's query into a signed image
URL, and GenerateSignedURL built one that failed: the route parsed
chi's wildcard, which keeps the path as the client wrote it, so one
%2F, %3B or %2C left the %3F undecoded and the signature failed. The
route now parses the decoded path, GenerateSignedURL writes Go's own
encoding of it, and the README says the ? before the query, any ? in it
and every % must be percent-encoded, with a signed example. The path
traversal check no longer reads the query and accepts a literal %, while
a double-encoded .. stays refused. A percent-escape in a source path is
now decoded once.

Model: opus-5-5
 2026-10-16 00:17:33 +02:00
clawbot f743930f71 Default max_concurrent_processing to half the CPUs pixa sees (closes #242)
check / check (push) Canceled after 0s
Details 
The default max_concurrent_processing was one image per CPU pixa sees,
so in a container with no CPU limit on a 48-CPU host pixa processed 48
images at once. It is now half the CPUs, rounded down and at least 1,
defined once in internal/config; PIXA_MAX_CONCURRENT_PROCESSING stays
the override, and a value that is not a whole number of at least 1
stops startup. "Running under upaas" gives the worst-case memory, 4.2
GiB per image (the largest peak measured over every source format and
kind at 8192x8192) plus the upstream buffers, and the temporary file
libvips keeps under /tmp. The two existing tests expect the new
default, as the owner allowed.

Model: opus-5-5
 2026-10-15 23:07:22 +02:00
clawbot 4e30dc7467 Re-vendor the canonical files from sneak/prompts at e3f79c4 (closes #221)
check / check (push) Canceled after 0s
Details 
The dotfiles, REPO_POLICIES.md and the standard scripts are fetched
whole from sneak/prompts e3f79c4, with pixa's own entries kept after
them. Check builds pass a new CACHEBUST build argument instead of
--no-cache, and every Dockerfile stage declares it first, so every RUN
step still runs while Go's build cache is kept. Lint runs golangci-lint
v2.14.0, and the test phase runs on the Debian Go image. Of the two new
lint findings, one is fixed and one (gosec G703 on the fixed config file
locations) is suppressed. The agent guidance moves unchanged from
CLAUDE.md to AGENTS.md.

Deviation: the workflow keeps its docker-smoke step while issue 203
waits on the owner.

Model: opus-5-5
 2026-10-15 22:40:38 +02:00
clawbot 16f9d1dfd1 Label request metrics with standard HTTP methods only (closes #274)
check / check (push) Canceled after 0s
Details 
The metrics middleware labelled each request with its method as sent,
and Go's HTTP server accepts any method name that is a valid token, so
each made-up method a client sent added series that were never removed.
The method label is now the request's method only for the nine standard
methods, and "other" for any other. The middleware hands the metrics
library a copy of the request carrying the label and serves the next
handler the original, so handlers still see the method the client sent.

Model: opus-5-5
 2026-10-15 21:47:11 +02:00
clawbot 742db9ceb1 Logging out ends every login session (closes #256)
check / check (push) Canceled after 0s
Details 
A login session was a cookie with nothing kept on the server, so
logging out removed it from one browser and a copy kept working for 30
days. The session manager now keeps a random value in the database,
made when it has none and replaced by each logout; a session carries
the value current at its login and is refused once it has changed. The
logout check compares no time, so the host's clock cannot block a
login, and losing the state directory ends every login session.
Logging out is now a POST with the CSRF token, sent by a form on the
generator page.

Model: opus-5-5
 2026-10-15 20:44:44 +02:00
clawbot 2ab96155ba Sync cached files and refuse ones left empty or partial (closes #252)
check / check (push) Canceled after 0s
Details 
A crash could leave a cached source or transformed image empty or cut
short, and pixa then served it, or fetched it again forever. Source and
transformed image files, and the .meta file beside each transformed
image, are now written under a temporary name, synced, renamed into
place and their directory synced; a failed directory sync removes the
renamed file. The reconciliation pass compares each cached file's size
with the size its row records and removes a file that is empty or of
another size, with its rows and a warning. Requests compare no sizes.

Not verified by a test: the syncs, and the removal after a failed sync.

Model: opus-5-5
 2026-10-15 19:27:33 +02:00
clawbot 3ba9ae9c39 Stop a transcode whose request has ended before it encodes (closes #243)
check / check (push) Canceled after 0s
Details 
Process looked at its context only while waiting for a processing
slot, so a transcode kept its slot and worked on after its request's
downstream_timeout. acquireSlot now gives the slot back when the
context has ended by the time it has one, and encode returns the
context's error without beginning. govips gives pixa no way to stop
libvips once an encode has begun, so that encode keeps its slot. A
request waiting on a shared transcode stopped at the first request's
deadline is answered 503, not 500. README.md "Routes" says both.

Model: opus-5-5
 2026-10-15 18:44:11 +02:00
clawbot a3aa73f6fd Tell upaas operators to set the cache size limit (closes #251)
check / check (push) Canceled after 0s
Details 
"Running under upaas" in README.md listed PIXA_CACHE_MAX_BYTES as
optional. Under upaas the volume is a directory on the host's
filesystem, and the default limit is 75% of the sum of that
filesystem's free space and what the cache holds, at least 500 MiB, so
one pixa can take three quarters or more of the space every app has
left, and all of it when that is 500 MiB or less. The section now tells
the operator to set the variable and says what the default does there.
The default is unchanged.

Model: opus-5-5
 2026-10-15 18:16:17 +02:00
clawbot 7956da1f59 Label request metrics by route, not by path (closes #254)
check / check (push) Canceled after 0s
Details 
The metrics middleware labelled each request with its path, so every
new path added series to the registry that were never removed, and
/metrics listed the token of every encrypted URL requested. The handler
label is now the pattern of the chi route the request will match, looked
up before routing, or "unmatched" when none does. /v1 and /v1/ ask the
image subrouter, and an empty path is looked up as /, as chi routes it.
TestMetricsRecordsServedRequest expects "unmatched" in place of "/test",
as the owner approved.

Model: opus-5-5
 2026-10-15 16:37:20 +02:00
clawbot 69f65d4302 Remove expired negative cache rows in the eviction loop (closes #244)
check / check (push) Canceled after 0s
Details 
A negative cache row was deleted only when its URL was requested
again, so a client asking for a new missing path each time grew the
database without limit. The eviction goroutine now removes expired rows
at startup and on every tick, so a row stays about NegativeTTL plus one
eviction interval. It deletes at most 1000 rows per statement, so a
request on the one database connection waits behind one batch at most.
The goroutine now also runs when the disk cache is disabled, since the
negative cache is used either way.

Model: opus-5-5
 2026-10-15 15:02:27 +02:00
clawbot 2790125cb2 Serve a cached variant while its source URL is in the negative cache (closes #245)
check / check (push) Canceled after 0s
Details 
Get checked the negative cache before the variant cache, so after one
failed fetch of a source every variant of it answered 502 for five
minutes, also one on disk or one that could be made from the cached
source. The check now sits at the start of fetchAndProcess, which runs
only when no cached source can be used. A refusal from the negative
cache now counts as a miss.

Model: opus-5-5
 2026-10-15 13:45:25 +02:00
clawbot c23585dd6d Lower-case the image URL's host and require a signature for a port (closes #246)
check / check (push) Canceled after 0s
Details 
With `cdn.example.com` allowlisted, other letter cases and a host with a port also passed the allowlist; each spelling was fetched and cached on its own with its own per-host connection limit, and a port let an unsigned URL reach any port of the host.

The image URL parser now lower-cases the host and refuses one holding `@` with 400. A host followed by `:`, with or without a port (read as Go's URL parser reads it, unbracketed IPv6 included), never counts as allowlisted, so it needs a signature. The host is signed lower-cased, as the URL writes it, so a signature made over an upper-case spelling is refused with 401.

Model: opus-5-5
 2026-10-15 12:31:14 +02:00
clawbot 5d1e3ea871 Escape : and % in the signed and cached path and query (closes #247)
check / check (push) Canceled after 0s
Details 
The signed data and the cache key joined host, path and query with ":" and no escaping, so path /a:b with no query and path /a with query b: gave the same string: a URL signed for https://h/a:b verified for https://h/a?b:, and the two shared a cached image. Both now write each % in the path and the query as %25 and each : as %3A before joining them. The host never holds / and the path always starts with /, so the joined string reads back one way only. A path and query holding neither character are signed and keyed as before, so the existing known-answer vectors and the README example are unchanged; new vectors cover the escaped cases. Signature Specification in README.md states the escaping.

Model: opus-5-5
 2026-10-15 09:37:01 +02:00
clawbot 761ade2e36 Accept a chunked upstream body of exactly the maximum size (closes #248)
check / check (push) Canceled after 0s
Details 
Once upstream_max_response_size bytes were read, the fetcher refused the next read without looking for the end of the body. Go's HTTP client returns the end of a body sent with a Content-Length together with its last bytes, but the end of a chunked body can come in a later read, so a chunked body of exactly the limit was refused after it was downloaded whole and the image routes answered 500. Once the limit is used up, the fetcher now reads one more byte: if the body has ended it is accepted, and if there is more it is refused with ErrResponseTooLarge.

Model: opus-5-5
 2026-10-15 07:44:09 +02:00
clawbot 88e1b1cdfa Allowlist no host in the example config (closes #249)
check / check (push) Canceled after 0s
Details 
The example config allowlisted github.com and user-images.githubusercontent.com, where anyone can put an image, so a deployment set up as Getting Started says served every image there to anyone without a signature. The example now allowlists nothing and shows one commented entry under example.com. README.md says in Getting Started and in "Running under upaas" that every image on an allowlisted host is served without a signature, so an allowlisted host must be one the operator controls. A new test loads the example config and fails if it allowlists any host.

Model: opus-5-5
 2026-10-15 06:26:23 +02:00
clawbot afb5dc5d0d Refuse a fit=outside size whose output exceeds the maximum (closes #257)
check / check (push) Successful in 15m51s
Details 
fit=outside scales the source to cover the requested size, and the side
scaled past it was held to no maximum: 1x8192?fit=outside of a 3000x600
source was resized to 40960x8192 and encoded. resize now refuses such an
output with ErrOutputTooLarge before resizing, so both image routes answer
it 400 with the error a size with one side 0 already gets. The other fit
modes make the requested size or one within it. README.md and TODO.md
say so.

Model: opus-5-5
 2026-10-13 10:44:45 +02:00
clawbot d8760c40a1 Refuse a size whose computed side exceeds the maximum (closes #241)
check / check (push) Successful in 14m47s
Details 
A side given as 0 is computed from the source's proportions, and
nothing held it to 8192: 0x8192 of a 3000x600 source was resized to
40960x8192 and encoded. Process now refuses such a size with
ErrOutputTooLarge once it knows the source's dimensions, before the
resize and the encode, and both image routes answer it 400 with the
JSON error body. It is a processing error, not a failed fetch, so it
is never negative-cached. imgcache.MaxDimension is now
imageprocessor.MaxOutputDimension, so the request limit and the output
limit are one value. README.md says it under Routes and Encrypted URLs.

Model: opus-5-5
 2026-10-13 09:44:44 +02:00
190 changed files with 19443 additions and 12946 deletions
Expand all files Collapse all files

No files matched your search

.dockerignore
+18 -9
View File
@@ -13,12 +13,21 @@
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and # `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context. # deletes the package directory from the context.
# Unlike the standard file, which leaves out all of .git, pixa sends # .git is sent without its config. Without a VERSION build argument the
# .git without its config. Without a VERSION build argument the stage # stage that compiles runs `git describe --tags --always` on .git, which
# that compiles runs `git describe --tags --always` on .git, which does # does not need .git/config; that file can hold a credential, such as a
# not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there. # password in a remote URL or the token the CI checkout step stores there.
.git/config # Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent. # Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root. # Anchored because it occurs once where agents run at the repo root.
@@ -42,7 +51,9 @@
**/[iI][dD]_[rR][sS][aA] **/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA] **/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA] **/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519 **/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in. # Dependencies: restored inside the image, never copied in.
**/node_modules **/node_modules
@@ -60,10 +71,8 @@
**/.vscode **/.vscode
**/*.sublime-* **/*.sublime-*
# pixa's own entries. Nothing in the build reads .gitignore. On the # pixa's own entries. On the host, `make build` writes bin/pixad, and the
# host, `make build` writes bin/pixad, and the example config keeps its # example config keeps its state directory in data/.
# state directory in data/.
.gitignore
/bin /bin
/data /data
.editorconfig
+3
View File
@@ -13,3 +13,6 @@ indent_style = tab
[*.go] [*.go]
indent_style = tab indent_style = tab
# This repository's own sections, such as one for another language it
# uses, go below this comment, and a re-vendor keeps them.
.gitea/workflows/check.yml
+9 -4
View File
@@ -1,15 +1,20 @@
name: check name: check
on: [push] on: [push]
# Free the shared runner: a new push cancels only the same branch's older run.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs: jobs:
check: check:
runs-on: ubuntu-latest runs-on: ubuntu-latest
# Free the shared runner from a hung build.
timeout-minutes: 20
steps: steps:
# actions/checkout v4.2.2, 2026-02-22 # actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
# The default clone is shallow and has no tags, so the # script/cibuild needs no token, so none is left in .git/config.
# version the build takes from `git describe` would be a
# bare commit; this fetches the whole history with its tags.
with: with:
persist-credentials: false
# All history and tags, so git describe finds the version tag.
fetch-depth: 0 fetch-depth: 0
- run: script/cibuild - run: script/cibuild
- run: script/docker-smoke
.gitignore
+37 -7
View File
@@ -17,15 +17,46 @@ Thumbs.db
# entry and must not be given a `**/` prefix on the way into one. # entry and must not be given a `**/` prefix on the way into one.
.claude/ .claude/
# Environment / secrets # Node
.env node_modules/
.env.*
*.pem # Secrets. Unanchored like every entry above, so each matches at every
*.key # depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation at the end of this file, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, such as its build outputs, go below
# this comment, and a re-vendor keeps them. Anchor a binary built at the
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
# Go
*.log
*.out
*.test
# Dependencies # Dependencies
vendor/ vendor/
node_modules/
# Build output # Build output
/bin/ /bin/
@@ -39,4 +70,3 @@ node_modules/
# Local dev configs # Local dev configs
config.yml config.yml
config.yaml config.yaml
config.dev.yml
.golangci.yml
+3
View File
@@ -17,6 +17,7 @@ linters:
disable: disable:
# Genuinely incompatible with project patterns # Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields - exhaustruct # Requires all struct fields
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
- godot # Requires comments to end with periods - godot # Requires comments to end with periods
- wrapcheck # Too verbose for internal packages - wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go - varnamelen # Short names like db, id are idiomatic Go
@@ -24,6 +25,8 @@ linters:
# silenced by disabling that name, not by enabling the successor. # silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5 - wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2 - gomodguard # Deprecated, replaced by gomodguard_v2
# Misses findings at random in v2.14.0; back once a pinned release fixes it
- canonicalheader
settings: settings:
lll: lll:
line-length: 88 line-length: 88
.prettierignore
-5
View File
@@ -1,7 +1,2 @@
node_modules/ node_modules/
yarn.lock yarn.lock
# A byte-for-byte copy of the one in sneak/prompts.
REPO_POLICIES.md
vendor/
AGENTS.md
+76
View File
@@ -0,0 +1,76 @@
# Repository Rules
Last Updated 2026-10-27
These rules MUST be followed at all times. This is very important.
- Never use `git add -A`. Stage specific changes for a deliberate commit. A
commit holds one change. After each change, make a commit with a good one-line
summary.
- NEVER modify the linter config without asking first.
- NEVER modify a test to exclude special cases or otherwise make it pass without
asking first. In almost all cases, change the code, NOT the test. You need
explicit user approval to modify an existing test. If you think a test needs
to change, make your case and ask for permission to proceed. Then stop. A bug
fix may change an existing test without asking only when that test pins
exactly the behavior the filed bug calls wrong. The change must then move the
test to the fixed behavior and nothing else. Any other change to an existing
test needs explicit user approval. Writing NEW tests needs no approval.
- When linting, assume the linter config is CORRECT. Treat each item the linter
reports as something that legitimately needs fixing in the code.
- When running tests, use `make test`.
- Before committing, run `make check`. It runs `make test`, `make lint` and
`make fmt-check`. Any issue it finds MUST be resolved before committing,
unless you are explicitly told otherwise.
- When fixing a bug, write a failing test for the bug FIRST. Add appropriate
logging to the test, to make sure the test is written correctly. Commit that.
Then work on the fix until the test passes, without changing the test further.
Then commit the fix.
- When adding a new feature, do the same: write a test first (test-driven
development, TDD). The test does not need to be very complex. Commit the test,
then commit the feature.
- When adding a new feature, use a feature branch. The feature branch reaches
`next` through a PR that passes review. It is merged only when the feature is
completely finished and the code is up to standards. The code is up to
standards when it passes `make check`. The branch is deleted after the merge.
Only the user merges `next` into `main`.
- Write `godoc` comments for all exported types and functions as you go along.
- ALWAYS be consistent in naming. If you name something one way in one place,
use the EXACT SAME name in every other place.
- Be descriptive and specific in naming. `al` is bad; `SourceHostAllowlist` is
good. `ConnsPerHost` is bad; `MaxConnectionsPerHost` is good.
- This is not prototype or teaching code. It is designed for production. Any
security issue (such as denial of service) or other web vulnerability is a P1
bug. It must be added at the top of `TODO.md`.
- Do not stub an implementation unless you are specifically instructed to. This
is production code, and we need working implementations.
- NEVER silently fall back to a different setting when a user's parameter
explicitly specifies a value. For example, if a user requests `format=webp`
and WebP encoding is not supported, return an error. Do NOT silently output
PNG instead. If a user specifies `fit=invalid` and that fit mode does not
exist, return an error. Do NOT silently default to `cover`. A silent fallback
violates the principle of least surprise and masks bugs. Defaults are
acceptable only for OMITTED parameters, never for INVALID explicit values.
- Avoid vendoring dependencies unless specifically instructed to. NEVER commit
the `vendor/` directory. NEVER commit compiled binaries. If such a directory
or file exists and `.gitignore` does not list it, add it to `.gitignore`.
Commit that `.gitignore` change. Keep the entire git repository, history
included, under 20 MiB. Go over 20 MiB only when you specifically must commit
a large file, such as a media file used as a test fixture. Only OUR source
code and its immediately supporting files, such as test examples, go into the
repository and its history.
CLAUDE.md
-71
View File
@@ -1,71 +0,0 @@
# Repository Rules
Last Updated 2026-01-08
These rules MUST be followed at all times, it is very important.
- Never use `git add -A` - add specific changes to a deliberate commit. A commit
should contain one change. After each change, make a commit with a good
one-line summary.
- NEVER modify the linter config without asking first.
- NEVER modify tests to exclude special cases or otherwise get them to pass
without asking first. In almost all cases, the code should be changed, NOT the
tests. If you think the test needs to be changed, make your case for that and
ask for permission to proceed, then stop. You need explicit user approval to
modify existing tests. (You do not need user approval for writing NEW tests.)
- When linting, assume the linter config is CORRECT, and that each item output
by the linter is something that legitimately needs fixing in the code.
- When running tests, use `make test`.
- Before commits, run `make check`. This runs `make lint` and `make test` and
`make check-fmt`. Any issues discovered MUST be resolved before committing
unless explicitly told otherwise.
- When fixing a bug, write a failing test for the bug FIRST. Add appropriate
logging to the test to ensure it is written correctly. Commit that. Then go
about fixing the bug until the test passes (without modifying the test
further). Then commit that.
- When adding a new feature, do the same - implement a test first (TDD). It
doesn't have to be super complex. Commit the test, then commit the feature.
- When adding a new feature, use a feature branch. When the feature is
completely finished and the code is up to standards (passes `make check`) then
and only then can the feature branch be merged into `main` and the branch
deleted.
- Write godoc documentation comments for all exported types and functions as you
go along.
- ALWAYS be consistent in naming. If you name something one thing in one place,
name it the EXACT SAME THING in another place.
- Be descriptive and specific in naming. `wl` is bad; `SourceHostWhitelist` is
good. `ConnsPerHost` is bad; `MaxConnectionsPerHost` is good.
- This is not prototype or teaching code - this is designed for production. Any
security issues (such as denial of service) or other web vulnerabilities are
P1 bugs and must be added to TODO.md at the top.
- As this is production code, no stubbing of implementations unless specifically
instructed. We need working implementations.
- NEVER silently fall back to a different setting when a user's parameter
explicitly specifies a value. If a user requests format=webp and WebP encoding
is not supported, return an error - do NOT silently output PNG instead. If a
user specifies fit=invalid and that fit mode doesn't exist, return an error -
do NOT silently default to "cover". Silent fallbacks violate the principle of
least surprise and mask bugs. The only acceptable defaults are for OMITTED
parameters, never for INVALID explicit values.
- Avoid vendoring deps unless specifically instructed to. NEVER commit the
vendor directory, NEVER commit compiled binaries. If these directories or
files exist, add them to .gitignore (and commit the .gitignore) if they are
not already in there. Keep the entire git repository (with history) small -
under 20MiB, unless you specifically must commit larger files (e.g. test
fixture example media files). Only OUR source code and immediately supporting
files (such as test examples) goes into the repo/history.
Dockerfile
+68 -41
View File
@@ -1,7 +1,16 @@
# In every stage ARG CACHEBUST comes right after the steps that install
# system packages and download Go's modules, or right after FROM in a
# stage with neither, and every step that lints, tests or compiles
# comes after it. script/ passes a new value on every build, which
# reruns every RUN step after it. Docker keeps the layers above it
# until the base image, their own lines or go.mod and go.sum change.
# The steps that compile mount Go's build cache, and golangci-lint run
# also mounts golangci-lint's; both survive from build to build.
# Lint phase. script/lint builds it alone. The linter is run directly: # Lint phase. script/lint builds it alone. The linter is run directly:
# `make lint` and script/lint are themselves a docker build. # `make lint` and script/lint are themselves a docker build.
# golangci/golangci-lint:v2.12.2, 2026-10-04 # golangci/golangci-lint:v2.14.0, 2026-10-15
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
# The linter compiles every package, and govips needs the libvips # The linter compiles every package, and govips needs the libvips
# headers for that. REPO_POLICIES.md has the lint phase install them # headers for that. REPO_POLICIES.md has the lint phase install them
@@ -13,70 +22,87 @@ RUN apt-get update \
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
RUN golangci-lint run --config .golangci.yml ./... RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/root/.cache/golangci-lint \
golangci-lint run --config .golangci.yml ./...
# Test phase. script/test builds it alone. # Test phase. script/test builds it alone. -race needs cgo and so a C
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too # compiler, which the Debian Go image ships and the alpine one does not.
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS test # golang:1.25.4-trixie, 2026-10-15
FROM golang@sha256:a02d35efc036053fdf0da8c15919276bf777a80cbfda6a35c5e9f087e652adfc AS test
# govips compiles against the libvips and libheif headers, and the tests
# need libvips' JPEG XL support; libvips-dev brings all three. The tests
# also encode AVIF, which needs libheif's AVIF encoder,
# libheif-plugin-aomenc: Debian only recommends it, so it is named here.
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev libheif-plugin-aomenc \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /src WORKDIR /src
# script/bootstrap --cgo installs the build dependencies (a C compiler,
# the libvips and libheif headers, and libvips' JPEG XL support, which
# the tests need) and downloads the Go modules.
COPY script/ ./script/
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN script/bootstrap --cgo RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
# Without -v first; on a failure, again with -v for the details, and # Without -v first; on a failure, again with -v for the details, and
# the step fails even if the second run passes. -parallel 4: by default # the step fails even if the second run passes. -count=1 because the
# a package runs as many of its tests at once as the host has CPUs, and # mounted cache also holds test results, which go test would otherwise
# on a busy host they then wait so long to be scheduled that a test # replay. -parallel 4: by default a package runs as many of its tests at
# that times a request can see it return a second late. # once as the host has CPUs, and on a busy host they then wait so long
RUN go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \ # to be scheduled that a test that times a request can see it return a
# second late.
RUN --mount=type=cache,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \ { echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; } go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; }
# Build stage. Nothing is wanted from the two phases above: these copies # Build stage. Nothing is wanted from the two phases above: these copies
# make BuildKit build them first, so this stage runs only when lint and # make BuildKit build them first, so nothing after them runs unless lint
# test passed. # and test passed.
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too # golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
# git for the version, and for govips a C compiler, pkg-config and the
# libvips and libheif headers
RUN apk add --no-cache git build-base pkgconfig vips-dev libheif-dev
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
WORKDIR /src # A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
# Build dependencies and Go modules, as in the test phase
COPY script/ ./script/
COPY go.mod go.sum ./
RUN script/bootstrap --cgo
# Copy source code # Copy source code
COPY . . COPY . .
# VERSION is declared here, not earlier: a new value reruns only the # The VERSION build arg when one is given, otherwise
# build, not script/bootstrap. Given none, the version is # `git describe --tags --always` on the .git in the build context: the
# `git describe --tags --always` of the .git in the build context (git # tag on a tagged commit, tag-N-gHASH after one, the short commit when
# comes from script/bootstrap): the tag on a tagged commit, tag-N-gHASH # no tag is reachable. With .git present, a version that is still empty,
# after one, the short commit when no tag is reachable. A context that # dev or unknown fails the build: git is missing or could not read the
# carries .git and still yields no version fails the build; one without # checkout. A context without .git, as from a source tarball, stamps an
# .git, as from a source tarball, stamps an empty version. CGO stays # empty version. CGO stays enabled for govips; -trimpath keeps build
# enabled for govips; -trimpath keeps build paths out of the binary, and # paths out of the binary, and -s -w leave out the symbol table and
# -s -w leave out the symbol table and debug information. # debug information.
ARG VERSION ARG VERSION
RUN version="${VERSION:-$(git describe --tags --always)}"; \ RUN --mount=type=cache,target=/root/.cache/go-build \
if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \ VERSION="${VERSION:-$(git describe --tags --always)}"; \
[ "$version" = unknown ]; }; then \ if [ -e .git ]; then \
echo "the build context carries .git but yields no version" >&2; \ case "$VERSION" in ""|dev|unknown) \
exit 1; \ echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \ fi; \
CGO_ENABLED=1 GOTOOLCHAIN=auto go build -trimpath \ CGO_ENABLED=1 GOTOOLCHAIN=auto go build -trimpath \
-ldflags "-s -w -X main.Version=${version}" \ -ldflags="-s -w -X main.Version=${VERSION}" \
-o /pixad ./cmd/pixad -o /pixad ./cmd/pixad
# Runtime stage, and the last one: a plain `docker build .` builds this # Runtime stage, and the last one: a plain `docker build .` builds this
@@ -95,6 +121,7 @@ RUN apk add --no-cache \
ca-certificates \ ca-certificates \
tzdata \ tzdata \
su-exec su-exec
ARG CACHEBUST
# Copy binary from builder # Copy binary from builder
COPY --from=builder /pixad /usr/local/bin/pixad COPY --from=builder /pixad /usr/local/bin/pixad
Makefile
+14 -5
View File
@@ -1,4 +1,4 @@
.PHONY: bootstrap setup check lint test fmt fmt-check build clean docker docker-smoke docker-versioned docker-test devserver devserver-stop hooks loadtest .PHONY: bootstrap setup check lint test fmt fmt-check build clean docker docker-smoke docker-versioned docker-test devserver devserver-stop hooks loadtest bench
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev") VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo "dev")
LDFLAGS := -X main.Version=$(VERSION) LDFLAGS := -X main.Version=$(VERSION)
@@ -64,6 +64,12 @@ docker-smoke:
loadtest: loadtest:
@script/loadtest @script/loadtest
# Serve cache hits in parallel through the image handler and report hits per
# second (needs Go on the host after `script/bootstrap --cgo`, not Docker; a
# benchmark, not part of check)
bench:
@script/bench
# Build Docker image as `make docker` does, and also tag it pixa:$(VERSION) # Build Docker image as `make docker` does, and also tag it pixa:$(VERSION)
docker-versioned: docker-versioned:
@script/docker @script/docker
@@ -73,12 +79,15 @@ docker-versioned:
docker-test: docker-test:
@script/test @script/test
# Run local dev server in Docker # Run local dev server in Docker, with debug on and a new random signing
# key each run. Log in to the page with the key it prints.
devserver: docker-versioned devserver-stop devserver: docker-versioned devserver-stop
@key="$$(head -c 32 /dev/urandom | base64)" && \
docker run -d --name pixad-dev -p 8080:8080 \ docker run -d --name pixad-dev -p 8080:8080 \
-v $(CURDIR)/config.dev.yml:/etc/pixa/config.yml:ro \ -e PIXA_SIGNING_KEY="$$key" -e PIXA_DEBUG=true \
pixa:latest pixa:latest >/dev/null && \
@echo "pixad running at http://localhost:8080" echo "pixad running at http://localhost:8080" && \
echo "signing key: $$key"
# Stop dev server # Stop dev server
devserver-stop: devserver-stop:
README.md
+1243 -577
View File
File diff suppressed because it is too large. Load diff
REPO_POLICIES.md
+295 -109
View File
@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-09-08 last_modified: 2026-10-18
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -54,34 +54,39 @@ style conventions are in separate documents:
`cibuild`. `script/bootstrap` installs all dependencies idempotently and `cibuild`. `script/bootstrap` installs all dependencies idempotently and
assumes nothing is present: base tools come from nix, apt, brew, or apk assumes nothing is present: base tools come from nix, apt, brew, or apk
(detected in that order; apt runs noninteractive). For node it uses the (detected in that order; apt runs noninteractive). For node it uses the
installed node if present; otherwise it installs a PINNED node version via installed node only when its major version is the pinned one; otherwise (node
nvm, first installing nvm itself if missing — from a hash-verified GitHub missing, or another major, such as the node 24 the CI runner image ships) it
release archive (never `curl | sh`), with bash installed as an explicit installs the PINNED node version via nvm and installs yarn and the packages
prerequisite since nvm requires bash. yarn is then pinned via under it. nvm itself is installed first if missing, from a hash-verified
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts"; GitHub release archive (never `curl | sh`), with bash installed as an explicit
always exact versions. `script/cibuild` runs the CI build: it changes to the prerequisite since nvm requires bash. Only the major version is compared
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image because the `Dockerfile` stages start from a node image whose exact version is
with the version; the Gitea workflow calls it. **`script/cibuild` runs not the pin, and nvm cannot install a prebuilt node on alpine. yarn is pinned
via `corepack prepare yarn@<version> --activate`. Never install "latest" or
"lts"; always exact versions. `script/cibuild` runs the CI build: it changes
to the repo root, runs `script/bootstrap`, runs `script/check`, and builds the
image with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/bootstrap` first**, because the workflow checks out the repo and runs `script/bootstrap` first**, because the workflow checks out the repo and runs
nothing else, while `script/fmt-check` runs the formatter on the host: on a nothing else, while `script/fmt-check` runs the formatter on the host: on a
pristine checkout with nothing installed the run dies there, after the pristine checkout with nothing installed the run dies there, after the
containerised gates have passed. **The bootstrap alone is not enough**: containerised gates have passed. **The bootstrap alone is not enough**: when
`script/bootstrap` installs node and yarn under nvm and leaves neither on the `script/bootstrap` installs node and yarn under nvm it leaves neither on the
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host `PATH` of the shell that called it, so a bare `yarn` either exits 127 or runs
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore under another node. The host entrypoints that need yarn — `script/fmt` and
source nvm for the pinned node version before invoking it, exactly as `script/fmt-check` — therefore use the same test as `script/bootstrap`: when
`script/bootstrap`'s own install step does. A runner carrying nothing but the node on `PATH` has the pinned major version they run the `yarn` on `PATH`,
docker and git then gets through `script/check`. Four further scripts are our and otherwise they load nvm through `$HOME/.nvm/nvm.sh` and run yarn under the
own extensions to the standard: `script/check` runs `script/test`, pinned node. A runner carrying nothing but docker and git then gets through
`script/lint` and `script/fmt-check`; `script/precommit` is what the git `script/check`. Four further scripts are our own extensions to the standard:
pre-commit hook runs, and it calls `script/check`; `script/install-precommit` `script/check` runs `script/test`, `script/lint` and `script/fmt-check`;
installs the git pre-commit hook (the `make hooks` target shims to it); and `script/precommit` is what the git pre-commit hook runs, and it calls
`script/projectname` (literally that filename) simply outputs the project's `script/check`; `script/install-precommit` installs the git pre-commit hook
name. Scripts that need the name call `script/projectname` — e.g. (the `make hooks` target shims to it); and `script/projectname` (literally
`script/docker` assembles its image tag from it — so those scripts stay that filename) simply outputs the project's name. Scripts that need the name
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g. call `script/projectname` — e.g. `script/docker` assembles its image tag from
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in it — so those scripts stay byte-identical across all repos. Repo-type-specific
the hook itself. Model scripts are at pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/precommit`, not in the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the must document the provided scripts in an **Entrypoints** section (see the
README requirements below). README requirements below).
@@ -104,18 +109,24 @@ style conventions are in separate documents:
`lint` phase and a `test` phase, with the final stage depending on both so the `lint` phase and a `test` phase, with the final stage depending on both so the
image cannot be built unless they pass. For non-server repos the final stage image cannot be built unless they pass. For non-server repos the final stage
brings up a development environment; for server repos it is the runtime image. brings up a development environment; for server repos it is the runtime image.
Dockerfiles install development prerequisites by running `script/bootstrap` The gate phases and the build stage start from their pinned base images and
rather than duplicating installs inline; COPY `script/` and the dependency install what those images lack either inline, as the canonical Go `Dockerfile`
manifests (`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before below does for `git`, or by running `script/bootstrap`, as the `prompts`
running it. repo's own `Dockerfile` does for its yarn packages. The development
environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is - **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
no separate lint file. `script/lint` and `script/test` each build one phase no separate lint file. `script/lint` and `script/test` each build one phase
and nothing else: and nothing else:
```sh ```sh
docker build --no-cache --target lint -t "$(script/projectname)-lint" . docker build --build-arg CACHEBUST="$(date +%s)-$$" \
docker build --no-cache --target test -t "$(script/projectname)-test" . --target lint --output type=cacheonly .
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--target test --output type=cacheonly .
``` ```
**A stage that is not the last one in the file is built only when the final **A stage that is not the last one in the file is built only when the final
@@ -125,10 +136,15 @@ style conventions are in separate documents:
plain `docker build .` builds the last stage alone and exits 0 having linted plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing. and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in **The gate builds write no image.** With `--output type=cacheonly` the phase
`script/cibuild` and `script/docker`. An untagged build leaves a dangling runs and a failing step fails the build, but the result is not exported.
image behind on every invocation, on every developer host and every CI Nothing uses those images, and writing one out is slow: a Go test phase's
runner; a tagged one replaces the previous image. image holds the toolchain. A build given neither `--output` nor `-t` writes
an untagged image and leaves it dangling, on every developer host and every
CI runner. `script/cibuild` and `script/docker` build the image that ships
and tag it, so each build replaces the previous image; each assigns the tag
on its own line before the build, so `set -e` stops it where
`script/projectname` fails.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`, Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are `eslint`, `prettier` — never through `make lint` or `script/test`, which are
@@ -143,66 +159,122 @@ style conventions are in separate documents:
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and org, in both directions. A container has its own `TMPDIR`, so the lock is
a digest-pinned binary, so neither is reachable. out of reach. The lint phase does keep golangci-lint's cache between check
builds (below), but it lints every checkout at `/src`, and a finding is
taken from that cache only for the same files, settings and golangci-lint
version.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates - **Any build that runs checks is given a new `CACHEBUST` value.** Docker
a `COPY` layer only when the copied content changes, so on an unchanged tree invalidates a `COPY` layer only when the copied content changes, so on an
the check `RUN` is served from cache, nothing executes, and the build still unchanged tree the check `RUN` is served from cache, nothing executes, and the
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`: build still exits 0. Every `docker build` in `script/` therefore passes
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the `--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the script's process
four, and there is no fifth — `script/check` runs the two gate phases and id, so no two builds share a value, not even the three in one
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is `script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
not evidence that anything ran: a sub-second build reporting success is a `script/docker` are the four, and there is no fifth — `script/check` runs the
cache hit, not a result. Never invalidate by pruning — `docker builder prune` two gate phases and `script/fmt-check`, and builds no image of its own. In
and friends destroy a build cache shared with every other build on the host. every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
that install system packages and download Go's modules, or right after `FROM`
in a stage with neither, and every step that lints, tests or compiles comes
after it. A step that installs anything else as well, such as a
`script/bootstrap` that also installs node packages, comes after it too. A new
value reruns every `RUN` step after that line; `COPY` steps are compared by
content and need nothing. Docker keeps the layers above the line until the
base image, their own lines or the files they copy change. A stage without the
`ARG` line is served from cache on an unchanged tree, so a repository
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
not pass `--no-cache`: it reruns every step too, but it also builds the layers
above the line again and empties every cache mount, so Go's build cache and
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
survive between check builds, would start empty every time. A bare
`docker build .` is not evidence that anything ran: a sub-second build
reporting success is a cache hit, not a result. Never invalidate by pruning —
`docker builder prune` and friends destroy a build cache shared with every
other build on the host. When a check is added or changed, prove it works by
planting a defect it must catch and watching the run fail on it, then revert
the defect. A green run alone shows neither that the check ran nor that it
covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.** - **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile, hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Go image. The canonical Go repo and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`: `Dockerfile`:
```dockerfile ```dockerfile
# In every stage ARG CACHEBUST comes right after the steps that install
# system packages and download Go's modules, or right after FROM in a
# stage with neither, and every step that lints, tests or compiles
# comes after it. script/ passes a new value on every build, which
# reruns every RUN step after it. Docker keeps the layers above it
# until the base image, their own lines or go.mod and go.sum change.
# The steps that compile mount Go's build cache, and golangci-lint run
# also mounts golangci-lint's; both survive from build to build.
# Lint phase # Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD # golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
RUN golangci-lint run --config .golangci.yml ./... RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/root/.cache/golangci-lint \
golangci-lint run --config .golangci.yml ./...
# Test phase # Test phase. -race needs cgo and so a C compiler, which the Debian Go
# golang:1.x-alpine, YYYY-MM-DD # image ships and the alpine one does not. -count=1 because the mounted
# cache also holds test results, which go test would otherwise replay.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test FROM golang@sha256:... AS test
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
RUN go test -timeout 90s -race -cover ./... || \ RUN --mount=type=cache,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \ { echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; } go test -count=1 -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies # Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run # are what make BuildKit build them first, so nothing after them can
# unless lint and test passed. # run unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD # golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null RUN apk add --no-cache git
COPY --from=test /src/go.sum /dev/null
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
COPY . . COPY . .
ARG VERSION=dev # The VERSION build arg when one is given, otherwise
RUN CGO_ENABLED=0 go build -trimpath \ # `git describe --tags --always` on the .git in the build context. With
-ldflags="-s -w -X main.Version=${VERSION}" \ # .git present, a version that is still empty, dev or unknown fails the
-o /app ./cmd/app/ # build: git is missing or could not read the checkout.
ARG VERSION
RUN --mount=type=cache,target=/root/.cache/go-build \
VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage, and the last one # Runtime stage, and the last one
FROM alpine@sha256:... FROM alpine@sha256:...
ARG CACHEBUST
COPY --from=builder /app /usr/local/bin/app COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"] ENTRYPOINT ["app"]
``` ```
@@ -217,23 +289,106 @@ style conventions are in separate documents:
- Keep the runtime stage last, and if you add a stage after it, give it the - Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain same two copies. A plain `docker build .` builds the last stage's chain
and nothing else. and nothing else.
- Check builds keep these, and none of them can hand back a stale result:
- The layers above `ARG CACHEBUST`: the system packages and Go's
modules. Docker installs the packages again when the base image or the
install line changes, and downloads the modules again when `go.mod` or
`go.sum` changes. Go checks every module against `go.sum` as it
downloads it, and no later build can change a kept layer. The modules
are kept in a layer rather than a cache mount because a cache mount is
shared by every build on the host, and Go builds from whatever files
it holds without checking them again (only `go mod verify` does), so a
file changed there would go unnoticed.
- Go's build cache (`/root/.cache/go-build`, where Go keeps it for
root), mounted on the steps that compile: `golangci-lint run`, both
`go test` runs and `go build`. Go reuses a compiled package only when
all its inputs are unchanged. The cache also holds test results, which
is why both `go test` runs carry `-count=1`.
- golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned
image keeps it), mounted on `golangci-lint run`. golangci-lint reuses
a package's findings only when the package's files and those of every
package it imports, `go.mod`, the linters and their settings in
`.golangci.yml`, and the golangci-lint version are all unchanged.
- If the project uses `//go:embed` directives that reference build artifacts - If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must (e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example: create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`. `RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
- If the project requires CGO or system libraries for linting (e.g. - If the project requires CGO or system libraries for linting, install them
`vips-dev`), install them in the lint phase with `apk add`. in the lint phase. The `golangci/golangci-lint` image is Debian-based and
- `ARG VERSION=dev` is declared in the stage that compiles and supplied by has no `apk`, so install with `apt-get` under the Debian package name
`script/docker` and `script/cibuild`; no stage may call `git describe`. (`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them. The test phase,
on the Debian Go image, installs the libraries its tests need the same
way. Either install goes right after `FROM`, above `ARG CACHEBUST`, so
Docker keeps it until the base image or the install line changes:
```dockerfile
FROM golangci/golangci-lint@sha256:... AS lint
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
A checkout whose `.git` is a file (a linked worktree, or a repository
checked out as a submodule) is the exception: that file points to a git
directory outside the build context, so the build cannot read the version
and a plain `docker build .` fails; pass the version with
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
do.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step. runs `script/cibuild` on push, and checks out the repo as its only other step,
That script bootstraps, runs the gate phases, and then builds the image, so a with `persist-credentials: false`: `script/cibuild` needs no token, and
successful run means every check passed; a bare `docker build .` does not without it the checkout leaves the job's token in `.git/config` for every
later step. The checkout step also sets `fetch-depth: 0`, which fetches the
tags `git describe` needs: by default it clones shallow with no tags, and a
tagged repository's CI build would stamp a bare short commit id. The
workflow's `concurrency` block groups runs by workflow and branch
(`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`,
so a new push cancels the older run on the same branch, queued or running, and
no other: runs for replaced commits do not hold up the shared runner.
`script/cibuild` bootstraps, runs the gate phases, and then builds the image,
so a successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the image build gets a `CACHEBUST` value of its own and so runs the gate phases a
price of the rule above, and it is worth paying: the image that ships is built second time. That is the price of the rule above, and it is worth paying: the
from a run of its own gates rather than from a cache entry. image that ships is built from a run of its own gates rather than from a cache
entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the
shared runner after 20 minutes. That allows for the three Docker builds
described above (the test phase, the lint phase, then the image), each held to
the 5-minute Docker build limit below, plus the bootstrap. A separate workflow
limited to `main` by a `branches` list under `on: push` cannot be checked by
review: to try a change to it, add the feature branch to that list and push,
then remove the branch from the list again before merging. Keep any job in it
that publishes behind `if: github.ref_name == 'main'`, so the run from the
feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for - Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -286,17 +441,18 @@ style conventions are in separate documents:
``` ```
`-count=1` is required on both invocations: it defeats Go's test _result_ `-count=1` is required on both invocations: it defeats Go's test _result_
cache, so the target cannot report a pass it did not earn, and the rerun cache, so neither run can report a stored pass in place of running the
reproduces a failure instead of replaying it. It leaves the build cache tests. It leaves the build cache alone, so it costs the runtime of the suite
alone, so it costs the runtime of the suite and no recompilation. and no recompilation.
Note that this is a second, independent cache, stacked below the Docker That cache is Go's own, separate from Docker's layer cache. Go stores a
layer cache that [issue #26](https://git.eeqj.de/sneak/prompts/issues/26) passing result in its cache directory (`GOCACHE`), and when the same tests
addresses. `CHECK_EPOCH` guarantees the `RUN make test` _step_ re-executes; run again on unchanged code it prints that result, marked `(cached)`,
it does not guarantee `go test` inside that step does any work, because the without running them. The directory lasts from one run to the next on a
`GOCACHE` baked into earlier image layers survives into the re-executed developer's machine, where this target runs, and in the `test` phase of the
step. They are two separate defects requiring two separate fixes, and a fix `Dockerfile`, which mounts it to keep compiled packages between check
for one must not be recorded as covering the other. builds, so the phase's `go test` runs carry `-count=1` too. A new
`CACHEBUST` value (above) is what makes that step run on an unchanged tree.
Python example: Python example:
@@ -324,9 +480,12 @@ style conventions are in separate documents:
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`), - `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`), editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore` `node_modules/`, and the repo's own build outputs. Fetch the standard
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when `.gitignore` from
setting up a new repo. These patterns are written to `.gitignore`'s own `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo. A repo's `.gitignore` is the standard file followed by the repo's
own entries, such as its binaries; a re-vendor replaces the standard part and
keeps those entries. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified. are not a `.dockerignore` and must not be transplanted into one unmodified.
@@ -340,7 +499,7 @@ style conventions are in separate documents:
— which is more dangerous than a short file with no secret patterns at all, — which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.git`, and the repo's own host-built root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is `cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
@@ -365,21 +524,24 @@ style conventions are in separate documents:
directory, so a repo running agents in subdirectories still ships directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there. `services/api/.claude/` and must add its own anchored entry there.
- **Excluding `.git` means `git describe` cannot run inside any build stage, and - **A plain `docker build .` of a clone stamps the version that
it fails quietly there.** In a build stage there is no repository, so `git describe --tags --always` gives**, derived from the `.git` in the build
`git describe` writes nothing to stdout, `-X main.Version=` comes out empty, context as the canonical `Dockerfile` above shows. Without its failure check,
the binary reports no version at all, and the build still exits 0. Compute the a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
version on the host and thread it in as a build arg. `script/docker` and and the build would still exit 0. `script/docker` and `script/cibuild` pass
`script/cibuild` do this, byte-identically across repos: the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh ```sh
# Own line: a failing command substitution inside an argument does not # The version and the tag each get their own line: a failing command
# trip `set -e`, so the inline form degrades to an empty constant. # substitution inside an argument does not trip `set -e`, so the inline
# form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)" version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown" [ -n "$version" ] || version="unknown"
docker build --no-cache \ tag="$(script/projectname)"
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--build-arg VERSION="$version" \ --build-arg VERSION="$version" \
-t "$(script/projectname)" . -t "$tag" .
``` ```
`--always` makes an untagged repo yield an abbreviated commit hash rather `--always` makes an untagged repo yield an abbreviated commit hash rather
@@ -387,12 +549,12 @@ style conventions are in separate documents:
fallback is applied — a live check that fires on a build from an export with fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION=dev` in the stage that compiles, declared Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a checkout action clones shallow and fetches no tags, so the canonical
tag-derived version must set `fetch-depth: 0` on its checkout step. `.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the - **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build patterns.** Plant files at the root _and_ at least two directories deep, build
@@ -426,12 +588,18 @@ style conventions are in separate documents:
`test-support` depguard rule, where a repo names its own test-support packages `test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is re-vendor carries its entries forward. The canonical golangci-lint version is
v2.12.2 (released 2026-05-06), pinned as the digest of the lint phase's base v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image image
(`golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240`, (`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.12.2 built with go1.26.2 from c0d3ddc9`). That digest is the which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
only pin, since no repo installs golangci-lint on the host: bumping the directive must not name a newer Go minor version than the one golangci-lint
version means changing it and nothing else. was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by - **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
@@ -453,8 +621,18 @@ style conventions are in separate documents:
function defined and never invoked has the same exit status and the same function defined and never invoked has the same exit status and the same
empty output as one that worked. empty output as one that worked.
Node is the exception to the whole-token comparison: the canonical
`script/bootstrap` compares only its major version, because the node 22
alpine image the `Dockerfile` stages start from is not the exact pin, and
nvm cannot install a prebuilt node on alpine.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`. Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference - When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD). with the version and date (YYYY-MM-DD).
@@ -564,13 +742,17 @@ style conventions are in separate documents:
Never edit existing migrations after release. Never edit existing migrations after release.
- All repos should have an `.editorconfig` enforcing the project's indentation - All repos should have an `.editorconfig` enforcing the project's indentation
settings. settings: the standard file from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, which sets
tabs for `Makefile` and Go files, followed by the repo's own sections, such as
one for another language it uses. A re-vendor replaces the standard part and
keeps those sections.
- Avoid putting files in the repo root unless necessary. Root should contain - Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `Makefile`, `Dockerfile`, only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and `Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
language-specific config). Everything else goes in a subdirectory. Canonical and language-specific config). Everything else goes in a subdirectory.
subdirectory names: Canonical subdirectory names:
- `bin/` — executable scripts and tools - `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose - `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in body is a single call into `internal/` or `pkg/`, no project logic in
@@ -601,3 +783,7 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml` - Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore` - JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml` - Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
TODO.md
+2 -761
View File
@@ -2,7 +2,7 @@
- branch per issue from `next` - branch per issue from `next`
- do the work in Next Step - do the work in Next Step
- move Next Step to the top of Completed Steps - delete Next Step from `TODO.md`
- `TODO.md` merges with git's union merge (`.gitattributes`), which never - `TODO.md` merges with git's union merge (`.gitattributes`), which never
reports a conflict: read the merged entries after every merge or rebase reports a conflict: read the merged entries after every merge or rebase
- move the top item of Future Steps into Next Step - move the top item of Future Steps into Next Step
@@ -16,775 +16,16 @@
# Status # Status
pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work lands on pre-1.0. The `1.0.0` milestone is in progress.
`next`, and `main` receives only the milestone PR that the owner merges. `next`
is at the canonical `golangci-lint` v2.12.2 config and is green. Recent work
extracted the internal/magic, internal/allowlist, internal/httpfetcher, and
internal/signature packages. The gosec findings from the 2026-07-06 survey are
resolved. The disk cache is now size-bounded with LRU eviction
(`cache_max_bytes`), closing the unbounded disk growth DoS vector.
# Next Step # Next Step
P2: security: per-origin rate limiting P2: security: per-origin rate limiting
# Completed Steps
- 2026-10-13 the unused cache lifetime is removed (closes #69):
`CacheConfig.CacheTTL`, which pixad set to `DefaultCacheTTL` (24 hours), was
never read, so cached images never expired. The field and the constant are
removed. Cached source and transformed images stay until eviction removes them
to keep the cache within `cache_max_bytes`.
- 2026-10-13 SVG sources are refused, and `orig` is never saved as JPEG in place
of the source's own format (closes #68): the upstream fetch no longer accepts
`image/svg+xml`, which libvips would render. Both image routes answer 415 for
an upstream `Content-Type` that is not one of the source formats, SVG
included, where they answered 500. Before decoding, the image processor
refuses with `ErrUnsupportedInputFormat`, also answered 415, any input that
govips' `DetermineImageType`, which picks the libvips loader from the first
bytes, does not name as a source format. That catches an SVG whose first bytes
pass the first-bytes check for another type, as an XML comment can make them
do for AVIF; an SVG whose first bytes do not pass that check answers 500, as
before, and so does a damaged image in a source format, whose first bytes
`DetermineImageType` names. `orig` is always the source's own format, where a
source the processor could not name was saved as JPEG. `README.md` no longer
lists SVG. The SVG detection in `internal/magic` stays: the request path does
not call it, and it goes with the unused code of
https://git.eeqj.de/sneak/pixa/issues/168.
- 2026-10-12 per-client limit on the image routes (closes #236): the new setting
`image_requests_per_minute` (`PIXA_IMAGE_REQUESTS_PER_MINUTE`), default 600,
is how many `GET` and `HEAD` requests one client may send to `/v1/image/` and
`/v1/e/` together per minute, cache hits included; the next is answered 429
with `Retry-After`. It uses the login limit's `RateLimit` middleware, so
clients are told apart the same way, and runs after the CORS middleware, so a
preflight is not counted. `0` turns the limit off; a negative or non-integer
value aborts startup.
- 2026-10-08 a request past its deadline no longer waits for the database to
count it (closes #224). On a busy host, `TestService_Get_ReturnsByItsDeadline`
failed because its request, past its deadline, still waited for its miss count
to be written, a write with no deadline, which in pixad waits for the one
database connection every request shares. Each count write now keeps the
request's deadline but not its cancellation. A count not written by then is
kept in memory, where `Stats` sees it, and one goroutine of the cache writes
those counts, one write at a time, and once more at shutdown before the
database closes. The test phase of the `Dockerfile` also runs at most 4 tests
of a package at once (`go test -parallel 4`), where it ran as many as the host
has CPUs: on a busy host they then wait so long to be scheduled that a test
that times a request can fail. `-p`, how many packages are tested at once, is
unchanged, as capping it also slows compiling.
- 2026-10-08 every output format is saved with settings pixa sets on purpose
(closes #232): each format has its own govips export, as JPEG XL does, in
place of govips' generic `Export`, which sent libvips a zero for some settings
it was not given. PNG gets libvips' default compression, 6, where it had none,
and WebP libvips' default effort, 4, where it had 0. GIF was already at
libvips' default effort, 7, and JPEG output is unchanged. AVIF is saved at
effort 1, the lowest govips can set, where it had libvips' default, 4. With
one libvips thread, an 8192x8192 image of random pixels, the worst case, takes
about 51 seconds as AVIF at effort 1 and 43 as WebP at effort 4, against the
default `downstream_timeout` of 60 seconds. On an image of milder noise, which
AVIF at effort 1 saves in about 12 seconds, effort 4 takes minutes. AVIF is
also saved with 8 bits per sample from a 16-bit source, which libvips would
save with 12: a 16-bit 8192x8192 image of milder noise takes about 54 seconds
at effort 1 with 12 bits, nearly all of the default `downstream_timeout`, and
about 12 with 8.
- 2026-10-08 JPEG XL is the default output (closes #222): a `/v1/image/` URL
whose last segment is a size with no format, such as `800x600` or `orig`, is
served as JPEG XL and signed as `jxl`, so it has the signature of the same URL
ending in `.jxl`. An encrypted URL whose token holds no format is served as
JPEG XL (`encurl.DefaultFormat`). The generator page selects JPEG XL until
another format is chosen, and a form with an empty format, or none, makes a
JPEG XL URL whose name ends in `.jxl`. The image processor no longer takes an
empty format as `orig`: both routes give every request a format, and it
refuses a request with none. `auto` still ends with JPEG.
- 2026-10-08 JPEG XL as an input and output format (part of #222): a source
whose bytes start with either JPEG XL signature, the bare codestream's `FF 0A`
or the container's, is detected as `image/jxl`, which the upstream fetch
accepts; `orig` of such a source is JPEG XL. The format `jxl` works in plain
and encrypted URLs and on the generator page, served as `image/jxl` and cached
like the other formats. `auto` chooses JPEG XL first when `Accept` names
`image/jxl`. govips sends libvips a JPEG XL distance with every save, so
libvips ignores the quality: pixa turns `q` into a distance with libvips' own
formula, keeping 100 lossy, and removes the metadata from the image before
saving, as govips cannot have libvips strip it from JPEG XL. libvips 8.16 and
later still write an EXIF block of their own into JPEG XL, from the image's
size, orientation and resolution, and fixed values; the image is upright and
is given 72 dpi before the save, so the block holds nothing from the source.
An image with an ICC profile is converted to sRGB before the JPEG XL save too,
and a CMYK image with none is converted to sRGB, as libvips cannot save CMYK
as JPEG XL. libvips' default effort, 7, is kept. JPEG XL is not yet the
default output.
- 2026-10-08 pixad runs on the Alpine release it is built on (closes #229): the
runtime stage of the `Dockerfile` uses `alpine:3.22`, the release of the
`golang:1.25.4-alpine3.22` image that the test phase and the build stage use,
so all three have libvips 8.16, where the runtime image had 8.15.
- 2026-10-08 requests no longer wait behind eviction queries that read a whole
table (closes #227): the new `cache_usage` table holds the total cache usage,
kept up to date by triggers on `source_content` and `variant_content` in the
statement that adds, removes or resizes a row, and `UsageBytes` reads it. The
reconciliation pass reads the content tables 1000 rows per query, sums them
and corrects the total when it differs, unless a row changed while it summed.
Source rows get `last_accessed_at` when added, so choosing source images to
evict reads that column's index instead of sorting the whole table.
- 2026-10-08 libvips' JPEG XL support is installed and required (part of #222):
`script/bootstrap --cgo` installs `vips-jxl` when its package manager is apk,
as Alpine's `vips` package lacks the support, and the runtime stage of the
`Dockerfile` installs it too. `imgcache.NewService` fails, naming the fix,
when `imageprocessor.CheckJPEGXLSupport` finds that libvips cannot load and
save JPEG XL, so pixad does not start without it. A test saves an image as
JPEG XL with govips and loads it back. JPEG XL is not yet a format pixa
serves.
- 2026-10-08 SQLite writes no longer fail with "database is locked" under load
(closes #223): `internal/database` opens the database with one connection, so
pixa's own reads and writes run on it one at a time instead of competing for
SQLite's lock, where a write that kept losing could wait past the five-second
busy timeout and be lost. The busy timeout stays, for another program writing
to the same file. No code in pixa keeps rows or a transaction open while it
runs another query, which with one connection would wait forever.
- 2026-10-05 the format `auto` (closes #88): a format in the `/v1/image/` path,
an encrypted URL's token and the generator page's format choice, chosen for
each request from `Accept` once the signature or token is checked: AVIF when
the header names `image/avif`, else WebP when it names `image/webp`, else JPEG
when the first of `image/jpeg`, `image/*` and `*/*` that it names allows it,
or when it names nothing; `q=0` refuses a format. AVIF and WebP must be named,
as clients that cannot show them send the wildcards too. A header that allows
none of the three answers 406, one that does not parse 400. The signature and
the token cover `auto` itself; the cache key and `ETag` use the format chosen.
Answers from the point the format is chosen carry `Vary: Accept`, next to the
CORS `Vary: Origin`; fixed-format answers do not.
- 2026-10-05 the markdown is formatted with prettier (closes #100): `script/fmt`
and `script/fmt-check` run prettier 3.8.1, pinned in `package.json` and
`yarn.lock`, on `**/*.md` after `gofmt`, with four-space tabs and
`proseWrap: always` as `.prettierrc` says; `.prettierignore` keeps it off
`REPO_POLICIES.md`, the copy from `sneak/prompts`, and `vendor/`. Plain
`script/bootstrap` installs Node and Yarn as the one in `sneak/prompts` does
and then prettier; `script/bootstrap --cgo` does not, as the `Dockerfile`
stages that run it format nothing. The HTML templates stay unformatted:
prettier cannot parse a Go template action inside a tag. The markdown was
reflowed in a commit of its own.
- 2026-10-05 lint and tests run as the `lint` and `test` phases of the
`Dockerfile`, built with `--no-cache` (closes #202): `script/check`,
`script/cibuild`, `script/docker`, `script/lint`, `script/test`,
`script/setup` and `script/install-precommit` are now the copies from
`sneak/prompts` `main`, unchanged. The `lint` phase runs golangci-lint from
the image `REPO_POLICIES.md` names, with `libvips-dev` from `apt-get`; the
`test` phase runs the tests with a 90-second timeout; the build stage depends
on both. `Dockerfile.lint` and the `CHECK_EPOCH` build argument are gone, the
formatting check runs on the host, and `make docker-versioned` and
`make docker-test` call the scripts. `script/bootstrap`, `script/fmt`,
`script/fmt-check`, `script/precommit` and `script/projectname` stay pixa's
own. `script/bootstrap` installs git, make and Go, refreshing apt's package
lists before its first apt install; with `--cgo`, which only the `test` phase
and the build stage pass, it also installs the C compiler and the libvips and
libheif libraries. The stage that compiles still takes the version from
`git describe` when no `VERSION` is given, per
https://git.eeqj.de/sneak/pixa/issues/166, so the copied scripts' comment that
`.dockerignore` leaves out `.git` does not hold for pixa.
- 2026-10-04 load test (closes #81): `script/loadtest [duration [clients]]`
(`make loadtest`, defaults `10s` and `4`), a benchmark that `script/check`
does not run, measures three scenarios, each against a new pixad container and
a new upstream host, `cmd/loadtest-origin`: `hit` (one cached image), `miss`
(a new source image every request) and `herd` (each new source image asked for
by all clients at once). For each it prints vegeta's report (requests per
second, latency percentiles, status codes), pixad's peak resident memory and
the requests that reached the origin. `README.md` says how to run it and read
it, and keeps 1-5k r/s as a target not yet measured. First measurement, with
the defaults on a shared 48-CPU machine with other work running: a baseline
for later changes, not a test of the target. `hit` 1413 r/s, p50 0.7 ms, p95
8.7 ms, p99 44 ms, peak 53 MiB (4 clients that each wait for their answer, so
not pixad's limit); `miss` 70 r/s, p50 52 ms, p95 91 ms, p99 122 ms, peak 100
MiB, one fetch per request; `herd` 74 r/s, p50 52 ms, p95 69 ms, p99 111 ms,
peak 60 MiB, 188 fetches for 749 requests.
- 2026-10-04 the CI checkout fetches the tags (closes #208): the checkout step
in `.gitea/workflows/check.yml` sets `fetch-depth: 0`, as `REPO_POLICIES.md`
asks of a repo that takes its version from the tags, so a CI build of a tagged
commit stamps the tag from `git describe` instead of a bare commit.
- 2026-10-04 `config.yml` stays out of git and the Docker build context (closes
#212): `.gitignore` now ignores `config.yml`, the config file Getting Started
creates with the signing key, and `.dockerignore` leaves it out in every
directory and in any letter case, as it already did `config.yaml` and
`config.dev.yml`.
- 2026-10-04 local config files stay out of the Docker build context (closes
#211): `.dockerignore` now leaves out `config.yaml` and `config.dev.yml` in
every directory and in any letter case, the local config files `.gitignore`
keeps out of git because they can hold the signing key.
`configs/config.example.yml` is still sent. `config.yml`, which Getting
Started creates, is in neither file:
https://git.eeqj.de/sneak/pixa/issues/212.
- 2026-10-04 `cmd/pixad/main.go` is one call into `internal/` (closes #206):
what it did (the command line and its `--config` flag, setting
`PIXA_CONFIG_PATH`, ignoring `SIGPIPE`, starting the fx app) is now `Run` in
`internal/app`, unchanged, and `main` calls it with `Version`, which the build
still sets through `-X main.Version`. That code had no tests to move.
- 2026-10-04 `.gitignore` ignores `.claude/` (closes #204): the entry and its
comment are copied from the canonical `.gitignore` in `sneak/prompts`,
unanchored so it matches at every depth. `.dockerignore` already has
`.claude`.
- 2026-10-04 `.dockerignore` keeps secrets out at every depth (closes #205): the
file is now the standard one from `sneak/prompts`, whose patterns match in
every directory and, for environment files and private keys, in any letter
case, so a nested `.env` or `server.key` no longer reaches the build context.
pixa still sends `.git` without `.git/config` in place of the standard file's
`.git` line, and still leaves out `.gitignore`, `/bin` and `/data`.
- 2026-10-04 `REPO_POLICIES.md` matches the canonical copy again (closes #196):
it is replaced, unchanged, by `prompts/REPO_POLICIES.md` from `sneak/prompts`
`main`. The rules it adds that pixa's tree breaks are filed:
https://git.eeqj.de/sneak/pixa/issues/202 (lint and tests as `Dockerfile`
phases built with `--no-cache`), https://git.eeqj.de/sneak/pixa/issues/203
(the workflow's `script/docker-smoke` step),
https://git.eeqj.de/sneak/pixa/issues/204 (`.claude/` in `.gitignore`),
https://git.eeqj.de/sneak/pixa/issues/205 (`.dockerignore` patterns at every
depth), https://git.eeqj.de/sneak/pixa/issues/206 (a thin `cmd/pixad/main.go`)
and https://git.eeqj.de/sneak/pixa/issues/208 (`fetch-depth: 0` on the CI
checkout, so the build sees the tags). Its rule that no build stage runs
`git describe` is not followed: pixa takes the version from the `.git` in the
build context, per https://git.eeqj.de/sneak/pixa/issues/166, as the copy on
`sneak/prompts` `next` already says.
- 2026-10-04 an integration test of the image proxy flow (closes #80):
`TestImageProxyFlow` in `internal/server` starts the database, handlers and
middleware from the constructors `pixad` uses, with a fresh state directory,
and replaces only the upstream origin with a local test server. For a resize
with a change to JPEG and for `orig`, the first request goes through the
router, the real fetcher, libvips, the disk cache and SQLite and answers 200
with the right content type and size and `X-Pixa-Cache: MISS`; the second
answers `HIT` with the same image and the upstream has had one request; the
source and the converted image are then in `cache/sources` and
`cache/variants`, with their rows in `source_content`, `source_metadata` and
`variant_content`. Two optional fields make this possible, which `pixad` does
not set and the config file and environment cannot:
`httpfetcher.Config.DialContext` connects in place of the dialer that refuses
internal addresses, the URL and redirect checks still running, and
`handlers.Params.Fetcher` replaces the fetcher the handlers build.
- 2026-10-04 a URL made on the generator page with a `ttl` is tested to expire
(closes #199): a new test in `internal/handlers` makes a URL on the generator
page with a `ttl` of one second, checks that `/v1/e/` serves it at once, waits
two seconds and checks that it then answers 410. The test waits for real, as
pixa reads the clock directly when it makes and checks a URL; it waits two
seconds because the time a URL expires is kept in whole seconds. Test only.
- 2026-10-04 referer blocklist (closes #90): `referer_blocklist`
(`PIXA_REFERER_BLOCKLIST`) lists hosts, written and matched as for
`allowlist_hosts` with the same matcher; an entry of either list that is
neither a host name (letters, digits, hyphens, underscores and dots, with at
most one leading dot) nor an IP address, such as one with a port or a `*.`
wildcard, aborts startup naming the setting and the entry. Both image routes
refuse a request whose `Referer` names a listed host with 403 and a JSON error
before the signature, the cache and the upstream fetch, so it fetches nothing
and is refused whether or not the image is cached. A request with no
`Referer`, or one that does not parse as a URL with a host, is served, so the
list is easily got around; `README.md` and `configs/config.example.yml` say
so. It does not apply to the login and generator pages.
- 2026-10-04 fewer files in the repository root (closes #97):
`config.example.yml` moved unchanged to `configs/config.example.yml`, and
`README.md`, the comments in `internal/config/config.go` and the startup error
for the placeholder signing key name the new path; `scripts/manual-test.sh`
and its directory are deleted, as the handler tests in `internal/handlers`
cover every check it made except two: fetching a real image from the internet,
and a URL made on the generator page with a `ttl` answering 410 once the `ttl`
has passed (https://git.eeqj.de/sneak/pixa/issues/199); `CONVENTIONS.md` is
deleted, as `REPO_POLICIES.md` links the canonical Go HTTP server conventions.
- 2026-10-04 SQLite writes no longer fail with "database is locked" (closes
#198): pixa adds `_pragma=busy_timeout(5000)` to every `db_url`, so a write
that finds another in progress on another connection waits up to five seconds
for it, and the default `db_url` turns on WAL mode with
`_pragma=journal_mode(WAL)`. The old default's `_journal_mode=WAL` is not a
parameter the driver reads, so the database was never in WAL mode.
- 2026-10-04 `TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup` only
passes through a periodic pass (closes #189): it slept for three eviction
intervals before writing its file, and a startup pass still running then could
adopt the file itself. It now holds the test database's only connection until
the startup pass waits for it after walking the empty variant directory,
writes the file and lets the connection go, as
`TestEvictionRunsOnPeriodicSchedule` does, so only a periodic reconciliation
pass can adopt the file. Test only.
- 2026-10-04 logging in, logging out, the URL generator and `/v1/e/` have
handler tests (closes #77): new tests in `internal/handlers`, with no network,
check that `GET /` without a login session shows the login form; a wrong key
shows it again with an error and sets no session cookie; the right key answers
303 to `/` with a session cookie marked `Secure`, `HttpOnly` and
`SameSite=Strict`, with which `GET /` shows the generator page; `GET /logout`
answers 303 to `/` with an empty session cookie sent with `Max-Age=0`;
`POST /generate` without a login session answers 303 to `/`; `/v1/e/` serves
the image for a valid token, answers 410 for an expired one and 400 for one
with a character changed, cut short or made with another signing key; and a
URL made on the generator page is served by `/v1/e/`. No code changes.
- 2026-10-04 `TODO.md` merges with git's union merge (closes #190): a root
`.gitattributes`, copied from `sneak/prompts`, marks it `merge=union`, so two
branches that each add an entry at the top of Completed Steps merge without a
conflict and keep both entries. Git now never reports a conflict in `TODO.md`:
a real one keeps both versions of the lines, and two entries that share an
identical line can end up one inside the other, which a rebase can do to an
entry already on `next`. The Workflow above says to read the merged entries
after every merge or rebase.
- 2026-10-04 the default `cache_max_bytes` no longer shrinks as the cache fills
(closes #184): for an omitted key, the cache works out the limit when it
opens, after the database is open, as 75% of the sum of the free space on the
filesystem containing `<state_dir>/cache/` and what the cache already holds by
its own size accounting, at least 500 MiB, so a cache filled to its limit
keeps that limit across a restart. The computation and its tests moved from
`internal/config` to `internal/imgcache`; the config only records whether the
key was set.
- 2026-10-04 `TestEvictionRunsOnPeriodicSchedule` no longer races the evictor
(closes #183): it wrote each variant file and then inserted its accounting row
by hand, and a reconciliation pass between the two adopted the file first, so
the insert failed. It now writes the files only, while holding the test
database's only connection so the evictor's startup pass waits after walking
the empty variant directory; a periodic reconciliation pass then adopts the
files and the eviction pass after it evicts them. No other test in
`internal/imgcache` inserts a row by hand after starting the evictor. Test
only.
- 2026-10-04 a config file pixa cannot read aborts startup (closes #176): of the
places pixa looks for its config file on its own, only one where the file does
not exist is passed over; any other error, such as a directory on the path
that pixa may not enter, aborts startup naming the file, as a file that does
not parse already did.
- 2026-10-04 `.golangci.yml` re-vendored from the canonical copy (closes #57):
the deprecated `gomodguard` is switched off, so lint runs print no deprecation
warning; its successor `gomodguard_v2` runs with the shared module block list,
and `depguard` keeps `net/http/httptest` out of files that are not tests. The
tree needed no code changes.
- 2026-10-04 the Content-Security-Policy allows no inline script or style
(closes #125): `script-src` and `style-src` are `'self'` only. The generator
page's two inline `onclick` handlers moved into
`internal/static/generator.js`, attached with `addEventListener`; the bundled
Tailwind script, which built styles in the browser, is replaced by a small
hand-written `internal/static/style.css` with only the rules the login and
generator pages use, the templates carrying a few plain class names in place
of Tailwind's. No build step. The pages keep their layout, not every pixel of
it.
- 2026-10-04 deployment guide and example Caddy config (closes #89):
"Deployment" in `README.md` says what the reverse proxy in front of pixa must
do (terminate TLS; pass `Host`, `Origin` and `Referer` on unchanged; set
`X-Forwarded-For`, with `trusted_proxies` to match; wait at least
`downstream_timeout`; optionally refuse `/metrics`) and what pixa does itself,
that the state directory needs a persistent volume and what `cache_max_bytes`
counts, the health check for a load balancer, what a stop does and its exit
codes, and what running outside Docker needs; `configs/Caddyfile` is the
example, checked with `caddy validate`.
- 2026-10-04 the metrics basic auth, CORS preflight, request logging and metrics
recording have tests (closes #79): `MetricsAuth` on its own answers 401 with a
challenge without credentials or with a wrong username or password and lets
the configured ones through; a preflight request gets `*` for any origin when
`access_control_allow_origin` is `*` and no `Access-Control-Allow-Origin` from
another origin than the configured one; a `POST /` carrying the signing key
leaves no trace of it in the request log line, and the login handler's own log
lines leave out the submitted key; the metrics middleware on its own records a
request it served, and the router records nothing while no metrics username is
set. Not tested: that the router puts the basic auth in front of `/metrics`
and records requests when a metrics username is set. Only one test per package
can set up `/metrics`, and in `internal/server` that is
`TestMaintenanceModeKeepsOtherRoutes`, which needs the owner's approval to
change; #180 holds it. Tests only; the basic auth library already compares the
password in constant time.
- 2026-10-04 the image route's signature check and error answers are tested
(closes #76): new tests in `internal/handlers`, with no network, check the
status and JSON error body for a missing, wrong, unpadded, upper-case or
expired signature on a host not on the allowlist, or a valid one sent for its
parent domain, a sibling host, a subdomain or the host with another domain
appended (401), an unparseable path (400), `localhost` as the upstream host
(403) and an upstream error (502); that an allowlisted host is served without
a signature, another host only with a valid one; and the answers of
`/robots.txt` and the health check. No code changes.
- 2026-10-04 request IDs returned and passed on, and `/v1/e/` revalidates
(closes #84): pixa's own `RequestID` middleware, in place of chi's, gives each
request an ID, its own `X-Request-ID` when that is at most 64 letters, digits,
`-`, `_` or `.` and a random one otherwise, stores it where chi's did and
sends it back as `X-Request-ID` on every response; the upstream fetch sends
that ID, and the "upstream fetched", "image converted" and "image served" log
lines carry it as `request_id`, a fetch shared by several requests carrying
the first request's; `/v1/e/` sets `ETag`, answers a matching `If-None-Match`
with 304 and is routed for `HEAD`, the `ETag` and 304 code being
`notModified`, which `/v1/image/` calls too; its token checks moved unchanged
into `parseImageEncRequest` to keep `HandleImageEnc` within the line limit; no
`Vary` is added, as no response depends on a request header except the image
routes' CORS headers, for which `go-chi/cors` already sends `Vary: Origin`;
`Vary: Accept` is left to #88.
- 2026-10-04 routes, encrypted URLs and config file documented (closes #75):
"Routes" in `README.md` lists every route with its method, purpose, what it
needs and the status codes it answers with, and says `q` and `fit` are part of
what is cached; "Encrypted URLs" covers logging in, making one on the
generator page, how long it lasts and the 410 once it has expired;
"Configuration" gives the order in which pixa looks for its config file;
`config.example.yml` lists `db_url` and `env` and gives every key's default;
`scripts/manual-test.sh` is left to #97.
- 2026-10-04 shutdown stops cache eviction in progress (closes #102):
`StartEviction` runs the eviction goroutine with its own context, which
`StopEviction` cancels, so a pass in progress stops at its next database call,
file, row or eviction candidate instead of running to completion, and no pass
starts after it, so a stop logs at most one warning; `StopEviction` takes a
context and, when that context ends before the goroutine exits, stops waiting
and returns its error; the handlers' stop hook passes fx's stop context, so an
eviction still running when fx's stop deadline ends fails the stop and makes
the exit code 1.
- 2026-10-04 dead code in `internal/imgcache` is gone (closes #73): `Purge`,
which only returned an error and which nothing called, is no longer part of
the `ImageCache` interface or `Service`; the `SignatureValidator`, `Allowlist`
and `Storage` interfaces, which nothing implemented or used, are deleted.
Nothing else changes.
- 2026-10-04 upstream host semaphores and variant `.meta` files no longer
outlive their use (closes #87): the fetcher counts the fetches holding or
waiting for a slot of each upstream host's semaphore and removes the host's
semaphore once none is left, so fetches from many hosts no longer leave one
semaphore each until restart; `VariantStorage.Delete` removes the variant's
`.meta` file along with it, a missing `.meta` file not being an error, and
`DeleteWithMeta`, which eviction called for that, is gone.
- 2026-10-04 `README.md` matches the code (closes #74): "Storage" names the
cache directories pixa uses (`cache/sources`, `cache/metadata`,
`cache/variants`) and how files are named in each, and the comments in
`001_schema.sql` name the same paths; the routes and the signature section
list the same output formats, `jpg` and `original` included; the TLS sentence
names `allow_http` as its exception; "Metrics" says only generic HTTP and Go
runtime metrics exist, measured and served only when the metrics username and
password are set.
- 2026-10-03 shutdown sets the exit code and waits for image processing (closes
#86): fx alone handles SIGINT and SIGTERM, and the server's own signal handler
is gone; fx's `Run` in `cmd/pixad` exits with the shutdown's code: 0 for a
signal, 1 when the HTTP server cannot listen or the app fails to start or to
stop; the server's stop hook, which fx waits for, stops the HTTP server, waits
for the images still being processed, both within 5 seconds, then flushes
Sentry; images still being processed after that are logged with their count
and make the exit code 1; a Sentry DSN that cannot be used fails startup, so
the stop hooks of what had already started run, instead of exiting the process
from a goroutine; the eviction loop is left to #102.
- 2026-10-03 every `script/cibuild` and `script/docker` run executes the checks
(closes #101): the `Dockerfile` declares `CHECK_EPOCH` above `make fmt-check`
and `make lint` in the lint stage and above `make test` in the build stage,
and each of those steps names it in its command; both scripts pass a new value
on every run, so Docker runs the checks instead of reusing cached results,
while the `script/bootstrap` steps stay cached; a plain `docker build .` still
works, leaves it empty, and reuses the check steps only for an identical build
context; the `script/cibuild` comment and `README.md` no longer say that any
successful build implies a green repo.
- 2026-09-29 share concurrent misses (closes #65): requests that miss the same
variant at once (the same cache key, so quality and fit included) share one
upstream fetch or cached source read and one transcode through
`golang.org/x/sync/singleflight`; the first request's processing ignores its
cancellation but keeps its deadline, and the others wait for its image or
error holding no upstream connection or processing slot, and stop waiting when
their own context ends; the request doing the processing waits for it even
then, up to its deadline; a request whose context has already ended starts
nothing; each request counts one miss, and the processing counts its fetch and
transcode once; a panic while processing is reported to Sentry when
`sentry_dsn` is set and becomes an error for every waiting request instead of
stopping pixad; documented in `README.md`.
- 2026-09-29 only the image routes send CORS headers (closes #98): the CORS
middleware, with the `access_control_allow_origin` origin, moved from the
router root onto a `/v1` subrouter holding `/v1/image/` and `/v1/e/`, where it
still answers a preflight `OPTIONS` request; the login and URL generator
pages, `/metrics` and the other routes send no `Access-Control-Allow-Origin`;
documented in `README.md` and `config.example.yml`.
- 2026-10-02 a plain `docker build .` stamps the tag or short commit, not `dev`
(closes #166): `.dockerignore` lets `.git` into the build context, without
`.git/config`; with no `VERSION` build argument the `Dockerfile` takes the
version from `git describe --tags --always`, and fails the build if the
context carries `.git` and no version comes out; `ARG VERSION` has no default;
pixad logs its version, with its name and architecture, as its first log line
at startup.
- 2026-09-29 the container makes `/var/lib/pixa` usable by itself (closes #159):
`deploy/docker-entrypoint.sh` creates the directory if it is missing, gives
the directory and everything in it to `pixad` when the directory or one of its
top-level entries belongs to another user or group, sets its mode to `750`,
then runs the server as `pixad`; data left by an earlier run under another uid
is taken over this way; "Running under upaas" in `README.md` no longer tells
the operator to create or chown the host directory.
- 2026-09-29 variant content types kept in memory (closes #70):
`Cache.metaCache` holds the content types of up to 10,000 variants in an LRU
(`github.com/hashicorp/golang-lru/v2`), filled by `StoreVariant` and by
`GetVariant` after it reads a `.meta` file, where a type `StoreVariant` added
meanwhile is kept over the one read, and never with the
`application/octet-stream` served for a variant without one; for a variant it
holds, `GetVariant` skips the `.meta` read, still opening the variant file and
taking the size from it; eviction removes the entry before deleting the files,
and `GetVariant` removes it when the file will not open; the cap is a
constant, not a setting; the unused `variantMeta` type is gone; `README.md`
describes it.
- 2026-09-29 maintenance mode refuses image requests (closes #71): while
`maintenance_mode` is on, `/v1/image/` and `/v1/e/` answer 503 with a
`Retry-After` header and the JSON error body, from one middleware in
`internal/server/routes.go`; the health check stays 200 and reports
`maintenance_mode`, as the image's Docker `HEALTHCHECK` requests it and upaas
marks a deploy failed when its container is unhealthy; the login and URL
generator pages and `/metrics` keep working; documented in `README.md` and
`config.example.yml`.
- 2026-09-29 bound concurrent image processing and upstream fetches (closes
#64): `max_concurrent_processing` (default the number of CPUs pixa can use)
limits the images decoded and encoded at once, and `upstream_connections`
(default 64) the connections to all upstream hosts together, on top of
`upstream_connections_per_host`; a fetch holds its connection until its image
has been processed, and a request whose source is cached reads it only once it
has a processing slot; a request that finds either limit reached waits up to
10 seconds for a free one, then gets 503 `server busy, try again later`;
libvips runs one worker thread per image with its operation cache off;
documented in `README.md` and `config.example.yml`.
- 2026-09-29 Dockerfiles install through `script/bootstrap` (closes #95): the
`Dockerfile` lint and build stages and `Dockerfile.lint` copy `script/`,
`go.mod` and `go.sum`, then run `script/bootstrap` in place of their own
`apk add` lines, so the build dependencies are listed in one place;
`script/bootstrap` now also installs a C compiler when `gcc` is missing; the
build uses `-trimpath` and `-s -w` and keeps `CGO_ENABLED=1` for govips;
`ARG VERSION` sits just above the build, so a new version reruns neither
`script/bootstrap` nor the tests.
- 2026-09-29 migrations at the path `REPO_POLICIES.md` sets (closes #96): the
migration files moved, contents unchanged, from `internal/database/schema/` to
`internal/db/migrations/` as `000_migration.sql` and `001_schema.sql`; the
`internal/db/migrations` package embeds them and `internal/database` reads
them through its `FS()`; the `internal/database` package itself stays; the
version still comes from the filename prefix, so a database that has recorded
versions 0 and 1 runs neither again.
- 2026-09-29 `trusted_proxies` advice and signature padding in `README.md`
(closes #150): the login-limit paragraph, the `trusted_proxies` entry and
`config.example.yml` say to set `trusted_proxies` to the address pixa sees for
requests that come through the proxy, which the request log shows as
`remoteIP` while it is not trusted; for a proxy on the Docker host that
connects over `127.0.0.1` that is the Docker network's gateway, not the
proxy's own address; the signature section says `sig` is base64url with the
`=` padding kept, and gives the example's `sig` for a stated signing key.
- 2026-09-29 fixed uid and gid for `pixad` (closes #151): the image creates the
`pixad` group with gid 65532 and the `pixad` user with uid 65532, instead of
the first free uid 1000, so a bind-mounted `/var/lib/pixa` given to `pixad` is
not owned on the host by a person's login account; the first-run step of
"Running under upaas" in `README.md` names the uid and gid.
- 2026-09-29 `max-age` never outlives an expiring URL (closes #63): both image
routes build `Cache-Control` from the request's `Expires`, which an encrypted
URL's expiry now fills too; `max-age` is one year, or the whole seconds left
until the `exp` of a `/v1/image/` URL or the expiry of an encrypted URL when
that is sooner, never negative; an allowlisted host's URL that has an `exp`
follows it too; `immutable` stays, as freshness now ends at the expiry;
documented in `README.md`.
- 2026-09-28 add the four settings `README.md` documented but pixa did not have,
which aborted startup as unknown keys (closes #61):
`access_control_allow_origin` (default `*`, the CORS origin),
`upstream_fetch_timeout` (default `30s`), `upstream_max_response_size`
(default 50 MiB) and `downstream_timeout` (default `60s`, both the server's
write timeout and the per-request timeout); each has a `PIXA_` variable;
durations are positive Go duration strings, the size a whole number of bytes
up to 1 GiB, the origin `*` or one `http` or `https` origin as `README.md`
describes it; an invalid value aborts startup naming the key and the value;
documented in `config.example.yml` and `README.md`.
- 2026-09-28 cache stats report real numbers (closes #56): `Cache.Stats` counts
the cached source images and processed variants (`source_content` plus
`variant_content`) and takes their size from `Cache.UsageBytes`, instead of
reading `request_cache` and `output_content`, which nothing writes; those two
tables are left in the schema; a disabled disk cache reports no items and no
size. A hit is counted even when the request context has ended. A miss is
counted after it is served or fails, also when the request context has ended
by then, with the bytes it read from upstream, so `upstream_fetch_count` and
`upstream_fetch_bytes` move, including for an upstream body that fails partway
or a fetched source that then fails the magic byte check; `transform_count`
counts each image the image processor transcodes.
- 2026-09-28 strip metadata from processed images (closes #82): every output is
exported with govips' `StripMetadata`, so it carries no EXIF, XMP, IPTC or ICC
profile; the image is first turned upright with `AutoRotate` (before sizes are
worked out) and, when it has an ICC profile, converted to sRGB; the `orig`
format is re-encoded and stripped like any other, as pixa never serves the
source bytes; there is no setting to keep metadata; documented in `README.md`.
- 2026-09-28 rate limit the login form (closes #66): `POST /` is limited to 5
attempts per minute per client address, and an attempt over the limit is
refused with 429 and a `Retry-After` header; the address is the one
`internal/clientip` resolves through `trusted_proxies`, an IPv6 client is
counted by its /64, and an IPv4-mapped address as the IPv4 address it carries;
the limit is a `RateLimit` middleware in `internal/middleware` on
`github.com/go-chi/httprate`, which the image routes can reuse; the library
keeps counts for the current and the previous minute only; documented in
`README.md`.
- 2026-09-28 refuse an unparseable `exp` on `/v1/image/` and log swallowed cache
errors (closes #72): an `exp` in the URL that is not a whole number, an empty
`exp=` included, is a 400 naming `exp` and the value, instead of being ignored
and answered with 401 as if the URL had no `exp`; only an `exp` missing from
the URL is unchanged; `README.md` says so where it documents `exp`. A failed
variant `.meta` write, source metadata JSON write, `Stats` count query, stats
counter update, negative cache write or expired negative cache delete is now
logged at `warn` with the path or key and the error, and stays non-fatal.
- 2026-09-28 refuse an empty `fit` on `/v1/image/` (closes #139): a `fit` in the
URL with an empty value (`fit=`) is a 400 naming `fit`, instead of being
served as `cover` and verified against a signature made for `cover`; only a
`fit` missing from the URL is still `cover`; any other value still goes
through the existing fit-mode check; `README.md` says so where it documents
`fit`.
- 2026-09-28 refuse an invalid `q` on `/v1/image/` (closes #134): a `q` that is
not a whole number from 1 to 100, an empty `q` included, is a 400 naming `q`
and the value, instead of being served at the default 85; the route reads `q`
with the generator's quality check (`parseFormInt` with `minQuality` and
`maxQuality`); only a `q` missing from the URL is still 85; a query string
that cannot be decoded, such as `q=80%`, is a 400 showing it; any query
parameter given more than once (`q`, `fit`, `sig`, `exp` alike) is a 400
naming it, so none is read from its first value only; `README.md` states the
range and both query-string rules.
- 2026-09-28 unknown `PIXA_` environment variables abort startup (closes #133):
a variable whose name starts with `PIXA_` but is neither a setting's variable
nor `PIXA_CONFIG_PATH` aborts startup naming it, as an unknown config key
does, and `PIXA_PORT` is named with a pointer to `PORT`; the check runs after
the config file loads, so the variables the file's `env:` section sets are
checked too; documented in `README.md`.
- 2026-09-28 start on a fresh upaas volume (closes #129): the image starts as
root only to give `/var/lib/pixa` to `pixad` when `pixad` does not own it
(`deploy/docker-entrypoint.sh`), then runs the server as `pixad` through
`su-exec`, so a root-owned host directory bind-mounted there no longer stops
the container at startup; `README.md` gains a "Running under upaas" section.
- 2026-09-28 run all linting in Docker via `Dockerfile.lint` + `script/lint`
(closes #104): `make lint` calls `script/lint`, the only way the linter is
run; inside a container (both Dockerfiles set `container=docker`) it runs
`golangci-lint`, anywhere else it builds the hash-pinned `Dockerfile.lint`,
whose last step runs `script/lint` again; the `Dockerfile` lint stage runs
`make lint`; no host or nix-shell `golangci-lint` path remains
(`script/bootstrap` installs no linter); a per-run `CACHEBUST` build-arg keeps
the lint step from being served from cache, and a tmpfs mount on that step
keeps Go's and golangci-lint's caches out of its layer, so a run leaves no
large build cache behind; `golangci-lint config verify` stays out, as it
fetches its schema over an unpinned live HTTPS call
- 2026-09-28 every setting as an environment variable (closes #128, also covers
#99): each config key can be set by `PIXA_` plus the key in upper case (`.`
written as `_`), and the port by `PORT`; a variable present in the
environment, even empty, wins over the config file, which wins over the
default; the typed getters read the variable first, so every existing check
applies to it and a bad value aborts startup naming the variable; lists are
comma-separated, and an empty variable (or `""` in the file) is an empty list;
the Docker image no longer bakes in `config.docker.yml` or passes `--config`,
and its `HEALTHCHECK` probes `PORT` (default `8080`); the config file is
looked for under `/etc/pixa` and `~/.config/pixa` instead of the daemon name
`pixad`; documented in `README.md` and `config.example.yml`.
- 2026-09-28 quality and fit in the URL signature (closes #60): the signed data
is now `host:path:query:width:height:format:expiration:quality:fit`, using
`85` and `cover` when the URL has no `q` or `fit`, so one signed URL can no
longer be replayed across other quality and fit values to create unauthorized
cache entries and transcodes; the known-answer vectors in
`internal/signature/golden_test.go` and the README signature specification
describe the new format.
- 2026-09-28 Docker image healthcheck (closes #111): a `HEALTHCHECK` in the
runtime stage probing `/.well-known/healthcheck.json` with busybox `wget`;
`script/docker-smoke` (`make docker-smoke`) builds the image, starts it with a
throwaway `PIXA_SIGNING_KEY`, and passes only once Docker reports it healthy
within 30 seconds, removing the container on exit; the Gitea workflow runs it
after `script/cibuild`.
- 2026-09-21 trusted-proxy client IP resolution (closes #94): a
`trusted_proxies` config key taking a list of CIDRs, parsed by the same
`net/netip` list parser as `blocked_networks` (an invalid entry aborts startup
naming the key and value; an omitted key defaults to the RFC 1918 private
ranges, an explicitly empty list trusts no one, and an explicit list replaces
the default); a new `internal/clientip` package resolves the client address by
honoring `X-Forwarded-For` only when the direct peer is a trusted proxy,
walking the chain right-to-left to the rightmost non-proxy entry, so a client
connecting directly cannot spoof its address; the resolved address is stored
in the request context by a new middleware and used by the request-logging
middleware and the login-attempt logs in place of the raw peer address;
documented in `README.md` and `config.example.yml`.
- 2026-09-21 blocked networks configuration extending SSRF protection: a
`blocked_networks` config key taking a list of CIDRs (parsed with `net/netip`,
an invalid entry aborts startup naming the key and value), added to the
built-in blocklist rather than replacing it; the built-in ranges extended to
CGNAT `100.64.0.0/10`, IETF protocol assignments `192.0.0.0/24`, benchmark
`198.18.0.0/15`, and NAT64 `64:ff9b::/96` (IPv4-mapped forms covered);
enforcement stays in the dial-time re-resolution so the DNS-rebinding window
remains closed; documented in `README.md` and `config.example.yml`.
- 2026-09-21 validate dimensions and fit mode on the encrypted-URL route and the
token generator (closes #62): `imgcache.ValidateDimension` alone holds the
`MaxDimension` bound and is used by the path parser, by the new
`ValidateImageRequest` (which also applies `ValidateFitMode`) and by the
generator; both the `/v1/image/` and `/v1/e/` routes call
`ValidateImageRequest`, so an over-limit size or an unknown fit mode is a 400
rather than an out-of-memory or a 500 from the processor; the URL generator
answers 400 naming the field for a `width` or `height` that is not a number or
fails the shared check, a `quality` that is not a number from 1 to 100, a
`ttl` that is not a number from 0 to the largest number of seconds the expiry
calculation can hold, or an unknown `fit`; an empty `quality` is 85 and an
empty `ttl` never expires; the form's width and height inputs stop at 8192
- 2026-09-21 http.Server hardening (closes #92): added `HTTPReadHeaderTimeout`
(10s, bounds the slowloris header dribble) and `HTTPIdleTimeout` (120s, bounds
keep-alive reuse) alongside the existing timeouts and wired them onto the
server; added a `LimitBody` middleware capping the two form POST bodies
(`POST /`, `POST /generate`) at `MaxFormBytes` (1 MiB) and returning 413,
applied ahead of the CSRF middleware so an oversized body is refused as 413
rather than being read as a missing CSRF token (403); left `WriteTimeout` at
60s unchanged
- 2026-08-07 update golangci-lint to v2.12.2 with the canonical `.golangci.yml`
(v2 schema, `default: all` minus six disabled linters, `lll` 88, tests
included): bumped the pinned `golangci/golangci-lint:v2.12.2-alpine` image in
`Dockerfile` and the release-archive sha256 pins in `script/bootstrap`; fixed
the findings the stricter config surfaced (notably `paralleltest`, `wsl_v5`,
`goconst`, `lll`, `noinlineerr`, `err113`, `errcheck`, `testpackage` —
white-box test files renamed to `*_internal_test.go`), including #55's code
absorbed after it merged, iterating the pinned linter to `0 issues.`; no
single finding total is substantiable, since golangci-lint's `uniq-by-line`
reveals new findings on a line as others there are fixed — the documented
re-measurements were 81 after the #53 merge and 149 after the #55 merge; three
behavior changes, so not a pure no-op: `Cache.StoreVariant` now takes a
`context.Context` (`noctx`), so a cancelled request skips its best-effort
accounting row; `MetadataStorage.Store`'s cleanup defer was dead on `main` and
leaked `.tmp-*.json` on failure, now fixed with explicit removals; and the
`signing_key` validation error text gained `value too short: `; the eviction
loop's uncancellable context is deferred to #102 under a
`//nolint:contextcheck`; three `//nolint:tagliatelle` directives keep the
snake_case JSON wire/disk formats unchanged; `make check` green
- 2026-08-07 implement cache size management and eviction (closes #51): new
`cache_max_bytes` config key validated by the startup framework (explicit
values used exactly with no floor, `0` disables the disk cache entirely,
omitted defaults to max(75% of free space on the filesystem containing
`<state_dir>/cache/`, 500 MiB), logged at startup); processed variants are now
tracked in the database (a new `variant_content` table and an LRU timestamp on
`source_content`) so total usage is two SUMs, never a directory scan on the
hot path; a background goroutine evicts globally least-recently-used entries
(variants and source blobs merged) to the limit, woken by a periodic ticker
and by write-pressure notifications from stores; a source blob and ALL of its
`source_metadata` references are deleted in one transaction before the file is
unlinked, so multi-referenced blobs are never removed while referenced and
rows never point at deleted files; a startup and periodic reconciliation pass
adopts untracked variant files, drops rows for missing files, removes
unreachable source blobs, and sweeps stale temp files
- 2026-08-07 validate configuration on startup, fail fast on bad config (closes
#52): a config value that is set but unparseable or invalid aborts startup
naming the key and value (defaults apply only to omitted keys), unknown config
keys abort startup, a malformed config file aborts instead of being skipped,
and `state_dir` is verified creatable and writable before the listener binds
- 2026-08-07 manual test pass of the auth and encrypted URL flows against a
locally built and running `pixad` (built from `main` at `6573b9d`, port 18099,
local throwaway config); all six checks passed, plus all nine tests in
`scripts/manual-test.sh` (closes #49):
- [x] visit `/` and see the login form: HTTP 200, `Pixa - Login` page with
`name="key"` password form
- [x] wrong key shows an error: POST `/` with `key=wrong-key` returned HTTP
200 login page containing "Invalid signing key"
- [x] correct signing key shows the generator form: POST `/` returned HTTP
303 to `/` with
`Set-Cookie: pixa_session=...; HttpOnly; Secure; SameSite=Strict`; GET
`/` with that cookie rendered `Pixa - URL Generator` with the
`/generate` form and logout link
- [x] a generated encrypted URL serves the image: POST `/generate`
(ttl=3600) produced a `/v1/e/<token>/img.jpeg` URL that returned HTTP
200, `Content-Type: image/jpeg`, an 800x600 baseline JPEG of 61706
bytes
- [x] an expired URL (short TTL) returns 410: a ttl=1 URL fetched after 3 s
returned HTTP 410 Gone with
`{"error":"URL has expired","status":410,...}`
- [x] logout redirects back to login: GET `/logout` returned HTTP 303 to `/`
with `Set-Cookie: pixa_session=; Max-Age=0`; subsequent GET `/`
rendered the login form again
- 2026-08-07 fix the two remaining gosec findings (G124 in internal/session):
session cookies now always carry Secure/HttpOnly/SameSite=Strict on both the
set and clear paths; `make check` green (closes #47)
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
shims, README Entrypoints section
- 2026-04-07 extract magic byte detection into internal/magic (#42)
- 2026-03-25 extract allowlist package from internal/imgcache (#41)
- 2026-03-25 move schema_migrations table creation into 000.sql (#36)
- 2026-03-20 enforce and document exact-match-only signature verification (#40)
- 2026-03-20 bound imageprocessor.Process input read to prevent unbounded memory
use (#37); consolidate appname into an internal/globals constant (#34)
- 2026-03-18 parse version prefix from migration filenames (#33)
- 2026-03-15 QA audit fixes for 1.0/MVP readiness (#25)
- 2026-03-02 split Dockerfile with pre-built golangci-lint stage for faster CI
(#23)
- 2026-02-25 repo policy compliance: CI workflow, hash-pinned images,
golangci-lint and gosec fixes of that date (#14); arm64 Docker build fix (#16)
- 2026-01-08 WebP and AVIF encoding support via govips (both former P0 image
processing items, now done)
# Future Steps # Future Steps
- P2: HTTP response handling - P2: HTTP response handling
- Last-Modified headers - Last-Modified headers
- P2: configuration
- YAML config file support
- P2: operational - P2: operational
- optional Sentry error reporting
- comprehensive request logging
- Prometheus performance metrics
- measure the 1k to 5k req/s target with `script/loadtest` on a machine not - measure the 1k to 5k req/s target with `script/loadtest` on a machine not
shared with other work shared with other work
configs/config.example.yml
+13 -12
View File
@@ -45,15 +45,14 @@ state_dir: ./data
signing_key: "CHANGE_ME_generate_with_openssl_rand_base64_32" signing_key: "CHANGE_ME_generate_with_openssl_rand_base64_32"
# Hosts that don't require signatures (default: none) # Hosts that don't require signatures (default: none)
# Every image on an allowlisted host is served to anyone without a
# signature, at any size and in any format, so list only hosts you
# control, where nobody else can put images.
# Use "." prefix for wildcard subdomain matching (e.g., ".example.com" matches "cdn.example.com") # Use "." prefix for wildcard subdomain matching (e.g., ".example.com" matches "cdn.example.com")
# An entry that is neither a host name nor an IP address (IPv6 without # An entry that is neither a host name nor an IP address (IPv6 without
# brackets), such as one with a port or a "*." wildcard, aborts startup. # brackets), such as one with a port or a "*." wildcard, aborts startup.
allowlist_hosts: # allowlist_hosts:
- s3.sneak.cloud # - images.example.com
- static.sneak.cloud
- sneak.berlin
- github.com
- user-images.githubusercontent.com
# Hosts whose pages may not show pixa's images, written as for # Hosts whose pages may not show pixa's images, written as for
# allowlist_hosts. A request to /v1/image/ or /v1/e/ whose Referer header # allowlist_hosts. A request to /v1/image/ or /v1/e/ whose Referer header
@@ -109,10 +108,11 @@ upstream_connections_per_host: 20
# downstream_timeout has ended first. # downstream_timeout has ended first.
upstream_connections: 64 upstream_connections: 64
# Maximum number of images decoded and encoded at once (default: the # Maximum number of images decoded and encoded at once (default: half the
# number of CPUs pixa can use, which follows a container's CPU limit). A # number of CPUs pixa can use, at least 1; the number of CPUs follows a
# request that finds none free waits up to 10 seconds for one, and if none # container's CPU limit). A request that finds none free waits up to 10
# frees up it is answered 503, unless downstream_timeout has ended first. # seconds for one, and if none frees up it is answered 503, unless
# downstream_timeout has ended first.
# max_concurrent_processing: 4 # max_concurrent_processing: 4
# How many requests to the image routes (/v1/image/ and /v1/e/) one client # How many requests to the image routes (/v1/image/ and /v1/e/) one client
@@ -158,8 +158,9 @@ access_control_allow_origin: "*"
# Sentry DSN for error reporting (default: empty, which turns it off) # Sentry DSN for error reporting (default: empty, which turns it off)
sentry_dsn: "" sentry_dsn: ""
# Username and password for /metrics, set together (default: unset). Metrics # Username and password for /metrics, set together (default: unset). pixa
# are measured and /metrics is served only when both are set. # serves /metrics, and measures HTTP requests, only when both are set. It
# measures the pixa_ metrics either way.
# metrics: # metrics:
# username: "admin" # username: "admin"
# password: "secret" # password: "secret"
go.mod
+1 -2
View File
@@ -14,10 +14,10 @@ require (
github.com/go-chi/httprate v0.16.0 github.com/go-chi/httprate v0.16.0
github.com/gorilla/csrf v1.7.3 github.com/gorilla/csrf v1.7.3
github.com/gorilla/securecookie v1.1.2 github.com/gorilla/securecookie v1.1.2
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/prometheus/client_golang v1.23.2 github.com/prometheus/client_golang v1.23.2
github.com/slok/go-http-metrics v0.13.0 github.com/slok/go-http-metrics v0.13.0
github.com/spf13/cobra v1.10.2 github.com/spf13/cobra v1.10.2
go.uber.org/dig v1.19.0
go.uber.org/fx v1.24.0 go.uber.org/fx v1.24.0
golang.org/x/crypto v0.41.0 golang.org/x/crypto v0.41.0
golang.org/x/sync v0.19.0 golang.org/x/sync v0.19.0
@@ -127,7 +127,6 @@ require (
go.opentelemetry.io/otel v1.36.0 // indirect go.opentelemetry.io/otel v1.36.0 // indirect
go.opentelemetry.io/otel/metric v1.36.0 // indirect go.opentelemetry.io/otel/metric v1.36.0 // indirect
go.opentelemetry.io/otel/trace v1.36.0 // indirect go.opentelemetry.io/otel/trace v1.36.0 // indirect
go.uber.org/dig v1.19.0 // indirect
go.uber.org/multierr v1.11.0 // indirect go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect go.uber.org/zap v1.27.0 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect go.yaml.in/yaml/v2 v2.4.2 // indirect
go.sum
-2
View File
@@ -228,8 +228,6 @@ github.com/hashicorp/go-version v1.2.1/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc=
github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/hashicorp/hcl v1.0.1-vault-7 h1:ag5OxFVy3QYTFTJODRzTKVZ6xvdfLLCA1cy/Y6xGI0I= github.com/hashicorp/hcl v1.0.1-vault-7 h1:ag5OxFVy3QYTFTJODRzTKVZ6xvdfLLCA1cy/Y6xGI0I=
github.com/hashicorp/hcl v1.0.1-vault-7/go.mod h1:XYhtn6ijBSAj6n4YqAaf7RBPS4I06AItNorpy+MoQNM= github.com/hashicorp/hcl v1.0.1-vault-7/go.mod h1:XYhtn6ijBSAj6n4YqAaf7RBPS4I06AItNorpy+MoQNM=
github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64= github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64=
internal/allowlist/allowlist.go
-10
View File
@@ -72,13 +72,3 @@ func (w *HostAllowList) IsAllowed(u *url.URL) bool {
return false return false
} }
// IsEmpty returns true if the allow list has no entries.
func (w *HostAllowList) IsEmpty() bool {
return len(w.exactHosts) == 0 && len(w.suffixHosts) == 0
}
// Count returns the total number of allow list entries.
func (w *HostAllowList) Count() int {
return len(w.exactHosts) + len(w.suffixHosts)
}
internal/allowlist/allowlist_test.go
-84
View File
@@ -139,87 +139,3 @@ func TestHostAllowList_IsAllowed_SuffixMatch(t *testing.T) {
}, },
}) })
} }
func TestHostAllowList_IsEmpty(t *testing.T) {
t.Parallel()
tests := []struct {
name string
patterns []string
want bool
}{
{
name: "empty",
patterns: []string{},
want: true,
},
{
name: "nil",
patterns: nil,
want: true,
},
{
name: "whitespace only",
patterns: []string{" ", ""},
want: true,
},
{
name: "has entries",
patterns: []string{"example.com"},
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
w := allowlist.New(tt.patterns)
if got := w.IsEmpty(); got != tt.want {
t.Errorf("IsEmpty() = %v, want %v", got, tt.want)
}
})
}
}
func TestHostAllowList_Count(t *testing.T) {
t.Parallel()
tests := []struct {
name string
patterns []string
want int
}{
{
name: "empty",
patterns: []string{},
want: 0,
},
{
name: "exact hosts only",
patterns: []string{"a.com", "b.com", "c.com"},
want: 3,
},
{
name: "suffix hosts only",
patterns: []string{".a.com", ".b.com"},
want: 2,
},
{
name: "mixed",
patterns: []string{"exact.com", ".suffix.com"},
want: 2,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
w := allowlist.New(tt.patterns)
if got := w.Count(); got != tt.want {
t.Errorf("Count() = %v, want %v", got, tt.want)
}
})
}
}
internal/app/app.go
+48 -2
View File
@@ -2,12 +2,16 @@
package app package app
import ( import (
"context"
"fmt" "fmt"
"io"
"os" "os"
"os/signal" "os/signal"
"syscall" "syscall"
"github.com/prometheus/client_golang/prometheus"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"go.uber.org/dig"
"go.uber.org/fx" "go.uber.org/fx"
"sneak.berlin/go/pixa/internal/config" "sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/database" "sneak.berlin/go/pixa/internal/database"
@@ -16,6 +20,7 @@ import (
"sneak.berlin/go/pixa/internal/healthcheck" "sneak.berlin/go/pixa/internal/healthcheck"
"sneak.berlin/go/pixa/internal/logger" "sneak.berlin/go/pixa/internal/logger"
"sneak.berlin/go/pixa/internal/middleware" "sneak.berlin/go/pixa/internal/middleware"
"sneak.berlin/go/pixa/internal/providers"
"sneak.berlin/go/pixa/internal/server" "sneak.berlin/go/pixa/internal/server"
) )
@@ -51,7 +56,10 @@ func run(_ *cobra.Command, _ []string) {
// A write to a closed stdout or stderr must not end the process. // A write to a closed stdout or stderr must not end the process.
signal.Ignore(syscall.SIGPIPE) signal.Ignore(syscall.SIGPIPE)
fx.New( app := fx.New(
// pixad logs with slog only; runApp prints a startup or shutdown
// error itself.
fx.NopLogger,
fx.Provide( fx.Provide(
config.New, config.New,
database.New, database.New,
@@ -61,10 +69,48 @@ func run(_ *cobra.Command, _ []string) {
server.New, server.New,
middleware.New, middleware.New,
healthcheck.New, healthcheck.New,
// The registry /metrics serves
func() prometheus.Registerer { return prometheus.DefaultRegisterer },
), ),
fx.Provide(providers.Constructors()...),
fx.Invoke( fx.Invoke(
func(log *logger.Logger) { log.Identify() }, func(log *logger.Logger) { log.Identify() },
func(*server.Server) {}, func(*server.Server) {},
), ),
).Run() )
exitCode := runApp(app, os.Stderr)
if exitCode != 0 {
os.Exit(exitCode)
}
}
// runApp starts app, waits for a signal or a shutdown request, stops app,
// and returns the status pixad exits with, as fx.App.Run does. It prints
// a start or stop error to stderr, once, as the error that caused it:
// dig.RootCause drops the frames fx and dig wrap around it.
func runApp(app *fx.App, stderr io.Writer) int {
startCtx, cancelStart := context.WithTimeout(context.Background(), app.StartTimeout())
defer cancelStart()
err := app.Start(startCtx)
if err != nil {
_, _ = fmt.Fprintln(stderr, dig.RootCause(err))
return 1
}
shutdown := <-app.Wait()
stopCtx, cancelStop := context.WithTimeout(context.Background(), app.StopTimeout())
defer cancelStop()
err = app.Stop(stopCtx)
if err != nil {
_, _ = fmt.Fprintln(stderr, dig.RootCause(err))
return 1
}
return shutdown.ExitCode
} }
internal/app/app_internal_test.go
+65
View File
@@ -0,0 +1,65 @@
package app
import (
"bytes"
"context"
"errors"
"io"
"testing"
"go.uber.org/fx"
)
var errTestStopFailed = errors.New("test stop hook failed")
// TestRunAppPrintsStopErrorOnce checks that when a stop hook fails, runApp
// prints the hook's error once and returns exit status 1.
func TestRunAppPrintsStopErrorOnce(t *testing.T) {
t.Parallel()
app := fx.New(
fx.NopLogger,
fx.Invoke(func(lifecycle fx.Lifecycle, shutdowner fx.Shutdowner) {
lifecycle.Append(fx.Hook{
// Ask to shut down as soon as the app has started, so
// that runApp goes on to stop it.
OnStart: func(context.Context) error { return shutdowner.Shutdown() },
OnStop: func(context.Context) error { return errTestStopFailed },
})
}),
)
var stderr bytes.Buffer
exitCode := runApp(app, &stderr)
if exitCode != 1 {
t.Errorf("runApp() = %d, want 1", exitCode)
}
want := errTestStopFailed.Error() + "\n"
if stderr.String() != want {
t.Errorf("stderr = %q, want %q", stderr.String(), want)
}
}
// TestRunAppReturnsShutdownExitCode checks that runApp returns the exit
// code a shutdown request names.
func TestRunAppReturnsShutdownExitCode(t *testing.T) {
t.Parallel()
app := fx.New(
fx.NopLogger,
fx.Invoke(func(lifecycle fx.Lifecycle, shutdowner fx.Shutdowner) {
lifecycle.Append(fx.Hook{
OnStart: func(context.Context) error {
return shutdowner.Shutdown(fx.ExitCode(3))
},
})
}),
)
exitCode := runApp(app, io.Discard)
if exitCode != 3 {
t.Errorf("runApp() = %d, want 3", exitCode)
}
}
internal/app/app_test.go
+177
View File
@@ -0,0 +1,177 @@
package app_test
import (
"bytes"
"errors"
"fmt"
"net"
"net/http"
"os"
"os/exec"
"strconv"
"syscall"
"testing"
"time"
"sneak.berlin/go/pixa/internal/app"
)
// These tests run pixad in a child process: the test binary itself,
// started again with runPixadInChildEnvVar set, which makes TestMain run
// pixad instead of the tests.
const runPixadInChildEnvVar = "TEST_RUN_PIXAD_IN_CHILD"
const validTestSigningKey = "test-signing-key-for-app-tests-0123456789"
func TestMain(m *testing.M) {
if os.Getenv(runPixadInChildEnvVar) == "1" {
os.Args = []string{"pixad"}
app.Run("test")
os.Exit(0)
}
os.Exit(m.Run())
}
// pixadCommand returns a command that runs pixad with only the given
// environment, in an empty directory with an empty HOME, so that no
// config file is found.
func pixadCommand(t *testing.T, env ...string) (*exec.Cmd, *bytes.Buffer) {
t.Helper()
testBinary := os.Args[0]
cmd := exec.CommandContext(t.Context(), testBinary) //nolint:gosec // not tainted
cmd.Dir = t.TempDir()
// A test binary built with -cover warns on stderr at exit when
// GOCOVERDIR is not set.
cmd.Env = append([]string{
runPixadInChildEnvVar + "=1",
"HOME=" + t.TempDir(),
"GOCOVERDIR=" + t.TempDir(),
}, env...)
var stderr bytes.Buffer
cmd.Stderr = &stderr
return cmd, &stderr
}
// TestRunPrintsStartupErrorOnce checks that an invalid setting is
// printed once, as the line that names it, and nothing else goes to
// stderr.
func TestRunPrintsStartupErrorOnce(t *testing.T) {
t.Parallel()
cmd, stderr := pixadCommand(t, "PIXA_SIGNING_KEY=short")
err := cmd.Run()
var exitErr *exec.ExitError
if !errors.As(err, &exitErr) || exitErr.ExitCode() != 1 {
t.Fatalf("pixad: got %v, want exit status 1; stderr:\n%s", err, stderr)
}
want := `config key "signing_key" (environment variable PIXA_SIGNING_KEY): ` +
"value too short: must be at least 32 characters, got 5\n"
if stderr.String() != want {
t.Errorf("stderr:\n%s\nwant:\n%s", stderr, want)
}
}
// TestRunWritesNothingToStderrOnStartAndStop starts pixad, waits until
// it answers its health check, stops it with SIGTERM, and checks that it
// wrote nothing to stderr.
func TestRunWritesNothingToStderrOnStartAndStop(t *testing.T) {
t.Parallel()
port := freePort(t)
cmd, stderr := pixadCommand(t,
"PORT="+strconv.Itoa(port),
"PIXA_STATE_DIR="+t.TempDir(),
"PIXA_SIGNING_KEY="+validTestSigningKey,
)
err := cmd.Start()
if err != nil {
t.Fatalf("starting pixad: %v", err)
}
waitForHealthCheck(t, cmd, port, stderr)
err = cmd.Process.Signal(syscall.SIGTERM)
if err != nil {
t.Fatalf("signalling pixad: %v", err)
}
err = cmd.Wait()
if err != nil {
t.Fatalf("pixad: got %v, want exit status 0; stderr:\n%s", err, stderr)
}
if stderr.Len() != 0 {
t.Errorf("stderr is not empty:\n%s", stderr)
}
}
// freePort returns a TCP port on the loopback address that nothing
// listens on.
func freePort(t *testing.T) int {
t.Helper()
var listenConfig net.ListenConfig
listener, err := listenConfig.Listen(t.Context(), "tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("finding a free port: %v", err)
}
addr, ok := listener.Addr().(*net.TCPAddr)
if !ok {
t.Fatalf("listener address %v is not a TCP address", listener.Addr())
}
err = listener.Close()
if err != nil {
t.Fatalf("closing listener: %v", err)
}
return addr.Port
}
// waitForHealthCheck waits up to 30 seconds for pixad, started as cmd, to
// answer its health check on port. If it does not, it stops pixad and
// fails the test with what pixad wrote to stderr.
func waitForHealthCheck(t *testing.T, cmd *exec.Cmd, port int, stderr *bytes.Buffer) {
t.Helper()
url := fmt.Sprintf("http://127.0.0.1:%d/.well-known/healthcheck.json", port)
deadline := time.Now().Add(30 * time.Second)
for time.Now().Before(deadline) {
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, url, nil)
if err != nil {
t.Fatalf("health check request: %v", err)
}
resp, err := http.DefaultClient.Do(req)
if err == nil {
_ = resp.Body.Close()
if resp.StatusCode == http.StatusOK {
return
}
}
time.Sleep(100 * time.Millisecond)
}
// Stop pixad first: stderr is complete, and safe to read, only once
// cmd.Wait has returned.
_ = cmd.Process.Kill()
_ = cmd.Wait()
t.Fatalf("pixad did not answer its health check on port %d; stderr:\n%s",
port, stderr)
}
internal/config/blocked_networks_internal_test.go
+53
View File
@@ -88,3 +88,56 @@ func TestBlockedNetworksInvalidAbortsStartup(t *testing.T) {
}, },
}) })
} }
// TestBlockedNetworksMappedEntryIsIPv4Range checks that an entry written in
// the IPv4-mapped form (::ffff:a.b.c.d) is kept as the IPv4 range it covers,
// since the fetcher matches an IPv4-mapped address as the IPv4 address it
// holds. /96, the shortest such entry, covers every IPv4 address.
func TestBlockedNetworksMappedEntryIsIPv4Range(t *testing.T) {
t.Parallel()
yamlContent := signingKeyLine + `blocked_networks:
- "::ffff:10.0.0.0/104"
- "::ffff:0:0/96"
`
c, err := configFromYAML(t, yamlContent)
if err != nil {
t.Fatalf("IPv4-mapped blocked_networks entries should load, got error: %v", err)
}
want := []string{"10.0.0.0/8", "0.0.0.0/0"}
if len(c.BlockedNetworks) != len(want) {
t.Fatalf("BlockedNetworks = %v, want %d entries", c.BlockedNetworks, len(want))
}
for i, w := range want {
if got := c.BlockedNetworks[i].String(); got != w {
t.Errorf("BlockedNetworks[%d] = %q, want %q", i, got, w)
}
}
}
// TestBlockedNetworksMappedEntryShorterThan96AbortsStartup checks that an
// entry in the IPv4-mapped form shorter than /96, which covers more than the
// IPv4 addresses, aborts startup with an error naming the key and the value.
func TestBlockedNetworksMappedEntryShorterThan96AbortsStartup(t *testing.T) {
t.Parallel()
runAbortCases(t, []abortCase{
{
name: "mapped-shorter-than-96",
yaml: signingKeyLine + `blocked_networks:
- "::ffff:0.0.0.0/80"
`,
wantErrSubstrings: []string{keyBlockedNetworks, "::ffff:0.0.0.0/80"},
},
{
name: "mapped-95",
yaml: signingKeyLine + `blocked_networks:
- "::ffff:0:0/95"
`,
wantErrSubstrings: []string{keyBlockedNetworks},
},
})
}
internal/config/cache_max_bytes_internal_test.go
+31 -3
View File
@@ -135,10 +135,38 @@ func TestCacheMaxBytesInvalidValuesAbortStartup(t *testing.T) {
} }
} }
// TestDefaultCacheMaxBytesAppliesFloor verifies that when 75% of the
// free space plus what the cache holds is below 500 MiB, the default is
// floored at DefaultCacheMaxBytesFloor.
func TestDefaultCacheMaxBytesAppliesFloor(t *testing.T) {
t.Parallel()
cases := []struct {
name string
freeBytes uint64
}{
{name: "100 MiB free", freeBytes: 104857600},
{name: "zero free", freeBytes: 0},
{name: "just below floor threshold", freeBytes: 699050665},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got := DefaultCacheMaxBytes(tc.freeBytes, 0)
if got != DefaultCacheMaxBytesFloor {
t.Errorf("DefaultCacheMaxBytes = %d, want floor %d",
got, DefaultCacheMaxBytesFloor)
}
})
}
}
// TestCacheMaxBytesExplicitIsRecorded verifies that an omitted // TestCacheMaxBytesExplicitIsRecorded verifies that an omitted
// cache_max_bytes is recorded as not explicit, so the cache works out // cache_max_bytes is recorded as not explicit, so pixad works out the
// the default when it opens, and that an explicit zero is recorded as // default when it builds the cache, and that an explicit zero is recorded
// explicit, so it disables the disk cache instead. // as explicit, so it disables the disk cache instead.
func TestCacheMaxBytesExplicitIsRecorded(t *testing.T) { func TestCacheMaxBytesExplicitIsRecorded(t *testing.T) {
t.Parallel() t.Parallel()
internal/config/concurrency_limits_internal_test.go
+52 -5
View File
@@ -1,6 +1,7 @@
package config package config
import ( import (
"fmt"
"runtime" "runtime"
"testing" "testing"
) )
@@ -12,8 +13,8 @@ const (
) )
// TestOmittedConcurrencyLimitsUseDefaults checks that an omitted // TestOmittedConcurrencyLimitsUseDefaults checks that an omitted
// max_concurrent_processing is the number of CPUs Go uses and an omitted // max_concurrent_processing is half the CPUs Go uses, and at least 1, and
// upstream_connections is 64. // an omitted upstream_connections is 64.
func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) { func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
t.Parallel() t.Parallel()
@@ -22,9 +23,9 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
t.Fatalf("minimal config should be valid, got error: %v", err) t.Fatalf("minimal config should be valid, got error: %v", err)
} }
if c.MaxConcurrentProcessing != runtime.GOMAXPROCS(0) { if c.MaxConcurrentProcessing != max(runtime.GOMAXPROCS(0)/2, 1) {
t.Errorf("MaxConcurrentProcessing = %d, want %d, one per CPU", t.Errorf("MaxConcurrentProcessing = %d, want %d, half the CPUs",
c.MaxConcurrentProcessing, runtime.GOMAXPROCS(0)) c.MaxConcurrentProcessing, max(runtime.GOMAXPROCS(0)/2, 1))
} }
if c.UpstreamConnections != 64 { if c.UpstreamConnections != 64 {
@@ -32,6 +33,36 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
} }
} }
// TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne checks
// that an omitted max_concurrent_processing is half the CPUs Go uses,
// rounded down, and 1 when Go uses one CPU.
//
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
func TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
cpusBefore := runtime.GOMAXPROCS(0)
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
for _, tc := range []struct{ cpus, want int }{
{1, 1}, {2, 1}, {3, 1}, {8, 4},
} {
runtime.GOMAXPROCS(tc.cpus)
c, err := configFromYAML(t, signingKeyLine)
if err != nil {
t.Fatalf("minimal config should be valid, got error: %v", err)
}
t.Logf("GOMAXPROCS %d: MaxConcurrentProcessing %d",
tc.cpus, c.MaxConcurrentProcessing)
if c.MaxConcurrentProcessing != tc.want {
t.Errorf("GOMAXPROCS %d: MaxConcurrentProcessing = %d, want %d",
tc.cpus, c.MaxConcurrentProcessing, tc.want)
}
}
}
// TestExplicitConcurrencyLimitsAreUsed checks that valid values for the // TestExplicitConcurrencyLimitsAreUsed checks that valid values for the
// two limits are used as given. // two limits are used as given.
func TestExplicitConcurrencyLimitsAreUsed(t *testing.T) { func TestExplicitConcurrencyLimitsAreUsed(t *testing.T) {
@@ -160,3 +191,19 @@ func TestConcurrencyLimitsFromEnvironment(t *testing.T) {
}) })
} }
} }
// TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup checks
// that a PIXA_MAX_CONCURRENT_PROCESSING that is not a whole number of at
// least 1, an empty one included, aborts startup naming the variable and
// the key instead of taking the default.
func TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup(t *testing.T) {
for _, value := range []string{"", " ", "0.5", "half", "-1", "4 images"} {
t.Run(fmt.Sprintf("value %q", value), func(t *testing.T) {
t.Setenv(testMaxConcurrentProcessingVar, value)
_, err := configFromYAML(t, signingKeyLine)
wantStartupError(t, err,
testMaxConcurrentProcessingVar, keyMaxConcurrentProcessing)
})
}
}
internal/config/config.go
+431 -593
View File
File diff suppressed because it is too large. Load diff
internal/config/config_internal_test.go
+14 -73
View File
@@ -1,94 +1,35 @@
package config package config
import ( import (
"os"
"path/filepath" "path/filepath"
"testing" "testing"
"git.eeqj.de/sneak/smartconfig" "git.eeqj.de/sneak/smartconfig"
) )
// writeTestConfig writes yamlContent to a temp config file and returns // TestExampleConfigAllowlistsNoHost loads configs/config.example.yml
// the file path. // with a real signing key, as Getting Started in README.md has it copied
func writeTestConfig(t *testing.T, yamlContent string) string { // and edited, and checks that it allowlists no host: every image on an
t.Helper() // allowlisted host is served to anyone without a signature.
func TestExampleConfigAllowlistsNoHost(t *testing.T) {
t.Setenv("PIXA_SIGNING_KEY", validTestSigningKey)
tmpDir := t.TempDir() examplePath := filepath.Join("..", "..", "configs", "config.example.yml")
configPath := filepath.Join(tmpDir, "config.yml")
err := os.WriteFile(configPath, []byte(yamlContent), 0o600) sc, err := loadTestConfig(examplePath)
if err != nil { if err != nil {
t.Fatalf("failed to write test config: %v", err) t.Fatalf("failed to load %s: %v", examplePath, err)
} }
return configPath c, err := newFromSmartConfig(sc)
}
// checkAllowlistHosts loads the config at configPath and asserts that
// getStringSlice returns the three expected hosts.
func checkAllowlistHosts(t *testing.T, configPath string) {
t.Helper()
sc, err := loadTestConfig(configPath)
if err != nil { if err != nil {
t.Fatalf("failed to load config: %v", err) t.Fatalf("%s is not a valid config: %v", examplePath, err)
} }
hosts := getStringSlice(sc) t.Logf("allowlist_hosts in %s: %v", examplePath, c.AllowlistHosts)
if len(hosts) != 3 { if len(c.AllowlistHosts) != 0 {
t.Errorf("expected 3 hosts, got %d: %v", len(hosts), hosts) t.Errorf("%s allowlists %v, want no host", examplePath, c.AllowlistHosts)
}
expected := []string{"static.sneak.cloud", "sneak.berlin", testHostS3}
for i, want := range expected {
if i >= len(hosts) {
t.Errorf("missing host at index %d: want %q", i, want)
continue
}
if hosts[i] != want {
t.Errorf("host[%d] = %q, want %q", i, hosts[i], want)
}
}
}
func TestGetStringSlice_YAMLList(t *testing.T) {
t.Parallel()
yamlContent := `
allowlist_hosts:
- static.sneak.cloud
- sneak.berlin
- s3.sneak.cloud
`
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
}
func TestGetStringSlice_CommaSeparated(t *testing.T) {
t.Parallel()
// Backwards compatibility with comma-separated string values.
yamlContent := `allowlist_hosts: "static.sneak.cloud, sneak.berlin, s3.sneak.cloud"`
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
}
func TestGetStringSlice_Empty(t *testing.T) {
t.Parallel()
configPath := writeTestConfig(t, `port: 8080`)
sc, err := loadTestConfig(configPath)
if err != nil {
t.Fatalf("failed to load config: %v", err)
}
hosts := getStringSlice(sc)
if len(hosts) != 0 {
t.Errorf("expected nil or empty slice, got %v", hosts)
} }
} }
internal/config/config_validation_internal_test.go
+127 -1
View File
@@ -15,8 +15,36 @@ import (
_ "modernc.org/sqlite" // SQLite driver registration _ "modernc.org/sqlite" // SQLite driver registration
) )
// The config keys the tests name in the errors they expect.
const (
keyDebug = "debug"
keyMaintenanceMode = "maintenance_mode"
keyPort = "port"
keyStateDir = "state_dir"
keySentryDSN = "sentry_dsn"
keyDBURL = "db_url"
keyMetrics = "metrics"
keyMetricsUsername = "metrics.username"
keyMetricsPassword = "metrics.password"
keySigningKey = "signing_key"
keyAllowlistHosts = "allowlist_hosts"
keyRefererBlocklist = "referer_blocklist"
keyAllowHTTP = "allow_http"
keyUpstreamConnectionsPerHost = "upstream_connections_per_host"
keyUpstreamConnections = "upstream_connections"
keyMaxConcurrentProcessing = "max_concurrent_processing"
keyCacheMaxBytes = "cache_max_bytes"
keyBlockedNetworks = "blocked_networks"
keyTrustedProxies = "trusted_proxies"
keyAccessControlAllowOrigin = "access_control_allow_origin"
keyUpstreamFetchTimeout = "upstream_fetch_timeout"
keyUpstreamMaxResponseSize = "upstream_max_response_size"
keyDownstreamTimeout = "downstream_timeout"
keyImageRequestsPerMinute = "image_requests_per_minute"
)
// validTestSigningKey is a 32-character signing key that satisfies the // validTestSigningKey is a 32-character signing key that satisfies the
// minimum length requirement in validate(). // minimum length requirement in validateSigningKey.
const validTestSigningKey = "0123456789abcdef0123456789abcdef" const validTestSigningKey = "0123456789abcdef0123456789abcdef"
// signingKeyLine is a valid signing_key config line used as the base of // signingKeyLine is a valid signing_key config line used as the base of
@@ -135,6 +163,104 @@ func TestDefaultDBURLOpensTheDatabaseInWALMode(t *testing.T) {
} }
} }
// TestDerivedDBURLOpensStateDirHoldingURLCharacters opens the db_url
// derived from a state_dir whose name holds %41, ? and #, and checks that
// the database is state.sqlite3 in that directory, in WAL mode. Put into
// a file: URL as it is, the name would lose its meaning: %41 is read as
// A, and the path ends at the ?, which also loses the journal mode.
func TestDerivedDBURLOpensStateDirHoldingURLCharacters(t *testing.T) {
t.Parallel()
stateDir := filepath.Join(t.TempDir(), "a%41b?c#d")
err := os.Mkdir(stateDir, 0o750)
if err != nil {
t.Fatalf("failed to create %q: %v", stateDir, err)
}
// Quoted so YAML takes the whole name as it is.
c, err := configFromYAML(t, signingKeyLine+"state_dir: '"+stateDir+"'\n")
if err != nil {
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
}
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(stateDir, "state.sqlite3"))
}
// TestDerivedDBURLOpensStateDirStartingWithTwoSlashes opens the db_url
// derived from a state_dir that starts with //, and checks that the
// database is state.sqlite3 in that directory, in WAL mode. Written right
// after file:, such a name gives a URL starting file://, and SQLite reads
// the first directory name as a host and refuses to open it.
func TestDerivedDBURLOpensStateDirStartingWithTwoSlashes(t *testing.T) {
t.Parallel()
dir := t.TempDir()
stateDir := "/" + dir
c, err := configFromYAML(t, signingKeyLine+"state_dir: "+stateDir+"\n")
if err != nil {
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
}
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(dir, "state.sqlite3"))
}
// TestDerivedDBURLForRootStateDir checks the db_url derived from a
// state_dir of /. The path written into the URL is then
// //state.sqlite3, and written right after file: it gives a URL starting
// file://, where SQLite reads state.sqlite3 as a host and refuses to open
// it. The test only checks the URL, since it cannot write to /.
func TestDerivedDBURLForRootStateDir(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, signingKeyLine+"state_dir: /\n")
if err != nil {
t.Fatalf("config with state_dir / should be valid, got: %v", err)
}
t.Logf("state_dir / gives db_url %q", c.DBURL)
wantDBURL := "file:////state.sqlite3?_pragma=journal_mode(WAL)"
if c.DBURL != wantDBURL {
t.Errorf("DBURL = %q, want %q", c.DBURL, wantDBURL)
}
}
// checkDBURLOpensFileInWALMode opens dbURL with the SQLite driver pixad
// uses, and checks that the database is in WAL mode and is the file
// dbFile.
func checkDBURLOpensFileInWALMode(t *testing.T, dbURL, dbFile string) {
t.Helper()
db, err := sql.Open("sqlite", dbURL)
if err != nil {
t.Fatalf("failed to open %q: %v", dbURL, err)
}
t.Cleanup(func() { _ = db.Close() })
var journalMode string
err = db.QueryRowContext(t.Context(), "PRAGMA journal_mode").Scan(&journalMode)
if err != nil {
t.Fatalf("failed to read the journal mode of %q: %v", dbURL, err)
}
if journalMode != "wal" {
t.Errorf("journal mode of %q = %q, want wal", dbURL, journalMode)
}
_, err = os.Stat(dbFile)
if err != nil {
t.Errorf("opening %q did not create %q: %v", dbURL, dbFile, err)
}
}
func TestExplicitValidValuesAreUsed(t *testing.T) { func TestExplicitValidValuesAreUsed(t *testing.T) {
t.Parallel() t.Parallel()
internal/config/documented_settings_internal_test.go
+127
View File
@@ -0,0 +1,127 @@
package config
import (
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
// TestReadmeNamesEverySetting checks that the list of settings in
// README.md names exactly the keys of the settings table, one item each.
// The list is the first one in the Configuration section, and it ends at
// the first blank line. Each item starts with its key in backticks.
func TestReadmeNamesEverySetting(t *testing.T) {
t.Parallel()
readme, err := os.ReadFile(filepath.Join("..", "..", "README.md"))
if err != nil {
t.Fatalf("failed to read README.md: %v", err)
}
_, section, found := strings.Cut(string(readme), "\n### Configuration\n")
if !found {
t.Fatal("README.md has no Configuration section")
}
// The section ends at the next heading of its level or above.
section, _, _ = strings.Cut(section, "\n## ")
section, _, _ = strings.Cut(section, "\n### ")
_, list, found := strings.Cut(section, "\n- ")
if !found {
t.Fatal("README.md's Configuration section has no list")
}
list, _, _ = strings.Cut("- "+list, "\n\n")
itemKey := regexp.MustCompile("^- `([^`]+)`")
named := map[string]bool{}
for line := range strings.SplitSeq(list, "\n") {
// A line that does not start with "- " goes on with the item above.
if !strings.HasPrefix(line, "- ") {
continue
}
match := itemKey.FindStringSubmatch(line)
if match == nil {
t.Errorf("README.md's list of settings has an item "+
"that does not start with a key in backticks: %s", line)
continue
}
if named[match[1]] {
t.Errorf("README.md's list of settings names %s twice", match[1])
}
named[match[1]] = true
}
for _, s := range new(Config).settings() {
if !named[s.key] {
t.Errorf("README.md's list of settings does not name %s", s.key)
}
delete(named, s.key)
}
for key := range named {
t.Errorf("README.md's list of settings names %s, "+
"which is not a setting", key)
}
}
// TestExampleConfigNamesEverySetting checks that
// configs/config.example.yml names exactly the keys of the settings
// table, each one set or commented out with "# ". A key line holds the
// key, a colon and at most a one-word value, which tells it from a
// comment line that starts with a word and a colon. An indented key is
// inside the map of the key above it, as username is inside metrics.
func TestExampleConfigNamesEverySetting(t *testing.T) {
t.Parallel()
const examplePath = "configs/config.example.yml"
example, err := os.ReadFile(filepath.Join("..", "..", examplePath))
if err != nil {
t.Fatalf("failed to read %s: %v", examplePath, err)
}
keyLine := regexp.MustCompile(`(?m)^(?:# )?( *)([a-z_]+):(?: \S+)?$`)
named := map[string]bool{}
var mapKey string
for _, match := range keyLine.FindAllStringSubmatch(string(example), -1) {
indent, key := match[1], match[2]
if indent == "" {
mapKey = key
named[key] = true
continue
}
// The key holding the map is not itself a setting.
delete(named, mapKey)
named[mapKey+"."+key] = true
}
// The env section sets environment variables; it is not a setting.
delete(named, "env")
for _, s := range new(Config).settings() {
if !named[s.key] {
t.Errorf("%s does not name %s", examplePath, s.key)
}
delete(named, s.key)
}
for key := range named {
t.Errorf("%s names %s, which is not a setting", examplePath, key)
}
}
internal/config/env_internal_test.go
+4 -3
View File
@@ -138,7 +138,8 @@ func TestPixaPortAbortsStartupPointingToPort(t *testing.T) {
// TestSettingVariablesAndConfigPathAreAccepted checks that every // TestSettingVariablesAndConfigPathAreAccepted checks that every
// setting's variable and PIXA_CONFIG_PATH pass the check for unknown // setting's variable and PIXA_CONFIG_PATH pass the check for unknown
// PIXA_ variables. TestEnvironmentSetsEveryKey pins the names in the list. // PIXA_ variables. TestEnvironmentSetsEveryKey pins the names in the
// settings table.
func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) { func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) {
// A config file's env section loaded by another test can leave a // A config file's env section loaded by another test can leave a
// PIXA_ variable set for the whole process, so every one is unset // PIXA_ variable set for the whole process, so every one is unset
@@ -159,8 +160,8 @@ func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) {
t.Setenv("PIXA_CONFIG_PATH", "/etc/pixa/config.yml") t.Setenv("PIXA_CONFIG_PATH", "/etc/pixa/config.yml")
for _, name := range envVarNames() { for _, s := range new(Config).settings() {
t.Setenv(name, "") t.Setenv(s.envVar, "")
} }
err := validateKnownEnvVars() err := validateKnownEnvVars()
internal/config/trusted_proxies_internal_test.go
+38
View File
@@ -83,3 +83,41 @@ func TestTrustedProxiesInvalidAbortsStartup(t *testing.T) {
}, },
}) })
} }
// TestTrustedProxiesMappedEntryIsIPv4Range checks that an entry written in
// the IPv4-mapped form (::ffff:a.b.c.d) is kept as the IPv4 range it covers,
// since the client address is matched as the IPv4 address it holds. /96, the
// shortest such entry, covers every IPv4 address.
func TestTrustedProxiesMappedEntryIsIPv4Range(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, signingKeyLine+
`trusted_proxies: ["::ffff:10.0.0.0/104", "::ffff:0:0/96"]`+"\n")
if err != nil {
t.Fatalf("IPv4-mapped trusted_proxies entries should load: %v", err)
}
got := make([]string, len(c.TrustedProxies))
for i, p := range c.TrustedProxies {
got[i] = p.String()
}
if joined := strings.Join(got, ","); joined != "10.0.0.0/8,0.0.0.0/0" {
t.Errorf("TrustedProxies = %v, want 10.0.0.0/8 and 0.0.0.0/0", got)
}
}
// TestTrustedProxiesMappedEntryShorterThan96AbortsStartup checks that an
// entry in the IPv4-mapped form shorter than /96, which covers more than the
// IPv4 addresses, aborts startup with an error naming the key and the value.
func TestTrustedProxiesMappedEntryShorterThan96AbortsStartup(t *testing.T) {
t.Parallel()
runAbortCases(t, []abortCase{
{
name: "mapped-95",
yaml: signingKeyLine + `trusted_proxies: ["::ffff:0:0/95"]` + "\n",
wantErrSubstrings: []string{keyTrustedProxies, "::ffff:0:0/95"},
},
})
}
internal/database/concurrent_writes_internal_test.go
+10 -19
View File
@@ -104,40 +104,31 @@ func writeConcurrently(t *testing.T, db *sql.DB) {
wg.Wait() wg.Wait()
var hits, sources int var sources int
err := db.QueryRowContext(ctx, ` err := db.QueryRowContext(ctx, `SELECT COUNT(*) FROM source_content`).
SELECT hit_count, (SELECT COUNT(*) FROM source_content) Scan(&sources)
FROM cache_stats WHERE id = 1
`).Scan(&hits, &sources)
if err != nil { if err != nil {
t.Fatalf("failed to count the writes: %v", err) t.Fatalf("failed to count the writes: %v", err)
} }
if hits != totalRequests || sources != totalRequests { if sources != totalRequests {
t.Errorf("hit_count = %d and %d source_content rows, want %d of each", t.Errorf("%d source_content rows, want %d", sources, totalRequests)
hits, sources, totalRequests)
} }
} }
// writeLikeOneRequest makes the writes one request and the eviction pass // writeLikeOneRequest makes the writes one request and the eviction pass
// make: it counts a cache hit, stores a source, records a transformed image // make: it stores a source, records a transformed image and deletes that
// and deletes that record again. // record again.
func writeLikeOneRequest(ctx context.Context, db *sql.DB, key string) error { func writeLikeOneRequest(ctx context.Context, db *sql.DB, key string) error {
_, err := db.ExecContext(ctx, _, err := db.ExecContext(ctx, `INSERT INTO source_content
`UPDATE cache_stats SET hit_count = hit_count + 1 WHERE id = 1`) (source_key, size_bytes) VALUES (?, 1)`, key)
if err != nil {
return fmt.Errorf("counting a cache hit: %w", err)
}
_, err = db.ExecContext(ctx, `INSERT INTO source_content
(content_hash, content_type, size_bytes) VALUES (?, 'image/png', 1)`, key)
if err != nil { if err != nil {
return fmt.Errorf("storing a source: %w", err) return fmt.Errorf("storing a source: %w", err)
} }
_, err = db.ExecContext(ctx, `INSERT INTO variant_content _, err = db.ExecContext(ctx, `INSERT INTO variant_content
(cache_key, size_bytes, content_type) VALUES (?, 1, 'image/png')`, key) (cache_key, size_bytes) VALUES (?, 1)`, key)
if err != nil { if err != nil {
return fmt.Errorf("recording a transformed image: %w", err) return fmt.Errorf("recording a transformed image: %w", err)
} }
internal/database/database.go
+11 -11
View File
@@ -82,7 +82,11 @@ func ParseMigrationVersion(filename string) (int, error) {
return version, nil return version, nil
} }
// New creates a new Database instance. // New opens the database and applies the migrations, so that the parts
// fx builds after it can read the database while they are built. Its
// stop hook closes the database. fx runs stop hooks in the reverse of the
// order it built their parts in, so the stop hooks of those parts run
// before this one.
func New(lc fx.Lifecycle, params Params) (*Database, error) { func New(lc fx.Lifecycle, params Params) (*Database, error) {
s := &Database{ s := &Database{
log: params.Logger.Get(), log: params.Logger.Get(),
@@ -91,20 +95,16 @@ func New(lc fx.Lifecycle, params Params) (*Database, error) {
s.log.Info("Database instantiated") s.log.Info("Database instantiated")
lc.Append(fx.Hook{ err := s.connect(context.Background())
OnStart: func(ctx context.Context) error { if err != nil {
s.log.Info("Database OnStart Hook") return nil, err
}
return s.connect(ctx) lc.Append(fx.Hook{
},
OnStop: func(_ context.Context) error { OnStop: func(_ context.Context) error {
s.log.Info("Database OnStop Hook") s.log.Info("Database OnStop Hook")
if s.db != nil { return s.db.Close()
return s.db.Close()
}
return nil
}, },
}) })
internal/database/database_internal_test.go
+1 -4
View File
@@ -159,10 +159,7 @@ func TestApplyMigrations_CreatesSchemaAndTables(t *testing.T) {
} }
// Verify that the application tables created by 001.sql exist. // Verify that the application tables created by 001.sql exist.
tables := []string{ tables := []string{"source_content", "negative_cache"}
"source_content", "source_metadata", "output_content",
"request_cache", "negative_cache", "cache_stats",
}
for _, table := range tables { for _, table := range tables {
var count int var count int
internal/db/migrations/001_schema.sql
+40 -81
View File
@@ -1,56 +1,51 @@
-- Migration 001: Initial schema -- Migration 001: Initial schema
-- Creates all tables for the pixa caching image proxy -- Creates all tables for the pixa caching image proxy
-- Source content blobs -- Source images, one row per source URL
-- Files stored at: cache/sources/<ab>/<cd>/<sha256> -- Files stored at: cache/sources/<ab>/<cd>/<source_key>, where source_key
-- is the SHA-256 of the source's host, path and query.
-- fetched_at is when the cached bytes were fetched. checked_at is when
-- upstream last sent them or answered that they had not changed.
-- upstream_etag and upstream_last_modified are the ETag and Last-Modified
-- headers upstream sent with the bytes, or '' when it sent none.
CREATE TABLE IF NOT EXISTS source_content ( CREATE TABLE IF NOT EXISTS source_content (
content_hash TEXT PRIMARY KEY, source_key TEXT PRIMARY KEY,
content_type TEXT NOT NULL,
size_bytes INTEGER NOT NULL, size_bytes INTEGER NOT NULL,
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP, last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP fetched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
checked_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
upstream_etag TEXT NOT NULL DEFAULT '',
upstream_last_modified TEXT NOT NULL DEFAULT ''
); );
CREATE INDEX IF NOT EXISTS idx_source_content_last_accessed CREATE INDEX IF NOT EXISTS idx_source_content_last_accessed
ON source_content(last_accessed_at); ON source_content(last_accessed_at);
CREATE INDEX IF NOT EXISTS idx_source_content_checked
-- Source URL metadata - maps URLs to content hashes ON source_content(checked_at);
-- JSON stored at: cache/metadata/<hostname>/<path_hash>.json
CREATE TABLE IF NOT EXISTS source_metadata (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_host TEXT NOT NULL,
source_path TEXT NOT NULL,
source_query TEXT NOT NULL DEFAULT '',
path_hash TEXT NOT NULL,
content_hash TEXT,
status_code INTEGER NOT NULL,
content_type TEXT,
response_headers TEXT,
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
expires_at DATETIME,
etag TEXT,
last_modified TEXT,
UNIQUE(source_host, source_path, source_query),
FOREIGN KEY (content_hash) REFERENCES source_content(content_hash)
);
CREATE INDEX IF NOT EXISTS idx_source_meta_host ON source_metadata(source_host);
CREATE INDEX IF NOT EXISTS idx_source_meta_path_hash ON source_metadata(path_hash);
CREATE INDEX IF NOT EXISTS idx_source_meta_expires ON source_metadata(expires_at);
CREATE INDEX IF NOT EXISTS idx_source_meta_content_hash ON source_metadata(content_hash);
-- Processed variant blobs -- Processed variant blobs
-- Files stored at: cache/variants/<ab>/<cd>/<cache_key> (plus a .meta -- Files stored at: cache/variants/<ab>/<cd>/<cache_key>. Tracked here
-- sidecar with the content type). Tracked here (like source content -- (like source content blobs above) so total cache usage is known
-- blobs above) so total cache usage is known without a directory scan, -- without a directory scan, and so LRU eviction has a timestamp to order
-- and so LRU eviction has a timestamp to order on. -- on. A cache hit does not read this table: it takes the content type
-- from the file's first bytes, and its last_accessed_at is written
-- later, in a batch. checked_at is the checked_at of the source the
-- variant was made from, when it was made or last renewed; the variant
-- file's modification time holds the same time for cache hits.
-- upstream_etag and upstream_last_modified are those of the source bytes
-- the variant was made from, so the variant can still be checked with
-- upstream after its source is evicted.
CREATE TABLE IF NOT EXISTS variant_content ( CREATE TABLE IF NOT EXISTS variant_content (
cache_key TEXT PRIMARY KEY, cache_key TEXT PRIMARY KEY,
size_bytes INTEGER NOT NULL, size_bytes INTEGER NOT NULL,
content_type TEXT NOT NULL, last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP, checked_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP upstream_etag TEXT NOT NULL DEFAULT '',
upstream_last_modified TEXT NOT NULL DEFAULT ''
); );
CREATE INDEX IF NOT EXISTS idx_variant_content_last_accessed CREATE INDEX IF NOT EXISTS idx_variant_content_last_accessed
ON variant_content(last_accessed_at); ON variant_content(last_accessed_at);
CREATE INDEX IF NOT EXISTS idx_variant_content_checked
ON variant_content(checked_at);
-- Total cache usage: the sum of size_bytes over source_content and -- Total cache usage: the sum of size_bytes over source_content and
-- variant_content, kept by the triggers below in the same statement -- variant_content, kept by the triggers below in the same statement
@@ -120,59 +115,23 @@ BEGIN
WHERE id = 1; WHERE id = 1;
END; END;
-- Output/transformed content blobs -- Negative cache: source URLs whose fetch failed with an upstream error
-- Not written: transformed images are stored in cache/variants and -- status, or whose first bytes did not match their Content-Type. pixa does
-- tracked in variant_content above. -- not fetch such a URL again until its row expires.
CREATE TABLE IF NOT EXISTS output_content (
content_hash TEXT PRIMARY KEY,
content_type TEXT NOT NULL,
size_bytes INTEGER NOT NULL,
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
-- Request cache - maps full request params to output content
CREATE TABLE IF NOT EXISTS request_cache (
id INTEGER PRIMARY KEY AUTOINCREMENT,
cache_key TEXT NOT NULL UNIQUE,
source_metadata_id INTEGER NOT NULL,
output_hash TEXT NOT NULL,
width INTEGER NOT NULL,
height INTEGER NOT NULL,
format TEXT NOT NULL,
quality INTEGER NOT NULL DEFAULT 85,
fit_mode TEXT NOT NULL DEFAULT 'cover',
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
access_count INTEGER NOT NULL DEFAULT 1,
FOREIGN KEY (source_metadata_id) REFERENCES source_metadata(id),
FOREIGN KEY (output_hash) REFERENCES output_content(content_hash)
);
CREATE INDEX IF NOT EXISTS idx_request_cache_key ON request_cache(cache_key);
CREATE INDEX IF NOT EXISTS idx_request_cache_source ON request_cache(source_metadata_id);
CREATE INDEX IF NOT EXISTS idx_request_cache_output ON request_cache(output_hash);
CREATE INDEX IF NOT EXISTS idx_request_cache_fetched ON request_cache(fetched_at);
-- Negative cache for failed fetches (404s, timeouts, etc.)
CREATE TABLE IF NOT EXISTS negative_cache ( CREATE TABLE IF NOT EXISTS negative_cache (
id INTEGER PRIMARY KEY AUTOINCREMENT, id INTEGER PRIMARY KEY AUTOINCREMENT,
source_host TEXT NOT NULL, source_host TEXT NOT NULL,
source_path TEXT NOT NULL, source_path TEXT NOT NULL,
source_query TEXT NOT NULL DEFAULT '', source_query TEXT NOT NULL DEFAULT '',
status_code INTEGER NOT NULL,
error_message TEXT,
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
expires_at DATETIME NOT NULL, expires_at DATETIME NOT NULL,
UNIQUE(source_host, source_path, source_query) UNIQUE(source_host, source_path, source_query)
); );
CREATE INDEX IF NOT EXISTS idx_negative_cache_expires ON negative_cache(expires_at); CREATE INDEX IF NOT EXISTS idx_negative_cache_expires ON negative_cache(expires_at);
-- Cache statistics for monitoring -- A random value that a login session carries from its login. pixa makes it
CREATE TABLE IF NOT EXISTS cache_stats ( -- when the table has no row, and each logout replaces it; a login session
-- carrying another value is refused, so logging out ends every login session.
CREATE TABLE IF NOT EXISTS logout (
id INTEGER PRIMARY KEY CHECK (id = 1), id INTEGER PRIMARY KEY CHECK (id = 1),
hit_count INTEGER NOT NULL DEFAULT 0, logout_value TEXT NOT NULL
miss_count INTEGER NOT NULL DEFAULT 0,
upstream_fetch_count INTEGER NOT NULL DEFAULT 0,
upstream_fetch_bytes INTEGER NOT NULL DEFAULT 0,
transform_count INTEGER NOT NULL DEFAULT 0,
last_updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
); );
INSERT OR IGNORE INTO cache_stats (id) VALUES (1);
internal/encurl/encurl.go
+62 -82
View File
@@ -7,18 +7,15 @@ import (
"github.com/fxamacker/cbor/v2" "github.com/fxamacker/cbor/v2"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal" "sneak.berlin/go/pixa/internal/seal"
) )
// Default values for optional fields. // Default values for optional fields.
const ( const (
DefaultQuality = 85 DefaultQuality = 85
DefaultFormat = imgcache.FormatJXL DefaultFormat = imagerequest.FormatJXL
DefaultFitMode = imgcache.FitCover DefaultFitMode = imagerequest.FitCover
// HKDF salt for URL encryption key derivation
urlKeySalt = "pixa-urlenc-v1"
) )
// Errors returned by encurl operations. // Errors returned by encurl operations.
@@ -28,19 +25,19 @@ var (
ErrDecryptFailed = errors.New("failed to decrypt URL") ErrDecryptFailed = errors.New("failed to decrypt URL")
) )
// Payload contains all image request parameters in a compact format. // payload contains all image request parameters in a compact format.
// Field names are short to minimize encoded size. // Field names are short to minimize encoded size.
// Fields with omitempty use sane defaults when absent. // Fields with omitempty use sane defaults when absent.
type Payload struct { type payload struct {
SourceHost string `cbor:"h"` // required SourceHost string `cbor:"h"` // required
SourcePath string `cbor:"p"` // required SourcePath string `cbor:"p"` // required
SourceQuery string `cbor:"q,omitempty"` // optional SourceQuery string `cbor:"q,omitempty"` // optional
Width int `cbor:"w,omitempty"` // 0 = original Width int `cbor:"w,omitempty"` // 0 = original
Height int `cbor:"ht,omitempty"` // 0 = original Height int `cbor:"ht,omitempty"` // 0 = original
Format imgcache.ImageFormat `cbor:"f,omitempty"` // default: jxl Format imagerequest.Format `cbor:"f,omitempty"` // default: jxl
Quality int `cbor:"ql,omitempty"` // default: 85 Quality int `cbor:"ql,omitempty"` // default: 85
FitMode imgcache.FitMode `cbor:"fm,omitempty"` // default: cover FitMode imagerequest.Fit `cbor:"fm,omitempty"` // default: cover
ExpiresAt int64 `cbor:"e,omitempty"` // 0 = never expires ExpiresAt int64 `cbor:"e,omitempty"` // 0 = never expires
} }
// Generator creates and parses encrypted URL tokens. // Generator creates and parses encrypted URL tokens.
@@ -48,20 +45,35 @@ type Generator struct {
key [seal.KeySize]byte key [seal.KeySize]byte
} }
// NewGenerator creates an encrypted URL generator with a key derived // NewGenerator creates an encrypted URL generator that encrypts its tokens
// from the signing key. // with key.
func NewGenerator(signingKey string) (*Generator, error) { func NewGenerator(key [seal.KeySize]byte) *Generator {
key, err := seal.DeriveKey([]byte(signingKey), urlKeySalt) return &Generator{key: key}
if err != nil {
return nil, err
}
return &Generator{key: key}, nil
} }
// Generate creates an encrypted URL token from the payload. // Generate creates an encrypted URL token from the request's source and
// The token is CBOR-encoded, encrypted with NaCl secretbox, and base64url-encoded. // transform, each as the request holds it, and from expires; an empty format,
func (g *Generator) Generate(p *Payload) (string, error) { // a quality of 0 or an empty fit mode is left out of the token, and a zero
// expires makes a token that never expires. The token is CBOR-encoded,
// encrypted with NaCl secretbox, and base64url-encoded.
func (g *Generator) Generate(
req *imagerequest.Request, expires time.Time,
) (string, error) {
p := &payload{
SourceHost: req.Source.Host,
SourcePath: req.Source.Path,
SourceQuery: req.Source.Query,
Width: req.Transform.Size.Width,
Height: req.Transform.Size.Height,
Format: req.Transform.Format,
Quality: req.Transform.Quality,
FitMode: req.Transform.FitMode,
}
if !expires.IsZero() {
p.ExpiresAt = expires.Unix()
}
// CBOR encode the payload // CBOR encode the payload
data, err := cbor.Marshal(p) data, err := cbor.Marshal(p)
if err != nil { if err != nil {
@@ -72,40 +84,35 @@ func (g *Generator) Generate(p *Payload) (string, error) {
return seal.Encrypt(g.key, data) return seal.Encrypt(g.key, data)
} }
// Parse decrypts and validates an encrypted URL token. // Parse decrypts and validates an encrypted URL token and returns the
// request it holds, with default values for omitted optional fields, and its
// expiry. A token that never expires gives the zero time.
// Returns ErrExpired if the token has expired, or other errors for invalid tokens. // Returns ErrExpired if the token has expired, or other errors for invalid tokens.
func (g *Generator) Parse(token string) (*Payload, error) { func (g *Generator) Parse(token string) (*imagerequest.Request, time.Time, error) {
// Decrypt // Decrypt
data, err := seal.Decrypt(g.key, token) data, err := seal.Decrypt(g.key, token)
if err != nil { if err != nil {
if errors.Is(err, seal.ErrDecryptionFailed) || if errors.Is(err, seal.ErrDecryptionFailed) ||
errors.Is(err, seal.ErrInvalidPayload) { errors.Is(err, seal.ErrInvalidPayload) {
return nil, ErrDecryptFailed return nil, time.Time{}, ErrDecryptFailed
} }
return nil, err return nil, time.Time{}, err
} }
// CBOR decode // CBOR decode
var p Payload var p payload
err = cbor.Unmarshal(data, &p) err = cbor.Unmarshal(data, &p)
if err != nil { if err != nil {
return nil, ErrInvalidFormat return nil, time.Time{}, ErrInvalidFormat
} }
// Check expiration (0 = never expires) // Check expiration (0 = never expires)
if p.ExpiresAt != 0 && time.Now().Unix() > p.ExpiresAt { if p.ExpiresAt != 0 && time.Now().Unix() > p.ExpiresAt {
return nil, ErrExpired return nil, time.Time{}, ErrExpired
} }
return &p, nil
}
// ToImageRequest converts the payload to an ImageRequest.
// Applies default values for omitted optional fields. An ExpiresAt of 0, a URL
// that never expires, gives the zero Expires.
func (p *Payload) ToImageRequest() *imgcache.ImageRequest {
format := p.Format format := p.Format
if format == "" { if format == "" {
format = DefaultFormat format = DefaultFormat
@@ -126,44 +133,17 @@ func (p *Payload) ToImageRequest() *imgcache.ImageRequest {
expires = time.Unix(p.ExpiresAt, 0) expires = time.Unix(p.ExpiresAt, 0)
} }
return &imgcache.ImageRequest{ return &imagerequest.Request{
SourceHost: p.SourceHost, Source: imagerequest.Source{
SourcePath: p.SourcePath, Host: p.SourceHost,
SourceQuery: p.SourceQuery, Path: p.SourcePath,
Size: imgcache.Size{ Query: p.SourceQuery,
Width: p.Width,
Height: p.Height,
}, },
Format: format, Transform: imagerequest.Transform{
Quality: quality, Size: imagerequest.Size{Width: p.Width, Height: p.Height},
FitMode: fitMode, Format: format,
Expires: expires, Quality: quality,
} FitMode: fitMode,
} },
}, expires, nil
// FromImageRequest creates a Payload from an ImageRequest with the given expiration.
func FromImageRequest(req *imgcache.ImageRequest, expiresAt time.Time) *Payload {
p := &Payload{
SourceHost: req.SourceHost,
SourcePath: req.SourcePath,
SourceQuery: req.SourceQuery,
Width: req.Size.Width,
Height: req.Size.Height,
ExpiresAt: expiresAt.Unix(),
}
// Only set non-default values to benefit from omitempty
if req.Format != "" && req.Format != DefaultFormat {
p.Format = req.Format
}
if req.Quality != 0 && req.Quality != DefaultQuality {
p.Quality = req.Quality
}
if req.FitMode != "" && req.FitMode != DefaultFitMode {
p.FitMode = req.FitMode
}
return p
} }
internal/encurl/encurl_test.go
+110 -188
View File
@@ -6,7 +6,8 @@ import (
"time" "time"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
) )
// Shared test fixture strings. // Shared test fixture strings.
@@ -19,24 +20,29 @@ const (
func TestGenerator_GenerateAndParse(t *testing.T) { func TestGenerator_GenerateAndParse(t *testing.T) {
t.Parallel() t.Parallel()
gen, err := encurl.NewGenerator("test-signing-key-12345") keys, err := seal.DeriveKeys("test-signing-key-12345")
if err != nil { if err != nil {
t.Fatalf("NewGenerator() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
payload := &encurl.Payload{ gen := encurl.NewGenerator(keys.EncryptedURLKey)
SourceHost: testSourceHost,
SourcePath: testSourcePath,
SourceQuery: testSourceQuery,
Width: 800,
Height: 600,
Format: imgcache.FormatWebP,
Quality: 90,
FitMode: imgcache.FitContain,
ExpiresAt: time.Now().Add(time.Hour).Unix(),
}
token, err := gen.Generate(payload) req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
Path: testSourcePath,
Query: testSourceQuery,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: imagerequest.FormatWebP,
Quality: 90,
FitMode: imagerequest.FitContain,
},
}
expires := time.Now().Add(time.Hour)
token, err := gen.Generate(req, expires)
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
@@ -45,66 +51,73 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
t.Fatal("Generate() returned empty token") t.Fatal("Generate() returned empty token")
} }
parsed, err := gen.Parse(token) parsed, parsedExpires, err := gen.Parse(token)
if err != nil { if err != nil {
t.Fatalf("Parse() error = %v", err) t.Fatalf("Parse() error = %v", err)
} }
// Verify all fields // Verify all fields
if parsed.SourceHost != payload.SourceHost { if parsed.Source.Host != req.Source.Host {
t.Errorf("SourceHost = %q, want %q", parsed.SourceHost, payload.SourceHost) t.Errorf("SourceHost = %q, want %q", parsed.Source.Host, req.Source.Host)
} }
if parsed.SourcePath != payload.SourcePath { if parsed.Source.Path != req.Source.Path {
t.Errorf("SourcePath = %q, want %q", parsed.SourcePath, payload.SourcePath) t.Errorf("SourcePath = %q, want %q", parsed.Source.Path, req.Source.Path)
} }
if parsed.SourceQuery != payload.SourceQuery { if parsed.Source.Query != req.Source.Query {
t.Errorf("SourceQuery = %q, want %q", parsed.SourceQuery, payload.SourceQuery) t.Errorf("SourceQuery = %q, want %q", parsed.Source.Query, req.Source.Query)
} }
if parsed.Width != payload.Width { if parsed.Transform.Size.Width != req.Transform.Size.Width {
t.Errorf("Width = %d, want %d", parsed.Width, payload.Width) t.Errorf("Width = %d, want %d",
parsed.Transform.Size.Width, req.Transform.Size.Width)
} }
if parsed.Height != payload.Height { if parsed.Transform.Size.Height != req.Transform.Size.Height {
t.Errorf("Height = %d, want %d", parsed.Height, payload.Height) t.Errorf("Height = %d, want %d",
parsed.Transform.Size.Height, req.Transform.Size.Height)
} }
if parsed.Format != payload.Format { if parsed.Transform.Format != req.Transform.Format {
t.Errorf("Format = %q, want %q", parsed.Format, payload.Format) t.Errorf("Format = %q, want %q",
parsed.Transform.Format, req.Transform.Format)
} }
if parsed.Quality != payload.Quality { if parsed.Transform.Quality != req.Transform.Quality {
t.Errorf("Quality = %d, want %d", parsed.Quality, payload.Quality) t.Errorf("Quality = %d, want %d",
parsed.Transform.Quality, req.Transform.Quality)
} }
if parsed.FitMode != payload.FitMode { if parsed.Transform.FitMode != req.Transform.FitMode {
t.Errorf("FitMode = %q, want %q", parsed.FitMode, payload.FitMode) t.Errorf("FitMode = %q, want %q",
parsed.Transform.FitMode, req.Transform.FitMode)
} }
if parsed.ExpiresAt != payload.ExpiresAt { if parsedExpires.Unix() != expires.Unix() {
t.Errorf("ExpiresAt = %d, want %d", parsed.ExpiresAt, payload.ExpiresAt) t.Errorf("ExpiresAt = %d, want %d", parsedExpires.Unix(), expires.Unix())
} }
} }
func TestGenerator_Parse_Expired(t *testing.T) { func TestGenerator_Parse_Expired(t *testing.T) {
t.Parallel() t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
payload := &encurl.Payload{ req := &imagerequest.Request{
SourceHost: testSourceHost, Source: imagerequest.Source{
SourcePath: testSourcePath, Host: testSourceHost,
ExpiresAt: time.Now().Add(-time.Hour).Unix(), // Already expired Path: testSourcePath,
},
} }
token, err := gen.Generate(payload) token, err := gen.Generate(req, time.Now().Add(-time.Hour)) // Already expired
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
_, err = gen.Parse(token) _, _, err = gen.Parse(token)
if err == nil { if err == nil {
t.Error("Parse() should fail for expired token") t.Error("Parse() should fail for expired token")
} }
@@ -117,9 +130,10 @@ func TestGenerator_Parse_Expired(t *testing.T) {
func TestGenerator_Parse_InvalidToken(t *testing.T) { func TestGenerator_Parse_InvalidToken(t *testing.T) {
t.Parallel() t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
_, err := gen.Parse("not-a-valid-token") _, _, err := gen.Parse("not-a-valid-token")
if err == nil { if err == nil {
t.Error("Parse() should fail for invalid token") t.Error("Parse() should fail for invalid token")
} }
@@ -128,15 +142,17 @@ func TestGenerator_Parse_InvalidToken(t *testing.T) {
func TestGenerator_Parse_TamperedToken(t *testing.T) { func TestGenerator_Parse_TamperedToken(t *testing.T) {
t.Parallel() t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
payload := &encurl.Payload{ req := &imagerequest.Request{
SourceHost: testSourceHost, Source: imagerequest.Source{
SourcePath: testSourcePath, Host: testSourceHost,
ExpiresAt: time.Now().Add(time.Hour).Unix(), Path: testSourcePath,
},
} }
token, _ := gen.Generate(payload) token, _ := gen.Generate(req, time.Now().Add(time.Hour))
// Tamper with the token // Tamper with the token
tampered := []byte(token) tampered := []byte(token)
@@ -144,7 +160,7 @@ func TestGenerator_Parse_TamperedToken(t *testing.T) {
tampered[10] ^= 0x01 tampered[10] ^= 0x01
} }
_, err := gen.Parse(string(tampered)) _, _, err := gen.Parse(string(tampered))
if err == nil { if err == nil {
t.Error("Parse() should fail for tampered token") t.Error("Parse() should fail for tampered token")
} }
@@ -153,172 +169,78 @@ func TestGenerator_Parse_TamperedToken(t *testing.T) {
func TestGenerator_Parse_WrongKey(t *testing.T) { func TestGenerator_Parse_WrongKey(t *testing.T) {
t.Parallel() t.Parallel()
gen1, _ := encurl.NewGenerator("signing-key-1") keys1, _ := seal.DeriveKeys("signing-key-1")
gen2, _ := encurl.NewGenerator("signing-key-2") keys2, _ := seal.DeriveKeys("signing-key-2")
gen1 := encurl.NewGenerator(keys1.EncryptedURLKey)
gen2 := encurl.NewGenerator(keys2.EncryptedURLKey)
payload := &encurl.Payload{ req := &imagerequest.Request{
SourceHost: testSourceHost, Source: imagerequest.Source{
SourcePath: testSourcePath, Host: testSourceHost,
ExpiresAt: time.Now().Add(time.Hour).Unix(), Path: testSourcePath,
},
} }
token, _ := gen1.Generate(payload) token, _ := gen1.Generate(req, time.Now().Add(time.Hour))
_, err := gen2.Parse(token) _, _, err := gen2.Parse(token)
if err == nil { if err == nil {
t.Error("Parse() should fail with different signing key") t.Error("Parse() should fail with different signing key")
} }
} }
func TestPayload_ToImageRequest(t *testing.T) { func TestGenerator_Parse_Defaults(t *testing.T) {
t.Parallel() t.Parallel()
payload := &encurl.Payload{ keys, _ := seal.DeriveKeys("test-signing-key-12345")
SourceHost: testSourceHost, gen := encurl.NewGenerator(keys.EncryptedURLKey)
SourcePath: testSourcePath,
SourceQuery: testSourceQuery, // Request with only required fields - should get defaults
Width: 800, token, err := gen.Generate(&imagerequest.Request{
Height: 600, Source: imagerequest.Source{
Format: imgcache.FormatWebP, Host: testSourceHost,
Quality: 90, Path: testSourcePath,
FitMode: imgcache.FitContain, },
ExpiresAt: time.Now().Add(time.Hour).Unix(), }, time.Now().Add(time.Hour))
if err != nil {
t.Fatalf("Generate() error = %v", err)
} }
req := payload.ToImageRequest() req, _, err := gen.Parse(token)
if err != nil {
if req.SourceHost != payload.SourceHost { t.Fatalf("Parse() error = %v", err)
t.Errorf("SourceHost = %q, want %q", req.SourceHost, payload.SourceHost)
} }
if req.SourcePath != payload.SourcePath { if req.Transform.Format != encurl.DefaultFormat {
t.Errorf("SourcePath = %q, want %q", req.SourcePath, payload.SourcePath) t.Errorf("Format = %q, want default %q",
req.Transform.Format, encurl.DefaultFormat)
} }
if req.SourceQuery != payload.SourceQuery { if req.Transform.Quality != encurl.DefaultQuality {
t.Errorf("SourceQuery = %q, want %q", req.SourceQuery, payload.SourceQuery) t.Errorf("Quality = %d, want default %d",
req.Transform.Quality, encurl.DefaultQuality)
} }
if req.Size.Width != payload.Width { if req.Transform.FitMode != encurl.DefaultFitMode {
t.Errorf("Width = %d, want %d", req.Size.Width, payload.Width) t.Errorf("FitMode = %q, want default %q",
} req.Transform.FitMode, encurl.DefaultFitMode)
if req.Size.Height != payload.Height {
t.Errorf("Height = %d, want %d", req.Size.Height, payload.Height)
}
if req.Format != payload.Format {
t.Errorf("Format = %q, want %q", req.Format, payload.Format)
}
if req.Quality != payload.Quality {
t.Errorf("Quality = %d, want %d", req.Quality, payload.Quality)
}
if req.FitMode != payload.FitMode {
t.Errorf("FitMode = %q, want %q", req.FitMode, payload.FitMode)
}
}
func TestPayload_ToImageRequest_Defaults(t *testing.T) {
t.Parallel()
// Payload with only required fields - should get defaults
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
ExpiresAt: time.Now().Add(time.Hour).Unix(),
}
req := payload.ToImageRequest()
if req.Format != encurl.DefaultFormat {
t.Errorf("Format = %q, want default %q", req.Format, encurl.DefaultFormat)
}
if req.Quality != encurl.DefaultQuality {
t.Errorf("Quality = %d, want default %d", req.Quality, encurl.DefaultQuality)
}
if req.FitMode != encurl.DefaultFitMode {
t.Errorf("FitMode = %q, want default %q", req.FitMode, encurl.DefaultFitMode)
}
}
func TestFromImageRequest(t *testing.T) {
t.Parallel()
req := &imgcache.ImageRequest{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
SourceQuery: testSourceQuery,
Size: imgcache.Size{Width: 800, Height: 600},
Format: imgcache.FormatWebP,
Quality: 90,
FitMode: imgcache.FitContain,
}
expiresAt := time.Now().Add(time.Hour)
payload := encurl.FromImageRequest(req, expiresAt)
if payload.SourceHost != req.SourceHost {
t.Errorf("SourceHost = %q, want %q", payload.SourceHost, req.SourceHost)
}
if payload.SourcePath != req.SourcePath {
t.Errorf("SourcePath = %q, want %q", payload.SourcePath, req.SourcePath)
}
if payload.Width != req.Size.Width {
t.Errorf("Width = %d, want %d", payload.Width, req.Size.Width)
}
if payload.ExpiresAt != expiresAt.Unix() {
t.Errorf("ExpiresAt = %d, want %d", payload.ExpiresAt, expiresAt.Unix())
}
}
func TestFromImageRequest_OmitsDefaults(t *testing.T) {
t.Parallel()
// Request with default values - payload should omit them for
// smaller encoding
req := &imgcache.ImageRequest{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
Format: encurl.DefaultFormat,
Quality: encurl.DefaultQuality,
FitMode: encurl.DefaultFitMode,
}
payload := encurl.FromImageRequest(req, time.Now().Add(time.Hour))
// These should be zero/empty because they match defaults
if payload.Format != "" {
t.Errorf("Format should be empty for default, got %q", payload.Format)
}
if payload.Quality != 0 {
t.Errorf("Quality should be 0 for default, got %d", payload.Quality)
}
if payload.FitMode != "" {
t.Errorf("FitMode should be empty for default, got %q", payload.FitMode)
} }
} }
func TestGenerator_TokenIsURLSafe(t *testing.T) { func TestGenerator_TokenIsURLSafe(t *testing.T) {
t.Parallel() t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
payload := &encurl.Payload{ req := &imagerequest.Request{
SourceHost: testSourceHost, Source: imagerequest.Source{
SourcePath: testSourcePath, Host: testSourceHost,
ExpiresAt: time.Now().Add(time.Hour).Unix(), Path: testSourcePath,
},
} }
token, _ := gen.Generate(payload) token, _ := gen.Generate(req, time.Now().Add(time.Hour))
// Check that token only contains URL-safe characters // Check that token only contains URL-safe characters
for _, c := range token { for _, c := range token {
internal/handlers/auth.go
+105 -35
View File
@@ -9,12 +9,14 @@ import (
"math" "math"
"net/http" "net/http"
"net/url" "net/url"
"slices"
"strconv" "strconv"
"strings"
"time" "time"
"sneak.berlin/go/pixa/internal/clientip" "sneak.berlin/go/pixa/internal/clientip"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/templates" "sneak.berlin/go/pixa/internal/templates"
) )
@@ -89,10 +91,18 @@ func (s *Handlers) handleLoginPost(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/", http.StatusSeeOther) http.Redirect(w, r, "/", http.StatusSeeOther)
} }
// HandleLogout clears the session and redirects to login. // HandleLogout logs out, which ends every login session (see
// session.Manager.Logout), and redirects to login.
func (s *Handlers) HandleLogout() http.HandlerFunc { func (s *Handlers) HandleLogout() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
s.sessMgr.ClearSession(w) err := s.sessMgr.Logout(w, r)
if err != nil {
s.log.Error("failed to log out", "error", err)
http.Error(w, "Failed to log out", http.StatusInternalServerError)
return
}
http.Redirect(w, r, "/", http.StatusSeeOther) http.Redirect(w, r, "/", http.StatusSeeOther)
} }
} }
@@ -117,7 +127,7 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
sourceURL := r.FormValue("url") sourceURL := r.FormValue("url")
// Validate source URL // Validate source URL
parsed, err := url.Parse(sourceURL) parsed, err := parseSourceURL(sourceURL)
if err != nil || parsed.Host == "" { if err != nil || parsed.Host == "" {
s.renderGeneratorWithForm(w, r, "Invalid source URL", r.Form, s.renderGeneratorWithForm(w, r, "Invalid source URL", r.Form,
http.StatusBadRequest) http.StatusBadRequest)
@@ -125,7 +135,17 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
return return
} }
payload, expiresAt, ttl, err := buildGeneratePayload(parsed, r.Form) // /v1/image/ refuses a ".." segment in the source path, and so does
// the generator. The path is decoded, so %2e%2e counts too
if slices.Contains(strings.Split(parsed.Path, "/"), "..") {
s.renderGeneratorWithForm(w, r,
"Source URL path cannot hold a .. segment", r.Form,
http.StatusBadRequest)
return
}
req, expiresAt, ttl, err := buildGenerateRequest(parsed, r.Form)
if err != nil { if err != nil {
s.renderGeneratorWithForm(w, r, err.Error(), r.Form, s.renderGeneratorWithForm(w, r, err.Error(), r.Form,
http.StatusBadRequest) http.StatusBadRequest)
@@ -133,8 +153,20 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
return return
} }
// The image URL must fetch the source URL as it was written. The
// fetcher fetches over http only when allow_http is set.
fetchURL := req.Source.URL(s.config.AllowHTTP)
if !fetchedAsWritten(parsed, fetchURL) {
s.renderGeneratorWithForm(w, r, "Source URL cannot be fetched as "+
"written: pixa would fetch "+fetchURL.String(),
r.Form, http.StatusBadRequest)
return
}
// Generate encrypted token // Generate encrypted token
token, err := s.encGen.Generate(payload) token, err := s.encGen.Generate(req, expiresAt)
if err != nil { if err != nil {
s.log.Error("failed to generate encrypted URL", "error", err) s.log.Error("failed to generate encrypted URL", "error", err)
s.renderGeneratorWithForm(w, r, "Failed to generate URL", r.Form, s.renderGeneratorWithForm(w, r, "Failed to generate URL", r.Form,
@@ -165,14 +197,55 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
} }
} }
// buildGeneratePayload parses the numeric form fields and assembles the // parseSourceURL parses the source URL entered on the generator page with its
// encrypted URL payload. ttl=0 means never expires (ExpiresAt stays 0). A // host lower-cased and an empty path written as "/", as /v1/image/ reads a
// non-numeric or out-of-range width, height, quality or ttl, or an // source URL, so that each spelling of one source URL makes one cache key and
// unrecognized fit mode, is a client error naming the offending field. The // one upstream request.
// format field is passed through unchecked. func parseSourceURL(sourceURL string) (*url.URL, error) {
func buildGeneratePayload( parsed, err := url.Parse(sourceURL)
if err != nil {
return nil, err
}
parsed.Host = strings.ToLower(parsed.Host)
if parsed.Path == "" {
parsed.Path = "/"
}
return parsed, nil
}
// fetchedAsWritten reports whether fetchURL, the URL pixa fetches for an
// image URL made from the source URL source, is source as it was written,
// apart from the scheme, which the token does not keep. It is not when source
// holds a user name and password, which the token does not keep either, or a
// "?" with no query after it. The token keeps the source path decoded and
// pixa fetches it percent-encoded again, so a path written another way, such
// as with %2F, is fetched as another path. url.URL keeps the path as written
// in RawPath when it is not the decoded path percent-encoded again.
// EscapedPath ignores RawPath when it also holds a character net/url encodes,
// such as a space, so RawPath is compared itself.
func fetchedAsWritten(source, fetchURL *url.URL) bool {
writtenPath := source.RawPath
if writtenPath == "" {
writtenPath = source.EscapedPath()
}
return source.User == nil && !source.ForceQuery &&
fetchURL.EscapedPath() == writtenPath &&
fetchURL.RawQuery == source.RawQuery
}
// buildGenerateRequest parses the numeric form fields and assembles the
// image request the encrypted URL's token is made from, and its expiry.
// ttl=0 means never expires (the expiry is the zero time). A non-numeric or
// out-of-range width, height, quality or ttl, or an unrecognized fit mode, is
// a client error naming the offending field. The format field is passed
// through unchecked.
func buildGenerateRequest(
parsed *url.URL, form url.Values, parsed *url.URL, form url.Values,
) (*encurl.Payload, time.Time, int, error) { ) (*imagerequest.Request, time.Time, int, error) {
width, err := parseFormDimension(form, "width") width, err := parseFormDimension(form, "width")
if err != nil { if err != nil {
return nil, time.Time{}, 0, err return nil, time.Time{}, 0, err
@@ -194,42 +267,39 @@ func buildGeneratePayload(
return nil, time.Time{}, 0, err return nil, time.Time{}, 0, err
} }
fitMode := imgcache.FitMode(form.Get("fit")) fitMode := imagerequest.Fit(form.Get("fit"))
err = imgcache.ValidateFitMode(fitMode) err = imagerequest.ValidateFitMode(fitMode)
if err != nil { if err != nil {
return nil, time.Time{}, 0, return nil, time.Time{}, 0,
fmt.Errorf("%w: %s", imgcache.ErrInvalidFitMode, form.Get("fit")) fmt.Errorf("%w: %s", imagerequest.ErrInvalidFitMode, form.Get("fit"))
} }
var ( var expiresAt time.Time
expiresAt time.Time
expiresAtUnix int64
)
if ttl > 0 { if ttl > 0 {
expiresAt = time.Now().Add(time.Duration(ttl) * time.Second) expiresAt = time.Now().Add(time.Duration(ttl) * time.Second)
expiresAtUnix = expiresAt.Unix()
} }
payload := &encurl.Payload{ req := &imagerequest.Request{
SourceHost: parsed.Host, Source: imagerequest.Source{
SourcePath: parsed.Path, Host: parsed.Host,
SourceQuery: parsed.RawQuery, Path: parsed.Path,
Width: width, Query: parsed.RawQuery,
Height: height, },
Format: imgcache.ImageFormat(form.Get("format")), Transform: imagerequest.Transform{
Quality: quality, Size: imagerequest.Size{Width: width, Height: height},
FitMode: fitMode, Format: imagerequest.Format(form.Get("format")),
ExpiresAt: expiresAtUnix, Quality: quality,
FitMode: fitMode,
},
} }
return payload, expiresAt, ttl, nil return req, expiresAt, ttl, nil
} }
// parseFormDimension reads an optional width or height form field. An empty // parseFormDimension reads an optional width or height form field. An empty
// value means "original size" (0). A non-numeric value, or one // value means "original size" (0). A non-numeric value, or one
// imgcache.ValidateDimension rejects, is an error naming the field. // imagerequest.ValidateDimension rejects, is an error naming the field.
func parseFormDimension(form url.Values, field string) (int, error) { func parseFormDimension(form url.Values, field string) (int, error) {
raw := form.Get(field) raw := form.Get(field)
if raw == "" { if raw == "" {
@@ -241,7 +311,7 @@ func parseFormDimension(form url.Values, field string) (int, error) {
return 0, fmt.Errorf("%w %s: not a number", errInvalidFormField, field) return 0, fmt.Errorf("%w %s: not a number", errInvalidFormField, field)
} }
err = imgcache.ValidateDimension(field, value) err = imagerequest.ValidateDimension(field, value)
if err != nil { if err != nil {
return 0, err return 0, err
} }
internal/handlers/auth_csrf_internal_test.go
+6 -15
View File
@@ -14,6 +14,7 @@ import (
"sneak.berlin/go/pixa/internal/config" "sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
@@ -44,27 +45,17 @@ func newCSRFTestRouter(t *testing.T) (*Handlers, http.Handler) {
cfg := &config.Config{SigningKey: testSigningKey, Debug: true} cfg := &config.Config{SigningKey: testSigningKey, Debug: true}
sessMgr, err := session.NewManager(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("session.NewManager() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
protect, err := newCSRFProtect(testSigningKey, cfg.Debug)
if err != nil {
t.Fatalf("newCSRFProtect() error = %v", err)
} }
h := &Handlers{ h := &Handlers{
log: slog.New(slog.DiscardHandler), log: slog.New(slog.DiscardHandler),
config: cfg, config: cfg,
sessMgr: sessMgr, sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
encGen: encGen, encGen: encurl.NewGenerator(keys.EncryptedURLKey),
csrfProtect: protect, csrfProtect: newCSRFProtect(keys.CSRFKey, cfg.Debug, slog.New(slog.DiscardHandler)),
} }
r := chi.NewRouter() r := chi.NewRouter()
internal/handlers/auth_generate_internal_test.go
+9 -9
View File
@@ -11,7 +11,7 @@ import (
"time" "time"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imagerequest"
) )
// Generator form field names, and a value that is not a number. // Generator form field names, and a value that is not a number.
@@ -125,10 +125,10 @@ func TestGeneratePostRejectsBadField(t *testing.T) {
} }
} }
// TestBuildGeneratePayloadDefaultAndLimits verifies that an empty quality // TestBuildGenerateRequestDefaultAndLimits verifies that an empty quality
// takes the default, and that the largest accepted width, height, quality and // takes the default, and that the largest accepted width, height, quality and
// ttl are accepted with an expiry still in the future. // ttl are accepted with an expiry still in the future.
func TestBuildGeneratePayloadDefaultAndLimits(t *testing.T) { func TestBuildGenerateRequestDefaultAndLimits(t *testing.T) {
t.Parallel() t.Parallel()
parsed, err := url.Parse(testSourceURL) parsed, err := url.Parse(testSourceURL)
@@ -136,19 +136,19 @@ func TestBuildGeneratePayloadDefaultAndLimits(t *testing.T) {
t.Fatalf("url.Parse() error = %v", err) t.Fatalf("url.Parse() error = %v", err)
} }
payload, _, _, err := buildGeneratePayload(parsed, url.Values{}) req, _, _, err := buildGenerateRequest(parsed, url.Values{})
if err != nil { if err != nil {
t.Fatalf("empty form: error = %v", err) t.Fatalf("empty form: error = %v", err)
} }
if payload.Quality != encurl.DefaultQuality { if req.Transform.Quality != encurl.DefaultQuality {
t.Errorf("empty quality gave %d, want %d", t.Errorf("empty quality gave %d, want %d",
payload.Quality, encurl.DefaultQuality) req.Transform.Quality, encurl.DefaultQuality)
} }
_, expiresAt, _, err := buildGeneratePayload(parsed, url.Values{ _, expiresAt, _, err := buildGenerateRequest(parsed, url.Values{
widthField: {strconv.Itoa(imgcache.MaxDimension)}, widthField: {strconv.Itoa(imagerequest.MaxDimension)},
heightField: {strconv.Itoa(imgcache.MaxDimension)}, heightField: {strconv.Itoa(imagerequest.MaxDimension)},
qualityField: {strconv.Itoa(maxQuality)}, qualityField: {strconv.Itoa(maxQuality)},
ttlField: {strconv.Itoa(maxTTL)}, ttlField: {strconv.Itoa(maxTTL)},
}) })
internal/handlers/auth_login_log_internal_test.go
+4 -3
View File
@@ -10,6 +10,7 @@ import (
"testing" "testing"
"sneak.berlin/go/pixa/internal/config" "sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
@@ -22,15 +23,15 @@ func TestLoginLogLeavesOutSubmittedKey(t *testing.T) {
var buf bytes.Buffer var buf bytes.Buffer
sessMgr, err := session.NewManager(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("session.NewManager() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
h := &Handlers{ h := &Handlers{
log: slog.New(slog.NewJSONHandler(&buf, nil)), log: slog.New(slog.NewJSONHandler(&buf, nil)),
config: &config.Config{SigningKey: testSigningKey}, config: &config.Config{SigningKey: testSigningKey},
sessMgr: sessMgr, sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
} }
submittedKeys := []string{wrongKey, testSigningKey} submittedKeys := []string{wrongKey, testSigningKey}
internal/handlers/auth_session_internal_test.go
+101 -3
View File
@@ -10,7 +10,8 @@ import (
"testing" "testing"
"time" "time"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
@@ -185,6 +186,103 @@ func TestHandleLogout_ClearsSessionCookie(t *testing.T) {
} }
} }
// TestHandleLogout_EndsCopiedSession verifies that after POST /logout a copy
// of the session cookie taken before it shows the login page, not the
// generator page.
func TestHandleLogout_EndsCopiedSession(t *testing.T) {
t.Parallel()
h, srv := newCSRFTestRouter(t)
copied := newSessionCookie(t, h)
req := httptest.NewRequestWithContext(
t.Context(), http.MethodPost, "/logout", nil)
req.AddCookie(copied)
rec := httptest.NewRecorder()
h.HandleLogout().ServeHTTP(rec, req)
if rec.Code != http.StatusSeeOther {
t.Fatalf("POST /logout status = %d, want %d",
rec.Code, http.StatusSeeOther)
}
req = httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req.AddCookie(copied)
rec = httptest.NewRecorder()
srv.ServeHTTP(rec, req)
body := rec.Body.String()
t.Logf("GET / with the copied cookie: status %d, login page %v",
rec.Code, strings.Contains(body, loginForm))
if !strings.Contains(body, loginForm) || strings.Contains(body, generatorForm) {
t.Error("a cookie copied before /logout still opens the generator page")
}
}
// TestHandleLogout_SaveFails_Answers500 verifies that when the logout cannot
// be saved in the database, POST /logout answers 500, leaves the session
// cookie in place and ends no session.
func TestHandleLogout_SaveFails_Answers500(t *testing.T) {
t.Parallel()
h, srv := newCSRFTestRouter(t)
db := setupTestDB(t)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
sessMgr, err := session.NewManagerWithDatabase(
t.Context(), keys.SessionHashKey, keys.SessionBlockKey, db)
if err != nil {
t.Fatalf("session.NewManagerWithDatabase() error = %v", err)
}
h.sessMgr = sessMgr
cookie := newSessionCookie(t, h)
// Saving the logout fails on a closed database.
err = db.Close()
if err != nil {
t.Fatalf("closing the database: %v", err)
}
req := httptest.NewRequestWithContext(
t.Context(), http.MethodPost, "/logout", nil)
req.AddCookie(cookie)
rec := httptest.NewRecorder()
h.HandleLogout().ServeHTTP(rec, req)
t.Logf("POST /logout: status %d, Set-Cookie %q",
rec.Code, rec.Header().Get("Set-Cookie"))
if rec.Code != http.StatusInternalServerError {
t.Errorf("POST /logout status = %d, want %d",
rec.Code, http.StatusInternalServerError)
}
if c := findSessionCookie(rec); c != nil {
t.Errorf("a logout that failed set the session cookie: %s", c)
}
req = httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req.AddCookie(cookie)
rec = httptest.NewRecorder()
srv.ServeHTTP(rec, req)
if !strings.Contains(rec.Body.String(), generatorForm) {
t.Error("a logout that failed ended the session")
}
}
// TestGeneratePost_NoSession_RedirectsToLogin verifies that POST /generate // TestGeneratePost_NoSession_RedirectsToLogin verifies that POST /generate
// with a valid CSRF token but no login session answers 303 to / and makes no // with a valid CSRF token but no login session answers 303 to / and makes no
// URL. // URL.
@@ -222,7 +320,7 @@ func TestGeneratePost_URLServesImage(t *testing.T) {
sourceURLField: {"https://" + signedHost + photoPath}, sourceURLField: {"https://" + signedHost + photoPath},
widthField: {"50"}, widthField: {"50"},
heightField: {"50"}, heightField: {"50"},
formatField: {string(imgcache.FormatJPEG)}, formatField: {string(imagerequest.FormatJPEG)},
}) })
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
@@ -255,7 +353,7 @@ func TestGeneratePost_URLWithTTLExpires(t *testing.T) {
sourceURLField: {"https://" + signedHost + photoPath}, sourceURLField: {"https://" + signedHost + photoPath},
widthField: {"50"}, widthField: {"50"},
heightField: {"50"}, heightField: {"50"},
formatField: {string(imgcache.FormatJPEG)}, formatField: {string(imagerequest.FormatJPEG)},
ttlField: {"1"}, ttlField: {"1"},
}) })
internal/handlers/avif_too_large_internal_test.go
+39
View File
@@ -0,0 +1,39 @@
package handlers
import (
"fmt"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError, which
// answers the errors of both image routes, answers 400 naming the AVIF pixel
// limit when the image processor refuses an AVIF output above it, wrapped as
// the service wraps it. The request's source only goes into a log line, so it
// is left empty.
func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
err := fmt.Errorf("image processing failed: %w: 4097x4096",
imageprocessor.ErrAVIFOutputTooLarge)
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
rec := httptest.NewRecorder()
h.respondImageError(rec, &imagerequest.Request{}, err)
t.Logf("status = %d, body = %q", rec.Code, rec.Body.String())
if rec.Code != http.StatusBadRequest {
t.Errorf("status = %d, want %d", rec.Code, http.StatusBadRequest)
}
if !strings.Contains(rec.Body.String(), wantBody) {
t.Errorf("body = %q, want it to say %q", rec.Body.String(), wantBody)
}
}
internal/handlers/bodylimit_internal_test.go
+6 -15
View File
@@ -11,6 +11,7 @@ import (
"sneak.berlin/go/pixa/internal/config" "sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
@@ -32,27 +33,17 @@ func newBodyLimitTestRouter(
cfg := &config.Config{SigningKey: testSigningKey, Debug: true} cfg := &config.Config{SigningKey: testSigningKey, Debug: true}
sessMgr, err := session.NewManager(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("session.NewManager() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
protect, err := newCSRFProtect(testSigningKey, cfg.Debug)
if err != nil {
t.Fatalf("newCSRFProtect() error = %v", err)
} }
h := &Handlers{ h := &Handlers{
log: slog.New(slog.DiscardHandler), log: slog.New(slog.DiscardHandler),
config: cfg, config: cfg,
sessMgr: sessMgr, sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
encGen: encGen, encGen: encurl.NewGenerator(keys.EncryptedURLKey),
csrfProtect: protect, csrfProtect: newCSRFProtect(keys.CSRFKey, cfg.Debug, slog.New(slog.DiscardHandler)),
} }
r := chi.NewRouter() r := chi.NewRouter()
internal/handlers/cache_hit_internal_test.go
+278
View File
@@ -0,0 +1,278 @@
package handlers
import (
"bytes"
"context"
"database/sql"
"database/sql/driver"
"image"
"image/jpeg"
"log/slog"
"net/http"
"net/http/httptest"
"path/filepath"
"sync/atomic"
"testing"
"testing/fstest"
"time"
"github.com/go-chi/chi/v5"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
)
// countingHost is the allowlisted host whose photoPath the server of
// newCountingDatabaseServer serves.
const countingHost = "counting.example.com"
// countingConnector opens connections to the SQLite database file name with
// sqliteDriver, each one counting in statements the statements run on it.
type countingConnector struct {
sqliteDriver driver.Driver
name string
statements *atomic.Int64
}
func (c countingConnector) Connect(context.Context) (driver.Conn, error) {
conn, err := c.sqliteDriver.Open(c.name)
if err != nil {
return nil, err
}
return countingConn{Conn: conn, statements: c.statements}, nil
}
func (c countingConnector) Driver() driver.Driver {
return c.sqliteDriver
}
// countingConn counts the statements run on its connection. It has only the
// methods of driver.Conn, so database/sql prepares every query and every
// other statement with Prepare, which counts it.
type countingConn struct {
driver.Conn
statements *atomic.Int64
}
func (c countingConn) Prepare(query string) (driver.Stmt, error) {
c.statements.Add(1)
return c.Conn.Prepare(query)
}
// newCountingDatabase returns a database with pixa's schema, in a file of
// its own and with one connection, and the count of the statements run on
// it once the schema is in place.
func newCountingDatabase(tb testing.TB) (*sql.DB, *atomic.Int64) {
tb.Helper()
path := filepath.Join(tb.TempDir(), "state.sqlite3")
schemaDB, err := sql.Open("sqlite", path)
if err != nil {
tb.Fatalf("opening the database: %v", err)
}
defer func() { _ = schemaDB.Close() }()
err = database.ApplyMigrations(tb.Context(), schemaDB, nil)
if err != nil {
tb.Fatalf("applying the migrations: %v", err)
}
statements := new(atomic.Int64)
db := sql.OpenDB(countingConnector{
sqliteDriver: schemaDB.Driver(),
name: path,
statements: statements,
})
db.SetMaxOpenConns(1)
tb.Cleanup(func() { _ = db.Close() })
return db, statements
}
// newCountingDatabaseServer returns a router for both image routes whose
// image cache uses newCountingDatabase, the count of the statements run on
// that database, and a URL on each route for a different image of
// photoPath on countingHost. Nothing writes the cache's access times in the
// background, so every statement counted comes from a request.
func newCountingDatabaseServer(
tb testing.TB,
) (http.Handler, *atomic.Int64, []string) {
tb.Helper()
db, statements := newCountingDatabase(tb)
var photo bytes.Buffer
err := jpeg.Encode(&photo, image.NewRGBA(image.Rect(0, 0, 100, 100)), nil)
if err != nil {
tb.Fatalf("encoding the photo: %v", err)
}
cache, err := imgcache.NewCache(db, imgcache.Config{
StateDir: tb.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
if err != nil {
tb.Fatalf("imgcache.NewCache() error = %v", err)
}
svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache,
Fetcher: newMockFetcher(fstest.MapFS{
countingHost + photoPath: &fstest.MapFile{Data: photo.Bytes()},
}),
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
Logger: slog.New(slog.DiscardHandler),
}, prometheus.NewRegistry())
if err != nil {
tb.Fatalf("imageproxy.New() error = %v", err)
}
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
tb.Fatalf("seal.DeriveKeys() error = %v", err)
}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
token, err := encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: countingHost, Path: photoPath},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 40, Height: 40},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}, time.Time{})
if err != nil {
tb.Fatalf("encGen.Generate() error = %v", err)
}
h := &Handlers{
log: slog.New(slog.DiscardHandler),
imgSvc: svc,
encGen: encGen,
signer: signature.New(testSigningKey),
allowlist: allowlist.New([]string{countingHost}),
}
r := chi.NewRouter()
r.Get("/v1/image/*", h.HandleImage())
r.Get("/v1/e/{token}/*", h.HandleImageEnc())
return r, statements, []string{
"/v1/image/" + countingHost + photoPath + "/50x50.jpeg",
"/v1/e/" + token + "/img.jpg",
}
}
// serveImage sends srv a GET for target and returns the response, failing
// the test unless it is a 200 with an image.
func serveImage(
tb testing.TB, srv http.Handler, target string,
) *httptest.ResponseRecorder {
tb.Helper()
rec := httptest.NewRecorder()
srv.ServeHTTP(rec,
httptest.NewRequestWithContext(tb.Context(), http.MethodGet, target, nil))
if rec.Code != http.StatusOK || rec.Body.Len() == 0 {
tb.Errorf("GET %s: %d with %d bytes, want 200 with an image",
target, rec.Code, rec.Body.Len())
}
return rec
}
// TestImageRoutes_CacheHitsMakeNoDatabaseCall serves an image on each image
// route once, which makes and caches it, then serves it again from the cache.
// Serving those cache hits must run no statement on the database.
func TestImageRoutes_CacheHitsMakeNoDatabaseCall(t *testing.T) {
t.Parallel()
srv, statements, targets := newCountingDatabaseServer(t)
for _, target := range targets {
before := statements.Load()
serveImage(t, srv, target)
afterMiss := statements.Load()
t.Logf("GET %s, a miss: %d statements", target, afterMiss-before)
if afterMiss == before {
t.Fatalf("GET %s, a miss: no statement counted, want some", target)
}
for range 3 {
rec := serveImage(t, srv, target)
cacheStatus := rec.Header().Get("X-Pixa-Cache")
if cacheStatus != string(imageproxy.CacheHit) {
t.Errorf("GET %s: X-Pixa-Cache = %q, want %s",
target, cacheStatus, imageproxy.CacheHit)
}
}
if hitStatements := statements.Load() - afterMiss; hitStatements != 0 {
t.Errorf("GET %s: 3 cache hits ran %d statements, want 0",
target, hitStatements)
}
}
}
// BenchmarkHandleImage_CacheHits serves one cached image through the
// /v1/image/ route from b.RunParallel's goroutines, and reports the cache
// hits served per second. It fails if serving them ran any statement on the
// database. script/bench runs it.
func BenchmarkHandleImage_CacheHits(b *testing.B) {
srv, statements, targets := newCountingDatabaseServer(b)
target := targets[0]
serveImage(b, srv, target)
before := statements.Load()
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
rec := serveImage(b, srv, target)
cacheStatus := rec.Header().Get("X-Pixa-Cache")
if cacheStatus != string(imageproxy.CacheHit) {
b.Errorf("GET %s: X-Pixa-Cache = %q, want %s",
target, cacheStatus, imageproxy.CacheHit)
return
}
}
})
b.ReportMetric(float64(b.N)/b.Elapsed().Seconds(), "hits/s")
if hitStatements := statements.Load() - before; hitStatements != 0 {
b.Fatalf("%d cache hits ran %d statements, want 0", b.N, hitStatements)
}
}
internal/handlers/cache_max_bytes_internal_test.go
-147
View File
@@ -1,147 +0,0 @@
package handlers
import (
"os"
"path/filepath"
"testing"
"go.uber.org/fx/fxtest"
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/globals"
"sneak.berlin/go/pixa/internal/logger"
)
// TestNewCacheConfigFromCacheMaxBytes checks the cache configuration
// built from cache_max_bytes: omitted, the cache works out the default
// limit; 0 turns the disk cache off; a positive value is the limit,
// unchanged.
func TestNewCacheConfigFromCacheMaxBytes(t *testing.T) {
t.Parallel()
const oneGiB = 1 << 30
cases := []struct {
name string
cacheMaxBytes int64
cacheMaxBytesExplicit bool
wantMaxBytes int64
wantUseDefaultMaxBytes bool
wantDisableDiskCache bool
}{
{
name: "cache_max_bytes omitted",
cacheMaxBytes: 0,
cacheMaxBytesExplicit: false,
wantMaxBytes: 0,
wantUseDefaultMaxBytes: true,
wantDisableDiskCache: false,
},
{
name: "cache_max_bytes: 0",
cacheMaxBytes: 0,
cacheMaxBytesExplicit: true,
wantMaxBytes: 0,
wantUseDefaultMaxBytes: false,
wantDisableDiskCache: true,
},
{
name: "cache_max_bytes: 1 GiB",
cacheMaxBytes: oneGiB,
cacheMaxBytesExplicit: true,
wantMaxBytes: oneGiB,
wantUseDefaultMaxBytes: false,
wantDisableDiskCache: false,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
cfg := &config.Config{
CacheMaxBytes: tc.cacheMaxBytes,
CacheMaxBytesExplicit: tc.cacheMaxBytesExplicit,
}
got := newCacheConfig(cfg, nil)
t.Logf("MaxBytes = %d, UseDefaultMaxBytes = %v, DisableDiskCache = %v",
got.MaxBytes, got.UseDefaultMaxBytes, got.DisableDiskCache)
if got.MaxBytes != tc.wantMaxBytes {
t.Errorf("MaxBytes = %d, want %d", got.MaxBytes, tc.wantMaxBytes)
}
if got.UseDefaultMaxBytes != tc.wantUseDefaultMaxBytes {
t.Errorf("UseDefaultMaxBytes = %v, want %v",
got.UseDefaultMaxBytes, tc.wantUseDefaultMaxBytes)
}
if got.DisableDiskCache != tc.wantDisableDiskCache {
t.Errorf("DisableDiskCache = %v, want %v",
got.DisableDiskCache, tc.wantDisableDiskCache)
}
})
}
}
// TestDiskCacheOffOnlyForExplicitZeroCacheMaxBytes starts the handlers
// once with cache_max_bytes omitted and once with cache_max_bytes: 0,
// and checks by whether the cache directories were created that the
// disk cache is on in the first case and off in the second.
func TestDiskCacheOffOnlyForExplicitZeroCacheMaxBytes(t *testing.T) {
t.Parallel()
cases := []struct {
name string
cacheMaxBytesExplicit bool
wantDiskCache bool
}{
{name: "cache_max_bytes omitted", cacheMaxBytesExplicit: false, wantDiskCache: true},
{name: "cache_max_bytes: 0", cacheMaxBytesExplicit: true, wantDiskCache: false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
stateDir := t.TempDir()
cfg := &config.Config{
SigningKey: testSigningKey,
StateDir: stateDir,
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
CacheMaxBytes: 0,
CacheMaxBytesExplicit: tc.cacheMaxBytesExplicit,
}
lc := fxtest.NewLifecycle(t)
log, err := logger.New(lc, logger.Params{Globals: &globals.Globals{}})
if err != nil {
t.Fatalf("logger.New() error = %v", err)
}
db, err := database.New(lc, database.Params{Logger: log, Config: cfg})
if err != nil {
t.Fatalf("database.New() error = %v", err)
}
_, err = New(lc, Params{Logger: log, Database: db, Config: cfg})
if err != nil {
t.Fatalf("New() error = %v", err)
}
lc.RequireStart()
t.Cleanup(lc.RequireStop)
_, err = os.Stat(filepath.Join(stateDir, "cache", "variants"))
gotDiskCache := err == nil
if gotDiskCache != tc.wantDiskCache {
t.Errorf("cache directories created = %v, want %v",
gotDiskCache, tc.wantDiskCache)
}
})
}
}
internal/handlers/client_gone_internal_test.go
+177
View File
@@ -0,0 +1,177 @@
package handlers
import (
"bytes"
"context"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"slices"
"testing"
"time"
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// heldFetcher sends on started when a fetch begins, and holds the fetch until
// release is closed. It then fails the fetch as an upstream error.
type heldFetcher struct {
started chan struct{}
release chan struct{}
}
// Fetch sends on f.started, waits for f.release to be closed, and returns
// httpfetcher.ErrUpstreamError.
func (f heldFetcher) Fetch(
context.Context, imagerequest.Source, string, string,
) (*httpfetcher.FetchResult, error) {
f.started <- struct{}{}
<-f.release
return nil, httpfetcher.ErrUpstreamError
}
// readLogLines decodes the JSON log lines in logs.
func readLogLines(t *testing.T, logs *bytes.Buffer) []map[string]any {
t.Helper()
t.Logf("logged:\n%s", logs.String())
var lines []map[string]any
dec := json.NewDecoder(logs)
for dec.More() {
var line map[string]any
err := dec.Decode(&line)
if err != nil {
t.Fatalf("decoding log line: %v", err)
}
lines = append(lines, line)
}
return lines
}
// TestImageRoutes_ClientGoneWhileWaitingAnswers499 checks that both image
// routes answer 499 to a request whose client goes away while it waits for an
// image another request is making. A first request's fetch is held, so the
// image is being made, and a second request for the same image waits for it
// until its client goes away. The second request must be logged once at info
// level with its request ID and the source host and path. Nothing may be
// logged at error level.
func TestImageRoutes_ClientGoneWhileWaitingAnswers499(t *testing.T) {
t.Parallel()
const requestID = "test-request-id"
for route, imageURL := range photoURLsByRoute() {
t.Run(route, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
fetcher := heldFetcher{started: make(chan struct{}), release: make(chan struct{})}
h, routes := newRoutesFetchingThrough(t, fetcher,
slog.New(slog.NewJSONHandler(&logs, nil)))
target := imageURL(t, h)
firstDone := make(chan struct{})
go func() {
defer close(firstDone)
sendGet(t, routes, target)
}()
<-fetcher.started
ctx, cancel := context.WithCancel(context.WithValue(t.Context(),
middleware.RequestIDKey, requestID))
rec := httptest.NewRecorder()
secondDone := make(chan struct{})
go func() {
defer close(secondDone)
routes.ServeHTTP(rec, httptest.NewRequestWithContext(
ctx, http.MethodGet, target, nil))
}()
// Give the second request time to start waiting for the first
// request's image. The fetch is still held, so the second request
// can end only because its client went away
time.Sleep(100 * time.Millisecond)
cancel()
<-secondDone
close(fetcher.release)
<-firstDone
if rec.Code != 499 {
t.Errorf("status = %d, want 499", rec.Code)
}
var clientGoneLines int
for _, line := range readLogLines(t, &logs) {
if line["level"] == slog.LevelError.String() {
t.Errorf("logged at error level: %v", line["msg"])
}
if line["level"] == slog.LevelInfo.String() &&
line["msg"] == "client went away before the image was ready" &&
line["request_id"] == requestID &&
line["host"] == signedHost &&
line["path"] == photoPath {
clientGoneLines++
}
}
if clientGoneLines != 1 {
t.Errorf("info line naming the request logged %d times, want 1",
clientGoneLines)
}
})
}
}
// TestImageRoutes_CanceledWithLiveRequestAnswers500 checks that both image
// routes answer 500, logged as "failed to get image" at error level, for an
// image service error that wraps context.Canceled while the request's own
// context is live. Its client has not gone away, so the cancellation is
// pixa's own failure.
func TestImageRoutes_CanceledWithLiveRequestAnswers500(t *testing.T) {
t.Parallel()
for route, imageURL := range photoURLsByRoute() {
t.Run(route, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h, routes := newRoutesFetchingThrough(t,
failingFetcher{err: context.Canceled},
slog.New(slog.NewJSONHandler(&logs, nil)))
rec := sendGet(t, routes, imageURL(t, h))
if rec.Code != http.StatusInternalServerError {
t.Errorf("status = %d, want %d",
rec.Code, http.StatusInternalServerError)
}
if !slices.ContainsFunc(readLogLines(t, &logs),
func(line map[string]any) bool {
return line["level"] == slog.LevelError.String() &&
line["msg"] == "failed to get image"
}) {
t.Error(`no "failed to get image" line logged at error level`)
}
})
}
}
internal/handlers/csrf.go
+108 -14
View File
@@ -1,23 +1,43 @@
package handlers package handlers
import ( import (
"bytes"
"errors"
"html/template" "html/template"
"log/slog"
"net/http" "net/http"
"net/url"
"github.com/go-chi/chi/v5/middleware"
"github.com/gorilla/csrf" "github.com/gorilla/csrf"
"sneak.berlin/go/pixa/internal/seal" "sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/templates"
) )
// csrfKeySalt provides domain separation for the CSRF authentication key, // The causes the form-refused page names, each with what to do about it.
// derived from the signing key so tokens survive restarts without extra const (
// configuration and never reuse the session or encrypted-URL key material. causePlainHTTP = "This page was opened over plain HTTP while debug is " +
const csrfKeySalt = "pixa-csrf-v1" "off. Open it over HTTPS, or turn debug on to use plain HTTP on " +
"your own machine."
causeHTTPSWithDebug = "This page was opened over HTTPS while debug is " +
"on. Turn debug off, or open the page over plain HTTP."
causeOtherHost = "The form was sent from another host than the one in " +
"the request's Host header. A reverse proxy in front of pixa must " +
"pass the Host header on unchanged."
causeNoReferer = "The browser sent neither an Origin nor a Referer " +
"header, and over HTTPS pixa needs one to check where the form " +
"came from. Let the browser send the Referer header to this site."
causeStaleToken = "The form token is missing or out of date. Reload " +
"the page and submit the form again."
)
// newCSRFProtect builds the CSRF-protection middleware for the // newCSRFProtect builds the CSRF-protection middleware for the
// state-mutating HTML form routes. The token lives in its own cookie, // state-mutating HTML form routes. The token lives in its own cookie,
// independent of the session cookie, so it also protects the login POST // independent of the session cookie, so it also protects the login POST
// where no session exists yet (login CSRF). // where no session exists yet (login CSRF). key signs that cookie. A
// refused form is answered by the handler that handleFormRefused returns.
// log gets a warning with the library's reason, such as "origin invalid".
// //
// When plaintext is true (local HTTP development), requests are marked // When plaintext is true (local HTTP development), requests are marked
// plaintext so the library neither demands an https Referer nor sets a // plaintext so the library neither demands an https Referer nor sets a
@@ -25,22 +45,18 @@ const csrfKeySalt = "pixa-csrf-v1"
// service runs behind a TLS-terminating proxy, so plaintext is false and // service runs behind a TLS-terminating proxy, so plaintext is false and
// the library enforces its https Referer origin check. // the library enforces its https Referer origin check.
func newCSRFProtect( func newCSRFProtect(
signingKey string, plaintext bool, key [seal.KeySize]byte, plaintext bool, log *slog.Logger,
) (func(http.Handler) http.Handler, error) { ) func(http.Handler) http.Handler {
key, err := seal.DeriveKey([]byte(signingKey), csrfKeySalt)
if err != nil {
return nil, err
}
protect := csrf.Protect( protect := csrf.Protect(
key[:], key[:],
csrf.Path("/"), csrf.Path("/"),
csrf.Secure(!plaintext), csrf.Secure(!plaintext),
csrf.SameSite(csrf.SameSiteStrictMode), csrf.SameSite(csrf.SameSiteStrictMode),
csrf.ErrorHandler(handleFormRefused(plaintext, log)),
) )
if !plaintext { if !plaintext {
return protect, nil return protect
} }
return func(next http.Handler) http.Handler { return func(next http.Handler) http.Handler {
@@ -49,7 +65,85 @@ func newCSRFProtect(
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
protected.ServeHTTP(w, csrf.PlaintextHTTPRequest(r)) protected.ServeHTTP(w, csrf.PlaintextHTTPRequest(r))
}) })
}, nil }
}
// handleFormRefused returns the handler the CSRF middleware calls when it
// refuses a request. It logs the library's reason with the request ID and
// answers 403 with a page naming the cause and what to do. A reason that is
// not a refusal, such as a failure to make a token, answers 500.
func handleFormRefused(plaintext bool, log *slog.Logger) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
reason := csrf.FailureReason(r)
requestID := middleware.GetReqID(r.Context())
cause := formRefusedCause(r, reason, plaintext)
if cause == "" {
log.Error("CSRF middleware failed",
"request_id", requestID, "error", reason)
http.Error(w, "Internal server error",
http.StatusInternalServerError)
return
}
log.Warn("form refused", "request_id", requestID, "reason", reason)
var page bytes.Buffer
err := templates.Render(&page, "form_refused.html",
struct{ Cause string }{Cause: cause})
if err != nil {
log.Error("failed to render form refused template", "error", err)
http.Error(w, "Internal server error",
http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(http.StatusForbidden)
_, err = page.WriteTo(w)
if err != nil {
log.Error("failed to write form refused page", "error", err)
}
})
}
// formRefusedCause returns the cause the form-refused page names for the
// CSRF middleware's reason, or "" when the reason is not a refusal.
func formRefusedCause(r *http.Request, reason error, plaintext bool) string {
switch {
case errors.Is(reason, csrf.ErrNoToken), errors.Is(reason, csrf.ErrBadToken):
return causeStaleToken
case errors.Is(reason, csrf.ErrNoReferer):
return causeNoReferer
case errors.Is(reason, csrf.ErrBadOrigin):
return pageAddressCause(r.Header.Get("Origin"), r.Host, plaintext)
case errors.Is(reason, csrf.ErrBadReferer):
return pageAddressCause(r.Referer(), r.Host, plaintext)
default:
return ""
}
}
// pageAddressCause returns the cause for a form whose page address, the
// Origin or Referer header, the CSRF middleware refused. The middleware
// expects the request's own host, with https when plaintext is false and
// http when it is true. So on the request's own host, only the scheme was
// wrong.
func pageAddressCause(pageAddress, host string, plaintext bool) string {
page, err := url.Parse(pageAddress)
if err != nil || page.Host != host {
return causeOtherHost
}
if plaintext {
return causeHTTPSWithDebug
}
return causePlainHTTP
} }
// CSRF returns the CSRF-protection middleware for the login and generator // CSRF returns the CSRF-protection middleware for the login and generator
internal/handlers/csrf_refused_internal_test.go
+205
View File
@@ -0,0 +1,205 @@
package handlers
import (
"bytes"
"context"
"encoding/json"
"html/template"
"log/slog"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/go-chi/chi/v5/middleware"
"github.com/gorilla/csrf"
"sneak.berlin/go/pixa/internal/seal"
)
// refusedFormRequestID is the request ID of the requests postRefusedForm
// sends, which the warning for a refused form must carry.
const refusedFormRequestID = "test-request-id"
// TestFormRefusedNamesCause verifies that a form the CSRF middleware refuses
// answers 403 with a page naming the cause, and logs a warning with the
// request ID and the middleware's reason.
func TestFormRefusedNamesCause(t *testing.T) {
t.Parallel()
tests := []struct {
name string
debug bool
origin string
referer string
wantCause string
wantReason error
}{
{
name: "plain HTTP origin with debug off",
origin: "http://example.com",
wantCause: causePlainHTTP,
wantReason: csrf.ErrBadOrigin,
},
{
name: "plain HTTP referer with debug off",
referer: "http://example.com/",
wantCause: causePlainHTTP,
wantReason: csrf.ErrBadReferer,
},
{
name: "origin on another host",
origin: "https://proxy.example.net",
wantCause: causeOtherHost,
wantReason: csrf.ErrBadOrigin,
},
{
name: "referer on another host",
referer: "https://proxy.example.net/",
wantCause: causeOtherHost,
wantReason: csrf.ErrBadReferer,
},
{
name: "HTTPS origin with debug on",
debug: true,
origin: "https://example.com",
wantCause: causeHTTPSWithDebug,
wantReason: csrf.ErrBadOrigin,
},
{
name: "no origin or referer with debug off",
wantCause: causeNoReferer,
wantReason: csrf.ErrNoReferer,
},
{
name: "missing token",
debug: true,
origin: "http://example.com",
wantCause: causeStaleToken,
wantReason: csrf.ErrNoToken,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
rec, logLine := postRefusedForm(t, tt.debug, tt.origin, tt.referer,
nil, "")
checkFormRefused(t, rec, logLine, tt.wantCause, tt.wantReason)
})
}
}
// TestFormRefusedNamesCauseForForeignToken verifies that a form whose token
// does not match its CSRF cookie answers 403 with a page naming
// causeStaleToken, and logs a warning with the request ID and the
// middleware's reason.
func TestFormRefusedNamesCauseForForeignToken(t *testing.T) {
t.Parallel()
// The router and postRefusedForm derive their CSRF key from
// testSigningKey, so cookiesA is valid there and only tokenB is wrong.
_, router := newCSRFTestRouter(t)
cookiesA, _ := csrfCredentials(t, router, nil)
_, tokenB := csrfCredentials(t, router, nil)
rec, logLine := postRefusedForm(t, true, "http://example.com", "",
cookiesA, tokenB)
checkFormRefused(t, rec, logLine, causeStaleToken, csrf.ErrBadToken)
}
// postRefusedForm sends a POST with the given cookies to the CSRF
// middleware built for debug. It adds the form token and the Origin and
// Referer headers when they are not empty. It returns the response and the
// one JSON log line the middleware wrote.
func postRefusedForm(
t *testing.T, debug bool, origin, referer string,
cookies []*http.Cookie, token string,
) (*httptest.ResponseRecorder, map[string]any) {
t.Helper()
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
var logs bytes.Buffer
log := slog.New(slog.NewJSONHandler(&logs, nil))
reached := http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) {
t.Error("the refused form reached the handler behind the middleware")
})
srv := newCSRFProtect(keys.CSRFKey, debug, log)(reached)
form := url.Values{loginKeyField: {"x"}}
if token != "" {
form.Set(csrfTokenField, token)
}
ctx := context.WithValue(context.Background(),
middleware.RequestIDKey, refusedFormRequestID)
req := httptest.NewRequestWithContext(ctx, http.MethodPost,
"http://example.com/", strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
for _, c := range cookies {
req.AddCookie(c)
}
if origin != "" {
req.Header.Set("Origin", origin)
}
if referer != "" {
req.Header.Set("Referer", referer)
}
rec := httptest.NewRecorder()
srv.ServeHTTP(rec, req)
var logLine map[string]any
err = json.Unmarshal(logs.Bytes(), &logLine)
if err != nil {
t.Fatalf("decoding log line %q: %v", logs.String(), err)
}
return rec, logLine
}
// checkFormRefused checks that rec is a 403 page naming wantCause and that
// logLine is a warning carrying the request ID and wantReason.
func checkFormRefused(
t *testing.T, rec *httptest.ResponseRecorder, logLine map[string]any,
wantCause string, wantReason error,
) {
t.Helper()
if rec.Code != http.StatusForbidden {
t.Errorf("status = %d, want %d", rec.Code, http.StatusForbidden)
}
body := rec.Body.String()
t.Logf("body: %s", body)
if !strings.Contains(body, template.HTMLEscapeString(wantCause)) {
t.Errorf("page does not name the cause %q", wantCause)
}
t.Logf("log line: %v", logLine)
if logLine["level"] != "WARN" || logLine["msg"] != "form refused" {
t.Errorf("log line = %v, want a WARN \"form refused\"", logLine)
}
if logLine["request_id"] != refusedFormRequestID {
t.Errorf("logged request_id = %v, want %q",
logLine["request_id"], refusedFormRequestID)
}
if logLine["reason"] != wantReason.Error() {
t.Errorf("logged reason = %v, want %q", logLine["reason"], wantReason)
}
}
internal/handlers/fetcher_internal_test.go
-61
View File
@@ -1,61 +0,0 @@
package handlers
import (
"net/http"
"net/netip"
"path/filepath"
"testing"
"time"
"github.com/go-chi/chi/v5"
"go.uber.org/fx"
"go.uber.org/fx/fxtest"
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/globals"
"sneak.berlin/go/pixa/internal/healthcheck"
"sneak.berlin/go/pixa/internal/logger"
)
// TestHandlersBuildTheirOwnFetcherWhenNoneIsProvided builds the handlers as
// pixad does, in an fx app that provides no fetcher, and requests an image
// from 192.0.2.10, which is on the allowlist and in blocked_networks. The URL
// check accepts that address; only the dialer that refuses internal
// addresses checks blocked_networks, so the answer is 403 only if the
// fetcher the handlers build from the config connects with that dialer. Any
// other dialer would try to connect until the upstream fetch timeout, which
// is short so that the test then fails quickly.
func TestHandlersBuildTheirOwnFetcherWhenNoneIsProvided(t *testing.T) {
t.Parallel()
const host = "192.0.2.10"
stateDir := t.TempDir()
cfg := &config.Config{
SigningKey: testSigningKey,
StateDir: stateDir,
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
AllowlistHosts: []string{host},
BlockedNetworks: []netip.Prefix{netip.MustParsePrefix("192.0.2.0/24")},
UpstreamFetchTimeout: 2 * time.Second,
// With no connection slots, the fetch would fail before dialing.
UpstreamConnections: config.DefaultUpstreamConnections,
}
var h *Handlers
app := fxtest.New(t,
fx.Supply(cfg),
fx.Provide(globals.New, logger.New, database.New, healthcheck.New, New),
fx.Populate(&h),
)
app.RequireStart()
t.Cleanup(app.RequireStop)
r := chi.NewRouter()
r.Get("/v1/image/*", h.HandleImage())
rec := sendGet(t, r, photoURL(host))
checkErrorBody(t, rec, http.StatusForbidden, "forbidden")
}
internal/handlers/first_bytes_internal_test.go
+203
View File
@@ -0,0 +1,203 @@
package handlers
import (
"bytes"
"encoding/json"
"image/color"
"log/slog"
"net/http"
"slices"
"testing"
"testing/fstest"
"time"
"github.com/go-chi/chi/v5"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
)
// mismatchedSourcePath is the path of the source that
// TestFirstBytesNotMatchingContentType serves on allowlistedHost. The mock
// fetcher names a file's Content-Type from its extension, so the source is
// sent as image/png.
const mismatchedSourcePath = "/images/source.png"
// newRoutesFetchingThrough returns both image routes, and the Handlers behind
// them, whose image service fetches through fetcher, and which have
// allowlistedHost on their allowlist. The handlers and the image service log
// to log.
func newRoutesFetchingThrough(
t *testing.T, fetcher httpfetcher.Fetcher, log *slog.Logger,
) (*Handlers, http.Handler) {
t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err)
}
svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache,
Fetcher: fetcher,
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
Logger: log,
}, prometheus.NewRegistry())
if err != nil {
t.Fatalf("imageproxy.New() error = %v", err)
}
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{
log: log,
imgSvc: svc,
encGen: encGen,
signer: signature.New(testSigningKey),
allowlist: allowlist.New([]string{allowlistedHost}),
}
r := chi.NewRouter()
r.Get("/v1/image/*", h.HandleImage())
r.Get("/v1/e/{token}/*", h.HandleImageEnc())
return h, r
}
// mismatchedSourcePlainURL returns the /v1/image/ URL of mismatchedSourcePath
// on allowlistedHost at 50x50 as a JPEG.
func mismatchedSourcePlainURL(*testing.T, *Handlers) string {
return "/v1/image/" + allowlistedHost + mismatchedSourcePath + "/50x50.jpeg"
}
// mismatchedSourceEncURL returns an encrypted /v1/e/ URL, made with h's
// generator, of mismatchedSourcePath on allowlistedHost at 50x50 as a JPEG.
func mismatchedSourceEncURL(t *testing.T, h *Handlers) string {
t.Helper()
token, err := h.encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{
Host: allowlistedHost,
Path: mismatchedSourcePath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, time.Time{})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
return "/v1/e/" + token + "/img.jpg"
}
// loggedLevels reads the JSON log lines in logs and returns the messages
// logged at each level.
func loggedLevels(t *testing.T, logs *bytes.Buffer) map[string][]string {
t.Helper()
t.Logf("logged:\n%s", logs.String())
messages := make(map[string][]string)
dec := json.NewDecoder(logs)
for dec.More() {
var line struct {
Level string `json:"level"`
Msg string `json:"msg"`
}
err := dec.Decode(&line)
if err != nil {
t.Fatalf("decoding log line: %v", err)
}
messages[line.Level] = append(messages[line.Level], line.Msg)
}
return messages
}
// TestFirstBytesNotMatchingContentType checks that both image routes answer
// 415 for an upstream body whose first bytes are not those of the format its
// Content-Type names, and log it as a refused source, not as an error of
// pixa's. The next request for that source must answer 502 without fetching
// it again, as after an upstream error status.
func TestFirstBytesNotMatchingContentType(t *testing.T) {
t.Parallel()
bodies := map[string][]byte{
"JPEG sent as image/png": generateTestJPEG(t, 10, 10, color.Black),
"text sent as image/png": []byte("This is just text, not a PNG"),
"body too short to name a format": []byte("short"),
}
// The source's URL on each image route.
sourceURLs := map[string]func(*testing.T, *Handlers) string{
"plain URL": mismatchedSourcePlainURL,
"encrypted URL": mismatchedSourceEncURL,
}
for bodyName, body := range bodies {
for urlName, sourceURL := range sourceURLs {
t.Run(urlName+", "+bodyName, func(t *testing.T) {
t.Parallel()
fetcher := &countingFetcher{
Fetcher: httpfetcher.NewMock(fstest.MapFS{
allowlistedHost + mismatchedSourcePath: &fstest.MapFile{
Data: body,
},
}),
}
var logs bytes.Buffer
h, routes := newRoutesFetchingThrough(t, fetcher,
slog.New(slog.NewJSONHandler(&logs, nil)))
target := sourceURL(t, h)
checkErrorBody(t, sendGet(t, routes, target),
http.StatusUnsupportedMediaType, "unsupported source format")
levels := loggedLevels(t, &logs)
if len(levels[slog.LevelError.String()]) != 0 {
t.Errorf("logged at error level: %q",
levels[slog.LevelError.String()])
}
if !slices.Contains(levels[slog.LevelWarn.String()], "source refused") {
t.Error(`no "source refused" line logged at warning level`)
}
checkErrorBody(t, sendGet(t, routes, target),
http.StatusBadGateway, "upstream error")
if n := fetcher.fetches.Load(); n != 1 {
t.Errorf("upstream fetched %d times, want 1", n)
}
})
}
}
}
internal/handlers/format_auto.go
+22 -23
View File
@@ -8,7 +8,7 @@ import (
"strconv" "strconv"
"strings" "strings"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imagerequest"
) )
// Errors for an Accept header that an auto URL cannot be served for. // Errors for an Accept header that an auto URL cannot be served for.
@@ -18,16 +18,17 @@ var (
"image/jxl, image/avif, image/webp or image/jpeg") "image/jxl, image/avif, image/webp or image/jpeg")
) )
// chooseAutoFormat replaces the format auto in req with the format // chooseAutoFormat returns the format to serve for the requested format: for
// formatForAccept chooses from r's Accept header, and adds Vary: Accept to the // the format auto, the format formatForAccept chooses from r's Accept header,
// and any other format as it is. For auto it adds Vary: Accept to the
// response, which then depends on that header. It answers 400 for an Accept // response, which then depends on that header. It answers 400 for an Accept
// header that is not valid and 406 for one that allows none of the formats, // header that is not valid and 406 for one that allows none of the formats,
// and reports whether req can be served. Any other format is left as it is. // and reports whether the image can be served.
func (s *Handlers) chooseAutoFormat( func (s *Handlers) chooseAutoFormat(
w http.ResponseWriter, r *http.Request, req *imgcache.ImageRequest, w http.ResponseWriter, r *http.Request, requested imagerequest.Format,
) bool { ) (imagerequest.Format, bool) {
if req.Format != imgcache.FormatAuto { if requested != imagerequest.FormatAuto {
return true return requested, true
} }
w.Header().Add("Vary", "Accept") w.Header().Add("Vary", "Accept")
@@ -36,18 +37,16 @@ func (s *Handlers) chooseAutoFormat(
if errors.Is(err, errNotAcceptable) { if errors.Is(err, errNotAcceptable) {
s.respondError(w, err.Error(), http.StatusNotAcceptable) s.respondError(w, err.Error(), http.StatusNotAcceptable)
return false return "", false
} }
if err != nil { if err != nil {
s.respondError(w, err.Error(), http.StatusBadRequest) s.respondError(w, err.Error(), http.StatusBadRequest)
return false return "", false
} }
req.Format = format return format, true
return true
} }
// formatForAccept returns the format an auto URL is served in for the Accept // formatForAccept returns the format an auto URL is served in for the Accept
@@ -56,30 +55,30 @@ func (s *Handlers) chooseAutoFormat(
// specific entry of image/jpeg, image/* and */* allows it, or when it names // specific entry of image/jpeg, image/* and */* allows it, or when it names
// nothing. A q of 0 refuses a format. JPEG XL, AVIF and WebP must be named, as // nothing. A q of 0 refuses a format. JPEG XL, AVIF and WebP must be named, as
// clients that cannot show them send image/* and */* too. // clients that cannot show them send image/* and */* too.
func formatForAccept(accept string) (imgcache.ImageFormat, error) { func formatForAccept(accept string) (imagerequest.Format, error) {
qualities, err := parseAccept(accept) qualities, err := parseAccept(accept)
if err != nil { if err != nil {
return "", err return "", err
} }
if len(qualities) == 0 { if len(qualities) == 0 {
return imgcache.FormatJPEG, nil return imagerequest.FormatJPEG, nil
} }
if qualities["image/jxl"] > 0 { if qualities[imagerequest.FormatJXL.MIMEType()] > 0 {
return imgcache.FormatJXL, nil return imagerequest.FormatJXL, nil
} }
if qualities["image/avif"] > 0 { if qualities[imagerequest.FormatAVIF.MIMEType()] > 0 {
return imgcache.FormatAVIF, nil return imagerequest.FormatAVIF, nil
} }
if qualities["image/webp"] > 0 { if qualities[imagerequest.FormatWebP.MIMEType()] > 0 {
return imgcache.FormatWebP, nil return imagerequest.FormatWebP, nil
} }
// For JPEG, the most specific entry the header has decides // For JPEG, the most specific entry the header has decides
quality, named := qualities["image/jpeg"] quality, named := qualities[imagerequest.FormatJPEG.MIMEType()]
if !named { if !named {
quality, named = qualities["image/*"] quality, named = qualities["image/*"]
} }
@@ -89,7 +88,7 @@ func formatForAccept(accept string) (imgcache.ImageFormat, error) {
} }
if quality > 0 { if quality > 0 {
return imgcache.FormatJPEG, nil return imagerequest.FormatJPEG, nil
} }
return "", errNotAcceptable return "", errNotAcceptable
internal/handlers/format_auto_internal_test.go
+44 -39
View File
@@ -12,8 +12,7 @@ import (
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
) )
// The content types the tests below expect. // The content types the tests below expect.
@@ -33,24 +32,24 @@ func TestFormatForAccept(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
accept string accept string
want imgcache.ImageFormat want imagerequest.Format
wantErr error wantErr error
}{ }{
{"AVIF-capable browser", {"AVIF-capable browser",
"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8", "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
imgcache.FormatAVIF, nil}, imagerequest.FormatAVIF, nil},
{"WebP-capable browser", {"WebP-capable browser",
"image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5", "image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5",
imgcache.FormatWebP, nil}, imagerequest.FormatWebP, nil},
{"WebP only", webpType, imgcache.FormatWebP, nil}, {"WebP only", webpType, imagerequest.FormatWebP, nil},
{"neither", "image/png,image/*;q=0.8,*/*;q=0.5", imgcache.FormatJPEG, nil}, {"neither", "image/png,image/*;q=0.8,*/*;q=0.5", imagerequest.FormatJPEG, nil},
{"wildcard only", "*/*", imgcache.FormatJPEG, nil}, {"wildcard only", "*/*", imagerequest.FormatJPEG, nil},
{"image wildcard only", "image/*", imgcache.FormatJPEG, nil}, {"image wildcard only", "image/*", imagerequest.FormatJPEG, nil},
{"absent", "", imgcache.FormatJPEG, nil}, {"absent", "", imagerequest.FormatJPEG, nil},
{"q=0 on AVIF", "image/avif;q=0,image/webp,*/*", imgcache.FormatWebP, nil}, {"q=0 on AVIF", "image/avif;q=0,image/webp,*/*", imagerequest.FormatWebP, nil},
{"AVIF named twice, once with q=0", "image/avif,image/avif;q=0.0,*/*", {"AVIF named twice, once with q=0", "image/avif,image/avif;q=0.0,*/*",
imgcache.FormatJPEG, nil}, imagerequest.FormatJPEG, nil},
{"upper case and spaces", " Image/AVIF ; Q=0.5 ", imgcache.FormatAVIF, nil}, {"upper case and spaces", " Image/AVIF ; Q=0.5 ", imagerequest.FormatAVIF, nil},
{"q=0 on JPEG", "image/jpeg;q=0,image/*", "", errNotAcceptable}, {"q=0 on JPEG", "image/jpeg;q=0,image/*", "", errNotAcceptable},
{"q=0 on everything", "*/*;q=0", "", errNotAcceptable}, {"q=0 on everything", "*/*;q=0", "", errNotAcceptable},
{"PNG only", "image/png", "", errNotAcceptable}, {"PNG only", "image/png", "", errNotAcceptable},
@@ -76,28 +75,33 @@ func TestFormatForAccept(t *testing.T) {
// autoPhotoURLs returns a signed /v1/image/ URL and an encrypted /v1/e/ URL, // autoPhotoURLs returns a signed /v1/image/ URL and an encrypted /v1/e/ URL,
// both valid for a minute, for the JPEG at photoPath on signedHost at 50x50 in // both valid for a minute, for the JPEG at photoPath on signedHost at 50x50 in
// the format auto, made with h's image service and generator. // the format auto. The first is signed with testSigningKey, the key the
// handlers that newSignedHostServer builds check signatures with. The second
// is made with h's generator.
func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) { func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) {
t.Helper() t.Helper()
signedURL, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{ signedURL := generateSignedURL(testSigningKey, &imagerequest.Request{
SourceHost: signedHost, Source: imagerequest.Source{
SourcePath: photoPath, Host: signedHost,
Size: imgcache.Size{Width: 50, Height: 50}, Path: photoPath,
Format: imgcache.FormatAuto, },
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatAuto,
},
}, time.Minute) }, time.Minute)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
token, err := h.encGen.Generate(&encurl.Payload{ token, err := h.encGen.Generate(&imagerequest.Request{
SourceHost: signedHost, Source: imagerequest.Source{
SourcePath: photoPath, Host: signedHost,
Width: 50, Path: photoPath,
Height: 50, },
Format: imgcache.FormatAuto, Transform: imagerequest.Transform{
ExpiresAt: time.Now().Add(time.Minute).Unix(), Size: imagerequest.Size{Width: 50, Height: 50},
}) Format: imagerequest.FormatAuto,
},
}, time.Now().Add(time.Minute))
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
@@ -182,17 +186,18 @@ func TestFormatAuto_ChosenFromAccept(t *testing.T) {
func TestFormatAuto_SignatureCoversAuto(t *testing.T) { func TestFormatAuto_SignatureCoversAuto(t *testing.T) {
t.Parallel() t.Parallel()
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler)) _, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
signedForAVIF, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{ signedForAVIF := generateSignedURL(testSigningKey, &imagerequest.Request{
SourceHost: signedHost, Source: imagerequest.Source{
SourcePath: photoPath, Host: signedHost,
Size: imgcache.Size{Width: 50, Height: 50}, Path: photoPath,
Format: imgcache.FormatAVIF, },
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatAVIF,
},
}, time.Minute) }, time.Minute)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
target := strings.Replace(signedForAVIF, "/50x50.avif?", "/50x50.auto?", 1) target := strings.Replace(signedForAVIF, "/50x50.avif?", "/50x50.auto?", 1)
if target == signedForAVIF { if target == signedForAVIF {
internal/handlers/handlers.go
+23 -128
View File
@@ -4,7 +4,6 @@ package handlers
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"errors"
"log/slog" "log/slog"
"net/http" "net/http"
"time" "time"
@@ -12,13 +11,13 @@ import (
"go.uber.org/fx" "go.uber.org/fx"
"sneak.berlin/go/pixa/internal/allowlist" "sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/config" "sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/healthcheck" "sneak.berlin/go/pixa/internal/healthcheck"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/logger" "sneak.berlin/go/pixa/internal/logger"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
"sneak.berlin/go/pixa/internal/signature"
) )
// Params defines dependencies for Handlers. // Params defines dependencies for Handlers.
@@ -27,70 +26,50 @@ type Params struct {
Logger *logger.Logger Logger *logger.Logger
Healthcheck *healthcheck.Healthcheck Healthcheck *healthcheck.Healthcheck
Database *database.Database
Config *config.Config Config *config.Config
Keys seal.Keys
// Fetcher, when provided, fetches upstream images in place of the ImageService *imageproxy.Proxy
// fetcher the handlers build from the config. Only tests provide one; SessionManager *session.Manager
// pixad does not. EncryptedURLGenerator *encurl.Generator
Fetcher httpfetcher.Fetcher `optional:"true"`
} }
// Handlers provides HTTP request handlers. // Handlers provides HTTP request handlers.
type Handlers struct { type Handlers struct {
log *slog.Logger log *slog.Logger
hc *healthcheck.Healthcheck hc *healthcheck.Healthcheck
db *database.Database
config *config.Config config *config.Config
fetcher httpfetcher.Fetcher imgSvc *imageproxy.Proxy
imgSvc *imgcache.Service
imgCache *imgcache.Cache
sessMgr *session.Manager sessMgr *session.Manager
encGen *encurl.Generator encGen *encurl.Generator
csrfProtect func(http.Handler) http.Handler csrfProtect func(http.Handler) http.Handler
// signer checks the signature of a /v1/image/ URL with signing_key.
signer *signature.Signer
// allowlist matches the hosts of allowlist_hosts, which a /v1/image/
// URL may name without a signature.
allowlist *allowlist.HostAllowList
// refererBlocklist matches the hosts of referer_blocklist; its IsAllowed // refererBlocklist matches the hosts of referer_blocklist; its IsAllowed
// reports whether a URL's host is on that list. // reports whether a URL's host is on that list.
refererBlocklist *allowlist.HostAllowList refererBlocklist *allowlist.HostAllowList
} }
// New creates a new Handlers instance. // New creates a new Handlers instance.
func New(lc fx.Lifecycle, params Params) (*Handlers, error) { func New(_ fx.Lifecycle, params Params) *Handlers {
csrfProtect, err := newCSRFProtect(params.Config.SigningKey, params.Config.Debug) log := params.Logger.Get()
if err != nil {
return nil, err
}
s := &Handlers{ return &Handlers{
log: params.Logger.Get(), log: log,
hc: params.Healthcheck, hc: params.Healthcheck,
db: params.Database,
config: params.Config, config: params.Config,
fetcher: params.Fetcher, imgSvc: params.ImageService,
csrfProtect: csrfProtect, sessMgr: params.SessionManager,
encGen: params.EncryptedURLGenerator,
csrfProtect: newCSRFProtect(params.Keys.CSRFKey, params.Config.Debug, log),
signer: signature.New(params.Config.SigningKey),
allowlist: allowlist.New(params.Config.AllowlistHosts),
refererBlocklist: allowlist.New(params.Config.RefererBlocklist), refererBlocklist: allowlist.New(params.Config.RefererBlocklist),
} }
lc.Append(fx.Hook{
//nolint:contextcheck // the cache's goroutines outlive OnStart; OnStop stops them
OnStart: func(_ context.Context) error {
return s.initImageService()
},
// The pending counts are written here, before the database's stop
// hook closes it.
OnStop: func(ctx context.Context) error {
if s.imgCache == nil {
return nil
}
return errors.Join(
s.imgCache.StopEviction(ctx),
s.imgCache.StopPendingCountWrites(ctx),
)
},
})
return s, nil
} }
// WaitForProcessing waits until no image is being processed, or until ctx // WaitForProcessing waits until no image is being processed, or until ctx
@@ -99,90 +78,6 @@ func (s *Handlers) WaitForProcessing(ctx context.Context) int {
return s.imgSvc.WaitForProcessing(ctx) return s.imgSvc.WaitForProcessing(ctx)
} }
// newCacheConfig builds the image cache's configuration from cfg.
// cache_max_bytes: 0 disables the disk cache entirely; any other value
// is the eviction limit in bytes; when it is omitted, the cache works
// out the default limit itself.
func newCacheConfig(cfg *config.Config, log *slog.Logger) imgcache.CacheConfig {
return imgcache.CacheConfig{
StateDir: cfg.StateDir,
NegativeTTL: imgcache.DefaultNegativeTTL,
MaxBytes: cfg.CacheMaxBytes,
UseDefaultMaxBytes: !cfg.CacheMaxBytesExplicit,
DisableDiskCache: cfg.CacheMaxBytesExplicit && cfg.CacheMaxBytes == 0,
Logger: log,
}
}
// initImageService initializes the image cache and service.
func (s *Handlers) initImageService() error {
cache, err := imgcache.NewCache(s.db.DB(), newCacheConfig(s.config, s.log))
if err != nil {
return err
}
s.imgCache = cache
// Background eviction: startup reconciliation, then periodic and
// write-pressure passes. No-op when the disk cache is disabled.
cache.StartEviction(imgcache.DefaultEvictionInterval)
// Writes the counts requests could not write by their deadline
cache.StartPendingCountWrites()
// Create the fetcher config
fetcherCfg := httpfetcher.DefaultConfig()
fetcherCfg.AllowHTTP = s.config.AllowHTTP
fetcherCfg.Timeout = s.config.UpstreamFetchTimeout
fetcherCfg.MaxResponseSize = s.config.UpstreamMaxResponseSize
if s.config.UpstreamConnectionsPerHost > 0 {
fetcherCfg.MaxConnectionsPerHost = s.config.UpstreamConnectionsPerHost
}
fetcherCfg.MaxConnections = s.config.UpstreamConnections
fetcherCfg.BlockedNetworks = s.config.BlockedNetworks
// Create the service. With no fetcher provided, it builds its own from
// fetcherCfg.
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
Cache: cache,
FetcherConfig: fetcherCfg,
Fetcher: s.fetcher,
SigningKey: s.config.SigningKey,
Allowlist: s.config.AllowlistHosts,
MaxConcurrentProcessing: s.config.MaxConcurrentProcessing,
Logger: s.log,
})
if err != nil {
return err
}
s.imgSvc = svc
s.log.Info("image service initialized")
// Initialize session manager (signing key is validated at config load
// time). Session cookies are always Secure/HttpOnly/SameSite=Strict.
sessMgr, err := session.NewManager(s.config.SigningKey)
if err != nil {
return err
}
s.sessMgr = sessMgr
// Initialize encrypted URL generator
encGen, err := encurl.NewGenerator(s.config.SigningKey)
if err != nil {
return err
}
s.encGen = encGen
s.log.Info("session manager and URL generator initialized")
return nil
}
func (s *Handlers) respondJSON(w http.ResponseWriter, data any, status int) { func (s *Handlers) respondJSON(w http.ResponseWriter, data any, status int) {
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status) w.WriteHeader(status)
internal/handlers/handlers_internal_test.go
+109 -16
View File
@@ -18,15 +18,20 @@ import (
"time" "time"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/database" "sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/signature"
) )
// testFixtures contains test data. // testFixtures contains test data.
type testFixtures struct { type testFixtures struct {
handler *Handlers handler *Handlers
service *imgcache.Service
goodHost string goodHost string
} }
@@ -45,32 +50,37 @@ func setupTestHandler(t *testing.T) *testFixtures {
tmpDir := t.TempDir() tmpDir := t.TempDir()
db := setupTestDB(t) db := setupTestDB(t)
cache, err := imgcache.NewCache(db, imgcache.CacheConfig{ cache, err := imgcache.NewCache(db, imgcache.Config{
StateDir: tmpDir, StateDir: tmpDir,
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
}) MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("failed to create cache: %v", err) t.Fatalf("failed to create cache: %v", err)
} }
svc, err := imgcache.NewService(&imgcache.ServiceConfig{ svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache, Cache: cache,
Fetcher: newMockFetcher(mockFS), Fetcher: newMockFetcher(mockFS),
SigningKey: "test-signing-key-must-be-32-chars", Processor: imageprocessor.New(imageprocessor.Params{
Allowlist: []string{goodHost}, MaxInputBytes: 50 << 20,
}) MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
}, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("failed to create service: %v", err) t.Fatalf("failed to create proxy: %v", err)
} }
h := &Handlers{ h := &Handlers{
imgSvc: svc, imgSvc: svc,
log: slog.Default(), log: slog.Default(),
signer: signature.New("test-signing-key-must-be-32-chars"),
allowlist: allowlist.New([]string{goodHost}),
} }
return &testFixtures{ return &testFixtures{
handler: h, handler: h,
service: svc,
goodHost: goodHost, goodHost: goodHost,
} }
} }
@@ -121,10 +131,9 @@ func newMockFetcher(fs fs.FS) *mockFetcher {
} }
func (f *mockFetcher) Fetch( func (f *mockFetcher) Fetch(
_ context.Context, url string, _ context.Context, source imagerequest.Source, _, _ string,
) (*httpfetcher.FetchResult, error) { ) (*httpfetcher.FetchResult, error) {
// Remove https:// prefix path := source.Host + source.Path
path := url[8:] // Remove "https://"
data, err := fs.ReadFile(f.fs, path) data, err := fs.ReadFile(f.fs, path)
if err != nil { if err != nil {
@@ -248,6 +257,90 @@ func TestHandleImage_ConditionalRequest_IfNoneMatch_DifferentETag(t *testing.T)
} }
} }
// TestHandleImage_ConditionalRequest_EarlierETagGetsChangedImage serves an
// image, changes it upstream, and requests it again with the ETag first
// served, once the cache TTL has passed. The changed image must be served,
// not 304 Not Modified.
func TestHandleImage_ConditionalRequest_EarlierETagGetsChangedImage(t *testing.T) {
t.Parallel()
const host = "goodhost.example.com"
upstream := fstest.MapFS{
host + photoPath: &fstest.MapFile{
Data: generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255}),
},
}
// A cache TTL of one nanosecond has passed by the second request
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute,
CacheTTL: time.Nanosecond,
MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err)
}
svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache,
Fetcher: newMockFetcher(upstream),
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
Logger: slog.New(slog.DiscardHandler),
}, prometheus.NewRegistry())
if err != nil {
t.Fatalf("imageproxy.New() error = %v", err)
}
h := &Handlers{
imgSvc: svc,
log: slog.New(slog.DiscardHandler),
signer: signature.New(testSigningKey),
allowlist: allowlist.New([]string{host}),
}
r := chi.NewRouter()
r.Get("/v1/image/*", h.HandleImage())
target := "/v1/image/" + host + photoPath + "/50x50.jpeg"
first := httptest.NewRecorder()
r.ServeHTTP(first,
httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil))
etag := first.Header().Get("ETag")
if first.Code != http.StatusOK || etag == "" {
t.Fatalf("first request: status = %d, ETag = %q, want %d and an ETag",
first.Code, etag, http.StatusOK)
}
upstream[host+photoPath] = &fstest.MapFile{
Data: generateTestJPEG(t, 100, 100, color.RGBA{0, 0, 255, 255}),
}
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil)
req.Header.Set("If-None-Match", etag)
second := httptest.NewRecorder()
r.ServeHTTP(second, req)
t.Logf("first ETag %s; with it after the change: status %d, ETag %s, %d bytes",
etag, second.Code, second.Header().Get("ETag"), second.Body.Len())
if second.Code != http.StatusOK {
t.Fatalf("status = %d, want %d with the changed image",
second.Code, http.StatusOK)
}
if bytes.Equal(second.Body.Bytes(), first.Body.Bytes()) {
t.Error("the image served is the one made before the change")
}
}
func TestHandleImage_ETagHeader(t *testing.T) { func TestHandleImage_ETagHeader(t *testing.T) {
t.Parallel() t.Parallel()
internal/handlers/image.go
+187 -101
View File
@@ -1,22 +1,31 @@
package handlers package handlers
import ( import (
"context"
"errors" "errors"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
"net/url" "net/url"
"strconv" "strconv"
"strings"
"time" "time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic"
) )
// statusClientClosedRequest is the status nginx calls "client closed
// request". The image routes answer it when the client goes away before its
// image is ready, unless its request is the one making the image. Nobody reads
// it; it is for logs and metrics.
const statusClientClosedRequest = 499
// HandleImage handles the main image proxy route: // HandleImage handles the main image proxy route:
// /v1/image/<host>/<path>/<width>x<height>.<format>, or with no format // /v1/image/<host>/<path>/<width>x<height>.<format>, or with no format
// /v1/image/<host>/<path>/<width>x<height> // /v1/image/<host>/<path>/<width>x<height>
@@ -26,17 +35,17 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
return return
} }
req, ok := s.parseImageRequest(w, r) req, sig, expires, ok := s.parseImageRequest(w, r)
if !ok { if !ok {
return return
} }
// Validate signature if required // Validate signature if required
err := s.imgSvc.ValidateRequest(req) err := s.validateSignature(req, sig, expires)
if err != nil { if err != nil {
s.log.Warn("signature validation failed", s.log.Warn("signature validation failed",
"host", req.SourceHost, "host", req.Source.Host,
"path", req.SourcePath, "path", req.Source.Path,
"error", err, "error", err,
) )
s.respondError(w, "unauthorized", http.StatusUnauthorized) s.respondError(w, "unauthorized", http.StatusUnauthorized)
@@ -44,27 +53,7 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
return return
} }
// The signature covers the format auto, not the format chosen s.serveImage(w, r, req, expires)
if !s.chooseAutoFormat(w, r, req) {
return
}
// Get cache key for logging
cacheKey := imgcache.CacheKey(req)
// Get the image (from cache or fetch/process)
startTime := time.Now()
resp, err := s.imgSvc.Get(r.Context(), req)
if err != nil {
s.respondImageError(w, req, err)
return
}
defer func() { _ = resp.Content.Close() }()
s.writeImageResponse(w, r, req, resp, cacheKey, startTime)
} }
} }
@@ -80,30 +69,28 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
} }
} }
// parseImageRequest parses the wildcard path and query parameters into // parseImageRequest parses the image URL's path and query parameters into
// an ImageRequest. On invalid input it writes an error response and // an image request, and returns it with the URL's signature and expiry. On
// returns false. // invalid input it writes an error response and returns false.
func (s *Handlers) parseImageRequest( func (s *Handlers) parseImageRequest(
w http.ResponseWriter, r *http.Request, w http.ResponseWriter, r *http.Request,
) (*imgcache.ImageRequest, bool) { ) (*imagerequest.Request, string, time.Time, bool) {
// Get the wildcard path from chi // The path is parsed decoded once, as r.URL.Path holds it, and not from
pathParam := chi.URLParam(r, "*") // chi's wildcard: chi routes on the path as the client wrote it whenever
// that is not Go's own encoding of it, so its wildcard would still hold
// Parse the URL path // the %3F before a source query when, say, a "/" in the query is written
parsed, err := imgcache.ParseImagePath(pathParam) // as %2F
req, err := imagerequest.ParseImageURL(r.URL.Path)
if err != nil { if err != nil {
s.log.Warn("failed to parse image URL", s.log.Warn("failed to parse image URL",
"path", pathParam, "path", r.URL.Path,
"error", err, "error", err,
) )
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest) s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
return nil, false return nil, "", time.Time{}, false
} }
// Convert to ImageRequest
req := parsed.ToImageRequest()
// Parse signature params from query string. r.URL.Query() would silently // Parse signature params from query string. r.URL.Query() would silently
// drop a pair it cannot decode, such as q=80%, so that q would be served // drop a pair it cannot decode, such as q=80%, so that q would be served
// at 85; a query string that cannot be decoded is refused instead. A // at 85; a query string that cannot be decoded is refused instead. A
@@ -114,7 +101,7 @@ func (s *Handlers) parseImageRequest(
s.respondError(w, fmt.Sprintf("invalid query string %q: %v", s.respondError(w, fmt.Sprintf("invalid query string %q: %v",
r.URL.RawQuery, err), http.StatusBadRequest) r.URL.RawQuery, err), http.StatusBadRequest)
return nil, false return nil, "", time.Time{}, false
} }
for name, values := range query { for name, values := range query {
@@ -122,17 +109,15 @@ func (s *Handlers) parseImageRequest(
s.respondError(w, fmt.Sprintf("invalid %s: given more than once", s.respondError(w, fmt.Sprintf("invalid %s: given more than once",
name), http.StatusBadRequest) name), http.StatusBadRequest)
return nil, false return nil, "", time.Time{}, false
} }
} }
req.Signature = query.Get("sig") expires, err := parseExpires(query)
req.Expires, err = parseExpires(query)
if err != nil { if err != nil {
s.respondError(w, err.Error(), http.StatusBadRequest) s.respondError(w, err.Error(), http.StatusBadRequest)
return nil, false return nil, "", time.Time{}, false
} }
// Parse optional quality and fit params. Only a q missing from the URL is // Parse optional quality and fit params. Only a q missing from the URL is
@@ -144,51 +129,76 @@ func (s *Handlers) parseImageRequest(
s.respondError(w, `invalid q: not a number, got ""`, s.respondError(w, `invalid q: not a number, got ""`,
http.StatusBadRequest) http.StatusBadRequest)
return nil, false return nil, "", time.Time{}, false
} }
req.Quality, err = parseFormInt(query, "q", req.Transform.Quality, err = parseFormInt(query, "q",
encurl.DefaultQuality, minQuality, maxQuality) encurl.DefaultQuality, minQuality, maxQuality)
if err != nil { if err != nil {
s.respondError(w, fmt.Sprintf("%v, got %q", err, qStr), s.respondError(w, fmt.Sprintf("%v, got %q", err, qStr),
http.StatusBadRequest) http.StatusBadRequest)
return nil, false return nil, "", time.Time{}, false
} }
// Only a fit missing from the URL is cover. A fit in the URL that is not a // Only a fit missing from the URL is cover. A fit in the URL that is not a
// fit mode is refused by the fit-mode check below; that check would take an // fit mode is refused by the fit-mode check below; that check would take an
// empty fit as missing, so an empty one is refused here. // empty fit as missing, so an empty one is refused here.
req.FitMode = imgcache.FitMode(query.Get("fit")) req.Transform.FitMode = imagerequest.Fit(query.Get("fit"))
if query.Has("fit") && req.FitMode == "" { if query.Has("fit") && req.Transform.FitMode == "" {
s.respondError(w, `invalid fit: not a fit mode, got ""`, http.StatusBadRequest) s.respondError(w, `invalid fit: not a fit mode, got ""`, http.StatusBadRequest)
return nil, false return nil, "", time.Time{}, false
} }
// Default fit mode if not set // Default fit mode if not set
if req.FitMode == "" { if req.Transform.FitMode == "" {
req.FitMode = imgcache.FitCover req.Transform.FitMode = imagerequest.FitCover
} }
// Enforce dimension and fit-mode bounds, shared with the encrypted-URL // Enforce dimension and fit-mode bounds, shared with the encrypted-URL
// route. Dimensions are already bounded by the path parser above; this // route. Dimensions are already bounded by the path parser above; this
// also rejects an unrecognized fit mode with 400 instead of letting it // also rejects an unrecognized fit mode with 400 instead of letting it
// reach the processor as a 500. // reach the processor as a 500.
err = imgcache.ValidateImageRequest(req) err = imagerequest.ValidateImageRequest(req)
if err != nil { if err != nil {
s.respondError(w, "invalid image request: "+err.Error(), s.respondError(w, "invalid image request: "+err.Error(),
http.StatusBadRequest) http.StatusBadRequest)
return nil, false return nil, "", time.Time{}, false
} }
return req, true return req, query.Get("sig"), expires, true
}
// validateSignature returns nil when the source host of req is on
// allowlist_hosts and names no port. Otherwise it verifies sig, the URL's
// signature of req, and expires, the URL's expiry.
func (s *Handlers) validateSignature(
req *imagerequest.Request, sig string, expires time.Time,
) error {
// An allowlist entry names no port, so a host followed by ":", with or
// without a port, needs a signature. Go's URL type reads the port as the
// fetcher does: in an IPv6 address written without brackets, such as
// 2001:db8::1:8443, it is what follows the last ":". This is checked here
// and not in allowlist.IsAllowed, which also matches referer_blocklist,
// where a Referer must be blocked whatever its port. Port and IsAllowed
// read only the host of a URL.
sourceURL := &url.URL{Host: req.Source.Host}
hostHasPort := sourceURL.Port() != "" ||
strings.HasSuffix(sourceURL.Host, ":")
if !hostHasPort && s.allowlist.IsAllowed(sourceURL) {
return nil
}
// Signature required for non-allowed hosts
return s.signer.Verify(req, sig, expires)
} }
// parseExpires reads the exp query parameter, a Unix time in seconds. An exp // parseExpires reads the exp query parameter, a Unix time in seconds. An exp
// missing from the URL gives the zero time, which the signature check takes // missing from the URL gives the zero time, which the signature check refuses
// as no expiration. An exp in the URL that is not a whole number, an empty // as a missing exp. An exp in the URL that is not a whole number, an empty
// one included, is an error naming exp and the value. // one included, is an error naming exp and the value.
func parseExpires(query url.Values) (time.Time, error) { func parseExpires(query url.Values) (time.Time, error) {
if !query.Has("exp") { if !query.Has("exp") {
@@ -216,46 +226,75 @@ func (s *Handlers) HandleTooManyRequests() http.HandlerFunc {
} }
} }
// respondImageError maps image retrieval errors to HTTP responses. // respondImageError answers err, the image service's error for req, on both
// image routes. It logs a 500 at error level, as pixa's own failure. It logs
// the other answers at warn level, since the client, the upstream or a full
// limit caused them. The 415 for a source refused for its first bytes is not
// logged here, because the image service logs it.
func (s *Handlers) respondImageError( func (s *Handlers) respondImageError(
w http.ResponseWriter, req *imgcache.ImageRequest, err error, w http.ResponseWriter, req *imagerequest.Request, err error,
) { ) {
s.log.Error("failed to get image", // An upstream body whose first bytes are not those of the format its
"host", req.SourceHost, // Content-Type names is the upstream's error, not pixa's: the image
"path", req.SourcePath, // service has logged it as a refused source.
"error", err, if errors.Is(err, magic.ErrMagicByteMismatch) ||
) errors.Is(err, magic.ErrUnknownFormat) ||
errors.Is(err, magic.ErrNotEnoughData) {
// Check for specific error types
if errors.Is(err, httpfetcher.ErrSSRFBlocked) {
s.respondError(w, "forbidden", http.StatusForbidden)
return
}
if errors.Is(err, httpfetcher.ErrUpstreamError) {
s.respondError(w, "upstream error", http.StatusBadGateway)
return
}
if errors.Is(err, httpfetcher.ErrTooManyConnections) ||
errors.Is(err, imageprocessor.ErrTooManyImages) {
s.respondError(w, "server busy, try again later",
http.StatusServiceUnavailable)
return
}
if errors.Is(err, httpfetcher.ErrInvalidContentType) ||
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat) {
s.respondError(w, "unsupported source format", s.respondError(w, "unsupported source format",
http.StatusUnsupportedMediaType) http.StatusUnsupportedMediaType)
return return
} }
s.respondError(w, "internal error", http.StatusInternalServerError) var (
status int
message string
)
switch {
case errors.Is(err, httpfetcher.ErrSSRFBlocked):
status, message = http.StatusForbidden, "forbidden"
case errors.Is(err, httpfetcher.ErrUpstreamError),
errors.Is(err, httpfetcher.ErrInvalidHost),
errors.Is(err, httpfetcher.ErrUpstreamRequestFailed):
status, message = http.StatusBadGateway, "upstream error"
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
status, message = http.StatusGatewayTimeout, "upstream timeout"
case errors.Is(err, httpfetcher.ErrTooManyConnections),
errors.Is(err, imageprocessor.ErrTooManyImages),
errors.Is(err, imageproxy.ErrProcessingTimedOut):
status = http.StatusServiceUnavailable
message = "server busy, try again later"
case errors.Is(err, httpfetcher.ErrInvalidContentType),
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat):
status = http.StatusUnsupportedMediaType
message = "unsupported source format"
case errors.Is(err, imageprocessor.ErrOutputTooLarge):
status = http.StatusBadRequest
message = fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d",
imageprocessor.MaxOutputDimension)
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
status = http.StatusBadRequest
message = fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels)
default:
s.log.Error("failed to get image",
"host", req.Source.Host,
"path", req.Source.Path,
"error", err,
)
s.respondError(w, "internal error", http.StatusInternalServerError)
return
}
s.log.Warn("failed to get image",
"host", req.Source.Host,
"path", req.Source.Path,
"error", err,
)
s.respondError(w, message, status)
} }
// cacheControl returns the Cache-Control header for an image served through a // cacheControl returns the Cache-Control header for an image served through a
@@ -312,13 +351,50 @@ func notModified(w http.ResponseWriter, r *http.Request, etag string) bool {
return true return true
} }
// writeImageResponse writes headers and streams the image content, // serveImage serves the image that req asks for. Both image routes call it
// handling conditional and HEAD requests. // once they have the request and its URL's expiry, which ends the
func (s *Handlers) writeImageResponse( // Cache-Control max-age. It chooses the format for the format auto, answers
// If-None-Match with 304 and HEAD with the headers only. It answers 499 when
// the image service stopped because the client went away, and any other error
// of the image service with respondImageError.
func (s *Handlers) serveImage(
w http.ResponseWriter, r *http.Request, w http.ResponseWriter, r *http.Request,
req *imgcache.ImageRequest, resp *imgcache.ImageResponse, req *imagerequest.Request, expires time.Time,
cacheKey imgcache.VariantKey, startTime time.Time,
) { ) {
// The signature or the token covers the format auto, not the format chosen
format, ok := s.chooseAutoFormat(w, r, req.Transform.Format)
if !ok {
return
}
req.Transform.Format = format
startTime := time.Now()
resp, err := s.imgSvc.Get(r.Context(), req)
// The client went away before its image was ready, which is not pixa's
// failure. A cancellation while the request's context is live is pixa's,
// and respondImageError answers it 500.
if errors.Is(err, context.Canceled) && r.Context().Err() != nil {
s.log.Info("client went away before the image was ready",
"request_id", middleware.GetReqID(r.Context()),
"host", req.Source.Host,
"path", req.Source.Path,
)
w.WriteHeader(statusClientClosedRequest)
return
}
if err != nil {
s.respondImageError(w, req, err)
return
}
defer func() { _ = resp.Content.Close() }()
// Set response headers // Set response headers
w.Header().Set("Content-Type", resp.ContentType) w.Header().Set("Content-Type", resp.ContentType)
@@ -327,7 +403,7 @@ func (s *Handlers) writeImageResponse(
} }
// Cache control headers // Cache control headers
w.Header().Set("Cache-Control", cacheControl(req.Expires)) w.Header().Set("Cache-Control", cacheControl(expires))
w.Header().Set("X-Pixa-Cache", string(resp.CacheStatus)) w.Header().Set("X-Pixa-Cache", string(resp.CacheStatus))
if notModified(w, r, resp.ETag) { if notModified(w, r, resp.ETag) {
@@ -344,21 +420,31 @@ func (s *Handlers) writeImageResponse(
// Stream the response // Stream the response
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
// A failed write is most often a client that has gone away, not pixa's
// failure, so it is logged at warn level
servedBytes, err := io.Copy(w, resp.Content) servedBytes, err := io.Copy(w, resp.Content)
if err != nil { if err != nil {
s.log.Error("failed to write response", s.log.Warn("failed to write response",
"request_id", middleware.GetReqID(r.Context()),
"host", req.Source.Host,
"path", req.Source.Path,
"served_bytes", servedBytes,
"error", err, "error", err,
) )
return
} }
// Log cache status and timing after serving // Log cache status and timing after serving
duration := time.Since(startTime) duration := time.Since(startTime)
s.log.Info("image served", s.log.Info("image served",
"request_id", middleware.GetReqID(r.Context()), "request_id", middleware.GetReqID(r.Context()),
"cache_key", cacheKey, "cache_key", imagerequest.CacheKey(req),
"host", req.Source.Host,
"path", req.Source.Path,
"cache_status", resp.CacheStatus, "cache_status", resp.CacheStatus,
"duration_ms", duration.Milliseconds(), "duration_ms", duration.Milliseconds(),
"format", req.Format, "format", req.Transform.Format,
"served_bytes", servedBytes, "served_bytes", servedBytes,
"fetched_bytes", resp.FetchedBytes, "fetched_bytes", resp.FetchedBytes,
) )
internal/handlers/image_cache_control_internal_test.go
+59 -32
View File
@@ -12,9 +12,16 @@ import (
"time" "time"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
) )
// photoPath is the path of the JPEG that newSignedHostServer serves. // photoPath is the path of the JPEG that newSignedHostServer serves.
@@ -29,37 +36,46 @@ func newSignedHostServer(
) (*Handlers, http.Handler) { ) (*Handlers, http.Handler) {
t.Helper() t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{ cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(), StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
}) MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err) t.Fatalf("imgcache.NewCache() error = %v", err)
} }
jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255}) jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
svc, err := imgcache.NewService(&imgcache.ServiceConfig{ svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache, Cache: cache,
Fetcher: newMockFetcher(fstest.MapFS{ Fetcher: newMockFetcher(fstest.MapFS{
signedHost + photoPath: &fstest.MapFile{Data: jpegData}, signedHost + photoPath: &fstest.MapFile{Data: jpegData},
}), }),
SigningKey: testSigningKey, Processor: imageprocessor.New(imageprocessor.Params{
Logger: log, MaxInputBytes: 50 << 20,
}) MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
Logger: log,
}, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("imgcache.NewService() error = %v", err) t.Fatalf("imageproxy.New() error = %v", err)
} }
encGen, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{ h := &Handlers{
log: log, log: log,
imgSvc: svc, imgSvc: svc,
encGen: encGen, encGen: encGen,
signer: signature.New(testSigningKey),
allowlist: allowlist.New(nil),
} }
r := chi.NewRouter() r := chi.NewRouter()
@@ -105,17 +121,18 @@ func getMaxAge(t *testing.T, srv http.Handler, target string) int {
func TestHandleImage_SignedURL_MaxAgeEndsAtExp(t *testing.T) { func TestHandleImage_SignedURL_MaxAgeEndsAtExp(t *testing.T) {
t.Parallel() t.Parallel()
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler)) _, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
signedURL, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{ signedURL := generateSignedURL(testSigningKey, &imagerequest.Request{
SourceHost: signedHost, Source: imagerequest.Source{
SourcePath: photoPath, Host: signedHost,
Size: imgcache.Size{Width: 50, Height: 50}, Path: photoPath,
Format: imgcache.FormatJPEG, },
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, time.Minute) }, time.Minute)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
maxAge := getMaxAge(t, srv, signedURL) maxAge := getMaxAge(t, srv, signedURL)
if maxAge < 50 || maxAge > 60 { if maxAge < 50 || maxAge > 60 {
@@ -168,12 +185,12 @@ func TestHandleImageEnc_MaxAge(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
expiresAt int64 ttl time.Duration
wantAtLeast int wantAtLeast int
wantAtMost int wantAtMost int
}{ }{
{"60 second TTL", time.Now().Add(time.Minute).Unix(), 50, 60}, {"60 second TTL", time.Minute, 50, 60},
{"two-year TTL", time.Now().Add(2 * 365 * 24 * time.Hour).Unix(), 31536000, 31536000}, {"two-year TTL", 2 * 365 * 24 * time.Hour, 31536000, 31536000},
{"no TTL", 0, 31536000, 31536000}, {"no TTL", 0, 31536000, 31536000},
} }
@@ -183,14 +200,24 @@ func TestHandleImageEnc_MaxAge(t *testing.T) {
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler)) h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
token, err := h.encGen.Generate(&encurl.Payload{ // The expiry is set here, not in the table: a parallel subtest
SourceHost: signedHost, // can wait behind other tests for more than the 10 seconds the
SourcePath: photoPath, // 60 second case allows.
Width: 50, var expires time.Time
Height: 50, if tt.ttl != 0 {
Format: imgcache.FormatJPEG, expires = time.Now().Add(tt.ttl)
ExpiresAt: tt.expiresAt, }
})
token, err := h.encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, expires)
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
internal/handlers/image_errors_internal_test.go
+352 -23
View File
@@ -1,19 +1,30 @@
package handlers package handlers
import ( import (
"bytes"
"context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"image/color" "image/color"
"io"
"log/slog" "log/slog"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"slices"
"strings" "strings"
"syscall"
"testing" "testing"
"testing/fstest" "testing/fstest"
"time" "time"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/signature" "sneak.berlin/go/pixa/internal/signature"
) )
@@ -23,29 +34,38 @@ import (
const allowlistedHost = "allowed.example.com" const allowlistedHost = "allowed.example.com"
// newImageRoute returns the image route of a Handlers whose service fetches // newImageRoute returns the image route of a Handlers whose service fetches
// with fetcher and checks signatures with testSigningKey. // with fetcher, and which checks signatures with testSigningKey.
func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler { func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
t.Helper() t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{ cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(), StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
}) MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("failed to create cache: %v", err) t.Fatalf("failed to create cache: %v", err)
} }
svc, err := imgcache.NewService(&imgcache.ServiceConfig{ svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache, Cache: cache,
Fetcher: fetcher, Fetcher: fetcher,
SigningKey: testSigningKey, Processor: imageprocessor.New(imageprocessor.Params{
Allowlist: []string{allowlistedHost}, MaxInputBytes: 50 << 20,
}) MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
}, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("failed to create service: %v", err) t.Fatalf("failed to create proxy: %v", err)
} }
h := &Handlers{imgSvc: svc, log: slog.New(slog.DiscardHandler)} h := &Handlers{
imgSvc: svc,
log: slog.New(slog.DiscardHandler),
signer: signature.New(testSigningKey),
allowlist: allowlist.New([]string{allowlistedHost}),
}
r := chi.NewRouter() r := chi.NewRouter()
r.Get("/v1/image/*", h.HandleImage()) r.Get("/v1/image/*", h.HandleImage())
@@ -54,7 +74,7 @@ func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
} }
// newPhotoFetcher returns a mock fetcher that serves a JPEG at photoPath on // newPhotoFetcher returns a mock fetcher that serves a JPEG at photoPath on
// each of hosts, and answers any other URL with an upstream error. // each of hosts, and answers any other source with an upstream error.
func newPhotoFetcher(t *testing.T, hosts ...string) *httpfetcher.MockFetcher { func newPhotoFetcher(t *testing.T, hosts ...string) *httpfetcher.MockFetcher {
t.Helper() t.Helper()
@@ -81,19 +101,27 @@ func photoURLWithSig(host, sig string, expires time.Time) string {
return fmt.Sprintf("%s?sig=%s&exp=%d", photoURL(host), sig, expires.Unix()) return fmt.Sprintf("%s?sig=%s&exp=%d", photoURL(host), sig, expires.Unix())
} }
// photoRequest returns the image request of photoURL(host) at the default
// quality and fit.
func photoRequest(host string) *imagerequest.Request {
return &imagerequest.Request{
Source: imagerequest.Source{
Host: host,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
}
// photoSignature returns the signature of photoURL(host) at the default // photoSignature returns the signature of photoURL(host) at the default
// quality and fit, made with key and expiring at expires. // quality and fit, made with key and expiring at expires.
func photoSignature(key, host string, expires time.Time) string { func photoSignature(key, host string, expires time.Time) string {
return signature.New(key).Sign(&signature.Request{ return signature.New(key).Sign(photoRequest(host), expires)
SourceHost: host,
SourcePath: photoPath,
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: expires,
})
} }
// sendGet sends a GET for target to route and returns the response. // sendGet sends a GET for target to route and returns the response.
@@ -164,7 +192,12 @@ func TestHandleImage_ErrorAnswers(t *testing.T) {
photos := newPhotoFetcher(t, photos := newPhotoFetcher(t,
signedHost, parentHost, siblingHost, subdomainHost, appendedHost) signedHost, parentHost, siblingHost, subdomainHost, appendedHost)
// The real fetcher refuses localhost before any lookup or connection. // The real fetcher refuses localhost before any lookup or connection.
realFetcher := httpfetcher.New(httpfetcher.DefaultConfig()) realFetcher := httpfetcher.New(&httpfetcher.Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
})
exp := time.Now().Add(time.Hour) exp := time.Now().Add(time.Hour)
expired := time.Now().Add(-time.Hour) expired := time.Now().Add(-time.Hour)
@@ -264,3 +297,299 @@ func TestHandleImage_AllowlistOrSignature(t *testing.T) {
}) })
} }
} }
// TestHandleImage_InvalidHost_Returns400 sends image URLs whose host no URL
// host can be, and checks that each is refused with 400 naming the host as
// decoded: allowlistedHost with a character after it that a URL host cannot
// hold, percent-encoded, an IPv6 address without brackets, and brackets
// around something that is not an IPv6 address. Later steps put the host
// unparsed into the allowlist check, the cache key and the upstream URL, so
// such a host is refused first.
func TestHandleImage_InvalidHost_Returns400(t *testing.T) {
t.Parallel()
photos := newPhotoFetcher(t, allowlistedHost)
tests := []struct {
name string
writtenHost string
decodedHost string
}{
{"? written as %3F", allowlistedHost + "%3Fx=1", allowlistedHost + "?x=1"},
{"# written as %23", allowlistedHost + "%23x", allowlistedHost + "#x"},
{"% written as %25", allowlistedHost + "%253F", allowlistedHost + "%3F"},
{`\ written as %5C`, allowlistedHost + "%5Cx", allowlistedHost + `\x`},
{"space written as %20", allowlistedHost + "%20", allowlistedHost + " "},
{"IPv6 address without brackets", "2001:db8::1", "2001:db8::1"},
{"IPv4 address in brackets", "[203.0.113.5]", "[203.0.113.5]"},
{"no address in brackets", "[abc]", "[abc]"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
rec := sendGet(t, newImageRoute(t, photos), photoURL(tt.writtenHost))
checkErrorBody(t, rec, http.StatusBadRequest,
fmt.Sprintf("invalid image URL: invalid source host: %q",
tt.decodedHost))
})
}
}
// TestHandleImage_HostForms_Served checks that the image route serves, with
// a valid signature, the image at photoPath on a host name, an IPv4 address,
// an IPv6 address in brackets, and a host followed by ":" with or without a
// port.
func TestHandleImage_HostForms_Served(t *testing.T) {
t.Parallel()
hosts := []string{
allowlistedHost,
"img_1.example.com",
"203.0.113.5",
"[2001:db8::1]",
allowlistedHost + ":8443",
"[2001:db8::1]:8443",
allowlistedHost + ":",
}
photos := newPhotoFetcher(t, hosts...)
exp := time.Now().Add(time.Hour)
for _, host := range hosts {
t.Run(host, func(t *testing.T) {
t.Parallel()
sig := photoSignature(testSigningKey, host, exp)
rec := sendGet(t, newImageRoute(t, photos),
photoURLWithSig(host, sig, exp))
if rec.Code != http.StatusOK {
t.Errorf("status = %d, want %d: %s",
rec.Code, http.StatusOK, rec.Body)
}
})
}
}
// TestHandleImage_HashInSourcePathOrQuery_Returns400 sends image URLs whose
// source path or source query holds a "#", written as %23, and checks that
// each is refused with 400, the one signed for that source path included.
// A "#" starts a URL's fragment, so neither the source path nor the source
// query may hold one.
func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
t.Parallel()
photos := newPhotoFetcher(t, allowlistedHost, signedHost)
exp := time.Now().Add(time.Hour)
sig := signature.New(testSigningKey).Sign(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath + "#x",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}, exp)
tests := []struct {
name string
target string
}{
{"# in the source path",
"/v1/image/" + allowlistedHost + photoPath + "%23x/50x50.jpeg"},
{"# in the source query",
"/v1/image/" + allowlistedHost + photoPath + "%3Fv=1%23x/50x50.jpeg"},
{"# in the source path, signed",
fmt.Sprintf("/v1/image/%s%s%%23x/50x50.jpeg?sig=%s&exp=%d",
signedHost, photoPath, sig, exp.Unix())},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
rec := sendGet(t, newImageRoute(t, photos), tt.target)
checkErrorBody(t, rec, http.StatusBadRequest,
`invalid image URL: invalid image path: `+
`the source path or query holds a "#"`)
})
}
}
// failingFetcher answers every fetch with its error.
type failingFetcher struct {
err error
}
// Fetch returns f's error.
func (f failingFetcher) Fetch(
context.Context, imagerequest.Source, string, string,
) (*httpfetcher.FetchResult, error) {
return nil, f.err
}
// photoURLsByRoute returns, for each image route, the function that makes the
// URL of the JPEG at photoPath on signedHost on that route.
func photoURLsByRoute() map[string]func(*testing.T, *Handlers) string {
return map[string]func(*testing.T, *Handlers) string{
"/v1/image/": signedPhotoURL,
"/v1/e/": encPhotoURL,
}
}
// TestImageRoutes_SameErrorAnswers checks that both image routes answer each
// upstream fetch error below with the same status. They log it as "failed to
// get image" at the same level: error for the 500 and warn for the others.
// Each request gets routes of its own, so no request finds another's failed
// fetch in the negative cache.
func TestImageRoutes_SameErrorAnswers(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
wantStatus int
wantLevel slog.Level
}{
{"blocked upstream address", httpfetcher.ErrSSRFBlocked,
http.StatusForbidden, slog.LevelWarn},
{"upstream error status",
fmt.Errorf("%w: status 404", httpfetcher.ErrUpstreamError),
http.StatusBadGateway, slog.LevelWarn},
{"upstream host with no address",
fmt.Errorf("%w: cdn.example.com", httpfetcher.ErrInvalidHost),
http.StatusBadGateway, slog.LevelWarn},
{"connection refused",
fmt.Errorf("%w: %w", httpfetcher.ErrUpstreamRequestFailed,
syscall.ECONNREFUSED),
http.StatusBadGateway, slog.LevelWarn},
{"no response headers within the fetch timeout",
httpfetcher.ErrUpstreamTimeout, http.StatusGatewayTimeout,
slog.LevelWarn},
{"no free upstream connection", httpfetcher.ErrTooManyConnections,
http.StatusServiceUnavailable, slog.LevelWarn},
{"Content-Type not a source format",
fmt.Errorf("%w: image/svg+xml", httpfetcher.ErrInvalidContentType),
http.StatusUnsupportedMediaType, slog.LevelWarn},
{"any other failure", io.ErrUnexpectedEOF,
http.StatusInternalServerError, slog.LevelError},
}
for _, tt := range tests {
for route, imageURL := range photoURLsByRoute() {
t.Run(route+" "+tt.name, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h, routes := newRoutesFetchingThrough(t,
failingFetcher{err: tt.err},
slog.New(slog.NewJSONHandler(&logs, nil)))
rec := sendGet(t, routes, imageURL(t, h))
if rec.Code != tt.wantStatus {
t.Errorf("status = %d, want %d: %s",
rec.Code, tt.wantStatus, rec.Body)
}
levels := loggedLevels(t, &logs)
if !slices.Contains(levels[tt.wantLevel.String()],
"failed to get image") {
t.Errorf(`no "failed to get image" line logged at level %s`,
tt.wantLevel)
}
if tt.wantLevel != slog.LevelError &&
len(levels[slog.LevelError.String()]) != 0 {
t.Errorf("logged at error level: %q",
levels[slog.LevelError.String()])
}
})
}
}
}
// failingResponseWriter is a response writer whose Write always fails, as when
// the client has gone away.
type failingResponseWriter struct {
*httptest.ResponseRecorder
}
// Write writes nothing and returns io.ErrClosedPipe.
func (failingResponseWriter) Write([]byte) (int, error) {
return 0, io.ErrClosedPipe
}
// TestImageRoutes_FailedWriteLoggedOnce checks that both image routes log a
// failed write of the image once, at warn level, with the request's ID, the
// source host and path, the bytes served and the error. Neither may also log
// the image as served.
func TestImageRoutes_FailedWriteLoggedOnce(t *testing.T) {
t.Parallel()
const requestID = "test-request-id"
for route, imageURL := range photoURLsByRoute() {
t.Run(route, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h, srv := newSignedHostServer(t,
slog.New(slog.NewJSONHandler(&logs, nil)))
ctx := context.WithValue(t.Context(),
middleware.RequestIDKey, requestID)
srv.ServeHTTP(failingResponseWriter{httptest.NewRecorder()},
httptest.NewRequestWithContext(
ctx, http.MethodGet, imageURL(t, h), nil))
t.Logf("logged:\n%s", logs.String())
var failedWrites []map[string]any
dec := json.NewDecoder(&logs)
for dec.More() {
var line map[string]any
err := dec.Decode(&line)
if err != nil {
t.Fatalf("decoding log line: %v", err)
}
switch line["msg"] {
case "image served":
t.Error(`"image served" logged after a failed write`)
case "failed to write response":
failedWrites = append(failedWrites, line)
}
}
if len(failedWrites) != 1 {
t.Fatalf(`"failed to write response" logged %d times, want 1`,
len(failedWrites))
}
want := map[string]any{
"level": slog.LevelWarn.String(),
"request_id": requestID,
"host": signedHost,
"path": photoPath,
"served_bytes": float64(0),
"error": io.ErrClosedPipe.Error(),
}
for key, wantValue := range want {
if got := failedWrites[0][key]; got != wantValue {
t.Errorf("%s = %v, want %v", key, got, wantValue)
}
}
})
}
}
internal/handlers/image_signature_internal_test.go
+240 -29
View File
@@ -5,11 +5,12 @@ import (
"fmt" "fmt"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/url"
"testing" "testing"
"time" "time"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/signature" "sneak.berlin/go/pixa/internal/signature"
) )
@@ -36,6 +37,47 @@ func getImage(t *testing.T, fix *testFixtures, target string) int {
return rec.Code return rec.Code
} }
// generateSignedURL returns the /v1/image/ URL of req, signed with key and
// expiring after ttl. It signs a copy of req, so req itself is not changed.
// An unset quality or fit in the copy is set to 85 or cover, the values the
// route uses when a URL has no q or fit. The URL carries q and fit next to sig
// and exp, so the route verifies it for the quality and fit it was signed
// with.
func generateSignedURL(
key string, req *imagerequest.Request, ttl time.Duration,
) string {
signed := *req
if signed.Transform.Quality == 0 {
signed.Transform.Quality = 85
}
if signed.Transform.FitMode == "" {
signed.Transform.FitMode = imagerequest.FitCover
}
expires := time.Now().Add(ttl)
sig := signature.New(key).Sign(&signed, expires)
// A source query follows the source path after a "?". The image route
// reads the path decoded once, so the whole path is written in Go's own
// percent-encoding of a URL path, which writes "?" as %3F and "%" as %25,
// and leaves "/", ";", ",", "=" and "&" as they are.
sourcePathAndQuery := signed.Source.Path
if signed.Source.Query != "" {
sourcePathAndQuery += "?" + signed.Source.Query
}
unescapedPath := fmt.Sprintf("/v1/image/%s%s/%dx%d.%s",
signed.Source.Host, sourcePathAndQuery,
signed.Transform.Size.Width, signed.Transform.Size.Height,
signed.Transform.Format)
return fmt.Sprintf("%s?sig=%s&exp=%d&q=%d&fit=%s",
(&url.URL{Path: unescapedPath}).EscapedPath(), sig, expires.Unix(),
signed.Transform.Quality, signed.Transform.FitMode)
}
// TestHandleImage_SignatureCoversQualityAndFit signs a URL for quality 85 // TestHandleImage_SignatureCoversQualityAndFit signs a URL for quality 85
// and fit cover, the values the route uses when a URL has no q or fit, and // and fit cover, the values the route uses when a URL has no q or fit, and
// sends that signature with each q and fit below. // sends that signature with each q and fit below.
@@ -44,16 +86,18 @@ func TestHandleImage_SignatureCoversQualityAndFit(t *testing.T) {
expires := time.Now().Add(time.Hour) expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars") signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&signature.Request{ sig := signer.Sign(&imagerequest.Request{
SourceHost: signedHost, Source: imagerequest.Source{
SourcePath: "/images/photo.jpg", Host: signedHost,
Width: 50, Path: "/images/photo.jpg",
Height: 50, },
Format: string(imgcache.FormatJPEG), Transform: imagerequest.Transform{
Quality: 85, Size: imagerequest.Size{Width: 50, Height: 50},
FitMode: string(imgcache.FitCover), Format: imagerequest.FormatJPEG,
Expires: expires, Quality: 85,
}) FitMode: imagerequest.FitCover,
},
}, expires)
signedURL := fmt.Sprintf("/v1/image/%s/images/photo.jpg/50x50.jpeg?sig=%s&exp=%d", signedURL := fmt.Sprintf("/v1/image/%s/images/photo.jpg/50x50.jpeg?sig=%s&exp=%d",
signedHost, sig, expires.Unix()) signedHost, sig, expires.Unix())
@@ -80,17 +124,17 @@ func TestHandleImage_SignatureCoversQualityAndFit(t *testing.T) {
} }
} }
// TestHandleImage_GeneratedSignedURLVerifies sends URLs built by the // TestHandleImage_GeneratedSignedURLVerifies sends URLs built by
// service's signed-URL generator to the route. // generateSignedURL to the route.
func TestHandleImage_GeneratedSignedURLVerifies(t *testing.T) { func TestHandleImage_GeneratedSignedURLVerifies(t *testing.T) {
t.Parallel() t.Parallel()
tests := []struct { tests := []struct {
name string name string
quality int quality int
fitMode imgcache.FitMode fitMode imagerequest.Fit
}{ }{
{"quality 40 and fit contain", 40, imgcache.FitContain}, {"quality 40 and fit contain", 40, imagerequest.FitContain},
{"quality and fit unset", 0, ""}, {"quality and fit unset", 0, ""},
} }
@@ -98,21 +142,21 @@ func TestHandleImage_GeneratedSignedURLVerifies(t *testing.T) {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
fix := setupTestHandler(t) signedURL := generateSignedURL("test-signing-key-must-be-32-chars",
&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: "/images/photo.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: tt.quality,
FitMode: tt.fitMode,
},
}, time.Hour)
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{ status := getImage(t, setupTestHandler(t), signedURL)
SourceHost: signedHost,
SourcePath: "/images/photo.jpg",
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
Quality: tt.quality,
FitMode: tt.fitMode,
}, time.Hour)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
status := getImage(t, fix, signedURL)
if status != http.StatusBadGateway { if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway) t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
} }
@@ -169,3 +213,170 @@ func TestHandleImage_InvalidExp_Returns400(t *testing.T) {
}) })
} }
} }
// TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies sends URLs built
// by generateSignedURL for source URLs with a query to the route, through a
// chi router: chi routes on the path as the client wrote it whenever that is
// not Go's own encoding of it. A 502 means the signature passed, so the route
// read the source path and query that were signed.
func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
t.Parallel()
tests := []struct {
name string
query string
}{
{"slash and colon", "url=https://origin.example.com/cat.jpg"},
{"semicolon", "a=1;b=2"},
{"comma", "crop=0,0,50,50"},
{"equals and ampersand", "a=1&b=2"},
{"percent-escape", "name=my%20cat"},
{"question mark", "next=/a?b=1"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
signedURL := generateSignedURL("test-signing-key-must-be-32-chars",
&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
Query: tt.query,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, time.Hour)
status := getImage(t, setupTestHandler(t), signedURL)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
// TestHandleImage_SourceQueryEncodedAnyWay signs one source URL with a query
// and writes it into the image URL in ways the README allows: the "?" before
// the query and each "%" in it percent-encoded, and every other character as
// itself or percent-encoded. A 502 means the signature passed.
func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
t.Parallel()
expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
Query: "url=https://origin.example.com/my%20cat.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}, expires)
tests := []struct {
name string
writtenQuery string
}{
{"only ? and % encoded",
"%3Furl=https://origin.example.com/my%2520cat.jpg"},
{"every reserved character encoded",
"%3Furl%3Dhttps%3A%2F%2Forigin.example.com%2Fmy%2520cat.jpg"},
{"lower-case hex digits",
"%3furl=https://origin.example.com/my%2520cat.jpg"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
target := fmt.Sprintf("/v1/image/%s%s%s/50x50.jpeg?sig=%s&exp=%d",
signedHost, photoPath, tt.writtenQuery, sig, expires.Unix())
status := getImage(t, setupTestHandler(t), target)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
// TestHandleImage_SourceURLHoldingPercent25Verifies signs a source path and a
// source query that each hold %25, the percent-escape of a literal "%", and
// sends each to the route written as the README says, with that "%" written as
// %25, and as generateSignedURL writes it. A 502 means the signature passed.
func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
t.Parallel()
tests := []struct {
name string
sourcePath string
sourceQuery string
// writtenPathAndQuery is the source path and query as the README says
// to write them into the image URL
writtenPathAndQuery string
}{
{"path", "/sale/50%25off.jpg", "", "/sale/50%2525off.jpg"},
{"query", photoPath, "discount=50%25", photoPath + "%3Fdiscount=50%2525"},
}
for _, tt := range tests {
t.Run(tt.name+" written by the README rule", func(t *testing.T) {
t.Parallel()
expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: tt.sourcePath,
Query: tt.sourceQuery,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}, expires)
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
signedHost, tt.writtenPathAndQuery, sig, expires.Unix())
status := getImage(t, setupTestHandler(t), target)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
t.Run(tt.name+" written by generateSignedURL", func(t *testing.T) {
t.Parallel()
signedURL := generateSignedURL("test-signing-key-must-be-32-chars",
&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: tt.sourcePath,
Query: tt.sourceQuery,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, time.Hour)
status := getImage(t, setupTestHandler(t), signedURL)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
internal/handlers/imageenc.go
+19 -113
View File
@@ -2,19 +2,13 @@ package handlers
import ( import (
"errors" "errors"
"fmt"
"io"
"net/http" "net/http"
"strconv"
"time" "time"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imgcache"
) )
// HandleImageEnc handles requests to /v1/e/{token}/* for encrypted // HandleImageEnc handles requests to /v1/e/{token}/* for encrypted
@@ -26,147 +20,59 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
return return
} }
ctx := r.Context() req, expires, ok := s.parseImageEncRequest(w, r)
start := time.Now() if !ok {
req, ok := s.parseImageEncRequest(w, r)
if !ok || !s.chooseAutoFormat(w, r, req) {
return return
} }
// Log the request s.serveImage(w, r, req, expires)
s.log.Debug("encrypted image request",
"host", req.SourceHost,
"path", req.SourcePath,
"dimensions", fmt.Sprintf("%dx%d", req.Size.Width, req.Size.Height),
"format", req.Format,
)
// Fetch and process the image (no signature validation
// needed - encrypted URL is trusted)
resp, err := s.imgSvc.Get(ctx, req)
if err != nil {
s.handleImageError(w, err)
return
}
defer func() { _ = resp.Content.Close() }()
// Set response headers
w.Header().Set("Content-Type", resp.ContentType)
if resp.ContentLength > 0 {
w.Header().Set("Content-Length", strconv.FormatInt(resp.ContentLength, 10))
}
// Cache headers: max-age ends at the URL's expiry
w.Header().Set("Cache-Control", cacheControl(req.Expires))
w.Header().Set("X-Pixa-Cache", string(resp.CacheStatus))
if notModified(w, r, resp.ETag) {
return
}
// A HEAD request gets the headers only
if r.Method == http.MethodHead {
w.WriteHeader(http.StatusOK)
return
}
// Stream the response
written, err := io.Copy(w, resp.Content)
if err != nil {
s.log.Error("failed to write response", "error", err)
return
}
// Log completion
duration := time.Since(start)
s.log.Info("image served",
"request_id", middleware.GetReqID(ctx),
"cache_key", imgcache.CacheKey(req),
"host", req.SourceHost,
"path", req.SourcePath,
"format", req.Format,
"cache_status", resp.CacheStatus,
"served_bytes", written,
"duration_ms", duration.Milliseconds(),
)
} }
} }
// parseImageEncRequest decrypts the token of an encrypted image URL into an // parseImageEncRequest decrypts the token of an encrypted image URL into an
// ImageRequest and checks it. On a token that is missing, does not decrypt, // image request and checks it, and returns it with the token's expiry. On a
// has expired or asks for something not valid, it writes an error response // token that is missing, does not decrypt, has expired or asks for something
// and returns false. // not valid, it writes an error response and returns false.
func (s *Handlers) parseImageEncRequest( func (s *Handlers) parseImageEncRequest(
w http.ResponseWriter, r *http.Request, w http.ResponseWriter, r *http.Request,
) (*imgcache.ImageRequest, bool) { ) (*imagerequest.Request, time.Time, bool) {
// Extract token from URL // Extract token from URL
token := chi.URLParam(r, "token") token := chi.URLParam(r, "token")
if token == "" { if token == "" {
s.respondError(w, "missing token", http.StatusBadRequest) s.respondError(w, "missing token", http.StatusBadRequest)
return nil, false return nil, time.Time{}, false
} }
// Decrypt and validate the payload // Decrypt and validate the payload
payload, err := s.encGen.Parse(token) req, expires, err := s.encGen.Parse(token)
if err != nil { if err != nil {
if errors.Is(err, encurl.ErrExpired) { if errors.Is(err, encurl.ErrExpired) {
s.log.Debug("encrypted URL expired", "error", err) s.log.Debug("encrypted URL expired", "error", err)
s.respondError(w, "URL has expired", http.StatusGone) s.respondError(w, "URL has expired", http.StatusGone)
return nil, false return nil, time.Time{}, false
} }
s.log.Debug("failed to decrypt URL", "error", err) s.log.Debug("invalid encrypted URL", "error", err)
s.respondError(w, "invalid encrypted URL", http.StatusBadRequest) s.respondError(w, "invalid encrypted URL", http.StatusBadRequest)
return nil, false return nil, time.Time{}, false
} }
// Convert payload to ImageRequest
req := payload.ToImageRequest()
// Apply the same dimension and fit-mode bounds as the plain image // Apply the same dimension and fit-mode bounds as the plain image
// route: a sealed payload is trusted for its origin, not for staying // route: a sealed payload is trusted for its origin, not for staying
// within limits, so an over-limit size or unknown fit mode is a 400 // within limits. This refuses an over-limit size before the source is
// here rather than an out-of-memory or a 500 from the processor. // fetched. An unknown fit mode is a 400 here rather than a 500 from the
err = imgcache.ValidateImageRequest(req) // processor.
err = imagerequest.ValidateImageRequest(req)
if err != nil { if err != nil {
s.log.Debug("encrypted URL failed validation", "error", err) s.log.Debug("encrypted URL failed validation", "error", err)
s.respondError(w, "invalid encrypted URL: "+err.Error(), s.respondError(w, "invalid encrypted URL: "+err.Error(),
http.StatusBadRequest) http.StatusBadRequest)
return nil, false return nil, time.Time{}, false
} }
return req, true return req, expires, true
}
// handleImageError converts image service errors to HTTP responses.
func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
switch {
case errors.Is(err, httpfetcher.ErrSSRFBlocked):
s.respondError(w, "forbidden", http.StatusForbidden)
case errors.Is(err, httpfetcher.ErrUpstreamError):
s.respondError(w, "upstream error", http.StatusBadGateway)
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
s.respondError(w, "upstream timeout", http.StatusGatewayTimeout)
case errors.Is(err, httpfetcher.ErrTooManyConnections),
errors.Is(err, imageprocessor.ErrTooManyImages):
s.respondError(w, "server busy, try again later",
http.StatusServiceUnavailable)
case errors.Is(err, httpfetcher.ErrInvalidContentType),
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat):
s.respondError(w, "unsupported source format",
http.StatusUnsupportedMediaType)
default:
s.log.Error("image request failed", "error", err)
s.respondError(w, "internal error", http.StatusInternalServerError)
}
} }
internal/handlers/imageenc_internal_test.go
+26 -16
View File
@@ -6,11 +6,13 @@ import (
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"testing" "testing"
"time"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
) )
// newEncTestServer builds a router serving the encrypted-URL route with a // newEncTestServer builds a router serving the encrypted-URL route with a
@@ -20,11 +22,13 @@ import (
func newEncTestServer(t *testing.T) (*encurl.Generator, http.Handler) { func newEncTestServer(t *testing.T) (*encurl.Generator, http.Handler) {
t.Helper() t.Helper()
encGen, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{ h := &Handlers{
log: slog.New(slog.DiscardHandler), log: slog.New(slog.DiscardHandler),
encGen: encGen, encGen: encGen,
@@ -54,12 +58,15 @@ func TestHandleImageEnc_OverLimitDimension_Returns400(t *testing.T) {
encGen, srv := newEncTestServer(t) encGen, srv := newEncTestServer(t)
token, err := encGen.Generate(&encurl.Payload{ token, err := encGen.Generate(&imagerequest.Request{
SourceHost: "cdn.example.com", Source: imagerequest.Source{
SourcePath: "/photo.jpg", Host: "cdn.example.com",
Width: 100000, Path: "/photo.jpg",
Height: 100000, },
}) Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 100000, Height: 100000},
},
}, time.Time{})
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
@@ -79,13 +86,16 @@ func TestHandleImageEnc_InvalidFitMode_Returns400(t *testing.T) {
encGen, srv := newEncTestServer(t) encGen, srv := newEncTestServer(t)
token, err := encGen.Generate(&encurl.Payload{ token, err := encGen.Generate(&imagerequest.Request{
SourceHost: "cdn.example.com", Source: imagerequest.Source{
SourcePath: "/photo.jpg", Host: "cdn.example.com",
Width: 800, Path: "/photo.jpg",
Height: 600, },
FitMode: imgcache.FitMode("bogus"), Transform: imagerequest.Transform{
}) Size: imagerequest.Size{Width: 800, Height: 600},
FitMode: imagerequest.Fit("bogus"),
},
}, time.Time{})
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
internal/handlers/imageenc_token_internal_test.go
+17 -13
View File
@@ -9,6 +9,8 @@ import (
"time" "time"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
) )
// requireServedPhoto requires that rec answers 200 with the JPEG at photoPath // requireServedPhoto requires that rec answers 200 with the JPEG at photoPath
@@ -59,32 +61,34 @@ func TestHandleImageEnc_RejectedToken(t *testing.T) {
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler)) h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
photo := encurl.Payload{ photo := imagerequest.Request{
SourceHost: signedHost, Source: imagerequest.Source{
SourcePath: photoPath, Host: signedHost,
Width: 50, Path: photoPath,
Height: 50, },
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
},
} }
valid, err := h.encGen.Generate(&photo) valid, err := h.encGen.Generate(&photo, time.Time{})
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
expiredPhoto := photo expired, err := h.encGen.Generate(&photo, time.Now().Add(-time.Minute))
expiredPhoto.ExpiresAt = time.Now().Add(-time.Minute).Unix()
expired, err := h.encGen.Generate(&expiredPhoto)
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
otherGen, err := encurl.NewGenerator("another-signing-key-fedcba9876543210") keys, err := seal.DeriveKeys("another-signing-key-fedcba9876543210")
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
otherKey, err := otherGen.Generate(&photo) otherGen := encurl.NewGenerator(keys.EncryptedURLKey)
otherKey, err := otherGen.Generate(&photo, time.Time{})
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
internal/handlers/jpegxl_default_internal_test.go
+22 -17
View File
@@ -14,7 +14,7 @@ import (
"github.com/davidbyttow/govips/v2/vips" "github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/signature" "sneak.berlin/go/pixa/internal/signature"
) )
@@ -58,16 +58,18 @@ func TestImageWithoutFormat_SignedAsJXL(t *testing.T) {
route := newImageRoute(t, newPhotoFetcher(t, signedHost)) route := newImageRoute(t, newPhotoFetcher(t, signedHost))
expires := time.Now().Add(time.Hour) expires := time.Now().Add(time.Hour)
sig := signature.New(testSigningKey).Sign(&signature.Request{ sig := signature.New(testSigningKey).Sign(&imagerequest.Request{
SourceHost: signedHost, Source: imagerequest.Source{
SourcePath: photoPath, Host: signedHost,
Width: 50, Path: photoPath,
Height: 50, },
Format: string(imgcache.FormatJXL), Transform: imagerequest.Transform{
Quality: encurl.DefaultQuality, Size: imagerequest.Size{Width: 50, Height: 50},
FitMode: string(imgcache.FitCover), Format: imagerequest.FormatJXL,
Expires: expires, Quality: encurl.DefaultQuality,
}) FitMode: imagerequest.FitCover,
},
}, expires)
query := fmt.Sprintf("?sig=%s&exp=%d", sig, expires.Unix()) query := fmt.Sprintf("?sig=%s&exp=%d", sig, expires.Unix())
for _, size := range []string{"50x50.jxl", "50x50"} { for _, size := range []string{"50x50.jxl", "50x50"} {
@@ -83,12 +85,15 @@ func TestImageEncWithoutFormat_ServesJPEGXL(t *testing.T) {
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler)) h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
token, err := h.encGen.Generate(&encurl.Payload{ token, err := h.encGen.Generate(&imagerequest.Request{
SourceHost: signedHost, Source: imagerequest.Source{
SourcePath: photoPath, Host: signedHost,
Width: 50, Path: photoPath,
Height: 50, },
}) Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
},
}, time.Time{})
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
internal/handlers/jpegxl_internal_test.go
+6 -6
View File
@@ -7,7 +7,7 @@ import (
"github.com/davidbyttow/govips/v2/vips" "github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imagerequest"
) )
// jxlType is the content type of JPEG XL. // jxlType is the content type of JPEG XL.
@@ -22,14 +22,14 @@ func TestFormatForAccept_JPEGXL(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
accept string accept string
want imgcache.ImageFormat want imagerequest.Format
}{ }{
{"JPEG XL-capable browser", {"JPEG XL-capable browser",
"image/jxl,image/avif,image/webp,image/*,*/*;q=0.8", imgcache.FormatJXL}, "image/jxl,image/avif,image/webp,image/*,*/*;q=0.8", imagerequest.FormatJXL},
{"JPEG XL only", jxlType, imgcache.FormatJXL}, {"JPEG XL only", jxlType, imagerequest.FormatJXL},
{"JPEG XL with a lower q than AVIF", "image/avif,image/jxl;q=0.5", {"JPEG XL with a lower q than AVIF", "image/avif,image/jxl;q=0.5",
imgcache.FormatJXL}, imagerequest.FormatJXL},
{"q=0 on JPEG XL", "image/jxl;q=0,image/avif,*/*", imgcache.FormatAVIF}, {"q=0 on JPEG XL", "image/jxl;q=0,image/avif,*/*", imagerequest.FormatAVIF},
} }
for _, tt := range tests { for _, tt := range tests {
internal/handlers/referer_blocklist_internal_test.go
+31 -16
View File
@@ -10,10 +10,16 @@ import (
"time" "time"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist" "sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
) )
// blockedReferer is a page on leech.example, which newRefererRoutes puts on // blockedReferer is a page on leech.example, which newRefererRoutes puts on
@@ -29,11 +35,12 @@ type countingFetcher struct {
// Fetch counts the fetch and passes it on. // Fetch counts the fetch and passes it on.
func (f *countingFetcher) Fetch( func (f *countingFetcher) Fetch(
ctx context.Context, url string, ctx context.Context, source imagerequest.Source,
ifNoneMatch, ifModifiedSince string,
) (*httpfetcher.FetchResult, error) { ) (*httpfetcher.FetchResult, error) {
f.fetches.Add(1) f.fetches.Add(1)
return f.Fetcher.Fetch(ctx, url) return f.Fetcher.Fetch(ctx, source, ifNoneMatch, ifModifiedSince)
} }
// newRefererRoutes returns both image routes of a Handlers whose // newRefererRoutes returns both image routes of a Handlers whose
@@ -47,33 +54,41 @@ func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher)
Fetcher: newPhotoFetcher(t, allowlistedHost, signedHost), Fetcher: newPhotoFetcher(t, allowlistedHost, signedHost),
} }
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{ cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(), StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
}) MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err) t.Fatalf("imgcache.NewCache() error = %v", err)
} }
svc, err := imgcache.NewService(&imgcache.ServiceConfig{ svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache, Cache: cache,
Fetcher: fetcher, Fetcher: fetcher,
SigningKey: testSigningKey, Processor: imageprocessor.New(imageprocessor.Params{
Allowlist: []string{allowlistedHost}, MaxInputBytes: 50 << 20,
}) MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
}, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("imgcache.NewService() error = %v", err) t.Fatalf("imageproxy.New() error = %v", err)
} }
encGen, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{ h := &Handlers{
log: slog.New(slog.DiscardHandler), log: slog.New(slog.DiscardHandler),
imgSvc: svc, imgSvc: svc,
encGen: encGen, encGen: encGen,
signer: signature.New(testSigningKey),
allowlist: allowlist.New([]string{allowlistedHost}),
refererBlocklist: allowlist.New( refererBlocklist: allowlist.New(
[]string{"leech.example", ".hotlinker.example"}), []string{"leech.example", ".hotlinker.example"}),
} }
internal/handlers/request_id_internal_test.go
+24 -21
View File
@@ -13,26 +13,26 @@ import (
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
) )
// signedPhotoURL returns a signed /v1/image/ URL, valid for a minute, for the // signedPhotoURL returns a signed /v1/image/ URL, valid for a minute, for the
// JPEG at photoPath on signedHost at 50x50, made with h's image service. // JPEG at photoPath on signedHost at 50x50. It is signed with testSigningKey,
func signedPhotoURL(t *testing.T, h *Handlers) string { // the key the handlers that newSignedHostServer builds check signatures with.
// It takes a Handlers it does not use, so that it has the type of encPhotoURL.
func signedPhotoURL(t *testing.T, _ *Handlers) string {
t.Helper() t.Helper()
signedURL, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{ return generateSignedURL(testSigningKey, &imagerequest.Request{
SourceHost: signedHost, Source: imagerequest.Source{
SourcePath: photoPath, Host: signedHost,
Size: imgcache.Size{Width: 50, Height: 50}, Path: photoPath,
Format: imgcache.FormatJPEG, },
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, time.Minute) }, time.Minute)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
return signedURL
} }
// encPhotoURL returns an encrypted /v1/e/ URL, which never expires, for the // encPhotoURL returns an encrypted /v1/e/ URL, which never expires, for the
@@ -40,13 +40,16 @@ func signedPhotoURL(t *testing.T, h *Handlers) string {
func encPhotoURL(t *testing.T, h *Handlers) string { func encPhotoURL(t *testing.T, h *Handlers) string {
t.Helper() t.Helper()
token, err := h.encGen.Generate(&encurl.Payload{ token, err := h.encGen.Generate(&imagerequest.Request{
SourceHost: signedHost, Source: imagerequest.Source{
SourcePath: photoPath, Host: signedHost,
Width: 50, Path: photoPath,
Height: 50, },
Format: imgcache.FormatJPEG, Transform: imagerequest.Transform{
}) Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, time.Time{})
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
internal/handlers/server_busy_internal_test.go
+46 -21
View File
@@ -1,6 +1,7 @@
package handlers package handlers
import ( import (
"context"
"fmt" "fmt"
"log/slog" "log/slog"
"net/http" "net/http"
@@ -10,40 +11,64 @@ import (
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestServerBusyAnswers503 checks that both image routes answer 503 with // TestServerBusyAnswers503 checks that respondImageError, which answers the
// a clear error when the image service gives up waiting for a free upstream // errors of both image routes, answers 503 with a clear error when the image
// connection or processing slot, wrapped as the service wraps them. // service gives up waiting for a free upstream connection or processing slot,
// wrapped as the service wraps them.
func TestServerBusyAnswers503(t *testing.T) { func TestServerBusyAnswers503(t *testing.T) {
t.Parallel() t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)} h := &Handlers{log: slog.New(slog.DiscardHandler)}
req := &imgcache.ImageRequest{SourceHost: "img.example.com", SourcePath: "/a.jpg"} req := &imagerequest.Request{
Source: imagerequest.Source{Host: "img.example.com", Path: "/a.jpg"},
}
for _, err := range []error{ for _, err := range []error{
fmt.Errorf("upstream fetch failed: %w", httpfetcher.ErrTooManyConnections), fmt.Errorf("upstream fetch failed: %w", httpfetcher.ErrTooManyConnections),
fmt.Errorf("image processing failed: %w", imageprocessor.ErrTooManyImages), fmt.Errorf("image processing failed: %w", imageprocessor.ErrTooManyImages),
} { } {
plain := httptest.NewRecorder() rec := httptest.NewRecorder()
h.respondImageError(plain, req, err) h.respondImageError(rec, req, err)
encrypted := httptest.NewRecorder() if rec.Code != http.StatusServiceUnavailable {
h.handleImageError(encrypted, err) t.Errorf("for %v: status = %d, want %d",
err, rec.Code, http.StatusServiceUnavailable)
}
for route, rec := range map[string]*httptest.ResponseRecorder{ if !strings.Contains(rec.Body.String(), "server busy, try again later") {
"/v1/image/": plain, "/v1/e/": encrypted, t.Errorf("for %v: body = %q, want the server busy error",
} { err, rec.Body.String())
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("%s for %v: status = %d, want %d",
route, err, rec.Code, http.StatusServiceUnavailable)
}
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("%s for %v: body = %q, want the server busy error",
route, err, rec.Body.String())
}
} }
} }
} }
// TestProcessingTimedOutAnswers503 checks that respondImageError, which
// answers the errors of both image routes, answers 503 with the server busy
// error when the image service reports that processing stopped at the
// deadline of the request doing it, wrapped as the service wraps it. A request
// that waited for that processing gets this error while its own deadline may
// have time left. The request's source only goes into a log line, so it is
// left empty.
func TestProcessingTimedOutAnswers503(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
err := fmt.Errorf("%w: failed to encode: %w",
imageproxy.ErrProcessingTimedOut, context.DeadlineExceeded)
rec := httptest.NewRecorder()
h.respondImageError(rec, &imagerequest.Request{}, err)
t.Logf("status = %d, body = %q", rec.Code, rec.Body.String())
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("status = %d, want %d", rec.Code, http.StatusServiceUnavailable)
}
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("body = %q, want the server busy error", rec.Body.String())
}
}
internal/handlers/unsupported_source_internal_test.go
+20 -25
View File
@@ -10,45 +10,40 @@ import (
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestUnsupportedSourceFormatAnswers415 checks that both image routes answer // TestUnsupportedSourceFormatAnswers415 checks that respondImageError, which
// 415 with a clear error when the upstream answers with a content type pixa // answers the errors of both image routes, answers 415 with a clear error when
// does not accept, such as SVG, and when orig is asked of a source in a // the upstream answers with a content type pixa does not accept, such as SVG,
// format the processor cannot name, wrapped as the service wraps them. // and when libvips does not read a source as the format its first bytes
// show, wrapped as the service wraps them.
func TestUnsupportedSourceFormatAnswers415(t *testing.T) { func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
t.Parallel() t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)} h := &Handlers{log: slog.New(slog.DiscardHandler)}
req := &imgcache.ImageRequest{SourceHost: "img.example.com", SourcePath: "/a.svg"} req := &imagerequest.Request{
Source: imagerequest.Source{Host: "img.example.com", Path: "/a.svg"},
}
for _, err := range []error{ for _, err := range []error{
fmt.Errorf("upstream fetch failed: %w: image/svg+xml", fmt.Errorf("upstream fetch failed: %w: image/svg+xml",
httpfetcher.ErrInvalidContentType), httpfetcher.ErrInvalidContentType),
fmt.Errorf("image processing failed: %w: unknown", fmt.Errorf("image processing failed: %w: libvips does not read the input as avif",
imageprocessor.ErrUnsupportedInputFormat), imageprocessor.ErrUnsupportedInputFormat),
} { } {
plain := httptest.NewRecorder() rec := httptest.NewRecorder()
h.respondImageError(plain, req, err) h.respondImageError(rec, req, err)
t.Logf("for %v: %d %s", err, rec.Code, rec.Body.String())
encrypted := httptest.NewRecorder() if rec.Code != http.StatusUnsupportedMediaType {
h.handleImageError(encrypted, err) t.Errorf("for %v: status = %d, want %d",
err, rec.Code, http.StatusUnsupportedMediaType)
}
for route, rec := range map[string]*httptest.ResponseRecorder{ if !strings.Contains(rec.Body.String(), "unsupported source format") {
"HandleImage": plain, "HandleImageEnc": encrypted, t.Errorf("for %v: body = %q, want the unsupported source format error",
} { err, rec.Body.String())
t.Logf("%s for %v: %d %s", route, err, rec.Code, rec.Body.String())
if rec.Code != http.StatusUnsupportedMediaType {
t.Errorf("%s for %v: status = %d, want %d",
route, err, rec.Code, http.StatusUnsupportedMediaType)
}
if !strings.Contains(rec.Body.String(), "unsupported source format") {
t.Errorf("%s for %v: body = %q, want the unsupported source format error",
route, err, rec.Body.String())
}
} }
} }
} }
internal/handlers/validate_signature_internal_test.go
+211
View File
@@ -0,0 +1,211 @@
package handlers
import (
"testing"
"time"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/signature"
)
// Addresses the port tests below put on the allowlist.
const (
allowlistedIPv4 = "203.0.113.5"
allowlistedIPv6 = "2001:db8::1"
)
// newSignatureChecker returns Handlers that check signatures with
// testSigningKey and have allowlistHosts on their allowlist, as New builds
// them from signing_key and allowlist_hosts.
func newSignatureChecker(allowlistHosts ...string) *Handlers {
return &Handlers{
signer: signature.New(testSigningKey),
allowlist: allowlist.New(allowlistHosts),
}
}
func TestValidateSignature_NonAllowlistedHost_NoSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedHost)
// Should fail validation - not allowlisted and no signature
err := h.validateSignature(photoRequest(signedHost), "", time.Time{})
if err == nil {
t.Error("validateSignature() expected error for non-allowlisted " +
"host without signature")
}
}
func TestValidateSignature_NonAllowlistedHost_ValidSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedHost)
req := photoRequest(signedHost)
// Generate a valid signature
expires := time.Now().Add(time.Hour)
sig := signature.New(testSigningKey).Sign(req, expires)
// Should pass validation
err := h.validateSignature(req, sig, expires)
if err != nil {
t.Errorf("validateSignature() error = %v", err)
}
}
func TestValidateSignature_NonAllowlistedHost_ExpiredSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedHost)
req := photoRequest(signedHost)
// Generate an expired signature
expires := time.Now().Add(-time.Hour)
sig := signature.New(testSigningKey).Sign(req, expires)
// Should fail validation
err := h.validateSignature(req, sig, expires)
if err == nil {
t.Error("validateSignature() expected error for expired signature")
}
}
func TestValidateSignature_NonAllowlistedHost_InvalidSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedHost)
// Should fail validation
err := h.validateSignature(photoRequest(signedHost), "invalid-signature",
time.Now().Add(time.Hour))
if err == nil {
t.Error("validateSignature() expected error for invalid signature")
}
}
// TestValidateSignature_NoAllowlist checks that with no host on the
// allowlist, a request without a signature fails validation.
func TestValidateSignature_NoAllowlist(t *testing.T) {
t.Parallel()
h := newSignatureChecker()
err := h.validateSignature(photoRequest(signedHost), "", time.Time{})
if err == nil {
t.Error("validateSignature() expected error when no host is allowlisted")
}
}
// TestValidateSignature_SignatureExactHostMatch verifies that
// validateSignature enforces exact host matching for signatures. A
// signature for one host must not verify for a different host, even
// if they share a domain suffix.
func TestValidateSignature_SignatureExactHostMatch(t *testing.T) {
t.Parallel()
h := newSignatureChecker()
// Sign a request for signedHost
signedReq := photoRequest(signedHost)
expires := time.Now().Add(time.Hour)
sig := signature.New(testSigningKey).Sign(signedReq, expires)
// The original request should pass validation
t.Run("exact host passes", func(t *testing.T) {
t.Parallel()
err := h.validateSignature(signedReq, sig, expires)
if err != nil {
t.Errorf("validateSignature() exact host failed: %v", err)
}
})
// Try to reuse the signature with different hosts
tests := []struct {
name string
host string
}{
{"parent domain", "example.com"},
{"sibling subdomain", "images.example.com"},
{"deeper subdomain", "a." + signedHost},
{"evil suffix domain", signedHost + ".evil.com"},
{"prefixed host", "evil" + signedHost},
}
for _, tt := range tests {
t.Run(tt.name+" rejected", func(t *testing.T) {
t.Parallel()
err := h.validateSignature(photoRequest(tt.host), sig, expires)
if err == nil {
t.Errorf("validateSignature() should reject signature for "+
"host %q (signed for %q)", tt.host, signedHost)
}
})
}
}
// TestValidateSignature_HostWithPortNeedsSignature verifies that a host
// followed by a port, or by a colon with no port, matches no allowlist entry,
// so that a request for it without a signature is refused, while the same
// host with no port is allowlisted.
func TestValidateSignature_HostWithPortNeedsSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedHost, allowlistedIPv4, allowlistedIPv6)
tests := []struct {
host string
allowlisted bool
}{
{allowlistedHost, true},
{allowlistedHost + ":8443", false},
{allowlistedHost + ":443", false},
{allowlistedHost + ":", false},
{allowlistedIPv4, true},
{allowlistedIPv4 + ":8080", false},
{"[" + allowlistedIPv6 + "]", true},
{"[" + allowlistedIPv6 + "]:8443", false},
}
for _, tt := range tests {
t.Run(tt.host, func(t *testing.T) {
t.Parallel()
err := h.validateSignature(photoRequest(tt.host), "", time.Time{})
t.Logf("validateSignature() for host %q with no signature: %v",
tt.host, err)
if allowlisted := err == nil; allowlisted != tt.allowlisted {
t.Errorf("host %q served without a signature = %v, want %v",
tt.host, allowlisted, tt.allowlisted)
}
})
}
}
// TestValidateSignature_UnbracketedIPv6PortNeedsSignature verifies that an
// allowlisted IPv6 address written without brackets and followed by a port,
// or by a colon with no port, needs a signature. Go's URL type reads
// 2001:db8::1:8443 as the host 2001:db8::1 with the port 8443.
func TestValidateSignature_UnbracketedIPv6PortNeedsSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedIPv6)
for _, host := range []string{allowlistedIPv6 + ":8443", allowlistedIPv6 + ":"} {
t.Run(host, func(t *testing.T) {
t.Parallel()
err := h.validateSignature(photoRequest(host), "", time.Time{})
t.Logf("validateSignature() for host %q with no signature: %v",
host, err)
if err == nil {
t.Errorf("host %q served without a signature, want refused",
host)
}
})
}
}
internal/httpfetcher/blocked_networks_internal_test.go
+10 -6
View File
@@ -7,6 +7,7 @@ import (
"net/http" "net/http"
"net/netip" "net/netip"
"testing" "testing"
"time"
) )
// TestIsPrivateIPBlocksSpecialRanges covers the internal and special-use // TestIsPrivateIPBlocksSpecialRanges covers the internal and special-use
@@ -70,12 +71,15 @@ func transportOf(t *testing.T, f *HTTPFetcher) *http.Transport {
func TestDialerEnforcesBlockedNetworks(t *testing.T) { func TestDialerEnforcesBlockedNetworks(t *testing.T) {
t.Parallel() t.Parallel()
cfg := DefaultConfig() transport := transportOf(t, New(&Config{
// TEST-NET-2 (198.51.100.0/24) is public to the built-in check, so Timeout: 30 * time.Second,
// blocking it can only come from the operator-supplied list. MaxResponseSize: 50 << 20,
cfg.BlockedNetworks = []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")} MaxConnectionsPerHost: 20,
MaxConnections: 64,
transport := transportOf(t, New(cfg)) // TEST-NET-2 (198.51.100.0/24) is public to the built-in check, so
// blocking it can only come from the operator-supplied list.
BlockedNetworks: []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")},
}))
blocked := []string{ blocked := []string{
"198.51.100.5:80", // operator-supplied range "198.51.100.5:80", // operator-supplied range
internal/httpfetcher/conditional_internal_test.go
+76
View File
@@ -0,0 +1,76 @@
package httpfetcher
import (
"errors"
"io"
"net/http"
"net/http/httptest"
"sync"
"testing"
)
// TestFetch_SendsConditionalHeadersAndHandlesNotModified checks that Fetch
// sends a non-empty ifNoneMatch and ifModifiedSince as request headers, that
// upstream's 304 to such a request is a result with status 304 and no
// content, and that a 304 to a request with neither is ErrUpstreamError.
func TestFetch_SendsConditionalHeadersAndHandlesNotModified(t *testing.T) {
t.Parallel()
const (
etag = `"abc"`
lastModified = "Mon, 02 Nov 2026 10:00:00 GMT"
)
var (
mu sync.Mutex
received http.Header
)
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
received = r.Header.Clone()
mu.Unlock()
w.WriteHeader(http.StatusNotModified)
}))
t.Cleanup(srv.Close)
f, _ := newServerFetcher(t, srv, nil)
res, err := f.Fetch(testContext(t), upstreamSource("/image"), etag, lastModified)
if err != nil {
t.Fatalf("conditional Fetch() error = %v", err)
}
body, err := io.ReadAll(res.Content)
_ = res.Content.Close()
mu.Lock()
t.Logf("status = %d, body = %q, request headers = %v", res.StatusCode, body, received)
gotIfNoneMatch := received.Get("If-None-Match")
gotIfModifiedSince := received.Get("If-Modified-Since")
mu.Unlock()
if err != nil || len(body) != 0 {
t.Errorf("304 content = %q, error = %v; want empty", body, err)
}
if res.StatusCode != http.StatusNotModified {
t.Errorf("StatusCode = %d, want 304", res.StatusCode)
}
if gotIfNoneMatch != etag || gotIfModifiedSince != lastModified {
t.Errorf("If-None-Match = %q, If-Modified-Since = %q; want %q and %q",
gotIfNoneMatch, gotIfModifiedSince, etag, lastModified)
}
if held := len(f.allHostsSemaphore); held != 0 {
t.Errorf("connection still held after the 304: %d held", held)
}
_, err = f.Fetch(testContext(t), upstreamSource("/image"), "", "")
if !errors.Is(err, ErrUpstreamError) {
t.Errorf("unconditional Fetch() answered 304: error = %v, want ErrUpstreamError", err)
}
}
internal/httpfetcher/dial_context_internal_test.go
+17 -7
View File
@@ -4,6 +4,7 @@ import (
"errors" "errors"
"net" "net"
"testing" "testing"
"time"
) )
// TestNewUsesCheckedDialerWithoutDialContext checks that a fetcher built // TestNewUsesCheckedDialerWithoutDialContext checks that a fetcher built
@@ -13,7 +14,12 @@ func TestNewUsesCheckedDialerWithoutDialContext(t *testing.T) {
t.Parallel() t.Parallel()
srv := startUpstream(t) srv := startUpstream(t)
transport := transportOf(t, New(DefaultConfig())) transport := transportOf(t, New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
}))
addr := srv.Listener.Addr().String() addr := srv.Listener.Addr().String()
@@ -32,21 +38,25 @@ func TestDialContextReplacesOnlyTheDialer(t *testing.T) {
srv := startUpstream(t) srv := startUpstream(t)
dialer := &recordingDialer{target: srv.Listener.Addr().String()} dialer := &recordingDialer{target: srv.Listener.Addr().String()}
cfg := DefaultConfig() f := New(&Config{
cfg.AllowHTTP = true Timeout: 30 * time.Second,
cfg.DialContext = dialer.dialContext MaxResponseSize: 50 << 20,
f := New(cfg) AllowHTTP: true,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
DialContext: dialer.dialContext,
})
if body := fetchBody(t, f, "/image"); body != imagePayload { if body := fetchBody(t, f, "/image"); body != imagePayload {
t.Errorf("body = %q, want %q", body, imagePayload) t.Errorf("body = %q, want %q", body, imagePayload)
} }
_, err := f.Fetch(testContext(t), "http://127.0.0.1/image") _, err := f.Fetch(testContext(t), imageSourceOn("127.0.0.1"), "", "")
if !errors.Is(err, ErrSSRFBlocked) { if !errors.Is(err, ErrSSRFBlocked) {
t.Errorf("Fetch(loopback URL) error = %v, want ErrSSRFBlocked", err) t.Errorf("Fetch(loopback URL) error = %v, want ErrSSRFBlocked", err)
} }
_, err = f.Fetch(testContext(t), upstreamURL("/redirect/private")) _, err = f.Fetch(testContext(t), upstreamSource("/redirect/private"), "", "")
if !errors.Is(err, ErrSSRFBlocked) { if !errors.Is(err, ErrSSRFBlocked) {
t.Errorf("Fetch(/redirect/private) error = %v, want ErrSSRFBlocked", err) t.Errorf("Fetch(/redirect/private) error = %v, want ErrSSRFBlocked", err)
} }
internal/httpfetcher/fetch_internal_test.go
+225 -18
View File
@@ -9,10 +9,13 @@ import (
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"slices" "slices"
"strconv"
"strings" "strings"
"sync" "sync"
"testing" "testing"
"time" "time"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// testPublicHost is a TEST-NET-1 (RFC 5737) literal. isPrivateIP treats it as // testPublicHost is a TEST-NET-1 (RFC 5737) literal. isPrivateIP treats it as
@@ -28,9 +31,9 @@ const imagePayload = "fake-jpeg-bytes"
// would mean SSRF protection let a forbidden target reach the transport. // would mean SSRF protection let a forbidden target reach the transport.
var errUnexpectedDial = errors.New("unexpected dial target") var errUnexpectedDial = errors.New("unexpected dial target")
// upstreamURL builds a fetch URL on the fake public host for the given path. // upstreamSource returns the source at path on the fake public host.
func upstreamURL(path string) string { func upstreamSource(path string) imagerequest.Source {
return "http://" + testPublicHost + path return imagerequest.Source{Host: testPublicHost, Path: path}
} }
// recordingDialer records every address the transport asks it to dial and // recordingDialer records every address the transport asks it to dial and
@@ -80,7 +83,7 @@ func startUpstream(t *testing.T) *httptest.Server {
mux := http.NewServeMux() mux := http.NewServeMux()
mux.HandleFunc("/image", func(w http.ResponseWriter, _ *http.Request) { mux.HandleFunc("/image", func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", contentTypeJPEG) w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
_, _ = io.WriteString(w, imagePayload) _, _ = io.WriteString(w, imagePayload)
}) })
mux.HandleFunc("/status/500", func(w http.ResponseWriter, _ *http.Request) { mux.HandleFunc("/status/500", func(w http.ResponseWriter, _ *http.Request) {
@@ -119,7 +122,12 @@ func newServerFetcher(
t.Helper() t.Helper()
if cfg == nil { if cfg == nil {
cfg = DefaultConfig() cfg = &Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
}
} }
cfg.AllowHTTP = true cfg.AllowHTTP = true
@@ -152,7 +160,7 @@ func testContext(t *testing.T) context.Context {
func fetchImage(t *testing.T, f *HTTPFetcher, path string) *FetchResult { func fetchImage(t *testing.T, f *HTTPFetcher, path string) *FetchResult {
t.Helper() t.Helper()
res, err := f.Fetch(testContext(t), upstreamURL(path)) res, err := f.Fetch(testContext(t), upstreamSource(path), "", "")
if err != nil { if err != nil {
t.Fatalf("Fetch(%s) error = %v", path, err) t.Fatalf("Fetch(%s) error = %v", path, err)
} }
@@ -164,7 +172,7 @@ func fetchImage(t *testing.T, f *HTTPFetcher, path string) *FetchResult {
func fetchExpectError(t *testing.T, f *HTTPFetcher, path string) error { func fetchExpectError(t *testing.T, f *HTTPFetcher, path string) error {
t.Helper() t.Helper()
res, err := f.Fetch(testContext(t), upstreamURL(path)) res, err := f.Fetch(testContext(t), upstreamSource(path), "", "")
if err == nil { if err == nil {
_ = res.Content.Close() _ = res.Content.Close()
@@ -216,7 +224,7 @@ func TestFetchRedirectToPrivateIPBlocked(t *testing.T) {
srv := startUpstream(t) srv := startUpstream(t)
f, dialer := newServerFetcher(t, srv, nil) f, dialer := newServerFetcher(t, srv, nil)
_, err := f.Fetch(testContext(t), upstreamURL("/redirect/private")) _, err := f.Fetch(testContext(t), upstreamSource("/redirect/private"), "", "")
if !errors.Is(err, ErrSSRFBlocked) { if !errors.Is(err, ErrSSRFBlocked) {
t.Fatalf("Fetch() error = %v, want ErrSSRFBlocked", err) t.Fatalf("Fetch() error = %v, want ErrSSRFBlocked", err)
} }
@@ -279,8 +287,12 @@ func TestFetchMaxResponseSizeEnforced(t *testing.T) {
srv := startUpstream(t) srv := startUpstream(t)
cfg := DefaultConfig() cfg := &Config{
cfg.MaxResponseSize = 8 Timeout: 30 * time.Second,
MaxResponseSize: 8,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
}
f, _ := newServerFetcher(t, srv, cfg) f, _ := newServerFetcher(t, srv, cfg)
@@ -297,13 +309,154 @@ func TestFetchMaxResponseSizeEnforced(t *testing.T) {
} }
} }
// fetchWithMaxResponseSize serves handler as the fake upstream and fetches
// its /image route with MaxResponseSize set to maxResponseSize.
func fetchWithMaxResponseSize(
t *testing.T,
maxResponseSize int64,
handler http.HandlerFunc,
) *FetchResult {
t.Helper()
srv := httptest.NewServer(handler)
t.Cleanup(srv.Close)
cfg := &Config{
Timeout: 30 * time.Second,
MaxResponseSize: maxResponseSize,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
}
f, _ := newServerFetcher(t, srv, cfg)
res := fetchImage(t, f, "/image")
t.Cleanup(func() { _ = res.Content.Close() })
return res
}
// TestFetchContentLengthBodyAtMaxResponseSize checks that a body sent with a
// Content-Length is read whole at exactly MaxResponseSize bytes and refused
// at one byte more.
func TestFetchContentLengthBodyAtMaxResponseSize(t *testing.T) {
t.Parallel()
const maxResponseSize = 8
for _, tc := range []struct {
size int
wantErr error
}{
{maxResponseSize, nil},
{maxResponseSize + 1, ErrResponseTooLarge},
} {
t.Run(strconv.Itoa(tc.size)+" bytes", func(t *testing.T) {
t.Parallel()
body := strings.Repeat("x", tc.size)
res := fetchWithMaxResponseSize(t, maxResponseSize,
func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
w.Header().Set("Content-Length", strconv.Itoa(tc.size))
_, _ = io.WriteString(w, body)
})
if res.ContentLength != int64(tc.size) {
t.Fatalf("ContentLength = %d, want %d", res.ContentLength, tc.size)
}
data, err := io.ReadAll(res.Content)
t.Logf("body of %d bytes: read %d, error %v", tc.size, len(data), err)
if !errors.Is(err, tc.wantErr) {
t.Fatalf("read error = %v, want %v", err, tc.wantErr)
}
if err == nil && string(data) != body {
t.Errorf("read %q, want %q", data, body)
}
})
}
}
// TestFetchChunkedBodyAtMaxResponseSize checks that a chunked body is read
// whole at exactly MaxResponseSize bytes and refused at one byte more. The
// upstream sends the end of the body only once the test has read
// MaxResponseSize bytes, so the read after them comes before the end of the
// body has arrived, as it can from a real upstream.
func TestFetchChunkedBodyAtMaxResponseSize(t *testing.T) {
t.Parallel()
const maxResponseSize = 8
for _, tc := range []struct {
size int
wantErr error
}{
{maxResponseSize, nil},
{maxResponseSize + 1, ErrResponseTooLarge},
} {
t.Run(strconv.Itoa(tc.size)+" bytes", func(t *testing.T) {
t.Parallel()
body := strings.Repeat("x", tc.size)
endBody := make(chan struct{})
res := fetchWithMaxResponseSize(t, maxResponseSize,
func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
_, _ = io.WriteString(w, body)
// With no Content-Length, flushing sends the body
// chunked; its end is sent when the handler returns.
_ = http.NewResponseController(w).Flush()
select {
case <-endBody:
case <-r.Context().Done():
}
})
if res.ContentLength != -1 {
t.Fatalf("ContentLength = %d, want -1 (chunked)", res.ContentLength)
}
head := make([]byte, maxResponseSize)
_, err := io.ReadFull(res.Content, head)
if err != nil {
t.Fatalf("read the first %d bytes: %v", maxResponseSize, err)
}
close(endBody)
rest, err := io.ReadAll(res.Content)
data := string(head) + string(rest)
t.Logf("body of %d bytes: read %d, error %v", tc.size, len(data), err)
if !errors.Is(err, tc.wantErr) {
t.Fatalf("read error = %v, want %v", err, tc.wantErr)
}
if err == nil && data != body {
t.Errorf("read %q, want %q", data, body)
}
})
}
}
func TestFetchSemaphoreReleasedOnError(t *testing.T) { func TestFetchSemaphoreReleasedOnError(t *testing.T) {
t.Parallel() t.Parallel()
srv := startUpstream(t) srv := startUpstream(t)
cfg := DefaultConfig() cfg := &Config{
cfg.MaxConnectionsPerHost = 1 Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 1,
MaxConnections: 64,
}
f, _ := newServerFetcher(t, srv, cfg) f, _ := newServerFetcher(t, srv, cfg)
@@ -332,8 +485,12 @@ func assertSlotReleasedByClose(
srv := startUpstream(t) srv := startUpstream(t)
cfg := DefaultConfig() cfg := &Config{
cfg.MaxConnectionsPerHost = 1 Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 1,
MaxConnections: 64,
}
f, _ := newServerFetcher(t, srv, cfg) f, _ := newServerFetcher(t, srv, cfg)
@@ -390,7 +547,7 @@ func TestFetchSemaphoreReleasedOnPartialReadClose(t *testing.T) {
}) })
} }
// The dial-time re-resolution in ssrfSafeDialer is what closes the DNS // The dial-time re-resolution in dialSSRFSafe is what closes the DNS
// rebinding window: even if validateURL saw a public answer earlier, the // rebinding window: even if validateURL saw a public answer earlier, the
// dialer independently re-checks the address it is about to connect to. A full // dialer independently re-checks the address it is about to connect to. A full
// rebinding simulation (a resolver returning public, then private) would mean // rebinding simulation (a resolver returning public, then private) would mean
@@ -410,9 +567,9 @@ func TestSSRFSafeDialerBlocksPrivateTarget(t *testing.T) {
t.Run(addr, func(t *testing.T) { t.Run(addr, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := ssrfSafeDialer(context.Background(), "tcp", addr) _, err := dialSSRFSafe(context.Background(), "tcp", addr, nil)
if !errors.Is(err, ErrSSRFBlocked) { if !errors.Is(err, ErrSSRFBlocked) {
t.Errorf("ssrfSafeDialer(%q) = %v, want ErrSSRFBlocked", addr, err) t.Errorf("dialSSRFSafe(%q) = %v, want ErrSSRFBlocked", addr, err)
} }
}) })
} }
@@ -426,7 +583,7 @@ func TestSSRFSafeDialerAllowsPublicTarget(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
cancel() cancel()
_, err := ssrfSafeDialer(ctx, "tcp", testPublicHost+":80") _, err := dialSSRFSafe(ctx, "tcp", testPublicHost+":80", nil)
if err == nil { if err == nil {
t.Fatal("expected a dial error for an unreachable public target") t.Fatal("expected a dial error for an unreachable public target")
} }
@@ -435,3 +592,53 @@ func TestSSRFSafeDialerAllowsPublicTarget(t *testing.T) {
t.Errorf("public target was SSRF-blocked: %v", err) t.Errorf("public target was SSRF-blocked: %v", err)
} }
} }
// TestFetchSchemeFollowsAllowHTTP checks that the fetcher fetches a source
// over https, on port 443, unless AllowHTTP is set, and then over http, on
// port 80. The test server speaks plain HTTP, so only the fetch over http
// succeeds.
func TestFetchSchemeFollowsAllowHTTP(t *testing.T) {
t.Parallel()
tests := []struct {
allowHTTP bool
wantAddr string
}{
{false, net.JoinHostPort(testPublicHost, "443")},
{true, net.JoinHostPort(testPublicHost, "80")},
}
for _, tt := range tests {
t.Run(fmt.Sprintf("AllowHTTP %v", tt.allowHTTP), func(t *testing.T) {
t.Parallel()
srv := startUpstream(t)
dialer := &recordingDialer{target: srv.Listener.Addr().String()}
f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
AllowHTTP: tt.allowHTTP,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
DialContext: dialer.dialContext,
})
res, err := f.Fetch(testContext(t), upstreamSource("/image"), "", "")
if err == nil {
_ = res.Content.Close()
}
dialed := dialer.dialedAddrs()
t.Logf("Fetch(): error %v, dialed %v", err, dialed)
if served := err == nil; served != tt.allowHTTP {
t.Errorf("served = %v, want %v", served, tt.allowHTTP)
}
if len(dialed) != 1 || dialed[0] != tt.wantAddr {
t.Errorf("dialed %v, want only %s", dialed, tt.wantAddr)
}
})
}
}
internal/httpfetcher/httpfetcher.go
+191 -125
View File
@@ -20,36 +20,24 @@ import (
"time" "time"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// Fetcher configuration constants. // Fetcher configuration constants.
const ( const (
DefaultFetchTimeout = 30 * time.Second DefaultTLSTimeout = 10 * time.Second
DefaultMaxResponseSize = 50 << 20 // 50MB DefaultMaxIdleConns = 100
DefaultTLSTimeout = 10 * time.Second DefaultIdleConnTimeout = 90 * time.Second
DefaultMaxIdleConns = 100 DefaultMaxRedirects = 10
DefaultIdleConnTimeout = 90 * time.Second
DefaultMaxRedirects = 10
DefaultMaxConnectionsPerHost = 20
DefaultMaxConnections = 64
) )
// userAgent is the User-Agent header sent to upstream servers.
const userAgent = "pixa/1.0"
// ConnectionWaitTimeout is how long Fetch waits for a free connection when // ConnectionWaitTimeout is how long Fetch waits for a free connection when
// MaxConnections fetches are already in progress. // MaxConnections fetches are already in progress.
const ConnectionWaitTimeout = 10 * time.Second const ConnectionWaitTimeout = 10 * time.Second
// MIME content types.
const (
contentTypeJPEG = "image/jpeg"
contentTypePNG = "image/png"
contentTypeGIF = "image/gif"
contentTypeWebP = "image/webp"
contentTypeAVIF = "image/avif"
contentTypeJXL = "image/jxl"
contentTypeSVG = "image/svg+xml"
contentTypeOctetStream = "application/octet-stream"
)
// Loopback addresses blocked by SSRF protection. // Loopback addresses blocked by SSRF protection.
const ( const (
localhostIPv4 = "127.0.0.1" localhostIPv4 = "127.0.0.1"
@@ -59,15 +47,17 @@ const (
// builtinBlockedPrefixes are internal or special-use ranges that Go's // builtinBlockedPrefixes are internal or special-use ranges that Go's
// net.IP predicates (IsPrivate, IsLinkLocalUnicast, and the like) do not // net.IP predicates (IsPrivate, IsLinkLocalUnicast, and the like) do not
// already cover. They are always blocked, in addition to any // already cover. They are always blocked, in addition to any
// operator-supplied networks. IPv4-mapped IPv6 addresses are unmapped // operator-supplied networks. IPv4-mapped and IPv4-compatible IPv6
// before matching, so these IPv4 ranges are caught in both forms. // addresses are read as the IPv4 address they hold before matching, so
// these IPv4 ranges are caught in all three forms.
// //
//nolint:gochecknoglobals // immutable built-in blocklist //nolint:gochecknoglobals // immutable built-in blocklist
var builtinBlockedPrefixes = []netip.Prefix{ var builtinBlockedPrefixes = []netip.Prefix{
netip.MustParsePrefix("100.64.0.0/10"), // RFC 6598 CGNAT / carrier-grade NAT netip.MustParsePrefix("100.64.0.0/10"), // RFC 6598 CGNAT / carrier-grade NAT
netip.MustParsePrefix("192.0.0.0/24"), // RFC 6890 IETF protocol assignments netip.MustParsePrefix("192.0.0.0/24"), // RFC 6890 IETF protocol assignments
netip.MustParsePrefix("198.18.0.0/15"), // RFC 2544 benchmarking range netip.MustParsePrefix("198.18.0.0/15"), // RFC 2544 benchmarking range
netip.MustParsePrefix("64:ff9b::/96"), // RFC 6052 NAT64 (maps onto IPv4) netip.MustParsePrefix("64:ff9b::/96"), // RFC 6052 NAT64 (maps onto IPv4)
netip.MustParsePrefix("64:ff9b:1::/48"), // RFC 8215 local-use NAT64 (maps onto IPv4)
} }
// Fetcher errors. // Fetcher errors.
@@ -79,7 +69,11 @@ var (
ErrInvalidContentType = errors.New("invalid or unsupported content type") ErrInvalidContentType = errors.New("invalid or unsupported content type")
ErrUpstreamError = errors.New("upstream server error") ErrUpstreamError = errors.New("upstream server error")
ErrUpstreamTimeout = errors.New("upstream request timeout") ErrUpstreamTimeout = errors.New("upstream request timeout")
ErrTooManyConnections = errors.New("too many concurrent upstream connections") // ErrUpstreamRequestFailed is a fetch that failed before the upstream
// answered, other than by timeout or cancellation: a failed connection
// or TLS handshake, or a redirect the fetcher refuses.
ErrUpstreamRequestFailed = errors.New("upstream request failed")
ErrTooManyConnections = errors.New("too many concurrent upstream connections")
) )
// Internal fetcher errors. // Internal fetcher errors.
@@ -90,8 +84,13 @@ var (
// Fetcher retrieves content from upstream origins. // Fetcher retrieves content from upstream origins.
type Fetcher interface { type Fetcher interface {
// Fetch retrieves content from the given URL. // Fetch retrieves the content of the source image. A non-empty
Fetch(ctx context.Context, url string) (*FetchResult, error) // ifNoneMatch or ifModifiedSince is sent as the request header of that
// name, and upstream may then answer 304 Not Modified with no content.
Fetch(
ctx context.Context, source imagerequest.Source,
ifNoneMatch, ifModifiedSince string,
) (*FetchResult, error)
} }
// FetchResult contains the result of fetching from upstream. // FetchResult contains the result of fetching from upstream.
@@ -118,17 +117,29 @@ type FetchResult struct {
TLSCipherSuite string TLSCipherSuite string
} }
// Config holds configuration for the upstream fetcher. // allowedContentTypes returns the content types an upstream response may
// have, which Fetch also sends as its Accept header: the MIME types of the
// formats pixa reads.
func allowedContentTypes() []string {
return []string{
imagerequest.FormatJPEG.MIMEType(),
imagerequest.FormatPNG.MIMEType(),
imagerequest.FormatGIF.MIMEType(),
imagerequest.FormatWebP.MIMEType(),
imagerequest.FormatAVIF.MIMEType(),
imagerequest.FormatJXL.MIMEType(),
}
}
// Config holds configuration for the upstream fetcher. Every field but
// AllowHTTP, BlockedNetworks and DialContext must be set.
type Config struct { type Config struct {
// Timeout for upstream requests. // Timeout for upstream requests.
Timeout time.Duration Timeout time.Duration
// MaxResponseSize is the maximum allowed response body size. // MaxResponseSize is the maximum allowed response body size.
MaxResponseSize int64 MaxResponseSize int64
// UserAgent to send to upstream servers. // AllowHTTP makes the fetcher fetch sources over http instead of https,
UserAgent string // and follow redirects to http (for testing only).
// AllowedContentTypes is an allow list of MIME types to accept.
AllowedContentTypes []string
// AllowHTTP allows non-TLS connections (for testing only).
AllowHTTP bool AllowHTTP bool
// MaxConnectionsPerHost limits concurrent connections to each upstream host. // MaxConnectionsPerHost limits concurrent connections to each upstream host.
MaxConnectionsPerHost int MaxConnectionsPerHost int
@@ -145,26 +156,6 @@ type Config struct {
DialContext func(ctx context.Context, network, addr string) (net.Conn, error) DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
} }
// DefaultConfig returns a Config with sensible defaults.
func DefaultConfig() *Config {
return &Config{
Timeout: DefaultFetchTimeout,
MaxResponseSize: DefaultMaxResponseSize,
UserAgent: "pixa/1.0",
AllowedContentTypes: []string{
contentTypeJPEG,
contentTypePNG,
contentTypeGIF,
contentTypeWebP,
contentTypeAVIF,
contentTypeJXL,
},
AllowHTTP: false,
MaxConnectionsPerHost: DefaultMaxConnectionsPerHost,
MaxConnections: DefaultMaxConnections,
}
}
// HTTPFetcher implements Fetcher with SSRF protection and connection limits // HTTPFetcher implements Fetcher with SSRF protection and connection limits
// per host and for all hosts together. // per host and for all hosts together.
type HTTPFetcher struct { type HTTPFetcher struct {
@@ -192,10 +183,6 @@ type hostSemaphore struct {
// New creates a new HTTPFetcher with SSRF protection. // New creates a new HTTPFetcher with SSRF protection.
func New(config *Config) *HTTPFetcher { func New(config *Config) *HTTPFetcher {
if config == nil {
config = DefaultConfig()
}
// Unless config.DialContext replaces it, the transport connects with // Unless config.DialContext replaces it, the transport connects with
// the SSRF-safe dialer, which re-resolves and re-checks at connect time // the SSRF-safe dialer, which re-resolves and re-checks at connect time
// (closing the DNS-rebinding window) against both the built-in ranges // (closing the DNS-rebinding window) against both the built-in ranges
@@ -224,7 +211,7 @@ func New(config *Config) *HTTPFetcher {
} }
// Validate the redirect target // Validate the redirect target
err := validateURL(req.Context(), req.URL.String(), config.AllowHTTP) err := validateURL(req.Context(), req.URL, config.AllowHTTP)
if err != nil { if err != nil {
return fmt.Errorf("redirect blocked: %w", err) return fmt.Errorf("redirect blocked: %w", err)
} }
@@ -242,18 +229,26 @@ func New(config *Config) *HTTPFetcher {
} }
} }
// Fetch retrieves content from the given URL with SSRF protection. When // Fetch retrieves the source image with SSRF protection, over https, or over
// MaxConnections fetches are already in progress, it waits up to // http when AllowHTTP is set. When MaxConnections fetches are already in
// ConnectionWaitTimeout for one to finish, then fails with // progress, it waits up to ConnectionWaitTimeout for one to finish, then
// ErrTooManyConnections. // fails with ErrTooManyConnections. A non-empty ifNoneMatch or
func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, error) { // ifModifiedSince is sent as the If-None-Match or If-Modified-Since header.
// Upstream's 304 Not Modified to such a request is a result with StatusCode
// 304 and empty Content; to any other request it is ErrUpstreamError.
func (f *HTTPFetcher) Fetch(
ctx context.Context, source imagerequest.Source,
ifNoneMatch, ifModifiedSince string,
) (*FetchResult, error) {
sourceURL := source.URL(f.config.AllowHTTP)
// Validate URL before making request // Validate URL before making request
err := validateURL(ctx, url, f.config.AllowHTTP) err := validateURL(ctx, sourceURL, f.config.AllowHTTP)
if err != nil { if err != nil {
return nil, err return nil, err
} }
release, err := f.acquireConnection(ctx, extractHost(url)) release, err := f.acquireConnection(ctx, source.Host)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -267,25 +262,10 @@ func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
} }
}() }()
parsedURL, err := neturl.Parse(url)
if err != nil {
return nil, fmt.Errorf("failed to parse URL: %w", err)
}
req := &http.Request{ req := &http.Request{
Method: http.MethodGet, Method: http.MethodGet,
URL: parsedURL, URL: sourceURL,
Header: make(http.Header), Header: requestHeaders(ctx, ifNoneMatch, ifModifiedSince),
}
req.Header.Set("User-Agent", f.config.UserAgent)
req.Header.Set("Accept", strings.Join(f.config.AllowedContentTypes, ", "))
// The ID of the request this fetch serves, so the fetch can be found in
// the upstream host's logs
requestID := middleware.GetReqID(ctx)
if requestID != "" {
req.Header.Set(middleware.RequestIDHeader, requestID)
} }
// Use httptrace to capture connection details // Use httptrace to capture connection details
@@ -307,11 +287,21 @@ func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
fetchDuration := time.Since(startTime) fetchDuration := time.Since(startTime)
if err != nil { if err != nil {
if errors.Is(err, context.DeadlineExceeded) { return nil, upstreamRequestError(err)
return nil, ErrUpstreamTimeout }
}
return nil, fmt.Errorf("upstream request failed: %w", err) // The deferred release gives the connection back, as no body follows
conditional := ifNoneMatch != "" || ifModifiedSince != ""
if conditional && resp.StatusCode == http.StatusNotModified {
_ = resp.Body.Close()
return &FetchResult{
Content: http.NoBody,
Headers: resp.Header,
StatusCode: resp.StatusCode,
FetchDurationMs: fetchDuration.Milliseconds(),
RemoteAddr: remoteAddr,
}, nil
} }
result, err := f.buildResult(resp, remoteAddr, fetchDuration, release) result, err := f.buildResult(resp, remoteAddr, fetchDuration, release)
@@ -326,6 +316,58 @@ func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
return result, nil return result, nil
} }
// upstreamRequestError returns the error Fetch reports when its upstream
// request fails with err:
// - ErrUpstreamTimeout for a timeout.
// - err, still matching context.Canceled, when the caller cancelled the
// fetch.
// - err wrapped in ErrUpstreamRequestFailed otherwise.
func upstreamRequestError(err error) error {
var netErr net.Error
switch {
case errors.Is(err, context.DeadlineExceeded),
errors.Is(err, ErrUpstreamTimeout),
errors.As(err, &netErr) && netErr.Timeout():
// A timeout other than the fetch timeout, such as the TLS
// handshake timeout, does not match context.DeadlineExceeded. An
// address lookup that timed out is ErrUpstreamTimeout already.
return ErrUpstreamTimeout
case errors.Is(err, context.Canceled):
// The caller cancelled the fetch; the upstream did not fail
return fmt.Errorf("upstream request cancelled: %w", err)
default:
return fmt.Errorf("%w: %w", ErrUpstreamRequestFailed, err)
}
}
// requestHeaders returns the headers of a fetch: User-Agent, Accept, the
// ID of the request the fetch serves, so the fetch can be found in the
// upstream host's logs, and a non-empty ifNoneMatch or ifModifiedSince.
func requestHeaders(
ctx context.Context, ifNoneMatch, ifModifiedSince string,
) http.Header {
header := make(http.Header)
header.Set("User-Agent", userAgent)
header.Set("Accept", strings.Join(allowedContentTypes(), ", "))
requestID := middleware.GetReqID(ctx)
if requestID != "" {
header.Set(middleware.RequestIDHeader, requestID)
}
if ifNoneMatch != "" {
header.Set("If-None-Match", ifNoneMatch)
}
if ifModifiedSince != "" {
header.Set("If-Modified-Since", ifModifiedSince)
}
return header
}
// acquireConnection takes a slot for host, then one of the slots shared by // acquireConnection takes a slot for host, then one of the slots shared by
// all hosts, and returns the func that gives both back. The host's slot // all hosts, and returns the func that gives both back. The host's slot
// comes first, so fetches queued for one busy host hold no shared slot. // comes first, so fetches queued for one busy host hold no shared slot.
@@ -458,7 +500,7 @@ func (f *HTTPFetcher) isAllowedContentType(contentType string) bool {
// Extract the MIME type without parameters // Extract the MIME type without parameters
mediaType := strings.TrimSpace(strings.Split(contentType, ";")[0]) mediaType := strings.TrimSpace(strings.Split(contentType, ";")[0])
for _, allowed := range f.config.AllowedContentTypes { for _, allowed := range allowedContentTypes() {
if strings.EqualFold(mediaType, allowed) { if strings.EqualFold(mediaType, allowed) {
return true return true
} }
@@ -468,23 +510,18 @@ func (f *HTTPFetcher) isAllowedContentType(contentType string) bool {
} }
// validateURL checks if a URL is safe to fetch (not internal/private). // validateURL checks if a URL is safe to fetch (not internal/private).
func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error { func validateURL(ctx context.Context, fetchURL *neturl.URL, allowHTTP bool) error {
if !allowHTTP && !strings.HasPrefix(rawURL, "https://") { if !allowHTTP && fetchURL.Scheme != "https" {
return ErrUnsupportedScheme return ErrUnsupportedScheme
} }
// Parse to extract host // Hostname drops any port, and the brackets of an IPv6 address whether
host := extractHost(rawURL) // or not a port follows them
host := fetchURL.Hostname()
if host == "" { if host == "" {
return ErrInvalidHost return ErrInvalidHost
} }
// Remove port if present
h, _, err := net.SplitHostPort(host)
if err == nil {
host = h
}
// Block obvious localhost patterns // Block obvious localhost patterns
if isLocalhost(host) { if isLocalhost(host) {
return ErrSSRFBlocked return ErrSSRFBlocked
@@ -493,7 +530,7 @@ func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
// Resolve the host to check IP addresses // Resolve the host to check IP addresses
addrs, err := net.DefaultResolver.LookupIPAddr(ctx, host) addrs, err := net.DefaultResolver.LookupIPAddr(ctx, host)
if err != nil { if err != nil {
return fmt.Errorf("%w: %s", ErrInvalidHost, host) return addressLookupError(host, err)
} }
private := slices.ContainsFunc(addrs, func(addr net.IPAddr) bool { private := slices.ContainsFunc(addrs, func(addr net.IPAddr) bool {
@@ -506,23 +543,15 @@ func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
return nil return nil
} }
// extractHost extracts the host from a URL string. // addressLookupError returns the error for a failed address lookup of host:
func extractHost(rawURL string) string { // ErrUpstreamTimeout when the lookup timed out, ErrInvalidHost otherwise.
// Simple extraction without full URL parsing func addressLookupError(host string, err error) error {
url := rawURL var netErr net.Error
if idx := strings.Index(url, "://"); idx != -1 { if errors.As(err, &netErr) && netErr.Timeout() {
url = url[idx+3:] return fmt.Errorf("%w: address lookup of %s", ErrUpstreamTimeout, host)
} }
if idx := strings.Index(url, "/"); idx != -1 { return fmt.Errorf("%w: %s", ErrInvalidHost, host)
url = url[:idx]
}
if idx := strings.Index(url, "?"); idx != -1 {
url = url[:idx]
}
return url
} }
// isLocalhost checks if the host is localhost. // isLocalhost checks if the host is localhost.
@@ -543,6 +572,9 @@ func isPrivateIP(ip net.IP) bool {
return true return true
} }
// An IPv4-compatible IPv6 address is judged as the IPv4 address it holds
ip = ipv4CompatibleToIPv4(ip)
// Check for loopback // Check for loopback
if ip.IsLoopback() { if ip.IsLoopback() {
return true return true
@@ -594,7 +626,12 @@ func isPrivateIP(ip net.IP) bool {
} }
// isBlockedIP reports whether ip is refused, either by the built-in // isBlockedIP reports whether ip is refused, either by the built-in
// internal-range check or by one of the operator-supplied prefixes. // internal-range check or by one of the operator-supplied prefixes. An
// IPv4-compatible address (in ::/96) is refused when a prefix contains the
// address as written or the IPv4 address it holds. An IPv4-mapped address (in
// ::ffff:0:0/96) is matched only as the IPv4 address it holds: Go keeps a
// plain IPv4 address in that form too, and an IPv6 prefix such as ::/0 must
// not refuse every IPv4 address.
func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool { func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
if isPrivateIP(ip) { if isPrivateIP(ip) {
return true return true
@@ -607,16 +644,26 @@ func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
addr = addr.Unmap() addr = addr.Unmap()
// ip read as IPv4 where it is or holds an IPv4 address, otherwise the
// zero netip.Addr, which no prefix contains. Matching asIPv4 as well as
// addr refuses more only for an IPv4-compatible address.
asIPv4, _ := netip.AddrFromSlice(ipv4CompatibleToIPv4(ip).To4())
return slices.ContainsFunc(blocked, func(prefix netip.Prefix) bool { return slices.ContainsFunc(blocked, func(prefix netip.Prefix) bool {
return prefix.Contains(addr) return prefix.Contains(addr) || prefix.Contains(asIPv4)
}) })
} }
// ssrfSafeDialer validates IP addresses against the built-in blocked ranges // ipv4CompatibleToIPv4 returns the IPv4 address that an IPv4-compatible IPv6
// before connecting. New wraps dialSSRFSafe with the operator-supplied // address (in ::/96, such as ::127.0.0.1) holds in its last four bytes, and
// blocklist; this entry point enforces the built-in ranges alone. // any other address unchanged.
func ssrfSafeDialer(ctx context.Context, network, addr string) (net.Conn, error) { func ipv4CompatibleToIPv4(ip net.IP) net.IP {
return dialSSRFSafe(ctx, network, addr, nil) addr, ok := netip.AddrFromSlice(ip)
if ok && netip.MustParsePrefix("::/96").Contains(addr) {
return ip[12:]
}
return ip
} }
// dialSSRFSafe re-resolves addr and refuses to connect to any built-in // dialSSRFSafe re-resolves addr and refuses to connect to any built-in
@@ -635,7 +682,7 @@ func dialSSRFSafe(
// Resolve the address // Resolve the address
ips, err := net.DefaultResolver.LookupIP(ctx, "ip", host) ips, err := net.DefaultResolver.LookupIP(ctx, "ip", host)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: %s", ErrInvalidHost, host) return nil, addressLookupError(host, err)
} }
// Check all resolved IPs // Check all resolved IPs
@@ -660,14 +707,33 @@ func dialSSRFSafe(
return nil, fmt.Errorf("%w to %s", errConnectFailed, host) return nil, fmt.Errorf("%w to %s", errConnectFailed, host)
} }
// limitedReader wraps a reader and limits the number of bytes read. // limitedReader wraps a reader and refuses a body longer than remaining
// bytes with ErrResponseTooLarge. remaining goes below 0 once the body is
// found to be longer.
type limitedReader struct { type limitedReader struct {
reader io.Reader reader io.Reader
remaining int64 remaining int64
} }
// Read reads at most the remaining bytes. Once they are read, it reads one
// more byte to tell the end of the body from more data: at the end it
// returns the wrapped reader's error, io.EOF; with more data it returns
// ErrResponseTooLarge, as does every later Read.
func (r *limitedReader) Read(p []byte) (int, error) { func (r *limitedReader) Read(p []byte) (int, error) {
if r.remaining <= 0 { if r.remaining < 0 {
return 0, ErrResponseTooLarge
}
if r.remaining == 0 {
var extra [1]byte
n, err := r.reader.Read(extra[:])
r.remaining -= int64(n)
if n == 0 {
return 0, err
}
return 0, ErrResponseTooLarge return 0, ErrResponseTooLarge
} }
internal/httpfetcher/httpfetcher_internal_test.go
+69 -91
View File
@@ -5,70 +5,34 @@ import (
"errors" "errors"
"io" "io"
"net" "net"
"net/url"
"testing" "testing"
"testing/fstest" "testing/fstest"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// testHost is the hostname used by mock fetch tests. // testHost is the hostname used by mock fetch tests.
const testHost = "example.com" const testHost = "example.com"
func TestDefaultConfig(t *testing.T) {
t.Parallel()
cfg := DefaultConfig()
if cfg.Timeout != DefaultFetchTimeout {
t.Errorf("Timeout = %v, want %v", cfg.Timeout, DefaultFetchTimeout)
}
if cfg.MaxResponseSize != DefaultMaxResponseSize {
t.Errorf("MaxResponseSize = %d, want %d", cfg.MaxResponseSize, DefaultMaxResponseSize)
}
if cfg.MaxConnectionsPerHost != DefaultMaxConnectionsPerHost {
t.Errorf("MaxConnectionsPerHost = %d, want %d",
cfg.MaxConnectionsPerHost, DefaultMaxConnectionsPerHost)
}
if cfg.AllowHTTP {
t.Error("AllowHTTP should default to false")
}
if len(cfg.AllowedContentTypes) == 0 {
t.Error("AllowedContentTypes should not be empty")
}
}
func TestNewWithNilConfigUsesDefaults(t *testing.T) {
t.Parallel()
f := New(nil)
if f == nil {
t.Fatal("New(nil) returned nil")
}
if f.config == nil {
t.Fatal("config should be populated from DefaultConfig")
}
if f.config.Timeout != DefaultFetchTimeout {
t.Errorf("Timeout = %v, want %v", f.config.Timeout, DefaultFetchTimeout)
}
}
func TestIsAllowedContentType(t *testing.T) { func TestIsAllowedContentType(t *testing.T) {
t.Parallel() t.Parallel()
f := New(DefaultConfig()) f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
})
tests := []struct { tests := []struct {
contentType string contentType string
want bool want bool
}{ }{
{contentTypeJPEG, true}, {imagerequest.FormatJPEG.MIMEType(), true},
{contentTypePNG, true}, {imagerequest.FormatPNG.MIMEType(), true},
{contentTypeWebP, true}, {imagerequest.FormatWebP.MIMEType(), true},
{"image/jpeg; charset=utf-8", true}, {"image/jpeg; charset=utf-8", true},
{"IMAGE/JPEG", true}, {"IMAGE/JPEG", true},
{"text/html", false}, {"text/html", false},
@@ -88,33 +52,6 @@ func TestIsAllowedContentType(t *testing.T) {
} }
} }
func TestExtractHost(t *testing.T) {
t.Parallel()
tests := []struct {
url string
want string
}{
{"https://example.com/path", testHost},
{"http://example.com:8080/path", "example.com:8080"},
{"https://example.com", testHost},
{"https://example.com?q=1", testHost},
{"example.com/path", testHost},
{"", ""},
}
for _, tc := range tests {
t.Run(tc.url, func(t *testing.T) {
t.Parallel()
got := extractHost(tc.url)
if got != tc.want {
t.Errorf("extractHost(%q) = %q, want %q", tc.url, got, tc.want)
}
})
}
}
func TestIsLocalhost(t *testing.T) { func TestIsLocalhost(t *testing.T) {
t.Parallel() t.Parallel()
@@ -186,10 +123,48 @@ func TestIsPrivateIP(t *testing.T) {
} }
} }
// TestIsPrivateIPJudgesEmbeddedIPv4Address checks IPv6 addresses that hold an
// IPv4 address in their last four bytes: the IPv4-compatible form (::/96) and
// the NAT64 form (64:ff9b::/96). One holding a loopback, private or
// link-local IPv4 address is refused as that address is, and one holding a
// public IPv4 address in the IPv4-compatible form passes.
func TestIsPrivateIPJudgesEmbeddedIPv4Address(t *testing.T) {
t.Parallel()
tests := []struct {
ip string
want bool
}{
{"::127.0.0.1", true}, // loopback
{"::10.0.0.1", true}, // private
{"::169.254.169.254", true}, // link-local
{"64:ff9b::7f00:1", true}, // NAT64 form of 127.0.0.1
{"::8.8.8.8", false}, // public
}
for _, tc := range tests {
t.Run(tc.ip, func(t *testing.T) {
t.Parallel()
ip := net.ParseIP(tc.ip)
if ip == nil {
t.Fatalf("failed to parse IP %q", tc.ip)
}
got := isPrivateIP(ip)
t.Logf("isPrivateIP(%s) = %v", ip, got)
if got != tc.want {
t.Errorf("isPrivateIP(%q) = %v, want %v", tc.ip, got, tc.want)
}
})
}
}
func TestValidateURL_RejectsNonHTTPS(t *testing.T) { func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
t.Parallel() t.Parallel()
err := validateURL(t.Context(), "http://example.com/path", false) err := validateURL(t.Context(), &url.URL{Scheme: "http", Host: testHost}, false)
if !errors.Is(err, ErrUnsupportedScheme) { if !errors.Is(err, ErrUnsupportedScheme) {
t.Errorf("validateURL http = %v, want ErrUnsupportedScheme", err) t.Errorf("validateURL http = %v, want ErrUnsupportedScheme", err)
} }
@@ -199,7 +174,8 @@ func TestValidateURL_AllowsHTTPWhenConfigured(t *testing.T) {
t.Parallel() t.Parallel()
// Use a host that won't resolve (explicit .invalid TLD) so we don't hit DNS. // Use a host that won't resolve (explicit .invalid TLD) so we don't hit DNS.
err := validateURL(t.Context(), "http://nonexistent.invalid/path", true) err := validateURL(t.Context(),
&url.URL{Scheme: "http", Host: "nonexistent.invalid"}, true)
// We expect a host resolution error, not ErrUnsupportedScheme. // We expect a host resolution error, not ErrUnsupportedScheme.
if errors.Is(err, ErrUnsupportedScheme) { if errors.Is(err, ErrUnsupportedScheme) {
t.Error("validateURL with AllowHTTP should not return ErrUnsupportedScheme") t.Error("validateURL with AllowHTTP should not return ErrUnsupportedScheme")
@@ -209,7 +185,7 @@ func TestValidateURL_AllowsHTTPWhenConfigured(t *testing.T) {
func TestValidateURL_RejectsLocalhost(t *testing.T) { func TestValidateURL_RejectsLocalhost(t *testing.T) {
t.Parallel() t.Parallel()
err := validateURL(t.Context(), "https://localhost/path", false) err := validateURL(t.Context(), &url.URL{Scheme: "https", Host: "localhost"}, false)
if !errors.Is(err, ErrSSRFBlocked) { if !errors.Is(err, ErrSSRFBlocked) {
t.Errorf("validateURL localhost = %v, want ErrSSRFBlocked", err) t.Errorf("validateURL localhost = %v, want ErrSSRFBlocked", err)
} }
@@ -218,7 +194,7 @@ func TestValidateURL_RejectsLocalhost(t *testing.T) {
func TestValidateURL_EmptyHost(t *testing.T) { func TestValidateURL_EmptyHost(t *testing.T) {
t.Parallel() t.Parallel()
err := validateURL(t.Context(), "https:///path", false) err := validateURL(t.Context(), &url.URL{Scheme: "https"}, false)
if !errors.Is(err, ErrInvalidHost) { if !errors.Is(err, ErrInvalidHost) {
t.Errorf("validateURL empty host = %v, want ErrInvalidHost", err) t.Errorf("validateURL empty host = %v, want ErrInvalidHost", err)
} }
@@ -233,14 +209,15 @@ func TestMockFetcher_FetchesFile(t *testing.T) {
m := NewMock(mockFS) m := NewMock(mockFS)
result, err := m.Fetch(context.Background(), "https://example.com/images/photo.jpg") result, err := m.Fetch(context.Background(),
imagerequest.Source{Host: testHost, Path: "/images/photo.jpg"}, "", "")
if err != nil { if err != nil {
t.Fatalf("Fetch() error = %v", err) t.Fatalf("Fetch() error = %v", err)
} }
defer func() { _ = result.Content.Close() }() defer func() { _ = result.Content.Close() }()
if result.ContentType != contentTypeJPEG { if result.ContentType != imagerequest.FormatJPEG.MIMEType() {
t.Errorf("ContentType = %q, want image/jpeg", result.ContentType) t.Errorf("ContentType = %q, want image/jpeg", result.ContentType)
} }
@@ -264,7 +241,8 @@ func TestMockFetcher_MissingFileReturnsUpstreamError(t *testing.T) {
mockFS := fstest.MapFS{} mockFS := fstest.MapFS{}
m := NewMock(mockFS) m := NewMock(mockFS)
_, err := m.Fetch(context.Background(), "https://example.com/missing.jpg") _, err := m.Fetch(context.Background(),
imagerequest.Source{Host: testHost, Path: "/missing.jpg"}, "", "")
if !errors.Is(err, ErrUpstreamError) { if !errors.Is(err, ErrUpstreamError) {
t.Errorf("Fetch() error = %v, want ErrUpstreamError", err) t.Errorf("Fetch() error = %v, want ErrUpstreamError", err)
} }
@@ -281,7 +259,7 @@ func TestMockFetcher_RespectsContextCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
cancel() cancel()
_, err := m.Fetch(ctx, "https://example.com/photo.jpg") _, err := m.Fetch(ctx, imagerequest.Source{Host: testHost, Path: "/photo.jpg"}, "", "")
if !errors.Is(err, context.Canceled) { if !errors.Is(err, context.Canceled) {
t.Errorf("Fetch() error = %v, want context.Canceled", err) t.Errorf("Fetch() error = %v, want context.Canceled", err)
} }
@@ -294,13 +272,13 @@ func TestDetectContentTypeFromPath(t *testing.T) {
path string path string
want string want string
}{ }{
{"foo/bar.jpg", contentTypeJPEG}, {"foo/bar.jpg", imagerequest.FormatJPEG.MIMEType()},
{"foo/bar.JPG", contentTypeJPEG}, {"foo/bar.JPG", imagerequest.FormatJPEG.MIMEType()},
{"foo/bar.jpeg", contentTypeJPEG}, {"foo/bar.jpeg", imagerequest.FormatJPEG.MIMEType()},
{"foo/bar.png", contentTypePNG}, {"foo/bar.png", imagerequest.FormatPNG.MIMEType()},
{"foo/bar.gif", contentTypeGIF}, {"foo/bar.gif", imagerequest.FormatGIF.MIMEType()},
{"foo/bar.webp", contentTypeWebP}, {"foo/bar.webp", imagerequest.FormatWebP.MIMEType()},
{"foo/bar.avif", contentTypeAVIF}, {"foo/bar.avif", imagerequest.FormatAVIF.MIMEType()},
{"foo/bar.svg", contentTypeSVG}, {"foo/bar.svg", contentTypeSVG},
{"foo/bar.bin", contentTypeOctetStream}, {"foo/bar.bin", contentTypeOctetStream},
{"foo/bar", contentTypeOctetStream}, {"foo/bar", contentTypeOctetStream},
internal/httpfetcher/jpegxl_internal_test.go
+14 -4
View File
@@ -1,16 +1,26 @@
package httpfetcher package httpfetcher
import "testing" import (
"testing"
"time"
)
// TestJPEGXLContentType verifies that the fetcher accepts an upstream answer // TestJPEGXLContentType verifies that the fetcher accepts an upstream answer
// of type image/jxl by default, and that the mock fetcher serves a .jxl file // of type image/jxl, and that the mock fetcher serves a .jxl file as
// as image/jxl. // image/jxl.
func TestJPEGXLContentType(t *testing.T) { func TestJPEGXLContentType(t *testing.T) {
t.Parallel() t.Parallel()
const jxlType = "image/jxl" const jxlType = "image/jxl"
if !New(DefaultConfig()).isAllowedContentType(jxlType) { f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
})
if !f.isAllowedContentType(jxlType) {
t.Errorf("isAllowedContentType(%q) = false, want true", jxlType) t.Errorf("isAllowedContentType(%q) = false, want true", jxlType)
} }
internal/httpfetcher/max_connections_internal_test.go
+52 -39
View File
@@ -8,22 +8,20 @@ import (
"sync" "sync"
"testing" "testing"
"time" "time"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// imageURLOnPort is the fake upstream's image route on testPublicHost at // imageSourceOn is the fake upstream's image route on host.
// port. Each port is a different host to the per-host limit, while the test func imageSourceOn(host string) imagerequest.Source {
// dialer sends every port to the one test server. return imagerequest.Source{Host: host, Path: "/image"}
func imageURLOnPort(port int) string {
return "http://" + net.JoinHostPort(testPublicHost, strconv.Itoa(port)) +
"/image"
} }
func TestDefaultConfigMaxConnections(t *testing.T) { // imageSourceOnPort is the fake upstream's image route on testPublicHost at
t.Parallel() // port. Each port is a different host to the per-host limit, while the test
// dialer sends every port to the one test server.
if got := DefaultConfig().MaxConnections; got != DefaultMaxConnections { func imageSourceOnPort(port int) imagerequest.Source {
t.Errorf("MaxConnections = %d, want %d", got, DefaultMaxConnections) return imageSourceOn(net.JoinHostPort(testPublicHost, strconv.Itoa(port)))
}
} }
// TestFetchLimitsConnectionsToAllHostsTogether checks that MaxConnections // TestFetchLimitsConnectionsToAllHostsTogether checks that MaxConnections
@@ -37,18 +35,22 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
srv := startUpstream(t) srv := startUpstream(t)
cfg := DefaultConfig() cfg := &Config{
cfg.MaxConnections = 2 Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 2,
}
f, _ := newServerFetcher(t, srv, cfg) f, _ := newServerFetcher(t, srv, cfg)
f.connectionWaitTimeout = 100 * time.Millisecond f.connectionWaitTimeout = 100 * time.Millisecond
first, err := f.Fetch(testContext(t), imageURLOnPort(81)) first, err := f.Fetch(testContext(t), imageSourceOnPort(81), "", "")
if err != nil { if err != nil {
t.Fatalf("first Fetch() error = %v", err) t.Fatalf("first Fetch() error = %v", err)
} }
second, err := f.Fetch(testContext(t), imageURLOnPort(82)) second, err := f.Fetch(testContext(t), imageSourceOnPort(82), "", "")
if err != nil { if err != nil {
t.Fatalf("second Fetch() error = %v", err) t.Fatalf("second Fetch() error = %v", err)
} }
@@ -57,7 +59,7 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
start := time.Now() start := time.Now()
_, err = f.Fetch(testContext(t), imageURLOnPort(83)) _, err = f.Fetch(testContext(t), imageSourceOnPort(83), "", "")
if !errors.Is(err, ErrTooManyConnections) { if !errors.Is(err, ErrTooManyConnections) {
t.Fatalf("third Fetch() error = %v, want ErrTooManyConnections", err) t.Fatalf("third Fetch() error = %v, want ErrTooManyConnections", err)
} }
@@ -76,7 +78,7 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
t.Fatalf("close first body: %v", err) t.Fatalf("close first body: %v", err)
} }
third, err := f.Fetch(testContext(t), imageURLOnPort(83)) third, err := f.Fetch(testContext(t), imageSourceOnPort(83), "", "")
if err != nil { if err != nil {
t.Fatalf("Fetch() after a response was closed: error = %v", err) t.Fatalf("Fetch() after a response was closed: error = %v", err)
} }
@@ -94,12 +96,16 @@ func TestFetchFreesHostSlotWhenContextEndsWaitingForConnection(t *testing.T) {
srv := startUpstream(t) srv := startUpstream(t)
cfg := DefaultConfig() cfg := &Config{
cfg.MaxConnections = 1 Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 1,
}
f, _ := newServerFetcher(t, srv, cfg) f, _ := newServerFetcher(t, srv, cfg)
first, err := f.Fetch(testContext(t), imageURLOnPort(81)) first, err := f.Fetch(testContext(t), imageSourceOnPort(81), "", "")
if err != nil { if err != nil {
t.Fatalf("first Fetch() error = %v", err) t.Fatalf("first Fetch() error = %v", err)
} }
@@ -109,7 +115,7 @@ func TestFetchFreesHostSlotWhenContextEndsWaitingForConnection(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond) ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel() defer cancel()
_, err = f.Fetch(ctx, imageURLOnPort(82)) _, err = f.Fetch(ctx, imageSourceOnPort(82), "", "")
if !errors.Is(err, context.DeadlineExceeded) { if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("second Fetch() error = %v, want context.DeadlineExceeded", err) t.Fatalf("second Fetch() error = %v, want context.DeadlineExceeded", err)
} }
@@ -134,7 +140,7 @@ func TestFetchRemovesIdleHostSemaphores(t *testing.T) {
for i := range 100 { for i := range 100 {
wg.Go(func() { wg.Go(func() {
res, err := f.Fetch(ctx, imageURLOnPort(1+i%50)) res, err := f.Fetch(ctx, imageSourceOnPort(1+i%50), "", "")
if err != nil { if err != nil {
t.Errorf("Fetch() error = %v", err) t.Errorf("Fetch() error = %v", err)
@@ -163,19 +169,22 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
srv := startUpstream(t) srv := startUpstream(t)
cfg := DefaultConfig() cfg := &Config{
cfg.MaxConnections = 1 Timeout: 30 * time.Second,
cfg.MaxConnectionsPerHost = 1 MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 1,
MaxConnections: 1,
}
f, _ := newServerFetcher(t, srv, cfg) f, _ := newServerFetcher(t, srv, cfg)
f.connectionWaitTimeout = 100 * time.Millisecond f.connectionWaitTimeout = 100 * time.Millisecond
open, err := f.Fetch(testContext(t), imageURLOnPort(81)) open, err := f.Fetch(testContext(t), imageSourceOnPort(81), "", "")
if err != nil { if err != nil {
t.Fatalf("first Fetch() error = %v", err) t.Fatalf("first Fetch() error = %v", err)
} }
_, err = f.Fetch(testContext(t), imageURLOnPort(82)) _, err = f.Fetch(testContext(t), imageSourceOnPort(82), "", "")
if !errors.Is(err, ErrTooManyConnections) { if !errors.Is(err, ErrTooManyConnections) {
t.Fatalf("Fetch() from another host: error = %v, "+ t.Fatalf("Fetch() from another host: error = %v, "+
"want ErrTooManyConnections", err) "want ErrTooManyConnections", err)
@@ -184,7 +193,7 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond) ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel() defer cancel()
_, err = f.Fetch(ctx, imageURLOnPort(81)) _, err = f.Fetch(ctx, imageSourceOnPort(81), "", "")
if !errors.Is(err, context.DeadlineExceeded) { if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("Fetch() from the busy host: error = %v, "+ t.Fatalf("Fetch() from the busy host: error = %v, "+
"want context.DeadlineExceeded", err) "want context.DeadlineExceeded", err)
@@ -196,7 +205,7 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
ctx, cancel = context.WithTimeout(t.Context(), 100*time.Millisecond) ctx, cancel = context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel() defer cancel()
_, err = f.Fetch(ctx, imageURLOnPort(83)) _, err = f.Fetch(ctx, imageSourceOnPort(83), "", "")
if !errors.Is(err, context.DeadlineExceeded) { if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("Fetch() from a host with nothing open: error = %v, "+ t.Fatalf("Fetch() from a host with nothing open: error = %v, "+
"want context.DeadlineExceeded", err) "want context.DeadlineExceeded", err)
@@ -219,15 +228,15 @@ func TestFetchReleasesConnectionOnError(t *testing.T) {
t.Parallel() t.Parallel()
cases := []struct { cases := []struct {
name string name string
url string source imagerequest.Source
want error want error
}{ }{
{"upstream answers 500", upstreamURL("/status/500"), ErrUpstreamError}, {"upstream answers 500", upstreamSource("/status/500"), ErrUpstreamError},
{"upstream sends HTML", upstreamURL("/html"), ErrInvalidContentType}, {"upstream sends HTML", upstreamSource("/html"), ErrInvalidContentType},
// 198.51.100.7 (TEST-NET-2) passes the SSRF checks, and the test // 198.51.100.7 (TEST-NET-2) passes the SSRF checks, and the test
// dialer refuses every host but testPublicHost. // dialer refuses every host but testPublicHost.
{"connecting fails", "http://198.51.100.7/image", errUnexpectedDial}, {"connecting fails", imageSourceOn("198.51.100.7"), errUnexpectedDial},
} }
for _, tc := range cases { for _, tc := range cases {
@@ -236,13 +245,17 @@ func TestFetchReleasesConnectionOnError(t *testing.T) {
srv := startUpstream(t) srv := startUpstream(t)
cfg := DefaultConfig() cfg := &Config{
cfg.MaxConnections = 1 Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 1,
}
f, _ := newServerFetcher(t, srv, cfg) f, _ := newServerFetcher(t, srv, cfg)
f.connectionWaitTimeout = 100 * time.Millisecond f.connectionWaitTimeout = 100 * time.Millisecond
_, err := f.Fetch(testContext(t), tc.url) _, err := f.Fetch(testContext(t), tc.source, "", "")
if !errors.Is(err, tc.want) { if !errors.Is(err, tc.want) {
t.Fatalf("Fetch() error = %v, want %v", err, tc.want) t.Fatalf("Fetch() error = %v, want %v", err, tc.want)
} }
internal/httpfetcher/mock.go
+21 -42
View File
@@ -7,14 +7,14 @@ import (
"io/fs" "io/fs"
"net/http" "net/http"
"strings" "strings"
)
// errEmptyURLPath is returned when a mock URL has no usable path. "sneak.berlin/go/pixa/internal/imagerequest"
var errEmptyURLPath = errors.New("empty URL path") )
// MockFetcher implements Fetcher using an embedded filesystem. // MockFetcher implements Fetcher using an embedded filesystem.
// Files are organized as: hostname/path/to/file.ext // Files are organized as: hostname/path/to/file.ext
// URLs like https://example.com/images/photo.jpg map to example.com/images/photo.jpg. // The source with host example.com and path /images/photo.jpg maps to
// example.com/images/photo.jpg; the source query is ignored.
type MockFetcher struct { type MockFetcher struct {
fs fs.FS fs fs.FS
} }
@@ -24,8 +24,11 @@ func NewMock(fsys fs.FS) *MockFetcher {
return &MockFetcher{fs: fsys} return &MockFetcher{fs: fsys}
} }
// Fetch retrieves content from the mock filesystem. // Fetch retrieves the source's content from the mock filesystem. It ignores
func (m *MockFetcher) Fetch(ctx context.Context, url string) (*FetchResult, error) { // ifNoneMatch and ifModifiedSince, and always sends the content.
func (m *MockFetcher) Fetch(
ctx context.Context, source imagerequest.Source, _, _ string,
) (*FetchResult, error) {
// Check context cancellation // Check context cancellation
select { select {
case <-ctx.Done(): case <-ctx.Done():
@@ -33,11 +36,7 @@ func (m *MockFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
default: default:
} }
// Parse URL to get filesystem path path := source.Host + source.Path
path, err := urlToFSPath(url)
if err != nil {
return nil, err
}
// Open the file // Open the file
f, err := m.fs.Open(path) f, err := m.fs.Open(path)
@@ -68,31 +67,11 @@ func (m *MockFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
}, nil }, nil
} }
// urlToFSPath converts a URL to a filesystem path. // The content types the mock fetcher gives files in no format pixa reads.
// https://example.com/images/photo.jpg -> example.com/images/photo.jpg const (
func urlToFSPath(rawURL string) (string, error) { contentTypeSVG = "image/svg+xml"
// Strip scheme contentTypeOctetStream = "application/octet-stream"
url := rawURL )
if idx := strings.Index(url, "://"); idx != -1 {
url = url[idx+3:]
}
// Remove query string
if idx := strings.Index(url, "?"); idx != -1 {
url = url[:idx]
}
// Remove fragment
if idx := strings.Index(url, "#"); idx != -1 {
url = url[:idx]
}
if url == "" {
return "", errEmptyURLPath
}
return url, nil
}
// detectContentTypeFromPath returns the MIME type based on file extension. // detectContentTypeFromPath returns the MIME type based on file extension.
func detectContentTypeFromPath(path string) string { func detectContentTypeFromPath(path string) string {
@@ -100,17 +79,17 @@ func detectContentTypeFromPath(path string) string {
switch { switch {
case strings.HasSuffix(path, ".jpg"), strings.HasSuffix(path, ".jpeg"): case strings.HasSuffix(path, ".jpg"), strings.HasSuffix(path, ".jpeg"):
return contentTypeJPEG return imagerequest.FormatJPEG.MIMEType()
case strings.HasSuffix(path, ".png"): case strings.HasSuffix(path, ".png"):
return contentTypePNG return imagerequest.FormatPNG.MIMEType()
case strings.HasSuffix(path, ".gif"): case strings.HasSuffix(path, ".gif"):
return contentTypeGIF return imagerequest.FormatGIF.MIMEType()
case strings.HasSuffix(path, ".webp"): case strings.HasSuffix(path, ".webp"):
return contentTypeWebP return imagerequest.FormatWebP.MIMEType()
case strings.HasSuffix(path, ".avif"): case strings.HasSuffix(path, ".avif"):
return contentTypeAVIF return imagerequest.FormatAVIF.MIMEType()
case strings.HasSuffix(path, ".jxl"): case strings.HasSuffix(path, ".jxl"):
return contentTypeJXL return imagerequest.FormatJXL.MIMEType()
case strings.HasSuffix(path, ".svg"): case strings.HasSuffix(path, ".svg"):
return contentTypeSVG return contentTypeSVG
default: default:
internal/httpfetcher/request_failed_internal_test.go
+148
View File
@@ -0,0 +1,148 @@
package httpfetcher
import (
"context"
"errors"
"net"
"os"
"syscall"
"testing"
"time"
)
// TestFetchFailedBeforeAnswer checks the error of a fetch whose connection
// fails:
// - refused: ErrUpstreamRequestFailed.
// - timed out: ErrUpstreamTimeout, even for a timeout error that does not
// match context.DeadlineExceeded, such as the TLS handshake timeout.
// - cancelled by its caller: context.Canceled, not ErrUpstreamRequestFailed.
func TestFetchFailedBeforeAnswer(t *testing.T) {
t.Parallel()
tests := []struct {
name string
dial func(cancel context.CancelFunc) error
wantErr error
wantNotErr error
}{
{
name: "connection refused",
dial: func(context.CancelFunc) error {
return syscall.ECONNREFUSED
},
wantErr: ErrUpstreamRequestFailed,
wantNotErr: context.Canceled,
},
{
name: "timed out while connecting",
dial: func(context.CancelFunc) error {
return os.ErrDeadlineExceeded
},
wantErr: ErrUpstreamTimeout,
wantNotErr: ErrUpstreamRequestFailed,
},
{
name: "cancelled while connecting",
dial: func(cancel context.CancelFunc) error {
cancel()
return context.Canceled
},
wantErr: context.Canceled,
wantNotErr: ErrUpstreamRequestFailed,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(testContext(t))
defer cancel()
f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
AllowHTTP: true,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
DialContext: func(context.Context, string, string) (net.Conn, error) {
return nil, tt.dial(cancel)
},
})
_, err := f.Fetch(ctx, upstreamSource("/image"), "", "")
t.Logf("Fetch() error: %v", err)
if !errors.Is(err, tt.wantErr) {
t.Errorf("Fetch() error = %v, want %v", err, tt.wantErr)
}
if errors.Is(err, tt.wantNotErr) {
t.Errorf("Fetch() error = %v, must not be %v", err, tt.wantNotErr)
}
})
}
}
// TestFetchAddressLookupTimedOut checks that a fetch whose address lookup
// times out fails with ErrUpstreamTimeout, both for the lookup of the source
// host and for the lookup while connecting. The error must match neither
// ErrInvalidHost nor ErrUpstreamRequestFailed, which the image routes answer
// 502. A context whose deadline has passed makes the lookup time out.
func TestFetchAddressLookupTimedOut(t *testing.T) {
t.Parallel()
checkTimedOut := func(t *testing.T, err error) {
t.Helper()
t.Logf("Fetch() error: %v", err)
if !errors.Is(err, ErrUpstreamTimeout) {
t.Errorf("Fetch() error = %v, want ErrUpstreamTimeout", err)
}
if errors.Is(err, ErrInvalidHost) || errors.Is(err, ErrUpstreamRequestFailed) {
t.Errorf("Fetch() error = %v, must match neither ErrInvalidHost "+
"nor ErrUpstreamRequestFailed", err)
}
}
t.Run("lookup of the source host", func(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithTimeout(testContext(t), 0)
defer cancel()
f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
})
_, err := f.Fetch(ctx, imageSourceOn("cdn.example.com"), "", "")
checkTimedOut(t, err)
})
t.Run("lookup while connecting", func(t *testing.T) {
t.Parallel()
// The source host is an address, which validateURL does not look up;
// the dialer looks up a name
f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) {
ctx, cancel := context.WithTimeout(ctx, 0)
defer cancel()
return dialSSRFSafe(ctx, network, "img.example.com:443", nil)
},
})
_, err := f.Fetch(testContext(t), upstreamSource("/image"), "", "")
checkTimedOut(t, err)
})
}
internal/httpfetcher/request_id_internal_test.go
+3 -2
View File
@@ -8,6 +8,7 @@ import (
"testing" "testing"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestFetchSendsRequestID verifies that a fetch sends the ID of the request // TestFetchSendsRequestID verifies that a fetch sends the ID of the request
@@ -25,7 +26,7 @@ func TestFetchSendsRequestID(t *testing.T) {
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
received <- r.Header.Get("X-Request-Id") received <- r.Header.Get("X-Request-Id")
w.Header().Set("Content-Type", contentTypeJPEG) w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
_, _ = io.WriteString(w, imagePayload) _, _ = io.WriteString(w, imagePayload)
})) }))
t.Cleanup(srv.Close) t.Cleanup(srv.Close)
@@ -34,7 +35,7 @@ func TestFetchSendsRequestID(t *testing.T) {
ctx := context.WithValue(testContext(t), middleware.RequestIDKey, requestID) ctx := context.WithValue(testContext(t), middleware.RequestIDKey, requestID)
res, err := f.Fetch(ctx, upstreamURL("/image")) res, err := f.Fetch(ctx, upstreamSource("/image"), "", "")
if err != nil { if err != nil {
t.Fatalf("Fetch() error = %v", err) t.Fatalf("Fetch() error = %v", err)
} }
internal/imageprocessor/avif_too_large_internal_test.go
+98
View File
@@ -0,0 +1,98 @@
package imageprocessor
import (
"bytes"
"encoding/binary"
"errors"
"os"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_RefusesAVIFAboveMaxPixels verifies that an AVIF output
// of more than MaxAVIFOutputPixels is refused with ErrAVIFOutputTooLarge
// before anything is decoded, whatever size and fit give it, the source's own
// size included, and that an AVIF output of at most that many pixels, or one
// in another format, is not. One input is a 64x32 PNG cut off in its pixel
// data. The other is testdata/red.avif, a 4x4 AVIF, with the size in its
// header changed to 4097x4096. libvips reads each input's size from its
// header but fails when it decodes its pixel data, so a request that gets
// that failure was not refused, and a refusal came before any decoding.
func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
t.Parallel()
whole := createTestPNG(t, 64, 32)
cutOff := whole[:len(whole)/2]
largeAVIF, err := os.ReadFile("testdata/red.avif")
if err != nil {
t.Fatalf("failed to read test AVIF: %v", err)
}
// The ispe box gives the width and then the height, 4 bytes each, after
// the box's type and its 4 bytes of version and flags
ispe := bytes.Index(largeAVIF, []byte("ispe"))
binary.BigEndian.PutUint32(largeAVIF[ispe+8:], 4097)
binary.BigEndian.PutUint32(largeAVIF[ispe+12:], 4096)
tests := []struct {
name string
input []byte
sourceFormat imagerequest.Format
format imagerequest.Format
width, height int
fit imagerequest.Fit
refused bool
}{
{"AVIF at the limit", cutOff, imagerequest.FormatPNG,
imagerequest.FormatAVIF, 4096, 4096, imagerequest.FitCover, false},
{"AVIF above the limit", cutOff, imagerequest.FormatPNG,
imagerequest.FormatAVIF, 4096, 4097, imagerequest.FitCover, true},
{"WebP above the AVIF limit", cutOff, imagerequest.FormatPNG,
imagerequest.FormatWebP, 4096, 4097, imagerequest.FitCover, false},
// FitInside does not enlarge, so the output stays 64x32
{"AVIF inside a larger size", cutOff, imagerequest.FormatPNG,
imagerequest.FormatAVIF, 8192, 8192, imagerequest.FitInside, false},
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
{"AVIF outside a smaller size", cutOff, imagerequest.FormatPNG,
imagerequest.FormatAVIF, 8192, 1, imagerequest.FitOutside, true},
// With no size, the output keeps the source's 4097x4096
{"AVIF of the source's size", largeAVIF, imagerequest.FormatAVIF,
imagerequest.FormatAVIF, 0, 0, imagerequest.FitCover, true},
{"orig of an AVIF source", largeAVIF, imagerequest.FormatAVIF,
imagerequest.FormatOriginal, 0, 0, imagerequest.FitCover, true},
{"JPEG of the source's size", largeAVIF, imagerequest.FormatAVIF,
imagerequest.FormatJPEG, 0, 0, imagerequest.FitCover, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
_, err := New(Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(), bytes.NewReader(tt.input), tt.sourceFormat,
&imagerequest.Transform{
Size: imagerequest.Size{Width: tt.width, Height: tt.height},
Format: tt.format,
FitMode: tt.fit,
})
t.Logf("Process() error = %v", err)
if tt.refused {
if !errors.Is(err, ErrAVIFOutputTooLarge) {
t.Errorf("Process() error = %v, want %v", err, ErrAVIFOutputTooLarge)
}
return
}
if err == nil || errors.Is(err, ErrAVIFOutputTooLarge) {
t.Errorf("Process() error = %v, want the failure to decode "+
"the input", err)
}
})
}
}
internal/imageprocessor/computed_side_internal_test.go
+141
View File
@@ -0,0 +1,141 @@
package imageprocessor
import (
"bytes"
"errors"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_RefusesComputedSideAboveMaximum verifies that a size
// with one side 0, which Process computes from the input's proportions, is
// served when that side comes out at MaxOutputDimension and refused with
// ErrOutputTooLarge when it would be above. A 4096x1 or 1x4096 input given
// 2 for its short side has 8192 for its long side.
func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
size imagerequest.Size
wantErr error
}{
{"width at the maximum", 4096, 1,
imagerequest.Size{Height: 2}, nil},
{"width above the maximum", 4096, 1,
imagerequest.Size{Height: 3}, ErrOutputTooLarge},
{"height at the maximum", 1, 4096,
imagerequest.Size{Width: 2}, nil},
{"height above the maximum", 1, 4096,
imagerequest.Size{Width: 3}, ErrOutputTooLarge},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
FitMode: imagerequest.FitCover,
})
if result != nil {
t.Logf("Process() made %dx%d", result.Width, result.Height)
if max(result.Width, result.Height) != MaxOutputDimension {
t.Errorf("Process() made %dx%d, want a side of %d",
result.Width, result.Height, MaxOutputDimension)
}
}
if !errors.Is(err, tt.wantErr) {
t.Errorf("Process() error = %v, want %v", err, tt.wantErr)
}
})
}
}
// TestImageProcessor_ComputedSideIsAtLeastOne verifies that, for a size with
// one side 0, the side computed from the input's proportions is 1 where it
// would round down to 0, and that Process makes that size in every fit mode
// but FitOutside, which scales the input to cover the size and so can make
// it longer. Each case checks the size targetDimensions gives before it
// calls Process: with that side at 0, an 8192x1 input at 1300x0 makes
// libvips ask for about 39 GB with FitCover.
func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
size imagerequest.Size
wantWidth, wantHeight int
}{
{"8192x1 at 100x0", 8192, 1, imagerequest.Size{Width: 100}, 100, 1},
{"8192x1 at 1300x0", 8192, 1, imagerequest.Size{Width: 1300}, 1300, 1},
{"8192x2 at 100x0", 8192, 2, imagerequest.Size{Width: 100}, 100, 1},
{"1x8192 at 0x100", 1, 8192, imagerequest.Size{Height: 100}, 1, 100},
{"1x8192 at 0x1300", 1, 8192, imagerequest.Size{Height: 1300}, 1, 1300},
{"2x8192 at 0x100", 2, 8192, imagerequest.Size{Height: 100}, 1, 100},
}
fits := []imagerequest.Fit{
imagerequest.FitCover, imagerequest.FitContain, imagerequest.FitFill,
imagerequest.FitInside, imagerequest.FitOutside,
}
for _, tt := range tests {
for _, fit := range fits {
t.Run(tt.name+" "+string(fit), func(t *testing.T) {
t.Parallel()
width, height, err := targetDimensions(tt.size,
tt.inputWidth, tt.inputHeight)
if err != nil || width != tt.wantWidth || height != tt.wantHeight {
t.Fatalf("targetDimensions() = %dx%d, %v, want %dx%d",
width, height, err, tt.wantWidth, tt.wantHeight)
}
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
FitMode: fit,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
t.Logf("Process() made %dx%d", result.Width, result.Height)
ok := result.Width == tt.wantWidth && result.Height == tt.wantHeight
if fit == imagerequest.FitOutside {
ok = result.Width >= tt.wantWidth && result.Height >= tt.wantHeight
}
if !ok {
t.Errorf("Process() made %dx%d, want %dx%d (at least, for %s)",
result.Width, result.Height, tt.wantWidth, tt.wantHeight,
imagerequest.FitOutside)
}
})
}
}
}
internal/imageprocessor/deadline_internal_test.go
+109
View File
@@ -0,0 +1,109 @@
package imageprocessor
import (
"bytes"
"context"
"errors"
"io"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// readerAfterDeadline is a Process input whose first Read waits until done
// is closed, so the deadline passes after Process has taken its slot and
// before it encodes.
type readerAfterDeadline struct {
data *bytes.Reader
done <-chan struct{}
waited bool
}
func (r *readerAfterDeadline) Read(p []byte) (int, error) {
if !r.waited {
r.waited = true
<-r.done
}
return r.data.Read(p)
}
// inputNotToBeRead is a Process input that fails the test when it is read.
type inputNotToBeRead struct {
t *testing.T
}
func (r inputNotToBeRead) Read([]byte) (int, error) {
r.t.Error("Process read its input although its deadline had passed")
return 0, io.EOF
}
// TestProcessStopsWhenDeadlinePassed checks that Process returns
// context.DeadlineExceeded, and holds no slot afterwards, when its context's
// deadline has passed before it starts, then without reading its input, or
// while it reads its input. govips gives pixa no way to stop libvips once an
// encode has begun, so Process must not begin one for a request that has
// ended.
func TestProcessStopsWhenDeadlinePassed(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, 50, 50)
t.Run("before Process starts", func(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithDeadline(t.Context(),
time.Now().Add(-time.Second))
defer cancel()
// With a slot free, the select in acquireSlot takes the slot or sees
// the ended context at random, so Process is called many times
for range 100 {
checkProcessStopsAtDeadline(ctx, t, inputNotToBeRead{t: t})
}
})
t.Run("while Process reads its input", func(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
checkProcessStopsAtDeadline(ctx, t, &readerAfterDeadline{
data: bytes.NewReader(input), done: ctx.Done(),
})
})
}
// checkProcessStopsAtDeadline runs Process on input with ctx, whose deadline
// has passed by the time Process could encode, and fails the test unless
// Process returns context.DeadlineExceeded and gives its slot back.
func checkProcessStopsAtDeadline(
ctx context.Context, t *testing.T, input io.Reader,
) {
t.Helper()
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
result, err := proc.Process(ctx, input, imagerequest.FormatJPEG,
smallJPEGTransform())
if result != nil {
t.Logf("Process() made a %dx%d image after its deadline passed",
result.Width, result.Height)
_ = result.Content.Close()
}
t.Logf("Process() error = %v", err)
if !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("Process() error = %v, want context.DeadlineExceeded", err)
}
if held := len(proc.processingSemaphore); held != 0 {
t.Errorf("%d slots still held after Process() returned", held)
}
}
internal/imageprocessor/empty_format_internal_test.go
+8 -2
View File
@@ -4,6 +4,8 @@ import (
"bytes" "bytes"
"errors" "errors"
"testing" "testing"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestImageProcessor_EmptyFormatRefused verifies that a request with no format // TestImageProcessor_EmptyFormatRefused verifies that a request with no format
@@ -12,8 +14,12 @@ import (
func TestImageProcessor_EmptyFormatRefused(t *testing.T) { func TestImageProcessor_EmptyFormatRefused(t *testing.T) {
t.Parallel() t.Parallel()
_, err := New(Params{}).Process( _, err := New(Params{
t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)), &Request{}, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(
t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)),
imagerequest.FormatJPEG, &imagerequest.Transform{},
) )
if !errors.Is(err, ErrUnsupportedOutputFormat) { if !errors.Is(err, ErrUnsupportedOutputFormat) {
t.Errorf("Process() error = %v, want %v", err, ErrUnsupportedOutputFormat) t.Errorf("Process() error = %v, want %v", err, ErrUnsupportedOutputFormat)
internal/imageprocessor/encoder_settings_internal_test.go
+50 -13
View File
@@ -8,14 +8,21 @@ import (
"testing" "testing"
"github.com/davidbyttow/govips/v2/vips" "github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// processedSize runs input through Process and returns the output's size in // processedSize runs input, an image in sourceFormat, through Process and
// bytes. // returns the output's size in bytes.
func processedSize(t *testing.T, input []byte, req *Request) int64 { func processedSize(
t *testing.T, input []byte, sourceFormat imagerequest.Format,
transform *imagerequest.Transform,
) int64 {
t.Helper() t.Helper()
result, err := New(Params{}).Process(t.Context(), bytes.NewReader(input), req) result, err := New(Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(), bytes.NewReader(input), sourceFormat, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -68,7 +75,8 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
} }
} }
size := processedSize(t, encodePNG(t, flat), &Request{Format: FormatPNG}) size := processedSize(t, encodePNG(t, flat), imagerequest.FormatPNG,
&imagerequest.Transform{Format: imagerequest.FormatPNG})
const rawBytes = width * height * 3 const rawBytes = width * height * 3
if size > rawBytes/10 { if size > rawBytes/10 {
@@ -77,27 +85,28 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
} }
} }
// TestImageProcessor_WebPAtDefaultEffort verifies that WebP is saved at // TestImageProcessor_WebPAtEffort2 verifies that WebP is saved at effort 2,
// libvips' default effort, 4: the output is the size of the same image saved // not libvips' default of 4: the output is the size of the same image saved
// at that effort. // at effort 2.
func TestImageProcessor_WebPAtDefaultEffort(t *testing.T) { func TestImageProcessor_WebPAtEffort2(t *testing.T) {
t.Parallel() t.Parallel()
input := createTestJPEG(t, 200, 150) input := createTestJPEG(t, 200, 150)
size := processedSize(t, input, &Request{Format: FormatWebP, Quality: 85}) size := processedSize(t, input, imagerequest.FormatJPEG,
&imagerequest.Transform{Format: imagerequest.FormatWebP, Quality: 85})
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{ want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
StripMetadata: true, StripMetadata: true,
Quality: 85, Quality: 85,
ReductionEffort: 4, ReductionEffort: 2,
}) })
if err != nil { if err != nil {
t.Fatalf("ExportWebp() error = %v", err) t.Fatalf("ExportWebp() error = %v", err)
} }
if size != int64(len(want)) { if size != int64(len(want)) {
t.Errorf("WebP output is %d bytes, want %d, the size at effort 4", t.Errorf("WebP output is %d bytes, want %d, the size at effort 2",
size, len(want)) size, len(want))
} }
} }
@@ -126,7 +135,8 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
input := encodePNG(t, source) input := encodePNG(t, source)
size := processedSize(t, input, &Request{Format: FormatAVIF, Quality: 85}) size := processedSize(t, input, imagerequest.FormatPNG,
&imagerequest.Transform{Format: imagerequest.FormatAVIF, Quality: 85})
want, _, err := decode(t, input).ExportAvif(&vips.AvifExportParams{ want, _, err := decode(t, input).ExportAvif(&vips.AvifExportParams{
StripMetadata: true, StripMetadata: true,
@@ -143,3 +153,30 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
"and 8 bits", size, len(want)) "and 8 bits", size, len(want))
} }
} }
// TestImageProcessor_JPEGXLAtEffort4 verifies that JPEG XL is saved at
// effort 4, not libvips' default of 7: the output is the size of the same
// image saved at effort 4, with the distance exportJXL gives quality 85.
func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, 200, 150)
size := processedSize(t, input, imagerequest.FormatJPEG,
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: 85})
params := vips.NewJxlExportParams()
params.Quality = 85
params.Distance = jxlDistance(85)
params.Effort = 4
want, _, err := decode(t, input).ExportJxl(params)
if err != nil {
t.Fatalf("ExportJxl() error = %v", err)
}
if size != int64(len(want)) {
t.Errorf("JPEG XL output is %d bytes, want %d, the size at effort 4",
size, len(want))
}
}
internal/imageprocessor/fit_cover_internal_test.go
+287
View File
@@ -0,0 +1,287 @@
package imageprocessor
import (
"bytes"
"image"
"image/color"
"image/png"
"math"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_FitCoverServesThinSource verifies that FitCover serves
// a thin source at the requested size. An 8192x1 source scaled to cover
// 8192x8192 is 8192 times as long as the output on its long side. The centre
// of an 8192x1 source with the proportions of 1x8192 is 1/8192 of a pixel
// wide. Each case is also turned on its side.
func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
size imagerequest.Size
}{
{"8192x1 at 8192x8192", 8192, 1, imagerequest.Size{Width: 8192, Height: 8192}},
{"8192x1 at 1x8192", 8192, 1, imagerequest.Size{Width: 1, Height: 8192}},
{"1x8192 at 8192x8192", 1, 8192, imagerequest.Size{Width: 8192, Height: 8192}},
{"1x8192 at 8192x1", 1, 8192, imagerequest.Size{Width: 8192, Height: 1}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
FitMode: imagerequest.FitCover,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
t.Logf("Process() made %dx%d", result.Width, result.Height)
if result.Width != tt.size.Width || result.Height != tt.size.Height {
t.Errorf("Process() made %dx%d, want %dx%d",
result.Width, result.Height, tt.size.Width, tt.size.Height)
}
})
}
}
// createBandsPNG creates a PNG of three equal bands, red, green and blue,
// side by side when it is wider than tall and top to bottom otherwise.
func createBandsPNG(t *testing.T, width, height int) []byte {
t.Helper()
bands := []color.RGBA{
{R: 255, A: 255},
{G: 255, A: 255},
{B: 255, A: 255},
}
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
band := 3 * y / height
if width > height {
band = 3 * x / width
}
img.Set(x, y, bands[band])
}
}
return encodePNG(t, img)
}
// TestImageProcessor_FitCoverCropsTheCentre verifies that FitCover keeps the
// centre of the source: a 300x100 source of red, green and blue bands side
// by side, or a 100x300 one with them top to bottom, comes out at 50x50 as
// its green band.
func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
}{
{"bands side by side", 300, 100},
{"bands top to bottom", 100, 300},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createBandsPNG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatPNG,
&imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatPNG,
FitMode: imagerequest.FitCover,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
output, err := png.Decode(result.Content)
if err != nil {
t.Fatalf("failed to decode output: %v", err)
}
bounds := output.Bounds()
t.Logf("Process() made %dx%d", bounds.Dx(), bounds.Dy())
if bounds.Dx() != 50 || bounds.Dy() != 50 {
t.Fatalf("Process() made %dx%d, want 50x50",
bounds.Dx(), bounds.Dy())
}
// Resampling may blend a little of the bands beside the
// green one into the output's edges.
for y := bounds.Min.Y; y < bounds.Max.Y; y++ {
for x := bounds.Min.X; x < bounds.Max.X; x++ {
r, g, b, _ := output.At(x, y).RGBA()
if r>>8 > 64 || g>>8 < 192 || b>>8 > 64 {
t.Fatalf("output pixel %d,%d is %d,%d,%d, want green",
x, y, r>>8, g>>8, b>>8)
}
}
}
})
}
}
// The greys of createSquarePNG, and the grey halfway between them.
const (
backgroundGrey = 192
squareGrey = 64
halfwayGrey = (backgroundGrey + squareGrey) / 2
)
// createSquarePNG creates a grey PNG side pixels square with a darker grey
// square squareSide pixels wide in its centre.
func createSquarePNG(t *testing.T, side, squareSide int) []byte {
t.Helper()
first := (side - squareSide) / 2
last := first + squareSide - 1
img := image.NewGray(image.Rect(0, 0, side, side))
for y := range side {
for x := range side {
grey := uint8(backgroundGrey)
if x >= first && x <= last && y >= first && y <= last {
grey = squareGrey
}
img.SetGray(x, y, color.Gray{Y: grey})
}
}
return encodePNG(t, img)
}
// squareLength returns how long the square is along greys, the greys of a
// row or column of the output that crosses it: the distance between the
// first and the last point where greys passes halfwayGrey, or 0 when there
// is no such point. Resampling blurs the square's edges over several
// pixels, and each point is placed between the two pixels on either side
// of it in proportion to their greys, so the length is measured to a
// fraction of a pixel.
func squareLength(greys []float64) float64 {
first, last := -1.0, -1.0
for i := 1; i < len(greys); i++ {
before, after := greys[i-1], greys[i]
if (before > halfwayGrey) == (after > halfwayGrey) {
continue
}
point := float64(i-1) + (before-halfwayGrey)/(before-after)
if first < 0 {
first = point
}
last = point
}
return last - first
}
// TestImageProcessor_FitCoverKeepsProportions verifies that FitCover scales
// both sides of the source by the same amount: a square in the centre of
// the source comes out square. Its width and height are measured along the
// output's middle row and column.
func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
t.Parallel()
tests := []struct {
name string
side, squareSide int
size imagerequest.Size
}{
{"16x16 at 100x60", 16, 8, imagerequest.Size{Width: 100, Height: 60}},
{"16x16 at 60x100", 16, 8, imagerequest.Size{Width: 60, Height: 100}},
{"10x10 at 100x35", 10, 2, imagerequest.Size{Width: 100, Height: 35}},
{"10x10 at 35x100", 10, 2, imagerequest.Size{Width: 35, Height: 100}},
{"64x64 at 300x100", 64, 16, imagerequest.Size{Width: 300, Height: 100}},
{"64x64 at 100x300", 64, 16, imagerequest.Size{Width: 100, Height: 300}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createSquarePNG(t, tt.side, tt.squareSide)
result, err := New(Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatPNG,
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatPNG,
FitMode: imagerequest.FitCover,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
output, err := png.Decode(result.Content)
if err != nil {
t.Fatalf("failed to decode output: %v", err)
}
bounds := output.Bounds()
if bounds.Dx() != tt.size.Width || bounds.Dy() != tt.size.Height {
t.Fatalf("Process() made %dx%d, want %dx%d",
bounds.Dx(), bounds.Dy(), tt.size.Width, tt.size.Height)
}
row := make([]float64, bounds.Dx())
for x := range row {
r, _, _, _ := output.At(x, bounds.Dy()/2).RGBA()
row[x] = float64(r >> 8)
}
column := make([]float64, bounds.Dy())
for y := range column {
r, _, _, _ := output.At(bounds.Dx()/2, y).RGBA()
column[y] = float64(r >> 8)
}
width, height := squareLength(row), squareLength(column)
t.Logf("square is %.2f wide and %.2f high", width, height)
if width == 0 || math.Abs(width-height) > 0.5 {
t.Errorf("square is %.2f wide and %.2f high, want it square",
width, height)
}
})
}
}
internal/imageprocessor/fit_outside_internal_test.go
+125
View File
@@ -0,0 +1,125 @@
package imageprocessor
import (
"bytes"
"errors"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_RefusesFitOutsideAboveMaximum verifies that FitOutside,
// which scales the input to cover the requested size and so past it on one
// side, is refused with ErrOutputTooLarge when that side would come out above
// MaxOutputDimension, and otherwise makes the scaled size. A 1024x256 or
// 256x1024 input scaled by 8 has 8192 for its long side.
func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
size imagerequest.Size
// wantWidth and wantHeight are the output's, 0 for a refusal
wantWidth, wantHeight int
}{
{"width above the maximum", 3000, 600,
imagerequest.Size{Width: 1, Height: 8192}, 0, 0},
{"height above the maximum", 600, 3000,
imagerequest.Size{Width: 8192, Height: 1}, 0, 0},
{"width at the maximum", 1024, 256,
imagerequest.Size{Width: 1, Height: 2048}, 8192, 2048},
{"height at the maximum", 256, 1024,
imagerequest.Size{Width: 2048, Height: 1}, 2048, 8192},
{"within the maximum", 3000, 600,
imagerequest.Size{Width: 1, Height: 1200}, 6000, 1200},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
FitMode: imagerequest.FitOutside,
})
if result != nil {
t.Logf("Process() made %dx%d", result.Width, result.Height)
}
if tt.wantWidth == 0 {
if !errors.Is(err, ErrOutputTooLarge) {
t.Errorf("Process() error = %v, want %v", err, ErrOutputTooLarge)
}
return
}
if err != nil {
t.Fatalf("Process() error = %v", err)
}
if result.Width != tt.wantWidth || result.Height != tt.wantHeight {
t.Errorf("Process() made %dx%d, want %dx%d",
result.Width, result.Height, tt.wantWidth, tt.wantHeight)
}
})
}
}
// TestImageProcessor_FitOutsideSideIsAtLeastOne verifies that FitOutside
// keeps each side at 1 or more when it works both sides out again from the
// larger scale. A side of 49 scaled to 1 is 49 times 1/49, which in floating
// point is just under 1 and so rounded down to 0, a width libvips refuses.
func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
size imagerequest.Size
wantWidth, wantHeight int
}{
{"49x8192 at 0x100", 49, 8192, imagerequest.Size{Height: 100}, 1, 167},
{"49x49 at 1x1", 49, 49, imagerequest.Size{Width: 1, Height: 1}, 1, 1},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
FitMode: imagerequest.FitOutside,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
t.Logf("Process() made %dx%d", result.Width, result.Height)
if result.Width != tt.wantWidth || result.Height != tt.wantHeight {
t.Errorf("Process() made %dx%d, want %dx%d",
result.Width, result.Height, tt.wantWidth, tt.wantHeight)
}
})
}
}
internal/imageprocessor/imageprocessor.go
+322 -288
View File
@@ -7,11 +7,11 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"runtime"
"sync" "sync"
"time" "time"
"github.com/davidbyttow/govips/v2/vips" "github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// vipsOnce ensures vips is initialized exactly once. // vipsOnce ensures vips is initialized exactly once.
@@ -20,11 +20,10 @@ import (
var vipsOnce sync.Once var vipsOnce sync.Once
// initVips initializes libvips with quiet logging, one worker thread per // initVips initializes libvips with quiet logging, one worker thread per
// image and no operation cache. Process already works on one image per CPU // image and no operation cache. Each request decodes different source
// by default, so more threads per image would only compete for the CPUs. // bytes, so the operation cache would rarely be hit and would hold memory
// Each request decodes different source bytes, so the operation cache // outside MaxConcurrentProcessing; repeated requests are served from
// would rarely be hit and would hold memory outside MaxConcurrentProcessing; // pixa's disk cache instead.
// repeated requests are served from pixa's disk cache instead.
func initVips() { func initVips() {
vipsOnce.Do(func() { vipsOnce.Do(func() {
vips.LoggingSettings(nil, vips.LogLevelError) vips.LoggingSettings(nil, vips.LogLevelError)
@@ -55,49 +54,6 @@ func CheckJPEGXLSupport() error {
return nil return nil
} }
// Format represents supported output image formats.
type Format string
// Supported image output formats.
const (
FormatOriginal Format = "orig"
FormatJPEG Format = "jpeg"
FormatPNG Format = "png"
FormatWebP Format = "webp"
FormatAVIF Format = "avif"
FormatJXL Format = "jxl"
FormatGIF Format = "gif"
)
// FitMode represents how to fit an image into requested dimensions.
type FitMode string
// Supported image fit modes.
const (
FitCover FitMode = "cover"
FitContain FitMode = "contain"
FitFill FitMode = "fill"
FitInside FitMode = "inside"
FitOutside FitMode = "outside"
)
// ErrInvalidFitMode is returned when an invalid fit mode is provided.
var ErrInvalidFitMode = errors.New("invalid fit mode")
// Size represents requested image dimensions.
type Size struct {
Width int
Height int
}
// Request holds the parameters for image processing.
type Request struct {
Size Size
Format Format
Quality int
FitMode FitMode
}
// Result contains the output of image processing. // Result contains the output of image processing.
type Result struct { type Result struct {
// Content is the processed image data. // Content is the processed image data.
@@ -114,18 +70,12 @@ type Result struct {
InputWidth int InputWidth int
// InputHeight is the original image height before processing. // InputHeight is the original image height before processing.
InputHeight int InputHeight int
// InputFormat is the detected input format (e.g., "jpeg", "png").
InputFormat string
} }
// MaxInputDimension is the maximum allowed width or height for input images. // MaxInputDimension is the maximum allowed width or height for input images.
// Images larger than this are rejected to prevent DoS via decompression bombs. // Images larger than this are rejected to prevent DoS via decompression bombs.
const MaxInputDimension = 8192 const MaxInputDimension = 8192
// DefaultMaxInputBytes is the default maximum input size in bytes (50 MiB).
// This matches the default upstream fetcher limit.
const DefaultMaxInputBytes = 50 << 20
// ErrInputTooLarge is returned when input image dimensions exceed MaxInputDimension. // ErrInputTooLarge is returned when input image dimensions exceed MaxInputDimension.
var ErrInputTooLarge = errors.New("input image dimensions exceed maximum") var ErrInputTooLarge = errors.New("input image dimensions exceed maximum")
@@ -133,14 +83,38 @@ var ErrInputTooLarge = errors.New("input image dimensions exceed maximum")
// configured byte limit. // configured byte limit.
var ErrInputDataTooLarge = errors.New("input data exceeds maximum allowed size") var ErrInputDataTooLarge = errors.New("input data exceeds maximum allowed size")
// MaxOutputDimension is the maximum width or height of an output image, and
// so of a requested size (imagerequest.MaxDimension).
const MaxOutputDimension = imagerequest.MaxDimension
// ErrOutputTooLarge is returned, before resizing, when the output would have a
// side above MaxOutputDimension: the side of a size with one side 0, computed
// from the input's proportions, or with FitOutside, the side scaled past the
// requested size.
var ErrOutputTooLarge = errors.New("output image dimensions exceed maximum")
// MaxAVIFOutputPixels is the most pixels an AVIF output may have, so that an
// encode takes at most 40 seconds of the default downstream_timeout of 60
// seconds; AVIF is already at the lowest effort govips can set (avifEffort).
// With one thread, as pixad runs libvips, saving 4096x4096, 8192x2048 or
// 2048x8192 random pixels with an alpha channel takes up to about 34
// seconds, the slowest at any quality and at 8 or 16 bits per sample, and
// 5120x4096 at 8 bits already takes about 36.
const MaxAVIFOutputPixels = 4096 * 4096
// ErrAVIFOutputTooLarge is returned, before anything is decoded, when an AVIF
// output would have more than MaxAVIFOutputPixels pixels.
var ErrAVIFOutputTooLarge = errors.New("AVIF output has too many pixels")
// ErrUnsupportedOutputFormat is returned when the requested output format is // ErrUnsupportedOutputFormat is returned when the requested output format is
// not supported. // not supported.
var ErrUnsupportedOutputFormat = errors.New("unsupported output format") var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
// ErrUnsupportedInputFormat is returned, before the input is decoded, when // ErrUnsupportedInputFormat is returned, before the input is decoded, when
// libvips reads the input's first bytes as another format than the source // libvips reads the input's first bytes as another format than its source
// formats, or as none: for example an SVG whose first bytes pass the // format, or as none. An SVG whose first bytes pass the first-bytes check for
// first-bytes check for AVIF. // AVIF is one example. A source format other than the six pixa reads is also
// refused with it.
var ErrUnsupportedInputFormat = errors.New("unsupported input format") var ErrUnsupportedInputFormat = errors.New("unsupported input format")
// ErrTooManyImages is returned when MaxConcurrentProcessing images are being // ErrTooManyImages is returned when MaxConcurrentProcessing images are being
@@ -162,47 +136,40 @@ type ImageProcessor struct {
processingWaitTimeout time.Duration processingWaitTimeout time.Duration
} }
// Params holds configuration for creating an ImageProcessor. // Params holds configuration for creating an ImageProcessor. Both fields
// Zero values use sensible defaults (MaxInputBytes defaults to DefaultMaxInputBytes). // must be at least 1.
type Params struct { type Params struct {
// MaxInputBytes is the maximum allowed input size in bytes. // MaxInputBytes is the maximum allowed input size in bytes.
// If <= 0, DefaultMaxInputBytes is used.
MaxInputBytes int64 MaxInputBytes int64
// MaxConcurrentProcessing is the most images processed at once. // MaxConcurrentProcessing is the most images processed at once.
// If <= 0, the number of CPUs Go uses (runtime.GOMAXPROCS(0)) is used.
MaxConcurrentProcessing int MaxConcurrentProcessing int
} }
// New creates a new image processor with the given parameters. // New creates a new image processor with the given parameters.
// A zero-value Params{} uses sensible defaults.
func New(params Params) *ImageProcessor { func New(params Params) *ImageProcessor {
initVips() initVips()
maxInputBytes := params.MaxInputBytes
if maxInputBytes <= 0 {
maxInputBytes = DefaultMaxInputBytes
}
maxConcurrentProcessing := params.MaxConcurrentProcessing
if maxConcurrentProcessing <= 0 {
maxConcurrentProcessing = runtime.GOMAXPROCS(0)
}
return &ImageProcessor{ return &ImageProcessor{
maxInputBytes: maxInputBytes, maxInputBytes: params.MaxInputBytes,
processingSemaphore: make(chan struct{}, maxConcurrentProcessing), processingSemaphore: make(chan struct{}, params.MaxConcurrentProcessing),
processingWaitTimeout: ProcessingWaitTimeout, processingWaitTimeout: ProcessingWaitTimeout,
} }
} }
// Process transforms an image according to the request. When // Process transforms the image read from input according to transform.
// MaxConcurrentProcessing images are already being processed, it waits up // sourceFormat is the input's format, which magic detected from its first
// to ProcessingWaitTimeout for one to finish, then fails with // bytes. When MaxConcurrentProcessing images are already being processed,
// ErrTooManyImages. // it waits up to ProcessingWaitTimeout for one to finish, then fails with
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
// encodes, it returns ctx's error and gives the slot back. govips gives pixa
// no way to stop libvips once an encode has begun, so an encode keeps its
// slot until it finishes. An AVIF output of more than MaxAVIFOutputPixels is
// refused with ErrAVIFOutputTooLarge before anything is decoded.
func (p *ImageProcessor) Process( func (p *ImageProcessor) Process(
ctx context.Context, ctx context.Context,
input io.Reader, input io.Reader,
req *Request, sourceFormat imagerequest.Format,
transform *imagerequest.Transform,
) (*Result, error) { ) (*Result, error) {
release, err := p.acquireSlot(ctx) release, err := p.acquireSlot(ctx)
if err != nil { if err != nil {
@@ -211,27 +178,14 @@ func (p *ImageProcessor) Process(
defer release() defer release()
// Read input with a size limit to prevent unbounded memory consumption. data, err := p.readInput(input)
// We read at most maxInputBytes+1 so we can detect if the input exceeds
// the limit without consuming additional memory.
limited := io.LimitReader(input, p.maxInputBytes+1)
data, err := io.ReadAll(limited)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read input: %w", err) return nil, err
} }
if int64(len(data)) > p.maxInputBytes { // Input libvips would not load as sourceFormat is refused before it is
return nil, ErrInputDataTooLarge // decoded
} err = checkInputFormat(data, sourceFormat)
// govips picks the libvips loader from the input's first bytes with
// DetermineImageType. Input whose first bytes it reads as no source
// format, such as SVG, which librsvg would render, is refused before
// libvips reads it.
inputFormat := p.detectFormat(vips.DetermineImageType(data))
sourceFormat, err := p.formatFromString(inputFormat)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -259,25 +213,37 @@ func (p *ImageProcessor) Process(
return nil, ErrInputTooLarge return nil, ErrInputTooLarge
} }
// Determine target dimensions // Determine target dimensions; too large ones are refused before resizing
targetWidth, targetHeight := targetDimensions(req.Size, origWidth, origHeight) targetWidth, targetHeight, err := targetDimensions(transform.Size,
origWidth, origHeight)
if err != nil {
return nil, err
}
// orig is the source's own format; encode refuses an empty format
outputFormat := transform.Format
if outputFormat == imagerequest.FormatOriginal {
outputFormat = sourceFormat
}
// Determine output dimensions; too large ones are refused before
// resizing, which with FitCover already decodes the input
outputWidth, outputHeight, err := outputDimensions(origWidth, origHeight,
targetWidth, targetHeight, transform.FitMode, outputFormat)
if err != nil {
return nil, err
}
// Resize if needed // Resize if needed
if targetWidth != origWidth || targetHeight != origHeight { if targetWidth != origWidth || targetHeight != origHeight {
err := p.resize(img, targetWidth, targetHeight, req.FitMode) err := p.resize(img, outputWidth, outputHeight, transform.FitMode)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to resize: %w", err) return nil, fmt.Errorf("failed to resize: %w", err)
} }
} }
// orig is the source's own format; encode refuses an empty format
outputFormat := req.Format
if outputFormat == FormatOriginal {
outputFormat = sourceFormat
}
// Encode to target format // Encode to target format
output, err := p.encode(img, outputFormat, req.Quality) output, err := p.encode(ctx, img, outputFormat, transform.Quality)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to encode: %w", err) return nil, fmt.Errorf("failed to encode: %w", err)
} }
@@ -285,88 +251,43 @@ func (p *ImageProcessor) Process(
return &Result{ return &Result{
Content: io.NopCloser(bytes.NewReader(output)), Content: io.NopCloser(bytes.NewReader(output)),
ContentLength: int64(len(output)), ContentLength: int64(len(output)),
ContentType: FormatToMIME(outputFormat), ContentType: outputFormat.MIMEType(),
Width: img.Width(), Width: img.Width(),
Height: img.Height(), Height: img.Height(),
InputWidth: origWidth, InputWidth: origWidth,
InputHeight: origHeight, InputHeight: origHeight,
InputFormat: inputFormat,
}, nil }, nil
} }
// targetDimensions calculates the output dimensions for a requested size, // targetDimensions calculates the output dimensions for a requested size,
// scaling proportionally when only one dimension is given and keeping the // scaling proportionally when only one dimension is given and keeping the
// original dimensions when both are zero. // original dimensions when both are zero. A side scaled proportionally is
func targetDimensions(size Size, origWidth, origHeight int) (int, int) { // rounded down but kept at 1 or more, as for a thin image it would round
// down to 0, a side of no pixels. It can come out above MaxOutputDimension,
// which is ErrOutputTooLarge.
func targetDimensions(
size imagerequest.Size, origWidth, origHeight int,
) (int, int, error) {
width, height := size.Width, size.Height
switch { switch {
case size.Width == 0 && size.Height == 0: case width == 0 && height == 0:
// Both are 0: keep original size // Both are 0: keep original size
return origWidth, origHeight return origWidth, origHeight, nil
case size.Width == 0: case width == 0:
// Only height specified: calculate width proportionally // Only height specified: calculate width proportionally
return origWidth * size.Height / origHeight, size.Height width = max(1, origWidth*height/origHeight)
case size.Height == 0: case height == 0:
// Only width specified: calculate height proportionally // Only width specified: calculate height proportionally
return size.Width, origHeight * size.Width / origWidth height = max(1, origHeight*width/origWidth)
default:
return size.Width, size.Height
} }
}
// MIME types for the supported image formats. if width > MaxOutputDimension || height > MaxOutputDimension {
const ( return 0, 0, fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
mimeJPEG = "image/jpeg" width, height, MaxOutputDimension)
mimePNG = "image/png"
mimeGIF = "image/gif"
mimeWebP = "image/webp"
mimeAVIF = "image/avif"
mimeJXL = "image/jxl"
)
// SupportedInputFormats returns MIME types this processor can read.
func (p *ImageProcessor) SupportedInputFormats() []string {
return []string{
mimeJPEG,
mimePNG,
mimeGIF,
mimeWebP,
mimeAVIF,
mimeJXL,
} }
}
// SupportedOutputFormats returns formats this processor can write. return width, height, nil
func (p *ImageProcessor) SupportedOutputFormats() []Format {
return []Format{
FormatJPEG,
FormatPNG,
FormatGIF,
FormatWebP,
FormatAVIF,
FormatJXL,
}
}
// FormatToMIME converts a Format to its MIME type string.
func FormatToMIME(format Format) string {
switch format {
case FormatJPEG:
return mimeJPEG
case FormatPNG:
return mimePNG
case FormatWebP:
return mimeWebP
case FormatGIF:
return mimeGIF
case FormatAVIF:
return mimeAVIF
case FormatJXL:
return mimeJXL
case FormatOriginal:
return "application/octet-stream"
default:
return "application/octet-stream"
}
} }
// WaitForProcessing waits until no image is being processed, or until ctx // WaitForProcessing waits until no image is being processed, or until ctx
@@ -396,112 +317,230 @@ func (p *ImageProcessor) WaitForProcessing(ctx context.Context) int {
// acquireSlot takes a slot in processingSemaphore, waiting at most // acquireSlot takes a slot in processingSemaphore, waiting at most
// processingWaitTimeout for one to free up, and returns the func that gives // processingWaitTimeout for one to free up, and returns the func that gives
// it back. A free slot is taken even when ctx has ended; only the wait for // it back. When ctx has ended by the time it has a slot, it gives the slot
// one stops when ctx ends, as the rest of Process does not check ctx. // back and returns ctx's error.
func (p *ImageProcessor) acquireSlot(ctx context.Context) (func(), error) { func (p *ImageProcessor) acquireSlot(ctx context.Context) (func(), error) {
release := func() { <-p.processingSemaphore }
select { select {
case p.processingSemaphore <- struct{}{}: case p.processingSemaphore <- struct{}{}:
return release, nil
default:
}
select {
case p.processingSemaphore <- struct{}{}:
return release, nil
case <-time.After(p.processingWaitTimeout): case <-time.After(p.processingWaitTimeout):
return nil, ErrTooManyImages return nil, ErrTooManyImages
case <-ctx.Done(): case <-ctx.Done():
return nil, ctx.Err() return nil, ctx.Err()
} }
release := func() { <-p.processingSemaphore }
// select takes the slot even when ctx has ended at the same moment
err := ctx.Err()
if err != nil {
release()
return nil, err
}
return release, nil
} }
// detectFormat returns the format string for a govips image type. // readInput reads input with a size limit to prevent unbounded memory
func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string { // consumption, refusing more than maxInputBytes with ErrInputDataTooLarge. It
switch imageType { // reads at most maxInputBytes+1 so it can detect if the input exceeds the
case vips.ImageTypeJPEG: // limit without consuming additional memory.
return "jpeg" func (p *ImageProcessor) readInput(input io.Reader) ([]byte, error) {
case vips.ImageTypePNG: data, err := io.ReadAll(io.LimitReader(input, p.maxInputBytes+1))
return "png" if err != nil {
case vips.ImageTypeGIF: return nil, fmt.Errorf("failed to read input: %w", err)
return "gif" }
case vips.ImageTypeWEBP:
return "webp" if int64(len(data)) > p.maxInputBytes {
case vips.ImageTypeAVIF, vips.ImageTypeHEIF: return nil, ErrInputDataTooLarge
return string(FormatAVIF) }
case vips.ImageTypeJXL:
return string(FormatJXL) return data, nil
case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF, }
vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP,
vips.ImageTypeJP2K: // checkInputFormat returns ErrUnsupportedInputFormat unless libvips would
return "unknown" // load data with the loader of sourceFormat. govips picks the libvips loader
// from the input's first bytes with DetermineImageType. It refuses an SVG
// whose first bytes pass the first-bytes check for AVIF, which librsvg would
// otherwise render.
func checkInputFormat(data []byte, sourceFormat imagerequest.Format) error {
// A format with no loader, such as orig or auto, has the type govips also
// gives input it cannot read, which the check below would let through.
sourceType := vipsImageType(sourceFormat)
if sourceType == vips.ImageTypeUnknown {
return fmt.Errorf("%w: %q is not a source format",
ErrUnsupportedInputFormat, sourceFormat)
}
if vips.DetermineImageType(data) != sourceType {
return fmt.Errorf("%w: libvips does not read the input as %s",
ErrUnsupportedInputFormat, sourceFormat)
}
return nil
}
// vipsImageType returns the govips image type whose libvips loader reads
// format. orig and auto stand for another format, so they have
// vips.ImageTypeUnknown, which govips has no loader for.
func vipsImageType(format imagerequest.Format) vips.ImageType {
switch format {
case imagerequest.FormatJPEG:
return vips.ImageTypeJPEG
case imagerequest.FormatPNG:
return vips.ImageTypePNG
case imagerequest.FormatGIF:
return vips.ImageTypeGIF
case imagerequest.FormatWebP:
return vips.ImageTypeWEBP
case imagerequest.FormatAVIF:
return vips.ImageTypeAVIF
case imagerequest.FormatJXL:
return vips.ImageTypeJXL
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
return vips.ImageTypeUnknown
default: default:
return "unknown" return vips.ImageTypeUnknown
} }
} }
// resize resizes the image according to the fit mode. // outputDimensions returns the dimensions resize gives an image of imgWidth x
// imgHeight for the target dimensions and fit mode, without resizing it, and
// refuses an output too large for format. With FitOutside one side can come
// out larger than the target; when it would be above MaxOutputDimension,
// outputDimensions returns ErrOutputTooLarge. An AVIF output of more than
// MaxAVIFOutputPixels is ErrAVIFOutputTooLarge.
func outputDimensions(
imgWidth, imgHeight, width, height int,
fit imagerequest.Fit, format imagerequest.Format,
) (int, int, error) {
scaleW := float64(width) / float64(imgWidth)
scaleH := float64(height) / float64(imgHeight)
outputWidth, outputHeight := width, height
switch fit {
case imagerequest.FitCover, imagerequest.FitFill, "":
// Cover crops, and fill stretches, to exactly the target dimensions
case imagerequest.FitContain:
// Fit within the target dimensions, maintaining aspect ratio. Scaled
// by the smaller scale, the short side of a thin image can round
// down to 0, which libvips refuses as a width, so each side is at
// least 1.
scale := min(scaleW, scaleH)
outputWidth = max(1, int(float64(imgWidth)*scale))
outputHeight = max(1, int(float64(imgHeight)*scale))
case imagerequest.FitInside:
// Same as contain, but only shrink
scale := min(scaleW, scaleH, 1)
outputWidth = max(1, int(float64(imgWidth)*scale))
outputHeight = max(1, int(float64(imgHeight)*scale))
case imagerequest.FitOutside:
// The smallest dimension fits, the other may exceed the target. A
// side scaled to 1 can come out just under 1 in floating point, as
// 49 times 1/49 does, and round down to 0, which libvips refuses as
// a width, so each side is at least 1.
scale := max(scaleW, scaleH)
outputWidth = max(1, int(float64(imgWidth)*scale))
outputHeight = max(1, int(float64(imgHeight)*scale))
default:
return 0, 0, fmt.Errorf("%w: %s", imagerequest.ErrInvalidFitMode, fit)
}
// The side FitOutside scales past the target is held to the maximum too
if max(outputWidth, outputHeight) > MaxOutputDimension {
return 0, 0, fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
outputWidth, outputHeight, MaxOutputDimension)
}
if format == imagerequest.FormatAVIF &&
outputWidth*outputHeight > MaxAVIFOutputPixels {
return 0, 0, fmt.Errorf("%w: %dx%d is above %d pixels",
ErrAVIFOutputTooLarge, outputWidth, outputHeight, MaxAVIFOutputPixels)
}
return outputWidth, outputHeight, nil
}
// resize resizes the image to width x height, the dimensions
// outputDimensions gives for the fit mode.
func (p *ImageProcessor) resize( func (p *ImageProcessor) resize(
img *vips.ImageRef, width, height int, fit FitMode, img *vips.ImageRef, width, height int, fit imagerequest.Fit,
) error { ) error {
switch fit { switch fit {
case FitCover, "": case imagerequest.FitCover, "":
// Resize and crop to fill exact dimensions (default) // Crop the centre and resize to fill exact dimensions (default)
return img.Thumbnail(width, height, vips.InterestingCentre) return cropCentreAndResize(img, width, height)
case FitContain: case imagerequest.FitFill:
// Resize to fit within dimensions, maintaining aspect ratio
imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := min(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
return img.Thumbnail(newW, newH, vips.InterestingNone)
case FitFill:
// Resize to exact dimensions (may distort) // Resize to exact dimensions (may distort)
return img.ThumbnailWithSize(width, height, vips.InterestingNone, vips.SizeForce) return img.ThumbnailWithSize(width, height, vips.InterestingNone, vips.SizeForce)
case FitInside: case imagerequest.FitInside:
// Same as contain, but only shrink // outputDimensions keeps the dimensions of an image that already fits
if img.Width() <= width && img.Height() <= height { if img.Width() == width && img.Height() == height {
return nil // Already fits return nil
} }
imgW, imgH := img.Width(), img.Height() return img.Thumbnail(width, height, vips.InterestingNone)
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := min(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
return img.Thumbnail(newW, newH, vips.InterestingNone) case imagerequest.FitContain, imagerequest.FitOutside:
return img.Thumbnail(width, height, vips.InterestingNone)
case FitOutside:
// Resize so smallest dimension fits, may exceed target on other dimension
imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := max(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
return img.Thumbnail(newW, newH, vips.InterestingNone)
default: default:
return fmt.Errorf("%w: %s", ErrInvalidFitMode, fit) return fmt.Errorf("%w: %s", imagerequest.ErrInvalidFitMode, fit)
} }
} }
// cropCentreAndResize scales img by the same amount on both sides, the
// least that makes it cover width x height, and cuts the centre width x
// height out of it, as libvips' thumbnail does with InterestingCentre.
// thumbnail alone scales the whole image and copies the result into memory
// before it cuts, and for a thin image that is far larger than the image or
// the output: 8192x64 at 8192x8192 is 1048576x8192, about 24 GiB. So the
// centre is cut from img first, widened to whole pixels of img, which adds
// less than one pixel of img on each side, and thumbnail scales and cuts
// only that.
func cropCentreAndResize(img *vips.ImageRef, width, height int) error {
imgW, imgH := img.Width(), img.Height()
// An image wider than the requested proportions keeps its height and
// loses width down to imgH*width/height, any other keeps its width and
// loses height down to imgW*height/width. Half of what is lost comes
// off each side, rounded down to whole pixels, so the part kept holds
// all of the centre.
left, top := 0, 0
if imgW*height > imgH*width {
left = (imgW*height - imgH*width) / (2 * height) //nolint:mnd // each side
} else {
top = (imgH*width - imgW*height) / (2 * width) //nolint:mnd // each side
}
err := img.ExtractArea(left, top, imgW-2*left, imgH-2*top)
if err != nil {
return err
}
return img.Thumbnail(width, height, vips.InterestingCentre)
}
const defaultQuality = 85 const defaultQuality = 85
// encode encodes an image to the specified format. // encode encodes an image to the specified format. govips gives pixa no way
// to stop libvips once an encode has begun, so when ctx has ended encode
// returns ctx's error without beginning one.
func (p *ImageProcessor) encode( func (p *ImageProcessor) encode(
img *vips.ImageRef, format Format, quality int, ctx context.Context, img *vips.ImageRef, format imagerequest.Format,
quality int,
) ([]byte, error) { ) ([]byte, error) {
err := ctx.Err()
if err != nil {
return nil, err
}
if quality <= 0 { if quality <= 0 {
quality = defaultQuality quality = defaultQuality
} }
@@ -511,32 +550,32 @@ func (p *ImageProcessor) encode(
// libvips' built-in profile; govips' own sRGB path variable is set on // libvips' built-in profile; govips' own sRGB path variable is set on
// first use but read without a lock, so concurrent requests race on it. // first use but read without a lock, so concurrent requests race on it.
if img.HasICCProfile() { if img.HasICCProfile() {
err := img.TransformICCProfileWithFallback("srgb", "srgb") err = img.TransformICCProfileWithFallback("srgb", "srgb")
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to convert to sRGB: %w", err) return nil, fmt.Errorf("failed to convert to sRGB: %w", err)
} }
} }
switch format { switch format {
case FormatJPEG: case imagerequest.FormatJPEG:
return exportJPEG(img, quality) return exportJPEG(img, quality)
case FormatPNG: case imagerequest.FormatPNG:
return exportPNG(img) return exportPNG(img)
case FormatGIF: case imagerequest.FormatGIF:
return exportGIF(img) return exportGIF(img)
case FormatWebP: case imagerequest.FormatWebP:
return exportWebP(img, quality) return exportWebP(img, quality)
case FormatAVIF: case imagerequest.FormatAVIF:
return exportAVIF(img, quality) return exportAVIF(img, quality)
case FormatJXL: case imagerequest.FormatJXL:
return exportJXL(img, quality) return exportJXL(img, quality)
case FormatOriginal: case imagerequest.FormatOriginal, imagerequest.FormatAuto:
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format) return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
default: default:
@@ -586,11 +625,16 @@ func exportGIF(img *vips.ImageRef) ([]byte, error) {
return output, err return output, err
} }
// webpEffort is libvips' default WebP effort, from 0 (fastest) to 6. // webpEffort is the WebP effort, from 0 (fastest) to 6. It is below libvips'
const webpEffort = 4 // default of 4 so that an encode takes at most 40 seconds of the default
// downstream_timeout of 60 seconds. With one thread, as pixad runs libvips,
// saving an 8192x8192 image of random pixels with an alpha channel takes up
// to about 23 seconds at 2, and up to about 16 without one, at any quality;
// with an alpha channel, 3 takes about 50 seconds and 4 about 61.
const webpEffort = 2
// exportWebP encodes img as lossy WebP at quality and libvips' default // exportWebP encodes img as lossy WebP at quality and webpEffort, without
// effort, without metadata. // metadata.
func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) { func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
output, _, err := img.ExportWebp(&vips.WebpExportParams{ output, _, err := img.ExportWebp(&vips.WebpExportParams{
StripMetadata: true, StripMetadata: true,
@@ -602,11 +646,13 @@ func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
} }
// avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest // avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest
// govips can set. With one thread, as pixad runs libvips, 1 takes about 51 // govips can set. With one thread, as pixad runs libvips, 1 takes about 52
// seconds to save an 8192x8192 image of random pixels, the worst case, // seconds to save an 8192x8192 RGB image of random pixels at quality 85,
// against the default downstream_timeout of 60 seconds. On an image of // and about 96 with an alpha channel, against the default
// milder noise, which 1 saves in about 12 seconds, 2 takes nearly a minute // downstream_timeout of 60 seconds, so an AVIF output is held to
// and libvips' default, 4, takes minutes. // MaxAVIFOutputPixels. On an image of milder noise, which 1 saves in about
// 12 seconds, 2 takes nearly a minute and libvips' default, 4, takes
// minutes.
const avifEffort = 1 const avifEffort = 1
// avifBitdepth is the AVIF bit depth, 8 bits per sample for every image. // avifBitdepth is the AVIF bit depth, 8 bits per sample for every image.
@@ -633,10 +679,18 @@ func exportAVIF(img *vips.ImageRef, quality int) ([]byte, error) {
// JPEG that names none. // JPEG that names none.
const jxlResolution = 72 / 25.4 const jxlResolution = 72 / 25.4
// exportJXL encodes img as JPEG XL at quality, with libvips' default effort // jxlEffort is the JPEG XL effort, from 1 (fastest) to 9. It is below
// and without metadata. govips sends libvips a distance, the JPEG XL // libvips' default of 7 so that an encode takes at most 40 seconds of the
// encoder's own measure of quality, along with the quality, and libvips then // default downstream_timeout of 60 seconds. With one thread, as pixad runs
// uses the distance alone, so the quality is also given as a distance. // libvips, saving an 8192x8192 image of random pixels with an alpha channel
// takes up to about 27 seconds at 4, and up to about 13 without one, at any
// quality; with an alpha channel, 5 takes about 43 seconds and 7 about 81.
const jxlEffort = 4
// exportJXL encodes img as JPEG XL at quality and jxlEffort, without
// metadata. govips sends libvips a distance, the JPEG XL encoder's own
// measure of quality, along with the quality, and libvips then uses the
// distance alone, so the quality is also given as a distance.
func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) { func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
// libvips converts a CMYK image to sRGB before it saves WebP, AVIF or // libvips converts a CMYK image to sRGB before it saves WebP, AVIF or
// PNG, but cannot save one as JPEG XL. encode has already converted // PNG, but cannot save one as JPEG XL. encode has already converted
@@ -675,6 +729,7 @@ func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
params := vips.NewJxlExportParams() params := vips.NewJxlExportParams()
params.Quality = quality params.Quality = quality
params.Distance = jxlDistance(quality) params.Distance = jxlDistance(quality)
params.Effort = jxlEffort
output, _, err := toSave.ExportJxl(params) output, _, err := toSave.ExportJxl(params)
if err != nil { if err != nil {
@@ -699,24 +754,3 @@ func jxlDistance(quality int) float64 {
return 53.0/3000.0*q*q - 23.0/20.0*q + 25.0 return 53.0/3000.0*q*q - 23.0/20.0*q + 25.0
} }
// formatFromString converts a format string from detectFormat to Format. A
// format detectFormat could not name is ErrUnsupportedInputFormat.
func (p *ImageProcessor) formatFromString(format string) (Format, error) {
switch format {
case "jpeg":
return FormatJPEG, nil
case "png":
return FormatPNG, nil
case "gif":
return FormatGIF, nil
case "webp":
return FormatWebP, nil
case string(FormatAVIF):
return FormatAVIF, nil
case string(FormatJXL):
return FormatJXL, nil
default:
return "", fmt.Errorf("%w: %s", ErrUnsupportedInputFormat, format)
}
}
internal/imageprocessor/imageprocessor_internal_test.go
+117 -126
View File
@@ -15,6 +15,7 @@ import (
"testing" "testing"
"github.com/davidbyttow/govips/v2/vips" "github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
@@ -87,29 +88,29 @@ func isAVIF(data []byte) bool {
brand := string(data[8:12]) brand := string(data[8:12])
return brand == string(FormatAVIF) || brand == "avis" return brand == string(imagerequest.FormatAVIF) || brand == "avis"
} }
// detectMIME is a minimal magic-byte detector for test assertions. // detectMIME is a minimal magic-byte detector for test assertions.
func detectMIME(data []byte) string { func detectMIME(data []byte) string {
if len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF { if len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF {
return mimeJPEG return imagerequest.FormatJPEG.MIMEType()
} }
if len(data) >= 8 && string(data[:8]) == "\x89PNG\r\n\x1a\n" { if len(data) >= 8 && string(data[:8]) == "\x89PNG\r\n\x1a\n" {
return mimePNG return imagerequest.FormatPNG.MIMEType()
} }
if len(data) >= 4 && string(data[:4]) == "GIF8" { if len(data) >= 4 && string(data[:4]) == "GIF8" {
return mimeGIF return imagerequest.FormatGIF.MIMEType()
} }
if len(data) >= 12 && string(data[:4]) == "RIFF" && string(data[8:12]) == "WEBP" { if len(data) >= 12 && string(data[:4]) == "RIFF" && string(data[8:12]) == "WEBP" {
return mimeWebP return imagerequest.FormatWebP.MIMEType()
} }
if isAVIF(data) { if isAVIF(data) {
return mimeAVIF return imagerequest.FormatAVIF.MIMEType()
} }
return "" return ""
@@ -118,19 +119,20 @@ func detectMIME(data []byte) string {
func TestImageProcessor_ResizeJPEG(t *testing.T) { func TestImageProcessor_ResizeJPEG(t *testing.T) {
t.Parallel() t.Parallel()
proc := New(Params{}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
ctx := context.Background() ctx := context.Background()
input := createTestJPEG(t, 800, 600) input := createTestJPEG(t, 800, 600)
req := &Request{ transform := &imagerequest.Transform{
Size: Size{Width: 400, Height: 300}, Size: imagerequest.Size{Width: 400, Height: 300},
Format: FormatJPEG, Format: imagerequest.FormatJPEG,
Quality: 85, Quality: 85,
FitMode: FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), req) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -156,7 +158,7 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
} }
mime := detectMIME(data) mime := detectMIME(data)
if mime != mimeJPEG { if mime != imagerequest.FormatJPEG.MIMEType() {
t.Errorf("Output format = %v, want image/jpeg", mime) t.Errorf("Output format = %v, want image/jpeg", mime)
} }
} }
@@ -164,18 +166,19 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
func TestImageProcessor_ConvertToPNG(t *testing.T) { func TestImageProcessor_ConvertToPNG(t *testing.T) {
t.Parallel() t.Parallel()
proc := New(Params{}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
ctx := context.Background() ctx := context.Background()
input := createTestJPEG(t, 200, 150) input := createTestJPEG(t, 200, 150)
req := &Request{ transform := &imagerequest.Transform{
Size: Size{Width: 200, Height: 150}, Size: imagerequest.Size{Width: 200, Height: 150},
Format: FormatPNG, Format: imagerequest.FormatPNG,
FitMode: FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), req) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -188,7 +191,7 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
} }
mime := detectMIME(data) mime := detectMIME(data)
if mime != mimePNG { if mime != imagerequest.FormatPNG.MIMEType() {
t.Errorf("Output format = %v, want image/png", mime) t.Errorf("Output format = %v, want image/png", mime)
} }
} }
@@ -196,23 +199,24 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
// processAndCheckSize processes a test JPEG of the given input dimensions // processAndCheckSize processes a test JPEG of the given input dimensions
// with the requested size and asserts the resulting dimensions. // with the requested size and asserts the resulting dimensions.
func processAndCheckSize( func processAndCheckSize(
t *testing.T, inputW, inputH int, size Size, wantW, wantH int, t *testing.T, inputW, inputH int, size imagerequest.Size, wantW, wantH int,
) { ) {
t.Helper() t.Helper()
proc := New(Params{}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
ctx := context.Background() ctx := context.Background()
input := createTestJPEG(t, inputW, inputH) input := createTestJPEG(t, inputW, inputH)
req := &Request{ transform := &imagerequest.Transform{
Size: size, Size: size,
Format: FormatJPEG, Format: imagerequest.FormatJPEG,
Quality: 85, Quality: 85,
FitMode: FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), req) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -232,27 +236,28 @@ func TestImageProcessor_OriginalSize(t *testing.T) {
t.Parallel() t.Parallel()
// Width and height 0: keep original size // Width and height 0: keep original size
processAndCheckSize(t, 640, 480, Size{Width: 0, Height: 0}, 640, 480) processAndCheckSize(t, 640, 480, imagerequest.Size{Width: 0, Height: 0}, 640, 480)
} }
func TestImageProcessor_FitContain(t *testing.T) { func TestImageProcessor_FitContain(t *testing.T) {
t.Parallel() t.Parallel()
proc := New(Params{}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
ctx := context.Background() ctx := context.Background()
// 800x400 image (2:1 aspect) into 400x400 box with contain // 800x400 image (2:1 aspect) into 400x400 box with contain
// Should result in 400x200 (maintaining aspect ratio) // Should result in 400x200 (maintaining aspect ratio)
input := createTestJPEG(t, 800, 400) input := createTestJPEG(t, 800, 400)
req := &Request{ transform := &imagerequest.Transform{
Size: Size{Width: 400, Height: 400}, Size: imagerequest.Size{Width: 400, Height: 400},
Format: FormatJPEG, Format: imagerequest.FormatJPEG,
Quality: 85, Quality: 85,
FitMode: FitContain, FitMode: imagerequest.FitContain,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), req) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -270,7 +275,7 @@ func TestImageProcessor_ProportionalScale_WidthOnly(t *testing.T) {
// 800x600 image, request width=400 height=0 // 800x600 image, request width=400 height=0
// Should scale proportionally to 400x300 // Should scale proportionally to 400x300
processAndCheckSize(t, 800, 600, Size{Width: 400, Height: 0}, 400, 300) processAndCheckSize(t, 800, 600, imagerequest.Size{Width: 400, Height: 0}, 400, 300)
} }
func TestImageProcessor_ProportionalScale_HeightOnly(t *testing.T) { func TestImageProcessor_ProportionalScale_HeightOnly(t *testing.T) {
@@ -278,24 +283,25 @@ func TestImageProcessor_ProportionalScale_HeightOnly(t *testing.T) {
// 800x600 image, request width=0 height=300 // 800x600 image, request width=0 height=300
// Should scale proportionally to 400x300 // Should scale proportionally to 400x300
processAndCheckSize(t, 800, 600, Size{Width: 0, Height: 300}, 400, 300) processAndCheckSize(t, 800, 600, imagerequest.Size{Width: 0, Height: 300}, 400, 300)
} }
func TestImageProcessor_ProcessPNG(t *testing.T) { func TestImageProcessor_ProcessPNG(t *testing.T) {
t.Parallel() t.Parallel()
proc := New(Params{}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
ctx := context.Background() ctx := context.Background()
input := createTestPNG(t, 400, 300) input := createTestPNG(t, 400, 300)
req := &Request{ transform := &imagerequest.Transform{
Size: Size{Width: 200, Height: 150}, Size: imagerequest.Size{Width: 200, Height: 150},
Format: FormatPNG, Format: imagerequest.FormatPNG,
FitMode: FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), req) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatPNG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -311,22 +317,6 @@ func TestImageProcessor_ProcessPNG(t *testing.T) {
} }
} }
func TestImageProcessor_SupportedFormats(t *testing.T) {
t.Parallel()
proc := New(Params{})
inputFormats := proc.SupportedInputFormats()
if len(inputFormats) == 0 {
t.Error("SupportedInputFormats() returned empty slice")
}
outputFormats := proc.SupportedOutputFormats()
if len(outputFormats) == 0 {
t.Error("SupportedOutputFormats() returned empty slice")
}
}
func TestImageProcessor_RejectsOversizedInput(t *testing.T) { func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
t.Parallel() t.Parallel()
@@ -345,18 +335,19 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
proc := New(Params{}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
ctx := context.Background() ctx := context.Background()
input := createTestJPEG(t, tt.width, tt.height) input := createTestJPEG(t, tt.width, tt.height)
req := &Request{ transform := &imagerequest.Transform{
Size: Size{Width: 100, Height: 100}, Size: imagerequest.Size{Width: 100, Height: 100},
Format: FormatJPEG, Format: imagerequest.FormatJPEG,
Quality: 85, Quality: 85,
FitMode: FitCover, FitMode: imagerequest.FitCover,
} }
_, err := proc.Process(ctx, bytes.NewReader(input), req) _, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err == nil { if err == nil {
t.Error("Process() should reject oversized input images") t.Error("Process() should reject oversized input images")
} }
@@ -371,20 +362,21 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) { func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
t.Parallel() t.Parallel()
proc := New(Params{}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
ctx := context.Background() ctx := context.Background()
// Create an image at exactly MaxInputDimension - should be accepted // Create an image at exactly MaxInputDimension - should be accepted
input := createTestJPEG(t, MaxInputDimension, 100) input := createTestJPEG(t, MaxInputDimension, 100)
req := &Request{ transform := &imagerequest.Transform{
Size: Size{Width: 100, Height: 100}, Size: imagerequest.Size{Width: 100, Height: 100},
Format: FormatJPEG, Format: imagerequest.FormatJPEG,
Quality: 85, Quality: 85,
FitMode: FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), req) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf( t.Fatalf(
"Process() should accept images at MaxInputDimension, got error: %v", "Process() should accept images at MaxInputDimension, got error: %v",
@@ -397,22 +389,25 @@ func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
// encodeAndCheck processes a 200x150 test JPEG into a 100x75 output of the // encodeAndCheck processes a 200x150 test JPEG into a 100x75 output of the
// given format and asserts the output MIME type and dimensions. // given format and asserts the output MIME type and dimensions.
func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) { func encodeAndCheck(
t *testing.T, format imagerequest.Format, quality int, wantMIME string,
) {
t.Helper() t.Helper()
proc := New(Params{}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
ctx := context.Background() ctx := context.Background()
input := createTestJPEG(t, 200, 150) input := createTestJPEG(t, 200, 150)
req := &Request{ transform := &imagerequest.Transform{
Size: Size{Width: 100, Height: 75}, Size: imagerequest.Size{Width: 100, Height: 75},
Format: format, Format: format,
Quality: quality, Quality: quality,
FitMode: FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), req) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v, want nil", err) t.Fatalf("Process() error = %v, want nil", err)
} }
@@ -443,13 +438,13 @@ func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) {
func TestImageProcessor_EncodeWebP(t *testing.T) { func TestImageProcessor_EncodeWebP(t *testing.T) {
t.Parallel() t.Parallel()
encodeAndCheck(t, FormatWebP, 80, mimeWebP) encodeAndCheck(t, imagerequest.FormatWebP, 80, imagerequest.FormatWebP.MIMEType())
} }
func TestImageProcessor_DecodeAVIF(t *testing.T) { func TestImageProcessor_DecodeAVIF(t *testing.T) {
t.Parallel() t.Parallel()
proc := New(Params{}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
ctx := context.Background() ctx := context.Background()
// Load test AVIF file // Load test AVIF file
@@ -459,14 +454,15 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
} }
// Request resize and convert to JPEG // Request resize and convert to JPEG
req := &Request{ transform := &imagerequest.Transform{
Size: Size{Width: 2, Height: 2}, Size: imagerequest.Size{Width: 2, Height: 2},
Format: FormatJPEG, Format: imagerequest.FormatJPEG,
Quality: 85, Quality: 85,
FitMode: FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), req) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatAVIF, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v, want nil (AVIF decoding should work)", err) t.Fatalf("Process() error = %v, want nil (AVIF decoding should work)", err)
} }
@@ -480,7 +476,7 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
} }
mime := detectMIME(data) mime := detectMIME(data)
if mime != mimeJPEG { if mime != imagerequest.FormatJPEG.MIMEType() {
t.Errorf("Output format = %v, want image/jpeg", mime) t.Errorf("Output format = %v, want image/jpeg", mime)
} }
} }
@@ -491,7 +487,7 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
// Create a processor with a very small byte limit // Create a processor with a very small byte limit
const limit = 1024 const limit = 1024
proc := New(Params{MaxInputBytes: limit}) proc := New(Params{MaxInputBytes: limit, MaxConcurrentProcessing: 1})
ctx := context.Background() ctx := context.Background()
// Create a valid JPEG that exceeds the byte limit // Create a valid JPEG that exceeds the byte limit
@@ -500,14 +496,15 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
t.Fatalf("test JPEG must exceed %d bytes, got %d", limit, len(input)) t.Fatalf("test JPEG must exceed %d bytes, got %d", limit, len(input))
} }
req := &Request{ transform := &imagerequest.Transform{
Size: Size{Width: 100, Height: 75}, Size: imagerequest.Size{Width: 100, Height: 75},
Format: FormatJPEG, Format: imagerequest.FormatJPEG,
Quality: 85, Quality: 85,
FitMode: FitCover, FitMode: imagerequest.FitCover,
} }
_, err := proc.Process(ctx, bytes.NewReader(input), req) _, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err == nil { if err == nil {
t.Fatal("Process() should reject input exceeding maxInputBytes") t.Fatal("Process() should reject input exceeding maxInputBytes")
} }
@@ -524,17 +521,18 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
input := createTestJPEG(t, 10, 10) input := createTestJPEG(t, 10, 10)
limit := int64(len(input)) * 10 // 10× headroom limit := int64(len(input)) * 10 // 10× headroom
proc := New(Params{MaxInputBytes: limit}) proc := New(Params{MaxInputBytes: limit, MaxConcurrentProcessing: 1})
ctx := context.Background() ctx := context.Background()
req := &Request{ transform := &imagerequest.Transform{
Size: Size{Width: 10, Height: 10}, Size: imagerequest.Size{Width: 10, Height: 10},
Format: FormatJPEG, Format: imagerequest.FormatJPEG,
Quality: 85, Quality: 85,
FitMode: FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), req) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v, want nil", err) t.Fatalf("Process() error = %v, want nil", err)
} }
@@ -542,35 +540,25 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
defer func() { _ = result.Content.Close() }() defer func() { _ = result.Content.Close() }()
} }
func TestImageProcessor_DefaultMaxInputBytes(t *testing.T) {
t.Parallel()
// Passing 0 should use the default
proc := New(Params{})
if proc.maxInputBytes != DefaultMaxInputBytes {
t.Errorf("maxInputBytes = %d, want %d", proc.maxInputBytes, DefaultMaxInputBytes)
}
// Passing negative should also use the default
proc = New(Params{MaxInputBytes: -1})
if proc.maxInputBytes != DefaultMaxInputBytes {
t.Errorf("maxInputBytes = %d, want %d", proc.maxInputBytes, DefaultMaxInputBytes)
}
}
func TestImageProcessor_EncodeAVIF(t *testing.T) { func TestImageProcessor_EncodeAVIF(t *testing.T) {
t.Parallel() t.Parallel()
encodeAndCheck(t, FormatAVIF, 85, mimeAVIF) encodeAndCheck(t, imagerequest.FormatAVIF, 85, imagerequest.FormatAVIF.MIMEType())
} }
// processAndDecode runs input through Process and decodes the output with // processAndDecode runs input, a JPEG, through Process and decodes the output
// vips, so a test can inspect the image a client would receive. // with vips, so a test can inspect the image a client would receive.
func processAndDecode(t *testing.T, input []byte, req *Request) *vips.ImageRef { func processAndDecode(
t *testing.T, input []byte, transform *imagerequest.Transform,
) *vips.ImageRef {
t.Helper() t.Helper()
result, err := New(Params{}).Process( result, err := New(Params{
context.Background(), bytes.NewReader(input), req, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(
context.Background(), bytes.NewReader(input), imagerequest.FormatJPEG,
transform,
) )
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
@@ -614,15 +602,17 @@ func TestImageProcessor_StripsEXIF(t *testing.T) {
t.Fatal("testdata/gps-exif.jpg has no GPS latitude") t.Fatal("testdata/gps-exif.jpg has no GPS latitude")
} }
formats := []Format{ formats := []imagerequest.Format{
FormatJPEG, FormatPNG, FormatWebP, FormatAVIF, FormatGIF, FormatOriginal, imagerequest.FormatJPEG, imagerequest.FormatPNG, imagerequest.FormatWebP,
imagerequest.FormatAVIF, imagerequest.FormatGIF, imagerequest.FormatOriginal,
} }
for _, format := range formats { for _, format := range formats {
t.Run(string(format), func(t *testing.T) { t.Run(string(format), func(t *testing.T) {
t.Parallel() t.Parallel()
output := processAndDecode(t, input, &Request{Format: format}) output := processAndDecode(t, input,
&imagerequest.Transform{Format: format})
if output.HasExif() { if output.HasExif() {
t.Errorf("output has EXIF: %v", output.GetExif()) t.Errorf("output has EXIF: %v", output.GetExif())
@@ -644,21 +634,21 @@ func TestImageProcessor_AppliesEXIFOrientation(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
size Size size imagerequest.Size
wantW int wantW int
wantH int wantH int
}{ }{
{name: "original size", size: Size{}, wantW: 8, wantH: 16}, {name: "original size", size: imagerequest.Size{}, wantW: 8, wantH: 16},
{name: "width only", size: Size{Width: 4}, wantW: 4, wantH: 8}, {name: "width only", size: imagerequest.Size{Width: 4}, wantW: 4, wantH: 8},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
output := processAndDecode(t, input, &Request{ output := processAndDecode(t, input, &imagerequest.Transform{
Size: tt.size, Size: tt.size,
Format: FormatPNG, Format: imagerequest.FormatPNG,
}) })
if output.Width() != tt.wantW || output.Height() != tt.wantH { if output.Width() != tt.wantW || output.Height() != tt.wantH {
@@ -694,7 +684,8 @@ func TestImageProcessor_ConvertsWideGamutToSRGB(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err) t.Fatalf("failed to read test JPEG: %v", err)
} }
output := processAndDecode(t, input, &Request{Format: FormatPNG}) output := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatPNG})
if output.HasICCProfile() { if output.HasICCProfile() {
t.Error("output has an ICC profile") t.Error("output has an ICC profile")
internal/imageprocessor/jpegxl_output_internal_test.go
+27 -10
View File
@@ -8,6 +8,7 @@ import (
"testing" "testing"
"github.com/davidbyttow/govips/v2/vips" "github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestImageProcessor_EncodeJPEGXL converts a JPEG to a smaller JPEG XL and // TestImageProcessor_EncodeJPEGXL converts a JPEG to a smaller JPEG XL and
@@ -15,10 +16,17 @@ import (
func TestImageProcessor_EncodeJPEGXL(t *testing.T) { func TestImageProcessor_EncodeJPEGXL(t *testing.T) {
t.Parallel() t.Parallel()
req := &Request{Size: Size{Width: 100, Height: 75}, Format: FormatJXL} transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 100, Height: 75},
Format: imagerequest.FormatJXL,
}
result, err := New(Params{}).Process( result, err := New(Params{
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)), req, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)),
imagerequest.FormatJPEG, transform,
) )
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
@@ -60,8 +68,12 @@ func TestImageProcessor_JPEGXLQuality(t *testing.T) {
outputBytes := make(map[int]int64) outputBytes := make(map[int]int64)
for _, quality := range []int{30, 90} { for _, quality := range []int{30, 90} {
result, err := New(Params{}).Process(t.Context(), result, err := New(Params{
bytes.NewReader(input), &Request{Format: FormatJXL, Quality: quality}) MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: quality})
if err != nil { if err != nil {
t.Fatalf("Process() at quality %d error = %v", quality, err) t.Fatalf("Process() at quality %d error = %v", quality, err)
} }
@@ -89,7 +101,8 @@ func TestImageProcessor_JPEGXLDropsSourceEXIF(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err) t.Fatalf("failed to read test JPEG: %v", err)
} }
exif := processAndDecode(t, input, &Request{Format: FormatJXL}).GetExif() exif := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL}).GetExif()
for _, field := range []string{ for _, field := range []string{
"exif-ifd0-Make", "exif-ifd0-Model", "exif-ifd2-BodySerialNumber", "exif-ifd0-Make", "exif-ifd0-Model", "exif-ifd2-BodySerialNumber",
@@ -115,7 +128,8 @@ func TestImageProcessor_JPEGXLDropsSourceResolution(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err) t.Fatalf("failed to read test JPEG: %v", err)
} }
output := processAndDecode(t, input, &Request{Format: FormatJXL}) output := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL})
// libvips gives the resolution in pixels per millimetre. // libvips gives the resolution in pixels per millimetre.
xDPI := math.Round(output.ResX() * 25.4) xDPI := math.Round(output.ResX() * 25.4)
@@ -140,7 +154,8 @@ func TestImageProcessor_JPEGXLAppliesEXIFOrientation(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err) t.Fatalf("failed to read test JPEG: %v", err)
} }
output := processAndDecode(t, input, &Request{Format: FormatJXL}) output := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL})
if output.Width() != 8 || output.Height() != 16 { if output.Width() != 8 || output.Height() != 16 {
t.Fatalf("output is %dx%d, want 8x16", output.Width(), output.Height()) t.Fatalf("output is %dx%d, want 8x16", output.Width(), output.Height())
@@ -176,7 +191,8 @@ func TestImageProcessor_JPEGXLConvertsWideGamutToSRGB(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err) t.Fatalf("failed to read test JPEG: %v", err)
} }
output := processAndDecode(t, input, &Request{Format: FormatJXL}) output := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL})
pixel, err := output.GetPoint(4, 4) pixel, err := output.GetPoint(4, 4)
if err != nil { if err != nil {
@@ -203,7 +219,8 @@ func TestImageProcessor_JPEGXLFromCMYK(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err) t.Fatalf("failed to read test JPEG: %v", err)
} }
output := processAndDecode(t, input, &Request{Format: FormatJXL}) output := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL})
if output.Bands() != 3 { if output.Bands() != 3 {
t.Fatalf("output has %d bands, want 3", output.Bands()) t.Fatalf("output has %d bands, want 3", output.Bands())
internal/imageprocessor/max_concurrent_processing_internal_test.go
+38 -43
View File
@@ -5,12 +5,13 @@ import (
"context" "context"
"errors" "errors"
"io" "io"
"runtime"
"strings" "strings"
"sync" "sync"
"testing" "testing"
"testing/iotest" "testing/iotest"
"time" "time"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// errTestReadFailed is the error the unreadable test input returns. // errTestReadFailed is the error the unreadable test input returns.
@@ -76,13 +77,13 @@ func (r *gatedReader) Read(p []byte) (int, error) {
return n, err return n, err
} }
// smallJPEGRequest asks for a 5x5 JPEG. // smallJPEGTransform asks for a 5x5 JPEG.
func smallJPEGRequest() *Request { func smallJPEGTransform() *imagerequest.Transform {
return &Request{ return &imagerequest.Transform{
Size: Size{Width: 5, Height: 5}, Size: imagerequest.Size{Width: 5, Height: 5},
Format: FormatJPEG, Format: imagerequest.FormatJPEG,
Quality: 85, Quality: 85,
FitMode: FitCover, FitMode: imagerequest.FitCover,
} }
} }
@@ -92,7 +93,8 @@ func processInBackground(
proc *ImageProcessor, reader *gatedReader, results chan<- error, proc *ImageProcessor, reader *gatedReader, results chan<- error,
) { ) {
go func() { go func() {
result, err := proc.Process(context.Background(), reader, smallJPEGRequest()) result, err := proc.Process(context.Background(), reader,
imagerequest.FormatJPEG, smallJPEGTransform())
if err == nil { if err == nil {
_ = result.Content.Close() _ = result.Content.Close()
} }
@@ -115,18 +117,10 @@ func waitForEntries(t *testing.T, entered <-chan struct{}, count int) {
} }
} }
func TestNewDefaultsMaxConcurrentProcessingToCPUs(t *testing.T) { func TestNewMakesMaxConcurrentProcessingSlots(t *testing.T) {
t.Parallel() t.Parallel()
for _, limit := range []int{0, -1} { proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 3})
proc := New(Params{MaxConcurrentProcessing: limit})
if got := cap(proc.processingSemaphore); got != runtime.GOMAXPROCS(0) {
t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, one per CPU",
limit, got, runtime.GOMAXPROCS(0))
}
}
proc := New(Params{MaxConcurrentProcessing: 3})
if got := cap(proc.processingSemaphore); got != 3 { if got := cap(proc.processingSemaphore); got != 3 {
t.Errorf("MaxConcurrentProcessing 3: %d slots, want 3", got) t.Errorf("MaxConcurrentProcessing 3: %d slots, want 3", got)
} }
@@ -145,7 +139,7 @@ func TestProcessNeverExceedsMaxConcurrentProcessing(t *testing.T) {
calls = 6 calls = 6
) )
proc := New(Params{MaxConcurrentProcessing: limit}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: limit})
input := createTestJPEG(t, 50, 50) input := createTestJPEG(t, 50, 50)
counter := &readingCounter{} counter := &readingCounter{}
@@ -192,7 +186,7 @@ func TestProcessNeverExceedsMaxConcurrentProcessing(t *testing.T) {
func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) { func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) {
t.Parallel() t.Parallel()
proc := New(Params{MaxConcurrentProcessing: 1}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
proc.processingWaitTimeout = 100 * time.Millisecond proc.processingWaitTimeout = 100 * time.Millisecond
input := createTestJPEG(t, 10, 10) input := createTestJPEG(t, 10, 10)
@@ -213,7 +207,7 @@ func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) {
start := time.Now() start := time.Now()
_, err := proc.Process(context.Background(), bytes.NewReader(input), _, err := proc.Process(context.Background(), bytes.NewReader(input),
smallJPEGRequest()) imagerequest.FormatJPEG, smallJPEGTransform())
if !errors.Is(err, ErrTooManyImages) { if !errors.Is(err, ErrTooManyImages) {
t.Fatalf("Process() error = %v, want ErrTooManyImages", err) t.Fatalf("Process() error = %v, want ErrTooManyImages", err)
} }
@@ -239,38 +233,38 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
valid := createTestJPEG(t, 10, 10) valid := createTestJPEG(t, 10, 10)
unsupported := smallJPEGRequest() unsupported := smallJPEGTransform()
unsupported.Format = "bmp" unsupported.Format = "bmp"
cases := []struct { cases := []struct {
name string name string
input io.Reader input io.Reader
req *Request transform *imagerequest.Transform
// want is the error Process must return; nil means any error. // want is the error Process must return; nil means any error.
want error want error
}{ }{
{ {
name: "input cannot be read", name: "input cannot be read",
input: iotest.ErrReader(errTestReadFailed), input: iotest.ErrReader(errTestReadFailed),
req: smallJPEGRequest(), transform: smallJPEGTransform(),
want: errTestReadFailed, want: errTestReadFailed,
}, },
{ {
name: "input over the byte limit", name: "input over the byte limit",
input: bytes.NewReader(createTestJPEG(t, 800, 600)), input: bytes.NewReader(createTestJPEG(t, 800, 600)),
req: smallJPEGRequest(), transform: smallJPEGTransform(),
want: ErrInputDataTooLarge, want: ErrInputDataTooLarge,
}, },
{ {
name: "input not an image", name: "input not an image",
input: strings.NewReader("not an image"), input: strings.NewReader("not an image"),
req: smallJPEGRequest(), transform: smallJPEGTransform(),
}, },
{ {
name: "output format not supported", name: "output format not supported",
input: bytes.NewReader(valid), input: bytes.NewReader(valid),
req: unsupported, transform: unsupported,
want: ErrUnsupportedOutputFormat, want: ErrUnsupportedOutputFormat,
}, },
} }
@@ -281,7 +275,8 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
proc := New(Params{MaxInputBytes: 4096, MaxConcurrentProcessing: 1}) proc := New(Params{MaxInputBytes: 4096, MaxConcurrentProcessing: 1})
proc.processingWaitTimeout = 100 * time.Millisecond proc.processingWaitTimeout = 100 * time.Millisecond
_, err := proc.Process(context.Background(), tc.input, tc.req) _, err := proc.Process(context.Background(), tc.input,
imagerequest.FormatJPEG, tc.transform)
if err == nil || (tc.want != nil && !errors.Is(err, tc.want)) { if err == nil || (tc.want != nil && !errors.Is(err, tc.want)) {
t.Fatalf("Process() error = %v, want %v", err, tc.want) t.Fatalf("Process() error = %v, want %v", err, tc.want)
} }
@@ -291,7 +286,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
} }
result, err := proc.Process(context.Background(), bytes.NewReader(valid), result, err := proc.Process(context.Background(), bytes.NewReader(valid),
smallJPEGRequest()) imagerequest.FormatJPEG, smallJPEGTransform())
if err != nil { if err != nil {
t.Fatalf("Process() after the error = %v, want nil", err) t.Fatalf("Process() after the error = %v, want nil", err)
} }
@@ -308,7 +303,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
func TestWaitForProcessing(t *testing.T) { func TestWaitForProcessing(t *testing.T) {
t.Parallel() t.Parallel()
proc := New(Params{MaxConcurrentProcessing: 2}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 2})
gate := make(chan struct{}) gate := make(chan struct{})
entered := make(chan struct{}, 1) entered := make(chan struct{}, 1)
internal/imageprocessor/orig_format_internal_test.go
+10 -4
View File
@@ -6,11 +6,13 @@ import (
"testing" "testing"
"github.com/davidbyttow/govips/v2/vips" "github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestImageProcessor_OrigOfUnnamedFormatRefused verifies that orig of a TIFF // TestImageProcessor_OrigOfUnnamedFormatRefused verifies that orig of a TIFF
// input, which is not a source format, is refused with // input is refused with ErrUnsupportedInputFormat before it is decoded, and
// ErrUnsupportedInputFormat before it is decoded, and not saved as JPEG. // not saved as JPEG. TIFF is not a source format, and the test passes the
// input to Process as a JPEG.
func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) { func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
t.Parallel() t.Parallel()
@@ -26,8 +28,12 @@ func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
t.Fatalf("failed to save test TIFF: %v", err) t.Fatalf("failed to save test TIFF: %v", err)
} }
result, err := New(Params{}).Process( result, err := New(Params{
t.Context(), bytes.NewReader(tiff), &Request{Format: FormatOriginal}, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(
t.Context(), bytes.NewReader(tiff), imagerequest.FormatJPEG,
&imagerequest.Transform{Format: imagerequest.FormatOriginal},
) )
if result != nil { if result != nil {
t.Logf("Process() returned %s", result.ContentType) t.Logf("Process() returned %s", result.ContentType)
internal/imageprocessor/source_formats_internal_test.go
+37 -8
View File
@@ -2,8 +2,11 @@ package imageprocessor
import ( import (
"bytes" "bytes"
"errors"
"io" "io"
"testing" "testing"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestImageProcessor_LoadsEachSourceFormat verifies that an image in each of // TestImageProcessor_LoadsEachSourceFormat verifies that an image in each of
@@ -12,13 +15,15 @@ import (
func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) { func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
t.Parallel() t.Parallel()
proc := New(Params{}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
for _, format := range []Format{ for _, format := range []imagerequest.Format{
FormatJPEG, FormatPNG, FormatGIF, FormatWebP, FormatAVIF, FormatJXL, imagerequest.FormatJPEG, imagerequest.FormatPNG, imagerequest.FormatGIF,
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
} { } {
saved, err := proc.Process(t.Context(), saved, err := proc.Process(t.Context(),
bytes.NewReader(createTestPNG(t, 20, 20)), &Request{Format: format}) bytes.NewReader(createTestPNG(t, 20, 20)), imagerequest.FormatPNG,
&imagerequest.Transform{Format: format})
if err != nil { if err != nil {
t.Fatalf("saving the test image as %s: %v", format, err) t.Fatalf("saving the test image as %s: %v", format, err)
} }
@@ -28,8 +33,8 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
t.Fatalf("reading the %s image: %v", format, err) t.Fatalf("reading the %s image: %v", format, err)
} }
result, err := proc.Process(t.Context(), bytes.NewReader(source), result, err := proc.Process(t.Context(), bytes.NewReader(source), format,
&Request{Format: FormatOriginal}) &imagerequest.Transform{Format: imagerequest.FormatOriginal})
if err != nil { if err != nil {
t.Errorf("orig of the %s image: %v", format, err) t.Errorf("orig of the %s image: %v", format, err)
@@ -38,9 +43,33 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
t.Logf("orig of the %s image: %s", format, result.ContentType) t.Logf("orig of the %s image: %s", format, result.ContentType)
if result.ContentType != FormatToMIME(format) { if result.ContentType != format.MIMEType() {
t.Errorf("orig of the %s image is %s, want %s", t.Errorf("orig of the %s image is %s, want %s",
format, result.ContentType, FormatToMIME(format)) format, result.ContentType, format.MIMEType())
}
}
}
// TestImageProcessor_NotASourceFormatRefused verifies that a source format
// other than the six, such as orig, auto or none, is refused with
// ErrUnsupportedInputFormat. The input is text, which libvips reads as no
// format.
func TestImageProcessor_NotASourceFormatRefused(t *testing.T) {
t.Parallel()
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
for _, sourceFormat := range []imagerequest.Format{
imagerequest.FormatOriginal, imagerequest.FormatAuto, "",
} {
result, err := proc.Process(t.Context(),
bytes.NewReader([]byte("this is text, not an image")), sourceFormat,
&imagerequest.Transform{Format: imagerequest.FormatJPEG})
t.Logf("source format %q: error = %v", sourceFormat, err)
if result != nil || !errors.Is(err, ErrUnsupportedInputFormat) {
t.Errorf("Process() with source format %q = %v, error %v; want %v",
sourceFormat, result, err, ErrUnsupportedInputFormat)
} }
} }
} }
internal/imageproxy/cache_hit_internal_test.go
+100
View File
@@ -0,0 +1,100 @@
package imageproxy
import (
"bytes"
"os"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// TestProxy_Get_MakesAgainVariantOfNoOutputFormat replaces a cached
// variant's file with text, and checks that the next request is a miss
// answered with the variant made again, and the one after a hit.
func TestProxy_Get_MakesAgainVariantOfNoOutputFormat(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := variantFilePath(svc, imagerequest.CacheKey(req))
err := os.WriteFile(path, []byte("this is text, not an image"),
imgcache.StorageFilePerm)
if err != nil {
t.Fatalf("replacing the variant file: %v", err)
}
got, status := getImage(t, svc, req)
t.Logf("after the replacement: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the replacement: Get() = %d bytes, %s; want the %d bytes "+
"made again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestProxy_Get_HitWaitsForNoDatabaseWrite holds the database's one
// connection while a cached variant is requested. The hit must be answered
// while the connection is still held, and its access time must reach the
// database once it is free.
func TestProxy_Get_HitWaitsForNoDatabaseWrite(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
key := imagerequest.CacheKey(req)
getImage(t, svc, req)
// Each connection to an in-memory SQLite database opens a new, empty one,
// so the goroutine that writes the access times must share the
// connection that holds the schema.
svc.db.SetMaxOpenConns(1)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
anHourAgo := time.Now().Add(-time.Hour)
setVariantLastAccessed(t, svc.db, key, anHourAgo)
releaseDatabase := holdDatabase(t, svc.db)
defer releaseDatabase()
results := startGet(t.Context(), svc, req)
select {
case got := <-results:
if got.err != nil {
t.Fatalf("Get() of the cached variant error = %v", got.err)
}
case <-time.After(pendingAccessTimesWait):
t.Fatal("the cache hit waited for the database")
}
releaseDatabase()
deadline := time.Now().Add(pendingAccessTimesWait)
for countRows(t, svc.db, `
SELECT COUNT(*) FROM variant_content
WHERE cache_key = ? AND last_accessed_at > ?
`, string(key), anHourAgo.UTC().Format(sqliteTimestampFormat)) == 0 {
if time.Now().After(deadline) {
t.Fatal("the hit's access time was not written")
}
time.Sleep(10 * time.Millisecond)
}
}
internal/imageproxy/cache_ttl_internal_test.go
+493
View File
@@ -0,0 +1,493 @@
package imageproxy
import (
"bytes"
"context"
"errors"
"fmt"
"image/color"
"io"
"net/http"
"os"
"sync"
"testing"
"time"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// Validators the test upstream sends with its image.
const (
testETag = `"v1"`
testLastModified = "Mon, 02 Nov 2026 10:00:00 GMT"
)
// errConnectionRefused stands for an upstream that cannot be reached.
var errConnectionRefused = errors.New("connection refused")
// upstreamFailure is one way a fetch from upstream fails.
type upstreamFailure struct {
name string
err error
}
// upstreamFailures returns the ways a check of a cached image with upstream
// can fail: an error status, a timeout and no connection.
func upstreamFailures() []upstreamFailure {
return []upstreamFailure{
{"error status", fmt.Errorf("%w: status 503", httpfetcher.ErrUpstreamError)},
{"timeout", httpfetcher.ErrUpstreamTimeout},
{"no connection", errConnectionRefused},
}
}
// conditionalUpstream serves one JPEG image with an ETag or a Last-Modified
// header, or both. It answers 304 Not Modified to a fetch whose If-None-Match
// is its ETag, or, when it has no ETag, whose If-Modified-Since is its
// Last-Modified. While err is set, every fetch fails with it. It records the
// conditional headers of every fetch.
type conditionalUpstream struct {
mu sync.Mutex
image []byte
etag string
lastModified string
err error
ifNoneMatch []string
ifModifiedSince []string
}
func (u *conditionalUpstream) Fetch(
_ context.Context, _ imagerequest.Source,
ifNoneMatch, ifModifiedSince string,
) (*httpfetcher.FetchResult, error) {
u.mu.Lock()
defer u.mu.Unlock()
u.ifNoneMatch = append(u.ifNoneMatch, ifNoneMatch)
u.ifModifiedSince = append(u.ifModifiedSince, ifModifiedSince)
if u.err != nil {
return nil, u.err
}
unchanged := ifNoneMatch == u.etag
if u.etag == "" {
unchanged = ifModifiedSince == u.lastModified
}
if unchanged {
return &httpfetcher.FetchResult{
Content: http.NoBody,
StatusCode: http.StatusNotModified,
}, nil
}
headers := http.Header{}
if u.etag != "" {
headers.Set("ETag", u.etag)
}
if u.lastModified != "" {
headers.Set("Last-Modified", u.lastModified)
}
return &httpfetcher.FetchResult{
Content: io.NopCloser(bytes.NewReader(u.image)),
ContentLength: int64(len(u.image)),
ContentType: testContentTypeJPEG,
Headers: headers,
StatusCode: http.StatusOK,
}, nil
}
// change replaces the upstream image and its ETag.
func (u *conditionalUpstream) change(image []byte, etag string) {
u.mu.Lock()
defer u.mu.Unlock()
u.image = image
u.etag = etag
}
// fail makes every later fetch fail with err.
func (u *conditionalUpstream) fail(err error) {
u.mu.Lock()
defer u.mu.Unlock()
u.err = err
}
// fetches returns how many fetches upstream was asked for.
func (u *conditionalUpstream) fetches() int {
u.mu.Lock()
defer u.mu.Unlock()
return len(u.ifNoneMatch)
}
// lastConditionalHeaders returns the If-None-Match and If-Modified-Since of
// the last fetch.
func (u *conditionalUpstream) lastConditionalHeaders() (string, string) {
u.mu.Lock()
defer u.mu.Unlock()
last := len(u.ifNoneMatch) - 1
return u.ifNoneMatch[last], u.ifModifiedSince[last]
}
// setupConditionalUpstreamProxy returns a proxy whose upstream is a
// conditionalUpstream serving a red JPEG with an ETag.
func setupConditionalUpstreamProxy(
t *testing.T,
) (*testProxy, *TestFixtures, *conditionalUpstream) {
t.Helper()
svc, fixtures := setupTestProxy(t)
upstream := &conditionalUpstream{
image: generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255}),
etag: testETag,
}
svc.fetcher = upstream
return svc, fixtures, upstream
}
// TestProxy_Get_FreshVariantAsksUpstreamNothing checks that a variant whose
// source was checked with upstream less than the cache TTL ago is served from
// the cache without asking upstream.
func TestProxy_Get_FreshVariantAsksUpstreamNothing(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
second, status := getImage(t, svc, variant)
t.Logf("cache status = %s, upstream fetches = %d", status, upstream.fetches())
if status != CacheHit {
t.Errorf("CacheStatus = %q, want %q", status, CacheHit)
}
if !bytes.Equal(first, second) {
t.Error("the cached variant differs from the one first served")
}
if n := upstream.fetches(); n != 1 {
t.Errorf("upstream fetches = %d, want 1", n)
}
}
// TestProxy_Get_UnchangedSourceAfterTTLIsNotDownloadedAgain caches a variant,
// sets its source's check time and the variant's file time two days back,
// and requests it again, with the source still cached and with the source's
// row and file removed, as eviction does. Upstream must be asked with the
// validator it sent, answer 304, and send no bytes; the variant must be served
// again without being made again, and then be served from the cache until the
// TTL passes.
func TestProxy_Get_UnchangedSourceAfterTTLIsNotDownloadedAgain(t *testing.T) {
t.Parallel()
tests := []struct {
name string
sourceEvicted bool
etag string
lastModified string
wantIfNoneMatch string
wantIfModifiedSince string
}{
{"ETag", false, testETag, "", testETag, ""},
{"Last-Modified only", false, "", testLastModified, "", testLastModified},
{"both", false, testETag, testLastModified, testETag, testLastModified},
{"ETag, source evicted", true, testETag, "", testETag, ""},
{"Last-Modified only, source evicted", true, "", testLastModified, "",
testLastModified},
{"both, source evicted", true, testETag, testLastModified, testETag,
testLastModified},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
upstream.etag = tt.etag
upstream.lastModified = tt.lastModified
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
if tt.sourceEvicted {
evictSource(t, svc, variant.Source)
}
second, status := getImage(t, svc, variant)
ifNoneMatch, ifModifiedSince := upstream.lastConditionalHeaders()
counters := readCounters(t, svc)
t.Logf("cache status = %s, If-None-Match = %q, If-Modified-Since = %q, "+
"counters = %+v", status, ifNoneMatch, ifModifiedSince, counters)
if ifNoneMatch != tt.wantIfNoneMatch {
t.Errorf("If-None-Match = %q, want %q", ifNoneMatch, tt.wantIfNoneMatch)
}
if ifModifiedSince != tt.wantIfModifiedSince {
t.Errorf("If-Modified-Since = %q, want %q",
ifModifiedSince, tt.wantIfModifiedSince)
}
if !bytes.Equal(first, second) {
t.Error("the variant served after the check differs from the cached one")
}
if counters.upstreamFetchCount != 1 || counters.transformCount != 1 {
t.Errorf("upstream downloads = %d, transforms = %d; want 1 and 1",
counters.upstreamFetchCount, counters.transformCount)
}
_, status = getImage(t, svc, variant)
if status != CacheHit {
t.Errorf("CacheStatus after the check = %q, want %q", status, CacheHit)
}
if n := upstream.fetches(); n != 2 {
t.Errorf("upstream fetches = %d, want 2", n)
}
})
}
}
// TestProxy_Get_ChangedSourceAfterTTLReplacesSourceAndVariants caches two
// variants of one source, changes the image upstream, and sets the source's
// check time and both variants' file times two days back. Each variant must
// then be made again from the new image, which is downloaded once.
func TestProxy_Get_ChangedSourceAfterTTLReplacesSourceAndVariants(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variants := []*imagerequest.Request{
photoVariant(fixtures, 85, imagerequest.FitCover),
photoVariant(fixtures, 40, imagerequest.FitCover),
}
oldImages := make([][]byte, len(variants))
for i, variant := range variants {
oldImages[i], _ = getImage(t, svc, variant)
}
newSource := generateTestJPEG(t, 120, 90, color.RGBA{0, 0, 255, 255})
upstream.change(newSource, `"v2"`)
twoDaysAgo := time.Now().AddDate(0, 0, -2)
for _, variant := range variants {
setBackSourceAndVariant(t, svc, variant, twoDaysAgo)
}
for i, variant := range variants {
image, _ := getImage(t, svc, variant)
if bytes.Equal(image, oldImages[i]) {
t.Errorf("variant %d is still made from the old image", i)
}
}
counters := readCounters(t, svc)
t.Logf("counters = %+v, upstream fetches = %d", counters, upstream.fetches())
if counters.upstreamFetchCount != 2 {
t.Errorf("upstream downloads = %d, want 2", counters.upstreamFetchCount)
}
cached, err := os.ReadFile(sourceFilePath(svc, variants[0].Source.Key()))
if err != nil {
t.Fatalf("reading the cached source: %v", err)
}
if !bytes.Equal(cached, newSource) {
t.Error("the cached source is not the new image")
}
}
// TestProxy_Get_UpstreamFailureAfterTTLServesCachedImage caches a variant,
// sets its source's check time and the variant's file time two days back, and
// makes upstream fail. The cached variant must still be served, and a variant
// not yet cached must be made from the cached source.
func TestProxy_Get_UpstreamFailureAfterTTLServesCachedImage(t *testing.T) {
t.Parallel()
for _, failure := range upstreamFailures() {
t.Run(failure.name, func(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
upstream.fail(failure.err)
second, _ := getImage(t, svc, variant)
t.Logf("upstream fetches = %d", upstream.fetches())
if upstream.fetches() != 2 {
t.Errorf("upstream fetches = %d, want 2", upstream.fetches())
}
if !bytes.Equal(first, second) {
t.Error("the variant served while upstream fails differs from the cached one")
}
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
})
}
}
// TestProxy_Get_FailingUpstreamIsAskedOnceWhileImageIsServedOften caches a
// variant, sets its source's check time and the variant's file time two days
// back, and makes every fetch time out. Serving the variant ten times, and a
// variant not yet cached, must ask upstream once: the failed check puts the
// source URL in the negative cache.
func TestProxy_Get_FailingUpstreamIsAskedOnceWhileImageIsServedOften(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
upstream.fail(httpfetcher.ErrUpstreamTimeout)
for range 10 {
getImage(t, svc, variant)
}
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
t.Logf("upstream fetches = %d", upstream.fetches())
if n := upstream.fetches(); n != 2 {
t.Errorf("upstream fetches = %d, want 2: the first fetch and one check", n)
}
}
// TestProxy_Get_UpstreamFailureAfterTTLServesVariantOfEvictedSource caches a
// variant, sets its file time two days back, removes its source's row and
// file, as eviction does, and makes upstream fail. The cached variant must
// still be served.
func TestProxy_Get_UpstreamFailureAfterTTLServesVariantOfEvictedSource(t *testing.T) {
t.Parallel()
for _, failure := range upstreamFailures() {
t.Run(failure.name, func(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
evictSource(t, svc, variant.Source)
upstream.fail(failure.err)
second, _ := getImage(t, svc, variant)
t.Logf("upstream fetches = %d", upstream.fetches())
if !bytes.Equal(first, second) {
t.Error("the variant served while upstream fails differs from the cached one")
}
if n := upstream.fetches(); n != 2 {
t.Errorf("upstream fetches = %d, want 2", n)
}
})
}
}
// TestProxy_Get_FailingUpstreamIsAskedOnceForVariantOfEvictedSource caches a
// variant, sets its file time two days back, removes its source's row and
// file, as eviction does, and makes every fetch time out. Serving the variant
// ten times must ask upstream once: the failed check puts the source URL in
// the negative cache.
func TestProxy_Get_FailingUpstreamIsAskedOnceForVariantOfEvictedSource(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
evictSource(t, svc, variant.Source)
upstream.fail(httpfetcher.ErrUpstreamTimeout)
for range 10 {
image, _ := getImage(t, svc, variant)
if !bytes.Equal(image, first) {
t.Fatal("the variant served while upstream fails differs from the cached one")
}
}
t.Logf("upstream fetches = %d", upstream.fetches())
if n := upstream.fetches(); n != 2 {
t.Errorf("upstream fetches = %d, want 2: the first fetch and one check", n)
}
}
// evictSource removes the row and then the file of src's cached source, as
// the cache's eviction does, and leaves its variants cached.
func evictSource(t *testing.T, svc *testProxy, src imagerequest.Source) {
t.Helper()
_, err := svc.db.ExecContext(t.Context(),
`DELETE FROM source_content WHERE source_key = ?`, string(src.Key()))
if err != nil {
t.Fatalf("deleting the source's row: %v", err)
}
err = os.Remove(sourceFilePath(svc, src.Key()))
if err != nil {
t.Fatalf("removing the source's file: %v", err)
}
}
// setBackSourceAndVariant sets back to when the times the cache keeps for
// req's source and for the variant req asks for: the source's fetch and check
// times, both last access times, the variant's check time, and the variant
// file's modification time.
func setBackSourceAndVariant(
t *testing.T, svc *testProxy, req *imagerequest.Request, when time.Time,
) {
t.Helper()
key := req.Source.Key()
setSourceLastAccessed(t, svc.db, key, when)
_, err := svc.db.ExecContext(t.Context(),
`UPDATE source_content SET fetched_at = ?, checked_at = ?
WHERE source_key = ?`, when.UTC(), when.UTC(), string(key))
if err != nil {
t.Fatalf("setting back the source's fetch and check times: %v", err)
}
variantKey := imagerequest.CacheKey(req)
setVariantLastAccessed(t, svc.db, variantKey, when)
_, err = svc.db.ExecContext(t.Context(),
`UPDATE variant_content SET checked_at = ? WHERE cache_key = ?`,
when.UTC(), string(variantKey))
if err != nil {
t.Fatalf("setting back the variant's check time: %v", err)
}
path := variantFilePath(svc, variantKey)
err = os.Chtimes(path, when, when)
if err != nil {
t.Fatalf("setting back the times of %s: %v", path, err)
}
}
internal/imgcache/concurrent_misses_internal_test.go → internal/imageproxy/concurrent_misses_internal_test.go
+119 -68
View File
@@ -1,4 +1,4 @@
package imgcache package imageproxy
import ( import (
"bytes" "bytes"
@@ -15,6 +15,8 @@ import (
"github.com/getsentry/sentry-go" "github.com/getsentry/sentry-go"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic" "sneak.berlin/go/pixa/internal/magic"
) )
@@ -34,7 +36,8 @@ type heldFetcher struct {
} }
func (f *heldFetcher) Fetch( func (f *heldFetcher) Fetch(
ctx context.Context, url string, ctx context.Context, source imagerequest.Source,
ifNoneMatch, ifModifiedSince string,
) (*httpfetcher.FetchResult, error) { ) (*httpfetcher.FetchResult, error) {
f.fetches.Add(1) f.fetches.Add(1)
@@ -46,7 +49,7 @@ func (f *heldFetcher) Fetch(
return nil, ctx.Err() return nil, ctx.Err()
} }
return f.upstream.Fetch(ctx, url) return f.upstream.Fetch(ctx, source, ifNoneMatch, ifModifiedSince)
} }
// releaseFetches lets every held fetch, and every later one, go on. // releaseFetches lets every held fetch, and every later one, go on.
@@ -54,15 +57,15 @@ func (f *heldFetcher) releaseFetches() {
f.releaseOnce.Do(func() { close(f.release) }) f.releaseOnce.Do(func() { close(f.release) })
} }
// setupHeldFetchService returns a test service whose fetches go through a // setupHeldFetchProxy returns a test proxy whose fetches go through a
// heldFetcher. Its database is limited to one connection: each connection to // heldFetcher. Its database is limited to one connection: each connection to
// an in-memory SQLite database opens a new, empty one, so requests running at // an in-memory SQLite database opens a new, empty one, so requests running at
// once must share the connection that holds the schema. // once must share the connection that holds the schema.
func setupHeldFetchService(t *testing.T) (*Service, *TestFixtures, *heldFetcher) { func setupHeldFetchProxy(t *testing.T) (*testProxy, *TestFixtures, *heldFetcher) {
t.Helper() t.Helper()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
svc.cache.db.SetMaxOpenConns(1) svc.db.SetMaxOpenConns(1)
fetcher := &heldFetcher{ fetcher := &heldFetcher{
upstream: svc.fetcher, upstream: svc.fetcher,
@@ -77,16 +80,22 @@ func setupHeldFetchService(t *testing.T) (*Service, *TestFixtures, *heldFetcher)
} }
// photoVariant asks for the test photo, 100x100, at 50x25 as a JPEG of the // photoVariant asks for the test photo, 100x100, at 50x25 as a JPEG of the
// given quality and fit mode. Each call returns a new request, as Get writes // given quality and fit mode. Each call returns a new request, which a test
// to the request it is given. // may change.
func photoVariant(fixtures *TestFixtures, quality int, fit FitMode) *ImageRequest { func photoVariant(
return &ImageRequest{ fixtures *TestFixtures, quality int, fit imagerequest.Fit,
SourceHost: fixtures.GoodHost, ) *imagerequest.Request {
SourcePath: testPathPhoto, return &imagerequest.Request{
Size: Size{Width: 50, Height: 25}, Source: imagerequest.Source{
Format: FormatJPEG, Host: fixtures.GoodHost,
Quality: quality, Path: testPathPhoto,
FitMode: fit, },
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 25},
Format: imagerequest.FormatJPEG,
Quality: quality,
FitMode: fit,
},
} }
} }
@@ -99,7 +108,7 @@ type getResult struct {
// startGet calls Get in a goroutine of its own and delivers what it returned // startGet calls Get in a goroutine of its own and delivers what it returned
// on the channel. // on the channel.
func startGet( func startGet(
ctx context.Context, svc *Service, req *ImageRequest, ctx context.Context, svc *testProxy, req *imagerequest.Request,
) <-chan getResult { ) <-chan getResult {
results := make(chan getResult, 1) results := make(chan getResult, 1)
@@ -131,21 +140,21 @@ func jpegSize(data []byte) (int, int) {
return config.Width, config.Height return config.Width, config.Height
} }
// TestService_Get_ConcurrentMissesShareOneFetch starts several requests for // TestProxy_Get_ConcurrentMissesShareOneFetch starts several requests for
// one uncached variant while the first one's fetch is held. Between them they // one uncached variant while the first one's fetch is held. Between them they
// must fetch the source once and transcode it once, every one must be answered // must fetch the source once and transcode it once, every one must be answered
// with the same 50x25 JPEG, and each must count one miss. // with the same 50x25 JPEG, and each must count one miss.
func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) { func TestProxy_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
const requests = 8 const requests = 8
pending := make([]<-chan getResult, 0, requests) pending := make([]<-chan getResult, 0, requests)
for range requests { for range requests {
pending = append(pending, pending = append(pending,
startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover))) startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover)))
} }
<-fetcher.started <-fetcher.started
@@ -175,7 +184,7 @@ func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
t.Errorf("%d requests made %d upstream fetches, want 1", requests, fetches) t.Errorf("%d requests made %d upstream fetches, want 1", requests, fetches)
} }
// NewTestFS builds the same files the test service's fetcher serves. // NewTestFS builds the same files the test proxy's fetcher serves.
testFS, _ := NewTestFS(t) testFS, _ := NewTestFS(t)
photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG) photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG)
@@ -183,24 +192,26 @@ func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
want := cacheStatsCounters{0, requests, 1, int64(len(photo)), 1} want := counterValues{0, requests, 1, int64(len(photo)), 1}
if got := readCacheStatsCounters(t, svc.cache); got != want { if got := readCounters(t, svc); got != want {
t.Errorf("counters = %+v, want %+v", got, want) t.Errorf("counters = %+v, want %+v", got, want)
} }
} }
// TestService_Get_ConcurrentVariantsStayApart requests three variants of the // TestProxy_Get_ConcurrentVariantsStayApart requests three variants of the
// test photo at once that differ only in quality or fit. Each must be made by // test photo at once that differ only in quality or fit. Each must be made by
// a fetch and a transcode of its own, and each answer must be its own variant. // a fetch and a transcode of its own, and each answer must be its own variant.
func TestService_Get_ConcurrentVariantsStayApart(t *testing.T) { func TestProxy_Get_ConcurrentVariantsStayApart(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
cover := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover)) cover := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
lowQuality := startGet(t.Context(), svc, photoVariant(fixtures, 40, FitCover)) lowQuality := startGet(t.Context(), svc,
contain := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitContain)) photoVariant(fixtures, 40, imagerequest.FitCover))
contain := startGet(t.Context(), svc,
photoVariant(fixtures, 85, imagerequest.FitContain))
for range 3 { for range 3 {
select { select {
@@ -243,21 +254,21 @@ func TestService_Get_ConcurrentVariantsStayApart(t *testing.T) {
} }
} }
// TestService_Get_WaiterStopsWhenItsContextEnds has a second request for a // TestProxy_Get_WaiterStopsWhenItsContextEnds has a second request for a
// variant join the first one's held fetch, then ends the second request's // variant join the first one's held fetch, then ends the second request's
// context. The second request must return at once with the context's error, // context. The second request must return at once with the context's error,
// while the fetch is still held, and the first must still be answered. // while the fetch is still held, and the first must still be answered.
func TestService_Get_WaiterStopsWhenItsContextEnds(t *testing.T) { func TestProxy_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
first := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover)) first := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
<-fetcher.started <-fetcher.started
waiterCtx, cancelWaiter := context.WithCancel(t.Context()) waiterCtx, cancelWaiter := context.WithCancel(t.Context())
waiter := startGet(waiterCtx, svc, photoVariant(fixtures, 85, FitCover)) waiter := startGet(waiterCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
time.Sleep(arrivalWait) time.Sleep(arrivalWait)
cancelWaiter() cancelWaiter()
@@ -288,22 +299,22 @@ func TestService_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
} }
} }
// TestService_Get_FirstRequestLeavingKeepsTheWork ends the context of the // TestProxy_Get_FirstRequestLeavingKeepsTheWork ends the context of the
// request whose fetch is held, after a second request has joined it. The fetch // request whose fetch is held, after a second request has joined it. The fetch
// and transcode must go on and answer the second request. The first request // and transcode must go on and answer the second request. The first request
// waits for its own work, as every request did before misses were shared, and // waits for its own work, as every request did before misses were shared, and
// is answered too. // is answered too.
func TestService_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) { func TestProxy_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
firstCtx, cancelFirst := context.WithCancel(t.Context()) firstCtx, cancelFirst := context.WithCancel(t.Context())
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, FitCover)) first := startGet(firstCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
<-fetcher.started <-fetcher.started
second := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover)) second := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
time.Sleep(arrivalWait) time.Sleep(arrivalWait)
cancelFirst() cancelFirst()
@@ -327,12 +338,11 @@ func TestService_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
} }
} }
// TestService_Get_ConcurrentMissesShareAFailure has several requests for an // TestProxy_Get_ConcurrentMissesShareAFailure has several requests for an
// image that cannot be served arrive while its fetch is held: the one fetch // image that cannot be served arrive while its fetch is held: the one fetch
// answers all of them with its error. The request after them is answered from // answers all of them with its error. The request after them is answered from
// the negative cache when the failure is kept there, and fetches again when it // the negative cache, where the failure is kept.
// is not. func TestProxy_Get_ConcurrentMissesShareAFailure(t *testing.T) {
func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
t.Parallel() t.Parallel()
tests := []struct { tests := []struct {
@@ -345,19 +355,19 @@ func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
{"upstream answers 404, kept in the negative cache", {"upstream answers 404, kept in the negative cache",
"/images/missing.jpg", httpfetcher.ErrUpstreamError, "/images/missing.jpg", httpfetcher.ErrUpstreamError,
ErrNegativeCached, 1}, ErrNegativeCached, 1},
{"source fails the magic byte check, not kept", {"source fails the magic byte check, kept in the negative cache",
"/images/text.png", magic.ErrUnknownFormat, magic.ErrUnknownFormat, 2}, "/images/text.png", magic.ErrUnknownFormat, ErrNegativeCached, 1},
} }
for _, tc := range tests { for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
request := func() *ImageRequest { request := func() *imagerequest.Request {
req := photoVariant(fixtures, 85, FitCover) req := photoVariant(fixtures, 85, imagerequest.FitCover)
req.SourcePath = tc.path req.Source.Path = tc.path
return req return req
} }
@@ -389,17 +399,17 @@ func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
} }
} }
// TestService_Get_EndedRequestFetchesNothing checks that a request whose // TestProxy_Get_EndedRequestFetchesNothing checks that a request whose
// context has already ended when it misses the cache starts no fetch. // context has already ended when it misses the cache starts no fetch.
func TestService_Get_EndedRequestFetchesNothing(t *testing.T) { func TestProxy_Get_EndedRequestFetchesNothing(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
ctx, cancel := context.WithCancel(t.Context()) ctx, cancel := context.WithCancel(t.Context())
cancel() cancel()
_, err := svc.Get(ctx, photoVariant(fixtures, 85, FitCover)) _, err := svc.Get(ctx, photoVariant(fixtures, 85, imagerequest.FitCover))
if !errors.Is(err, context.Canceled) { if !errors.Is(err, context.Canceled) {
t.Errorf("Get() error = %v, want %v", err, context.Canceled) t.Errorf("Get() error = %v, want %v", err, context.Canceled)
} }
@@ -409,21 +419,21 @@ func TestService_Get_EndedRequestFetchesNothing(t *testing.T) {
} }
} }
// TestService_Get_ReturnsByItsDeadline gives a request a deadline and holds // TestProxy_Get_ReturnsByItsDeadline gives a request a deadline and holds
// its fetch until the fetch's context ends, as the fetcher does while it waits // its fetch until the fetch's context ends, as the fetcher does while it waits
// for a free connection to the host. The request must return by its deadline // for a free connection to the host. The request must return by its deadline
// with the deadline's error. // with the deadline's error.
func TestService_Get_ReturnsByItsDeadline(t *testing.T) { func TestProxy_Get_ReturnsByItsDeadline(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, _ := setupHeldFetchService(t) svc, fixtures, _ := setupHeldFetchProxy(t)
const timeout = 200 * time.Millisecond const timeout = 200 * time.Millisecond
ctx, cancel := context.WithTimeout(t.Context(), timeout) ctx, cancel := context.WithTimeout(t.Context(), timeout)
defer cancel() defer cancel()
results := startGet(ctx, svc, photoVariant(fixtures, 85, FitCover)) results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
select { select {
case got := <-results: case got := <-results:
@@ -435,22 +445,63 @@ func TestService_Get_ReturnsByItsDeadline(t *testing.T) {
} }
} }
// TestProxy_Get_WaiterGetsProcessingTimedOut has a request with no deadline
// join the held fetch of a first request with a short one. With the only
// processing slot held, the first request's transcode waits for it until the
// first request's deadline ends, and stops then. The waiting request, whose
// own context has not ended, must get ErrProcessingTimedOut, which the image
// routes answer 503 as when pixa is busy.
func TestProxy_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc.processor = imageprocessor.New(
imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1},
)
holdProcessingSlot(t, svc.processor)
const firstTimeout = 500 * time.Millisecond
firstCtx, cancelFirst := context.WithTimeout(t.Context(), firstTimeout)
defer cancelFirst()
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
<-fetcher.started
waiter := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
time.Sleep(arrivalWait)
fetcher.releaseFetches()
t.Logf("first request: Get() error = %v", (<-first).err)
got := <-waiter
t.Logf("waiting request: Get() error = %v", got.err)
if !errors.Is(got.err, ErrProcessingTimedOut) {
t.Errorf("waiting request: Get() error = %v, want %v",
got.err, ErrProcessingTimedOut)
}
}
// panickingFetcher panics on every fetch. // panickingFetcher panics on every fetch.
type panickingFetcher struct{} type panickingFetcher struct{}
func (panickingFetcher) Fetch( func (panickingFetcher) Fetch(
context.Context, string, context.Context, imagerequest.Source, string, string,
) (*httpfetcher.FetchResult, error) { ) (*httpfetcher.FetchResult, error) {
panic("upstream fetcher panicked") panic("upstream fetcher panicked")
} }
// TestService_Get_PanicBecomesAnError checks that a panic while a variant is // TestProxy_Get_PanicBecomesAnError checks that a panic while a variant is
// being made reaches its request as an error naming the panic, instead of // being made reaches its request as an error naming the panic, instead of
// being raised again. // being raised again.
func TestService_Get_PanicBecomesAnError(t *testing.T) { func TestProxy_Get_PanicBecomesAnError(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
svc.fetcher = panickingFetcher{} svc.fetcher = panickingFetcher{}
var err error var err error
@@ -462,7 +513,7 @@ func TestService_Get_PanicBecomesAnError(t *testing.T) {
} }
}() }()
_, err = svc.Get(t.Context(), photoVariant(fixtures, 85, FitCover)) _, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
}() }()
if err == nil || !strings.Contains(err.Error(), "upstream fetcher panicked") { if err == nil || !strings.Contains(err.Error(), "upstream fetcher panicked") {
@@ -470,13 +521,13 @@ func TestService_Get_PanicBecomesAnError(t *testing.T) {
} }
} }
// TestService_Get_PanicIsReportedToSentry checks that a panic while a variant // TestProxy_Get_PanicIsReportedToSentry checks that a panic while a variant
// is being made is reported through the Sentry hub on the request's context, // is being made is reported through the Sentry hub on the request's context,
// where the Sentry middleware puts one when sentry_dsn is set. // where the Sentry middleware puts one when sentry_dsn is set.
func TestService_Get_PanicIsReportedToSentry(t *testing.T) { func TestProxy_Get_PanicIsReportedToSentry(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
svc.fetcher = panickingFetcher{} svc.fetcher = panickingFetcher{}
transport := &sentry.MockTransport{} transport := &sentry.MockTransport{}
@@ -499,7 +550,7 @@ func TestService_Get_PanicIsReportedToSentry(t *testing.T) {
} }
}() }()
_, _ = svc.Get(ctx, photoVariant(fixtures, 85, FitCover)) _, _ = svc.Get(ctx, photoVariant(fixtures, 85, imagerequest.FitCover))
}() }()
events := transport.Events() events := transport.Events()
internal/imageproxy/counters_internal_test.go
+77
View File
@@ -0,0 +1,77 @@
package imageproxy
import (
"errors"
"testing"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/testutil"
"sneak.berlin/go/pixa/internal/imgcache"
)
// TestNew_RegistersItsCountersOnItsRegistry checks that New registers the
// proxy's counters on the registry it is given, under the names README.md
// gives them. The cache registers its own counters on another registry.
func TestNew_RegistersItsCountersOnItsRegistry(t *testing.T) {
t.Parallel()
cache, err := imgcache.NewCache(setupTestDB(t),
imgcache.Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err)
}
registry := prometheus.NewRegistry()
_, err = New(&Config{Cache: cache}, registry)
if err != nil {
t.Fatalf("New() error = %v", err)
}
names := []string{
"pixa_upstream_fetches_total",
"pixa_upstream_fetch_bytes_total",
"pixa_transforms_total",
}
count, err := testutil.GatherAndCount(registry, names...)
if err != nil {
t.Fatalf("testutil.GatherAndCount() error = %v", err)
}
if count != len(names) {
t.Errorf("%d of the counters %v are on the registry, want %d",
count, names, len(names))
}
}
// TestNew_FailsWhenItsCountersCannotBeRegistered makes a second proxy on
// the registry of a first. Its counters have the same names, so the
// registry refuses them, and New must return that error.
func TestNew_FailsWhenItsCountersCannotBeRegistered(t *testing.T) {
t.Parallel()
cache, err := imgcache.NewCache(setupTestDB(t),
imgcache.Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err)
}
registry := prometheus.NewRegistry()
_, err = New(&Config{Cache: cache}, registry)
if err != nil {
t.Fatalf("New() of the first proxy error = %v", err)
}
_, err = New(&Config{Cache: cache}, registry)
t.Logf("New() of the second proxy error = %v", err)
var alreadyRegistered prometheus.AlreadyRegisteredError
if !errors.As(err, &alreadyRegistered) {
t.Errorf("New() of the second proxy error = %v, want %T",
err, alreadyRegistered)
}
}
internal/imgcache/divzero_internal_test.go → internal/imageproxy/divzero_internal_test.go
+1 -1
View File
@@ -1,4 +1,4 @@
package imgcache package imageproxy
import ( import (
"fmt" "fmt"
internal/imageproxy/imageproxy.go
+933
View File
@@ -0,0 +1,933 @@
// Package imageproxy fetches, transforms, caches and serves images. Its
// Proxy serves an image from the cache when it can, and otherwise makes it
// from the cached or fetched source image.
package imageproxy
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"runtime/debug"
"time"
"github.com/dustin/go-humanize"
"github.com/getsentry/sentry-go"
"github.com/go-chi/chi/v5/middleware"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/sync/singleflight"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic"
)
// Proxy orchestrates the cache, fetcher, and processor.
type Proxy struct {
cache *imgcache.Cache
fetcher httpfetcher.Fetcher
processor *imageprocessor.ImageProcessor
log *slog.Logger
maxResponseSize int64
// variantsInProgress lets the requests that miss the same variant at the
// same time share one fetch and one transcode.
variantsInProgress singleflight.Group
// The metrics of upstream fetches and transforms, made and registered by
// New. They are kept in memory, so updating one waits for nothing.
upstreamFetches prometheus.Counter
upstreamFetchBytes prometheus.Counter
upstreamFetchDurations prometheus.Histogram
upstreamFetchErrors prometheus.Counter
transforms prometheus.Counter
transformDurations prometheus.Histogram
transformsInProgress prometheus.Gauge
}
// Config holds the proxy's configuration.
type Config struct {
// Cache is the cache instance
Cache *imgcache.Cache
// Fetcher fetches source images from upstream hosts
Fetcher httpfetcher.Fetcher
// Processor converts and resizes the images
Processor *imageprocessor.ImageProcessor
// MaxResponseSize is the largest source accepted, in bytes; a cached
// source larger than this is fetched again
MaxResponseSize int64
// Logger for logging
Logger *slog.Logger
}
// errCacheRequired is returned when the proxy is built without a cache.
var errCacheRequired = errors.New("cache is required")
// New creates a new image proxy and registers its metrics on
// metricsRegistry. pixad passes Prometheus' default registry, the one
// /metrics serves. It fails when metricsRegistry already holds metrics of
// the same names.
func New(
cfg *Config, metricsRegistry prometheus.Registerer,
) (*Proxy, error) {
if cfg.Cache == nil {
return nil, errCacheRequired
}
log := cfg.Logger
if log == nil {
log = slog.Default()
}
// JPEG XL is the default output format, so pixad does not start
// without it.
err := imageprocessor.CheckJPEGXLSupport()
if err != nil {
return nil, err
}
p := &Proxy{
cache: cfg.Cache,
fetcher: cfg.Fetcher,
processor: cfg.Processor,
log: log,
maxResponseSize: cfg.MaxResponseSize,
upstreamFetches: prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetches_total",
Help: "Fetches of a source image from upstream that read any bytes.",
}),
upstreamFetchBytes: prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetch_bytes_total",
Help: "Bytes read from upstream by fetches of source images.",
}),
upstreamFetchDurations: prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "pixa_upstream_fetch_duration_seconds",
Help: "Duration of the fetches pixa_upstream_fetches_total counts, " +
"until the response's last byte.",
}),
upstreamFetchErrors: prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetch_errors_total",
Help: "Fetches of a source image from upstream that failed, " +
"before or while reading the response.",
}),
transforms: prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_transforms_total",
Help: "Images the image processor transformed.",
}),
transformDurations: prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "pixa_transform_duration_seconds",
Help: "Duration of the transforms pixa_transforms_total counts, " +
"waiting for a processing slot included.",
}),
transformsInProgress: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "pixa_transforms_in_progress",
Help: "Images being transformed or waiting for a processing slot.",
}),
}
for _, metric := range []prometheus.Collector{
p.upstreamFetches, p.upstreamFetchBytes, p.upstreamFetchDurations,
p.upstreamFetchErrors, p.transforms, p.transformDurations,
p.transformsInProgress,
} {
err = metricsRegistry.Register(metric)
if err != nil {
return nil, fmt.Errorf("failed to register the proxy's metrics: %w", err)
}
}
return p, nil
}
// ErrNegativeCached is returned when a URL is in the negative cache (recently failed).
var ErrNegativeCached = errors.New("request is in negative cache (recently failed)")
// ErrProcessingTimedOut is returned when processing an image stopped because
// the downstream_timeout of the request doing it ended, to that request and to
// every request that waited for its image.
var ErrProcessingTimedOut = errors.New(
"image processing stopped at the deadline of the request doing it")
// ImageResponse represents a processed image ready to serve
type ImageResponse struct {
// Content is the image data reader
Content io.ReadCloser
// ContentLength is the size in bytes (-1 if unknown)
ContentLength int64
// ContentType is the MIME type of the response
ContentType string
// ETag is the entity tag for caching
ETag string
// CacheStatus indicates HIT or MISS
CacheStatus CacheStatus
// FetchedBytes is the number of bytes fetched from upstream (0 if cache hit)
FetchedBytes int64
}
// CacheStatus indicates whether the response was served from cache.
type CacheStatus string
// Cache status values for response headers.
const (
CacheHit CacheStatus = "HIT"
CacheMiss CacheStatus = "MISS"
)
// Get retrieves a processed image, fetching and processing if necessary.
// While its source URL is in the negative cache, a cached variant is still
// served and a variant not in the cache is still made from a cached source;
// only a request that needs the source fetched is refused.
//
// A cache hit opens the variant file and makes no database call: the cache
// hands its access time to the goroutine that writes access times, without
// waiting for it, and its hit is counted in memory. A variant whose source
// was last checked with upstream CacheTTL or more ago is a miss, which checks
// the source first (see processFromSourceOrFetch). A variant file that
// cannot be served, such as one whose first bytes are not those of an output
// format, is a miss, and the variant is made again.
func (p *Proxy) Get(
ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) {
cacheKey := imagerequest.CacheKey(req)
cached, err := p.cache.Variant(cacheKey)
if err == nil {
p.cache.IncrementStats(true)
return &ImageResponse{
Content: cached.Content,
ContentLength: cached.Size,
ContentType: cached.ContentType,
CacheStatus: CacheHit,
ETag: formatETag(cacheKey, cached.CheckedAt),
}, nil
}
if !errors.Is(err, imgcache.ErrNotFound) && !errors.Is(err, imgcache.ErrStale) {
p.log.Warn("cached variant not served, making it again",
"cache_key", cacheKey, "error", err)
}
// Cache miss - get the variant, processed once for all the requests that
// miss it at the same time, then count this request's miss, also when it
// failed or the request context has ended meanwhile
response, err := p.processOrWait(ctx, req)
p.cache.IncrementStats(false)
if err != nil {
return nil, err
}
response.CacheStatus = CacheMiss
return response, nil
}
// WaitForProcessing waits until no image is being processed, or until ctx
// ends, and returns how many images were still being processed then.
func (p *Proxy) WaitForProcessing(ctx context.Context) int {
return p.processor.WaitForProcessing(ctx)
}
// errPanicked is returned when processing a variant panicked.
var errPanicked = errors.New("panic while processing image")
// processOrWait returns the variant req asks for. The first of the requests
// that miss a variant at the same time processes it, and singleflight hands
// its result, or its error, to the others: they fetch nothing, read no source
// and take no upstream connection or processing slot. The processing ignores
// the first request's cancellation, so the others are still served if that
// client goes away, but keeps its deadline.
func (p *Proxy) processOrWait(
ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) {
// A request that has already ended starts no processing
if ctx.Err() != nil {
return nil, ctx.Err()
}
cacheKey := imagerequest.CacheKey(req)
// Closed when this request's own function runs, which singleflight does
// only when no other request is processing the variant
processing := make(chan struct{})
results := p.variantsInProgress.DoChan(string(cacheKey),
func() (_ any, err error) {
close(processing)
// singleflight would raise a panic again in a goroutine of its
// own, where no handler recovers it, and stop pixad. It is
// reported through the Sentry hub that the Sentry middleware puts
// on the request's context when sentry_dsn is set.
defer func() {
recovered := recover()
if recovered != nil {
p.log.Error("panic while processing image",
"host", req.Source.Host, "path", req.Source.Path,
"panic", recovered, "stack", string(debug.Stack()))
if hub := sentry.GetHubFromContext(ctx); hub != nil {
hub.RecoverWithContext(ctx, recovered)
}
err = fmt.Errorf("%w: %v", errPanicked, recovered)
}
}()
processingCtx, cancel := withoutCancelKeepingDeadline(ctx)
defer cancel()
return p.processFromSourceOrFetch(processingCtx, req, cacheKey)
})
var result singleflight.Result
select {
case result = <-results:
case <-ctx.Done():
select {
case <-processing:
// This request is processing the variant: it waits for the
// result, as every request did before misses were shared
result = <-results
default:
// Another request is processing the variant, or this request's
// function has not started yet; the processing goes on without it
return nil, ctx.Err()
}
}
if result.Err != nil {
return nil, result.Err
}
variant, _ := result.Val.(*processedVariant)
return &ImageResponse{
Content: io.NopCloser(bytes.NewReader(variant.data)),
ContentLength: int64(len(variant.data)),
ContentType: variant.contentType,
FetchedBytes: variant.fetchedBytes,
ETag: formatETag(cacheKey, variant.checkedAt),
}, nil
}
// withoutCancelKeepingDeadline returns ctx without its cancellation but with
// its deadline, if it has one, and the func that releases the returned
// context.
func withoutCancelKeepingDeadline(
ctx context.Context,
) (context.Context, context.CancelFunc) {
deadline, hasDeadline := ctx.Deadline()
if !hasDeadline {
return context.WithoutCancel(ctx), func() {}
}
return context.WithDeadline(context.WithoutCancel(ctx), deadline)
}
// loadCachedSource opens src's cached source, reading only the first bytes,
// which give its format, and returns it with its size and its format. It
// returns nil if the source is not in the cache, cannot be read, is in no
// source format or exceeds maxResponseSize.
func (p *Proxy) loadCachedSource(
ctx context.Context, src imagerequest.Source,
) (io.ReadCloser, int64, imagerequest.Format) {
reader, size, format, err := p.cache.Source(ctx, src)
if errors.Is(err, imgcache.ErrNotFound) {
return nil, 0, ""
}
if err != nil {
p.log.Warn("failed to load cached source, fetching", "error", err)
return nil, 0, ""
}
p.log.Debug("using cached source", "source_key", src.Key())
if size > p.maxResponseSize {
_ = reader.Close()
p.log.Warn("cached source exceeds max response size, discarding",
"source_key", src.Key(),
"max_bytes", p.maxResponseSize,
)
return nil, 0, ""
}
return reader, size, format
}
// processFromSourceOrFetch makes the variant req asks for, from the cached
// source when there is one and from a fetched source otherwise. It counts a
// source cache miss when it fetches the source because the source has no row
// in the cache or its cached file cannot be used, and a source cache hit
// otherwise.
//
// A cached source last checked with upstream CacheTTL or more ago is checked
// first: upstream is asked to send it only if it changed since. When it
// changed, the new bytes replace the cached source and the variant is made
// from them. When it did not change, or the check fails, the cached source
// is used: the variant's cached file is served again if it was made from the
// cached source's bytes, and otherwise the variant is made from them. When
// the source is not cached but the variant is, the variant is checked with
// upstream the same way (see fetchAndProcess).
func (p *Proxy) processFromSourceOrFetch(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
) (*processedVariant, error) {
cached, err := p.cache.LookupSource(ctx, req.Source)
if err != nil {
if !errors.Is(err, imgcache.ErrNotFound) {
p.log.Warn("failed to look up cached source, fetching", "error", err)
}
p.cache.IncrementSourceStats(false)
return p.fetchAndProcess(ctx, req, cacheKey)
}
if p.cache.Stale(cached.CheckedAt) {
fetched, err := p.fetchSource(ctx, req, cached.ETag, cached.LastModified)
switch {
case errors.Is(err, errSourceNotModified):
cached.CheckedAt = time.Now().UTC()
p.renewSource(ctx, req.Source, cached.CheckedAt)
case err != nil:
p.recordFailedCheck(ctx, req, err)
default:
p.cache.IncrementSourceStats(true)
return p.processFetched(ctx, req, cacheKey, fetched)
}
}
variant := p.renewVariant(ctx, cacheKey, cached)
if variant != nil {
p.cache.IncrementSourceStats(true)
return variant, nil
}
source, sourceSize, sourceFormat := p.loadCachedSource(ctx, req.Source)
p.cache.IncrementSourceStats(source != nil)
// Fetch from upstream if we don't have source data or it's empty
if source == nil {
return p.fetchAndProcess(ctx, req, cacheKey)
}
defer func() { _ = source.Close() }()
// Process using cached source; nothing was fetched from upstream. The
// image processor reads the source only once it has a processing slot,
// so a request waiting for one holds none of it in memory.
return p.processAndStore(ctx, req, cacheKey, cached, source, sourceFormat,
sourceSize)
}
// recordFailedCheck logs a failed check of req's source with upstream, after
// which the image is still served from the cache. It puts the source URL in
// the negative cache, whatever the failure, a timeout included, so upstream
// is asked at most once per NegativeTTL while it fails, however often the
// image is served. A failure that was a negative cache hit records nothing.
func (p *Proxy) recordFailedCheck(
ctx context.Context, req *imagerequest.Request, err error,
) {
if errors.Is(err, ErrNegativeCached) {
return
}
p.log.Warn("checking source with upstream failed, serving from the cache",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host, "path", req.Source.Path, "error", err)
p.storeNegative(ctx, req)
}
// renewSource records that upstream answered at checkedAt that src's cached
// source had not changed. A failure to record it is only logged: the source
// is then checked again on its next use.
func (p *Proxy) renewSource(
ctx context.Context, src imagerequest.Source, checkedAt time.Time,
) {
err := p.cache.RenewSource(ctx, src, checkedAt)
if err != nil {
p.log.Warn("failed to renew cached source", "source_key", src.Key(),
"error", err)
}
}
// renewVariant returns the cached variant of cacheKey when it was made from
// the bytes of the cached source that source describes, and renews it to
// source.CheckedAt (see imgcache.Cache.RenewVariant). It returns nil when
// there is no such variant or it cannot be read.
func (p *Proxy) renewVariant(
ctx context.Context, cacheKey imagerequest.VariantKey,
source imgcache.SourceInfo,
) *processedVariant {
cached, err := p.cache.RenewVariant(ctx, cacheKey, source)
if err != nil {
if !errors.Is(err, imgcache.ErrNotFound) {
p.log.Warn("cached variant not renewed, making it again",
"cache_key", cacheKey, "error", err)
}
return nil
}
return p.readCachedVariant(cacheKey, cached)
}
// cachedVariantIgnoringTTL returns the cached variant of cacheKey whatever
// its age, with what its row keeps about the source bytes it was made from
// (see imgcache.Cache.LookupVariant). It returns nil when the variant is not
// cached or cannot be read. The SourceInfo is empty when the variant is not
// returned, or its row is missing or cannot be read.
func (p *Proxy) cachedVariantIgnoringTTL(
ctx context.Context, cacheKey imagerequest.VariantKey,
) (*processedVariant, imgcache.SourceInfo) {
cached, err := p.cache.VariantIgnoringTTL(cacheKey)
if err != nil {
if !errors.Is(err, imgcache.ErrNotFound) {
p.log.Warn("cached variant not served", "cache_key", cacheKey,
"error", err)
}
return nil, imgcache.SourceInfo{}
}
variant := p.readCachedVariant(cacheKey, cached)
if variant == nil {
return nil, imgcache.SourceInfo{}
}
madeFrom, err := p.cache.LookupVariant(ctx, cacheKey)
if err != nil && !errors.Is(err, imgcache.ErrNotFound) {
p.log.Warn("failed to look up cached variant", "cache_key", cacheKey,
"error", err)
}
return variant, madeFrom
}
// readCachedVariant reads the cached variant of cacheKey from its opened file,
// and closes the file. It returns nil when the variant cannot be read.
func (p *Proxy) readCachedVariant(
cacheKey imagerequest.VariantKey, file imgcache.VariantFile,
) *processedVariant {
defer func() { _ = file.Content.Close() }()
data, err := io.ReadAll(file.Content)
if err != nil {
p.log.Warn("failed to read cached variant", "cache_key", cacheKey,
"error", err)
return nil
}
return &processedVariant{
data: data,
contentType: file.ContentType,
checkedAt: file.CheckedAt,
}
}
// fetchAndProcess fetches the source from upstream, stores it, and makes
// and stores the variant from it.
//
// The variant can still be cached after eviction removed its source. The
// fetch then sends the ETag and Last-Modified of the source bytes the variant
// was made from, so upstream sends the source only if it changed since. When
// it did not change, the cached variant is renewed and served, with no
// download and no conversion. When the fetch fails, the cached variant is
// served whatever its age, and the failure is recorded as a failed check (see
// recordFailedCheck).
func (p *Proxy) fetchAndProcess(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
) (*processedVariant, error) {
variant, madeFrom := p.cachedVariantIgnoringTTL(ctx, cacheKey)
fetched, err := p.fetchSource(ctx, req, madeFrom.ETag, madeFrom.LastModified)
switch {
case err == nil:
return p.processFetched(ctx, req, cacheKey, fetched)
case variant == nil:
return nil, err
case errors.Is(err, errSourceNotModified):
variant.checkedAt = time.Now().UTC()
err = p.cache.SetVariantCheckedAt(ctx, cacheKey, variant.checkedAt)
if err != nil {
p.log.Warn("failed to renew cached variant", "cache_key", cacheKey,
"error", err)
}
default:
p.recordFailedCheck(ctx, req, err)
}
return variant, nil
}
// processFetched makes and stores the variant from the source fetchSource
// fetched, then closes the fetch's response.
func (p *Proxy) processFetched(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
fetched *fetchedSource,
) (*processedVariant, error) {
// Closing the response frees the upstream connection. It is closed only
// after processing, so the fetcher's connection limit also bounds the
// fetched sources held in memory while their requests wait for a
// processing slot.
defer func() { _ = fetched.response.Close() }()
return p.processAndStore(ctx, req, cacheKey, fetched.info,
bytes.NewReader(fetched.data), fetched.format, int64(len(fetched.data)))
}
// errSourceNotModified is returned by fetchSource when upstream answers 304
// Not Modified: the cached source has not changed.
var errSourceNotModified = errors.New("upstream source not modified")
// fetchedSource is a source fetchSource fetched and stored in the cache.
type fetchedSource struct {
data []byte
format imagerequest.Format
// info is what the cache keeps about the source
info imgcache.SourceInfo
// response is the fetch's response, still open; closing it frees the
// upstream connection
response io.Closer
}
// fetchSource fetches req's source from upstream and stores it in the cache.
// A non-empty ifNoneMatch or ifModifiedSince asks upstream to send the source
// only if it changed since; errSourceNotModified says it did not. It counts
// the fetch with the bytes read from upstream and its duration, including
// when reading the response or a later step fails, and counts a fetch that
// fails or whose response cannot be read as an upstream fetch error. It
// fetches nothing while the source URL is in the negative cache. The caller
// closes the returned response.
func (p *Proxy) fetchSource(
ctx context.Context,
req *imagerequest.Request,
ifNoneMatch, ifModifiedSince string,
) (*fetchedSource, error) {
err := p.refuseNegativeCached(ctx, req)
if err != nil {
return nil, err
}
// Fetch from upstream
p.log.Debug("fetching from upstream", "host", req.Source.Host,
"path", req.Source.Path, "query", req.Source.Query)
fetchStart := time.Now()
fetchResult, err := p.fetcher.Fetch(ctx, req.Source,
ifNoneMatch, ifModifiedSince)
if err != nil {
p.upstreamFetchErrors.Inc()
// Store negative cache for certain errors
if isNegativeCacheable(err) {
p.storeNegative(ctx, req)
}
return nil, fmt.Errorf("upstream fetch failed: %w", err)
}
if fetchResult.StatusCode == http.StatusNotModified {
_ = fetchResult.Content.Close()
p.log.Info("upstream source not modified",
"request_id", middleware.GetReqID(ctx), "host", req.Source.Host,
"path", req.Source.Path, "fetch_ms", fetchResult.FetchDurationMs)
return nil, errSourceNotModified
}
// Read and validate the source content
sourceData, err := io.ReadAll(fetchResult.Content)
fetchBytes := int64(len(sourceData))
// Counted also when the request context has ended meanwhile
p.incrementUpstreamFetch(fetchBytes, time.Since(fetchStart))
if err != nil {
p.upstreamFetchErrors.Inc()
_ = fetchResult.Content.Close()
return nil, fmt.Errorf("failed to read upstream response: %w", err)
}
p.logUpstreamFetch(ctx, req, fetchResult, fetchBytes)
// The source's format, detected from its first bytes, which must match
// its content type. A body that does not is the upstream's error, so it
// gets a negative cache entry, as an upstream error status does.
sourceFormat, err := magic.ValidateMagicBytes(sourceData,
fetchResult.ContentType)
if err != nil {
p.log.Warn("source refused", "request_id", middleware.GetReqID(ctx),
"host", req.Source.Host, "path", req.Source.Path,
"content_type", fetchResult.ContentType, "error", err)
p.storeNegative(ctx, req)
_ = fetchResult.Content.Close()
return nil, fmt.Errorf("content validation failed: %w", err)
}
info := fetchedSourceInfo(fetchResult)
// Store source content
_, err = p.cache.PutSource(ctx, req.Source, bytes.NewReader(sourceData), info)
if err != nil {
p.log.Warn("failed to store source content", "error", err)
// Continue even if caching fails
}
return &fetchedSource{
data: sourceData,
format: sourceFormat,
info: info,
response: fetchResult.Content,
}, nil
}
// refuseNegativeCached returns an error wrapping ErrNegativeCached while
// req's source URL is in the negative cache, and nil otherwise. A failure to
// read the negative cache is only logged, and returns nil.
func (p *Proxy) refuseNegativeCached(
ctx context.Context, req *imagerequest.Request,
) error {
negHit, err := p.cache.Failure(ctx, req.Source)
if err != nil {
p.log.Warn("negative cache check failed", "error", err)
}
if !negHit {
return nil
}
p.log.Debug("negative cache hit",
"host", req.Source.Host,
"path", req.Source.Path,
)
return fmt.Errorf("%w: %w", httpfetcher.ErrUpstreamError, ErrNegativeCached)
}
// fetchedSourceInfo returns what the cache keeps about a source upstream
// just sent: fetched and checked now, with the ETag and Last-Modified
// headers of fetchResult.
func fetchedSourceInfo(fetchResult *httpfetcher.FetchResult) imgcache.SourceInfo {
now := time.Now().UTC()
upstreamHeaders := http.Header(fetchResult.Headers)
return imgcache.SourceInfo{
FetchedAt: now,
CheckedAt: now,
ETag: upstreamHeaders.Get("ETag"),
LastModified: upstreamHeaders.Get("Last-Modified"),
}
}
// logUpstreamFetch logs a fetch of req's source that read fetchBytes bytes,
// with its download rate and connection details.
func (p *Proxy) logUpstreamFetch(
ctx context.Context,
req *imagerequest.Request,
fetchResult *httpfetcher.FetchResult,
fetchBytes int64,
) {
var downloadRate string
if fetchResult.FetchDurationMs > 0 {
seconds := float64(fetchResult.FetchDurationMs) / 1000.0 //nolint:mnd // ms to seconds
bitsPerSecond := float64(fetchBytes*8) / seconds //nolint:mnd // bytes to bits
downloadRate = humanize.SI(bitsPerSecond, "bps")
}
p.log.Info("upstream fetched",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host,
"path", req.Source.Path,
"bytes", fetchBytes,
"fetch_ms", fetchResult.FetchDurationMs,
"rate", downloadRate,
"remote_addr", fetchResult.RemoteAddr,
"http", fetchResult.HTTPVersion,
"tls", fetchResult.TLSVersion,
"cipher", fetchResult.TLSCipherSuite,
)
}
// incrementUpstreamFetch counts one upstream fetch that read fetchBytes bytes
// and took duration. A fetch that read no bytes is not counted.
func (p *Proxy) incrementUpstreamFetch(fetchBytes int64, duration time.Duration) {
if fetchBytes <= 0 {
return
}
p.upstreamFetches.Inc()
p.upstreamFetchBytes.Add(float64(fetchBytes))
p.upstreamFetchDurations.Observe(duration.Seconds())
}
// storeNegative stores a negative cache entry for req's source, whose fetch
// failed, so the source is not fetched again until the entry expires. A
// failure to store the entry is only logged.
func (p *Proxy) storeNegative(ctx context.Context, req *imagerequest.Request) {
err := p.cache.PutFailure(ctx, req.Source)
if err != nil {
p.log.Warn("failed to store negative cache entry",
"host", req.Source.Host, "path", req.Source.Path, "error", err)
}
}
// processedVariant is a variant as processAndStore made it. Each request that
// shared its processing serves it through a reader of its own.
type processedVariant struct {
data []byte
contentType string
fetchedBytes int64
// checkedAt is its source's check time when the variant was made or last
// renewed, which its cached file has as its modification time
checkedAt time.Time
}
// processAndStore processes the image read from source, which is in
// sourceFormat, and stores the result. sourceInfo describes the source's
// bytes, and is stored with the variant (see imgcache.Cache.PutVariant).
func (p *Proxy) processAndStore(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
sourceInfo imgcache.SourceInfo,
source io.Reader,
sourceFormat imagerequest.Format,
fetchBytes int64,
) (*processedVariant, error) {
// Process the image
processStart := time.Now()
// ctx has the deadline of the request doing the processing, which can
// end while a request waiting for its image has time left
processResult, err := p.transform(ctx, source, sourceFormat, &req.Transform)
if errors.Is(err, context.DeadlineExceeded) {
return nil, fmt.Errorf("%w: %w", ErrProcessingTimedOut, err)
}
if err != nil {
return nil, fmt.Errorf("image processing failed: %w", err)
}
processDuration := time.Since(processStart)
// Counted also when the request context has ended meanwhile
p.transforms.Inc()
p.transformDurations.Observe(processDuration.Seconds())
// Read processed content
processedData, err := io.ReadAll(processResult.Content)
_ = processResult.Content.Close()
if err != nil {
return nil, fmt.Errorf("failed to read processed content: %w", err)
}
// Log conversion details
outputSize := int64(len(processedData))
var sizePercent float64
if fetchBytes > 0 {
//nolint:mnd // percentage calculation
sizePercent = float64(outputSize) / float64(fetchBytes) * 100.0
}
p.log.Info("image converted",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host,
"path", req.Source.Path,
"src_format", sourceFormat,
"dst_format", req.Transform.Format,
"src_bytes", fetchBytes,
"dst_bytes", outputSize,
"src_dimensions", fmt.Sprintf("%dx%d",
processResult.InputWidth, processResult.InputHeight),
"dst_dimensions", fmt.Sprintf("%dx%d",
processResult.Width, processResult.Height),
"size_ratio", fmt.Sprintf("%.1f%%", sizePercent),
"convert_ms", processDuration.Milliseconds(),
"quality", req.Transform.Quality,
"fit", req.Transform.FitMode,
)
// Store variant to cache
err = p.cache.PutVariant(ctx, cacheKey, sourceInfo,
bytes.NewReader(processedData))
if err != nil {
p.log.Warn("failed to store variant", "error", err)
// Continue even if caching fails
}
return &processedVariant{
data: processedData,
contentType: processResult.ContentType,
fetchedBytes: fetchBytes,
checkedAt: sourceInfo.CheckedAt,
}, nil
}
// transform has the image processor transform the image read from source
// as requested says, and counts it in pixa_transforms_in_progress until the
// processor returns or panics.
func (p *Proxy) transform(
ctx context.Context,
source io.Reader,
sourceFormat imagerequest.Format,
requested *imagerequest.Transform,
) (*imageprocessor.Result, error) {
p.transformsInProgress.Inc()
defer p.transformsInProgress.Dec()
return p.processor.Process(ctx, source, sourceFormat, requested)
}
// isNegativeCacheable returns true if the error should be cached.
func isNegativeCacheable(err error) bool {
return errors.Is(err, httpfetcher.ErrUpstreamError)
}
// etagHashLength is the number of hash characters to use for ETags.
const etagHashLength = 16
// formatETag returns the quoted ETag of the variant of key: the first
// etagHashLength characters of key, then checkedAt in Unix nanoseconds.
// checkedAt is its source's check time when the variant was made or last
// renewed, so a variant made again from new source bytes gets another ETag.
func formatETag(key imagerequest.VariantKey, checkedAt time.Time) string {
hash := string(key)
// Use first 16 characters of hash for a shorter but still unique ETag
if len(hash) > etagHashLength {
hash = hash[:etagHashLength]
}
return fmt.Sprintf(`"%s-%d"`, hash, checkedAt.UnixNano())
}
internal/imageproxy/imageproxy_internal_test.go
+482
View File
@@ -0,0 +1,482 @@
package imageproxy
import (
"context"
"io"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic"
)
// Test data literals used repeatedly in this file (goconst).
const (
testPathPhoto = "/images/photo.jpg"
testPathUpload = "/uploads/image.jpg"
)
func TestProxy_Get_GoodHost(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
resp, err := svc.Get(ctx, req)
if err != nil {
t.Fatalf("Get() error = %v", err)
}
defer func() { _ = resp.Content.Close() }()
// Verify we got content
data, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("failed to read response: %v", err)
}
if len(data) == 0 {
t.Error("expected non-empty response")
}
if resp.ContentType != testContentTypeJPEG {
t.Errorf("ContentType = %q, want %q", resp.ContentType, testContentTypeJPEG)
}
}
// TestProxy_Get_OtherHost checks that the proxy serves a second host as it
// serves GoodHost.
func TestProxy_Get_OtherHost(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.OtherHost,
Path: testPathUpload,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Should fetch successfully
resp, err := svc.Get(ctx, req)
if err != nil {
t.Fatalf("Get() error = %v", err)
}
defer func() { _ = resp.Content.Close() }()
data, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("failed to read response: %v", err)
}
if len(data) == 0 {
t.Error("expected non-empty response")
}
}
func TestProxy_Get_InvalidFile(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := photoVariant(fixtures, 85, imagerequest.FitCover)
req.Source.Path = "/images/fake.jpg"
req.Transform.Size = imagerequest.Size{Width: 50, Height: 50}
// Should fail because magic bytes don't match
_, err := svc.Get(ctx, req)
if err == nil {
t.Error("Get() expected error for invalid file")
}
}
func TestProxy_Get_NotFound(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := photoVariant(fixtures, 85, imagerequest.FitCover)
req.Source.Path = "/images/nonexistent.jpg"
req.Transform.Size = imagerequest.Size{Width: 50, Height: 50}
_, err := svc.Get(ctx, req)
if err == nil {
t.Error("Get() expected error for nonexistent file")
}
}
func TestProxy_Get_FormatConversion(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
tests := []struct {
name string
sourcePath string
outFormat imagerequest.Format
wantMIME string
}{
{
name: "JPEG to PNG",
sourcePath: testPathPhoto,
outFormat: imagerequest.FormatPNG,
wantMIME: "image/png",
},
{
name: "PNG to JPEG",
sourcePath: "/images/logo.png",
outFormat: imagerequest.FormatJPEG,
wantMIME: testContentTypeJPEG,
},
{
name: "GIF to PNG",
sourcePath: "/images/animation.gif",
outFormat: imagerequest.FormatPNG,
wantMIME: "image/png",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
req := &imagerequest.Request{
Source: imagerequest.Source{Host: fixtures.GoodHost, Path: tt.sourcePath},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: tt.outFormat, Quality: 85, FitMode: imagerequest.FitCover,
},
}
resp, err := svc.Get(ctx, req)
if err != nil {
t.Fatalf("Get() error = %v", err)
}
defer func() { _ = resp.Content.Close() }()
if resp.ContentType != tt.wantMIME {
t.Errorf("ContentType = %q, want %q", resp.ContentType, tt.wantMIME)
}
// Verify magic bytes match the expected format
data, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("failed to read response: %v", err)
}
detectedFormat, err := magic.DetectFormat(data)
if err != nil {
t.Fatalf("failed to detect format: %v", err)
}
if detectedFormat.MIMEType() != tt.wantMIME {
t.Errorf("detected format = %q, want the format of %s",
detectedFormat, tt.wantMIME)
}
})
}
}
func TestProxy_Get_Caching(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// First request - should be a cache miss
resp1, err := svc.Get(ctx, req)
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
if resp1.CacheStatus != CacheMiss {
t.Errorf("first request CacheStatus = %q, want %q", resp1.CacheStatus, CacheMiss)
}
data1, err := io.ReadAll(resp1.Content)
if err != nil {
t.Fatalf("failed to read first response: %v", err)
}
_ = resp1.Content.Close()
// Second request - should be a cache hit
resp2, err := svc.Get(ctx, req)
if err != nil {
t.Fatalf("Get() second request error = %v", err)
}
if resp2.CacheStatus != CacheHit {
t.Errorf("second request CacheStatus = %q, want %q", resp2.CacheStatus, CacheHit)
}
data2, err := io.ReadAll(resp2.Content)
if err != nil {
t.Fatalf("failed to read second response: %v", err)
}
_ = resp2.Content.Close()
// Content should be identical
if len(data1) != len(data2) {
t.Errorf("response sizes differ: %d vs %d", len(data1), len(data2))
}
}
func TestProxy_Get_DifferentSizes(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
// Request same image at different sizes
sizes := []imagerequest.Size{
{Width: 25, Height: 25},
{Width: 50, Height: 50},
{Width: 75, Height: 75},
}
responses := make([][]byte, 0, len(sizes))
for _, size := range sizes {
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: size,
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
resp, err := svc.Get(ctx, req)
if err != nil {
t.Fatalf("Get() error for size %v = %v", size, err)
}
data, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("failed to read response: %v", err)
}
_ = resp.Content.Close()
responses = append(responses, data)
}
// All responses should be different sizes (different cache entries)
for i := range len(responses) - 1 {
if len(responses[i]) == len(responses[i+1]) {
// Not necessarily an error, but worth noting
t.Logf("responses %d and %d have same size: %d bytes",
i, i+1, len(responses[i]))
}
}
}
func TestProxy_Get_ContextCancellation(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
ctx, cancel := context.WithCancel(context.Background())
cancel() // Cancel immediately
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
_, err := svc.Get(ctx, req)
if err == nil {
t.Error("Get() expected error for cancelled context")
}
}
func TestProxy_Get_ReturnsETag(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
resp, err := svc.Get(ctx, req)
if err != nil {
t.Fatalf("Get() error = %v", err)
}
defer func() { _ = resp.Content.Close() }()
// ETag should be set
if resp.ETag == "" {
t.Error("ETag should be set in response")
}
// ETag should be properly quoted
if len(resp.ETag) < 2 || resp.ETag[0] != '"' || resp.ETag[len(resp.ETag)-1] != '"' {
t.Errorf("ETag should be quoted, got %q", resp.ETag)
}
}
func TestProxy_Get_ETagConsistency(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// First request
resp1, err := svc.Get(ctx, req)
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
etag1 := resp1.ETag
_ = resp1.Content.Close()
// Second request (from cache)
resp2, err := svc.Get(ctx, req)
if err != nil {
t.Fatalf("Get() second request error = %v", err)
}
etag2 := resp2.ETag
_ = resp2.Content.Close()
// ETags should be identical for the same content
if etag1 != etag2 {
t.Errorf("ETags should match: %q != %q", etag1, etag2)
}
}
func TestProxy_Get_DifferentETagsForDifferentContent(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
// Request same image at different sizes - should get different ETags
req1 := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 25, Height: 25},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
req2 := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
resp1, err := svc.Get(ctx, req1)
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
etag1 := resp1.ETag
_ = resp1.Content.Close()
resp2, err := svc.Get(ctx, req2)
if err != nil {
t.Fatalf("Get() second request error = %v", err)
}
etag2 := resp2.ETag
_ = resp2.Content.Close()
// ETags should be different for different content
if etag1 == etag2 {
t.Error("ETags should differ for different sizes")
}
}
internal/imgcache/max_concurrent_processing_internal_test.go → internal/imageproxy/max_concurrent_processing_internal_test.go
+33 -24
View File
@@ -1,4 +1,4 @@
package imgcache package imageproxy
import ( import (
"image/color" "image/color"
@@ -9,18 +9,23 @@ import (
"time" "time"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// widthOnlyRequest asks for the test photo at width, its height scaled to // widthOnlyRequest asks for the test photo at width, its height scaled to
// keep the photo's aspect ratio. // keep the photo's aspect ratio.
func widthOnlyRequest(fixtures *TestFixtures, width int) *ImageRequest { func widthOnlyRequest(fixtures *TestFixtures, width int) *imagerequest.Request {
return &ImageRequest{ return &imagerequest.Request{
SourceHost: fixtures.GoodHost, Source: imagerequest.Source{
SourcePath: testPathPhoto, Host: fixtures.GoodHost,
Size: Size{Width: width}, Path: testPathPhoto,
Format: FormatJPEG, },
Quality: 85, Transform: imagerequest.Transform{
FitMode: FitCover, Size: imagerequest.Size{Width: width},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
} }
} }
@@ -36,7 +41,8 @@ func holdProcessingSlot(
input, feed := io.Pipe() input, feed := io.Pipe()
go func() { go func() {
_, _ = proc.Process(t.Context(), input, &imageprocessor.Request{}) _, _ = proc.Process(t.Context(), input, imagerequest.FormatJPEG,
&imagerequest.Transform{})
}() }()
_, err := feed.Write([]byte{0}) _, err := feed.Write([]byte{0})
@@ -50,18 +56,19 @@ func holdProcessingSlot(
return release return release
} }
// TestService_Get_WaitsForSlotBeforeReadingCachedSource checks that a // TestProxy_Get_WaitsForSlotBeforeReadingCachedSource checks that a
// request whose source is cached holds none of it while it waits for a // request whose source is cached holds none of it while it waits for a
// processing slot: it reads the cached file only once it has a slot. With // processing slot: past the first bytes, which give the source's format, it
// the only slot held, a request for a new width of the cached 100x100 photo // reads the cached file only once it has a slot. With the only slot held, a
// waits; the cached file is then rewritten as a 100x50 image before the slot // request for a new width of the cached 100x100 photo waits; the cached file
// is freed, so the request must answer with that image scaled to 40x20. // is then rewritten as a 100x50 image before the slot is freed, so the
func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) { // request must answer with that image scaled to 40x20.
func TestProxy_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
svc.processor = imageprocessor.New( svc.processor = imageprocessor.New(
imageprocessor.Params{MaxConcurrentProcessing: 1}, imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1},
) )
// A first request caches the photo as a source. // A first request caches the photo as a source.
@@ -72,13 +79,15 @@ func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
_ = resp.Content.Close() _ = resp.Content.Close()
contentHash, _, err := svc.cache.LookupSource(t.Context(), cachedSource := widthOnlyRequest(fixtures, 50).Source
widthOnlyRequest(fixtures, 50))
if err != nil || contentHash == "" { reader, _, _, err := svc.cache.Source(t.Context(), cachedSource)
t.Fatalf("LookupSource() = %q, %v; want the cached source", if err != nil {
contentHash, err) t.Fatalf("Source() error = %v; want the cached source", err)
} }
_ = reader.Close()
release := holdProcessingSlot(t, svc.processor) release := holdProcessingSlot(t, svc.processor)
var ( var (
@@ -98,7 +107,7 @@ func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
// before waiting, it would have read it by now. // before waiting, it would have read it by now.
time.Sleep(100 * time.Millisecond) time.Sleep(100 * time.Millisecond)
err = os.WriteFile(svc.cache.srcContent.hashToPath(contentHash), err = os.WriteFile(sourceFilePath(svc, cachedSource.Key()),
generateTestJPEG(t, 100, 50, color.RGBA{0, 0, 255, 255}), 0o600) generateTestJPEG(t, 100, 50, color.RGBA{0, 0, 255, 255}), 0o600)
if err != nil { if err != nil {
t.Fatalf("failed to rewrite the cached source: %v", err) t.Fatalf("failed to rewrite the cached source: %v", err)
Loaded 100 of 190 files, more files were not shown because too many files have changed in this diff. Show more