sneak/pixa
1
0
Fork 0
Code Issues 7 Pull Requests 3 Actions 18 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/pixa:cb3439da3d1cf25aad6d05ea6b0b80db67430393
Branches Tags
sneak/pixa:main sneak/pixa:next sneak/pixa:issue-327-readme-rewrite sneak/pixa:prod
sneak/pixa:v1.0.0-rc3 sneak/pixa:v1.0.0-rc2 sneak/pixa:v1.0.0-rc1
..
compare: sneak/pixa:next
Branches Tags
sneak/pixa:next sneak/pixa:issue-327-readme-rewrite sneak/pixa:main sneak/pixa:prod
sneak/pixa:v1.0.0-rc3 sneak/pixa:v1.0.0-rc2 sneak/pixa:v1.0.0-rc1
15 Commits
cb3439da3d .. next
Author SHA1 Message Date
clawbot 9f1e9f5253 Rewrite AGENTS.md in short plain sentences (#327)
check / check (push) Waiting to run
Details 
Every rule keeps its meaning and emphasis. Sentences are split so each
holds one idea, with the main rule before its exception and every
command, file name and identifier in backticks.

Three statements were false against the repo or the standing rulings
on record and are corrected:

- the format check is `make fmt-check`, not `make check-fmt`
- a feature branch reaches `next` through a PR that passes review, is
  merged only when the feature is completely finished and the code is
  up to standards, and is deleted after the merge; only the user
  merges `next` into `main`
- the naming example uses the code's term: `al` is bad,
  `SourceHostAllowlist` is good

The `Last Updated` line takes today's date.

Model: fable-5-1
 2026-10-27 05:06:36 +01:00
clawbot 2c90144e7b Go on past cache entries the reconciliation pass cannot read (closes #343)
check / check (push) Canceled after 0s
Details 
The walk of each storage directory returned the error of a directory it
could not list, or of a source or variant file whose info it could not
read. The pass then stopped before its later checks, every time it ran.
Now reconcileFiles skips a directory it cannot list and a file whose
info it cannot read, temp files included. It logs a warning naming the
path and the error, and the walk goes on. It reads each file's info
once and hands it to sweepStaleTempFile, reconcileVariantFile or
reconcileSourceFile. A file renamed or removed after its directory was
listed is skipped with no warning. A cancelled context still stops the
walk with its error. README.md says so under Storage.

Model: opus-5-5
 2026-10-27 01:13:55 +00:00
clawbot 26b5160aa8 Test that reconciliation goes on past entries it cannot read 
Each storage directory gets a directory the pass cannot list and a file
whose info it cannot read. Their paths are longer than Linux accepts,
even for root. The test wants a warning naming each path and the error.
It also wants the rest of the pass done: a variant file with no row
adopted, the row of a missing variant file dropped, a source file with
no row removed, and a wrong usage total corrected. A second test puts a
temp file whose info cannot be read in each storage directory and wants
the same warning. On next the first test's pass stops at the first such
directory and returns its error. The second test gets no warning.

Model: opus-5-5
 2026-10-27 01:12:34 +00:00
clawbot 33045a24b5 Keep a variant's accounting row when its file cannot be read (closes #340)
check / check (push) Canceled after 0s
Details 
The reconciliation pass dropped a variant's accounting row on any error
reading its file's size, and logged every such file as missing. A file
still on disk then had no row, so eviction no longer counted it against
cache_max_bytes. Now only a missing file (ErrNotFound, which
FileStore.Size returns for it) loses its row, with the same info line.
Any other error keeps the row and logs a warning naming the variant key
and the error, and the pass goes on to the next key. README.md says so
under Storage.

Model: opus-5-5
 2026-10-26 23:56:34 +00:00
clawbot 701ed9b94b Test that reconciliation drops a variant's row only when its file is missing
check / check (push) Canceled after 0s
Details 
reconcileVariantRows is given two tracked variants. The first one's
file is a symbolic link to itself, so reading its size fails with an
error other than a missing file. The test wants its row kept, with a
warning naming its key and the error. The second one has no file, and
the test wants its row dropped with the info line for a missing file.
The test fails on next, which drops both rows and logs both files as
missing. The fix follows in the next commit.

Model: opus-5-5
 2026-10-26 23:54:24 +00:00
clawbot 233765033a Answer 499 to a client that goes away while waiting for an image (closes #339)
check / check (push) Canceled after 0s
Details 
When a request waits for an image another request is making and its
client goes away, the image proxy returns context.Canceled. serveImage
now answers it 499, the status nginx calls "client closed request". It
logs it once at info level with the request_id, host and path. It was
answered 500 and logged at error level, as pixa's own failure. A
cancellation while the request's context is live is still answered 500.
The request making the image is still answered as before. README.md
lists the 499 on both image routes.

Model: opus-5-5
 2026-10-26 23:37:35 +00:00
clawbot d33316caf3 Test that a client leaving while another request makes its image gets 499 
On both image routes, a request waits for an image another request is
making, and its client goes away. The first test wants that request
answered 499 and logged once at info level. It fails on next, which
answers 500 and logs it at error level. The second test pins the 500
that a cancellation keeps while the request's context is live; it
passes on next. The fix follows in the next commit.

Model: opus-5-5
 2026-10-26 23:26:07 +00:00
clawbot 6de69f7349 Name pixa's concepts at the start of the README's Design section (closes #321)
check / check (push) Canceled after 0s
Details 
The Design section starts with a Concepts list. Each concept gets one
sentence, under the name the code gives it. Storage calls the
transformed images variants, and the sentences it changed for that are
shorter. Its paths end in the source key and the variant key, and it
lists the image cache's tables in the database. Architecture names
internal/providers and lists the packages that work with the concepts.
No code or test changes.

Model: opus-5-5
 2026-10-26 23:19:19 +01:00
clawbot 93b13d6638 Load every setting from one table in internal/config (closes #320)
check / check (push) Canceled after 0s
Details 
Config.settings lists each setting once: its key, its environment variable
and the function that loads it. Most rows load through field, whose
arguments give the type, the default and the check. db_url and
cache_max_bytes keep their own load functions. Loading, the unknown-key
and unknown-variable checks and the names in error messages all read the
table. allowlist_hosts now loads through parseHostList.
isKnownConfigKey, envVarNames, strictLoader, validate,
validateAllowlistHostsValue and getStringSlice are gone, with the
getStringSlice tests. The key constants move to the tests, their only
users. README.md lists the settings one item each instead of in a table,
with shorter sentences above the list. A new test checks that this list
and configs/config.example.yml name exactly the keys of the table.

Model: opus-5-5
 2026-10-26 22:32:03 +01:00
clawbot b73bbde46e Detect a fetched image's format from its bytes once (closes #319)
check / check (push) Canceled after 0s
Details 
magic.DetectFormat returns an imagerequest.Format. ValidateMagicBytes
returns the format it checked against the upstream Content-Type. The
cache reads a cached source's format from its file's first bytes, as it
reads a variant's. The proxy hands the format to Process. Process
refuses a format that is not a source format, and input libvips would
load with another loader, such as an SVG passing as AVIF.
Format.MIMEType declares each MIME string once, and the fetcher, the
cache's variant read and the Accept matching use it. magic.MIMEType,
magic.ImageFormat, MIMEToImageFormat, magic's SVG detection,
detectFormat, formatFromString, FormatToMIME and Result.InputFormat are
gone.

Model: opus-5-5
 2026-10-26 21:21:39 +01:00
clawbot ac850823f4 Keep cached sources and variants in one FileStore type (closes #318)
check / check (push) Canceled after 0s
Details 
imgcache.FileStore is a directory of files named by a hex key, at
<ab>/<cd>/<key>. It writes a file under a temp name, syncs it and renames
it into place. It opens, reports the size of and deletes the file for a
key. The cache keeps its sources in one FileStore and its variants in
another, so ContentStorage and VariantStorage are gone. FileStore has no
Exists: the reconciliation pass checks a variant's file with Size, and the
tests look for the file on disk. Cache.Variant reads the content type from
the file's first bytes itself. storage.go is now filestore.go, and the
unused testdata/red.avif is gone.

Model: opus-5-5
 2026-10-26 20:24:33 +01:00
clawbot 17376d885b Split internal/imgcache into the image proxy and the cache (closes #317)
check / check (push) Canceled after 0s
Details 
The new package internal/imageproxy holds Proxy, which was imgcache.Service,
with its Config, ImageResponse, CacheStatus and errors. imageproxy.New makes
and registers the counters of upstream fetches and transforms.
internal/imgcache holds the cache only, with its hit and miss counters, and
imgcache.CacheConfig is imgcache.Config. The cache reads and writes each kind
it keeps with one pair of methods: Variant and PutVariant, Source and
PutSource, Failure and PutFailure. Source replaces LookupSource followed by
GetSourceContent. imgcache.go and module.go are gone. The proxy's tests moved
with it. The cache's tests check that the reconciliation pass removes an empty
or cut file and its row. The proxy's tests remove them by hand, then check
that Get fetches or converts again.

Model: opus-5-5
 2026-10-26 20:00:25 +01:00
clawbot 87cb1ab5f6 Serve both image routes with one function and one error mapping (closes #316)
check / check (push) Canceled after 0s
Details 
HandleImage parses the path and verifies the signature, and HandleImageEnc
decrypts the token. Each then calls serveImage, which chooses the format for
auto, gets the image and serves it. respondImageError answers the image
service's errors on both routes, and handleImageError is gone. /v1/image/ now
answers 504 when the upstream sends no response headers within
upstream_fetch_timeout, as README.md documented for /v1/e/. respondImageError
logs a 500 at error level and the other answers at warn. The image service
alone logs a source refused for its first bytes. A failed write is logged once,
at warn, with the request and the bytes served. README.md lists each image
route's answers as bullets. Tests send the same errors through both routes.

Model: opus-5-5
 2026-10-26 15:24:24 +01:00
clawbot 77d55abb7e Derive every key from the signing key in one place (closes #315)
check / check (push) Canceled after 0s
Details 
seal.DeriveKeys derives the session hash and block keys, the encrypted URL
key and the CSRF key from the signing key, with their salts declared beside
it, and returns them as one seal.Keys. internal/providers calls it once at
startup and provides the value to the fx graph. The session manager, the
encrypted URL generator and the CSRF protection each take their own key.
session.NewManager, encurl.NewGenerator, newCSRFProtect and handlers.New
return no error, as nothing in them can fail now. Each key is derived as
before, with seal.DeriveKey and the same salt. The signing key itself stays
the HMAC key of signed URLs and the login password. README.md says under
signing_key what it protects.

Model: opus-5-5
 2026-10-26 14:10:20 +01:00
clawbot 7b6712538e Take the signature, the expiry and the URL scheme off the image request (closes #314)
check / check (push) Canceled after 0s
Details 
The image request now holds only its source and its transform, exactly what
the variant key covers. The /v1/image/ handler checks the allowlist and
verifies the signature itself with Signer.Verify(req, sig, expires). Both
image handlers keep the URL's expiry as a local value for Cache-Control. The
image service holds no signer, allowlist or AllowHTTP, and ValidateRequest is
gone. The fetcher takes the source and builds its URL with Source.URL from its
own AllowHTTP, so no URL string is built and parsed again. chooseAutoFormat
returns the format it chooses instead of writing into the request. The signed
URL format and every route's answers are unchanged.

Model: opus-5-5
 2026-10-26 13:49:12 +01:00
99 changed files with 4525 additions and 3500 deletions
Expand all files Collapse all files
AGENTS.md
+52 -51
View File
@@ -1,75 +1,76 @@
# Repository Rules # Repository Rules
Last Updated 2026-01-08 Last Updated 2026-10-27
These rules MUST be followed at all times, it is very important. These rules MUST be followed at all times. This is very important.
- Never use `git add -A` - add specific changes to a deliberate commit. A commit - Never use `git add -A`. Stage specific changes for a deliberate commit. A
should contain one change. After each change, make a commit with a good commit holds one change. After each change, make a commit with a good one-line
one-line summary. summary.
- NEVER modify the linter config without asking first. - NEVER modify the linter config without asking first.
- NEVER modify tests to exclude special cases or otherwise get them to pass - NEVER modify a test to exclude special cases or otherwise make it pass without
without asking first. In almost all cases, the code should be changed, NOT the asking first. In almost all cases, change the code, NOT the test. You need
tests. If you think the test needs to be changed, make your case for that and explicit user approval to modify an existing test. If you think a test needs
ask for permission to proceed, then stop. You need explicit user approval to to change, make your case and ask for permission to proceed. Then stop. A bug
modify existing tests. A bug fix may change an existing test without asking fix may change an existing test without asking only when that test pins
when that test pins exactly the behavior the filed bug calls wrong, and the exactly the behavior the filed bug calls wrong. The change must then move the
change moves it to the fixed behavior and nothing else; any other change to an test to the fixed behavior and nothing else. Any other change to an existing
existing test still needs explicit user approval. (You do not need user test needs explicit user approval. Writing NEW tests needs no approval.
approval for writing NEW tests.)
- When linting, assume the linter config is CORRECT, and that each item output - When linting, assume the linter config is CORRECT. Treat each item the linter
by the linter is something that legitimately needs fixing in the code. reports as something that legitimately needs fixing in the code.
- When running tests, use `make test`. - When running tests, use `make test`.
- Before commits, run `make check`. This runs `make lint` and `make test` and - Before committing, run `make check`. It runs `make test`, `make lint` and
`make check-fmt`. Any issues discovered MUST be resolved before committing `make fmt-check`. Any issue it finds MUST be resolved before committing,
unless explicitly told otherwise. unless you are explicitly told otherwise.
- When fixing a bug, write a failing test for the bug FIRST. Add appropriate - When fixing a bug, write a failing test for the bug FIRST. Add appropriate
logging to the test to ensure it is written correctly. Commit that. Then go logging to the test, to make sure the test is written correctly. Commit that.
about fixing the bug until the test passes (without modifying the test Then work on the fix until the test passes, without changing the test further.
further). Then commit that. Then commit the fix.
- When adding a new feature, do the same - implement a test first (TDD). It - When adding a new feature, do the same: write a test first (test-driven
doesn't have to be super complex. Commit the test, then commit the feature. development, TDD). The test does not need to be very complex. Commit the test,
then commit the feature.
- When adding a new feature, use a feature branch. When the feature is - When adding a new feature, use a feature branch. The feature branch reaches
completely finished and the code is up to standards (passes `make check`) then `next` through a PR that passes review. It is merged only when the feature is
and only then can the feature branch be merged into `main` and the branch completely finished and the code is up to standards. The code is up to
deleted. standards when it passes `make check`. The branch is deleted after the merge.
Only the user merges `next` into `main`.
- Write godoc documentation comments for all exported types and functions as you - Write `godoc` comments for all exported types and functions as you go along.
go along.
- ALWAYS be consistent in naming. If you name something one thing in one place, - ALWAYS be consistent in naming. If you name something one way in one place,
name it the EXACT SAME THING in another place. use the EXACT SAME name in every other place.
- Be descriptive and specific in naming. `wl` is bad; `SourceHostWhitelist` is - Be descriptive and specific in naming. `al` is bad; `SourceHostAllowlist` is
good. `ConnsPerHost` is bad; `MaxConnectionsPerHost` is good. good. `ConnsPerHost` is bad; `MaxConnectionsPerHost` is good.
- This is not prototype or teaching code - this is designed for production. Any - This is not prototype or teaching code. It is designed for production. Any
security issues (such as denial of service) or other web vulnerabilities are security issue (such as denial of service) or other web vulnerability is a P1
P1 bugs and must be added to TODO.md at the top. bug. It must be added at the top of `TODO.md`.
- As this is production code, no stubbing of implementations unless specifically - Do not stub an implementation unless you are specifically instructed to. This
instructed. We need working implementations. is production code, and we need working implementations.
- NEVER silently fall back to a different setting when a user's parameter - NEVER silently fall back to a different setting when a user's parameter
explicitly specifies a value. If a user requests format=webp and WebP encoding explicitly specifies a value. For example, if a user requests `format=webp`
is not supported, return an error - do NOT silently output PNG instead. If a and WebP encoding is not supported, return an error. Do NOT silently output
user specifies fit=invalid and that fit mode doesn't exist, return an error - PNG instead. If a user specifies `fit=invalid` and that fit mode does not
do NOT silently default to "cover". Silent fallbacks violate the principle of exist, return an error. Do NOT silently default to `cover`. A silent fallback
least surprise and mask bugs. The only acceptable defaults are for OMITTED violates the principle of least surprise and masks bugs. Defaults are
parameters, never for INVALID explicit values. acceptable only for OMITTED parameters, never for INVALID explicit values.
- Avoid vendoring deps unless specifically instructed to. NEVER commit the - Avoid vendoring dependencies unless specifically instructed to. NEVER commit
vendor directory, NEVER commit compiled binaries. If these directories or the `vendor/` directory. NEVER commit compiled binaries. If such a directory
files exist, add them to .gitignore (and commit the .gitignore) if they are or file exists and `.gitignore` does not list it, add it to `.gitignore`.
not already in there. Keep the entire git repository (with history) small - Commit that `.gitignore` change. Keep the entire git repository, history
under 20MiB, unless you specifically must commit larger files (e.g. test included, under 20 MiB. Go over 20 MiB only when you specifically must commit
fixture example media files). Only OUR source code and immediately supporting a large file, such as a media file used as a test fixture. Only OUR source
files (such as test examples) goes into the repo/history. code and its immediately supporting files, such as test examples, go into the
repository and its history.
README.md
+245 -105
View File
@@ -133,7 +133,7 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
filesystem's free space and what the cache holds, and at least 500 MiB, so filesystem's free space and what the cache holds, and at least 500 MiB, so
the cache can grow to three quarters or more of the space they all have the cache can grow to three quarters or more of the space they all have
left, and to all of it when that is 500 MiB or less. left, and to all of it when that is 500 MiB or less.
- the rest are in the table under Configuration below - the rest are in the list under Configuration below
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for - **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
the images it processes at once, plus `upstream_connections` times the images it processes at once, plus `upstream_connections` times
`upstream_max_response_size` for the source images it has fetched that wait `upstream_max_response_size` for the source images it has fetched that wait
@@ -178,46 +178,100 @@ allowlisted source hosts for open access.
## Design ## Design
### Concepts
These terms are the names the code uses:
- **Source**: the image at one upstream URL, named by its host, path and query.
- **Source key**: the SHA-256 of a source's host, path and query, which names
the source's file and its row in the database.
- **Transform**: the output size, format, quality and fit an image request asks
for; with the format `auto`, pixa chooses the output format from the `Accept`
header.
- **Image request**: a source and a transform, which each image route reads from
its URL.
- **Variant**: the image pixa serves for an image request, made from its source
with its transform.
- **Variant key**: the SHA-256 of a variant's source, size, format, quality and
fit, which names the variant's file and its row in the database, and gives its
`ETag`.
- **Signed URL**: a `/v1/image/` URL whose `sig` signs its image request and its
expiry `exp` with `signing_key`.
- **Encrypted URL**: a `/v1/e/` URL whose token holds its image request and any
expiry, encrypted with a key derived from `signing_key`.
- **Login session**: an encrypted cookie that keeps a browser logged in to the
generator page for 30 days, or until a logout replaces the random value pixa
keeps in the `logout` table (see Encrypted URLs).
- **Negative cache**: the source URLs pixa does not fetch again for 5 minutes,
after one of these failed fetches:
- the upstream answered with an error status
- the body's first bytes were not those of the format its `Content-Type`
names
- **Image cache**: the part of pixa that keeps the sources, the variants and the
negative cache (see Storage).
- **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached
sources and variants may reach together.
- **Image proxy**: the part of pixa that serves a variant from the image cache
when it can, and otherwise makes it from the source.
### Storage ### Storage
- **Source images**: - **Sources**: `<state_dir>/cache/sources/<ab>/<cd>/<source key>`, one file for
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of host, path and query>`, one each source URL
file for each source URL
- **Database**: `<state_dir>/state.sqlite3` (SQLite) - **Database**: `<state_dir>/state.sqlite3` (SQLite)
- **Transformed images**: - **Variants**: `<state_dir>/cache/variants/<ab>/<cd>/<variant key>`
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`
`<ab>` and `<cd>` are the first and second pairs of characters of the file's `<ab>` and `<cd>` are the first and second pairs of characters of the file's
name. name.
Each source image and each transformed image has one row in the database, The image cache keeps these tables in the database:
holding its size and when it was last used. Two source URLs that serve the same
bytes are stored twice, once under each URL. A source image is stored by writing
its file and then its row; when the row cannot be written, the file is removed
again.
Source and transformed image files are written under a temporary name, synced to - `source_content`: a row for each source, by source key, with its size and when
disk, and only then renamed into place, so a crash leaves no file under its own it was last used
name with part of its data. When pixa starts, and every five minutes after, it - `variant_content`: a row for each variant, by variant key (column
removes each cached source or transformed image whose file is empty, or not the `cache_key`), with its size and when it was last used
size the database records for it, with a warning naming the file; the next - `cache_usage`: the total size of the cached sources and variants, which
request for it fetches the source or converts the image again. Looking for those eviction reads
files reads the database once for every cached file, so pixa skips each - `negative_cache`: a row for each source URL in the negative cache, with when
five-minute look that would start less than ten times as long after the last it expires
look ended as that look took. After a look that took one minute, the next one
starts 10 to 15 minutes after that look ended. pixa reads a cached source image's format from its file's first bytes. When they
match none of the source formats, pixa fetches the source again. Two source URLs
that serve the same bytes are stored twice, once under each URL. A source image
is stored by writing its file and then its row; when the row cannot be written,
the file is removed again.
Each source and variant file is written under a temporary name and synced to
disk before it is renamed into place. So a crash never leaves part of a file
under its own name. When pixa starts, and every five minutes after, it removes
each source or variant whose file is empty, or not the size its row records. It
logs a warning naming the file. A later request for that image fetches the
source or makes the variant again. Looking for those files reads the database
once for every cached file, so pixa skips each five-minute look that would start
less than ten times as long after the last look ended as that look took. After a
look that took one minute, the next one starts 10 to 15 minutes after that look
ended.
Each look also drops the rows of variants whose files are missing. When pixa
cannot read a variant file's size for another reason, it keeps the variant's
row. It logs a warning naming the variant key and the error.
During a look, pixa skips a directory it cannot list and a file whose size it
cannot read. It logs a warning naming the path and the error, and goes on with
the rest of the look. A file renamed or removed after pixa listed its directory
gets no warning.
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
Test). Toward it, a cache hit opens the transformed image's file and nothing Test). Toward it, a cache hit opens the variant's file and nothing else, and
else, and makes no database call. The content type comes from the file's first makes no database call. The content type comes from the file's first bytes,
bytes, which are those of the format pixa wrote it in; a file whose first bytes which are those of the format pixa wrote it in; a file whose first bytes are not
are not those of a format pixa writes is not served, and the image is converted those of a format pixa writes is not served, and the image is converted again. A
again. A hit hands the time it was served to a background writer and does not hit hands the time it was served to a background writer and does not wait for
wait for the writer. Eviction orders by that time. The hit is counted in memory. the writer. Eviction orders by that time. The hit is counted in memory. A miss
A miss that uses a cached source image also hands that writer the time it used that uses a cached source image also hands that writer the time it used the
the image. Once a second that writer writes the waiting times in one image. Once a second that writer writes the waiting times in one transaction. Up
transaction. While 10,000 times of transformed images, or 10,000 of source to 10,000 access times of variants, and as many of sources, can wait to be
images, wait to be written, a further one is dropped, and that image only looks written. A time handed over beyond that is dropped. Its image then only looks
less recently used to eviction. `script/bench` measures how many hits pixa less recently used to eviction. `script/bench` measures how many hits pixa
serves per second (see Load Test). serves per second (see Load Test).
@@ -248,32 +302,61 @@ path under `/v1/` answers 200, in maintenance mode too.
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or - `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or
`/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized `/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized
and converted (below). Needs: a signature, unless the host is allowlisted (see and converted (below). Needs: a signature, unless the host is allowlisted (see
Source Hosts). Answers: 200; 304 when `If-None-Match` matches the image's Source Hosts). Answers:
`ETag`; 400 for a URL or parameter that is not valid, or for the format `auto` - 200 with the image.
an `Accept` header that is not valid; 406 for the format `auto` when `Accept` - 304 when `If-None-Match` matches the image's `ETag`.
allows none of the formats it chooses from; 401 for a missing or wrong - 400 for a URL or parameter that is not valid.
signature, a missing `exp` or an `exp` in the past; 403 when the request's - 400 when the format is `auto` and the `Accept` header is not valid.
`Referer` names a host in `referer_blocklist`, checked before the signature, - 406 when the format is `auto` and `Accept` allows none of the formats it
the cache and the upstream fetch; 403 when the upstream host, or a host it chooses from.
redirects to, is `localhost`, ends in `.localhost` or `.local`, or has an - 401 for a missing or wrong signature.
address in a blocked network (see `blocked_networks`); 429 over - 401 for a missing `exp`.
`image_requests_per_minute` (below); 415 when the upstream answers with a - 401 for an `exp` in the past.
`Content-Type` that is not one of the source formats (below), with a body - 403 when the request's `Referer` names a host in `referer_blocklist`,
whose first bytes are not those of the format its `Content-Type` names, or checked before the signature, the cache and the upstream fetch.
with an image whose first bytes libvips reads as another format; 502 when the - 403 when the upstream host, or a host it redirects to, is `localhost` or
upstream answered with an error status, and for 5 minutes after that, or after ends in `.localhost` or `.local`.
a body whose first bytes are not those of its `Content-Type`, for an image of - 403 when the upstream host, or a host it redirects to, has an address in a
the same source URL that pixa can neither serve from its cache nor make from blocked network (see `blocked_networks`).
the source image still in its cache; 503 when pixa is busy or in maintenance - 429 over `image_requests_per_minute` (below).
mode; 500 for any other failure. - 499, with no body, when the client goes away before its image is ready,
unless its request is the one making the image (below).
- 415 when the upstream answers with a `Content-Type` that is not one of the
source formats (below).
- 415 when the upstream sends a body whose first bytes are not those of the
format its `Content-Type` names.
- 415 when the upstream sends an image whose first bytes libvips reads as
another format.
- 502 when the upstream answers with an error status.
- 502 for 5 minutes after that for the same source URL, when pixa has
neither the image nor its source image in its cache.
- 502 likewise for 5 minutes after a body whose first bytes are not those of
its `Content-Type`.
- 503 when pixa is busy.
- 503 when pixa is in maintenance mode.
- 504 when the upstream has not sent its response headers within
`upstream_fetch_timeout`.
- 500 when that time runs out while the image itself is still arriving.
- 500 for any other failure.
- `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL - `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL
(see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when (see Encrypted URLs). Needs: nothing but the URL. Answers:
`If-None-Match` matches the image's `ETag`; 400 for a token that does not - 200 with the image.
decrypt, or that asks for a size or fit that is not valid; 410 once it has - 304 when `If-None-Match` matches the image's `ETag`.
expired; 504 when the upstream has not sent its response headers within - 400 for a token that does not decrypt.
`upstream_fetch_timeout`, but 500 when that time runs out while the image - 400 for a token that asks for a size or fit that is not valid.
itself is still arriving; 400 for an `Accept` header that is not valid, and - 410 once the token has expired.
406, 403, 415, 429, 502, 503 and 500, as for `/v1/image/`. - 400 when the token's format is `auto` and the `Accept` header is not
valid.
- These answers, for the same causes as on `/v1/image/`:
- 406
- 403
- 415
- 429
- 499, with no body
- 502
- 503
- 504
- 500
- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs: - `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs:
nothing. Answers: 200. nothing. Answers: 200.
- `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`, - `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`,
@@ -651,54 +734,85 @@ refused with 400.
Every setting can be given as an environment variable, in a YAML config file Every setting can be given as an environment variable, in a YAML config file
(`--config`), or both. A variable present in the environment wins over the file, (`--config`), or both. A variable present in the environment wins over the file,
even when it is empty, and the file wins over the built-in default. The one even when it is empty, and the file wins over the built-in default. The one
exception is a variable named in the file's `env:` section: it is set while the exception is a variable set in the file's `env:` section. pixa sets that
file loads, so it overrides both the environment the process was started with variable while loading the file, so the variable wins over the environment pixa
and the file's own key. A variable's value is parsed as the same text in the started with. It also wins over the file's key for the same setting. A
file would be. The three lists take comma-separated entries, with the spaces variable's value is parsed as the same text in the file would be. A setting that
around each trimmed; an empty variable is an empty list. A value that does not holds a list takes comma-separated entries, with the spaces around each trimmed.
parse or is invalid aborts startup, naming the variable. A variable whose name An empty variable is an empty list. A value that does not parse or is invalid
starts with `PIXA_` but is not in the table below, such as a misspelled one or aborts startup, naming the variable. pixa refuses to start when a variable
`PIXA_PORT`, aborts startup naming it, as an unknown config key does. The one starts with `PIXA_` but is not in the list below, such as a misspelled one or
other accepted name is `PIXA_CONFIG_PATH`, the config file's path (like `PIXA_PORT`. The error names the variable, as it names an unknown config key.
`--config`). The variables set by the file's `env:` section are checked the same pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like
`--config`. The variables set by the file's `env:` section are checked the same
way. way.
pixa reads at most one config file: the one given with `--config` (or `-c`), pixa reads at most one config file. It reads the file given with `--config` (or
otherwise the one `PIXA_CONFIG_PATH` names, otherwise the first of these that `-c`), or else the one `PIXA_CONFIG_PATH` names. pixa refuses to start when that
pixa finds: `/etc/pixa/config.yml`, `/etc/pixa/config.yaml`, file does not exist, cannot be read or does not parse. Without `--config` or
`~/.config/pixa/config.yml`, `~/.config/pixa/config.yaml`, then `config.yml` and `PIXA_CONFIG_PATH`, pixa looks in `/etc/pixa/`, then in `~/.config/pixa/`, then
`config.yaml` in the working directory. A named file that does not exist, cannot in the working directory. In each directory it looks for `config.yml`, then
be read or does not parse aborts startup. Of the files pixa looks for on its `config.yaml`. It reads the first file it finds. It skips a missing file without
own, only one that does not exist is passed over, without a message. One that a message. pixa refuses to start when a file it finds cannot be read or does not
pixa cannot read or parse aborts startup, naming the file. So does one in a parse, and the error names the file. It also refuses to start when it may not
directory pixa may not enter, whether or not it is there, since pixa cannot enter one of these directories, whether or not the file is there, since it
tell. With no file, pixa uses the environment and the defaults. cannot tell. With no file, pixa uses the environment and the defaults.
| Variable | Config key | Meaning | Each item below is one setting, named by its config key:
| ------------------------------------ | ------------------------------- | ---------------------------------------------------------------------------- |
| `PIXA_SIGNING_KEY` | `signing_key` | Required: secret for signed and encrypted URLs and login, 32+ characters | - `signing_key`: variable `PIXA_SIGNING_KEY`, required. The secret for signed
| `PORT` | `port` | Port to listen on; default `8080` | URLs, encrypted URLs and login. It must be 32 characters or longer.
| `PIXA_STATE_DIR` | `state_dir` | Directory for the database and the disk cache; default `/var/lib/pixa` | - `port`: variable `PORT`, default `8080`. The port to listen on.
| `PIXA_DB_URL` | `db_url` | SQLite database URL; default `state.sqlite3` in the state directory | - `state_dir`: variable `PIXA_STATE_DIR`, default `/var/lib/pixa`. The directory
| `PIXA_CACHE_MAX_BYTES` | `cache_max_bytes` | Disk cache limit in bytes; `0` disables it; default 75% of (free + cached) | for the database and the disk cache.
| `PIXA_ALLOWLIST_HOSTS` | `allowlist_hosts` | Upstream hosts served without a signature | - `db_url`: variable `PIXA_DB_URL`, default `state.sqlite3` in the state
| `PIXA_REFERER_BLOCKLIST` | `referer_blocklist` | Hosts whose pages the image routes refuse with 403, by `Referer` | directory. The URL of the SQLite database.
| `PIXA_BLOCKED_NETWORKS` | `blocked_networks` | CIDR ranges never fetched from, on top of the built-in ones | - `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default 75% of the sum of
| `PIXA_TRUSTED_PROXIES` | `trusted_proxies` | CIDR ranges of proxies whose `X-Forwarded-For` is believed; default RFC 1918 | the free space and what the cache holds. The disk cache limit in bytes. `0`
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` | turns the disk cache off.
| `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` | - `allowlist_hosts`: variable `PIXA_ALLOWLIST_HOSTS`, default none. The upstream
| `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` | hosts whose images pixa serves without a signature.
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default half the CPUs, at least 1 | - `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. The
| `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` | image routes refuse a request with 403 when its `Referer` header names one of
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` | these hosts.
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB | - `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. The CIDR
| `PIXA_DOWNSTREAM_TIMEOUT` | `downstream_timeout` | Time allowed for answering one client request; default `60s` | ranges pixa never fetches from, on top of the built-in ones.
| `PIXA_ACCESS_CONTROL_ALLOW_ORIGIN` | `access_control_allow_origin` | CORS origin allowed to read image responses: `*` or one origin; default `*` | - `trusted_proxies`: variable `PIXA_TRUSTED_PROXIES`, default the RFC 1918
| `PIXA_METRICS_USERNAME` | `metrics.username` | Username for `/metrics`, which is served only when both are set | ranges. The CIDR ranges of the proxies whose `X-Forwarded-For` header pixa
| `PIXA_METRICS_PASSWORD` | `metrics.password` | Password for `/metrics`; set together with the username | believes.
| `PIXA_SENTRY_DSN` | `sentry_dsn` | Sentry DSN for error reporting; empty disables it | - `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. Allows plain-HTTP
| `PIXA_DEBUG` | `debug` | Debug logging and plain-HTTP local development; default `false` | upstream hosts. It is for testing only.
| `PIXA_MAINTENANCE_MODE` | `maintenance_mode` | Answer image requests with 503; the health check stays 200; default `false` | - `upstream_connections_per_host`: variable
`PIXA_UPSTREAM_CONNECTIONS_PER_HOST`, default `20`. The most connections to
one upstream host at once.
- `upstream_connections`: variable `PIXA_UPSTREAM_CONNECTIONS`, default `64`.
The most connections to all upstream hosts together at once.
- `max_concurrent_processing`: variable `PIXA_MAX_CONCURRENT_PROCESSING`,
default half the CPUs, at least 1. The most images processed at once.
- `image_requests_per_minute`: variable `PIXA_IMAGE_REQUESTS_PER_MINUTE`,
default `600`. How many image requests one client may send per minute. `0`
turns the limit off.
- `upstream_fetch_timeout`: variable `PIXA_UPSTREAM_FETCH_TIMEOUT`, default
`30s`. The time allowed for one fetch from an upstream host.
- `upstream_max_response_size`: variable `PIXA_UPSTREAM_MAX_RESPONSE_SIZE`,
default 50 MiB. The largest upstream response accepted, in bytes.
- `downstream_timeout`: variable `PIXA_DOWNSTREAM_TIMEOUT`, default `60s`. The
time allowed for answering one client request.
- `access_control_allow_origin`: variable `PIXA_ACCESS_CONTROL_ALLOW_ORIGIN`,
default `*`. The CORS origin allowed to read image responses. It is `*` or one
origin.
- `metrics.username`: variable `PIXA_METRICS_USERNAME`, default empty. The
username for `/metrics`. pixa serves `/metrics` only when the username and the
password are both set.
- `metrics.password`: variable `PIXA_METRICS_PASSWORD`, default empty. The
password for `/metrics`. Set it together with the username.
- `sentry_dsn`: variable `PIXA_SENTRY_DSN`, default empty. The Sentry DSN for
error reporting. An empty DSN turns error reporting off.
- `debug`: variable `PIXA_DEBUG`, default `false`. Turns on debug logging and
plain-HTTP local development.
- `maintenance_mode`: variable `PIXA_MAINTENANCE_MODE`, default `false`. While
`true`, pixa answers image requests with 503. The health check still
answers 200.
Key settings in more detail: Key settings in more detail:
@@ -765,7 +879,17 @@ Key settings in more detail:
duration; default `60s`. The upstream fetch counts toward it, and so do the duration; default `60s`. The upstream fetch counts toward it, and so do the
waits for an upstream connection and for a processing slot (up to 10 seconds waits for an upstream connection and for a processing slot (up to 10 seconds
each), so keep it longer than `upstream_fetch_timeout` plus 20 seconds each), so keep it longer than `upstream_fetch_timeout` plus 20 seconds
- `signing_key` — HMAC secret for URL signatures - `signing_key` — the secret that protects signed URLs, encrypted URLs and
logins. pixa uses it as it is for two things:
- the HMAC key of signed URLs (see Signature Specification)
- the password of the login page
pixa derives every other key it uses from it at startup, with HKDF-SHA256:
- a key that signs the session cookie
- a key that encrypts the session cookie
- a key that encrypts the token of an encrypted URL
- a key that signs the CSRF token cookie of the login and generator pages
- `db_url` — the SQLite database to open; omitted, it is - `db_url` — the SQLite database to open; omitted, it is
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the `file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
database in WAL mode. `state_dir` is percent-encoded in it, so a `%`, `?` or database in WAL mode. `state_dir` is percent-encoded in it, so a `%`, `?` or
@@ -812,9 +936,25 @@ See `configs/config.example.yml` for all options with defaults.
### Architecture ### Architecture
- **Dependency injection**: Uber fx. `internal/app` builds every part of pixad - **Dependency injection**: Uber fx. `internal/app` puts every part of pixad in
from the config in one fx graph. Each setting's default is declared once, in one fx graph. `internal/providers` builds the image cache, the image proxy and
`internal/config`. the other parts that serve images and logins. Each setting's default is
declared once, in `internal/config`.
- **Packages**: these packages under `internal/` work with the concepts above:
- `handlers`: the routes, which read the image request from the URL, check
the signed or encrypted URL, and make the login session
- `imagerequest`: the image request, with the source key and the variant key
- `imageproxy`: the image proxy
- `imgcache`: the image cache, which keeps its sources in one `FileStore`
and its variants in another
- `httpfetcher`: fetching sources from upstream hosts
- `magic`: reading an image's format from its first bytes
- `imageprocessor`: making a variant from a source with libvips
- `signature`: checking the `sig` of a signed URL
- `encurl`: making and reading the tokens of encrypted URLs
- `seal`: the keys derived from `signing_key`, and the encryption of those
tokens
- `session`: the login session
- **HTTP router**: go-chi - **HTTP router**: go-chi
- **Image processing**: govips (CGO wrapper for libvips) - **Image processing**: govips (CGO wrapper for libvips)
- **Database**: SQLite via modernc.org/sqlite - **Database**: SQLite via modernc.org/sqlite
TODO.md
+173
View File
@@ -30,6 +30,179 @@ P2: security: per-origin rate limiting
# Completed Steps # Completed Steps
- 2026-10-27 the reconciliation pass skips a directory it cannot list and a file
whose info it cannot read, and goes on (closes #343). It stopped before its
later checks at a directory it could not list, or at a source or variant file
whose info it could not read.
- It logs the warning "skipped unreadable entry in cache directory", with
the path and the error.
- A file renamed or removed after its directory was listed gets no warning.
- `reconcileFiles` reads each file's info once, before the temp-file check,
and hands it to `sweepStaleTempFile`, `reconcileVariantFile` or
`reconcileSourceFile`.
- A cancelled context still stops the walk with its error.
- 2026-10-26 the reconciliation pass drops a variant's accounting row only when
its file is missing (closes #340). Any other error reading the file's size
keeps the row. It dropped the row on any error and logged the file as missing.
- The kept row gets the warning "kept accounting row for unreadable variant
file", with the variant key and the error.
- The pass checks for `ErrNotFound`, which `FileStore.Size` returns for a
missing file.
- 2026-10-26 on both image routes, a request is answered 499 and logged at info
level when its client goes away before its image is ready, unless it is the
request making the image (closes #339). It was answered 500 and logged at
error level, as pixa's own failure.
- The line is "client went away before the image was ready". It carries the
`request_id`, the host and the path.
- The 499 has no body, since nobody reads it.
- `serveImage` answers it before calling `respondImageError`, which cannot
see the request's context.
- An error wrapping `context.Canceled` while the request's context is live
is still answered 500 and logged at error level.
- 2026-10-26 `README.md`'s Design section starts with a Concepts list (closes
#321). Each concept gets one sentence, under the name the code gives it.
- Storage calls the transformed images variants, in shorter sentences.
- Storage's paths end in the source key and the variant key.
- Storage lists the image cache's tables in the database.
- Architecture names `internal/providers`, and lists the packages that work
with the concepts.
- 2026-10-26 one table in `internal/config` lists every setting once (closes
#320). `Config.settings` gives each setting's key, its environment variable
and the function that loads it.
- Most rows load through `field`, whose arguments give the setting's type,
its default and its check.
- `db_url` and `cache_max_bytes` have their own load functions. The default
of `db_url` comes from `state_dir`, and `cache_max_bytes` records whether
it is set.
- Loading, the unknown-key check, the unknown-variable check and the names
in error messages all read the table.
- `allowlist_hosts` loads as `referer_blocklist` does, through
`parseHostList`.
- The metrics username and password are checked together once the table has
loaded.
- When a config holds more than one invalid value, the error names the one
that comes first in the table.
- `README.md` lists the settings one item each, where it had a table. A test
checks that this list and `configs/config.example.yml` name exactly the
keys of the table.
- The key constants moved to the tests, their only users.
- Removed:
- `isKnownConfigKey`, `envVarNames`, `strictLoader` and `validate`
- `validateAllowlistHostsValue` and `getStringSlice`, with the three
tests of `getStringSlice`
- 2026-10-26 a fetched image's format is detected from its bytes once, as an
`imagerequest.Format` (closes #319). The upstream `Content-Type` check in the
fetcher stays as the filter before downloading.
- `magic.DetectFormat` returns an `imagerequest.Format`, and
`magic.ValidateMagicBytes` returns the format it checked against the
`Content-Type`.
- `Format.MIMEType` gives each format's MIME type. The fetcher, the mock
fetcher, the cache's read of a variant's type and the handler's `Accept`
matching use it.
- `Cache.Source` returns a source's format, read from its file's first bytes
as `Cache.Variant` reads a variant's. A file in no source format is an
error, and the proxy fetches the source again.
- The proxy hands the format to `ImageProcessor.Process`. The processor
refuses input that libvips would load as another format, as before, with
`ErrUnsupportedInputFormat`. It refuses a format that is not one of the
six source formats the same way.
- `internal/magic` no longer detects SVG, which pixa does not read. SVG data
is `ErrUnknownFormat`.
- Removed:
- `magic.MIMEType`, `magic.ImageFormat` and `magic.MIMEToImageFormat`
- `detectFormat`, `formatFromString`, `FormatToMIME` and
`Result.InputFormat` of the processor
- the fetcher's MIME constants of the six formats
- `errNotAnOutputFormat` of the cache
- 2026-10-26 the cache keeps its source images in one `imgcache.FileStore` and
its variants in another (closes #318). A `FileStore` is a directory of files,
each named by a hex key, at `<ab>/<cd>/<key>`.
- `ContentStorage` and `VariantStorage` are gone.
- `FileStore` has these methods:
- `Write`, which writes a file under a temp name, syncs it and renames
it into place
- `Open`, which returns the file and its size
- `Size`
- `Delete`
- `FileStore` has no `Exists`. The reconciliation pass checks that a
variant's file is there with `Size`. The tests look for the file on disk.
- `Cache.Variant` reads the content type from the variant file's first
bytes, as `VariantStorage.LoadWithContentType` did. A cache hit still
makes no database call.
- `storage.go` is now `filestore.go`. `internal/imgcache/testdata/red.avif`,
which no test used, is gone.
- 2026-10-26 `internal/imgcache` holds the cache only (closes #317). The new
package `internal/imageproxy` fetches, transforms, caches and serves images.
It imports `imgcache`, and `imgcache` does not import it.
- `imgcache.Service` is `imageproxy.Proxy`, built by `imageproxy.New` from
an `imageproxy.Config`.
- `ImageResponse`, `CacheStatus`, `ErrNegativeCached` and
`ErrProcessingTimedOut` moved with it, under their names.
- `imgcache.CacheConfig` is `imgcache.Config`.
- The cache has one read and one write for each kind it keeps. Each takes
the place of an earlier method:
- `Variant`, of `GetVariant`
- `PutVariant`, of `StoreVariant`
- `Source`, of `LookupSource` followed by `GetSourceContent`
- `PutSource`, of `StoreSource`
- `Failure`, of `checkNegativeCache`
- `PutFailure`, of `StoreNegative`
- `Variant` and `Source` each open the file and hand its access time to the
background writer. A cache hit still makes no database call.
- `imgcache.go` and `module.go` are gone.
- `imageproxy.New` makes the counters of upstream fetches and transforms,
and registers them on the registry it is given. The cache keeps only the
hit and miss counters.
- A cached source whose row cannot be read is logged as "failed to load
cached source, fetching". It was logged as "source lookup failed".
- The proxy's tests moved to `internal/imageproxy`. They reach the cache
through its database, its state directory and its metrics registry.
- The tests of an empty or cut cached file are split in two:
- `internal/imgcache` checks that the reconciliation pass removes the
file and its row, with a warning naming the file.
- `internal/imageproxy` removes the file and its row as the pass does,
and checks that `Get` fetches or converts again.
- 2026-10-26 both image routes serve an image with one function, `serveImage`,
and answer its errors with one, `respondImageError` (closes #316).
`HandleImage` parses the path and verifies the signature, and `HandleImageEnc`
decrypts the token. Each then calls `serveImage`.
- `handleImageError`, the encrypted route's own mapping, is gone.
- `/v1/image/` answers 504 when the upstream has not sent its response
headers within `upstream_fetch_timeout`, as `/v1/e/` did. It answered 500.
- `respondImageError` logs the image service's errors as "failed to get
image". Each answer gets the same level on `/v1/image/` and on `/v1/e/`:
- 500: error, as pixa's own failure
- 400: warn
- 403: warn
- 415: warn
- 502: warn
- 503: warn
- 504: warn
- Before, `/v1/image/` logged all of these answers at error level. `/v1/e/`
logged only its 500, as "image request failed", without the host and the
path.
- `respondImageError` does not log the 415 for a body refused for its first
bytes, on either route. The image service logs it as "source refused", at
warn level.
- The "image served" line carries the host, the path and `fetched_bytes` on
both routes. The debug line "encrypted image request" is gone.
- The debug line "failed to decrypt URL" is now "invalid encrypted URL". A
token that decrypts to a payload that does not decode also gets it.
- A failed write is logged once, as "failed to write response" at warn
level, since most often the client has gone away. It carries the
`request_id`, the host, the path, `served_bytes` and the error, and is not
also logged as "image served".
- 2026-10-26 every key derived from the signing key is derived in one place
(closes #315): `seal.DeriveKeys` returns them as one `seal.Keys`, with their
salts declared beside it. `internal/providers` calls it once at startup. Each
key is derived as before, with `seal.DeriveKey` and the same salt.
- The session manager, the encrypted URL generator and the CSRF protection
each take their own key instead of the signing key.
- `session.NewManager`, `encurl.NewGenerator`, `newCSRFProtect` and
`handlers.New` return no error, as nothing in them can fail now.
- The signing key itself stays the HMAC key of signed URLs and the login
password.
- `README.md` says under `signing_key` what it protects.
- 2026-10-26 the image request holds only what names the cached object (closes - 2026-10-26 the image request holds only what names the cached object (closes
#314): its source and its transform, which the variant key covers. The signed #314): its source and its transform, which the variant key covers. The signed
URL format, the signing algorithm and every route's answers are unchanged. URL format, the signing algorithm and every route's answers are unchanged.
internal/config/config.go
+343 -584
View File
File diff suppressed because it is too large Load Diff
internal/config/config_internal_test.go
-85
View File
@@ -1,97 +1,12 @@
package config package config
import ( import (
"os"
"path/filepath" "path/filepath"
"testing" "testing"
"git.eeqj.de/sneak/smartconfig" "git.eeqj.de/sneak/smartconfig"
) )
// writeTestConfig writes yamlContent to a temp config file and returns
// the file path.
func writeTestConfig(t *testing.T, yamlContent string) string {
t.Helper()
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.yml")
err := os.WriteFile(configPath, []byte(yamlContent), 0o600)
if err != nil {
t.Fatalf("failed to write test config: %v", err)
}
return configPath
}
// checkAllowlistHosts loads the config at configPath and asserts that
// getStringSlice returns the three expected hosts.
func checkAllowlistHosts(t *testing.T, configPath string) {
t.Helper()
sc, err := loadTestConfig(configPath)
if err != nil {
t.Fatalf("failed to load config: %v", err)
}
hosts := getStringSlice(sc)
if len(hosts) != 3 {
t.Errorf("expected 3 hosts, got %d: %v", len(hosts), hosts)
}
expected := []string{"static.sneak.cloud", "sneak.berlin", testHostS3}
for i, want := range expected {
if i >= len(hosts) {
t.Errorf("missing host at index %d: want %q", i, want)
continue
}
if hosts[i] != want {
t.Errorf("host[%d] = %q, want %q", i, hosts[i], want)
}
}
}
func TestGetStringSlice_YAMLList(t *testing.T) {
t.Parallel()
yamlContent := `
allowlist_hosts:
- static.sneak.cloud
- sneak.berlin
- s3.sneak.cloud
`
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
}
func TestGetStringSlice_CommaSeparated(t *testing.T) {
t.Parallel()
// Backwards compatibility with comma-separated string values.
yamlContent := `allowlist_hosts: "static.sneak.cloud, sneak.berlin, s3.sneak.cloud"`
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
}
func TestGetStringSlice_Empty(t *testing.T) {
t.Parallel()
configPath := writeTestConfig(t, `port: 8080`)
sc, err := loadTestConfig(configPath)
if err != nil {
t.Fatalf("failed to load config: %v", err)
}
hosts := getStringSlice(sc)
if len(hosts) != 0 {
t.Errorf("expected nil or empty slice, got %v", hosts)
}
}
// TestExampleConfigAllowlistsNoHost loads configs/config.example.yml // TestExampleConfigAllowlistsNoHost loads configs/config.example.yml
// with a real signing key, as Getting Started in README.md has it copied // with a real signing key, as Getting Started in README.md has it copied
// and edited, and checks that it allowlists no host: every image on an // and edited, and checks that it allowlists no host: every image on an
internal/config/config_validation_internal_test.go
+29 -1
View File
@@ -15,8 +15,36 @@ import (
_ "modernc.org/sqlite" // SQLite driver registration _ "modernc.org/sqlite" // SQLite driver registration
) )
// The config keys the tests name in the errors they expect.
const (
keyDebug = "debug"
keyMaintenanceMode = "maintenance_mode"
keyPort = "port"
keyStateDir = "state_dir"
keySentryDSN = "sentry_dsn"
keyDBURL = "db_url"
keyMetrics = "metrics"
keyMetricsUsername = "metrics.username"
keyMetricsPassword = "metrics.password"
keySigningKey = "signing_key"
keyAllowlistHosts = "allowlist_hosts"
keyRefererBlocklist = "referer_blocklist"
keyAllowHTTP = "allow_http"
keyUpstreamConnectionsPerHost = "upstream_connections_per_host"
keyUpstreamConnections = "upstream_connections"
keyMaxConcurrentProcessing = "max_concurrent_processing"
keyCacheMaxBytes = "cache_max_bytes"
keyBlockedNetworks = "blocked_networks"
keyTrustedProxies = "trusted_proxies"
keyAccessControlAllowOrigin = "access_control_allow_origin"
keyUpstreamFetchTimeout = "upstream_fetch_timeout"
keyUpstreamMaxResponseSize = "upstream_max_response_size"
keyDownstreamTimeout = "downstream_timeout"
keyImageRequestsPerMinute = "image_requests_per_minute"
)
// validTestSigningKey is a 32-character signing key that satisfies the // validTestSigningKey is a 32-character signing key that satisfies the
// minimum length requirement in validate(). // minimum length requirement in validateSigningKey.
const validTestSigningKey = "0123456789abcdef0123456789abcdef" const validTestSigningKey = "0123456789abcdef0123456789abcdef"
// signingKeyLine is a valid signing_key config line used as the base of // signingKeyLine is a valid signing_key config line used as the base of
internal/config/documented_settings_internal_test.go
+127
View File
@@ -0,0 +1,127 @@
package config
import (
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
// TestReadmeNamesEverySetting checks that the list of settings in
// README.md names exactly the keys of the settings table, one item each.
// The list is the first one in the Configuration section, and it ends at
// the first blank line. Each item starts with its key in backticks.
func TestReadmeNamesEverySetting(t *testing.T) {
t.Parallel()
readme, err := os.ReadFile(filepath.Join("..", "..", "README.md"))
if err != nil {
t.Fatalf("failed to read README.md: %v", err)
}
_, section, found := strings.Cut(string(readme), "\n### Configuration\n")
if !found {
t.Fatal("README.md has no Configuration section")
}
// The section ends at the next heading of its level or above.
section, _, _ = strings.Cut(section, "\n## ")
section, _, _ = strings.Cut(section, "\n### ")
_, list, found := strings.Cut(section, "\n- ")
if !found {
t.Fatal("README.md's Configuration section has no list")
}
list, _, _ = strings.Cut("- "+list, "\n\n")
itemKey := regexp.MustCompile("^- `([^`]+)`")
named := map[string]bool{}
for line := range strings.SplitSeq(list, "\n") {
// A line that does not start with "- " goes on with the item above.
if !strings.HasPrefix(line, "- ") {
continue
}
match := itemKey.FindStringSubmatch(line)
if match == nil {
t.Errorf("README.md's list of settings has an item "+
"that does not start with a key in backticks: %s", line)
continue
}
if named[match[1]] {
t.Errorf("README.md's list of settings names %s twice", match[1])
}
named[match[1]] = true
}
for _, s := range new(Config).settings() {
if !named[s.key] {
t.Errorf("README.md's list of settings does not name %s", s.key)
}
delete(named, s.key)
}
for key := range named {
t.Errorf("README.md's list of settings names %s, "+
"which is not a setting", key)
}
}
// TestExampleConfigNamesEverySetting checks that
// configs/config.example.yml names exactly the keys of the settings
// table, each one set or commented out with "# ". A key line holds the
// key, a colon and at most a one-word value, which tells it from a
// comment line that starts with a word and a colon. An indented key is
// inside the map of the key above it, as username is inside metrics.
func TestExampleConfigNamesEverySetting(t *testing.T) {
t.Parallel()
const examplePath = "configs/config.example.yml"
example, err := os.ReadFile(filepath.Join("..", "..", examplePath))
if err != nil {
t.Fatalf("failed to read %s: %v", examplePath, err)
}
keyLine := regexp.MustCompile(`(?m)^(?:# )?( *)([a-z_]+):(?: \S+)?$`)
named := map[string]bool{}
var mapKey string
for _, match := range keyLine.FindAllStringSubmatch(string(example), -1) {
indent, key := match[1], match[2]
if indent == "" {
mapKey = key
named[key] = true
continue
}
// The key holding the map is not itself a setting.
delete(named, mapKey)
named[mapKey+"."+key] = true
}
// The env section sets environment variables; it is not a setting.
delete(named, "env")
for _, s := range new(Config).settings() {
if !named[s.key] {
t.Errorf("%s does not name %s", examplePath, s.key)
}
delete(named, s.key)
}
for key := range named {
t.Errorf("%s names %s, which is not a setting", examplePath, key)
}
}
internal/config/env_internal_test.go
+4 -3
View File
@@ -138,7 +138,8 @@ func TestPixaPortAbortsStartupPointingToPort(t *testing.T) {
// TestSettingVariablesAndConfigPathAreAccepted checks that every // TestSettingVariablesAndConfigPathAreAccepted checks that every
// setting's variable and PIXA_CONFIG_PATH pass the check for unknown // setting's variable and PIXA_CONFIG_PATH pass the check for unknown
// PIXA_ variables. TestEnvironmentSetsEveryKey pins the names in the list. // PIXA_ variables. TestEnvironmentSetsEveryKey pins the names in the
// settings table.
func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) { func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) {
// A config file's env section loaded by another test can leave a // A config file's env section loaded by another test can leave a
// PIXA_ variable set for the whole process, so every one is unset // PIXA_ variable set for the whole process, so every one is unset
@@ -159,8 +160,8 @@ func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) {
t.Setenv("PIXA_CONFIG_PATH", "/etc/pixa/config.yml") t.Setenv("PIXA_CONFIG_PATH", "/etc/pixa/config.yml")
for _, name := range envVarNames() { for _, s := range new(Config).settings() {
t.Setenv(name, "") t.Setenv(s.envVar, "")
} }
err := validateKnownEnvVars() err := validateKnownEnvVars()
internal/encurl/encurl.go
+4 -12
View File
@@ -16,9 +16,6 @@ const (
DefaultQuality = 85 DefaultQuality = 85
DefaultFormat = imagerequest.FormatJXL DefaultFormat = imagerequest.FormatJXL
DefaultFitMode = imagerequest.FitCover DefaultFitMode = imagerequest.FitCover
// HKDF salt for URL encryption key derivation
urlKeySalt = "pixa-urlenc-v1"
) )
// Errors returned by encurl operations. // Errors returned by encurl operations.
@@ -48,15 +45,10 @@ type Generator struct {
key [seal.KeySize]byte key [seal.KeySize]byte
} }
// NewGenerator creates an encrypted URL generator with a key derived // NewGenerator creates an encrypted URL generator that encrypts its tokens
// from the signing key. // with key.
func NewGenerator(signingKey string) (*Generator, error) { func NewGenerator(key [seal.KeySize]byte) *Generator {
key, err := seal.DeriveKey([]byte(signingKey), urlKeySalt) return &Generator{key: key}
if err != nil {
return nil, err
}
return &Generator{key: key}, nil
} }
// Generate creates an encrypted URL token from the request's source and // Generate creates an encrypted URL token from the request's source and
internal/encurl/encurl_test.go
+19 -9
View File
@@ -7,6 +7,7 @@ import (
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
) )
// Shared test fixture strings. // Shared test fixture strings.
@@ -19,11 +20,13 @@ const (
func TestGenerator_GenerateAndParse(t *testing.T) { func TestGenerator_GenerateAndParse(t *testing.T) {
t.Parallel() t.Parallel()
gen, err := encurl.NewGenerator("test-signing-key-12345") keys, err := seal.DeriveKeys("test-signing-key-12345")
if err != nil { if err != nil {
t.Fatalf("NewGenerator() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
gen := encurl.NewGenerator(keys.EncryptedURLKey)
req := &imagerequest.Request{ req := &imagerequest.Request{
Source: imagerequest.Source{ Source: imagerequest.Source{
Host: testSourceHost, Host: testSourceHost,
@@ -99,7 +102,8 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
func TestGenerator_Parse_Expired(t *testing.T) { func TestGenerator_Parse_Expired(t *testing.T) {
t.Parallel() t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
req := &imagerequest.Request{ req := &imagerequest.Request{
Source: imagerequest.Source{ Source: imagerequest.Source{
@@ -126,7 +130,8 @@ func TestGenerator_Parse_Expired(t *testing.T) {
func TestGenerator_Parse_InvalidToken(t *testing.T) { func TestGenerator_Parse_InvalidToken(t *testing.T) {
t.Parallel() t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
_, _, err := gen.Parse("not-a-valid-token") _, _, err := gen.Parse("not-a-valid-token")
if err == nil { if err == nil {
@@ -137,7 +142,8 @@ func TestGenerator_Parse_InvalidToken(t *testing.T) {
func TestGenerator_Parse_TamperedToken(t *testing.T) { func TestGenerator_Parse_TamperedToken(t *testing.T) {
t.Parallel() t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
req := &imagerequest.Request{ req := &imagerequest.Request{
Source: imagerequest.Source{ Source: imagerequest.Source{
@@ -163,8 +169,10 @@ func TestGenerator_Parse_TamperedToken(t *testing.T) {
func TestGenerator_Parse_WrongKey(t *testing.T) { func TestGenerator_Parse_WrongKey(t *testing.T) {
t.Parallel() t.Parallel()
gen1, _ := encurl.NewGenerator("signing-key-1") keys1, _ := seal.DeriveKeys("signing-key-1")
gen2, _ := encurl.NewGenerator("signing-key-2") keys2, _ := seal.DeriveKeys("signing-key-2")
gen1 := encurl.NewGenerator(keys1.EncryptedURLKey)
gen2 := encurl.NewGenerator(keys2.EncryptedURLKey)
req := &imagerequest.Request{ req := &imagerequest.Request{
Source: imagerequest.Source{ Source: imagerequest.Source{
@@ -184,7 +192,8 @@ func TestGenerator_Parse_WrongKey(t *testing.T) {
func TestGenerator_Parse_Defaults(t *testing.T) { func TestGenerator_Parse_Defaults(t *testing.T) {
t.Parallel() t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
// Request with only required fields - should get defaults // Request with only required fields - should get defaults
token, err := gen.Generate(&imagerequest.Request{ token, err := gen.Generate(&imagerequest.Request{
@@ -221,7 +230,8 @@ func TestGenerator_Parse_Defaults(t *testing.T) {
func TestGenerator_TokenIsURLSafe(t *testing.T) { func TestGenerator_TokenIsURLSafe(t *testing.T) {
t.Parallel() t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
req := &imagerequest.Request{ req := &imagerequest.Request{
Source: imagerequest.Source{ Source: imagerequest.Source{
internal/handlers/auth_csrf_internal_test.go
+6 -15
View File
@@ -14,6 +14,7 @@ import (
"sneak.berlin/go/pixa/internal/config" "sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
@@ -44,27 +45,17 @@ func newCSRFTestRouter(t *testing.T) (*Handlers, http.Handler) {
cfg := &config.Config{SigningKey: testSigningKey, Debug: true} cfg := &config.Config{SigningKey: testSigningKey, Debug: true}
sessMgr, err := session.NewManager(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("session.NewManager() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
protect, err := newCSRFProtect(testSigningKey, cfg.Debug)
if err != nil {
t.Fatalf("newCSRFProtect() error = %v", err)
} }
h := &Handlers{ h := &Handlers{
log: slog.New(slog.DiscardHandler), log: slog.New(slog.DiscardHandler),
config: cfg, config: cfg,
sessMgr: sessMgr, sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
encGen: encGen, encGen: encurl.NewGenerator(keys.EncryptedURLKey),
csrfProtect: protect, csrfProtect: newCSRFProtect(keys.CSRFKey, cfg.Debug),
} }
r := chi.NewRouter() r := chi.NewRouter()
internal/handlers/auth_login_log_internal_test.go
+4 -3
View File
@@ -10,6 +10,7 @@ import (
"testing" "testing"
"sneak.berlin/go/pixa/internal/config" "sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
@@ -22,15 +23,15 @@ func TestLoginLogLeavesOutSubmittedKey(t *testing.T) {
var buf bytes.Buffer var buf bytes.Buffer
sessMgr, err := session.NewManager(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("session.NewManager() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
h := &Handlers{ h := &Handlers{
log: slog.New(slog.NewJSONHandler(&buf, nil)), log: slog.New(slog.NewJSONHandler(&buf, nil)),
config: &config.Config{SigningKey: testSigningKey}, config: &config.Config{SigningKey: testSigningKey},
sessMgr: sessMgr, sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
} }
submittedKeys := []string{wrongKey, testSigningKey} submittedKeys := []string{wrongKey, testSigningKey}
internal/handlers/auth_session_internal_test.go
+7 -1
View File
@@ -11,6 +11,7 @@ import (
"time" "time"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
@@ -232,8 +233,13 @@ func TestHandleLogout_SaveFails_Answers500(t *testing.T) {
db := setupTestDB(t) db := setupTestDB(t)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
sessMgr, err := session.NewManagerWithDatabase( sessMgr, err := session.NewManagerWithDatabase(
t.Context(), testSigningKey, db) t.Context(), keys.SessionHashKey, keys.SessionBlockKey, db)
if err != nil { if err != nil {
t.Fatalf("session.NewManagerWithDatabase() error = %v", err) t.Fatalf("session.NewManagerWithDatabase() error = %v", err)
} }
internal/handlers/avif_too_large_internal_test.go
+13 -23
View File
@@ -12,11 +12,11 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError and // TestAVIFOutputTooLargeAnswers400 checks that respondImageError, which
// handleImageError, which answer the errors of the two image routes, answer // answers the errors of both image routes, answers 400 naming the AVIF pixel
// 400 naming the AVIF pixel limit when the image processor refuses an AVIF // limit when the image processor refuses an AVIF output above it, wrapped as
// output above it, wrapped as the service wraps it. The request's source only // the service wraps it. The request's source only goes into a log line, so it
// goes into a log line, so it is left empty. // is left empty.
func TestAVIFOutputTooLargeAnswers400(t *testing.T) { func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
t.Parallel() t.Parallel()
@@ -25,25 +25,15 @@ func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
imageprocessor.ErrAVIFOutputTooLarge) imageprocessor.ErrAVIFOutputTooLarge)
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels) wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
plain := httptest.NewRecorder() rec := httptest.NewRecorder()
h.respondImageError(plain, &imagerequest.Request{}, err) h.respondImageError(rec, &imagerequest.Request{}, err)
t.Logf("status = %d, body = %q", rec.Code, rec.Body.String())
encrypted := httptest.NewRecorder() if rec.Code != http.StatusBadRequest {
h.handleImageError(encrypted, err) t.Errorf("status = %d, want %d", rec.Code, http.StatusBadRequest)
}
for name, rec := range map[string]*httptest.ResponseRecorder{ if !strings.Contains(rec.Body.String(), wantBody) {
"respondImageError": plain, "handleImageError": encrypted, t.Errorf("body = %q, want it to say %q", rec.Body.String(), wantBody)
} {
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status = %d, want %d",
name, rec.Code, http.StatusBadRequest)
}
if !strings.Contains(rec.Body.String(), wantBody) {
t.Errorf("%s: body = %q, want it to say %q",
name, rec.Body.String(), wantBody)
}
} }
} }
internal/handlers/bodylimit_internal_test.go
+6 -15
View File
@@ -11,6 +11,7 @@ import (
"sneak.berlin/go/pixa/internal/config" "sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
@@ -32,27 +33,17 @@ func newBodyLimitTestRouter(
cfg := &config.Config{SigningKey: testSigningKey, Debug: true} cfg := &config.Config{SigningKey: testSigningKey, Debug: true}
sessMgr, err := session.NewManager(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("session.NewManager() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
protect, err := newCSRFProtect(testSigningKey, cfg.Debug)
if err != nil {
t.Fatalf("newCSRFProtect() error = %v", err)
} }
h := &Handlers{ h := &Handlers{
log: slog.New(slog.DiscardHandler), log: slog.New(slog.DiscardHandler),
config: cfg, config: cfg,
sessMgr: sessMgr, sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
encGen: encGen, encGen: encurl.NewGenerator(keys.EncryptedURLKey),
csrfProtect: protect, csrfProtect: newCSRFProtect(keys.CSRFKey, cfg.Debug),
} }
r := chi.NewRouter() r := chi.NewRouter()
internal/handlers/cache_hit_internal_test.go
+14 -10
View File
@@ -23,8 +23,10 @@ import (
"sneak.berlin/go/pixa/internal/database" "sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature" "sneak.berlin/go/pixa/internal/signature"
) )
@@ -121,7 +123,7 @@ func newCountingDatabaseServer(
tb.Fatalf("encoding the photo: %v", err) tb.Fatalf("encoding the photo: %v", err)
} }
cache, err := imgcache.NewCache(db, imgcache.CacheConfig{ cache, err := imgcache.NewCache(db, imgcache.Config{
StateDir: tb.TempDir(), StateDir: tb.TempDir(),
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30, MaxBytes: 1 << 30,
@@ -130,7 +132,7 @@ func newCountingDatabaseServer(
tb.Fatalf("imgcache.NewCache() error = %v", err) tb.Fatalf("imgcache.NewCache() error = %v", err)
} }
svc, err := imgcache.NewService(&imgcache.ServiceConfig{ svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache, Cache: cache,
Fetcher: newMockFetcher(fstest.MapFS{ Fetcher: newMockFetcher(fstest.MapFS{
countingHost + photoPath: &fstest.MapFile{Data: photo.Bytes()}, countingHost + photoPath: &fstest.MapFile{Data: photo.Bytes()},
@@ -141,16 +143,18 @@ func newCountingDatabaseServer(
}), }),
MaxResponseSize: 50 << 20, MaxResponseSize: 50 << 20,
Logger: slog.New(slog.DiscardHandler), Logger: slog.New(slog.DiscardHandler),
}) }, prometheus.NewRegistry())
if err != nil { if err != nil {
tb.Fatalf("imgcache.NewService() error = %v", err) tb.Fatalf("imageproxy.New() error = %v", err)
} }
encGen, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
tb.Fatalf("encurl.NewGenerator() error = %v", err) tb.Fatalf("seal.DeriveKeys() error = %v", err)
} }
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
token, err := encGen.Generate(&imagerequest.Request{ token, err := encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: countingHost, Path: photoPath}, Source: imagerequest.Source{Host: countingHost, Path: photoPath},
Transform: imagerequest.Transform{ Transform: imagerequest.Transform{
@@ -225,9 +229,9 @@ func TestImageRoutes_CacheHitsMakeNoDatabaseCall(t *testing.T) {
rec := serveImage(t, srv, target) rec := serveImage(t, srv, target)
cacheStatus := rec.Header().Get("X-Pixa-Cache") cacheStatus := rec.Header().Get("X-Pixa-Cache")
if cacheStatus != string(imgcache.CacheHit) { if cacheStatus != string(imageproxy.CacheHit) {
t.Errorf("GET %s: X-Pixa-Cache = %q, want %s", t.Errorf("GET %s: X-Pixa-Cache = %q, want %s",
target, cacheStatus, imgcache.CacheHit) target, cacheStatus, imageproxy.CacheHit)
} }
} }
@@ -257,9 +261,9 @@ func BenchmarkHandleImage_CacheHits(b *testing.B) {
rec := serveImage(b, srv, target) rec := serveImage(b, srv, target)
cacheStatus := rec.Header().Get("X-Pixa-Cache") cacheStatus := rec.Header().Get("X-Pixa-Cache")
if cacheStatus != string(imgcache.CacheHit) { if cacheStatus != string(imageproxy.CacheHit) {
b.Errorf("GET %s: X-Pixa-Cache = %q, want %s", b.Errorf("GET %s: X-Pixa-Cache = %q, want %s",
target, cacheStatus, imgcache.CacheHit) target, cacheStatus, imageproxy.CacheHit)
return return
} }
internal/handlers/client_gone_internal_test.go
+177
View File
@@ -0,0 +1,177 @@
package handlers
import (
"bytes"
"context"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"slices"
"testing"
"time"
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// heldFetcher sends on started when a fetch begins, and holds the fetch until
// release is closed. It then fails the fetch as an upstream error.
type heldFetcher struct {
started chan struct{}
release chan struct{}
}
// Fetch sends on f.started, waits for f.release to be closed, and returns
// httpfetcher.ErrUpstreamError.
func (f heldFetcher) Fetch(
context.Context, imagerequest.Source,
) (*httpfetcher.FetchResult, error) {
f.started <- struct{}{}
<-f.release
return nil, httpfetcher.ErrUpstreamError
}
// readLogLines decodes the JSON log lines in logs.
func readLogLines(t *testing.T, logs *bytes.Buffer) []map[string]any {
t.Helper()
t.Logf("logged:\n%s", logs.String())
var lines []map[string]any
dec := json.NewDecoder(logs)
for dec.More() {
var line map[string]any
err := dec.Decode(&line)
if err != nil {
t.Fatalf("decoding log line: %v", err)
}
lines = append(lines, line)
}
return lines
}
// TestImageRoutes_ClientGoneWhileWaitingAnswers499 checks that both image
// routes answer 499 to a request whose client goes away while it waits for an
// image another request is making. A first request's fetch is held, so the
// image is being made, and a second request for the same image waits for it
// until its client goes away. The second request must be logged once at info
// level with its request ID and the source host and path. Nothing may be
// logged at error level.
func TestImageRoutes_ClientGoneWhileWaitingAnswers499(t *testing.T) {
t.Parallel()
const requestID = "test-request-id"
for route, imageURL := range photoURLsByRoute() {
t.Run(route, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
fetcher := heldFetcher{started: make(chan struct{}), release: make(chan struct{})}
h, routes := newRoutesFetchingThrough(t, fetcher,
slog.New(slog.NewJSONHandler(&logs, nil)))
target := imageURL(t, h)
firstDone := make(chan struct{})
go func() {
defer close(firstDone)
sendGet(t, routes, target)
}()
<-fetcher.started
ctx, cancel := context.WithCancel(context.WithValue(t.Context(),
middleware.RequestIDKey, requestID))
rec := httptest.NewRecorder()
secondDone := make(chan struct{})
go func() {
defer close(secondDone)
routes.ServeHTTP(rec, httptest.NewRequestWithContext(
ctx, http.MethodGet, target, nil))
}()
// Give the second request time to start waiting for the first
// request's image. The fetch is still held, so the second request
// can end only because its client went away
time.Sleep(100 * time.Millisecond)
cancel()
<-secondDone
close(fetcher.release)
<-firstDone
if rec.Code != 499 {
t.Errorf("status = %d, want 499", rec.Code)
}
var clientGoneLines int
for _, line := range readLogLines(t, &logs) {
if line["level"] == slog.LevelError.String() {
t.Errorf("logged at error level: %v", line["msg"])
}
if line["level"] == slog.LevelInfo.String() &&
line["msg"] == "client went away before the image was ready" &&
line["request_id"] == requestID &&
line["host"] == signedHost &&
line["path"] == photoPath {
clientGoneLines++
}
}
if clientGoneLines != 1 {
t.Errorf("info line naming the request logged %d times, want 1",
clientGoneLines)
}
})
}
}
// TestImageRoutes_CanceledWithLiveRequestAnswers500 checks that both image
// routes answer 500, logged as "failed to get image" at error level, for an
// image service error that wraps context.Canceled while the request's own
// context is live. Its client has not gone away, so the cancellation is
// pixa's own failure.
func TestImageRoutes_CanceledWithLiveRequestAnswers500(t *testing.T) {
t.Parallel()
for route, imageURL := range photoURLsByRoute() {
t.Run(route, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h, routes := newRoutesFetchingThrough(t,
failingFetcher{err: context.Canceled},
slog.New(slog.NewJSONHandler(&logs, nil)))
rec := sendGet(t, routes, imageURL(t, h))
if rec.Code != http.StatusInternalServerError {
t.Errorf("status = %d, want %d",
rec.Code, http.StatusInternalServerError)
}
if !slices.ContainsFunc(readLogLines(t, &logs),
func(line map[string]any) bool {
return line["level"] == slog.LevelError.String() &&
line["msg"] == "failed to get image"
}) {
t.Error(`no "failed to get image" line logged at error level`)
}
})
}
}
internal/handlers/csrf.go
+5 -15
View File
@@ -9,15 +9,10 @@ import (
"sneak.berlin/go/pixa/internal/seal" "sneak.berlin/go/pixa/internal/seal"
) )
// csrfKeySalt provides domain separation for the CSRF authentication key,
// derived from the signing key so tokens survive restarts without extra
// configuration and never reuse the session or encrypted-URL key material.
const csrfKeySalt = "pixa-csrf-v1"
// newCSRFProtect builds the CSRF-protection middleware for the // newCSRFProtect builds the CSRF-protection middleware for the
// state-mutating HTML form routes. The token lives in its own cookie, // state-mutating HTML form routes. The token lives in its own cookie,
// independent of the session cookie, so it also protects the login POST // independent of the session cookie, so it also protects the login POST
// where no session exists yet (login CSRF). // where no session exists yet (login CSRF). key signs that cookie.
// //
// When plaintext is true (local HTTP development), requests are marked // When plaintext is true (local HTTP development), requests are marked
// plaintext so the library neither demands an https Referer nor sets a // plaintext so the library neither demands an https Referer nor sets a
@@ -25,13 +20,8 @@ const csrfKeySalt = "pixa-csrf-v1"
// service runs behind a TLS-terminating proxy, so plaintext is false and // service runs behind a TLS-terminating proxy, so plaintext is false and
// the library enforces its https Referer origin check. // the library enforces its https Referer origin check.
func newCSRFProtect( func newCSRFProtect(
signingKey string, plaintext bool, key [seal.KeySize]byte, plaintext bool,
) (func(http.Handler) http.Handler, error) { ) func(http.Handler) http.Handler {
key, err := seal.DeriveKey([]byte(signingKey), csrfKeySalt)
if err != nil {
return nil, err
}
protect := csrf.Protect( protect := csrf.Protect(
key[:], key[:],
csrf.Path("/"), csrf.Path("/"),
@@ -40,7 +30,7 @@ func newCSRFProtect(
) )
if !plaintext { if !plaintext {
return protect, nil return protect
} }
return func(next http.Handler) http.Handler { return func(next http.Handler) http.Handler {
@@ -49,7 +39,7 @@ func newCSRFProtect(
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
protected.ServeHTTP(w, csrf.PlaintextHTTPRequest(r)) protected.ServeHTTP(w, csrf.PlaintextHTTPRequest(r))
}) })
}, nil }
} }
// CSRF returns the CSRF-protection middleware for the login and generator // CSRF returns the CSRF-protection middleware for the login and generator
internal/handlers/first_bytes_internal_test.go
+10 -6
View File
@@ -17,8 +17,10 @@ import (
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature" "sneak.berlin/go/pixa/internal/signature"
) )
@@ -37,7 +39,7 @@ func newRoutesFetchingThrough(
) (*Handlers, http.Handler) { ) (*Handlers, http.Handler) {
t.Helper() t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{ cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(), StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30, MaxBytes: 1 << 30,
@@ -46,7 +48,7 @@ func newRoutesFetchingThrough(
t.Fatalf("imgcache.NewCache() error = %v", err) t.Fatalf("imgcache.NewCache() error = %v", err)
} }
svc, err := imgcache.NewService(&imgcache.ServiceConfig{ svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache, Cache: cache,
Fetcher: fetcher, Fetcher: fetcher,
Processor: imageprocessor.New(imageprocessor.Params{ Processor: imageprocessor.New(imageprocessor.Params{
@@ -55,16 +57,18 @@ func newRoutesFetchingThrough(
}), }),
MaxResponseSize: 50 << 20, MaxResponseSize: 50 << 20,
Logger: log, Logger: log,
}) }, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("imgcache.NewService() error = %v", err) t.Fatalf("imageproxy.New() error = %v", err)
} }
encGen, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{ h := &Handlers{
log: log, log: log,
imgSvc: svc, imgSvc: svc,
internal/handlers/format_auto.go
+4 -4
View File
@@ -65,20 +65,20 @@ func formatForAccept(accept string) (imagerequest.Format, error) {
return imagerequest.FormatJPEG, nil return imagerequest.FormatJPEG, nil
} }
if qualities["image/jxl"] > 0 { if qualities[imagerequest.FormatJXL.MIMEType()] > 0 {
return imagerequest.FormatJXL, nil return imagerequest.FormatJXL, nil
} }
if qualities["image/avif"] > 0 { if qualities[imagerequest.FormatAVIF.MIMEType()] > 0 {
return imagerequest.FormatAVIF, nil return imagerequest.FormatAVIF, nil
} }
if qualities["image/webp"] > 0 { if qualities[imagerequest.FormatWebP.MIMEType()] > 0 {
return imagerequest.FormatWebP, nil return imagerequest.FormatWebP, nil
} }
// For JPEG, the most specific entry the header has decides // For JPEG, the most specific entry the header has decides
quality, named := qualities["image/jpeg"] quality, named := qualities[imagerequest.FormatJPEG.MIMEType()]
if !named { if !named {
quality, named = qualities["image/*"] quality, named = qualities["image/*"]
} }
internal/handlers/format_auto_internal_test.go
+3 -3
View File
@@ -75,9 +75,9 @@ func TestFormatForAccept(t *testing.T) {
// autoPhotoURLs returns a signed /v1/image/ URL and an encrypted /v1/e/ URL, // autoPhotoURLs returns a signed /v1/image/ URL and an encrypted /v1/e/ URL,
// both valid for a minute, for the JPEG at photoPath on signedHost at 50x50 in // both valid for a minute, for the JPEG at photoPath on signedHost at 50x50 in
// the format auto. The first is signed with testSigningKey, the key of the // the format auto. The first is signed with testSigningKey, the key the
// image service newSignedHostServer builds, and the second is made with h's // handlers that newSignedHostServer builds check signatures with. The second
// generator. // is made with h's generator.
func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) { func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) {
t.Helper() t.Helper()
internal/handlers/handlers.go
+8 -11
View File
@@ -13,8 +13,9 @@ import (
"sneak.berlin/go/pixa/internal/config" "sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/healthcheck" "sneak.berlin/go/pixa/internal/healthcheck"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/logger" "sneak.berlin/go/pixa/internal/logger"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
"sneak.berlin/go/pixa/internal/signature" "sneak.berlin/go/pixa/internal/signature"
) )
@@ -26,8 +27,9 @@ type Params struct {
Logger *logger.Logger Logger *logger.Logger
Healthcheck *healthcheck.Healthcheck Healthcheck *healthcheck.Healthcheck
Config *config.Config Config *config.Config
Keys seal.Keys
ImageService *imgcache.Service ImageService *imageproxy.Proxy
SessionManager *session.Manager SessionManager *session.Manager
EncryptedURLGenerator *encurl.Generator EncryptedURLGenerator *encurl.Generator
} }
@@ -37,7 +39,7 @@ type Handlers struct {
log *slog.Logger log *slog.Logger
hc *healthcheck.Healthcheck hc *healthcheck.Healthcheck
config *config.Config config *config.Config
imgSvc *imgcache.Service imgSvc *imageproxy.Proxy
sessMgr *session.Manager sessMgr *session.Manager
encGen *encurl.Generator encGen *encurl.Generator
csrfProtect func(http.Handler) http.Handler csrfProtect func(http.Handler) http.Handler
@@ -53,12 +55,7 @@ type Handlers struct {
} }
// New creates a new Handlers instance. // New creates a new Handlers instance.
func New(_ fx.Lifecycle, params Params) (*Handlers, error) { func New(_ fx.Lifecycle, params Params) *Handlers {
csrfProtect, err := newCSRFProtect(params.Config.SigningKey, params.Config.Debug)
if err != nil {
return nil, err
}
return &Handlers{ return &Handlers{
log: params.Logger.Get(), log: params.Logger.Get(),
hc: params.Healthcheck, hc: params.Healthcheck,
@@ -66,11 +63,11 @@ func New(_ fx.Lifecycle, params Params) (*Handlers, error) {
imgSvc: params.ImageService, imgSvc: params.ImageService,
sessMgr: params.SessionManager, sessMgr: params.SessionManager,
encGen: params.EncryptedURLGenerator, encGen: params.EncryptedURLGenerator,
csrfProtect: csrfProtect, csrfProtect: newCSRFProtect(params.Keys.CSRFKey, params.Config.Debug),
signer: signature.New(params.Config.SigningKey), signer: signature.New(params.Config.SigningKey),
allowlist: allowlist.New(params.Config.AllowlistHosts), allowlist: allowlist.New(params.Config.AllowlistHosts),
refererBlocklist: allowlist.New(params.Config.RefererBlocklist), refererBlocklist: allowlist.New(params.Config.RefererBlocklist),
}, nil }
} }
// WaitForProcessing waits until no image is being processed, or until ctx // WaitForProcessing waits until no image is being processed, or until ctx
internal/handlers/handlers_internal_test.go
+5 -4
View File
@@ -23,6 +23,7 @@ import (
"sneak.berlin/go/pixa/internal/database" "sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/signature" "sneak.berlin/go/pixa/internal/signature"
@@ -49,7 +50,7 @@ func setupTestHandler(t *testing.T) *testFixtures {
tmpDir := t.TempDir() tmpDir := t.TempDir()
db := setupTestDB(t) db := setupTestDB(t)
cache, err := imgcache.NewCache(db, imgcache.CacheConfig{ cache, err := imgcache.NewCache(db, imgcache.Config{
StateDir: tmpDir, StateDir: tmpDir,
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30, MaxBytes: 1 << 30,
@@ -58,7 +59,7 @@ func setupTestHandler(t *testing.T) *testFixtures {
t.Fatalf("failed to create cache: %v", err) t.Fatalf("failed to create cache: %v", err)
} }
svc, err := imgcache.NewService(&imgcache.ServiceConfig{ svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache, Cache: cache,
Fetcher: newMockFetcher(mockFS), Fetcher: newMockFetcher(mockFS),
Processor: imageprocessor.New(imageprocessor.Params{ Processor: imageprocessor.New(imageprocessor.Params{
@@ -66,9 +67,9 @@ func setupTestHandler(t *testing.T) *testFixtures {
MaxConcurrentProcessing: 2, MaxConcurrentProcessing: 2,
}), }),
MaxResponseSize: 50 << 20, MaxResponseSize: 50 << 20,
}) }, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("failed to create service: %v", err) t.Fatalf("failed to create proxy: %v", err)
} }
h := &Handlers{ h := &Handlers{
internal/handlers/image.go
+113 -85
View File
@@ -1,6 +1,7 @@
package handlers package handlers
import ( import (
"context"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -14,11 +15,17 @@ import (
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic" "sneak.berlin/go/pixa/internal/magic"
) )
// statusClientClosedRequest is the status nginx calls "client closed
// request". The image routes answer it when the client goes away before its
// image is ready, unless its request is the one making the image. Nobody reads
// it; it is for logs and metrics.
const statusClientClosedRequest = 499
// HandleImage handles the main image proxy route: // HandleImage handles the main image proxy route:
// /v1/image/<host>/<path>/<width>x<height>.<format>, or with no format // /v1/image/<host>/<path>/<width>x<height>.<format>, or with no format
// /v1/image/<host>/<path>/<width>x<height> // /v1/image/<host>/<path>/<width>x<height>
@@ -46,30 +53,7 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
return return
} }
// The signature covers the format auto, not the format chosen s.serveImage(w, r, req, expires)
format, ok := s.chooseAutoFormat(w, r, req.Transform.Format)
if !ok {
return
}
req.Transform.Format = format
// Get cache key for logging
cacheKey := imagerequest.CacheKey(req)
// Get the image (from cache or fetch/process)
startTime := time.Now()
resp, err := s.imgSvc.Get(r.Context(), req)
if err != nil {
s.respondImageError(w, req, err)
return
}
defer func() { _ = resp.Content.Close() }()
s.writeImageResponse(w, r, req, expires, resp, cacheKey, startTime)
} }
} }
@@ -213,8 +197,8 @@ func (s *Handlers) validateSignature(
} }
// parseExpires reads the exp query parameter, a Unix time in seconds. An exp // parseExpires reads the exp query parameter, a Unix time in seconds. An exp
// missing from the URL gives the zero time, which the signature check takes // missing from the URL gives the zero time, which the signature check refuses
// as no expiration. An exp in the URL that is not a whole number, an empty // as a missing exp. An exp in the URL that is not a whole number, an empty
// one included, is an error naming exp and the value. // one included, is an error naming exp and the value.
func parseExpires(query url.Values) (time.Time, error) { func parseExpires(query url.Values) (time.Time, error) {
if !query.Has("exp") { if !query.Has("exp") {
@@ -242,7 +226,11 @@ func (s *Handlers) HandleTooManyRequests() http.HandlerFunc {
} }
} }
// respondImageError maps image retrieval errors to HTTP responses. // respondImageError answers err, the image service's error for req, on both
// image routes. It logs a 500 at error level, as pixa's own failure. It logs
// the other answers at warn level, since the client, the upstream or a full
// limit caused them. The 415 for a source refused for its first bytes is not
// logged here, because the image service logs it.
func (s *Handlers) respondImageError( func (s *Handlers) respondImageError(
w http.ResponseWriter, req *imagerequest.Request, err error, w http.ResponseWriter, req *imagerequest.Request, err error,
) { ) {
@@ -258,59 +246,53 @@ func (s *Handlers) respondImageError(
return return
} }
s.log.Error("failed to get image", var (
status int
message string
)
switch {
case errors.Is(err, httpfetcher.ErrSSRFBlocked):
status, message = http.StatusForbidden, "forbidden"
case errors.Is(err, httpfetcher.ErrUpstreamError):
status, message = http.StatusBadGateway, "upstream error"
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
status, message = http.StatusGatewayTimeout, "upstream timeout"
case errors.Is(err, httpfetcher.ErrTooManyConnections),
errors.Is(err, imageprocessor.ErrTooManyImages),
errors.Is(err, imageproxy.ErrProcessingTimedOut):
status = http.StatusServiceUnavailable
message = "server busy, try again later"
case errors.Is(err, httpfetcher.ErrInvalidContentType),
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat):
status = http.StatusUnsupportedMediaType
message = "unsupported source format"
case errors.Is(err, imageprocessor.ErrOutputTooLarge):
status = http.StatusBadRequest
message = fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d",
imageprocessor.MaxOutputDimension)
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
status = http.StatusBadRequest
message = fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels)
default:
s.log.Error("failed to get image",
"host", req.Source.Host,
"path", req.Source.Path,
"error", err,
)
s.respondError(w, "internal error", http.StatusInternalServerError)
return
}
s.log.Warn("failed to get image",
"host", req.Source.Host, "host", req.Source.Host,
"path", req.Source.Path, "path", req.Source.Path,
"error", err, "error", err,
) )
s.respondError(w, message, status)
// Check for specific error types
if errors.Is(err, httpfetcher.ErrSSRFBlocked) {
s.respondError(w, "forbidden", http.StatusForbidden)
return
}
if errors.Is(err, httpfetcher.ErrUpstreamError) {
s.respondError(w, "upstream error", http.StatusBadGateway)
return
}
if errors.Is(err, httpfetcher.ErrTooManyConnections) ||
errors.Is(err, imageprocessor.ErrTooManyImages) ||
errors.Is(err, imgcache.ErrProcessingTimedOut) {
s.respondError(w, "server busy, try again later",
http.StatusServiceUnavailable)
return
}
if errors.Is(err, httpfetcher.ErrInvalidContentType) ||
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat) {
s.respondError(w, "unsupported source format",
http.StatusUnsupportedMediaType)
return
}
if errors.Is(err, imageprocessor.ErrOutputTooLarge) {
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d",
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
return
}
if errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge) {
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
http.StatusBadRequest)
return
}
s.respondError(w, "internal error", http.StatusInternalServerError)
} }
// cacheControl returns the Cache-Control header for an image served through a // cacheControl returns the Cache-Control header for an image served through a
@@ -367,14 +349,50 @@ func notModified(w http.ResponseWriter, r *http.Request, etag string) bool {
return true return true
} }
// writeImageResponse writes headers and streams the image content, // serveImage serves the image that req asks for. Both image routes call it
// handling conditional and HEAD requests. expires is the URL's expiry, which // once they have the request and its URL's expiry, which ends the
// ends the Cache-Control max-age. // Cache-Control max-age. It chooses the format for the format auto, answers
func (s *Handlers) writeImageResponse( // If-None-Match with 304 and HEAD with the headers only. It answers 499 when
// the image service stopped because the client went away, and any other error
// of the image service with respondImageError.
func (s *Handlers) serveImage(
w http.ResponseWriter, r *http.Request, w http.ResponseWriter, r *http.Request,
req *imagerequest.Request, expires time.Time, resp *imgcache.ImageResponse, req *imagerequest.Request, expires time.Time,
cacheKey imagerequest.VariantKey, startTime time.Time,
) { ) {
// The signature or the token covers the format auto, not the format chosen
format, ok := s.chooseAutoFormat(w, r, req.Transform.Format)
if !ok {
return
}
req.Transform.Format = format
startTime := time.Now()
resp, err := s.imgSvc.Get(r.Context(), req)
// The client went away before its image was ready, which is not pixa's
// failure. A cancellation while the request's context is live is pixa's,
// and respondImageError answers it 500.
if errors.Is(err, context.Canceled) && r.Context().Err() != nil {
s.log.Info("client went away before the image was ready",
"request_id", middleware.GetReqID(r.Context()),
"host", req.Source.Host,
"path", req.Source.Path,
)
w.WriteHeader(statusClientClosedRequest)
return
}
if err != nil {
s.respondImageError(w, req, err)
return
}
defer func() { _ = resp.Content.Close() }()
// Set response headers // Set response headers
w.Header().Set("Content-Type", resp.ContentType) w.Header().Set("Content-Type", resp.ContentType)
@@ -400,18 +418,28 @@ func (s *Handlers) writeImageResponse(
// Stream the response // Stream the response
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
// A failed write is most often a client that has gone away, not pixa's
// failure, so it is logged at warn level
servedBytes, err := io.Copy(w, resp.Content) servedBytes, err := io.Copy(w, resp.Content)
if err != nil { if err != nil {
s.log.Error("failed to write response", s.log.Warn("failed to write response",
"request_id", middleware.GetReqID(r.Context()),
"host", req.Source.Host,
"path", req.Source.Path,
"served_bytes", servedBytes,
"error", err, "error", err,
) )
return
} }
// Log cache status and timing after serving // Log cache status and timing after serving
duration := time.Since(startTime) duration := time.Since(startTime)
s.log.Info("image served", s.log.Info("image served",
"request_id", middleware.GetReqID(r.Context()), "request_id", middleware.GetReqID(r.Context()),
"cache_key", cacheKey, "cache_key", imagerequest.CacheKey(req),
"host", req.Source.Host,
"path", req.Source.Path,
"cache_status", resp.CacheStatus, "cache_status", resp.CacheStatus,
"duration_ms", duration.Milliseconds(), "duration_ms", duration.Milliseconds(),
"format", req.Transform.Format, "format", req.Transform.Format,
internal/handlers/image_cache_control_internal_test.go
+10 -6
View File
@@ -17,8 +17,10 @@ import (
"sneak.berlin/go/pixa/internal/allowlist" "sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature" "sneak.berlin/go/pixa/internal/signature"
) )
@@ -34,7 +36,7 @@ func newSignedHostServer(
) (*Handlers, http.Handler) { ) (*Handlers, http.Handler) {
t.Helper() t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{ cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(), StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30, MaxBytes: 1 << 30,
@@ -45,7 +47,7 @@ func newSignedHostServer(
jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255}) jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
svc, err := imgcache.NewService(&imgcache.ServiceConfig{ svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache, Cache: cache,
Fetcher: newMockFetcher(fstest.MapFS{ Fetcher: newMockFetcher(fstest.MapFS{
signedHost + photoPath: &fstest.MapFile{Data: jpegData}, signedHost + photoPath: &fstest.MapFile{Data: jpegData},
@@ -56,16 +58,18 @@ func newSignedHostServer(
}), }),
MaxResponseSize: 50 << 20, MaxResponseSize: 50 << 20,
Logger: log, Logger: log,
}) }, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("imgcache.NewService() error = %v", err) t.Fatalf("imageproxy.New() error = %v", err)
} }
encGen, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{ h := &Handlers{
log: log, log: log,
imgSvc: svc, imgSvc: svc,
internal/handlers/image_errors_internal_test.go
+182 -13
View File
@@ -1,22 +1,28 @@
package handlers package handlers
import ( import (
"bytes"
"context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"image/color" "image/color"
"io"
"log/slog" "log/slog"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"slices"
"strings" "strings"
"testing" "testing"
"testing/fstest" "testing/fstest"
"time" "time"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist" "sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/signature" "sneak.berlin/go/pixa/internal/signature"
@@ -31,7 +37,7 @@ const allowlistedHost = "allowed.example.com"
func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler { func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
t.Helper() t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{ cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(), StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30, MaxBytes: 1 << 30,
@@ -40,7 +46,7 @@ func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
t.Fatalf("failed to create cache: %v", err) t.Fatalf("failed to create cache: %v", err)
} }
svc, err := imgcache.NewService(&imgcache.ServiceConfig{ svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache, Cache: cache,
Fetcher: fetcher, Fetcher: fetcher,
Processor: imageprocessor.New(imageprocessor.Params{ Processor: imageprocessor.New(imageprocessor.Params{
@@ -48,9 +54,9 @@ func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
MaxConcurrentProcessing: 2, MaxConcurrentProcessing: 2,
}), }),
MaxResponseSize: 50 << 20, MaxResponseSize: 50 << 20,
}) }, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("failed to create service: %v", err) t.Fatalf("failed to create proxy: %v", err)
} }
h := &Handlers{ h := &Handlers{
@@ -67,7 +73,7 @@ func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
} }
// newPhotoFetcher returns a mock fetcher that serves a JPEG at photoPath on // newPhotoFetcher returns a mock fetcher that serves a JPEG at photoPath on
// each of hosts, and answers any other URL with an upstream error. // each of hosts, and answers any other source with an upstream error.
func newPhotoFetcher(t *testing.T, hosts ...string) *httpfetcher.MockFetcher { func newPhotoFetcher(t *testing.T, hosts ...string) *httpfetcher.MockFetcher {
t.Helper() t.Helper()
@@ -295,10 +301,9 @@ func TestHandleImage_AllowlistOrSignature(t *testing.T) {
// host can be, and checks that each is refused with 400 naming the host as // host can be, and checks that each is refused with 400 naming the host as
// decoded: allowlistedHost with a character after it that a URL host cannot // decoded: allowlistedHost with a character after it that a URL host cannot
// hold, percent-encoded, an IPv6 address without brackets, and brackets // hold, percent-encoded, an IPv6 address without brackets, and brackets
// around something that is not an IPv6 address. A "?" or "#" after // around something that is not an IPv6 address. Later steps put the host
// allowlistedHost would otherwise end the host when the source URL is read, // unparsed into the allowlist check, the cache key and the upstream URL, so
// so the request would pass the allowlist and be fetched and cached under a // such a host is refused first.
// host that is not the one fetched.
func TestHandleImage_InvalidHost_Returns400(t *testing.T) { func TestHandleImage_InvalidHost_Returns400(t *testing.T) {
t.Parallel() t.Parallel()
@@ -369,10 +374,9 @@ func TestHandleImage_HostForms_Served(t *testing.T) {
// TestHandleImage_HashInSourcePathOrQuery_Returns400 sends image URLs whose // TestHandleImage_HashInSourcePathOrQuery_Returns400 sends image URLs whose
// source path or source query holds a "#", written as %23, and checks that // source path or source query holds a "#", written as %23, and checks that
// each is refused with 400, the one signed for that source path included. The // each is refused with 400, the one signed for that source path included.
// source URL would otherwise end at the "#" when it is fetched, so each text // A "#" starts a URL's fragment, so neither the source path nor the source
// after it would fetch the image at photoPath and cache it under keys of its // query may hold one.
// own.
func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) { func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
t.Parallel() t.Parallel()
@@ -416,3 +420,168 @@ func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
}) })
} }
} }
// failingFetcher answers every fetch with its error.
type failingFetcher struct {
err error
}
// Fetch returns f's error.
func (f failingFetcher) Fetch(
context.Context, imagerequest.Source,
) (*httpfetcher.FetchResult, error) {
return nil, f.err
}
// photoURLsByRoute returns, for each image route, the function that makes the
// URL of the JPEG at photoPath on signedHost on that route.
func photoURLsByRoute() map[string]func(*testing.T, *Handlers) string {
return map[string]func(*testing.T, *Handlers) string{
"/v1/image/": signedPhotoURL,
"/v1/e/": encPhotoURL,
}
}
// TestImageRoutes_SameErrorAnswers checks that both image routes answer each
// upstream fetch error below with the same status. They log it as "failed to
// get image" at the same level: error for the 500 and warn for the others.
// Each request gets routes of its own, so no request finds another's failed
// fetch in the negative cache.
func TestImageRoutes_SameErrorAnswers(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
wantStatus int
wantLevel slog.Level
}{
{"blocked upstream address", httpfetcher.ErrSSRFBlocked,
http.StatusForbidden, slog.LevelWarn},
{"upstream error status",
fmt.Errorf("%w: status 404", httpfetcher.ErrUpstreamError),
http.StatusBadGateway, slog.LevelWarn},
{"no response headers within the fetch timeout",
httpfetcher.ErrUpstreamTimeout, http.StatusGatewayTimeout,
slog.LevelWarn},
{"no free upstream connection", httpfetcher.ErrTooManyConnections,
http.StatusServiceUnavailable, slog.LevelWarn},
{"Content-Type not a source format",
fmt.Errorf("%w: image/svg+xml", httpfetcher.ErrInvalidContentType),
http.StatusUnsupportedMediaType, slog.LevelWarn},
{"any other failure", io.ErrUnexpectedEOF,
http.StatusInternalServerError, slog.LevelError},
}
for _, tt := range tests {
for route, imageURL := range photoURLsByRoute() {
t.Run(route+" "+tt.name, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h, routes := newRoutesFetchingThrough(t,
failingFetcher{err: tt.err},
slog.New(slog.NewJSONHandler(&logs, nil)))
rec := sendGet(t, routes, imageURL(t, h))
if rec.Code != tt.wantStatus {
t.Errorf("status = %d, want %d: %s",
rec.Code, tt.wantStatus, rec.Body)
}
levels := loggedLevels(t, &logs)
if !slices.Contains(levels[tt.wantLevel.String()],
"failed to get image") {
t.Errorf(`no "failed to get image" line logged at level %s`,
tt.wantLevel)
}
if tt.wantLevel != slog.LevelError &&
len(levels[slog.LevelError.String()]) != 0 {
t.Errorf("logged at error level: %q",
levels[slog.LevelError.String()])
}
})
}
}
}
// failingResponseWriter is a response writer whose Write always fails, as when
// the client has gone away.
type failingResponseWriter struct {
*httptest.ResponseRecorder
}
// Write writes nothing and returns io.ErrClosedPipe.
func (failingResponseWriter) Write([]byte) (int, error) {
return 0, io.ErrClosedPipe
}
// TestImageRoutes_FailedWriteLoggedOnce checks that both image routes log a
// failed write of the image once, at warn level, with the request's ID, the
// source host and path, the bytes served and the error. Neither may also log
// the image as served.
func TestImageRoutes_FailedWriteLoggedOnce(t *testing.T) {
t.Parallel()
const requestID = "test-request-id"
for route, imageURL := range photoURLsByRoute() {
t.Run(route, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h, srv := newSignedHostServer(t,
slog.New(slog.NewJSONHandler(&logs, nil)))
ctx := context.WithValue(t.Context(),
middleware.RequestIDKey, requestID)
srv.ServeHTTP(failingResponseWriter{httptest.NewRecorder()},
httptest.NewRequestWithContext(
ctx, http.MethodGet, imageURL(t, h), nil))
t.Logf("logged:\n%s", logs.String())
var failedWrites []map[string]any
dec := json.NewDecoder(&logs)
for dec.More() {
var line map[string]any
err := dec.Decode(&line)
if err != nil {
t.Fatalf("decoding log line: %v", err)
}
switch line["msg"] {
case "image served":
t.Error(`"image served" logged after a failed write`)
case "failed to write response":
failedWrites = append(failedWrites, line)
}
}
if len(failedWrites) != 1 {
t.Fatalf(`"failed to write response" logged %d times, want 1`,
len(failedWrites))
}
want := map[string]any{
"level": slog.LevelWarn.String(),
"request_id": requestID,
"host": signedHost,
"path": photoPath,
"served_bytes": float64(0),
"error": io.ErrClosedPipe.Error(),
}
for key, wantValue := range want {
if got := failedWrites[0][key]; got != wantValue {
t.Errorf("%s = %v, want %v", key, got, wantValue)
}
}
})
}
}
internal/handlers/imageenc.go
+5 -118
View File
@@ -2,21 +2,13 @@ package handlers
import ( import (
"errors" "errors"
"fmt"
"io"
"net/http" "net/http"
"strconv"
"time" "time"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic"
) )
// HandleImageEnc handles requests to /v1/e/{token}/* for encrypted // HandleImageEnc handles requests to /v1/e/{token}/* for encrypted
@@ -28,83 +20,12 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
return return
} }
ctx := r.Context()
start := time.Now()
req, expires, ok := s.parseImageEncRequest(w, r) req, expires, ok := s.parseImageEncRequest(w, r)
if !ok { if !ok {
return return
} }
format, ok := s.chooseAutoFormat(w, r, req.Transform.Format) s.serveImage(w, r, req, expires)
if !ok {
return
}
req.Transform.Format = format
// Log the request
s.log.Debug("encrypted image request",
"host", req.Source.Host,
"path", req.Source.Path,
"dimensions", fmt.Sprintf("%dx%d",
req.Transform.Size.Width, req.Transform.Size.Height),
"format", req.Transform.Format,
)
// Fetch and process the image (no signature validation
// needed - encrypted URL is trusted)
resp, err := s.imgSvc.Get(ctx, req)
if err != nil {
s.handleImageError(w, err)
return
}
defer func() { _ = resp.Content.Close() }()
// Set response headers
w.Header().Set("Content-Type", resp.ContentType)
if resp.ContentLength > 0 {
w.Header().Set("Content-Length", strconv.FormatInt(resp.ContentLength, 10))
}
// Cache headers: max-age ends at the URL's expiry
w.Header().Set("Cache-Control", cacheControl(expires))
w.Header().Set("X-Pixa-Cache", string(resp.CacheStatus))
if notModified(w, r, resp.ETag) {
return
}
// A HEAD request gets the headers only
if r.Method == http.MethodHead {
w.WriteHeader(http.StatusOK)
return
}
// Stream the response
written, err := io.Copy(w, resp.Content)
if err != nil {
s.log.Error("failed to write response", "error", err)
return
}
// Log completion
duration := time.Since(start)
s.log.Info("image served",
"request_id", middleware.GetReqID(ctx),
"cache_key", imagerequest.CacheKey(req),
"host", req.Source.Host,
"path", req.Source.Path,
"format", req.Transform.Format,
"cache_status", resp.CacheStatus,
"served_bytes", written,
"duration_ms", duration.Milliseconds(),
)
} }
} }
@@ -133,7 +54,7 @@ func (s *Handlers) parseImageEncRequest(
return nil, time.Time{}, false return nil, time.Time{}, false
} }
s.log.Debug("failed to decrypt URL", "error", err) s.log.Debug("invalid encrypted URL", "error", err)
s.respondError(w, "invalid encrypted URL", http.StatusBadRequest) s.respondError(w, "invalid encrypted URL", http.StatusBadRequest)
return nil, time.Time{}, false return nil, time.Time{}, false
@@ -141,8 +62,9 @@ func (s *Handlers) parseImageEncRequest(
// Apply the same dimension and fit-mode bounds as the plain image // Apply the same dimension and fit-mode bounds as the plain image
// route: a sealed payload is trusted for its origin, not for staying // route: a sealed payload is trusted for its origin, not for staying
// within limits, so an over-limit size or unknown fit mode is a 400 // within limits. This refuses an over-limit size before the source is
// here rather than an out-of-memory or a 500 from the processor. // fetched. An unknown fit mode is a 400 here rather than a 500 from the
// processor.
err = imagerequest.ValidateImageRequest(req) err = imagerequest.ValidateImageRequest(req)
if err != nil { if err != nil {
s.log.Debug("encrypted URL failed validation", "error", err) s.log.Debug("encrypted URL failed validation", "error", err)
@@ -154,38 +76,3 @@ func (s *Handlers) parseImageEncRequest(
return req, expires, true return req, expires, true
} }
// handleImageError converts image service errors to HTTP responses.
func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
switch {
case errors.Is(err, httpfetcher.ErrSSRFBlocked):
s.respondError(w, "forbidden", http.StatusForbidden)
case errors.Is(err, httpfetcher.ErrUpstreamError):
s.respondError(w, "upstream error", http.StatusBadGateway)
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
s.respondError(w, "upstream timeout", http.StatusGatewayTimeout)
case errors.Is(err, httpfetcher.ErrTooManyConnections),
errors.Is(err, imageprocessor.ErrTooManyImages),
errors.Is(err, imgcache.ErrProcessingTimedOut):
s.respondError(w, "server busy, try again later",
http.StatusServiceUnavailable)
case errors.Is(err, httpfetcher.ErrInvalidContentType),
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat),
errors.Is(err, magic.ErrMagicByteMismatch),
errors.Is(err, magic.ErrUnknownFormat),
errors.Is(err, magic.ErrNotEnoughData):
s.respondError(w, "unsupported source format",
http.StatusUnsupportedMediaType)
case errors.Is(err, imageprocessor.ErrOutputTooLarge):
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d",
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
http.StatusBadRequest)
default:
s.log.Error("image request failed", "error", err)
s.respondError(w, "internal error", http.StatusInternalServerError)
}
}
internal/handlers/imageenc_internal_test.go
+5 -2
View File
@@ -12,6 +12,7 @@ import (
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
) )
// newEncTestServer builds a router serving the encrypted-URL route with a // newEncTestServer builds a router serving the encrypted-URL route with a
@@ -21,11 +22,13 @@ import (
func newEncTestServer(t *testing.T) (*encurl.Generator, http.Handler) { func newEncTestServer(t *testing.T) (*encurl.Generator, http.Handler) {
t.Helper() t.Helper()
encGen, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{ h := &Handlers{
log: slog.New(slog.DiscardHandler), log: slog.New(slog.DiscardHandler),
encGen: encGen, encGen: encGen,
internal/handlers/imageenc_token_internal_test.go
+5 -2
View File
@@ -10,6 +10,7 @@ import (
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
) )
// requireServedPhoto requires that rec answers 200 with the JPEG at photoPath // requireServedPhoto requires that rec answers 200 with the JPEG at photoPath
@@ -80,11 +81,13 @@ func TestHandleImageEnc_RejectedToken(t *testing.T) {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
} }
otherGen, err := encurl.NewGenerator("another-signing-key-fedcba9876543210") keys, err := seal.DeriveKeys("another-signing-key-fedcba9876543210")
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
otherGen := encurl.NewGenerator(keys.EncryptedURLKey)
otherKey, err := otherGen.Generate(&photo, time.Time{}) otherKey, err := otherGen.Generate(&photo, time.Time{})
if err != nil { if err != nil {
t.Fatalf("Generate() error = %v", err) t.Fatalf("Generate() error = %v", err)
internal/handlers/referer_blocklist_internal_test.go
+10 -6
View File
@@ -15,8 +15,10 @@ import (
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature" "sneak.berlin/go/pixa/internal/signature"
) )
@@ -51,7 +53,7 @@ func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher)
Fetcher: newPhotoFetcher(t, allowlistedHost, signedHost), Fetcher: newPhotoFetcher(t, allowlistedHost, signedHost),
} }
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{ cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(), StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30, MaxBytes: 1 << 30,
@@ -60,7 +62,7 @@ func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher)
t.Fatalf("imgcache.NewCache() error = %v", err) t.Fatalf("imgcache.NewCache() error = %v", err)
} }
svc, err := imgcache.NewService(&imgcache.ServiceConfig{ svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache, Cache: cache,
Fetcher: fetcher, Fetcher: fetcher,
Processor: imageprocessor.New(imageprocessor.Params{ Processor: imageprocessor.New(imageprocessor.Params{
@@ -68,16 +70,18 @@ func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher)
MaxConcurrentProcessing: 2, MaxConcurrentProcessing: 2,
}), }),
MaxResponseSize: 50 << 20, MaxResponseSize: 50 << 20,
}) }, prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("imgcache.NewService() error = %v", err) t.Fatalf("imageproxy.New() error = %v", err)
} }
encGen, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{ h := &Handlers{
log: slog.New(slog.DiscardHandler), log: slog.New(slog.DiscardHandler),
imgSvc: svc, imgSvc: svc,
internal/handlers/request_id_internal_test.go
+2 -2
View File
@@ -18,8 +18,8 @@ import (
// signedPhotoURL returns a signed /v1/image/ URL, valid for a minute, for the // signedPhotoURL returns a signed /v1/image/ URL, valid for a minute, for the
// JPEG at photoPath on signedHost at 50x50. It is signed with testSigningKey, // JPEG at photoPath on signedHost at 50x50. It is signed with testSigningKey,
// the key of the image service newSignedHostServer builds. It takes a Handlers // the key the handlers that newSignedHostServer builds check signatures with.
// it does not use, so that it has the type of encPhotoURL. // It takes a Handlers it does not use, so that it has the type of encPhotoURL.
func signedPhotoURL(t *testing.T, _ *Handlers) string { func signedPhotoURL(t *testing.T, _ *Handlers) string {
t.Helper() t.Helper()
internal/handlers/server_busy_internal_test.go
+30 -46
View File
@@ -11,13 +11,14 @@ import (
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
) )
// TestServerBusyAnswers503 checks that both image routes answer 503 with // TestServerBusyAnswers503 checks that respondImageError, which answers the
// a clear error when the image service gives up waiting for a free upstream // errors of both image routes, answers 503 with a clear error when the image
// connection or processing slot, wrapped as the service wraps them. // service gives up waiting for a free upstream connection or processing slot,
// wrapped as the service wraps them.
func TestServerBusyAnswers503(t *testing.T) { func TestServerBusyAnswers503(t *testing.T) {
t.Parallel() t.Parallel()
@@ -30,61 +31,44 @@ func TestServerBusyAnswers503(t *testing.T) {
fmt.Errorf("upstream fetch failed: %w", httpfetcher.ErrTooManyConnections), fmt.Errorf("upstream fetch failed: %w", httpfetcher.ErrTooManyConnections),
fmt.Errorf("image processing failed: %w", imageprocessor.ErrTooManyImages), fmt.Errorf("image processing failed: %w", imageprocessor.ErrTooManyImages),
} { } {
plain := httptest.NewRecorder() rec := httptest.NewRecorder()
h.respondImageError(plain, req, err) h.respondImageError(rec, req, err)
encrypted := httptest.NewRecorder() if rec.Code != http.StatusServiceUnavailable {
h.handleImageError(encrypted, err) t.Errorf("for %v: status = %d, want %d",
err, rec.Code, http.StatusServiceUnavailable)
}
for route, rec := range map[string]*httptest.ResponseRecorder{ if !strings.Contains(rec.Body.String(), "server busy, try again later") {
"/v1/image/": plain, "/v1/e/": encrypted, t.Errorf("for %v: body = %q, want the server busy error",
} { err, rec.Body.String())
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("%s for %v: status = %d, want %d",
route, err, rec.Code, http.StatusServiceUnavailable)
}
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("%s for %v: body = %q, want the server busy error",
route, err, rec.Body.String())
}
} }
} }
} }
// TestProcessingTimedOutAnswers503 checks that respondImageError and // TestProcessingTimedOutAnswers503 checks that respondImageError, which
// handleImageError, which answer the errors of the two image routes, answer // answers the errors of both image routes, answers 503 with the server busy
// 503 with the server busy error when the image service reports that // error when the image service reports that processing stopped at the
// processing stopped at the deadline of the request doing it, wrapped as the // deadline of the request doing it, wrapped as the service wraps it. A request
// service wraps it. A request that waited for that processing gets this error // that waited for that processing gets this error while its own deadline may
// while its own deadline may have time left. The request's source only goes // have time left. The request's source only goes into a log line, so it is
// into a log line, so it is left empty. // left empty.
func TestProcessingTimedOutAnswers503(t *testing.T) { func TestProcessingTimedOutAnswers503(t *testing.T) {
t.Parallel() t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)} h := &Handlers{log: slog.New(slog.DiscardHandler)}
err := fmt.Errorf("%w: failed to encode: %w", err := fmt.Errorf("%w: failed to encode: %w",
imgcache.ErrProcessingTimedOut, context.DeadlineExceeded) imageproxy.ErrProcessingTimedOut, context.DeadlineExceeded)
plain := httptest.NewRecorder() rec := httptest.NewRecorder()
h.respondImageError(plain, &imagerequest.Request{}, err) h.respondImageError(rec, &imagerequest.Request{}, err)
t.Logf("status = %d, body = %q", rec.Code, rec.Body.String())
encrypted := httptest.NewRecorder() if rec.Code != http.StatusServiceUnavailable {
h.handleImageError(encrypted, err) t.Errorf("status = %d, want %d", rec.Code, http.StatusServiceUnavailable)
}
for name, rec := range map[string]*httptest.ResponseRecorder{ if !strings.Contains(rec.Body.String(), "server busy, try again later") {
"respondImageError": plain, "handleImageError": encrypted, t.Errorf("body = %q, want the server busy error", rec.Body.String())
} {
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("%s: status = %d, want %d",
name, rec.Code, http.StatusServiceUnavailable)
}
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("%s: body = %q, want the server busy error",
name, rec.Body.String())
}
} }
} }
internal/handlers/unsupported_source_internal_test.go
+16 -23
View File
@@ -13,10 +13,11 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestUnsupportedSourceFormatAnswers415 checks that both image routes answer // TestUnsupportedSourceFormatAnswers415 checks that respondImageError, which
// 415 with a clear error when the upstream answers with a content type pixa // answers the errors of both image routes, answers 415 with a clear error when
// does not accept, such as SVG, and when orig is asked of a source in a // the upstream answers with a content type pixa does not accept, such as SVG,
// format the processor cannot name, wrapped as the service wraps them. // and when libvips does not read a source as the format its first bytes
// show, wrapped as the service wraps them.
func TestUnsupportedSourceFormatAnswers415(t *testing.T) { func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
t.Parallel() t.Parallel()
@@ -28,29 +29,21 @@ func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
for _, err := range []error{ for _, err := range []error{
fmt.Errorf("upstream fetch failed: %w: image/svg+xml", fmt.Errorf("upstream fetch failed: %w: image/svg+xml",
httpfetcher.ErrInvalidContentType), httpfetcher.ErrInvalidContentType),
fmt.Errorf("image processing failed: %w: unknown", fmt.Errorf("image processing failed: %w: libvips does not read the input as avif",
imageprocessor.ErrUnsupportedInputFormat), imageprocessor.ErrUnsupportedInputFormat),
} { } {
plain := httptest.NewRecorder() rec := httptest.NewRecorder()
h.respondImageError(plain, req, err) h.respondImageError(rec, req, err)
t.Logf("for %v: %d %s", err, rec.Code, rec.Body.String())
encrypted := httptest.NewRecorder() if rec.Code != http.StatusUnsupportedMediaType {
h.handleImageError(encrypted, err) t.Errorf("for %v: status = %d, want %d",
err, rec.Code, http.StatusUnsupportedMediaType)
}
for route, rec := range map[string]*httptest.ResponseRecorder{ if !strings.Contains(rec.Body.String(), "unsupported source format") {
"HandleImage": plain, "HandleImageEnc": encrypted, t.Errorf("for %v: body = %q, want the unsupported source format error",
} { err, rec.Body.String())
t.Logf("%s for %v: %d %s", route, err, rec.Code, rec.Body.String())
if rec.Code != http.StatusUnsupportedMediaType {
t.Errorf("%s for %v: status = %d, want %d",
route, err, rec.Code, http.StatusUnsupportedMediaType)
}
if !strings.Contains(rec.Body.String(), "unsupported source format") {
t.Errorf("%s for %v: body = %q, want the unsupported source format error",
route, err, rec.Body.String())
}
} }
} }
} }
internal/httpfetcher/fetch_internal_test.go
+3 -3
View File
@@ -83,7 +83,7 @@ func startUpstream(t *testing.T) *httptest.Server {
mux := http.NewServeMux() mux := http.NewServeMux()
mux.HandleFunc("/image", func(w http.ResponseWriter, _ *http.Request) { mux.HandleFunc("/image", func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", contentTypeJPEG) w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
_, _ = io.WriteString(w, imagePayload) _, _ = io.WriteString(w, imagePayload)
}) })
mux.HandleFunc("/status/500", func(w http.ResponseWriter, _ *http.Request) { mux.HandleFunc("/status/500", func(w http.ResponseWriter, _ *http.Request) {
@@ -358,7 +358,7 @@ func TestFetchContentLengthBodyAtMaxResponseSize(t *testing.T) {
res := fetchWithMaxResponseSize(t, maxResponseSize, res := fetchWithMaxResponseSize(t, maxResponseSize,
func(w http.ResponseWriter, _ *http.Request) { func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", contentTypeJPEG) w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
w.Header().Set("Content-Length", strconv.Itoa(tc.size)) w.Header().Set("Content-Length", strconv.Itoa(tc.size))
_, _ = io.WriteString(w, body) _, _ = io.WriteString(w, body)
}) })
@@ -406,7 +406,7 @@ func TestFetchChunkedBodyAtMaxResponseSize(t *testing.T) {
res := fetchWithMaxResponseSize(t, maxResponseSize, res := fetchWithMaxResponseSize(t, maxResponseSize,
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", contentTypeJPEG) w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
_, _ = io.WriteString(w, body) _, _ = io.WriteString(w, body)
// With no Content-Length, flushing sends the body // With no Content-Length, flushing sends the body
// chunked; its end is sent when the handler returns. // chunked; its end is sent when the handler returns.
internal/httpfetcher/httpfetcher.go
+8 -19
View File
@@ -38,18 +38,6 @@ const userAgent = "pixa/1.0"
// MaxConnections fetches are already in progress. // MaxConnections fetches are already in progress.
const ConnectionWaitTimeout = 10 * time.Second const ConnectionWaitTimeout = 10 * time.Second
// MIME content types.
const (
contentTypeJPEG = "image/jpeg"
contentTypePNG = "image/png"
contentTypeGIF = "image/gif"
contentTypeWebP = "image/webp"
contentTypeAVIF = "image/avif"
contentTypeJXL = "image/jxl"
contentTypeSVG = "image/svg+xml"
contentTypeOctetStream = "application/octet-stream"
)
// Loopback addresses blocked by SSRF protection. // Loopback addresses blocked by SSRF protection.
const ( const (
localhostIPv4 = "127.0.0.1" localhostIPv4 = "127.0.0.1"
@@ -121,15 +109,16 @@ type FetchResult struct {
} }
// allowedContentTypes returns the content types an upstream response may // allowedContentTypes returns the content types an upstream response may
// have, which Fetch also sends as its Accept header. // have, which Fetch also sends as its Accept header: the MIME types of the
// formats pixa reads.
func allowedContentTypes() []string { func allowedContentTypes() []string {
return []string{ return []string{
contentTypeJPEG, imagerequest.FormatJPEG.MIMEType(),
contentTypePNG, imagerequest.FormatPNG.MIMEType(),
contentTypeGIF, imagerequest.FormatGIF.MIMEType(),
contentTypeWebP, imagerequest.FormatWebP.MIMEType(),
contentTypeAVIF, imagerequest.FormatAVIF.MIMEType(),
contentTypeJXL, imagerequest.FormatJXL.MIMEType(),
} }
} }
internal/httpfetcher/httpfetcher_internal_test.go
+11 -11
View File
@@ -30,9 +30,9 @@ func TestIsAllowedContentType(t *testing.T) {
contentType string contentType string
want bool want bool
}{ }{
{contentTypeJPEG, true}, {imagerequest.FormatJPEG.MIMEType(), true},
{contentTypePNG, true}, {imagerequest.FormatPNG.MIMEType(), true},
{contentTypeWebP, true}, {imagerequest.FormatWebP.MIMEType(), true},
{"image/jpeg; charset=utf-8", true}, {"image/jpeg; charset=utf-8", true},
{"IMAGE/JPEG", true}, {"IMAGE/JPEG", true},
{"text/html", false}, {"text/html", false},
@@ -217,7 +217,7 @@ func TestMockFetcher_FetchesFile(t *testing.T) {
defer func() { _ = result.Content.Close() }() defer func() { _ = result.Content.Close() }()
if result.ContentType != contentTypeJPEG { if result.ContentType != imagerequest.FormatJPEG.MIMEType() {
t.Errorf("ContentType = %q, want image/jpeg", result.ContentType) t.Errorf("ContentType = %q, want image/jpeg", result.ContentType)
} }
@@ -272,13 +272,13 @@ func TestDetectContentTypeFromPath(t *testing.T) {
path string path string
want string want string
}{ }{
{"foo/bar.jpg", contentTypeJPEG}, {"foo/bar.jpg", imagerequest.FormatJPEG.MIMEType()},
{"foo/bar.JPG", contentTypeJPEG}, {"foo/bar.JPG", imagerequest.FormatJPEG.MIMEType()},
{"foo/bar.jpeg", contentTypeJPEG}, {"foo/bar.jpeg", imagerequest.FormatJPEG.MIMEType()},
{"foo/bar.png", contentTypePNG}, {"foo/bar.png", imagerequest.FormatPNG.MIMEType()},
{"foo/bar.gif", contentTypeGIF}, {"foo/bar.gif", imagerequest.FormatGIF.MIMEType()},
{"foo/bar.webp", contentTypeWebP}, {"foo/bar.webp", imagerequest.FormatWebP.MIMEType()},
{"foo/bar.avif", contentTypeAVIF}, {"foo/bar.avif", imagerequest.FormatAVIF.MIMEType()},
{"foo/bar.svg", contentTypeSVG}, {"foo/bar.svg", contentTypeSVG},
{"foo/bar.bin", contentTypeOctetStream}, {"foo/bar.bin", contentTypeOctetStream},
{"foo/bar", contentTypeOctetStream}, {"foo/bar", contentTypeOctetStream},
internal/httpfetcher/mock.go
+12 -6
View File
@@ -66,23 +66,29 @@ func (m *MockFetcher) Fetch(
}, nil }, nil
} }
// The content types the mock fetcher gives files in no format pixa reads.
const (
contentTypeSVG = "image/svg+xml"
contentTypeOctetStream = "application/octet-stream"
)
// detectContentTypeFromPath returns the MIME type based on file extension. // detectContentTypeFromPath returns the MIME type based on file extension.
func detectContentTypeFromPath(path string) string { func detectContentTypeFromPath(path string) string {
path = strings.ToLower(path) path = strings.ToLower(path)
switch { switch {
case strings.HasSuffix(path, ".jpg"), strings.HasSuffix(path, ".jpeg"): case strings.HasSuffix(path, ".jpg"), strings.HasSuffix(path, ".jpeg"):
return contentTypeJPEG return imagerequest.FormatJPEG.MIMEType()
case strings.HasSuffix(path, ".png"): case strings.HasSuffix(path, ".png"):
return contentTypePNG return imagerequest.FormatPNG.MIMEType()
case strings.HasSuffix(path, ".gif"): case strings.HasSuffix(path, ".gif"):
return contentTypeGIF return imagerequest.FormatGIF.MIMEType()
case strings.HasSuffix(path, ".webp"): case strings.HasSuffix(path, ".webp"):
return contentTypeWebP return imagerequest.FormatWebP.MIMEType()
case strings.HasSuffix(path, ".avif"): case strings.HasSuffix(path, ".avif"):
return contentTypeAVIF return imagerequest.FormatAVIF.MIMEType()
case strings.HasSuffix(path, ".jxl"): case strings.HasSuffix(path, ".jxl"):
return contentTypeJXL return imagerequest.FormatJXL.MIMEType()
case strings.HasSuffix(path, ".svg"): case strings.HasSuffix(path, ".svg"):
return contentTypeSVG return contentTypeSVG
default: default:
internal/httpfetcher/request_id_internal_test.go
+2 -1
View File
@@ -8,6 +8,7 @@ import (
"testing" "testing"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestFetchSendsRequestID verifies that a fetch sends the ID of the request // TestFetchSendsRequestID verifies that a fetch sends the ID of the request
@@ -25,7 +26,7 @@ func TestFetchSendsRequestID(t *testing.T) {
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
received <- r.Header.Get("X-Request-Id") received <- r.Header.Get("X-Request-Id")
w.Header().Set("Content-Type", contentTypeJPEG) w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
_, _ = io.WriteString(w, imagePayload) _, _ = io.WriteString(w, imagePayload)
})) }))
t.Cleanup(srv.Close) t.Cleanup(srv.Close)
internal/imageprocessor/avif_too_large_internal_test.go
+17 -16
View File
@@ -39,30 +39,31 @@ func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
input []byte input []byte
sourceFormat imagerequest.Format
format imagerequest.Format format imagerequest.Format
width, height int width, height int
fit imagerequest.Fit fit imagerequest.Fit
refused bool refused bool
}{ }{
{"AVIF at the limit", cutOff, imagerequest.FormatAVIF, {"AVIF at the limit", cutOff, imagerequest.FormatPNG,
4096, 4096, imagerequest.FitCover, false}, imagerequest.FormatAVIF, 4096, 4096, imagerequest.FitCover, false},
{"AVIF above the limit", cutOff, imagerequest.FormatAVIF, {"AVIF above the limit", cutOff, imagerequest.FormatPNG,
4096, 4097, imagerequest.FitCover, true}, imagerequest.FormatAVIF, 4096, 4097, imagerequest.FitCover, true},
{"WebP above the AVIF limit", cutOff, imagerequest.FormatWebP, {"WebP above the AVIF limit", cutOff, imagerequest.FormatPNG,
4096, 4097, imagerequest.FitCover, false}, imagerequest.FormatWebP, 4096, 4097, imagerequest.FitCover, false},
// FitInside does not enlarge, so the output stays 64x32 // FitInside does not enlarge, so the output stays 64x32
{"AVIF inside a larger size", cutOff, imagerequest.FormatAVIF, {"AVIF inside a larger size", cutOff, imagerequest.FormatPNG,
8192, 8192, imagerequest.FitInside, false}, imagerequest.FormatAVIF, 8192, 8192, imagerequest.FitInside, false},
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096 // FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
{"AVIF outside a smaller size", cutOff, imagerequest.FormatAVIF, {"AVIF outside a smaller size", cutOff, imagerequest.FormatPNG,
8192, 1, imagerequest.FitOutside, true}, imagerequest.FormatAVIF, 8192, 1, imagerequest.FitOutside, true},
// With no size, the output keeps the source's 4097x4096 // With no size, the output keeps the source's 4097x4096
{"AVIF of the source's size", largeAVIF, imagerequest.FormatAVIF, {"AVIF of the source's size", largeAVIF, imagerequest.FormatAVIF,
0, 0, imagerequest.FitCover, true}, imagerequest.FormatAVIF, 0, 0, imagerequest.FitCover, true},
{"orig of an AVIF source", largeAVIF, imagerequest.FormatOriginal, {"orig of an AVIF source", largeAVIF, imagerequest.FormatAVIF,
0, 0, imagerequest.FitCover, true}, imagerequest.FormatOriginal, 0, 0, imagerequest.FitCover, true},
{"JPEG of the source's size", largeAVIF, imagerequest.FormatJPEG, {"JPEG of the source's size", largeAVIF, imagerequest.FormatAVIF,
0, 0, imagerequest.FitCover, false}, imagerequest.FormatJPEG, 0, 0, imagerequest.FitCover, false},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -72,7 +73,7 @@ func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
_, err := New(Params{ _, err := New(Params{
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process(t.Context(), bytes.NewReader(tt.input), }).Process(t.Context(), bytes.NewReader(tt.input), tt.sourceFormat,
&imagerequest.Transform{ &imagerequest.Transform{
Size: imagerequest.Size{Width: tt.width, Height: tt.height}, Size: imagerequest.Size{Width: tt.width, Height: tt.height},
Format: tt.format, Format: tt.format,
internal/imageprocessor/computed_side_internal_test.go
+2 -2
View File
@@ -42,7 +42,7 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process(t.Context(), }).Process(t.Context(),
bytes.NewReader(input), bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{ &imagerequest.Transform{
Size: tt.size, Size: tt.size,
Format: imagerequest.FormatJPEG, Format: imagerequest.FormatJPEG,
@@ -111,7 +111,7 @@ func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process(t.Context(), }).Process(t.Context(),
bytes.NewReader(input), bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{ &imagerequest.Transform{
Size: tt.size, Size: tt.size,
Format: imagerequest.FormatJPEG, Format: imagerequest.FormatJPEG,
internal/imageprocessor/deadline_internal_test.go
+4 -1
View File
@@ -7,6 +7,8 @@ import (
"io" "io"
"testing" "testing"
"time" "time"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// readerAfterDeadline is a Process input whose first Read waits until done // readerAfterDeadline is a Process input whose first Read waits until done
@@ -86,7 +88,8 @@ func checkProcessStopsAtDeadline(
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1}) proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
result, err := proc.Process(ctx, input, smallJPEGTransform()) result, err := proc.Process(ctx, input, imagerequest.FormatJPEG,
smallJPEGTransform())
if result != nil { if result != nil {
t.Logf("Process() made a %dx%d image after its deadline passed", t.Logf("Process() made a %dx%d image after its deadline passed",
result.Width, result.Height) result.Width, result.Height)
internal/imageprocessor/empty_format_internal_test.go
+1 -1
View File
@@ -19,7 +19,7 @@ func TestImageProcessor_EmptyFormatRefused(t *testing.T) {
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process( }).Process(
t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)), t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)),
&imagerequest.Transform{}, imagerequest.FormatJPEG, &imagerequest.Transform{},
) )
if !errors.Is(err, ErrUnsupportedOutputFormat) { if !errors.Is(err, ErrUnsupportedOutputFormat) {
t.Errorf("Process() error = %v, want %v", err, ErrUnsupportedOutputFormat) t.Errorf("Process() error = %v, want %v", err, ErrUnsupportedOutputFormat)
internal/imageprocessor/encoder_settings_internal_test.go
+9 -8
View File
@@ -11,17 +11,18 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
) )
// processedSize runs input through Process and returns the output's size in // processedSize runs input, an image in sourceFormat, through Process and
// bytes. // returns the output's size in bytes.
func processedSize( func processedSize(
t *testing.T, input []byte, transform *imagerequest.Transform, t *testing.T, input []byte, sourceFormat imagerequest.Format,
transform *imagerequest.Transform,
) int64 { ) int64 {
t.Helper() t.Helper()
result, err := New(Params{ result, err := New(Params{
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process(t.Context(), bytes.NewReader(input), transform) }).Process(t.Context(), bytes.NewReader(input), sourceFormat, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -74,7 +75,7 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
} }
} }
size := processedSize(t, encodePNG(t, flat), size := processedSize(t, encodePNG(t, flat), imagerequest.FormatPNG,
&imagerequest.Transform{Format: imagerequest.FormatPNG}) &imagerequest.Transform{Format: imagerequest.FormatPNG})
const rawBytes = width * height * 3 const rawBytes = width * height * 3
@@ -92,7 +93,7 @@ func TestImageProcessor_WebPAtEffort2(t *testing.T) {
input := createTestJPEG(t, 200, 150) input := createTestJPEG(t, 200, 150)
size := processedSize(t, input, size := processedSize(t, input, imagerequest.FormatJPEG,
&imagerequest.Transform{Format: imagerequest.FormatWebP, Quality: 85}) &imagerequest.Transform{Format: imagerequest.FormatWebP, Quality: 85})
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{ want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
@@ -134,7 +135,7 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
input := encodePNG(t, source) input := encodePNG(t, source)
size := processedSize(t, input, size := processedSize(t, input, imagerequest.FormatPNG,
&imagerequest.Transform{Format: imagerequest.FormatAVIF, Quality: 85}) &imagerequest.Transform{Format: imagerequest.FormatAVIF, Quality: 85})
want, _, err := decode(t, input).ExportAvif(&vips.AvifExportParams{ want, _, err := decode(t, input).ExportAvif(&vips.AvifExportParams{
@@ -161,7 +162,7 @@ func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
input := createTestJPEG(t, 200, 150) input := createTestJPEG(t, 200, 150)
size := processedSize(t, input, size := processedSize(t, input, imagerequest.FormatJPEG,
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: 85}) &imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: 85})
params := vips.NewJxlExportParams() params := vips.NewJxlExportParams()
internal/imageprocessor/fit_cover_internal_test.go
+3 -3
View File
@@ -40,7 +40,7 @@ func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process(t.Context(), }).Process(t.Context(),
bytes.NewReader(input), bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{ &imagerequest.Transform{
Size: tt.size, Size: tt.size,
Format: imagerequest.FormatJPEG, Format: imagerequest.FormatJPEG,
@@ -113,7 +113,7 @@ func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process(t.Context(), }).Process(t.Context(),
bytes.NewReader(input), bytes.NewReader(input), imagerequest.FormatPNG,
&imagerequest.Transform{ &imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50}, Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatPNG, Format: imagerequest.FormatPNG,
@@ -240,7 +240,7 @@ func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process(t.Context(), }).Process(t.Context(),
bytes.NewReader(input), bytes.NewReader(input), imagerequest.FormatPNG,
&imagerequest.Transform{ &imagerequest.Transform{
Size: tt.size, Size: tt.size,
Format: imagerequest.FormatPNG, Format: imagerequest.FormatPNG,
internal/imageprocessor/fit_outside_internal_test.go
+2 -2
View File
@@ -45,7 +45,7 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process(t.Context(), }).Process(t.Context(),
bytes.NewReader(input), bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{ &imagerequest.Transform{
Size: tt.size, Size: tt.size,
Format: imagerequest.FormatJPEG, Format: imagerequest.FormatJPEG,
@@ -102,7 +102,7 @@ func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process(t.Context(), }).Process(t.Context(),
bytes.NewReader(input), bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{ &imagerequest.Transform{
Size: tt.size, Size: tt.size,
Format: imagerequest.FormatJPEG, Format: imagerequest.FormatJPEG,
internal/imageprocessor/imageprocessor.go
+55 -92
View File
@@ -70,8 +70,6 @@ type Result struct {
InputWidth int InputWidth int
// InputHeight is the original image height before processing. // InputHeight is the original image height before processing.
InputHeight int InputHeight int
// InputFormat is the detected input format (e.g., "jpeg", "png").
InputFormat string
} }
// MaxInputDimension is the maximum allowed width or height for input images. // MaxInputDimension is the maximum allowed width or height for input images.
@@ -113,9 +111,10 @@ var ErrAVIFOutputTooLarge = errors.New("AVIF output has too many pixels")
var ErrUnsupportedOutputFormat = errors.New("unsupported output format") var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
// ErrUnsupportedInputFormat is returned, before the input is decoded, when // ErrUnsupportedInputFormat is returned, before the input is decoded, when
// libvips reads the input's first bytes as another format than the source // libvips reads the input's first bytes as another format than its source
// formats, or as none: for example an SVG whose first bytes pass the // format, or as none. An SVG whose first bytes pass the first-bytes check for
// first-bytes check for AVIF. // AVIF is one example. A source format other than the six pixa reads is also
// refused with it.
var ErrUnsupportedInputFormat = errors.New("unsupported input format") var ErrUnsupportedInputFormat = errors.New("unsupported input format")
// ErrTooManyImages is returned when MaxConcurrentProcessing images are being // ErrTooManyImages is returned when MaxConcurrentProcessing images are being
@@ -157,9 +156,10 @@ func New(params Params) *ImageProcessor {
} }
} }
// Process transforms an image according to transform. When // Process transforms the image read from input according to transform.
// MaxConcurrentProcessing images are already being processed, it waits up // sourceFormat is the input's format, which magic detected from its first
// to ProcessingWaitTimeout for one to finish, then fails with // bytes. When MaxConcurrentProcessing images are already being processed,
// it waits up to ProcessingWaitTimeout for one to finish, then fails with
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it // ErrTooManyImages. When ctx has ended once Process has a slot, or before it
// encodes, it returns ctx's error and gives the slot back. govips gives pixa // encodes, it returns ctx's error and gives the slot back. govips gives pixa
// no way to stop libvips once an encode has begun, so an encode keeps its // no way to stop libvips once an encode has begun, so an encode keeps its
@@ -168,6 +168,7 @@ func New(params Params) *ImageProcessor {
func (p *ImageProcessor) Process( func (p *ImageProcessor) Process(
ctx context.Context, ctx context.Context,
input io.Reader, input io.Reader,
sourceFormat imagerequest.Format,
transform *imagerequest.Transform, transform *imagerequest.Transform,
) (*Result, error) { ) (*Result, error) {
release, err := p.acquireSlot(ctx) release, err := p.acquireSlot(ctx)
@@ -182,13 +183,9 @@ func (p *ImageProcessor) Process(
return nil, err return nil, err
} }
// govips picks the libvips loader from the input's first bytes with // Input libvips would not load as sourceFormat is refused before it is
// DetermineImageType. Input whose first bytes it reads as no source // decoded
// format, such as SVG, which librsvg would render, is refused before err = checkInputFormat(data, sourceFormat)
// libvips reads it.
inputFormat := p.detectFormat(vips.DetermineImageType(data))
sourceFormat, err := p.formatFromString(inputFormat)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -254,12 +251,11 @@ func (p *ImageProcessor) Process(
return &Result{ return &Result{
Content: io.NopCloser(bytes.NewReader(output)), Content: io.NopCloser(bytes.NewReader(output)),
ContentLength: int64(len(output)), ContentLength: int64(len(output)),
ContentType: FormatToMIME(outputFormat), ContentType: outputFormat.MIMEType(),
Width: img.Width(), Width: img.Width(),
Height: img.Height(), Height: img.Height(),
InputWidth: origWidth, InputWidth: origWidth,
InputHeight: origHeight, InputHeight: origHeight,
InputFormat: inputFormat,
}, nil }, nil
} }
@@ -294,38 +290,6 @@ func targetDimensions(
return width, height, nil return width, height, nil
} }
// MIME types for the supported image formats.
const (
mimeJPEG = "image/jpeg"
mimePNG = "image/png"
mimeGIF = "image/gif"
mimeWebP = "image/webp"
mimeAVIF = "image/avif"
mimeJXL = "image/jxl"
)
// FormatToMIME converts a Format to its MIME type string.
func FormatToMIME(format imagerequest.Format) string {
switch format {
case imagerequest.FormatJPEG:
return mimeJPEG
case imagerequest.FormatPNG:
return mimePNG
case imagerequest.FormatWebP:
return mimeWebP
case imagerequest.FormatGIF:
return mimeGIF
case imagerequest.FormatAVIF:
return mimeAVIF
case imagerequest.FormatJXL:
return mimeJXL
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
return "application/octet-stream"
default:
return "application/octet-stream"
}
}
// WaitForProcessing waits until no image is being processed, or until ctx // WaitForProcessing waits until no image is being processed, or until ctx
// ends, and returns how many images were still being processed then. It // ends, and returns how many images were still being processed then. It
// waits by taking each slot in processingSemaphore as it frees up until it // waits by taking each slot in processingSemaphore as it frees up until it
@@ -394,27 +358,49 @@ func (p *ImageProcessor) readInput(input io.Reader) ([]byte, error) {
return data, nil return data, nil
} }
// detectFormat returns the format string for a govips image type. // checkInputFormat returns ErrUnsupportedInputFormat unless libvips would
func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string { // load data with the loader of sourceFormat. govips picks the libvips loader
switch imageType { // from the input's first bytes with DetermineImageType. It refuses an SVG
case vips.ImageTypeJPEG: // whose first bytes pass the first-bytes check for AVIF, which librsvg would
return "jpeg" // otherwise render.
case vips.ImageTypePNG: func checkInputFormat(data []byte, sourceFormat imagerequest.Format) error {
return "png" // A format with no loader, such as orig or auto, has the type govips also
case vips.ImageTypeGIF: // gives input it cannot read, which the check below would let through.
return "gif" sourceType := vipsImageType(sourceFormat)
case vips.ImageTypeWEBP: if sourceType == vips.ImageTypeUnknown {
return "webp" return fmt.Errorf("%w: %q is not a source format",
case vips.ImageTypeAVIF, vips.ImageTypeHEIF: ErrUnsupportedInputFormat, sourceFormat)
return string(imagerequest.FormatAVIF) }
case vips.ImageTypeJXL:
return string(imagerequest.FormatJXL) if vips.DetermineImageType(data) != sourceType {
case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF, return fmt.Errorf("%w: libvips does not read the input as %s",
vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP, ErrUnsupportedInputFormat, sourceFormat)
vips.ImageTypeJP2K: }
return "unknown"
return nil
}
// vipsImageType returns the govips image type whose libvips loader reads
// format. orig and auto stand for another format, so they have
// vips.ImageTypeUnknown, which govips has no loader for.
func vipsImageType(format imagerequest.Format) vips.ImageType {
switch format {
case imagerequest.FormatJPEG:
return vips.ImageTypeJPEG
case imagerequest.FormatPNG:
return vips.ImageTypePNG
case imagerequest.FormatGIF:
return vips.ImageTypeGIF
case imagerequest.FormatWebP:
return vips.ImageTypeWEBP
case imagerequest.FormatAVIF:
return vips.ImageTypeAVIF
case imagerequest.FormatJXL:
return vips.ImageTypeJXL
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
return vips.ImageTypeUnknown
default: default:
return "unknown" return vips.ImageTypeUnknown
} }
} }
@@ -768,26 +754,3 @@ func jxlDistance(quality int) float64 {
return 53.0/3000.0*q*q - 23.0/20.0*q + 25.0 return 53.0/3000.0*q*q - 23.0/20.0*q + 25.0
} }
// formatFromString converts a format string from detectFormat to Format. A
// format detectFormat could not name is ErrUnsupportedInputFormat.
func (p *ImageProcessor) formatFromString(
format string,
) (imagerequest.Format, error) {
switch format {
case "jpeg":
return imagerequest.FormatJPEG, nil
case "png":
return imagerequest.FormatPNG, nil
case "gif":
return imagerequest.FormatGIF, nil
case "webp":
return imagerequest.FormatWebP, nil
case string(imagerequest.FormatAVIF):
return imagerequest.FormatAVIF, nil
case string(imagerequest.FormatJXL):
return imagerequest.FormatJXL, nil
default:
return "", fmt.Errorf("%w: %s", ErrUnsupportedInputFormat, format)
}
}
internal/imageprocessor/imageprocessor_internal_test.go
+36 -24
View File
@@ -94,23 +94,23 @@ func isAVIF(data []byte) bool {
// detectMIME is a minimal magic-byte detector for test assertions. // detectMIME is a minimal magic-byte detector for test assertions.
func detectMIME(data []byte) string { func detectMIME(data []byte) string {
if len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF { if len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF {
return mimeJPEG return imagerequest.FormatJPEG.MIMEType()
} }
if len(data) >= 8 && string(data[:8]) == "\x89PNG\r\n\x1a\n" { if len(data) >= 8 && string(data[:8]) == "\x89PNG\r\n\x1a\n" {
return mimePNG return imagerequest.FormatPNG.MIMEType()
} }
if len(data) >= 4 && string(data[:4]) == "GIF8" { if len(data) >= 4 && string(data[:4]) == "GIF8" {
return mimeGIF return imagerequest.FormatGIF.MIMEType()
} }
if len(data) >= 12 && string(data[:4]) == "RIFF" && string(data[8:12]) == "WEBP" { if len(data) >= 12 && string(data[:4]) == "RIFF" && string(data[8:12]) == "WEBP" {
return mimeWebP return imagerequest.FormatWebP.MIMEType()
} }
if isAVIF(data) { if isAVIF(data) {
return mimeAVIF return imagerequest.FormatAVIF.MIMEType()
} }
return "" return ""
@@ -131,7 +131,8 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
FitMode: imagerequest.FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), transform) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -157,7 +158,7 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
} }
mime := detectMIME(data) mime := detectMIME(data)
if mime != mimeJPEG { if mime != imagerequest.FormatJPEG.MIMEType() {
t.Errorf("Output format = %v, want image/jpeg", mime) t.Errorf("Output format = %v, want image/jpeg", mime)
} }
} }
@@ -176,7 +177,8 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
FitMode: imagerequest.FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), transform) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -189,7 +191,7 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
} }
mime := detectMIME(data) mime := detectMIME(data)
if mime != mimePNG { if mime != imagerequest.FormatPNG.MIMEType() {
t.Errorf("Output format = %v, want image/png", mime) t.Errorf("Output format = %v, want image/png", mime)
} }
} }
@@ -213,7 +215,8 @@ func processAndCheckSize(
FitMode: imagerequest.FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), transform) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -253,7 +256,8 @@ func TestImageProcessor_FitContain(t *testing.T) {
FitMode: imagerequest.FitContain, FitMode: imagerequest.FitContain,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), transform) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -296,7 +300,8 @@ func TestImageProcessor_ProcessPNG(t *testing.T) {
FitMode: imagerequest.FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), transform) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatPNG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
} }
@@ -341,7 +346,8 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
FitMode: imagerequest.FitCover, FitMode: imagerequest.FitCover,
} }
_, err := proc.Process(ctx, bytes.NewReader(input), transform) _, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err == nil { if err == nil {
t.Error("Process() should reject oversized input images") t.Error("Process() should reject oversized input images")
} }
@@ -369,7 +375,8 @@ func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
FitMode: imagerequest.FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), transform) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf( t.Fatalf(
"Process() should accept images at MaxInputDimension, got error: %v", "Process() should accept images at MaxInputDimension, got error: %v",
@@ -399,7 +406,8 @@ func encodeAndCheck(
FitMode: imagerequest.FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), transform) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v, want nil", err) t.Fatalf("Process() error = %v, want nil", err)
} }
@@ -430,7 +438,7 @@ func encodeAndCheck(
func TestImageProcessor_EncodeWebP(t *testing.T) { func TestImageProcessor_EncodeWebP(t *testing.T) {
t.Parallel() t.Parallel()
encodeAndCheck(t, imagerequest.FormatWebP, 80, mimeWebP) encodeAndCheck(t, imagerequest.FormatWebP, 80, imagerequest.FormatWebP.MIMEType())
} }
func TestImageProcessor_DecodeAVIF(t *testing.T) { func TestImageProcessor_DecodeAVIF(t *testing.T) {
@@ -453,7 +461,8 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
FitMode: imagerequest.FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), transform) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatAVIF, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v, want nil (AVIF decoding should work)", err) t.Fatalf("Process() error = %v, want nil (AVIF decoding should work)", err)
} }
@@ -467,7 +476,7 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
} }
mime := detectMIME(data) mime := detectMIME(data)
if mime != mimeJPEG { if mime != imagerequest.FormatJPEG.MIMEType() {
t.Errorf("Output format = %v, want image/jpeg", mime) t.Errorf("Output format = %v, want image/jpeg", mime)
} }
} }
@@ -494,7 +503,8 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
FitMode: imagerequest.FitCover, FitMode: imagerequest.FitCover,
} }
_, err := proc.Process(ctx, bytes.NewReader(input), transform) _, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err == nil { if err == nil {
t.Fatal("Process() should reject input exceeding maxInputBytes") t.Fatal("Process() should reject input exceeding maxInputBytes")
} }
@@ -521,7 +531,8 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
FitMode: imagerequest.FitCover, FitMode: imagerequest.FitCover,
} }
result, err := proc.Process(ctx, bytes.NewReader(input), transform) result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
if err != nil { if err != nil {
t.Fatalf("Process() error = %v, want nil", err) t.Fatalf("Process() error = %v, want nil", err)
} }
@@ -532,11 +543,11 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
func TestImageProcessor_EncodeAVIF(t *testing.T) { func TestImageProcessor_EncodeAVIF(t *testing.T) {
t.Parallel() t.Parallel()
encodeAndCheck(t, imagerequest.FormatAVIF, 85, mimeAVIF) encodeAndCheck(t, imagerequest.FormatAVIF, 85, imagerequest.FormatAVIF.MIMEType())
} }
// processAndDecode runs input through Process and decodes the output with // processAndDecode runs input, a JPEG, through Process and decodes the output
// vips, so a test can inspect the image a client would receive. // with vips, so a test can inspect the image a client would receive.
func processAndDecode( func processAndDecode(
t *testing.T, input []byte, transform *imagerequest.Transform, t *testing.T, input []byte, transform *imagerequest.Transform,
) *vips.ImageRef { ) *vips.ImageRef {
@@ -546,7 +557,8 @@ func processAndDecode(
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process( }).Process(
context.Background(), bytes.NewReader(input), transform, context.Background(), bytes.NewReader(input), imagerequest.FormatJPEG,
transform,
) )
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
internal/imageprocessor/jpegxl_output_internal_test.go
+3 -2
View File
@@ -25,7 +25,8 @@ func TestImageProcessor_EncodeJPEGXL(t *testing.T) {
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process( }).Process(
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)), transform, t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)),
imagerequest.FormatJPEG, transform,
) )
if err != nil { if err != nil {
t.Fatalf("Process() error = %v", err) t.Fatalf("Process() error = %v", err)
@@ -71,7 +72,7 @@ func TestImageProcessor_JPEGXLQuality(t *testing.T) {
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process(t.Context(), }).Process(t.Context(),
bytes.NewReader(input), bytes.NewReader(input), imagerequest.FormatJPEG,
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: quality}) &imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: quality})
if err != nil { if err != nil {
t.Fatalf("Process() at quality %d error = %v", quality, err) t.Fatalf("Process() at quality %d error = %v", quality, err)
internal/imageprocessor/max_concurrent_processing_internal_test.go
+6 -4
View File
@@ -93,7 +93,8 @@ func processInBackground(
proc *ImageProcessor, reader *gatedReader, results chan<- error, proc *ImageProcessor, reader *gatedReader, results chan<- error,
) { ) {
go func() { go func() {
result, err := proc.Process(context.Background(), reader, smallJPEGTransform()) result, err := proc.Process(context.Background(), reader,
imagerequest.FormatJPEG, smallJPEGTransform())
if err == nil { if err == nil {
_ = result.Content.Close() _ = result.Content.Close()
} }
@@ -206,7 +207,7 @@ func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) {
start := time.Now() start := time.Now()
_, err := proc.Process(context.Background(), bytes.NewReader(input), _, err := proc.Process(context.Background(), bytes.NewReader(input),
smallJPEGTransform()) imagerequest.FormatJPEG, smallJPEGTransform())
if !errors.Is(err, ErrTooManyImages) { if !errors.Is(err, ErrTooManyImages) {
t.Fatalf("Process() error = %v, want ErrTooManyImages", err) t.Fatalf("Process() error = %v, want ErrTooManyImages", err)
} }
@@ -274,7 +275,8 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
proc := New(Params{MaxInputBytes: 4096, MaxConcurrentProcessing: 1}) proc := New(Params{MaxInputBytes: 4096, MaxConcurrentProcessing: 1})
proc.processingWaitTimeout = 100 * time.Millisecond proc.processingWaitTimeout = 100 * time.Millisecond
_, err := proc.Process(context.Background(), tc.input, tc.transform) _, err := proc.Process(context.Background(), tc.input,
imagerequest.FormatJPEG, tc.transform)
if err == nil || (tc.want != nil && !errors.Is(err, tc.want)) { if err == nil || (tc.want != nil && !errors.Is(err, tc.want)) {
t.Fatalf("Process() error = %v, want %v", err, tc.want) t.Fatalf("Process() error = %v, want %v", err, tc.want)
} }
@@ -284,7 +286,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
} }
result, err := proc.Process(context.Background(), bytes.NewReader(valid), result, err := proc.Process(context.Background(), bytes.NewReader(valid),
smallJPEGTransform()) imagerequest.FormatJPEG, smallJPEGTransform())
if err != nil { if err != nil {
t.Fatalf("Process() after the error = %v, want nil", err) t.Fatalf("Process() after the error = %v, want nil", err)
} }
internal/imageprocessor/orig_format_internal_test.go
+4 -3
View File
@@ -10,8 +10,9 @@ import (
) )
// TestImageProcessor_OrigOfUnnamedFormatRefused verifies that orig of a TIFF // TestImageProcessor_OrigOfUnnamedFormatRefused verifies that orig of a TIFF
// input, which is not a source format, is refused with // input is refused with ErrUnsupportedInputFormat before it is decoded, and
// ErrUnsupportedInputFormat before it is decoded, and not saved as JPEG. // not saved as JPEG. TIFF is not a source format, and the test passes the
// input to Process as a JPEG.
func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) { func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
t.Parallel() t.Parallel()
@@ -31,7 +32,7 @@ func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
MaxInputBytes: 50 << 20, MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1, MaxConcurrentProcessing: 1,
}).Process( }).Process(
t.Context(), bytes.NewReader(tiff), t.Context(), bytes.NewReader(tiff), imagerequest.FormatJPEG,
&imagerequest.Transform{Format: imagerequest.FormatOriginal}, &imagerequest.Transform{Format: imagerequest.FormatOriginal},
) )
if result != nil { if result != nil {
internal/imageprocessor/source_formats_internal_test.go
+29 -4
View File
@@ -2,6 +2,7 @@ package imageprocessor
import ( import (
"bytes" "bytes"
"errors"
"io" "io"
"testing" "testing"
@@ -21,7 +22,7 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL, imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
} { } {
saved, err := proc.Process(t.Context(), saved, err := proc.Process(t.Context(),
bytes.NewReader(createTestPNG(t, 20, 20)), bytes.NewReader(createTestPNG(t, 20, 20)), imagerequest.FormatPNG,
&imagerequest.Transform{Format: format}) &imagerequest.Transform{Format: format})
if err != nil { if err != nil {
t.Fatalf("saving the test image as %s: %v", format, err) t.Fatalf("saving the test image as %s: %v", format, err)
@@ -32,7 +33,7 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
t.Fatalf("reading the %s image: %v", format, err) t.Fatalf("reading the %s image: %v", format, err)
} }
result, err := proc.Process(t.Context(), bytes.NewReader(source), result, err := proc.Process(t.Context(), bytes.NewReader(source), format,
&imagerequest.Transform{Format: imagerequest.FormatOriginal}) &imagerequest.Transform{Format: imagerequest.FormatOriginal})
if err != nil { if err != nil {
t.Errorf("orig of the %s image: %v", format, err) t.Errorf("orig of the %s image: %v", format, err)
@@ -42,9 +43,33 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
t.Logf("orig of the %s image: %s", format, result.ContentType) t.Logf("orig of the %s image: %s", format, result.ContentType)
if result.ContentType != FormatToMIME(format) { if result.ContentType != format.MIMEType() {
t.Errorf("orig of the %s image is %s, want %s", t.Errorf("orig of the %s image is %s, want %s",
format, result.ContentType, FormatToMIME(format)) format, result.ContentType, format.MIMEType())
}
}
}
// TestImageProcessor_NotASourceFormatRefused verifies that a source format
// other than the six, such as orig, auto or none, is refused with
// ErrUnsupportedInputFormat. The input is text, which libvips reads as no
// format.
func TestImageProcessor_NotASourceFormatRefused(t *testing.T) {
t.Parallel()
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
for _, sourceFormat := range []imagerequest.Format{
imagerequest.FormatOriginal, imagerequest.FormatAuto, "",
} {
result, err := proc.Process(t.Context(),
bytes.NewReader([]byte("this is text, not an image")), sourceFormat,
&imagerequest.Transform{Format: imagerequest.FormatJPEG})
t.Logf("source format %q: error = %v", sourceFormat, err)
if result != nil || !errors.Is(err, ErrUnsupportedInputFormat) {
t.Errorf("Process() with source format %q = %v, error %v; want %v",
sourceFormat, result, err, ErrUnsupportedInputFormat)
} }
} }
} }
internal/imageproxy/cache_hit_internal_test.go
+100
View File
@@ -0,0 +1,100 @@
package imageproxy
import (
"bytes"
"os"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// TestProxy_Get_MakesAgainVariantOfNoOutputFormat replaces a cached
// variant's file with text, and checks that the next request is a miss
// answered with the variant made again, and the one after a hit.
func TestProxy_Get_MakesAgainVariantOfNoOutputFormat(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := variantFilePath(svc, imagerequest.CacheKey(req))
err := os.WriteFile(path, []byte("this is text, not an image"),
imgcache.StorageFilePerm)
if err != nil {
t.Fatalf("replacing the variant file: %v", err)
}
got, status := getImage(t, svc, req)
t.Logf("after the replacement: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the replacement: Get() = %d bytes, %s; want the %d bytes "+
"made again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestProxy_Get_HitWaitsForNoDatabaseWrite holds the database's one
// connection while a cached variant is requested. The hit must be answered
// while the connection is still held, and its access time must reach the
// database once it is free.
func TestProxy_Get_HitWaitsForNoDatabaseWrite(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
key := imagerequest.CacheKey(req)
getImage(t, svc, req)
// Each connection to an in-memory SQLite database opens a new, empty one,
// so the goroutine that writes the access times must share the
// connection that holds the schema.
svc.db.SetMaxOpenConns(1)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
anHourAgo := time.Now().Add(-time.Hour)
setVariantLastAccessed(t, svc.db, key, anHourAgo)
releaseDatabase := holdDatabase(t, svc.db)
defer releaseDatabase()
results := startGet(t.Context(), svc, req)
select {
case got := <-results:
if got.err != nil {
t.Fatalf("Get() of the cached variant error = %v", got.err)
}
case <-time.After(pendingAccessTimesWait):
t.Fatal("the cache hit waited for the database")
}
releaseDatabase()
deadline := time.Now().Add(pendingAccessTimesWait)
for countRows(t, svc.db, `
SELECT COUNT(*) FROM variant_content
WHERE cache_key = ? AND last_accessed_at > ?
`, string(key), anHourAgo.UTC().Format(sqliteTimestampFormat)) == 0 {
if time.Now().After(deadline) {
t.Fatal("the hit's access time was not written")
}
time.Sleep(10 * time.Millisecond)
}
}
internal/imgcache/concurrent_misses_internal_test.go → internal/imageproxy/concurrent_misses_internal_test.go
+40 -40
View File
@@ -1,4 +1,4 @@
package imgcache package imageproxy
import ( import (
"bytes" "bytes"
@@ -56,15 +56,15 @@ func (f *heldFetcher) releaseFetches() {
f.releaseOnce.Do(func() { close(f.release) }) f.releaseOnce.Do(func() { close(f.release) })
} }
// setupHeldFetchService returns a test service whose fetches go through a // setupHeldFetchProxy returns a test proxy whose fetches go through a
// heldFetcher. Its database is limited to one connection: each connection to // heldFetcher. Its database is limited to one connection: each connection to
// an in-memory SQLite database opens a new, empty one, so requests running at // an in-memory SQLite database opens a new, empty one, so requests running at
// once must share the connection that holds the schema. // once must share the connection that holds the schema.
func setupHeldFetchService(t *testing.T) (*Service, *TestFixtures, *heldFetcher) { func setupHeldFetchProxy(t *testing.T) (*testProxy, *TestFixtures, *heldFetcher) {
t.Helper() t.Helper()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
svc.cache.db.SetMaxOpenConns(1) svc.db.SetMaxOpenConns(1)
fetcher := &heldFetcher{ fetcher := &heldFetcher{
upstream: svc.fetcher, upstream: svc.fetcher,
@@ -79,8 +79,8 @@ func setupHeldFetchService(t *testing.T) (*Service, *TestFixtures, *heldFetcher)
} }
// photoVariant asks for the test photo, 100x100, at 50x25 as a JPEG of the // photoVariant asks for the test photo, 100x100, at 50x25 as a JPEG of the
// given quality and fit mode. Each call returns a new request, as Get writes // given quality and fit mode. Each call returns a new request, which a test
// to the request it is given. // may change.
func photoVariant( func photoVariant(
fixtures *TestFixtures, quality int, fit imagerequest.Fit, fixtures *TestFixtures, quality int, fit imagerequest.Fit,
) *imagerequest.Request { ) *imagerequest.Request {
@@ -107,7 +107,7 @@ type getResult struct {
// startGet calls Get in a goroutine of its own and delivers what it returned // startGet calls Get in a goroutine of its own and delivers what it returned
// on the channel. // on the channel.
func startGet( func startGet(
ctx context.Context, svc *Service, req *imagerequest.Request, ctx context.Context, svc *testProxy, req *imagerequest.Request,
) <-chan getResult { ) <-chan getResult {
results := make(chan getResult, 1) results := make(chan getResult, 1)
@@ -139,14 +139,14 @@ func jpegSize(data []byte) (int, int) {
return config.Width, config.Height return config.Width, config.Height
} }
// TestService_Get_ConcurrentMissesShareOneFetch starts several requests for // TestProxy_Get_ConcurrentMissesShareOneFetch starts several requests for
// one uncached variant while the first one's fetch is held. Between them they // one uncached variant while the first one's fetch is held. Between them they
// must fetch the source once and transcode it once, every one must be answered // must fetch the source once and transcode it once, every one must be answered
// with the same 50x25 JPEG, and each must count one miss. // with the same 50x25 JPEG, and each must count one miss.
func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) { func TestProxy_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
const requests = 8 const requests = 8
@@ -183,7 +183,7 @@ func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
t.Errorf("%d requests made %d upstream fetches, want 1", requests, fetches) t.Errorf("%d requests made %d upstream fetches, want 1", requests, fetches)
} }
// NewTestFS builds the same files the test service's fetcher serves. // NewTestFS builds the same files the test proxy's fetcher serves.
testFS, _ := NewTestFS(t) testFS, _ := NewTestFS(t)
photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG) photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG)
@@ -193,18 +193,18 @@ func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
want := counterValues{0, requests, 1, int64(len(photo)), 1} want := counterValues{0, requests, 1, int64(len(photo)), 1}
if got := readCounters(svc.cache); got != want { if got := readCounters(t, svc); got != want {
t.Errorf("counters = %+v, want %+v", got, want) t.Errorf("counters = %+v, want %+v", got, want)
} }
} }
// TestService_Get_ConcurrentVariantsStayApart requests three variants of the // TestProxy_Get_ConcurrentVariantsStayApart requests three variants of the
// test photo at once that differ only in quality or fit. Each must be made by // test photo at once that differ only in quality or fit. Each must be made by
// a fetch and a transcode of its own, and each answer must be its own variant. // a fetch and a transcode of its own, and each answer must be its own variant.
func TestService_Get_ConcurrentVariantsStayApart(t *testing.T) { func TestProxy_Get_ConcurrentVariantsStayApart(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
cover := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover)) cover := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
lowQuality := startGet(t.Context(), svc, lowQuality := startGet(t.Context(), svc,
@@ -253,14 +253,14 @@ func TestService_Get_ConcurrentVariantsStayApart(t *testing.T) {
} }
} }
// TestService_Get_WaiterStopsWhenItsContextEnds has a second request for a // TestProxy_Get_WaiterStopsWhenItsContextEnds has a second request for a
// variant join the first one's held fetch, then ends the second request's // variant join the first one's held fetch, then ends the second request's
// context. The second request must return at once with the context's error, // context. The second request must return at once with the context's error,
// while the fetch is still held, and the first must still be answered. // while the fetch is still held, and the first must still be answered.
func TestService_Get_WaiterStopsWhenItsContextEnds(t *testing.T) { func TestProxy_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
first := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover)) first := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
@@ -298,15 +298,15 @@ func TestService_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
} }
} }
// TestService_Get_FirstRequestLeavingKeepsTheWork ends the context of the // TestProxy_Get_FirstRequestLeavingKeepsTheWork ends the context of the
// request whose fetch is held, after a second request has joined it. The fetch // request whose fetch is held, after a second request has joined it. The fetch
// and transcode must go on and answer the second request. The first request // and transcode must go on and answer the second request. The first request
// waits for its own work, as every request did before misses were shared, and // waits for its own work, as every request did before misses were shared, and
// is answered too. // is answered too.
func TestService_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) { func TestProxy_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
firstCtx, cancelFirst := context.WithCancel(t.Context()) firstCtx, cancelFirst := context.WithCancel(t.Context())
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover)) first := startGet(firstCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
@@ -337,11 +337,11 @@ func TestService_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
} }
} }
// TestService_Get_ConcurrentMissesShareAFailure has several requests for an // TestProxy_Get_ConcurrentMissesShareAFailure has several requests for an
// image that cannot be served arrive while its fetch is held: the one fetch // image that cannot be served arrive while its fetch is held: the one fetch
// answers all of them with its error. The request after them is answered from // answers all of them with its error. The request after them is answered from
// the negative cache, where the failure is kept. // the negative cache, where the failure is kept.
func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) { func TestProxy_Get_ConcurrentMissesShareAFailure(t *testing.T) {
t.Parallel() t.Parallel()
tests := []struct { tests := []struct {
@@ -362,7 +362,7 @@ func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
request := func() *imagerequest.Request { request := func() *imagerequest.Request {
req := photoVariant(fixtures, 85, imagerequest.FitCover) req := photoVariant(fixtures, 85, imagerequest.FitCover)
@@ -398,12 +398,12 @@ func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
} }
} }
// TestService_Get_EndedRequestFetchesNothing checks that a request whose // TestProxy_Get_EndedRequestFetchesNothing checks that a request whose
// context has already ended when it misses the cache starts no fetch. // context has already ended when it misses the cache starts no fetch.
func TestService_Get_EndedRequestFetchesNothing(t *testing.T) { func TestProxy_Get_EndedRequestFetchesNothing(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
ctx, cancel := context.WithCancel(t.Context()) ctx, cancel := context.WithCancel(t.Context())
cancel() cancel()
@@ -418,14 +418,14 @@ func TestService_Get_EndedRequestFetchesNothing(t *testing.T) {
} }
} }
// TestService_Get_ReturnsByItsDeadline gives a request a deadline and holds // TestProxy_Get_ReturnsByItsDeadline gives a request a deadline and holds
// its fetch until the fetch's context ends, as the fetcher does while it waits // its fetch until the fetch's context ends, as the fetcher does while it waits
// for a free connection to the host. The request must return by its deadline // for a free connection to the host. The request must return by its deadline
// with the deadline's error. // with the deadline's error.
func TestService_Get_ReturnsByItsDeadline(t *testing.T) { func TestProxy_Get_ReturnsByItsDeadline(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, _ := setupHeldFetchService(t) svc, fixtures, _ := setupHeldFetchProxy(t)
const timeout = 200 * time.Millisecond const timeout = 200 * time.Millisecond
@@ -444,16 +444,16 @@ func TestService_Get_ReturnsByItsDeadline(t *testing.T) {
} }
} }
// TestService_Get_WaiterGetsProcessingTimedOut has a request with no deadline // TestProxy_Get_WaiterGetsProcessingTimedOut has a request with no deadline
// join the held fetch of a first request with a short one. With the only // join the held fetch of a first request with a short one. With the only
// processing slot held, the first request's transcode waits for it until the // processing slot held, the first request's transcode waits for it until the
// first request's deadline ends, and stops then. The waiting request, whose // first request's deadline ends, and stops then. The waiting request, whose
// own context has not ended, must get ErrProcessingTimedOut, which the image // own context has not ended, must get ErrProcessingTimedOut, which the image
// routes answer 503 as when pixa is busy. // routes answer 503 as when pixa is busy.
func TestService_Get_WaiterGetsProcessingTimedOut(t *testing.T) { func TestProxy_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t) svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc.processor = imageprocessor.New( svc.processor = imageprocessor.New(
imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1}, imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1},
) )
@@ -494,13 +494,13 @@ func (panickingFetcher) Fetch(
panic("upstream fetcher panicked") panic("upstream fetcher panicked")
} }
// TestService_Get_PanicBecomesAnError checks that a panic while a variant is // TestProxy_Get_PanicBecomesAnError checks that a panic while a variant is
// being made reaches its request as an error naming the panic, instead of // being made reaches its request as an error naming the panic, instead of
// being raised again. // being raised again.
func TestService_Get_PanicBecomesAnError(t *testing.T) { func TestProxy_Get_PanicBecomesAnError(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
svc.fetcher = panickingFetcher{} svc.fetcher = panickingFetcher{}
var err error var err error
@@ -520,13 +520,13 @@ func TestService_Get_PanicBecomesAnError(t *testing.T) {
} }
} }
// TestService_Get_PanicIsReportedToSentry checks that a panic while a variant // TestProxy_Get_PanicIsReportedToSentry checks that a panic while a variant
// is being made is reported through the Sentry hub on the request's context, // is being made is reported through the Sentry hub on the request's context,
// where the Sentry middleware puts one when sentry_dsn is set. // where the Sentry middleware puts one when sentry_dsn is set.
func TestService_Get_PanicIsReportedToSentry(t *testing.T) { func TestProxy_Get_PanicIsReportedToSentry(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
svc.fetcher = panickingFetcher{} svc.fetcher = panickingFetcher{}
transport := &sentry.MockTransport{} transport := &sentry.MockTransport{}
internal/imageproxy/counters_internal_test.go
+77
View File
@@ -0,0 +1,77 @@
package imageproxy
import (
"errors"
"testing"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/testutil"
"sneak.berlin/go/pixa/internal/imgcache"
)
// TestNew_RegistersItsCountersOnItsRegistry checks that New registers the
// proxy's counters on the registry it is given, under the names README.md
// gives them. The cache registers its own counters on another registry.
func TestNew_RegistersItsCountersOnItsRegistry(t *testing.T) {
t.Parallel()
cache, err := imgcache.NewCache(setupTestDB(t),
imgcache.Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err)
}
registry := prometheus.NewRegistry()
_, err = New(&Config{Cache: cache}, registry)
if err != nil {
t.Fatalf("New() error = %v", err)
}
names := []string{
"pixa_upstream_fetches_total",
"pixa_upstream_fetch_bytes_total",
"pixa_transforms_total",
}
count, err := testutil.GatherAndCount(registry, names...)
if err != nil {
t.Fatalf("testutil.GatherAndCount() error = %v", err)
}
if count != len(names) {
t.Errorf("%d of the counters %v are on the registry, want %d",
count, names, len(names))
}
}
// TestNew_FailsWhenItsCountersCannotBeRegistered makes a second proxy on
// the registry of a first. Its counters have the same names, so the
// registry refuses them, and New must return that error.
func TestNew_FailsWhenItsCountersCannotBeRegistered(t *testing.T) {
t.Parallel()
cache, err := imgcache.NewCache(setupTestDB(t),
imgcache.Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err)
}
registry := prometheus.NewRegistry()
_, err = New(&Config{Cache: cache}, registry)
if err != nil {
t.Fatalf("New() of the first proxy error = %v", err)
}
_, err = New(&Config{Cache: cache}, registry)
t.Logf("New() of the second proxy error = %v", err)
var alreadyRegistered prometheus.AlreadyRegisteredError
if !errors.As(err, &alreadyRegistered) {
t.Errorf("New() of the second proxy error = %v, want %T",
err, alreadyRegistered)
}
}
internal/imgcache/divzero_internal_test.go → internal/imageproxy/divzero_internal_test.go
+1 -1
View File
@@ -1,4 +1,4 @@
package imgcache package imageproxy
import ( import (
"fmt" "fmt"
internal/imgcache/service.go → internal/imageproxy/imageproxy.go
+167 -103
View File
@@ -1,4 +1,7 @@
package imgcache // Package imageproxy fetches, transforms, caches and serves images. Its
// Proxy serves an image from the cache when it can, and otherwise makes it
// from the cached or fetched source image.
package imageproxy
import ( import (
"bytes" "bytes"
@@ -13,16 +16,18 @@ import (
"github.com/dustin/go-humanize" "github.com/dustin/go-humanize"
"github.com/getsentry/sentry-go" "github.com/getsentry/sentry-go"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/sync/singleflight" "golang.org/x/sync/singleflight"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic" "sneak.berlin/go/pixa/internal/magic"
) )
// Service orchestrates the cache, fetcher, and processor. // Proxy orchestrates the cache, fetcher, and processor.
type Service struct { type Proxy struct {
cache *Cache cache *imgcache.Cache
fetcher httpfetcher.Fetcher fetcher httpfetcher.Fetcher
processor *imageprocessor.ImageProcessor processor *imageprocessor.ImageProcessor
log *slog.Logger log *slog.Logger
@@ -30,12 +35,18 @@ type Service struct {
// variantsInProgress lets the requests that miss the same variant at the // variantsInProgress lets the requests that miss the same variant at the
// same time share one fetch and one transcode. // same time share one fetch and one transcode.
variantsInProgress singleflight.Group variantsInProgress singleflight.Group
// The counts of upstream fetches and transforms, made and registered by
// New. They are kept in memory, so adding to one waits for nothing.
upstreamFetches prometheus.Counter
upstreamFetchBytes prometheus.Counter
transforms prometheus.Counter
} }
// ServiceConfig holds configuration for the image service. // Config holds the proxy's configuration.
type ServiceConfig struct { type Config struct {
// Cache is the cache instance // Cache is the cache instance
Cache *Cache Cache *imgcache.Cache
// Fetcher fetches source images from upstream hosts // Fetcher fetches source images from upstream hosts
Fetcher httpfetcher.Fetcher Fetcher httpfetcher.Fetcher
// Processor converts and resizes the images // Processor converts and resizes the images
@@ -47,11 +58,16 @@ type ServiceConfig struct {
Logger *slog.Logger Logger *slog.Logger
} }
// errCacheRequired is returned when the service is built without a cache. // errCacheRequired is returned when the proxy is built without a cache.
var errCacheRequired = errors.New("cache is required") var errCacheRequired = errors.New("cache is required")
// NewService creates a new image service. // New creates a new image proxy and registers its counters on
func NewService(cfg *ServiceConfig) (*Service, error) { // metricsRegistry. pixad passes Prometheus' default registry, the one
// /metrics serves. It fails when metricsRegistry already holds counters of
// the same names.
func New(
cfg *Config, metricsRegistry prometheus.Registerer,
) (*Proxy, error) {
if cfg.Cache == nil { if cfg.Cache == nil {
return nil, errCacheRequired return nil, errCacheRequired
} }
@@ -68,13 +84,36 @@ func NewService(cfg *ServiceConfig) (*Service, error) {
return nil, err return nil, err
} }
return &Service{ p := &Proxy{
cache: cfg.Cache, cache: cfg.Cache,
fetcher: cfg.Fetcher, fetcher: cfg.Fetcher,
processor: cfg.Processor, processor: cfg.Processor,
log: log, log: log,
maxResponseSize: cfg.MaxResponseSize, maxResponseSize: cfg.MaxResponseSize,
}, nil upstreamFetches: prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetches_total",
Help: "Fetches of a source image from upstream that read any bytes.",
}),
upstreamFetchBytes: prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetch_bytes_total",
Help: "Bytes read from upstream by fetches of source images.",
}),
transforms: prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_transforms_total",
Help: "Images the image processor transformed.",
}),
}
for _, counter := range []prometheus.Counter{
p.upstreamFetches, p.upstreamFetchBytes, p.transforms,
} {
err = metricsRegistry.Register(counter)
if err != nil {
return nil, fmt.Errorf("failed to register the proxy's counters: %w", err)
}
}
return p, nil
} }
// ErrNegativeCached is returned when a URL is in the negative cache (recently failed). // ErrNegativeCached is returned when a URL is in the negative cache (recently failed).
@@ -86,25 +125,49 @@ var ErrNegativeCached = errors.New("request is in negative cache (recently faile
var ErrProcessingTimedOut = errors.New( var ErrProcessingTimedOut = errors.New(
"image processing stopped at the deadline of the request doing it") "image processing stopped at the deadline of the request doing it")
// ImageResponse represents a processed image ready to serve
type ImageResponse struct {
// Content is the image data reader
Content io.ReadCloser
// ContentLength is the size in bytes (-1 if unknown)
ContentLength int64
// ContentType is the MIME type of the response
ContentType string
// ETag is the entity tag for caching
ETag string
// CacheStatus indicates HIT or MISS
CacheStatus CacheStatus
// FetchedBytes is the number of bytes fetched from upstream (0 if cache hit)
FetchedBytes int64
}
// CacheStatus indicates whether the response was served from cache.
type CacheStatus string
// Cache status values for response headers.
const (
CacheHit CacheStatus = "HIT"
CacheMiss CacheStatus = "MISS"
)
// Get retrieves a processed image, fetching and processing if necessary. // Get retrieves a processed image, fetching and processing if necessary.
// While its source URL is in the negative cache, a cached variant is still // While its source URL is in the negative cache, a cached variant is still
// served and a variant not in the cache is still made from a cached source; // served and a variant not in the cache is still made from a cached source;
// only a request that needs the source fetched is refused. // only a request that needs the source fetched is refused.
// //
// A cache hit opens the variant file and makes no database call: its access // A cache hit opens the variant file and makes no database call: the cache
// time is handed to the goroutine that StartPendingAccessTimeWrites starts, // hands its access time to the goroutine that writes access times, without
// without waiting for it, and its hit is counted in memory. A variant file // waiting for it, and its hit is counted in memory. A variant file that
// that cannot be served, such as one whose first bytes are not those of an // cannot be served, such as one whose first bytes are not those of an output
// output format, is a miss, and the variant is made again. // format, is a miss, and the variant is made again.
func (s *Service) Get( func (p *Proxy) Get(
ctx context.Context, req *imagerequest.Request, ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) { ) (*ImageResponse, error) {
cacheKey := imagerequest.CacheKey(req) cacheKey := imagerequest.CacheKey(req)
reader, size, contentType, err := s.cache.GetVariant(cacheKey) reader, size, contentType, err := p.cache.Variant(cacheKey)
if err == nil { if err == nil {
s.cache.touchVariant(cacheKey) p.cache.IncrementStats(true)
s.cache.IncrementStats(true, 0)
return &ImageResponse{ return &ImageResponse{
Content: reader, Content: reader,
@@ -115,17 +178,17 @@ func (s *Service) Get(
}, nil }, nil
} }
if !errors.Is(err, ErrNotFound) { if !errors.Is(err, imgcache.ErrNotFound) {
s.log.Warn("cached variant not served, making it again", p.log.Warn("cached variant not served, making it again",
"cache_key", cacheKey, "error", err) "cache_key", cacheKey, "error", err)
} }
// Cache miss - get the variant, processed once for all the requests that // Cache miss - get the variant, processed once for all the requests that
// miss it at the same time, then count this request's miss, also when it // miss it at the same time, then count this request's miss, also when it
// failed or the request context has ended meanwhile // failed or the request context has ended meanwhile
response, err := s.processOrWait(ctx, req) response, err := p.processOrWait(ctx, req)
s.cache.IncrementStats(false, 0) p.cache.IncrementStats(false)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -138,8 +201,8 @@ func (s *Service) Get(
// WaitForProcessing waits until no image is being processed, or until ctx // WaitForProcessing waits until no image is being processed, or until ctx
// ends, and returns how many images were still being processed then. // ends, and returns how many images were still being processed then.
func (s *Service) WaitForProcessing(ctx context.Context) int { func (p *Proxy) WaitForProcessing(ctx context.Context) int {
return s.processor.WaitForProcessing(ctx) return p.processor.WaitForProcessing(ctx)
} }
// errPanicked is returned when processing a variant panicked. // errPanicked is returned when processing a variant panicked.
@@ -151,7 +214,7 @@ var errPanicked = errors.New("panic while processing image")
// and take no upstream connection or processing slot. The processing ignores // and take no upstream connection or processing slot. The processing ignores
// the first request's cancellation, so the others are still served if that // the first request's cancellation, so the others are still served if that
// client goes away, but keeps its deadline. // client goes away, but keeps its deadline.
func (s *Service) processOrWait( func (p *Proxy) processOrWait(
ctx context.Context, req *imagerequest.Request, ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) { ) (*ImageResponse, error) {
// A request that has already ended starts no processing // A request that has already ended starts no processing
@@ -165,7 +228,7 @@ func (s *Service) processOrWait(
// only when no other request is processing the variant // only when no other request is processing the variant
processing := make(chan struct{}) processing := make(chan struct{})
results := s.variantsInProgress.DoChan(string(cacheKey), results := p.variantsInProgress.DoChan(string(cacheKey),
func() (_ any, err error) { func() (_ any, err error) {
close(processing) close(processing)
@@ -176,7 +239,7 @@ func (s *Service) processOrWait(
defer func() { defer func() {
recovered := recover() recovered := recover()
if recovered != nil { if recovered != nil {
s.log.Error("panic while processing image", p.log.Error("panic while processing image",
"host", req.Source.Host, "path", req.Source.Path, "host", req.Source.Host, "path", req.Source.Path,
"panic", recovered, "stack", string(debug.Stack())) "panic", recovered, "stack", string(debug.Stack()))
@@ -191,7 +254,7 @@ func (s *Service) processOrWait(
processingCtx, cancel := withoutCancelKeepingDeadline(ctx) processingCtx, cancel := withoutCancelKeepingDeadline(ctx)
defer cancel() defer cancel()
return s.processFromSourceOrFetch(processingCtx, req, cacheKey) return p.processFromSourceOrFetch(processingCtx, req, cacheKey)
}) })
var result singleflight.Result var result singleflight.Result
@@ -240,65 +303,52 @@ func withoutCancelKeepingDeadline(
return context.WithDeadline(context.WithoutCancel(ctx), deadline) return context.WithDeadline(context.WithoutCancel(ctx), deadline)
} }
// loadCachedSource opens source content from cache, without reading it, and // loadCachedSource opens src's cached source, reading only the first bytes,
// returns it with its size; nil if the cached data is unavailable, empty or // which give its format, and returns it with its size and its format. It
// exceeds maxResponseSize. // returns nil if the source is not in the cache, cannot be read, is in no
func (s *Service) loadCachedSource( // source format or exceeds maxResponseSize.
sourceKey imagerequest.SourceKey, func (p *Proxy) loadCachedSource(
) (io.ReadCloser, int64) { ctx context.Context, src imagerequest.Source,
reader, size, err := s.cache.GetSourceContent(sourceKey) ) (io.ReadCloser, int64, imagerequest.Format) {
if err != nil { reader, size, format, err := p.cache.Source(ctx, src)
s.log.Warn("failed to load cached source, fetching", "error", err) if errors.Is(err, imgcache.ErrNotFound) {
return nil, 0, ""
return nil, 0
} }
if size > s.maxResponseSize { if err != nil {
p.log.Warn("failed to load cached source, fetching", "error", err)
return nil, 0, ""
}
p.log.Debug("using cached source", "source_key", src.Key())
if size > p.maxResponseSize {
_ = reader.Close() _ = reader.Close()
s.log.Warn("cached source exceeds max response size, discarding", p.log.Warn("cached source exceeds max response size, discarding",
"source_key", sourceKey, "source_key", src.Key(),
"max_bytes", s.maxResponseSize, "max_bytes", p.maxResponseSize,
) )
return nil, 0 return nil, 0, ""
} }
if size == 0 { return reader, size, format
_ = reader.Close()
return nil, 0
}
return reader, size
} }
// processFromSourceOrFetch processes an image, using cached source content // processFromSourceOrFetch processes an image, using cached source content
// if available. // if available.
func (s *Service) processFromSourceOrFetch( func (p *Proxy) processFromSourceOrFetch(
ctx context.Context, ctx context.Context,
req *imagerequest.Request, req *imagerequest.Request,
cacheKey imagerequest.VariantKey, cacheKey imagerequest.VariantKey,
) (*processedVariant, error) { ) (*processedVariant, error) {
// Check if we have cached source content source, sourceSize, sourceFormat := p.loadCachedSource(ctx, req.Source)
sourceKey, err := s.cache.LookupSource(ctx, req)
if err != nil {
s.log.Warn("source lookup failed", "error", err)
}
var (
source io.ReadCloser
sourceSize int64
)
if sourceKey != "" {
s.log.Debug("using cached source", "source_key", sourceKey)
source, sourceSize = s.loadCachedSource(sourceKey)
}
// Fetch from upstream if we don't have source data or it's empty // Fetch from upstream if we don't have source data or it's empty
if source == nil { if source == nil {
return s.fetchAndProcess(ctx, req, cacheKey) return p.fetchAndProcess(ctx, req, cacheKey)
} }
defer func() { _ = source.Close() }() defer func() { _ = source.Close() }()
@@ -306,26 +356,26 @@ func (s *Service) processFromSourceOrFetch(
// Process using cached source; nothing was fetched from upstream. The // Process using cached source; nothing was fetched from upstream. The
// image processor reads the source only once it has a processing slot, // image processor reads the source only once it has a processing slot,
// so a request waiting for one holds none of it in memory. // so a request waiting for one holds none of it in memory.
return s.processAndStore(ctx, req, cacheKey, source, sourceSize) return p.processAndStore(ctx, req, cacheKey, source, sourceFormat, sourceSize)
} }
// fetchAndProcess fetches from upstream, processes, and caches the result. // fetchAndProcess fetches from upstream, processes, and caches the result.
// It counts the fetch with the bytes read from upstream, including when // It counts the fetch with the bytes read from upstream, including when
// reading the response or a later step fails. It fetches nothing while the // reading the response or a later step fails. It fetches nothing while the
// source URL is in the negative cache. // source URL is in the negative cache.
func (s *Service) fetchAndProcess( func (p *Proxy) fetchAndProcess(
ctx context.Context, ctx context.Context,
req *imagerequest.Request, req *imagerequest.Request,
cacheKey imagerequest.VariantKey, cacheKey imagerequest.VariantKey,
) (*processedVariant, error) { ) (*processedVariant, error) {
// Check negative cache - skip fetching for recently-failed URLs // Check negative cache - skip fetching for recently-failed URLs
negHit, err := s.cache.checkNegativeCache(ctx, req) negHit, err := p.cache.Failure(ctx, req.Source)
if err != nil { if err != nil {
s.log.Warn("negative cache check failed", "error", err) p.log.Warn("negative cache check failed", "error", err)
} }
if negHit { if negHit {
s.log.Debug("negative cache hit", p.log.Debug("negative cache hit",
"host", req.Source.Host, "host", req.Source.Host,
"path", req.Source.Path, "path", req.Source.Path,
) )
@@ -334,14 +384,14 @@ func (s *Service) fetchAndProcess(
} }
// Fetch from upstream // Fetch from upstream
s.log.Debug("fetching from upstream", "host", req.Source.Host, p.log.Debug("fetching from upstream", "host", req.Source.Host,
"path", req.Source.Path, "query", req.Source.Query) "path", req.Source.Path, "query", req.Source.Query)
fetchResult, err := s.fetcher.Fetch(ctx, req.Source) fetchResult, err := p.fetcher.Fetch(ctx, req.Source)
if err != nil { if err != nil {
// Store negative cache for certain errors // Store negative cache for certain errors
if isNegativeCacheable(err) { if isNegativeCacheable(err) {
s.storeNegative(ctx, req) p.storeNegative(ctx, req)
} }
return nil, fmt.Errorf("upstream fetch failed: %w", err) return nil, fmt.Errorf("upstream fetch failed: %w", err)
@@ -358,7 +408,7 @@ func (s *Service) fetchAndProcess(
fetchBytes := int64(len(sourceData)) fetchBytes := int64(len(sourceData))
// Counted also when the request context has ended meanwhile // Counted also when the request context has ended meanwhile
s.cache.IncrementUpstreamFetch(fetchBytes) p.incrementUpstreamFetch(fetchBytes)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read upstream response: %w", err) return nil, fmt.Errorf("failed to read upstream response: %w", err)
@@ -374,7 +424,7 @@ func (s *Service) fetchAndProcess(
} }
// Log upstream fetch details // Log upstream fetch details
s.log.Info("upstream fetched", p.log.Info("upstream fetched",
"request_id", middleware.GetReqID(ctx), "request_id", middleware.GetReqID(ctx),
"host", req.Source.Host, "host", req.Source.Host,
"path", req.Source.Path, "path", req.Source.Path,
@@ -387,38 +437,50 @@ func (s *Service) fetchAndProcess(
"cipher", fetchResult.TLSCipherSuite, "cipher", fetchResult.TLSCipherSuite,
) )
// Validate magic bytes match content type. A body that does not is the // The source's format, detected from its first bytes, which must match
// upstream's error, so it gets a negative cache entry, as an upstream // its content type. A body that does not is the upstream's error, so it
// error status does. // gets a negative cache entry, as an upstream error status does.
err = magic.ValidateMagicBytes(sourceData, fetchResult.ContentType) sourceFormat, err := magic.ValidateMagicBytes(sourceData,
fetchResult.ContentType)
if err != nil { if err != nil {
s.log.Warn("source refused", "request_id", middleware.GetReqID(ctx), p.log.Warn("source refused", "request_id", middleware.GetReqID(ctx),
"host", req.Source.Host, "path", req.Source.Path, "host", req.Source.Host, "path", req.Source.Path,
"content_type", fetchResult.ContentType, "error", err) "content_type", fetchResult.ContentType, "error", err)
s.storeNegative(ctx, req) p.storeNegative(ctx, req)
return nil, fmt.Errorf("content validation failed: %w", err) return nil, fmt.Errorf("content validation failed: %w", err)
} }
// Store source content // Store source content
_, err = s.cache.StoreSource(ctx, req, bytes.NewReader(sourceData)) _, err = p.cache.PutSource(ctx, req.Source, bytes.NewReader(sourceData))
if err != nil { if err != nil {
s.log.Warn("failed to store source content", "error", err) p.log.Warn("failed to store source content", "error", err)
// Continue even if caching fails // Continue even if caching fails
} }
return s.processAndStore( return p.processAndStore(
ctx, req, cacheKey, bytes.NewReader(sourceData), fetchBytes, ctx, req, cacheKey, bytes.NewReader(sourceData), sourceFormat, fetchBytes,
) )
} }
// incrementUpstreamFetch counts one upstream fetch that read fetchBytes bytes.
// A fetch that read no bytes is not counted.
func (p *Proxy) incrementUpstreamFetch(fetchBytes int64) {
if fetchBytes <= 0 {
return
}
p.upstreamFetches.Inc()
p.upstreamFetchBytes.Add(float64(fetchBytes))
}
// storeNegative stores a negative cache entry for req's source, whose fetch // storeNegative stores a negative cache entry for req's source, whose fetch
// failed, so the source is not fetched again until the entry expires. A // failed, so the source is not fetched again until the entry expires. A
// failure to store the entry is only logged. // failure to store the entry is only logged.
func (s *Service) storeNegative(ctx context.Context, req *imagerequest.Request) { func (p *Proxy) storeNegative(ctx context.Context, req *imagerequest.Request) {
err := s.cache.StoreNegative(ctx, req) err := p.cache.PutFailure(ctx, req.Source)
if err != nil { if err != nil {
s.log.Warn("failed to store negative cache entry", p.log.Warn("failed to store negative cache entry",
"host", req.Source.Host, "path", req.Source.Path, "error", err) "host", req.Source.Host, "path", req.Source.Path, "error", err)
} }
} }
@@ -431,13 +493,14 @@ type processedVariant struct {
fetchedBytes int64 fetchedBytes int64
} }
// processAndStore processes the image read from source and stores the // processAndStore processes the image read from source, which is in
// result. // sourceFormat, and stores the result.
func (s *Service) processAndStore( func (p *Proxy) processAndStore(
ctx context.Context, ctx context.Context,
req *imagerequest.Request, req *imagerequest.Request,
cacheKey imagerequest.VariantKey, cacheKey imagerequest.VariantKey,
source io.Reader, source io.Reader,
sourceFormat imagerequest.Format,
fetchBytes int64, fetchBytes int64,
) (*processedVariant, error) { ) (*processedVariant, error) {
// Process the image // Process the image
@@ -445,7 +508,8 @@ func (s *Service) processAndStore(
// ctx has the deadline of the request doing the processing, which can // ctx has the deadline of the request doing the processing, which can
// end while a request waiting for its image has time left // end while a request waiting for its image has time left
processResult, err := s.processor.Process(ctx, source, &req.Transform) processResult, err := p.processor.Process(ctx, source, sourceFormat,
&req.Transform)
if errors.Is(err, context.DeadlineExceeded) { if errors.Is(err, context.DeadlineExceeded) {
return nil, fmt.Errorf("%w: %w", ErrProcessingTimedOut, err) return nil, fmt.Errorf("%w: %w", ErrProcessingTimedOut, err)
} }
@@ -457,7 +521,7 @@ func (s *Service) processAndStore(
processDuration := time.Since(processStart) processDuration := time.Since(processStart)
// Counted also when the request context has ended meanwhile // Counted also when the request context has ended meanwhile
s.cache.IncrementTransformCount() p.transforms.Inc()
// Read processed content // Read processed content
processedData, err := io.ReadAll(processResult.Content) processedData, err := io.ReadAll(processResult.Content)
@@ -476,11 +540,11 @@ func (s *Service) processAndStore(
sizePercent = float64(outputSize) / float64(fetchBytes) * 100.0 sizePercent = float64(outputSize) / float64(fetchBytes) * 100.0
} }
s.log.Info("image converted", p.log.Info("image converted",
"request_id", middleware.GetReqID(ctx), "request_id", middleware.GetReqID(ctx),
"host", req.Source.Host, "host", req.Source.Host,
"path", req.Source.Path, "path", req.Source.Path,
"src_format", processResult.InputFormat, "src_format", sourceFormat,
"dst_format", req.Transform.Format, "dst_format", req.Transform.Format,
"src_bytes", fetchBytes, "src_bytes", fetchBytes,
"dst_bytes", outputSize, "dst_bytes", outputSize,
@@ -495,9 +559,9 @@ func (s *Service) processAndStore(
) )
// Store variant to cache // Store variant to cache
err = s.cache.StoreVariant(ctx, cacheKey, bytes.NewReader(processedData)) err = p.cache.PutVariant(ctx, cacheKey, bytes.NewReader(processedData))
if err != nil { if err != nil {
s.log.Warn("failed to store variant", "error", err) p.log.Warn("failed to store variant", "error", err)
// Continue even if caching fails // Continue even if caching fails
} }
internal/imgcache/service_internal_test.go → internal/imageproxy/imageproxy_internal_test.go
+43 -53
View File
@@ -1,4 +1,4 @@
package imgcache package imageproxy
import ( import (
"context" "context"
@@ -17,10 +17,10 @@ const (
testPathUpload = "/uploads/image.jpg" testPathUpload = "/uploads/image.jpg"
) )
func TestService_Get_AllowlistedHost(t *testing.T) { func TestProxy_Get_GoodHost(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx := context.Background() ctx := context.Background()
req := &imagerequest.Request{ req := &imagerequest.Request{
@@ -58,13 +58,12 @@ func TestService_Get_AllowlistedHost(t *testing.T) {
} }
} }
// TestService_Get_NonAllowlistedHost checks that the image service serves a // TestProxy_Get_OtherHost checks that the proxy serves a second host as it
// host that is not on the allowlist as any other: the /v1/image/ handler checks // serves GoodHost.
// the signature for such a host before it asks the service for the image. func TestProxy_Get_OtherHost(t *testing.T) {
func TestService_Get_NonAllowlistedHost(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx := context.Background() ctx := context.Background()
req := &imagerequest.Request{ req := &imagerequest.Request{
@@ -98,10 +97,10 @@ func TestService_Get_NonAllowlistedHost(t *testing.T) {
} }
} }
func TestService_Get_InvalidFile(t *testing.T) { func TestProxy_Get_InvalidFile(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx := context.Background() ctx := context.Background()
req := photoVariant(fixtures, 85, imagerequest.FitCover) req := photoVariant(fixtures, 85, imagerequest.FitCover)
@@ -115,10 +114,10 @@ func TestService_Get_InvalidFile(t *testing.T) {
} }
} }
func TestService_Get_NotFound(t *testing.T) { func TestProxy_Get_NotFound(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx := context.Background() ctx := context.Background()
req := photoVariant(fixtures, 85, imagerequest.FitCover) req := photoVariant(fixtures, 85, imagerequest.FitCover)
@@ -131,10 +130,10 @@ func TestService_Get_NotFound(t *testing.T) {
} }
} }
func TestService_Get_FormatConversion(t *testing.T) { func TestProxy_Get_FormatConversion(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx := context.Background() ctx := context.Background()
tests := []struct { tests := []struct {
@@ -192,32 +191,23 @@ func TestService_Get_FormatConversion(t *testing.T) {
t.Fatalf("failed to read response: %v", err) t.Fatalf("failed to read response: %v", err)
} }
detectedMIME, err := magic.DetectFormat(data) detectedFormat, err := magic.DetectFormat(data)
if err != nil { if err != nil {
t.Fatalf("failed to detect format: %v", err) t.Fatalf("failed to detect format: %v", err)
} }
expectedFormat, ok := magic.MIMEToImageFormat(tt.wantMIME) if detectedFormat.MIMEType() != tt.wantMIME {
if !ok { t.Errorf("detected format = %q, want the format of %s",
t.Fatalf("unknown format for MIME type: %s", tt.wantMIME) detectedFormat, tt.wantMIME)
}
detectedFormat, ok := magic.MIMEToImageFormat(string(detectedMIME))
if !ok {
t.Fatalf("unknown format for detected MIME type: %s", detectedMIME)
}
if detectedFormat != expectedFormat {
t.Errorf("detected format = %q, want %q", detectedFormat, expectedFormat)
} }
}) })
} }
} }
func TestService_Get_Caching(t *testing.T) { func TestProxy_Get_Caching(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx := context.Background() ctx := context.Background()
req := &imagerequest.Request{ req := &imagerequest.Request{
@@ -273,7 +263,7 @@ func TestService_Get_Caching(t *testing.T) {
} }
} }
// TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo caches a source and // TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo caches a source and
// a variant of it and sets back two days every time pixa stored for them. // a variant of it and sets back two days every time pixa stored for them.
// For the source: last_accessed_at in source_content, and the access and // For the source: last_accessed_at in source_content, and the access and
// modification times of its file. For the variant: last_accessed_at in // modification times of its file. For the variant: last_accessed_at in
@@ -284,10 +274,10 @@ func TestService_Get_Caching(t *testing.T) {
// Left at the present: the files' status change and birth times, which a // Left at the present: the files' status change and birth times, which a
// program cannot set, and the times of the directories holding the files, // program cannot set, and the times of the directories holding the files,
// which other entries share. // which other entries share.
func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) { func TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover) variant := photoVariant(fixtures, 85, imagerequest.FitCover)
resp, err := svc.Get(t.Context(), variant) resp, err := svc.Get(t.Context(), variant)
@@ -298,8 +288,8 @@ func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
_ = resp.Content.Close() _ = resp.Content.Close()
twoDaysAgo := time.Now().AddDate(0, 0, -2) twoDaysAgo := time.Now().AddDate(0, 0, -2)
backdateSource(t, svc.cache, variant, twoDaysAgo) backdateSource(t, svc, variant, twoDaysAgo)
backdateVariant(t, svc.cache, variant, twoDaysAgo) backdateVariant(t, svc, variant, twoDaysAgo)
resp, err = svc.Get(t.Context(), variant) resp, err = svc.Get(t.Context(), variant)
if err != nil { if err != nil {
@@ -321,23 +311,23 @@ func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
_ = resp.Content.Close() _ = resp.Content.Close()
fetches := readCounters(svc.cache).upstreamFetchCount fetches := readCounters(t, svc).upstreamFetchCount
if fetches != 1 { if fetches != 1 {
t.Errorf("upstream fetches = %d, want 1", fetches) t.Errorf("upstream fetches = %d, want 1", fetches)
} }
} }
// backdateSource sets every time pixa stored for req's cached source to when, // backdateSource sets every time pixa stored for req's cached source to when,
// as listed on TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo. // as listed on TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo.
func backdateSource( func backdateSource(
t *testing.T, cache *Cache, req *imagerequest.Request, when time.Time, t *testing.T, svc *testProxy, req *imagerequest.Request, when time.Time,
) { ) {
t.Helper() t.Helper()
key := req.Source.Key() key := req.Source.Key()
setSourceLastAccessed(t, cache, key, when) setSourceLastAccessed(t, svc.db, key, when)
path := cache.srcContent.keyToPath(key) path := sourceFilePath(svc, key)
err := os.Chtimes(path, when, when) err := os.Chtimes(path, when, when)
if err != nil { if err != nil {
@@ -347,16 +337,16 @@ func backdateSource(
// backdateVariant sets every time pixa stored for the cached variant req asks // backdateVariant sets every time pixa stored for the cached variant req asks
// for to when, as listed on // for to when, as listed on
// TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo. // TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo.
func backdateVariant( func backdateVariant(
t *testing.T, cache *Cache, req *imagerequest.Request, when time.Time, t *testing.T, svc *testProxy, req *imagerequest.Request, when time.Time,
) { ) {
t.Helper() t.Helper()
key := imagerequest.CacheKey(req) key := imagerequest.CacheKey(req)
path := cache.variants.keyToPath(key) path := variantFilePath(svc, key)
setVariantLastAccessed(t, cache, key, when) setVariantLastAccessed(t, svc.db, key, when)
err := os.Chtimes(path, when, when) err := os.Chtimes(path, when, when)
if err != nil { if err != nil {
@@ -364,10 +354,10 @@ func backdateVariant(
} }
} }
func TestService_Get_DifferentSizes(t *testing.T) { func TestProxy_Get_DifferentSizes(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx := context.Background() ctx := context.Background()
// Request same image at different sizes // Request same image at different sizes
@@ -418,10 +408,10 @@ func TestService_Get_DifferentSizes(t *testing.T) {
} }
} }
func TestService_Get_ContextCancellation(t *testing.T) { func TestProxy_Get_ContextCancellation(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
cancel() // Cancel immediately cancel() // Cancel immediately
@@ -445,10 +435,10 @@ func TestService_Get_ContextCancellation(t *testing.T) {
} }
} }
func TestService_Get_ReturnsETag(t *testing.T) { func TestProxy_Get_ReturnsETag(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx := context.Background() ctx := context.Background()
req := &imagerequest.Request{ req := &imagerequest.Request{
@@ -482,10 +472,10 @@ func TestService_Get_ReturnsETag(t *testing.T) {
} }
} }
func TestService_Get_ETagConsistency(t *testing.T) { func TestProxy_Get_ETagConsistency(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx := context.Background() ctx := context.Background()
req := &imagerequest.Request{ req := &imagerequest.Request{
@@ -527,10 +517,10 @@ func TestService_Get_ETagConsistency(t *testing.T) {
} }
} }
func TestService_Get_DifferentETagsForDifferentContent(t *testing.T) { func TestProxy_Get_DifferentETagsForDifferentContent(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx := context.Background() ctx := context.Background()
// Request same image at different sizes - should get different ETags // Request same image at different sizes - should get different ETags
internal/imgcache/max_concurrent_processing_internal_test.go → internal/imageproxy/max_concurrent_processing_internal_test.go
+19 -15
View File
@@ -1,4 +1,4 @@
package imgcache package imageproxy
import ( import (
"image/color" "image/color"
@@ -41,7 +41,8 @@ func holdProcessingSlot(
input, feed := io.Pipe() input, feed := io.Pipe()
go func() { go func() {
_, _ = proc.Process(t.Context(), input, &imagerequest.Transform{}) _, _ = proc.Process(t.Context(), input, imagerequest.FormatJPEG,
&imagerequest.Transform{})
}() }()
_, err := feed.Write([]byte{0}) _, err := feed.Write([]byte{0})
@@ -55,16 +56,17 @@ func holdProcessingSlot(
return release return release
} }
// TestService_Get_WaitsForSlotBeforeReadingCachedSource checks that a // TestProxy_Get_WaitsForSlotBeforeReadingCachedSource checks that a
// request whose source is cached holds none of it while it waits for a // request whose source is cached holds none of it while it waits for a
// processing slot: it reads the cached file only once it has a slot. With // processing slot: past the first bytes, which give the source's format, it
// the only slot held, a request for a new width of the cached 100x100 photo // reads the cached file only once it has a slot. With the only slot held, a
// waits; the cached file is then rewritten as a 100x50 image before the slot // request for a new width of the cached 100x100 photo waits; the cached file
// is freed, so the request must answer with that image scaled to 40x20. // is then rewritten as a 100x50 image before the slot is freed, so the
func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) { // request must answer with that image scaled to 40x20.
func TestProxy_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
svc.processor = imageprocessor.New( svc.processor = imageprocessor.New(
imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1}, imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1},
) )
@@ -77,13 +79,15 @@ func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
_ = resp.Content.Close() _ = resp.Content.Close()
sourceKey, err := svc.cache.LookupSource(t.Context(), cachedSource := widthOnlyRequest(fixtures, 50).Source
widthOnlyRequest(fixtures, 50))
if err != nil || sourceKey == "" { reader, _, _, err := svc.cache.Source(t.Context(), cachedSource)
t.Fatalf("LookupSource() = %q, %v; want the cached source", if err != nil {
sourceKey, err) t.Fatalf("Source() error = %v; want the cached source", err)
} }
_ = reader.Close()
release := holdProcessingSlot(t, svc.processor) release := holdProcessingSlot(t, svc.processor)
var ( var (
@@ -103,7 +107,7 @@ func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
// before waiting, it would have read it by now. // before waiting, it would have read it by now.
time.Sleep(100 * time.Millisecond) time.Sleep(100 * time.Millisecond)
err = os.WriteFile(svc.cache.srcContent.keyToPath(sourceKey), err = os.WriteFile(sourceFilePath(svc, cachedSource.Key()),
generateTestJPEG(t, 100, 50, color.RGBA{0, 0, 255, 255}), 0o600) generateTestJPEG(t, 100, 50, color.RGBA{0, 0, 255, 255}), 0o600)
if err != nil { if err != nil {
t.Fatalf("failed to rewrite the cached source: %v", err) t.Fatalf("failed to rewrite the cached source: %v", err)
internal/imageproxy/negative_cache_internal_test.go
+96
View File
@@ -0,0 +1,96 @@
package imageproxy
import (
"errors"
"os"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestProxy_Get_ServesCachedVariantOfNegativeCachedURL caches a variant of
// the test photo, then keeps a failed fetch of the photo's URL in the negative
// cache, as a failed fetch for another variant does once the source has been
// evicted. The cached variant is still served, and a variant not in the cache
// is made from the cached source. Once the source is evicted, a variant not in
// the cache needs a fetch and is refused.
func TestProxy_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
resp, err := svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
_ = resp.Content.Close()
err = svc.cache.PutFailure(t.Context(),
photoVariant(fixtures, 85, imagerequest.FitCover).Source)
if err != nil {
t.Fatalf("PutFailure() error = %v", err)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() of the cached variant error = %v", err)
}
_ = resp.Content.Close()
if resp.CacheStatus != CacheHit {
t.Errorf("cached variant CacheStatus = %q, want %q",
resp.CacheStatus, CacheHit)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
t.Logf("Get() of a variant made from the cached source: error = %v", err)
if err != nil {
t.Fatalf("Get() of a variant made from the cached source error = %v", err)
}
_ = resp.Content.Close()
source := photoVariant(fixtures, 85, imagerequest.FitCover).Source
reader, _, _, err := svc.cache.Source(t.Context(), source)
if err != nil {
t.Fatalf("Source() error = %v, want the cached source", err)
}
_ = reader.Close()
// Evicts the source as the cache's eviction does: its row, then its file
_, err = svc.db.ExecContext(t.Context(),
`DELETE FROM source_content WHERE source_key = ?`, string(source.Key()))
if err != nil {
t.Fatalf("deleting the source's row: %v", err)
}
err = os.Remove(sourceFilePath(svc, source.Key()))
if err != nil {
t.Fatalf("removing the source's file: %v", err)
}
_, err = svc.Get(t.Context(), photoVariant(fixtures, 60, imagerequest.FitCover))
t.Logf("Get() of a variant whose source was evicted: error = %v", err)
if !errors.Is(err, ErrNegativeCached) {
t.Errorf("Get() of a variant whose source was evicted error = %v, want %v",
err, ErrNegativeCached)
}
}
func TestProxy_Get_ReturnsErrorForNegativeCachedURL(t *testing.T) {
t.Parallel()
// This test verifies that Proxy.Get() checks the negative cache
// We can't easily test the full pipeline without vips, but we can
// verify the error type
err := ErrNegativeCached
if !errors.Is(err, ErrNegativeCached) {
t.Error("ErrNegativeCached should be identifiable with errors.Is")
}
}
internal/imageproxy/partial_cache_files_internal_test.go
+184
View File
@@ -0,0 +1,184 @@
package imageproxy
import (
"bytes"
"io"
"os"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// getImage calls Get for req and returns the image it answered with and
// its cache status.
func getImage(
t *testing.T, svc *testProxy, req *imagerequest.Request,
) ([]byte, CacheStatus) {
t.Helper()
resp, err := svc.Get(t.Context(), req)
if err != nil {
t.Fatalf("Get() of quality %d error = %v", req.Transform.Quality, err)
}
defer func() { _ = resp.Content.Close() }()
body, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("reading the image of quality %d: %v", req.Transform.Quality, err)
}
return body, resp.CacheStatus
}
// fileSize returns the size of the file at path.
func fileSize(t *testing.T, path string) int64 {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
return info.Size()
}
// cachedSourcePath returns the path of the cached source file req uses.
func cachedSourcePath(
t *testing.T, svc *testProxy, req *imagerequest.Request,
) string {
t.Helper()
reader, _, _, err := svc.cache.Source(t.Context(), req.Source)
if err != nil {
t.Fatalf("Source() error = %v; want the cached source", err)
}
_ = reader.Close()
return sourceFilePath(svc, req.Source.Key())
}
// TestProxy_Get_FetchesAgainForRemovedCachedSource removes the cached
// source's row and file, as the reconciliation pass does when the file is
// empty or cut short. internal/imgcache tests that the pass does so. The
// test then asks for two new sizes of the source. The first must fetch it
// again and cache it whole, so the second is made from the cache without a
// third fetch.
func TestProxy_Get_FetchesAgainForRemovedCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
source := photoVariant(fixtures, 85, imagerequest.FitCover).Source
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
// Removes the source as the reconciliation pass does: its row, then its file
_, err := svc.db.ExecContext(t.Context(),
`DELETE FROM source_content WHERE source_key = ?`, string(source.Key()))
if err != nil {
t.Fatalf("deleting the source's row: %v", err)
}
err = os.Remove(path)
if err != nil {
t.Fatalf("removing the source's file: %v", err)
}
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
fetches := readCounters(t, svc).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// TestProxy_Get_FetchesAgainSourceOfNoSourceFormat replaces the cached
// source's file with text. The next request for a new size of the source
// must fetch it again and cache it whole.
func TestProxy_Get_FetchesAgainSourceOfNoSourceFormat(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
err := os.WriteFile(path, []byte("this is text, not an image"),
imgcache.StorageFilePerm)
if err != nil {
t.Fatalf("replacing the source's file: %v", err)
}
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
fetches := readCounters(t, svc).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// TestProxy_Get_ConvertsAgainForRemovedCachedVariant caches a variant of
// the photo and removes its row and file, as the reconciliation pass does
// when the file is empty or cut short. internal/imgcache tests that the
// pass does so. The test then checks that Get converts the image again, and
// that the next Get serves the whole variant from the cache.
func TestProxy_Get_ConvertsAgainForRemovedCachedVariant(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
key := imagerequest.CacheKey(req)
// Removes the variant as the reconciliation pass does: its row, then its file
_, err := svc.db.ExecContext(t.Context(),
`DELETE FROM variant_content WHERE cache_key = ?`, string(key))
if err != nil {
t.Fatalf("deleting the variant's row: %v", err)
}
err = os.Remove(variantFilePath(svc, key))
if err != nil {
t.Fatalf("removing the variant's file: %v", err)
}
got, status := getImage(t, svc, req)
t.Logf("after the removal: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the removal: Get() = %d bytes, %s; want the %d bytes "+
"converted again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
t.Logf("next request: %d bytes, %s", len(got), status)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
internal/imageproxy/pendingaccesstimes_internal_test.go
+78
View File
@@ -0,0 +1,78 @@
package imageproxy
import (
"context"
"database/sql"
"errors"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// pendingAccessTimesWait is how long a test waits for the database's
// connection, for a cache hit to be answered, for an access time to reach
// the database, or for a request to return past its deadline.
const pendingAccessTimesWait = 5 * time.Second
// holdDatabase takes the one connection of db, so that every other query
// waits for it, and returns the func that frees it.
func holdDatabase(t *testing.T, db *sql.DB) func() {
t.Helper()
ctx, cancel := context.WithTimeout(t.Context(), pendingAccessTimesWait)
defer cancel()
conn, err := db.Conn(ctx)
if err != nil {
t.Fatalf("failed to take the database connection: %v", err)
}
return func() { _ = conn.Close() }
}
// TestProxy_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy holds the
// database's one connection while a request whose fetch is held reaches its
// deadline. The request must return with the deadline's error while the
// connection is still held, as it waits for no database write past its
// deadline.
func TestProxy_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
// Room for the request to reach its fetch on a busy host
const timeout = 2 * time.Second
ctx, cancel := context.WithTimeout(t.Context(), timeout)
defer cancel()
deadline, _ := ctx.Deadline()
results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
// The request has made its database reads by the time it fetches.
select {
case <-fetcher.started:
case <-time.After(timeout):
t.Fatal("request did not reach its fetch by its deadline")
}
releaseDatabase := holdDatabase(t, svc.db)
defer releaseDatabase()
select {
case got := <-results:
t.Logf("Get() returned %v after its deadline, error = %v",
time.Since(deadline), got.err)
if !errors.Is(got.err, context.DeadlineExceeded) {
t.Errorf("Get() error = %v, want %v", got.err, context.DeadlineExceeded)
}
case <-time.After(time.Until(deadline) + pendingAccessTimesWait):
t.Fatal("request did not return while the database was busy")
}
}
internal/imgcache/stats_internal_test.go → internal/imageproxy/stats_internal_test.go
+22 -20
View File
@@ -1,4 +1,4 @@
package imgcache package imageproxy
import ( import (
"bytes" "bytes"
@@ -12,15 +12,16 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
) )
// TestService_Get_CountsStats walks Get through a miss that fetches the // TestProxy_Get_CountsStats walks Get through a miss that fetches the
// source, a hit, a miss that reuses the cached source, and two misses whose // source, a hit, a miss that reuses the cached source, and two misses whose
// source cannot be used, checking every counter of the cache after each. // source cannot be used, checking every counter of the proxy and its cache
func TestService_Get_CountsStats(t *testing.T) { // after each.
func TestProxy_Get_CountsStats(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
// NewTestFS builds the same files the test service's fetcher serves. // NewTestFS builds the same files the test proxy's fetcher serves.
testFS, _ := NewTestFS(t) testFS, _ := NewTestFS(t)
photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG) photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG)
@@ -76,7 +77,7 @@ func TestService_Get_CountsStats(t *testing.T) {
_ = resp.Content.Close() _ = resp.Content.Close()
} }
got := readCounters(svc.cache) got := readCounters(t, svc)
if got != step.want { if got != step.want {
t.Fatalf("after the %s: counters = %+v, want %+v", step.name, got, step.want) t.Fatalf("after the %s: counters = %+v, want %+v", step.name, got, step.want)
} }
@@ -114,10 +115,11 @@ func (u *fakeUpstream) Read(p []byte) (int, error) {
return 0, u.err return 0, u.err
} }
// TestService_Get_CountsInterruptedMisses checks every counter of the cache // TestProxy_Get_CountsInterruptedMisses checks every counter of the proxy
// after a miss whose request context ends during or after the upstream // and its cache after a miss whose request context ends during or after the
// fetch, and after a miss whose upstream body is over the size limit. // upstream fetch, and after a miss whose upstream body is over the size
func TestService_Get_CountsInterruptedMisses(t *testing.T) { // limit.
func TestProxy_Get_CountsInterruptedMisses(t *testing.T) {
t.Parallel() t.Parallel()
photo := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255}) photo := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
@@ -145,7 +147,7 @@ func TestService_Get_CountsInterruptedMisses(t *testing.T) {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
ctx, cancel := context.WithCancel(t.Context()) ctx, cancel := context.WithCancel(t.Context())
defer cancel() defer cancel()
@@ -182,7 +184,7 @@ func TestService_Get_CountsInterruptedMisses(t *testing.T) {
_ = resp.Content.Close() _ = resp.Content.Close()
} }
got := readCounters(svc.cache) got := readCounters(t, svc)
if got != tc.want { if got != tc.want {
t.Errorf("counters = %+v, want %+v", got, tc.want) t.Errorf("counters = %+v, want %+v", got, tc.want)
} }
@@ -190,13 +192,13 @@ func TestService_Get_CountsInterruptedMisses(t *testing.T) {
} }
} }
// TestService_Get_CountsHitAfterRequestEnds checks every counter of the // TestProxy_Get_CountsHitAfterRequestEnds checks every counter of the
// cache after a hit served with a request context that has already ended: // proxy and its cache after a hit served with a request context that has
// only the hit count moves. // already ended: only the hit count moves.
func TestService_Get_CountsHitAfterRequestEnds(t *testing.T) { func TestProxy_Get_CountsHitAfterRequestEnds(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) svc, fixtures := setupTestProxy(t)
req := &imagerequest.Request{ req := &imagerequest.Request{
Source: imagerequest.Source{ Source: imagerequest.Source{
@@ -219,7 +221,7 @@ func TestService_Get_CountsHitAfterRequestEnds(t *testing.T) {
_ = resp.Content.Close() _ = resp.Content.Close()
want := readCounters(svc.cache) want := readCounters(t, svc)
want.hitCount++ want.hitCount++
ctx, cancel := context.WithCancel(t.Context()) ctx, cancel := context.WithCancel(t.Context())
@@ -236,7 +238,7 @@ func TestService_Get_CountsHitAfterRequestEnds(t *testing.T) {
t.Fatalf("CacheStatus = %v, want %v", resp.CacheStatus, CacheHit) t.Fatalf("CacheStatus = %v, want %v", resp.CacheStatus, CacheHit)
} }
got := readCounters(svc.cache) got := readCounters(t, svc)
if got != want { if got != want {
t.Errorf("counters = %+v, want %+v", got, want) t.Errorf("counters = %+v, want %+v", got, want)
} }
internal/imageproxy/testutil_internal_test.go
+355
View File
@@ -0,0 +1,355 @@
package imageproxy
import (
"bytes"
"context"
"database/sql"
"image"
"image/color"
"image/gif"
"image/jpeg"
"image/png"
"io/fs"
"path/filepath"
"testing"
"testing/fstest"
"time"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// Shared test data literals, extracted as constants for goconst.
const (
testContentTypeJPEG = "image/jpeg"
)
// sqliteTimestampFormat matches the format SQLite's CURRENT_TIMESTAMP
// produces, so injected timestamps compare correctly against ones the
// cache writes.
const sqliteTimestampFormat = "2006-01-02 15:04:05"
// TestFixtures contains paths to test files in the mock filesystem.
type TestFixtures struct {
// Valid image files
GoodHostJPEG string // valid JPEG on GoodHost
GoodHostPNG string // valid PNG on GoodHost
GoodHostGIF string // valid GIF on GoodHost
OtherHostJPEG string // valid JPEG on OtherHost
OtherHostPNG string // valid PNG on OtherHost
// Invalid/edge case files
InvalidFile string // file with wrong magic bytes
EmptyFile string // zero-byte file
TextFile string // text file masquerading as image
// Hostnames
GoodHost string // a host the mock fetcher serves
OtherHost string // a second host the mock fetcher serves
}
// DefaultFixtures returns the standard test fixture paths.
func DefaultFixtures() *TestFixtures {
return &TestFixtures{
GoodHostJPEG: "goodhost.example.com/images/photo.jpg",
GoodHostPNG: "goodhost.example.com/images/logo.png",
GoodHostGIF: "goodhost.example.com/images/animation.gif",
OtherHostJPEG: "otherhost.example.com/uploads/image.jpg",
OtherHostPNG: "otherhost.example.com/uploads/icon.png",
InvalidFile: "goodhost.example.com/images/fake.jpg",
EmptyFile: "goodhost.example.com/images/empty.jpg",
TextFile: "goodhost.example.com/images/text.png",
GoodHost: "goodhost.example.com",
OtherHost: "otherhost.example.com",
}
}
// NewTestFS creates a mock filesystem with test images.
func NewTestFS(t *testing.T) (fs.FS, *TestFixtures) {
t.Helper()
fixtures := DefaultFixtures()
// Generate test images
jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
pngData := generateTestPNG(t, 100, 100, color.RGBA{0, 255, 0, 255})
gifData := generateTestGIF(t, 100, 100, color.RGBA{0, 0, 255, 255})
// Create the mock filesystem
mockFS := fstest.MapFS{
// Good host files
fixtures.GoodHostJPEG: &fstest.MapFile{Data: jpegData},
fixtures.GoodHostPNG: &fstest.MapFile{Data: pngData},
fixtures.GoodHostGIF: &fstest.MapFile{Data: gifData},
// Other host files
fixtures.OtherHostJPEG: &fstest.MapFile{Data: jpegData},
fixtures.OtherHostPNG: &fstest.MapFile{Data: pngData},
// Invalid files
fixtures.InvalidFile: &fstest.MapFile{Data: []byte("not a real image file content")},
fixtures.EmptyFile: &fstest.MapFile{Data: []byte{}},
fixtures.TextFile: &fstest.MapFile{Data: []byte("This is just text, not a PNG")},
}
return mockFS, fixtures
}
// generateTestJPEG creates a minimal valid JPEG image.
func generateTestJPEG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 85})
if err != nil {
t.Fatalf("failed to encode test JPEG: %v", err)
}
return buf.Bytes()
}
// generateTestPNG creates a minimal valid PNG image.
func generateTestPNG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
err := png.Encode(&buf, img)
if err != nil {
t.Fatalf("failed to encode test PNG: %v", err)
}
return buf.Bytes()
}
// generateTestGIF creates a minimal valid GIF image.
func generateTestGIF(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewPaletted(
image.Rect(0, 0, width, height),
[]color.Color{c, color.White},
)
for y := range height {
for x := range width {
img.SetColorIndex(x, y, 0)
}
}
var buf bytes.Buffer
err := gif.Encode(&buf, img, nil)
if err != nil {
t.Fatalf("failed to encode test GIF: %v", err)
}
return buf.Bytes()
}
// testProxy is a Proxy with the database and the state directory of its
// cache, through which a test reads and changes the cache behind the proxy.
// registry is the metrics registry the proxy and its cache share.
type testProxy struct {
*Proxy
db *sql.DB
stateDir string
registry *prometheus.Registry
}
// setupTestProxy creates a Proxy with mock fetcher for testing.
func setupTestProxy(t *testing.T) (*testProxy, *TestFixtures) {
t.Helper()
mockFS, fixtures := NewTestFS(t)
// Create temp directory for cache
tmpDir := t.TempDir()
// Create in-memory database
db := setupTestDB(t)
registry := prometheus.NewRegistry()
cache, err := imgcache.NewCache(db, imgcache.Config{
StateDir: tmpDir,
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
}, registry)
if err != nil {
t.Fatalf("failed to create cache: %v", err)
}
proxy, err := New(&Config{
Cache: cache,
Fetcher: httpfetcher.NewMock(mockFS),
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
}, registry)
if err != nil {
t.Fatalf("failed to create proxy: %v", err)
}
return &testProxy{
Proxy: proxy,
db: db,
stateDir: tmpDir,
registry: registry,
}, fixtures
}
// counterValues holds the values of the counters of a proxy and its cache.
type counterValues struct {
hitCount int64
missCount int64
upstreamFetchCount int64
upstreamFetchBytes int64
transformCount int64
}
// readCounters reads the counters of svc and its cache from their metrics
// registry.
func readCounters(t *testing.T, svc *testProxy) counterValues {
t.Helper()
families, err := svc.registry.Gather()
if err != nil {
t.Fatalf("failed to gather the counters: %v", err)
}
values := make(map[string]int64, len(families))
for _, family := range families {
values[family.GetName()] = int64(family.GetMetric()[0].GetCounter().GetValue())
}
return counterValues{
hitCount: values["pixa_cache_hits_total"],
missCount: values["pixa_cache_misses_total"],
upstreamFetchCount: values["pixa_upstream_fetches_total"],
upstreamFetchBytes: values["pixa_upstream_fetch_bytes_total"],
transformCount: values["pixa_transforms_total"],
}
}
// sourceFilePath returns the path of the file of the source key in svc's
// cache, as README.md gives it: <state_dir>/cache/sources/<ab>/<cd>/<key>.
func sourceFilePath(svc *testProxy, key imagerequest.SourceKey) string {
name := string(key)
return filepath.Join(svc.stateDir, "cache", "sources", name[0:2], name[2:4], name)
}
// variantFilePath returns the path of the file of the variant key in svc's
// cache, as README.md gives it: <state_dir>/cache/variants/<ab>/<cd>/<key>.
func variantFilePath(svc *testProxy, key imagerequest.VariantKey) string {
name := string(key)
return filepath.Join(svc.stateDir, "cache", "variants", name[0:2], name[2:4], name)
}
// setVariantLastAccessed backdates the last access time of a tracked
// variant.
func setVariantLastAccessed(
t *testing.T, db *sql.DB, key imagerequest.VariantKey, when time.Time,
) {
t.Helper()
res, err := db.ExecContext(t.Context(),
`UPDATE variant_content SET last_accessed_at = ? WHERE cache_key = ?`,
when.UTC().Format(sqliteTimestampFormat), string(key),
)
if err != nil {
t.Fatalf("failed to set variant last_accessed_at: %v", err)
}
affected, err := res.RowsAffected()
if err != nil {
t.Fatalf("failed to read affected rows: %v", err)
}
if affected != 1 {
t.Fatalf("variant %s has no accounting row (affected=%d); "+
"stores must track variants in the database", key, affected)
}
}
// setSourceLastAccessed backdates the last access time of a tracked
// source content blob.
func setSourceLastAccessed(
t *testing.T, db *sql.DB, key imagerequest.SourceKey, when time.Time,
) {
t.Helper()
res, err := db.ExecContext(t.Context(),
`UPDATE source_content SET last_accessed_at = ? WHERE source_key = ?`,
when.UTC().Format(sqliteTimestampFormat), string(key),
)
if err != nil {
t.Fatalf("failed to set source last_accessed_at: %v", err)
}
affected, err := res.RowsAffected()
if err != nil {
t.Fatalf("failed to read affected rows: %v", err)
}
if affected != 1 {
t.Fatalf("source %s has no accounting row (affected=%d)", key, affected)
}
}
// countRows returns the number of rows the given query yields.
func countRows(t *testing.T, db *sql.DB, query string, args ...any) int {
t.Helper()
var n int
err := db.QueryRowContext(t.Context(), query, args...).Scan(&n)
if err != nil {
t.Fatalf("count query %q failed: %v", query, err)
}
return n
}
// setupTestDB creates an in-memory SQLite database for testing
// using the production schema.
func setupTestDB(t *testing.T) *sql.DB {
t.Helper()
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatalf("failed to open test db: %v", err)
}
// Use the real production schema via migrations
err = database.ApplyMigrations(context.Background(), db, nil)
if err != nil {
t.Fatalf("failed to apply migrations: %v", err)
}
return db
}
internal/imagerequest/imagerequest.go
+30 -4
View File
@@ -1,7 +1,8 @@
// Package imagerequest holds the image request type, its parts, its checks, // Package imagerequest holds the image request type, its parts, its checks,
// the parsing of the /v1/image/ path, the variant key, the source key and the // the parsing of the /v1/image/ path, the variant key, the source key and the
// source's upstream URL. It imports no other package of this repo, so every // source's upstream URL. It also holds the image format type and the MIME type
// package that handles an image request can import it. // of each format. It imports no other package of this repo, so every package
// that handles an image request can import it.
package imagerequest package imagerequest
import ( import (
@@ -12,10 +13,11 @@ import (
// ErrInvalidFitMode is returned when an invalid fit mode is provided. // ErrInvalidFitMode is returned when an invalid fit mode is provided.
var ErrInvalidFitMode = errors.New("invalid fit mode") var ErrInvalidFitMode = errors.New("invalid fit mode")
// Format represents supported output image formats. // Format is an image format. pixa reads and writes the formats from JPEG to
// GIF below. In a request, orig and auto stand for one of them.
type Format string type Format string
// Supported image output formats. // Supported image formats.
const ( const (
FormatOriginal Format = "orig" FormatOriginal Format = "orig"
FormatJPEG Format = "jpeg" FormatJPEG Format = "jpeg"
@@ -31,6 +33,30 @@ const (
FormatAuto Format = "auto" FormatAuto Format = "auto"
) )
// MIMEType returns the MIME type of the format, such as image/jpeg for
// FormatJPEG. orig and auto stand for another format, so they have none:
// MIMEType returns "" for them.
func (f Format) MIMEType() string {
switch f {
case FormatJPEG:
return "image/jpeg"
case FormatPNG:
return "image/png"
case FormatWebP:
return "image/webp"
case FormatAVIF:
return "image/avif"
case FormatJXL:
return "image/jxl"
case FormatGIF:
return "image/gif"
case FormatOriginal, FormatAuto:
return ""
default:
return ""
}
}
// Size represents requested image dimensions // Size represents requested image dimensions
type Size struct { type Size struct {
Width int Width int
internal/imagerequest/mimetype_internal_test.go
+30
View File
@@ -0,0 +1,30 @@
package imagerequest
import "testing"
// TestFormat_MIMEType checks the MIME type of each format pixa reads and
// writes, and that orig and auto have none.
func TestFormat_MIMEType(t *testing.T) {
t.Parallel()
tests := []struct {
format Format
want string
}{
{FormatJPEG, "image/jpeg"},
{FormatPNG, "image/png"},
{FormatWebP, "image/webp"},
{FormatAVIF, "image/avif"},
{FormatJXL, "image/jxl"},
{FormatGIF, "image/gif"},
{FormatOriginal, ""},
{FormatAuto, ""},
}
for _, tt := range tests {
got := tt.format.MIMEType()
if got != tt.want {
t.Errorf("%s MIMEType() = %q, want %q", tt.format, got, tt.want)
}
}
}
internal/imagerequest/urlparser.go
+2 -1
View File
@@ -133,7 +133,8 @@ func parseImageComponents(remainder string) (*Request, error) {
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host) return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
} }
// Go's URL parser refuses brackets around anything but an IPv6 address // The allowlist check and the fetcher drop the brackets, so brackets
// around anything but an IPv6 address would give a host a second spelling
if inBrackets := hostMatch[1]; inBrackets != "" { if inBrackets := hostMatch[1]; inBrackets != "" {
addr, err := netip.ParseAddr(inBrackets) addr, err := netip.ParseAddr(inBrackets)
if err != nil || !addr.Is6() { if err != nil || !addr.Is6() {
internal/imgcache/cache.go
+176 -151
View File
@@ -1,3 +1,10 @@
// Package imgcache is the image cache. It keeps:
// - the source images fetched from upstream
// - the variants made from them
// - the negative cache: the source URLs whose fetch failed recently
//
// Each image is a file with a row in the database. Eviction removes the
// least recently used images to keep the cache under its size limit.
package imgcache package imgcache
import ( import (
@@ -12,6 +19,7 @@ import (
"github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic"
) )
// maxPendingAccessTimes is how many access times of variants served from // maxPendingAccessTimes is how many access times of variants served from
@@ -26,8 +34,12 @@ const maxPendingAccessTimes = 10000
// have expired. // have expired.
const cleanExpiredBatchSize = 1000 const cleanExpiredBatchSize = 1000
// CacheConfig holds cache configuration. // DefaultNegativeTTL is how long pixad keeps a failed fetch in the negative
type CacheConfig struct { // cache.
const DefaultNegativeTTL = 5 * time.Minute
// Config holds cache configuration.
type Config struct {
StateDir string StateDir string
NegativeTTL time.Duration NegativeTTL time.Duration
@@ -46,9 +58,9 @@ type CacheConfig struct {
// Cache implements the caching layer for the image proxy. // Cache implements the caching layer for the image proxy.
type Cache struct { type Cache struct {
db *sql.DB db *sql.DB
srcContent *ContentStorage // source images by source key srcContent *FileStore // source images by source key
variants *VariantStorage // processed variants by cache key variants *FileStore // processed variants by cache key
config CacheConfig config Config
log *slog.Logger log *slog.Logger
// Eviction machinery. The channels are created in NewCache so // Eviction machinery. The channels are created in NewCache so
@@ -73,13 +85,10 @@ type Cache struct {
pendingAccessTimesDone chan struct{} pendingAccessTimesDone chan struct{}
pendingAccessTimesCancel context.CancelFunc pendingAccessTimesCancel context.CancelFunc
// The counts of what the cache does, made and registered by // The counts of cache hits and misses, made and registered by
// registerCounters (see counters.go) // registerCounters (see counters.go)
cacheHits prometheus.Counter cacheHits prometheus.Counter
cacheMisses prometheus.Counter cacheMisses prometheus.Counter
upstreamFetches prometheus.Counter
upstreamFetchBytes prometheus.Counter
transforms prometheus.Counter
// reconciliationPageSize is the most rows one read of a content table // reconciliationPageSize is the most rows one read of a content table
// returns in the reconciliation pass. NewCache sets it to // returns in the reconciliation pass. NewCache sets it to
@@ -97,7 +106,7 @@ type Cache struct {
// metricsRegistry. pixad passes Prometheus' default registry, the one // metricsRegistry. pixad passes Prometheus' default registry, the one
// /metrics serves. It makes no database call. // /metrics serves. It makes no database call.
func NewCache( func NewCache(
db *sql.DB, config CacheConfig, metricsRegistry prometheus.Registerer, db *sql.DB, config Config, metricsRegistry prometheus.Registerer,
) (*Cache, error) { ) (*Cache, error) {
log := config.Logger log := config.Logger
if log == nil { if log == nil {
@@ -127,14 +136,14 @@ func NewCache(
return c, nil return c, nil
} }
srcContent, err := NewContentStorage( srcContent, err := NewFileStore(
filepath.Join(config.StateDir, "cache", "sources"), filepath.Join(config.StateDir, "cache", "sources"),
) )
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to create source content storage: %w", err) return nil, fmt.Errorf("failed to create source content storage: %w", err)
} }
variants, err := NewVariantStorage( variants, err := NewFileStore(
filepath.Join(config.StateDir, "cache", "variants"), filepath.Join(config.StateDir, "cache", "variants"),
) )
if err != nil { if err != nil {
@@ -147,86 +156,60 @@ func NewCache(
return c, nil return c, nil
} }
// GetVariant opens a cached variant and returns a reader, its size and its // Variant opens a cached variant and returns a reader, its size and its
// content type, read from the variant file's first bytes. It opens that one // content type, read from the variant file's first bytes. It opens that one
// file and makes no database call. A variant not in the cache, or any // file and makes no database call. A variant it opens has its access time
// variant of a disabled cache, is ErrNotFound; a variant file whose first // handed to the background writer without waiting (see touchVariant). A
// bytes are not those of an output format is another error. // variant not in the cache, or any variant of a disabled cache, is
func (c *Cache) GetVariant( // ErrNotFound; a variant file whose first bytes are not those of an output
// format is another error.
func (c *Cache) Variant(
cacheKey imagerequest.VariantKey, cacheKey imagerequest.VariantKey,
) (io.ReadCloser, int64, string, error) { ) (io.ReadCloser, int64, string, error) {
if c.config.MaxBytes == 0 { if c.config.MaxBytes == 0 {
return nil, 0, "", ErrNotFound return nil, 0, "", ErrNotFound
} }
return c.variants.LoadWithContentType(cacheKey) file, size, err := c.variants.Open(string(cacheKey))
if err != nil {
return nil, 0, "", err
}
format, err := readFormat(file)
if err != nil {
_ = file.Close()
return nil, 0, "", err
}
c.touchVariant(cacheKey)
return file, size, format.MIMEType(), nil
} }
// StoreSource stores fetched source content: it writes the file named by // PutVariant stores a processed variant by its cache key and records
// the source's key, then the source's row. When the row cannot be written
// the file is removed and the store fails, as a file with no row is never
// served or evicted; if that removal fails too, the next reconciliation
// pass removes the file. On a disabled cache it is a no-op returning an
// empty key.
func (c *Cache) StoreSource(
ctx context.Context,
req *imagerequest.Request,
content io.Reader,
) (imagerequest.SourceKey, error) {
if c.config.MaxBytes == 0 {
return "", nil
}
data, err := io.ReadAll(content)
if err != nil {
return "", fmt.Errorf("failed to read source content: %w", err)
}
key := req.Source.Key()
size, err := c.srcContent.Store(key, data)
if err != nil {
return "", fmt.Errorf("failed to store source content: %w", err)
}
_, err = c.db.ExecContext(ctx, `
INSERT INTO source_content (source_key, size_bytes)
VALUES (?, ?)
ON CONFLICT(source_key) DO UPDATE SET
size_bytes = excluded.size_bytes,
last_accessed_at = CURRENT_TIMESTAMP
`, string(key), size)
if err != nil {
deleteErr := c.srcContent.Delete(key)
if deleteErr != nil {
c.log.Warn("failed to remove source file whose row was not written",
"source_key", key, "error", deleteErr)
}
return "", fmt.Errorf("failed to insert source content: %w", err)
}
c.notifyWritePressure()
return key, nil
}
// StoreVariant stores a processed variant by its cache key and records
// it in the size accounting. On a disabled cache it is a no-op. The // it in the size accounting. On a disabled cache it is a no-op. The
// accounting insert is best-effort (the startup reconciliation pass // accounting insert is best-effort (the startup reconciliation pass
// adopts any variant file that misses its accounting row). // adopts any variant file that misses its accounting row).
func (c *Cache) StoreVariant( func (c *Cache) PutVariant(
ctx context.Context, cacheKey imagerequest.VariantKey, content io.Reader, ctx context.Context, cacheKey imagerequest.VariantKey, content io.Reader,
) error { ) error {
if c.config.MaxBytes == 0 { if c.config.MaxBytes == 0 {
return nil return nil
} }
size, err := c.variants.Store(cacheKey, content) data, err := io.ReadAll(content)
if err != nil {
return fmt.Errorf("failed to read content: %w", err)
}
err = c.variants.Write(string(cacheKey), data)
if err != nil { if err != nil {
return err return err
} }
size := int64(len(data))
_, err = c.db.ExecContext(ctx, ` _, err = c.db.ExecContext(ctx, `
INSERT INTO variant_content (cache_key, size_bytes) INSERT INTO variant_content (cache_key, size_bytes)
VALUES (?, ?) VALUES (?, ?)
@@ -244,19 +227,20 @@ func (c *Cache) StoreVariant(
return nil return nil
} }
// LookupSource checks if we have cached source content for a request. // Source opens the cached source of src and returns a reader, its size in
// Returns the source key if found, or an empty key if not; a row whose // bytes and its format, read from the file's first bytes. A source whose row
// file is missing is not found. A source found has its access time // or file is missing is ErrNotFound, as is any source of a disabled cache; a
// handed to the background writer without waiting (see // file whose first bytes are not those of a source format is another error.
// touchSourceContent); a disabled cache always reports no cached source. // A source it opens has its access time handed to the background writer
func (c *Cache) LookupSource( // without waiting (see touchSourceContent).
ctx context.Context, req *imagerequest.Request, func (c *Cache) Source(
) (imagerequest.SourceKey, error) { ctx context.Context, src imagerequest.Source,
) (io.ReadCloser, int64, imagerequest.Format, error) {
if c.config.MaxBytes == 0 { if c.config.MaxBytes == 0 {
return "", nil return nil, 0, "", ErrNotFound
} }
key := req.Source.Key() key := src.Key()
var rowExists bool var rowExists bool
@@ -264,25 +248,109 @@ func (c *Cache) LookupSource(
SELECT EXISTS(SELECT 1 FROM source_content WHERE source_key = ?) SELECT EXISTS(SELECT 1 FROM source_content WHERE source_key = ?)
`, string(key)).Scan(&rowExists) `, string(key)).Scan(&rowExists)
if err != nil { if err != nil {
return "", fmt.Errorf("failed to lookup source: %w", err) return nil, 0, "", fmt.Errorf("failed to lookup source: %w", err)
} }
if !rowExists { if !rowExists {
return "", nil return nil, 0, "", ErrNotFound
} }
// Verify the content file exists file, size, err := c.srcContent.Open(string(key))
if !c.srcContent.Exists(key) { if err != nil {
return "", nil return nil, 0, "", err
}
format, err := readFormat(file)
if err != nil {
_ = file.Close()
return nil, 0, "", err
} }
c.touchSourceContent(key) c.touchSourceContent(key)
return file, size, format, nil
}
// PutSource stores fetched source content: it writes the file named by
// the source's key, then the source's row. When the row cannot be written
// the file is removed and the store fails, as a file with no row is never
// served or evicted; if that removal fails too, the next reconciliation
// pass removes the file. On a disabled cache it is a no-op returning an
// empty key.
func (c *Cache) PutSource(
ctx context.Context,
src imagerequest.Source,
content io.Reader,
) (imagerequest.SourceKey, error) {
if c.config.MaxBytes == 0 {
return "", nil
}
data, err := io.ReadAll(content)
if err != nil {
return "", fmt.Errorf("failed to read source content: %w", err)
}
key := src.Key()
err = c.srcContent.Write(string(key), data)
if err != nil {
return "", fmt.Errorf("failed to store source content: %w", err)
}
size := int64(len(data))
_, err = c.db.ExecContext(ctx, `
INSERT INTO source_content (source_key, size_bytes)
VALUES (?, ?)
ON CONFLICT(source_key) DO UPDATE SET
size_bytes = excluded.size_bytes,
last_accessed_at = CURRENT_TIMESTAMP
`, string(key), size)
if err != nil {
deleteErr := c.srcContent.Delete(string(key))
if deleteErr != nil {
c.log.Warn("failed to remove source file whose row was not written",
"source_key", key, "error", deleteErr)
}
return "", fmt.Errorf("failed to insert source content: %w", err)
}
c.notifyWritePressure()
return key, nil return key, nil
} }
// StoreNegative stores a negative cache entry for a failed fetch. // Failure reports whether src is in the negative cache. It only reads: a
func (c *Cache) StoreNegative(ctx context.Context, req *imagerequest.Request) error { // row past its expiry is not a hit, and the eviction goroutine deletes it
// with the other expired rows (see CleanExpired).
func (c *Cache) Failure(
ctx context.Context, src imagerequest.Source,
) (bool, error) {
var expiresAt time.Time
err := c.db.QueryRowContext(ctx, `
SELECT expires_at FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, src.Host, src.Path, src.Query).Scan(&expiresAt)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("failed to check negative cache: %w", err)
}
expired := time.Now().After(expiresAt)
return !expired, nil
}
// PutFailure stores a negative cache entry for src, whose fetch failed.
func (c *Cache) PutFailure(ctx context.Context, src imagerequest.Source) error {
expiresAt := time.Now().UTC().Add(c.config.NegativeTTL) expiresAt := time.Now().UTC().Add(c.config.NegativeTTL)
_, err := c.db.ExecContext(ctx, ` _, err := c.db.ExecContext(ctx, `
@@ -291,7 +359,7 @@ func (c *Cache) StoreNegative(ctx context.Context, req *imagerequest.Request) er
VALUES (?, ?, ?, ?) VALUES (?, ?, ?, ?)
ON CONFLICT(source_host, source_path, source_query) DO UPDATE SET ON CONFLICT(source_host, source_path, source_query) DO UPDATE SET
expires_at = excluded.expires_at expires_at = excluded.expires_at
`, req.Source.Host, req.Source.Path, req.Source.Query, expiresAt) `, src.Host, src.Path, src.Query, expiresAt)
if err != nil { if err != nil {
return fmt.Errorf("failed to insert negative cache: %w", err) return fmt.Errorf("failed to insert negative cache: %w", err)
} }
@@ -299,18 +367,6 @@ func (c *Cache) StoreNegative(ctx context.Context, req *imagerequest.Request) er
return nil return nil
} }
// GetSourceContent returns a reader for cached source content by its key,
// and the content's size in bytes.
func (c *Cache) GetSourceContent(
key imagerequest.SourceKey,
) (io.ReadCloser, int64, error) {
if c.config.MaxBytes == 0 {
return nil, 0, ErrNotFound
}
return c.srcContent.LoadWithSize(key)
}
// CleanExpired removes the negative cache rows past their expiry, one // CleanExpired removes the negative cache rows past their expiry, one
// batch per statement, until none is left. Once ctx ends it starts no // batch per statement, until none is left. Once ctx ends it starts no
// further batch and returns ctx's error. The eviction goroutine calls it // further batch and returns ctx's error. The eviction goroutine calls it
@@ -334,34 +390,15 @@ func (c *Cache) CleanExpired(ctx context.Context) error {
} }
} }
// IncrementStats counts a cache hit or miss, and an upstream fetch that read // IncrementStats counts a cache hit or miss. It only adds to the cache's
// fetchBytes bytes, as IncrementUpstreamFetch does. Like the other Increment // Prometheus counters in memory (see counters.go), so it never waits for the
// methods, it only adds to the cache's Prometheus counters in memory (see // database or fails.
// counters.go), so it never waits for the database or fails. func (c *Cache) IncrementStats(hit bool) {
func (c *Cache) IncrementStats(hit bool, fetchBytes int64) {
if hit { if hit {
c.cacheHits.Inc() c.cacheHits.Inc()
} else { } else {
c.cacheMisses.Inc() c.cacheMisses.Inc()
} }
c.IncrementUpstreamFetch(fetchBytes)
}
// IncrementUpstreamFetch counts one upstream fetch that read fetchBytes bytes.
// A fetch that read no bytes is not counted.
func (c *Cache) IncrementUpstreamFetch(fetchBytes int64) {
if fetchBytes <= 0 {
return
}
c.upstreamFetches.Inc()
c.upstreamFetchBytes.Add(float64(fetchBytes))
}
// IncrementTransformCount counts one image transcoded by the image processor.
func (c *Cache) IncrementTransformCount() {
c.transforms.Inc()
} }
// touchVariant hands the access time of a variant served from the cache to // touchVariant hands the access time of a variant served from the cache to
@@ -388,32 +425,6 @@ func (c *Cache) touchSourceContent(key imagerequest.SourceKey) {
} }
} }
// checkNegativeCache checks if a request is in the negative cache. It only
// reads: a row past its expiry is not a hit, and the eviction goroutine
// deletes it with the other expired rows (see CleanExpired).
func (c *Cache) checkNegativeCache(
ctx context.Context, req *imagerequest.Request,
) (bool, error) {
var expiresAt time.Time
err := c.db.QueryRowContext(ctx, `
SELECT expires_at FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.Source.Host, req.Source.Path, req.Source.Query).Scan(&expiresAt)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("failed to check negative cache: %w", err)
}
expired := time.Now().After(expiresAt)
return !expired, nil
}
// cleanExpiredBatch deletes up to cleanExpiredBatchSize negative cache // cleanExpiredBatch deletes up to cleanExpiredBatchSize negative cache
// rows past their expiry in one statement, and returns how many it // rows past their expiry in one statement, and returns how many it
// deleted. DELETE ... LIMIT needs a SQLite build option, so the subquery // deleted. DELETE ... LIMIT needs a SQLite build option, so the subquery
@@ -437,3 +448,17 @@ func (c *Cache) cleanExpiredBatch(ctx context.Context) (int64, error) {
return deleted, nil return deleted, nil
} }
// readFormat reads the first bytes of a source or variant file, without
// moving its offset, and returns the format they show. Sources and variants
// are in the same six formats.
func readFormat(content io.ReaderAt) (imagerequest.Format, error) {
firstBytes := make([]byte, magic.MinMagicBytes)
_, err := content.ReadAt(firstBytes, 0)
if err != nil {
return "", fmt.Errorf("failed to read the first bytes: %w", err)
}
return magic.DetectFormat(firstBytes)
}
internal/imgcache/cache_hit_internal_test.go
+20 -109
View File
@@ -6,7 +6,6 @@ import (
"fmt" "fmt"
"image/color" "image/color"
"io" "io"
"os"
"testing" "testing"
"time" "time"
@@ -15,10 +14,10 @@ import (
"sneak.berlin/go/pixa/internal/magic" "sneak.berlin/go/pixa/internal/magic"
) )
// TestGetVariant_ContentTypeFromFirstBytes stores an image in each output // TestVariant_ContentTypeFromFirstBytes stores an image in each output
// format as a variant and checks that GetVariant serves it whole with the // format as a variant and checks that Variant serves it whole with the
// content type of its format, read from the file. // content type of its format, read from the file.
func TestGetVariant_ContentTypeFromFirstBytes(t *testing.T) { func TestVariant_ContentTypeFromFirstBytes(t *testing.T) {
t.Parallel() t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30) cache, _ := newEvictionTestCache(t, 1<<30)
@@ -33,7 +32,7 @@ func TestGetVariant_ContentTypeFromFirstBytes(t *testing.T) {
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL, imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
} { } {
result, err := processor.Process(t.Context(), bytes.NewReader(source), result, err := processor.Process(t.Context(), bytes.NewReader(source),
&imagerequest.Transform{ imagerequest.FormatJPEG, &imagerequest.Transform{
Size: imagerequest.Size{Width: 20, Height: 15}, Size: imagerequest.Size{Width: 20, Height: 15},
Format: format, Format: format,
Quality: 85, Quality: 85,
@@ -50,14 +49,14 @@ func TestGetVariant_ContentTypeFromFirstBytes(t *testing.T) {
key := imagerequest.VariantKey("aabbccdd10" + string(format)) key := imagerequest.VariantKey("aabbccdd10" + string(format))
err = cache.StoreVariant(t.Context(), key, bytes.NewReader(image)) err = cache.PutVariant(t.Context(), key, bytes.NewReader(image))
if err != nil { if err != nil {
t.Fatalf("StoreVariant(%s) error = %v", format, err) t.Fatalf("PutVariant(%s) error = %v", format, err)
} }
reader, size, contentType, err := cache.GetVariant(key) reader, size, contentType, err := cache.Variant(key)
if err != nil { if err != nil {
t.Fatalf("GetVariant(%s) error = %v", format, err) t.Fatalf("Variant(%s) error = %v", format, err)
} }
served, err := io.ReadAll(reader) served, err := io.ReadAll(reader)
@@ -69,7 +68,7 @@ func TestGetVariant_ContentTypeFromFirstBytes(t *testing.T) {
t.Logf("%s: %d bytes served as %s", format, size, contentType) t.Logf("%s: %d bytes served as %s", format, size, contentType)
if want := imageprocessor.FormatToMIME(format); contentType != want { if want := format.MIMEType(); contentType != want {
t.Errorf("%s variant content type = %q, want %q", format, contentType, want) t.Errorf("%s variant content type = %q, want %q", format, contentType, want)
} }
@@ -80,11 +79,11 @@ func TestGetVariant_ContentTypeFromFirstBytes(t *testing.T) {
} }
} }
// TestGetVariant_RefusesFileOfNoOutputFormat checks that GetVariant refuses // TestVariant_RefusesFileOfNoOutputFormat checks that Variant refuses
// a variant file whose first bytes are not those of an output format, with // a variant file whose first bytes are not those of an output format, with
// an error other than ErrNotFound, and that a variant with no file, or any // an error other than ErrNotFound, and that a variant with no file, or any
// variant of a disabled cache, is ErrNotFound. // variant of a disabled cache, is ErrNotFound.
func TestGetVariant_RefusesFileOfNoOutputFormat(t *testing.T) { func TestVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
t.Parallel() t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30) cache, _ := newEvictionTestCache(t, 1<<30)
@@ -97,7 +96,7 @@ func TestGetVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
}{ }{
{"text", []byte("this is text, not an image"), magic.ErrUnknownFormat}, {"text", []byte("this is text, not an image"), magic.ErrUnknownFormat},
{"SVG", []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`), {"SVG", []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`),
errNotAnOutputFormat}, magic.ErrUnknownFormat},
{"JPEG cut to 10 bytes", jpegImage[:10], io.EOF}, {"JPEG cut to 10 bytes", jpegImage[:10], io.EOF},
{"empty", nil, io.EOF}, {"empty", nil, io.EOF},
} }
@@ -105,123 +104,35 @@ func TestGetVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
for i, tt := range tests { for i, tt := range tests {
key := imagerequest.VariantKey(fmt.Sprintf("aabbccdd20%02d", i)) key := imagerequest.VariantKey(fmt.Sprintf("aabbccdd20%02d", i))
_, err := cache.variants.Store(key, bytes.NewReader(tt.content)) err := cache.variants.Write(string(key), tt.content)
if err != nil { if err != nil {
t.Fatalf("%s: storing the variant file: %v", tt.name, err) t.Fatalf("%s: storing the variant file: %v", tt.name, err)
} }
reader, _, contentType, err := cache.GetVariant(key) reader, _, contentType, err := cache.Variant(key)
t.Logf("%s: GetVariant() error = %v", tt.name, err) t.Logf("%s: Variant() error = %v", tt.name, err)
if reader != nil || contentType != "" || !errors.Is(err, tt.wantErr) { if reader != nil || contentType != "" || !errors.Is(err, tt.wantErr) {
t.Errorf("%s: GetVariant() = reader %v, content type %q, error %v; "+ t.Errorf("%s: Variant() = reader %v, content type %q, error %v; "+
"want no reader and %v", tt.name, reader, contentType, err, tt.wantErr) "want no reader and %v", tt.name, reader, contentType, err, tt.wantErr)
} }
} }
reader, _, _, err := cache.GetVariant(testVariantKeyFour) reader, _, _, err := cache.Variant(testVariantKeyFour)
if reader != nil || !errors.Is(err, ErrNotFound) { if reader != nil || !errors.Is(err, ErrNotFound) {
t.Errorf("GetVariant() of a variant with no file = reader %v, error %v; "+ t.Errorf("Variant() of a variant with no file = reader %v, error %v; "+
"want no reader and %v", reader, err, ErrNotFound) "want no reader and %v", reader, err, ErrNotFound)
} }
disabled, _ := newEvictionTestCache(t, 0) disabled, _ := newEvictionTestCache(t, 0)
reader, _, _, err = disabled.GetVariant(testVariantKeyOne) reader, _, _, err = disabled.Variant(testVariantKeyOne)
if reader != nil || !errors.Is(err, ErrNotFound) { if reader != nil || !errors.Is(err, ErrNotFound) {
t.Errorf("GetVariant() on a disabled cache = reader %v, error %v; "+ t.Errorf("Variant() on a disabled cache = reader %v, error %v; "+
"want no reader and %v", reader, err, ErrNotFound) "want no reader and %v", reader, err, ErrNotFound)
} }
} }
// TestService_Get_MakesAgainVariantOfNoOutputFormat replaces a cached
// variant's file with text, and checks that the next request is a miss
// answered with the variant made again, and the one after a hit.
func TestService_Get_MakesAgainVariantOfNoOutputFormat(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := svc.cache.variants.keyToPath(imagerequest.CacheKey(req))
err := os.WriteFile(path, []byte("this is text, not an image"), StorageFilePerm)
if err != nil {
t.Fatalf("replacing the variant file: %v", err)
}
got, status := getImage(t, svc, req)
t.Logf("after the replacement: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the replacement: Get() = %d bytes, %s; want the %d bytes "+
"made again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestService_Get_HitWaitsForNoDatabaseWrite holds the database's one
// connection while a cached variant is requested. The hit must be answered
// while the connection is still held, and its access time must reach the
// database once it is free.
func TestService_Get_HitWaitsForNoDatabaseWrite(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
key := imagerequest.CacheKey(req)
getImage(t, svc, req)
// Each connection to an in-memory SQLite database opens a new, empty one,
// so the goroutine that writes the access times must share the
// connection that holds the schema.
svc.cache.db.SetMaxOpenConns(1)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
anHourAgo := time.Now().Add(-time.Hour)
setVariantLastAccessed(t, svc.cache, key, anHourAgo)
releaseDatabase := holdDatabase(t, svc.cache)
defer releaseDatabase()
results := startGet(t.Context(), svc, req)
select {
case got := <-results:
if got.err != nil {
t.Fatalf("Get() of the cached variant error = %v", got.err)
}
case <-time.After(pendingAccessTimesWait):
t.Fatal("the cache hit waited for the database")
}
releaseDatabase()
deadline := time.Now().Add(pendingAccessTimesWait)
for countRows(t, svc.cache, `
SELECT COUNT(*) FROM variant_content
WHERE cache_key = ? AND last_accessed_at > ?
`, string(key), anHourAgo.UTC().Format(sqliteTimestampFormat)) == 0 {
if time.Now().After(deadline) {
t.Fatal("the hit's access time was not written")
}
time.Sleep(10 * time.Millisecond)
}
}
// TestTouchVariantAndTouchSourceContent_DropAccessTimesPastTheLimit hands one // TestTouchVariantAndTouchSourceContent_DropAccessTimesPastTheLimit hands one
// more access time of a variant, and one more of a source, than may wait to // more access time of a variant, and one more of a source, than may wait to
// be written. touchVariant and touchSourceContent must return each time // be written. touchVariant and touchSourceContent must return each time
internal/imgcache/cache_internal_test.go
+2 -2
View File
@@ -50,7 +50,7 @@ func TestCache_CleanExpired(t *testing.T) {
tmpDir := t.TempDir() tmpDir := t.TempDir()
db := setupTestDB(t) db := setupTestDB(t)
cache, _ := NewCache(db, CacheConfig{ cache, _ := NewCache(db, Config{
StateDir: tmpDir, StateDir: tmpDir,
NegativeTTL: 1 * time.Millisecond, NegativeTTL: 1 * time.Millisecond,
MaxBytes: 1 << 30, MaxBytes: 1 << 30,
@@ -103,7 +103,7 @@ func TestCache_StorageDirectoriesCreated(t *testing.T) {
tmpDir := t.TempDir() tmpDir := t.TempDir()
db := setupTestDB(t) db := setupTestDB(t)
_, err := NewCache(db, CacheConfig{ _, err := NewCache(db, Config{
StateDir: tmpDir, StateDir: tmpDir,
MaxBytes: 1 << 30, MaxBytes: 1 << 30,
}, prometheus.NewRegistry()) }, prometheus.NewRegistry())
internal/imgcache/counters.go
+1 -15
View File
@@ -19,22 +19,8 @@ func (c *Cache) registerCounters(metricsRegistry prometheus.Registerer) error {
Name: "pixa_cache_misses_total", Name: "pixa_cache_misses_total",
Help: "Image requests not served from the cache.", Help: "Image requests not served from the cache.",
}) })
c.upstreamFetches = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetches_total",
Help: "Fetches of a source image from upstream that read any bytes.",
})
c.upstreamFetchBytes = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetch_bytes_total",
Help: "Bytes read from upstream by fetches of source images.",
})
c.transforms = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_transforms_total",
Help: "Images the image processor transformed.",
})
for _, counter := range []prometheus.Counter{ for _, counter := range []prometheus.Counter{c.cacheHits, c.cacheMisses} {
c.cacheHits, c.cacheMisses, c.upstreamFetches, c.upstreamFetchBytes, c.transforms,
} {
err := metricsRegistry.Register(counter) err := metricsRegistry.Register(counter)
if err != nil { if err != nil {
return fmt.Errorf("failed to register the cache's counters: %w", err) return fmt.Errorf("failed to register the cache's counters: %w", err)
internal/imgcache/counters_internal_test.go
+3 -26
View File
@@ -8,26 +8,6 @@ import (
"github.com/prometheus/client_golang/prometheus/testutil" "github.com/prometheus/client_golang/prometheus/testutil"
) )
// counterValues holds the values of a cache's counters.
type counterValues struct {
hitCount int64
missCount int64
upstreamFetchCount int64
upstreamFetchBytes int64
transformCount int64
}
// readCounters reads cache's counters.
func readCounters(cache *Cache) counterValues {
return counterValues{
hitCount: int64(testutil.ToFloat64(cache.cacheHits)),
missCount: int64(testutil.ToFloat64(cache.cacheMisses)),
upstreamFetchCount: int64(testutil.ToFloat64(cache.upstreamFetches)),
upstreamFetchBytes: int64(testutil.ToFloat64(cache.upstreamFetchBytes)),
transformCount: int64(testutil.ToFloat64(cache.transforms)),
}
}
// TestNewCache_RegistersItsCountersOnItsRegistry checks that NewCache // TestNewCache_RegistersItsCountersOnItsRegistry checks that NewCache
// registers the cache's counters on the registry it is given, under the // registers the cache's counters on the registry it is given, under the
// names README.md gives them. // names README.md gives them.
@@ -37,7 +17,7 @@ func TestNewCache_RegistersItsCountersOnItsRegistry(t *testing.T) {
registry := prometheus.NewRegistry() registry := prometheus.NewRegistry()
_, err := NewCache(evictionTestDB(t), _, err := NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry) Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
if err != nil { if err != nil {
t.Fatalf("NewCache() error = %v", err) t.Fatalf("NewCache() error = %v", err)
} }
@@ -45,9 +25,6 @@ func TestNewCache_RegistersItsCountersOnItsRegistry(t *testing.T) {
names := []string{ names := []string{
"pixa_cache_hits_total", "pixa_cache_hits_total",
"pixa_cache_misses_total", "pixa_cache_misses_total",
"pixa_upstream_fetches_total",
"pixa_upstream_fetch_bytes_total",
"pixa_transforms_total",
} }
count, err := testutil.GatherAndCount(registry, names...) count, err := testutil.GatherAndCount(registry, names...)
@@ -70,13 +47,13 @@ func TestNewCache_FailsWhenItsCountersCannotBeRegistered(t *testing.T) {
registry := prometheus.NewRegistry() registry := prometheus.NewRegistry()
_, err := NewCache(evictionTestDB(t), _, err := NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry) Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
if err != nil { if err != nil {
t.Fatalf("NewCache() of the first cache error = %v", err) t.Fatalf("NewCache() of the first cache error = %v", err)
} }
_, err = NewCache(evictionTestDB(t), _, err = NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry) Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
t.Logf("NewCache() of the second cache error = %v", err) t.Logf("NewCache() of the second cache error = %v", err)
var alreadyRegistered prometheus.AlreadyRegisteredError var alreadyRegistered prometheus.AlreadyRegisteredError
internal/imgcache/eviction.go
+58 -41
View File
@@ -48,7 +48,7 @@ const staleTempFileAge = time.Hour
const sqliteTimestampLayout = "2006-01-02 15:04:05" const sqliteTimestampLayout = "2006-01-02 15:04:05"
// tempFilePrefix is the prefix os.CreateTemp uses for in-flight cache // tempFilePrefix is the prefix os.CreateTemp uses for in-flight cache
// writes (".tmp-*" patterns in the storage layer). // writes (the ".tmp-*" pattern in writeFileSynced).
const tempFilePrefix = ".tmp-" const tempFilePrefix = ".tmp-"
// UsageBytes returns the total number of bytes of cache content // UsageBytes returns the total number of bytes of cache content
@@ -306,7 +306,7 @@ func (c *Cache) evictVariant(
return fmt.Errorf("failed to delete variant accounting row: %w", err) return fmt.Errorf("failed to delete variant accounting row: %w", err)
} }
err = c.variants.Delete(cacheKey) err = c.variants.Delete(string(cacheKey))
if err != nil { if err != nil {
return err return err
} }
@@ -317,7 +317,7 @@ func (c *Cache) evictVariant(
// evictSourceBlob removes one source: its row first, then its file, as // evictSourceBlob removes one source: its row first, then its file, as
// evictVariant does for a variant. It holds no lock: a store of the same // evictVariant does for a variant. It holds no lock: a store of the same
// source in between can leave the source's row with no file, which // source in between can leave the source's row with no file, which
// LookupSource treats as a miss. // Source treats as a miss.
func (c *Cache) evictSourceBlob(ctx context.Context, key imagerequest.SourceKey) error { func (c *Cache) evictSourceBlob(ctx context.Context, key imagerequest.SourceKey) error {
_, err := c.db.ExecContext(ctx, _, err := c.db.ExecContext(ctx,
`DELETE FROM source_content WHERE source_key = ?`, string(key)) `DELETE FROM source_content WHERE source_key = ?`, string(key))
@@ -325,7 +325,7 @@ func (c *Cache) evictSourceBlob(ctx context.Context, key imagerequest.SourceKey)
return fmt.Errorf("failed to delete source content row: %w", err) return fmt.Errorf("failed to delete source content row: %w", err)
} }
return c.srcContent.Delete(key) return c.srcContent.Delete(string(key))
} }
// notifyWritePressure wakes the background evictor after a store, so // notifyWritePressure wakes the background evictor after a store, so
@@ -479,7 +479,7 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
// then on the first periodic eviction tick that comes at least ten times // then on the first periodic eviction tick that comes at least ten times
// as long after the last pass ended as that pass took, off the request // as long after the last pass ended as that pass took, off the request
// hot path: it adopts variant files that predate the accounting table // hot path: it adopts variant files that predate the accounting table
// (or whose accounting insert failed, e.g. StoreVariant's best-effort // (or whose accounting insert failed, e.g. PutVariant's best-effort
// insert under transient DB contention), drops variant accounting rows // insert under transient DB contention), drops variant accounting rows
// whose files are missing, removes source blob files the database does // whose files are missing, removes source blob files the database does
// not know, removes source and variant files that are empty or not the // not know, removes source and variant files that are empty or not the
@@ -515,12 +515,15 @@ func (c *Cache) reconcileAccounting(ctx context.Context) error {
} }
// reconcileFiles walks baseDir, one of the cache's storage directories, // reconcileFiles walks baseDir, one of the cache's storage directories,
// sweeping stale temp files and passing each other file, with its name, to // sweeping stale temp files and passing each other file, with its info and
// reconcileFile. Once ctx is cancelled it stops at the next file and // name, to reconcileFile. It skips a directory it cannot list and a file
// returns ctx's error. // whose info it cannot read, so one such entry does not stop the pass. It
// logs a warning naming the path and the error. A file renamed or removed
// after its directory was listed is skipped with no warning. Once ctx is
// cancelled it stops at the next file and returns ctx's error.
func (c *Cache) reconcileFiles( func (c *Cache) reconcileFiles(
ctx context.Context, baseDir string, ctx context.Context, baseDir string,
reconcileFile func(path string, entry fs.DirEntry, name string) error, reconcileFile func(path string, info fs.FileInfo, name string) error,
) error { ) error {
return filepath.WalkDir( return filepath.WalkDir(
baseDir, baseDir,
@@ -529,19 +532,38 @@ func (c *Cache) reconcileFiles(
return ctx.Err() return ctx.Err()
} }
if err != nil || entry.IsDir() { if err != nil {
return err c.log.Warn("skipped unreadable entry in cache directory",
"path", path, "error", err)
return nil
}
if entry.IsDir() {
return nil
}
info, err := entry.Info()
if errors.Is(err, fs.ErrNotExist) {
return nil
}
if err != nil {
c.log.Warn("skipped unreadable entry in cache directory",
"path", path, "error", err)
return nil
} }
name := entry.Name() name := entry.Name()
if strings.HasPrefix(name, tempFilePrefix) { if strings.HasPrefix(name, tempFilePrefix) {
c.sweepStaleTempFile(path, entry) c.sweepStaleTempFile(path, info)
return nil return nil
} }
return reconcileFile(path, entry, name) return reconcileFile(path, info, name)
}, },
) )
} }
@@ -551,8 +573,8 @@ func (c *Cache) reconcileFiles(
// and sweeping stale temp files. // and sweeping stale temp files.
func (c *Cache) reconcileVariantFiles(ctx context.Context) error { func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
return c.reconcileFiles(ctx, c.variants.baseDir, return c.reconcileFiles(ctx, c.variants.baseDir,
func(path string, entry fs.DirEntry, name string) error { func(path string, info fs.FileInfo, name string) error {
return c.reconcileVariantFile(ctx, path, entry, imagerequest.VariantKey(name)) return c.reconcileVariantFile(ctx, path, info, imagerequest.VariantKey(name))
}) })
} }
@@ -562,16 +584,11 @@ func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
// crash or a restored snapshot can leave it, is removed with its row and // crash or a restored snapshot can leave it, is removed with its row and
// a warning naming it, so the variant is converted again. // a warning naming it, so the variant is converted again.
func (c *Cache) reconcileVariantFile( func (c *Cache) reconcileVariantFile(
ctx context.Context, path string, entry fs.DirEntry, cacheKey imagerequest.VariantKey, ctx context.Context, path string, info fs.FileInfo, cacheKey imagerequest.VariantKey,
) error { ) error {
info, err := entry.Info()
if err != nil {
return fmt.Errorf("failed to stat variant file: %w", err)
}
var recordedSize int64 var recordedSize int64
err = c.db.QueryRowContext(ctx, err := c.db.QueryRowContext(ctx,
`SELECT size_bytes FROM variant_content WHERE cache_key = ?`, string(cacheKey), `SELECT size_bytes FROM variant_content WHERE cache_key = ?`, string(cacheKey),
).Scan(&recordedSize) ).Scan(&recordedSize)
@@ -613,7 +630,9 @@ func (c *Cache) adoptVariantFile(
} }
// reconcileVariantRows drops accounting rows whose variant files are // reconcileVariantRows drops accounting rows whose variant files are
// missing, so the database never references deleted content. // missing, so the database never references deleted content. A file whose
// size cannot be read for another reason may still be on disk, so its row
// stays, with a warning naming its key and the error.
func (c *Cache) reconcileVariantRows(ctx context.Context) error { func (c *Cache) reconcileVariantRows(ctx context.Context) error {
var after imagerequest.VariantKey var after imagerequest.VariantKey
@@ -636,11 +655,19 @@ func (c *Cache) reconcileVariantRows(ctx context.Context) error {
return ctx.Err() return ctx.Err()
} }
if c.variants.Exists(key) { _, err := c.variants.Size(string(key))
if err == nil {
continue continue
} }
_, err := c.db.ExecContext(ctx, if !errors.Is(err, ErrNotFound) {
c.log.Warn("kept accounting row for unreadable variant file",
"cache_key", key, "error", err)
continue
}
_, err = c.db.ExecContext(ctx,
`DELETE FROM variant_content WHERE cache_key = ?`, string(key)) `DELETE FROM variant_content WHERE cache_key = ?`, string(key))
if err != nil { if err != nil {
return fmt.Errorf("failed to drop stale variant accounting row: %w", err) return fmt.Errorf("failed to drop stale variant accounting row: %w", err)
@@ -704,8 +731,8 @@ func queryStringColumn[T ~string](
// and sweeping stale temp files. // and sweeping stale temp files.
func (c *Cache) reconcileSourceFiles(ctx context.Context) error { func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
return c.reconcileFiles(ctx, c.srcContent.baseDir, return c.reconcileFiles(ctx, c.srcContent.baseDir,
func(path string, entry fs.DirEntry, name string) error { func(path string, info fs.FileInfo, name string) error {
return c.reconcileSourceFile(ctx, path, entry, imagerequest.SourceKey(name)) return c.reconcileSourceFile(ctx, path, info, imagerequest.SourceKey(name))
}) })
} }
@@ -716,16 +743,11 @@ func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
// is removed with its row and a warning naming it, so the source is // is removed with its row and a warning naming it, so the source is
// fetched again. // fetched again.
func (c *Cache) reconcileSourceFile( func (c *Cache) reconcileSourceFile(
ctx context.Context, path string, entry fs.DirEntry, key imagerequest.SourceKey, ctx context.Context, path string, info fs.FileInfo, key imagerequest.SourceKey,
) error { ) error {
info, err := entry.Info()
if err != nil {
return fmt.Errorf("failed to stat source content file: %w", err)
}
var recordedSize int64 var recordedSize int64
err = c.db.QueryRowContext(ctx, err := c.db.QueryRowContext(ctx,
`SELECT size_bytes FROM source_content WHERE source_key = ?`, `SELECT size_bytes FROM source_content WHERE source_key = ?`,
string(key), string(key),
).Scan(&recordedSize) ).Scan(&recordedSize)
@@ -890,17 +912,12 @@ func (c *Cache) correctUsageTotal(
// sweepStaleTempFile removes a temp file left behind by a crashed // sweepStaleTempFile removes a temp file left behind by a crashed
// write once it is old enough that no in-flight store can own it. // write once it is old enough that no in-flight store can own it.
func (c *Cache) sweepStaleTempFile(path string, entry fs.DirEntry) { func (c *Cache) sweepStaleTempFile(path string, info fs.FileInfo) {
info, err := entry.Info()
if err != nil {
return
}
if time.Since(info.ModTime()) < staleTempFileAge { if time.Since(info.ModTime()) < staleTempFileAge {
return return
} }
err = os.Remove(path) err := os.Remove(path)
if err != nil && !os.IsNotExist(err) { if err != nil && !os.IsNotExist(err) {
c.log.Warn("failed to remove stale temp file", "path", path, "error", err) c.log.Warn("failed to remove stale temp file", "path", path, "error", err)
internal/imgcache/eviction_internal_test.go
+327 -35
View File
@@ -9,8 +9,10 @@ import (
"log/slog" "log/slog"
"os" "os"
"path/filepath" "path/filepath"
"slices"
"strings" "strings"
"sync" "sync"
"syscall"
"testing" "testing"
"time" "time"
@@ -67,7 +69,7 @@ func newEvictionTestCache(t *testing.T, maxBytes int64) (*Cache, string) {
db := evictionTestDB(t) db := evictionTestDB(t)
// maxBytes zero turns the disk cache off, as cache_max_bytes: 0 does. // maxBytes zero turns the disk cache off, as cache_max_bytes: 0 does.
cache, err := NewCache(db, CacheConfig{ cache, err := NewCache(db, Config{
StateDir: tmpDir, StateDir: tmpDir,
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
MaxBytes: maxBytes, MaxBytes: maxBytes,
@@ -98,11 +100,11 @@ func storeEvictionTestSource(
}, },
} }
key, err := cache.StoreSource( key, err := cache.PutSource(
context.Background(), req, bytes.NewReader(content), context.Background(), req.Source, bytes.NewReader(content),
) )
if err != nil { if err != nil {
t.Fatalf("StoreSource(%s%s) failed: %v", host, path, err) t.Fatalf("PutSource(%s%s) failed: %v", host, path, err)
} }
return key return key
@@ -115,9 +117,9 @@ func storeEvictionTestVariant(
) { ) {
t.Helper() t.Helper()
err := cache.StoreVariant(t.Context(), key, bytes.NewReader(content)) err := cache.PutVariant(t.Context(), key, bytes.NewReader(content))
if err != nil { if err != nil {
t.Fatalf("StoreVariant(%s) failed: %v", key, err) t.Fatalf("PutVariant(%s) failed: %v", key, err)
} }
} }
@@ -209,7 +211,8 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
t.Fatalf("failed to scan source_key: %v", err) t.Fatalf("failed to scan source_key: %v", err)
} }
if !cache.srcContent.Exists(imagerequest.SourceKey(key)) { _, err = os.Stat(cache.srcContent.keyToPath(key))
if err != nil {
t.Errorf("source_content references content %s but the file is missing", t.Errorf("source_content references content %s but the file is missing",
key) key)
} }
@@ -236,7 +239,8 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
t.Fatalf("failed to scan cache_key: %v", err) t.Fatalf("failed to scan cache_key: %v", err)
} }
if !cache.variants.Exists(imagerequest.VariantKey(key)) { _, err = os.Stat(cache.variants.keyToPath(key))
if err != nil {
t.Errorf("variant_content references key %s but the file is missing", key) t.Errorf("variant_content references key %s but the file is missing", key)
} }
} }
@@ -334,7 +338,8 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
t.Errorf("usage after eviction = %d, want <= %d", usage, limit) t.Errorf("usage after eviction = %d, want <= %d", usage, limit)
} }
if cache.variants.Exists(keys[0]) { _, err = os.Stat(cache.variants.keyToPath(string(keys[0])))
if err == nil {
t.Errorf("least recently used variant %s must be evicted", keys[0]) t.Errorf("least recently used variant %s must be evicted", keys[0])
} }
@@ -345,7 +350,8 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
} }
for _, key := range keys[1:] { for _, key := range keys[1:] {
if !cache.variants.Exists(key) { _, err = os.Stat(cache.variants.keyToPath(string(key)))
if err != nil {
t.Errorf("more recently used variant %s must survive eviction", key) t.Errorf("more recently used variant %s must survive eviction", key)
} }
} }
@@ -373,7 +379,8 @@ func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
} }
for _, key := range keys { for _, key := range keys {
if !cache.srcContent.Exists(key) { _, err = os.Stat(cache.srcContent.keyToPath(string(key)))
if err != nil {
t.Errorf("source %s must not be evicted while usage is under the limit", key) t.Errorf("source %s must not be evicted while usage is under the limit", key)
} }
@@ -384,7 +391,8 @@ func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
} }
} }
if !cache.variants.Exists(testVariantKeyOne) { _, err = os.Stat(cache.variants.keyToPath(string(testVariantKeyOne)))
if err != nil {
t.Error("variant must not be evicted while usage is under the limit") t.Error("variant must not be evicted while usage is under the limit")
} }
@@ -423,20 +431,20 @@ func assertDisabledCacheWritesAreNoOps(
ctx := t.Context() ctx := t.Context()
err := cache.StoreVariant( err := cache.PutVariant(
ctx, imagerequest.CacheKey(req), bytes.NewReader([]byte("data")), ctx, imagerequest.CacheKey(req), bytes.NewReader([]byte("data")),
) )
if err != nil { if err != nil {
t.Fatalf("StoreVariant on disabled cache must be a no-op, got error: %v", err) t.Fatalf("PutVariant on disabled cache must be a no-op, got error: %v", err)
} }
key, err := cache.StoreSource(ctx, req, bytes.NewReader([]byte("data"))) key, err := cache.PutSource(ctx, req.Source, bytes.NewReader([]byte("data")))
if err != nil { if err != nil {
t.Fatalf("StoreSource on disabled cache must be a no-op, got error: %v", err) t.Fatalf("PutSource on disabled cache must be a no-op, got error: %v", err)
} }
if key != "" { if key != "" {
t.Errorf("StoreSource on disabled cache returned key %q, want empty", key) t.Errorf("PutSource on disabled cache returned key %q, want empty", key)
} }
} }
@@ -449,13 +457,13 @@ func assertDisabledCacheReadsAlwaysMiss(
ctx := t.Context() ctx := t.Context()
key, err := cache.LookupSource(ctx, req) reader, _, _, err := cache.Source(ctx, req.Source)
if err != nil { if !errors.Is(err, ErrNotFound) {
t.Fatalf("LookupSource on disabled cache failed: %v", err) t.Fatalf("Source on disabled cache error = %v, want %v", err, ErrNotFound)
} }
if key != "" { if reader != nil {
t.Errorf("LookupSource on disabled cache = %q, want empty", key) t.Errorf("Source on disabled cache = reader %v, want none", reader)
} }
} }
@@ -544,7 +552,7 @@ func TestEvictionRunsUnderWritePressure(t *testing.T) {
} }
// TestEvictionRunsOnPeriodicSchedule writes three variant files straight // TestEvictionRunsOnPeriodicSchedule writes three variant files straight
// to disk, bypassing StoreVariant, so they have no accounting rows and no // to disk, bypassing PutVariant, so they have no accounting rows and no
// write-pressure notification fires. Only a periodic reconciliation pass // write-pressure notification fires. Only a periodic reconciliation pass
// can then adopt them, and only the eviction pass that follows it can // can then adopt them, and only the eviction pass that follows it can
// evict them. // evict them.
@@ -586,7 +594,7 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
for i, key := range keys { for i, key := range keys {
content := bytes.Repeat([]byte{fills[i]}, 1000) content := bytes.Repeat([]byte{fills[i]}, 1000)
_, err = cache.variants.Store(key, bytes.NewReader(content)) err = cache.variants.Write(string(key), content)
if err != nil { if err != nil {
t.Fatalf("failed to store variant file: %v", err) t.Fatalf("failed to store variant file: %v", err)
} }
@@ -605,7 +613,8 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
stored = 0 stored = 0
for _, key := range keys { for _, key := range keys {
if cache.variants.Exists(key) { _, err = os.Stat(cache.variants.keyToPath(string(key)))
if err == nil {
stored++ stored++
} }
} }
@@ -637,9 +646,7 @@ func TestStartEvictionReconcilesAccountingWithDisk(t *testing.T) {
// feature existed) must be adopted into the accounting. // feature existed) must be adopted into the accounting.
untracked := bytes.Repeat([]byte{0x31}, 1000) untracked := bytes.Repeat([]byte{0x31}, 1000)
_, err := cache.variants.Store( err := cache.variants.Write(string(testVariantKeyOne), untracked)
testVariantKeyOne, bytes.NewReader(untracked),
)
if err != nil { if err != nil {
t.Fatalf("failed to store untracked variant file: %v", err) t.Fatalf("failed to store untracked variant file: %v", err)
} }
@@ -694,10 +701,295 @@ func TestStartEvictionReconcilesAccountingWithDisk(t *testing.T) {
} }
} }
// logLineNaming returns the first line of logs, written by slog's JSON
// handler, that is at level and names the variant key, or "" if none is.
func logLineNaming(logs, level string, key imagerequest.VariantKey) string {
for line := range strings.Lines(logs) {
if strings.Contains(line, `"level":"`+level+`"`) &&
strings.Contains(line, `"cache_key":"`+string(key)+`"`) {
return line
}
}
return ""
}
// TestReconcileVariantRowsDropsOnlyRowsOfMissingFiles gives
// reconcileVariantRows two tracked variants. The first one's file is a
// symbolic link to itself, so its size cannot be read. Its row must stay,
// with a warning naming its key and the error. The second one has no file.
// Its row must go, with the info line for a missing file.
func TestReconcileVariantRowsDropsOnlyRowsOfMissingFiles(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
storeEvictionTestVariant(t, cache, testVariantKeyOne, []byte("variant one"))
storeEvictionTestVariant(t, cache, testVariantKeyTwo, []byte("variant two"))
unreadablePath := cache.variants.keyToPath(string(testVariantKeyOne))
err := os.Remove(unreadablePath)
if err != nil {
t.Fatalf("failed to remove the first variant's file: %v", err)
}
err = os.Symlink(unreadablePath, unreadablePath)
if err != nil {
t.Fatalf("failed to link the first variant's path to itself: %v", err)
}
_, statErr := os.Stat(unreadablePath)
if statErr == nil || errors.Is(statErr, fs.ErrNotExist) {
t.Fatalf("os.Stat() of the first variant: error = %v, "+
"want an error other than a missing file", statErr)
}
err = os.Remove(cache.variants.keyToPath(string(testVariantKeyTwo)))
if err != nil {
t.Fatalf("failed to remove the second variant's file: %v", err)
}
var logBuf bytes.Buffer
cache.log = slog.New(slog.NewJSONHandler(&logBuf, nil))
err = cache.reconcileVariantRows(t.Context())
t.Logf("reconcileVariantRows() error = %v, log: %s", err, logBuf.String())
if err != nil {
t.Fatalf("reconcileVariantRows() error = %v", err)
}
rowQuery := `SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`
if n := countRows(t, cache, rowQuery, string(testVariantKeyOne)); n != 1 {
t.Errorf("the variant whose size cannot be read has %d rows, want 1", n)
}
if n := countRows(t, cache, rowQuery, string(testVariantKeyTwo)); n != 0 {
t.Errorf("the variant with no file has %d rows, want 0", n)
}
logs := logBuf.String()
warning := logLineNaming(logs, "WARN", testVariantKeyOne)
if !strings.Contains(warning, statErr.Error()) {
t.Errorf("no warning names %s and the error %q", testVariantKeyOne, statErr)
}
if line := logLineNaming(logs, "INFO", testVariantKeyOne); line != "" {
t.Errorf("the variant whose size cannot be read was logged at info: %s", line)
}
dropped := logLineNaming(logs, "INFO", testVariantKeyTwo)
if !strings.Contains(dropped, "dropped accounting row for missing variant file") {
t.Errorf("no info line says the file of %s is missing", testVariantKeyTwo)
}
}
// openDirectoryNearPathLimit makes a chain of directories inside dir, each
// named with 255 zeros, the longest name Linux allows. Linux refuses a path
// of 4096 bytes or more, and the chain goes as deep as it can below that.
// It returns the chain's deepest directory as an os.Root, closed when the
// test ends. An entry made through it with a 255-byte name has a path even
// the root user cannot read. The os.Root can make that entry because it
// names the entry only by its name, not by the whole path.
func openDirectoryNearPathLimit(t *testing.T, dir string) *os.Root {
t.Helper()
name := strings.Repeat("0", 255)
deepest := dir
for len(filepath.Join(deepest, name)) < 4096 {
deepest = filepath.Join(deepest, name)
}
err := os.MkdirAll(deepest, StorageDirPerm)
if err != nil {
t.Fatalf("failed to make a chain of directories: %v", err)
}
root, err := os.OpenRoot(deepest)
if err != nil {
t.Fatalf("os.OpenRoot(%s) error = %v", deepest, err)
}
t.Cleanup(func() { _ = root.Close() })
return root
}
// makeEntriesTooLongToRead makes a new directory and a new file in the
// deepest directory of the chain that openDirectoryNearPathLimit makes
// inside dir. Even the root user can then neither list the new directory
// nor read the new file's info. It returns the paths of the new directory
// and the new file.
func makeEntriesTooLongToRead(t *testing.T, dir string) []string {
t.Helper()
deepest := openDirectoryNearPathLimit(t, dir)
dirName := strings.Repeat("0", 255)
err := deepest.Mkdir(dirName, StorageDirPerm)
if err != nil {
t.Fatalf("failed to make a directory too deep to list: %v", err)
}
fileName := strings.Repeat("1", 255)
err = deepest.WriteFile(fileName, []byte("unreadable"), StorageFilePerm)
if err != nil {
t.Fatalf("failed to make a file too deep to read: %v", err)
}
return []string{
filepath.Join(deepest.Name(), dirName),
filepath.Join(deepest.Name(), fileName),
}
}
// makeTempFileTooLongToRead makes a temp file in the deepest directory of
// the chain that openDirectoryNearPathLimit makes inside dir. Even the
// root user then cannot read the temp file's info. It returns the temp
// file's path.
func makeTempFileTooLongToRead(t *testing.T, dir string) string {
t.Helper()
deepest := openDirectoryNearPathLimit(t, dir)
name := tempFilePrefix + strings.Repeat("1", 255-len(tempFilePrefix))
err := deepest.WriteFile(name, []byte("partial"), StorageFilePerm)
if err != nil {
t.Fatalf("failed to make a temp file too deep to read: %v", err)
}
return filepath.Join(deepest.Name(), name)
}
// assertSkippedEntryWarning checks that logs hold the warning "skipped
// unreadable entry in cache directory" naming path, with the error
// ENAMETOOLONG for that path.
func assertSkippedEntryWarning(t *testing.T, logs, path string) {
t.Helper()
for line := range strings.Lines(logs) {
if strings.Contains(line, `"level":"WARN"`) &&
strings.Contains(line, `"msg":"skipped unreadable entry in cache directory"`) &&
strings.Contains(line, `"path":"`+path+`"`) &&
strings.Contains(line, path+": "+syscall.ENAMETOOLONG.Error()) {
return
}
}
t.Errorf("no warning %q names %s with the error %q",
"skipped unreadable entry in cache directory", path, syscall.ENAMETOOLONG)
}
// TestReconciliationGoesOnPastEntriesItCannotRead puts, in each storage
// directory, a directory the pass cannot list and a file whose info it
// cannot read. The pass walks each directory in name order, so it reaches
// them before the cache's files, whose keys start with a letter. It must
// log a warning naming each one's path and the error, and still do the
// rest of its work:
// - adopt a variant file that has no row
// - drop the row of a variant whose file is missing
// - remove a source file that has no row
// - correct a wrong total cache usage
func TestReconciliationGoesOnPastEntriesItCannotRead(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
ctx := t.Context()
untrackedVariant := []byte("variant with no row")
err := cache.variants.Write(string(testVariantKeyOne), untrackedVariant)
if err != nil {
t.Fatalf("failed to write a variant file with no row: %v", err)
}
storeEvictionTestVariant(t, cache, testVariantKeyTwo, []byte("variant two"))
err = os.Remove(cache.variants.keyToPath(string(testVariantKeyTwo)))
if err != nil {
t.Fatalf("failed to remove the second variant's file: %v", err)
}
untrackedSourceKey := "ffeeddcc0001"
err = cache.srcContent.Write(untrackedSourceKey, []byte("source with no row"))
if err != nil {
t.Fatalf("failed to write a source file with no row: %v", err)
}
_, err = cache.db.ExecContext(ctx,
`UPDATE cache_usage SET total_size_bytes = 1 WHERE id = 1`)
if err != nil {
t.Fatalf("failed to set a wrong total: %v", err)
}
unreadable := slices.Concat(
makeEntriesTooLongToRead(t, cache.variants.baseDir),
makeEntriesTooLongToRead(t, cache.srcContent.baseDir),
)
logs := reconcileAsAtStartup(t, cache)
for _, path := range unreadable {
assertSkippedEntryWarning(t, logs, path)
}
rowQuery := `SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`
if n := countRows(t, cache, rowQuery, string(testVariantKeyOne)); n != 1 {
t.Errorf("the variant file with no row got %d rows, want 1", n)
}
if n := countRows(t, cache, rowQuery, string(testVariantKeyTwo)); n != 0 {
t.Errorf("the variant with no file kept %d rows, want 0", n)
}
_, err = os.Stat(cache.srcContent.keyToPath(untrackedSourceKey))
if err == nil {
t.Error("the source file with no row is still on disk")
}
usage, err := cache.UsageBytes(ctx)
if err != nil {
t.Fatalf("UsageBytes() error = %v", err)
}
if usage != int64(len(untrackedVariant)) {
t.Errorf("UsageBytes() = %d, want %d, the size of the adopted variant",
usage, len(untrackedVariant))
}
}
// TestReconciliationWarnsOfTempFilesWhoseInfoItCannotRead puts, in each
// storage directory, a temp file whose info the pass cannot read. The
// pass must log a warning naming each one's path and the error, as it
// does for any other file whose info it cannot read.
func TestReconciliationWarnsOfTempFilesWhoseInfoItCannotRead(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
unreadable := []string{
makeTempFileTooLongToRead(t, cache.variants.baseDir),
makeTempFileTooLongToRead(t, cache.srcContent.baseDir),
}
logs := reconcileAsAtStartup(t, cache)
for _, path := range unreadable {
assertSkippedEntryWarning(t, logs, path)
}
}
// TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup proves // TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup proves
// reconciliation is not a one-shot startup-only pass: it must also run // reconciliation is not a one-shot startup-only pass: it must also run
// on the periodic ticker, so a variant file that lands on disk with no // on the periodic ticker, so a variant file that lands on disk with no
// accounting row well after startup (e.g. because StoreVariant's // accounting row well after startup (e.g. because PutVariant's
// best-effort accounting insert failed under transient contention, or // best-effort accounting insert failed under transient contention, or
// any other cause of an untracked file appearing during steady-state // any other cause of an untracked file appearing during steady-state
// operation) is still adopted into accounting eventually, rather than // operation) is still adopted into accounting eventually, rather than
@@ -733,14 +1025,12 @@ func TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup(t *testing.T) {
// Simulate a variant whose accounting insert failed after the // Simulate a variant whose accounting insert failed after the
// process was already running and serving requests: the content // process was already running and serving requests: the content
// file is written directly, bypassing StoreVariant's (and thus its // file is written directly, bypassing PutVariant's (and thus its
// accounting insert) entirely, exactly as would happen if that // accounting insert) entirely, exactly as would happen if that
// insert had failed and only the file write had succeeded. // insert had failed and only the file write had succeeded.
untracked := bytes.Repeat([]byte{0x41}, 900) untracked := bytes.Repeat([]byte{0x41}, 900)
_, err = cache.variants.Store( err = cache.variants.Write("aabbccdd0099", untracked)
"aabbccdd0099", bytes.NewReader(untracked),
)
if err != nil { if err != nil {
t.Fatalf("failed to store untracked variant file: %v", err) t.Fatalf("failed to store untracked variant file: %v", err)
} }
@@ -1046,13 +1336,15 @@ func assertOnlyOldestSourceEvicted(
) { ) {
t.Helper() t.Helper()
if cache.srcContent.Exists(keys[0]) { _, err := os.Stat(cache.srcContent.keyToPath(string(keys[0])))
if err == nil {
t.Errorf("source %s, evicted when the context was cancelled, "+ t.Errorf("source %s, evicted when the context was cancelled, "+
"is still on disk", keys[0]) "is still on disk", keys[0])
} }
for _, key := range keys[1:] { for _, key := range keys[1:] {
if !cache.srcContent.Exists(key) { _, err = os.Stat(cache.srcContent.keyToPath(string(key)))
if err != nil {
t.Errorf("source %s was evicted after the context was cancelled", key) t.Errorf("source %s was evicted after the context was cancelled", key)
} }
internal/imgcache/filestore.go
+185
View File
@@ -0,0 +1,185 @@
package imgcache
import (
"errors"
"fmt"
"os"
"path/filepath"
)
// Storage constants.
const (
// StorageDirPerm is the permission mode for storage directories.
StorageDirPerm = 0750
// StorageFilePerm is the permission mode for storage files.
StorageFilePerm = 0600
// MinHashLength is the minimum hash length for path splitting.
MinHashLength = 4
)
// Storage errors.
var (
ErrNotFound = errors.New("content not found")
)
// FileStore is a directory of files, each named by a hex key. The file for
// a key is at <basedir>/<ab>/<cd>/<key>, where <ab> and <cd> are the key's
// first and second pairs of characters. The cache keeps its source images in
// one FileStore and its variants in another.
type FileStore struct {
baseDir string
}
// NewFileStore creates the directory baseDir if it is missing and returns
// the FileStore of the files in it.
func NewFileStore(baseDir string) (*FileStore, error) {
err := os.MkdirAll(baseDir, StorageDirPerm)
if err != nil {
return nil, fmt.Errorf("failed to create storage directory: %w", err)
}
return &FileStore{baseDir: baseDir}, nil
}
// Write writes data with writeFileSynced to the file for key, replacing any
// file already there.
func (s *FileStore) Write(key string, data []byte) error {
path := s.keyToPath(key)
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
if err != nil {
return fmt.Errorf("failed to create directory: %w", err)
}
return writeFileSynced(path, data)
}
// Open opens the file for key and returns it with its size in bytes. A
// missing file is ErrNotFound.
func (s *FileStore) Open(key string) (*os.File, int64, error) {
f, err := os.Open(s.keyToPath(key))
if err != nil {
if os.IsNotExist(err) {
return nil, 0, ErrNotFound
}
return nil, 0, fmt.Errorf("failed to open content: %w", err)
}
stat, err := f.Stat()
if err != nil {
_ = f.Close()
return nil, 0, fmt.Errorf("failed to stat content: %w", err)
}
return f, stat.Size(), nil
}
// Size returns the size in bytes of the file for key. A missing file is
// ErrNotFound.
func (s *FileStore) Size(key string) (int64, error) {
stat, err := os.Stat(s.keyToPath(key))
if err != nil {
if os.IsNotExist(err) {
return 0, ErrNotFound
}
return 0, fmt.Errorf("failed to stat content: %w", err)
}
return stat.Size(), nil
}
// Delete removes the file for key. A missing file is not an error.
func (s *FileStore) Delete(key string) error {
err := os.Remove(s.keyToPath(key))
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("failed to delete content: %w", err)
}
return nil
}
// writeFileSynced writes data to path through a temp file in the same
// directory: the temp file is synced to disk, renamed to path, and the
// directory synced after, so that neither a concurrent reader nor a
// crash finds a partial file under path. When the directory sync fails,
// the file is removed from path, so that a failed write leaves no file
// there.
func writeFileSynced(path string, data []byte) error {
dir := filepath.Dir(path)
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
if err != nil {
return fmt.Errorf("failed to create temp file: %w", err)
}
tmpPath := tmpFile.Name()
_, err = tmpFile.Write(data)
if err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to write temp file: %w", err)
}
err = tmpFile.Sync()
if err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to sync temp file: %w", err)
}
err = tmpFile.Close()
if err != nil {
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to close temp file: %w", err)
}
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
if err != nil {
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to rename temp file: %w", err)
}
err = syncDirectory(dir)
if err != nil {
_ = os.Remove(path)
return err
}
return nil
}
// syncDirectory syncs the directory dir to disk, so that a file just
// renamed into it is still there after a crash.
func syncDirectory(dir string) error {
d, err := os.Open(filepath.Clean(dir))
if err != nil {
return fmt.Errorf("failed to open directory: %w", err)
}
err = d.Sync()
_ = d.Close()
if err != nil {
return fmt.Errorf("failed to sync directory: %w", err)
}
return nil
}
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
func (s *FileStore) keyToPath(key string) string {
if len(key) < MinHashLength {
return filepath.Join(s.baseDir, key)
}
return filepath.Join(s.baseDir, key[0:2], key[2:4], key)
}
internal/imgcache/filestore_internal_test.go
+184
View File
@@ -0,0 +1,184 @@
package imgcache
import (
"bytes"
"errors"
"io"
"os"
"path/filepath"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
func TestFileStore_WriteAndOpen(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewFileStore(tmpDir)
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
content := []byte("hello world")
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
err = storage.Write(key, content)
if err != nil {
t.Fatalf("Write() error = %v", err)
}
// Verify file exists at expected path
expectedPath := filepath.Join(tmpDir, key[0:2], key[2:4], key)
_, err = os.Stat(expectedPath)
if err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
// Open and verify size and content
r, size, err := storage.Open(key)
if err != nil {
t.Fatalf("Open() error = %v", err)
}
defer func() { _ = r.Close() }()
if size != int64(len(content)) {
t.Errorf("Open() size = %d, want %d", size, len(content))
}
loaded, err := io.ReadAll(r)
if err != nil {
t.Fatalf("ReadAll() error = %v", err)
}
if !bytes.Equal(loaded, content) {
t.Errorf("Open() content = %q, want %q", loaded, content)
}
}
func TestFileStore_OpenNotFound(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewFileStore(tmpDir)
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
_, _, err = storage.Open("nonexistent")
if !errors.Is(err, ErrNotFound) {
t.Errorf("Open() error = %v, want ErrNotFound", err)
}
}
// TestFileStore_Size checks that Size returns the size of a written file,
// and ErrNotFound for a key with no file.
func TestFileStore_Size(t *testing.T) {
t.Parallel()
storage, err := NewFileStore(t.TempDir())
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
content := []byte("the file's content")
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
err = storage.Write(key, content)
if err != nil {
t.Fatalf("Write() error = %v", err)
}
size, err := storage.Size(key)
if err != nil || size != int64(len(content)) {
t.Errorf("Size() = %d, error %v; want %d", size, err, len(content))
}
_, err = storage.Size("nonexistent")
if !errors.Is(err, ErrNotFound) {
t.Errorf("Size() of a missing file error = %v, want ErrNotFound", err)
}
}
func TestFileStore_Delete(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewFileStore(tmpDir)
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
content := []byte("to be deleted")
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
path := storage.keyToPath(key)
err = storage.Write(key, content)
if err != nil {
t.Fatalf("Write() error = %v", err)
}
_, err = os.Stat(path)
if err != nil {
t.Errorf("file missing before Delete(): %v", err)
}
err = storage.Delete(key)
if err != nil {
t.Fatalf("Delete() error = %v", err)
}
_, err = os.Stat(path)
if err == nil {
t.Error("file still on disk after Delete()")
}
}
func TestFileStore_DeleteNonexistent(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewFileStore(tmpDir)
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
// Should not error
err = storage.Delete("nonexistent")
if err != nil {
t.Errorf("Delete() error = %v, want nil", err)
}
}
func TestFileStore_KeyToPath(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewFileStore(tmpDir)
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
// Test by writing and verifying the resulting path structure
content := []byte("test content for path verification")
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
err = storage.Write(key, content)
if err != nil {
t.Fatalf("Write() error = %v", err)
}
expectedPath := filepath.Join(tmpDir, key[0:2], key[2:4], key)
_, err = os.Stat(expectedPath)
if err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
}
internal/imgcache/imgcache.go
-29
View File
@@ -1,29 +0,0 @@
// Package imgcache provides types for the image caching proxy.
package imgcache
import "io"
// ImageResponse represents a processed image ready to serve
type ImageResponse struct {
// Content is the image data reader
Content io.ReadCloser
// ContentLength is the size in bytes (-1 if unknown)
ContentLength int64
// ContentType is the MIME type of the response
ContentType string
// ETag is the entity tag for caching
ETag string
// CacheStatus indicates HIT or MISS
CacheStatus CacheStatus
// FetchedBytes is the number of bytes fetched from upstream (0 if cache hit)
FetchedBytes int64
}
// CacheStatus indicates whether the response was served from cache.
type CacheStatus string
// Cache status values for response headers.
const (
CacheHit CacheStatus = "HIT"
CacheMiss CacheStatus = "MISS"
)
internal/imgcache/module.go
-8
View File
@@ -1,8 +0,0 @@
package imgcache
import "time"
// CacheConfig defaults.
const (
DefaultNegativeTTL = 5 * time.Minute
)
internal/imgcache/negative_cache_internal_test.go
+13 -90
View File
@@ -17,7 +17,7 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
db := setupTestDB(t) db := setupTestDB(t)
dir := t.TempDir() dir := t.TempDir()
cache, err := NewCache(db, CacheConfig{ cache, err := NewCache(db, Config{
StateDir: dir, StateDir: dir,
NegativeTTL: 5 * time.Minute, NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30, MaxBytes: 1 << 30,
@@ -35,7 +35,7 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
} }
// Initially should not be in negative cache // Initially should not be in negative cache
hit, err := cache.checkNegativeCache(ctx, req) hit, err := cache.Failure(ctx, req.Source)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -45,13 +45,13 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
} }
// Store a negative entry // Store a negative entry
err = cache.StoreNegative(ctx, req) err = cache.PutFailure(ctx, req.Source)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
// Now should be in negative cache // Now should be in negative cache
hit, err = cache.checkNegativeCache(ctx, req) hit, err = cache.Failure(ctx, req.Source)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -67,7 +67,7 @@ func TestNegativeCache_Expired(t *testing.T) {
db := setupTestDB(t) db := setupTestDB(t)
dir := t.TempDir() dir := t.TempDir()
cache, err := NewCache(db, CacheConfig{ cache, err := NewCache(db, Config{
StateDir: dir, StateDir: dir,
NegativeTTL: 1 * time.Millisecond, // very short TTL NegativeTTL: 1 * time.Millisecond, // very short TTL
MaxBytes: 1 << 30, MaxBytes: 1 << 30,
@@ -85,7 +85,7 @@ func TestNegativeCache_Expired(t *testing.T) {
} }
// Store a negative entry with very short TTL // Store a negative entry with very short TTL
err = cache.StoreNegative(ctx, req) err = cache.PutFailure(ctx, req.Source)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -94,7 +94,7 @@ func TestNegativeCache_Expired(t *testing.T) {
time.Sleep(10 * time.Millisecond) time.Sleep(10 * time.Millisecond)
// Should no longer be in negative cache // Should no longer be in negative cache
hit, err := cache.checkNegativeCache(ctx, req) hit, err := cache.Failure(ctx, req.Source)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -104,71 +104,6 @@ func TestNegativeCache_Expired(t *testing.T) {
} }
} }
// TestService_Get_ServesCachedVariantOfNegativeCachedURL caches a variant of
// the test photo, then keeps a failed fetch of the photo's URL in the negative
// cache, as a failed fetch for another variant does once the source has been
// evicted. The cached variant is still served, and a variant not in the cache
// is made from the cached source. Once the source is evicted, a variant not in
// the cache needs a fetch and is refused.
func TestService_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
resp, err := svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
_ = resp.Content.Close()
err = svc.cache.StoreNegative(t.Context(),
photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("StoreNegative() error = %v", err)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() of the cached variant error = %v", err)
}
_ = resp.Content.Close()
if resp.CacheStatus != CacheHit {
t.Errorf("cached variant CacheStatus = %q, want %q",
resp.CacheStatus, CacheHit)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
t.Logf("Get() of a variant made from the cached source: error = %v", err)
if err != nil {
t.Fatalf("Get() of a variant made from the cached source error = %v", err)
}
_ = resp.Content.Close()
sourceKey, err := svc.cache.LookupSource(t.Context(),
photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil || sourceKey == "" {
t.Fatalf("LookupSource() = %q, %v, want the cached source", sourceKey, err)
}
err = svc.cache.evictSourceBlob(t.Context(), sourceKey)
if err != nil {
t.Fatalf("evictSourceBlob() error = %v", err)
}
_, err = svc.Get(t.Context(), photoVariant(fixtures, 60, imagerequest.FitCover))
t.Logf("Get() of a variant whose source was evicted: error = %v", err)
if !errors.Is(err, ErrNegativeCached) {
t.Errorf("Get() of a variant whose source was evicted error = %v, want %v",
err, ErrNegativeCached)
}
}
// TestEvictionLoopRemovesExpiredNegativeCacheRows stores a failed fetch // TestEvictionLoopRemovesExpiredNegativeCacheRows stores a failed fetch
// that expires a second later and starts the eviction loop with a short // that expires a second later and starts the eviction loop with a short
// interval: a later tick must remove its row although its URL is never // interval: a later tick must remove its row although its URL is never
@@ -190,7 +125,7 @@ func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
cache, err := NewCache(evictionTestDB(t), CacheConfig{ cache, err := NewCache(evictionTestDB(t), Config{
StateDir: t.TempDir(), StateDir: t.TempDir(),
NegativeTTL: time.Second, NegativeTTL: time.Second,
MaxBytes: tt.maxBytes, MaxBytes: tt.maxBytes,
@@ -203,9 +138,9 @@ func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
Source: imagerequest.Source{Host: testHostExample, Path: "/gone.jpg"}, Source: imagerequest.Source{Host: testHostExample, Path: "/gone.jpg"},
} }
err = cache.StoreNegative(t.Context(), expired) err = cache.PutFailure(t.Context(), expired.Source)
if err != nil { if err != nil {
t.Fatalf("StoreNegative failed: %v", err) t.Fatalf("PutFailure failed: %v", err)
} }
const notExpiredPath = "/later.jpg" const notExpiredPath = "/later.jpg"
@@ -269,7 +204,7 @@ func TestCleanExpiredDeletesInBatches(t *testing.T) {
t.Parallel() t.Parallel()
cache, err := NewCache(evictionTestDB(t), cache, err := NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry()) prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("failed to create cache: %v", err) t.Fatalf("failed to create cache: %v", err)
@@ -325,7 +260,7 @@ func TestCleanExpiredRemovesMoreThanOneBatch(t *testing.T) {
t.Parallel() t.Parallel()
cache, err := NewCache(evictionTestDB(t), cache, err := NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry()) prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("failed to create cache: %v", err) t.Fatalf("failed to create cache: %v", err)
@@ -357,7 +292,7 @@ func TestCleanExpiredStopsOnceContextEnds(t *testing.T) {
t.Parallel() t.Parallel()
cache, err := NewCache(evictionTestDB(t), cache, err := NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry()) prometheus.NewRegistry())
if err != nil { if err != nil {
t.Fatalf("failed to create cache: %v", err) t.Fatalf("failed to create cache: %v", err)
@@ -410,15 +345,3 @@ func negativeCacheTableRowCount(tb testing.TB, cache *Cache) int {
return count return count
} }
func TestService_Get_ReturnsErrorForNegativeCachedURL(t *testing.T) {
t.Parallel()
// This test verifies that Service.Get() checks the negative cache
// We can't easily test the full pipeline without vips, but we can
// verify the error type
err := ErrNegativeCached
if !errors.Is(err, ErrNegativeCached) {
t.Error("ErrNegativeCached should be identifiable with errors.Is")
}
}
internal/imgcache/partial_cache_files_internal_test.go
+69 -187
View File
@@ -12,67 +12,6 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
) )
// getImage calls Get for req and returns the image it answered with and
// its cache status.
func getImage(
t *testing.T, svc *Service, req *imagerequest.Request,
) ([]byte, CacheStatus) {
t.Helper()
resp, err := svc.Get(t.Context(), req)
if err != nil {
t.Fatalf("Get() of quality %d error = %v", req.Transform.Quality, err)
}
defer func() { _ = resp.Content.Close() }()
body, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("reading the image of quality %d: %v", req.Transform.Quality, err)
}
return body, resp.CacheStatus
}
// fileSize returns the size of the file at path.
func fileSize(t *testing.T, path string) int64 {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
return info.Size()
}
// truncateFile cuts the file at path down to size bytes.
func truncateFile(t *testing.T, path string, size int64) {
t.Helper()
t.Logf("cutting %s from %d to %d bytes",
filepath.Base(path), fileSize(t, path), size)
err := os.Truncate(path, size)
if err != nil {
t.Fatalf("truncating %s: %v", path, err)
}
}
// cachedSourcePath returns the path of the cached source file req uses.
func cachedSourcePath(
t *testing.T, svc *Service, req *imagerequest.Request,
) string {
t.Helper()
sourceKey, err := svc.cache.LookupSource(t.Context(), req)
if err != nil || sourceKey == "" {
t.Fatalf("LookupSource() = %q, %v; want the cached source", sourceKey, err)
}
return svc.cache.srcContent.keyToPath(sourceKey)
}
// reconcileAsAtStartup runs the cache's reconciliation pass, as pixa does // reconcileAsAtStartup runs the cache's reconciliation pass, as pixa does
// when it starts, and returns what the pass logged. // when it starts, and returns what the pass logged.
func reconcileAsAtStartup(t *testing.T, cache *Cache) string { func reconcileAsAtStartup(t *testing.T, cache *Cache) string {
@@ -107,127 +46,67 @@ func assertWarningNames(t *testing.T, logs, path string) {
t.Errorf("reconciliation logged no warning naming %s", path) t.Errorf("reconciliation logged no warning naming %s", path)
} }
// TestService_Get_FetchesAgainForEmptyCachedSource empties the cached source // TestReconciliationRemovesDamagedFiles empties a cached source file and a
// file, as a crash or a restored snapshot could, and runs the reconciliation // cached variant file, or cuts them to half their size, as a crash or a
// pass, which must remove it with a warning naming it. It then asks for two // restored snapshot could. It checks that the reconciliation pass removes
// new sizes of the source: the first must fetch it again and cache it whole, // each file and its row, with a warning naming the file.
// so that the second is made from the cache without a third fetch. func TestReconciliationRemovesDamagedFiles(t *testing.T) {
func TestService_Get_FetchesAgainForEmptyCachedSource(t *testing.T) {
t.Parallel() t.Parallel()
svc, fixtures := SetupTestService(t) content := []byte("the whole cached image")
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover)) for name, size := range map[string]int64{
"empty": 0,
"partial": int64(len(content) / 2),
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover)) cache, _ := newEvictionTestCache(t, 1<<30)
wholeSize := fileSize(t, path)
truncateFile(t, path, 0) sourceKey := storeEvictionTestSource(t, cache, testHostCDN, testPathCat, content)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path) storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover)) sourcePath := cache.srcContent.keyToPath(string(sourceKey))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover)) variantPath := cache.variants.keyToPath(string(testVariantKeyOne))
fetches := readCounters(svc.cache).upstreamFetchCount for _, path := range []string{sourcePath, variantPath} {
t.Logf("upstream fetches = %d", fetches) err := os.Truncate(path, size)
if err != nil {
t.Fatalf("cutting %s to %d bytes: %v", path, size, err)
}
}
if fetches != 2 { logs := reconcileAsAtStartup(t, cache)
t.Errorf("upstream fetches = %d, want 2: the empty cached source "+
"was kept and fetched again on every new size", fetches) assertWarningNames(t, logs, sourcePath)
assertWarningNames(t, logs, variantPath)
_, err := os.Stat(sourcePath)
if err == nil {
t.Error("the damaged source file is still on disk")
}
_, err = os.Stat(variantPath)
if err == nil {
t.Error("the damaged variant file is still on disk")
}
rows := countRows(t, cache,
`SELECT COUNT(*) FROM source_content WHERE source_key = ?`,
string(sourceKey))
if rows != 0 {
t.Errorf("the damaged source file kept %d rows, want 0", rows)
}
rows = countRows(t, cache,
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`,
string(testVariantKeyOne))
if rows != 0 {
t.Errorf("the damaged variant file kept %d rows, want 0", rows)
}
})
} }
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// TestService_Get_FetchesAgainForPartialCachedSource cuts the cached source
// file to half its size, as a crash or a restored snapshot could, and runs
// the reconciliation pass, which must remove it with a warning naming it.
// It then asks for two new sizes of the source: the first must fetch it
// again rather than fail to decode the partial file, and the second is
// made from the cache.
func TestService_Get_FetchesAgainForPartialCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
truncateFile(t, path, wholeSize/2)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
fetches := readCounters(svc.cache).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// assertConvertedAgain caches a variant of the photo, cuts its file to size
// bytes and runs the reconciliation pass, which must remove it with a
// warning naming it. It then checks that Get converts the image again
// instead of serving the file, and that the next Get serves the whole
// variant from the cache.
func assertConvertedAgain(t *testing.T, size int64) {
t.Helper()
svc, fixtures := SetupTestService(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := svc.cache.variants.keyToPath(imagerequest.CacheKey(req))
truncateFile(t, path, size)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
got, status := getImage(t, svc, req)
t.Logf("after the cut: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the cut: Get() = %d bytes, %s; want the %d bytes "+
"converted again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
t.Logf("next request: %d bytes, %s", len(got), status)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestService_Get_ConvertsAgainForEmptyCachedVariant empties a cached
// variant file, as a crash or a restored snapshot could, and checks that it
// is converted again rather than served empty.
func TestService_Get_ConvertsAgainForEmptyCachedVariant(t *testing.T) {
t.Parallel()
assertConvertedAgain(t, 0)
}
// TestService_Get_ConvertsAgainForPartialCachedVariant cuts a cached
// variant file to ten bytes, as a crash or a restored snapshot could, and
// checks that it is converted again rather than served cut short.
func TestService_Get_ConvertsAgainForPartialCachedVariant(t *testing.T) {
t.Parallel()
assertConvertedAgain(t, 10)
} }
// TestReconciliationRemovesEmptyVariantFileWithoutRow checks that the // TestReconciliationRemovesEmptyVariantFileWithoutRow checks that the
@@ -239,16 +118,19 @@ func TestReconciliationRemovesEmptyVariantFileWithoutRow(t *testing.T) {
cache, _ := newEvictionTestCache(t, 1<<30) cache, _ := newEvictionTestCache(t, 1<<30)
_, err := cache.variants.Store(testVariantKeyOne, bytes.NewReader(nil)) err := cache.variants.Write(string(testVariantKeyOne), nil)
if err != nil { if err != nil {
t.Fatalf("failed to store an empty variant file: %v", err) t.Fatalf("failed to store an empty variant file: %v", err)
} }
variantPath := cache.variants.keyToPath(string(testVariantKeyOne))
logs := reconcileAsAtStartup(t, cache) logs := reconcileAsAtStartup(t, cache)
assertWarningNames(t, logs, cache.variants.keyToPath(testVariantKeyOne)) assertWarningNames(t, logs, variantPath)
if cache.variants.Exists(testVariantKeyOne) { _, err = os.Stat(variantPath)
if err == nil {
t.Error("the empty variant file is still on disk") t.Error("the empty variant file is still on disk")
} }
@@ -260,19 +142,19 @@ func TestReconciliationRemovesEmptyVariantFileWithoutRow(t *testing.T) {
} }
} }
// TestContentStorage_StoreReplacesFileOfAnotherSize checks that a // TestFileStore_WriteReplacesFileOfAnotherSize checks that a write over a
// store over a file of another size at the source's path, such as a // file of another size at the source's path, such as a partial file a crash
// partial file a crash left, replaces it with the content. // left, replaces it with the content.
func TestContentStorage_StoreReplacesFileOfAnotherSize(t *testing.T) { func TestFileStore_WriteReplacesFileOfAnotherSize(t *testing.T) {
t.Parallel() t.Parallel()
storage, err := NewContentStorage(t.TempDir()) storage, err := NewFileStore(t.TempDir())
if err != nil { if err != nil {
t.Fatalf("NewContentStorage() error = %v", err) t.Fatalf("NewFileStore() error = %v", err)
} }
content := []byte("the whole source image") content := []byte("the whole source image")
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key() key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
path := storage.keyToPath(key) path := storage.keyToPath(key)
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm) err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
@@ -285,14 +167,14 @@ func TestContentStorage_StoreReplacesFileOfAnotherSize(t *testing.T) {
t.Fatalf("writing a partial file: %v", err) t.Fatalf("writing a partial file: %v", err)
} }
_, err = storage.Store(key, content) err = storage.Write(key, content)
if err != nil { if err != nil {
t.Fatalf("Store() error = %v", err) t.Fatalf("Write() error = %v", err)
} }
stored, _, err := storage.LoadWithSize(key) stored, _, err := storage.Open(key)
if err != nil { if err != nil {
t.Fatalf("LoadWithSize() error = %v", err) t.Fatalf("Open() error = %v", err)
} }
defer func() { _ = stored.Close() }() defer func() { _ = stored.Close() }()
internal/imgcache/pendingaccesstimes_internal_test.go
+20 -59
View File
@@ -5,6 +5,8 @@ import (
"context" "context"
"database/sql" "database/sql"
"errors" "errors"
"image/color"
"io"
"log/slog" "log/slog"
"strings" "strings"
"testing" "testing"
@@ -15,8 +17,7 @@ import (
// pendingAccessTimesWait is how long a test waits for the database's // pendingAccessTimesWait is how long a test waits for the database's
// connection, for an access time to reach the database, for a write to start // connection, for an access time to reach the database, for a write to start
// waiting for the connection, for a request to return past its deadline, or // waiting for the connection, or for StopPendingAccessTimeWrites to return.
// for StopPendingAccessTimeWrites to return.
const pendingAccessTimesWait = 5 * time.Second const pendingAccessTimesWait = 5 * time.Second
// otherWritesWait is how long a test gives other writes, once one write waits // otherWritesWait is how long a test gives other writes, once one write waits
@@ -56,52 +57,6 @@ func waitForConnectionWaits(t *testing.T, db *sql.DB, waits int64) {
} }
} }
// TestService_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy holds the
// database's one connection while a request whose fetch is held reaches its
// deadline. The request must return with the deadline's error while the
// connection is still held, as it waits for no database write past its
// deadline.
func TestService_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
// Room for the request to reach its fetch on a busy host
const timeout = 2 * time.Second
ctx, cancel := context.WithTimeout(t.Context(), timeout)
defer cancel()
deadline, _ := ctx.Deadline()
results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
// The request has made its database reads by the time it fetches.
select {
case <-fetcher.started:
case <-time.After(timeout):
t.Fatal("request did not reach its fetch by its deadline")
}
releaseDatabase := holdDatabase(t, svc.cache)
defer releaseDatabase()
select {
case got := <-results:
t.Logf("Get() returned %v after its deadline, error = %v",
time.Since(deadline), got.err)
if !errors.Is(got.err, context.DeadlineExceeded) {
t.Errorf("Get() error = %v, want %v", got.err, context.DeadlineExceeded)
}
case <-time.After(time.Until(deadline) + pendingAccessTimesWait):
t.Fatal("request did not return while the database was busy")
}
}
// TestStopPendingAccessTimeWrites_LetsTheWriteUnderWayFinish holds the // TestStopPendingAccessTimeWrites_LetsTheWriteUnderWayFinish holds the
// database's one connection while the goroutine that writes the pending // database's one connection while the goroutine that writes the pending
// access times waits for it, hands over another access time, then stops that // access times waits for it, hands over another access time, then stops that
@@ -242,16 +197,16 @@ func TestStopPendingAccessTimeWrites_ReturnsWhenItsContextEnds(t *testing.T) {
} }
} }
// TestLookupSource_HandsAccessTimeToTheBackgroundWriter finds a cached source // TestSource_HandsAccessTimeToTheBackgroundWriter finds a cached source
// whose access time is an hour old. LookupSource must not write the access // whose access time is an hour old. Source must not write the access
// time itself but hand it over to be written, and StopPendingAccessTimeWrites // time itself but hand it over to be written, and StopPendingAccessTimeWrites
// must then write it. // must then write it.
func TestLookupSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) { func TestSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) {
t.Parallel() t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30) cache, _ := newEvictionTestCache(t, 1<<30)
key := storeEvictionTestSource(t, cache, "example.com", "/photo.jpg", content := generateTestJPEG(t, 10, 10, color.RGBA{0, 0, 255, 255})
[]byte("source bytes")) key := storeEvictionTestSource(t, cache, "example.com", "/photo.jpg", content)
anHourAgo := time.Now().Add(-time.Hour) anHourAgo := time.Now().Add(-time.Hour)
setSourceLastAccessed(t, cache, key, anHourAgo) setSourceLastAccessed(t, cache, key, anHourAgo)
@@ -261,15 +216,21 @@ func TestLookupSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) {
` `
anHourAgoInDatabase := anHourAgo.UTC().Format(sqliteTimestampFormat) anHourAgoInDatabase := anHourAgo.UTC().Format(sqliteTimestampFormat)
found, err := cache.LookupSource(t.Context(), &imagerequest.Request{ reader, _, _, err := cache.Source(t.Context(),
Source: imagerequest.Source{Host: "example.com", Path: "/photo.jpg"}, imagerequest.Source{Host: "example.com", Path: "/photo.jpg"})
}) if err != nil {
if err != nil || found != key { t.Fatalf("Source() error = %v; want the source of %q", err, key)
t.Fatalf("LookupSource() = %q, %v; want %q", found, err, key) }
found, err := io.ReadAll(reader)
_ = reader.Close()
if err != nil || !bytes.Equal(found, content) {
t.Fatalf("Source() reads %q, %v; want the source of %q", found, err, key)
} }
if countRows(t, cache, touchedQuery, string(key), anHourAgoInDatabase) != 0 { if countRows(t, cache, touchedQuery, string(key), anHourAgoInDatabase) != 0 {
t.Error("LookupSource wrote the source's access time itself") t.Error("Source wrote the source's access time itself")
} }
if waiting := len(cache.pendingSourceAccessTimes); waiting != 1 { if waiting := len(cache.pendingSourceAccessTimes); waiting != 1 {
internal/imgcache/source_content_internal_test.go
+118 -21
View File
@@ -2,19 +2,24 @@ package imgcache
import ( import (
"bytes" "bytes"
"errors"
"fmt"
"image/color"
"io" "io"
"os" "os"
"path/filepath" "path/filepath"
"testing" "testing"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic"
) )
// TestStoreSource_StoresIdenticalBytesOfTwoURLsTwice stores the same bytes // TestPutSource_StoresIdenticalBytesOfTwoURLsTwice stores the same bytes
// for two source URLs and checks that each URL has a source_content row of // for two source URLs and checks that each URL has a source_content row of
// its own and a file of its own, named by its source key under // its own and a file of its own, named by its source key under
// cache/sources/<ab>/<cd>/. // cache/sources/<ab>/<cd>/.
func TestStoreSource_StoresIdenticalBytesOfTwoURLsTwice(t *testing.T) { func TestPutSource_StoresIdenticalBytesOfTwoURLsTwice(t *testing.T) {
t.Parallel() t.Parallel()
cache, stateDir := newEvictionTestCache(t, 1<<30) cache, stateDir := newEvictionTestCache(t, 1<<30)
@@ -64,7 +69,7 @@ func TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed(t *testing.T) {
t.Parallel() t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30) cache, _ := newEvictionTestCache(t, 1<<30)
content := bytes.Repeat([]byte{0x5B}, 400) content := generateTestJPEG(t, 20, 20, color.RGBA{0x5B, 0x5B, 0x5B, 255})
evicted := storeEvictionTestSource(t, cache, testHostCDN, "/evicted.jpg", content) evicted := storeEvictionTestSource(t, cache, testHostCDN, "/evicted.jpg", content)
storeEvictionTestSource(t, cache, testHostCDN, "/kept.jpg", content) storeEvictionTestSource(t, cache, testHostCDN, "/kept.jpg", content)
@@ -74,7 +79,8 @@ func TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed(t *testing.T) {
t.Fatalf("evictSourceBlob() error = %v", err) t.Fatalf("evictSourceBlob() error = %v", err)
} }
if cache.srcContent.Exists(evicted) { _, err = os.Stat(cache.srcContent.keyToPath(string(evicted)))
if err == nil {
t.Error("the evicted source's file is still on disk") t.Error("the evicted source's file is still on disk")
} }
@@ -84,18 +90,12 @@ func TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed(t *testing.T) {
t.Errorf("the evicted source still has %d source_content rows", n) t.Errorf("the evicted source still has %d source_content rows", n)
} }
kept, err := cache.LookupSource(t.Context(), &imagerequest.Request{ reader, _, _, err := cache.Source(t.Context(),
Source: imagerequest.Source{Host: testHostCDN, Path: "/kept.jpg"}, imagerequest.Source{Host: testHostCDN, Path: "/kept.jpg"})
}) t.Logf("Source() of the kept source: error %v", err)
t.Logf("LookupSource() of the kept source = %q, error %v", kept, err)
if err != nil || kept == "" {
t.Fatalf("LookupSource() of the kept source = %q, %v; want it cached", kept, err)
}
reader, _, err := cache.GetSourceContent(kept)
if err != nil { if err != nil {
t.Fatalf("GetSourceContent() of the kept source error = %v", err) t.Fatalf("Source() of the kept source error = %v; want it cached", err)
} }
defer func() { _ = reader.Close() }() defer func() { _ = reader.Close() }()
@@ -111,10 +111,10 @@ func TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed(t *testing.T) {
} }
} }
// TestStoreSource_FailedRowWriteLeavesNoFile drops source_content, so that // TestPutSource_FailedRowWriteLeavesNoFile drops source_content, so that
// StoreSource cannot write the source's row, and checks that the store fails // PutSource cannot write the source's row, and checks that the store fails
// and leaves no file under the source's key. // and leaves no file under the source's key.
func TestStoreSource_FailedRowWriteLeavesNoFile(t *testing.T) { func TestPutSource_FailedRowWriteLeavesNoFile(t *testing.T) {
t.Parallel() t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30) cache, _ := newEvictionTestCache(t, 1<<30)
@@ -127,15 +127,112 @@ func TestStoreSource_FailedRowWriteLeavesNoFile(t *testing.T) {
req := &imagerequest.Request{ req := &imagerequest.Request{
Source: imagerequest.Source{Host: testHostCDN, Path: testPathCat}, Source: imagerequest.Source{Host: testHostCDN, Path: testPathCat},
} }
key, err := cache.StoreSource(t.Context(), req, key, err := cache.PutSource(t.Context(), req.Source,
bytes.NewReader([]byte("source bytes"))) bytes.NewReader([]byte("source bytes")))
t.Logf("StoreSource() = %q, error = %v", key, err) t.Logf("PutSource() = %q, error = %v", key, err)
if err == nil { if err == nil {
t.Error("StoreSource() error = nil, want the failed row write") t.Error("PutSource() error = nil, want the failed row write")
} }
if cache.srcContent.Exists(req.Source.Key()) { _, err = os.Stat(cache.srcContent.keyToPath(string(req.Source.Key())))
if err == nil {
t.Error("the source's file is on disk although its row was not written") t.Error("the source's file is on disk although its row was not written")
} }
} }
// TestSource_FormatFromFirstBytes stores an image in each source format
// under one source URL, in turn. After each store, Source must return that
// image whole, with its format read from the file.
func TestSource_FormatFromFirstBytes(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
processor := imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
})
jpegImage := generateTestJPEG(t, 40, 30, color.RGBA{200, 40, 40, 255})
src := imagerequest.Source{Host: testHostCDN, Path: testPathCat}
for _, format := range []imagerequest.Format{
imagerequest.FormatJPEG, imagerequest.FormatPNG, imagerequest.FormatGIF,
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
} {
result, err := processor.Process(t.Context(), bytes.NewReader(jpegImage),
imagerequest.FormatJPEG, &imagerequest.Transform{Format: format})
if err != nil {
t.Fatalf("making a %s image: %v", format, err)
}
image, err := io.ReadAll(result.Content)
if err != nil {
t.Fatalf("reading the %s image: %v", format, err)
}
_, err = cache.PutSource(t.Context(), src, bytes.NewReader(image))
if err != nil {
t.Fatalf("PutSource(%s) error = %v", format, err)
}
reader, size, got, err := cache.Source(t.Context(), src)
if err != nil {
t.Fatalf("Source() of the %s image error = %v", format, err)
}
served, err := io.ReadAll(reader)
_ = reader.Close()
if err != nil {
t.Fatalf("reading the %s source: %v", format, err)
}
t.Logf("%s: %d bytes read as %s", format, size, got)
if got != format {
t.Errorf("Source() of the %s image format = %q, want %q",
format, got, format)
}
if size != int64(len(image)) || !bytes.Equal(served, image) {
t.Errorf("%s source read as %d bytes, size %d, want the %d bytes stored",
format, len(served), size, len(image))
}
}
}
// TestSource_RefusesFileOfNoSourceFormat checks that Source refuses a
// cached source whose file's first bytes are not those of a source format,
// with an error other than ErrNotFound.
func TestSource_RefusesFileOfNoSourceFormat(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
jpegImage := generateTestJPEG(t, 10, 10, color.RGBA{0, 0, 255, 255})
tests := []struct {
name string
content []byte
wantErr error
}{
{"text", []byte("this is text, not an image"), magic.ErrUnknownFormat},
{"SVG", []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`),
magic.ErrUnknownFormat},
{"JPEG cut to 10 bytes", jpegImage[:10], io.EOF},
{"empty file", nil, io.EOF},
}
for i, tt := range tests {
path := fmt.Sprintf("/refused%d.jpg", i)
storeEvictionTestSource(t, cache, testHostCDN, path, tt.content)
reader, _, format, err := cache.Source(t.Context(),
imagerequest.Source{Host: testHostCDN, Path: path})
t.Logf("%s: Source() error = %v", tt.name, err)
if reader != nil || format != "" || !errors.Is(err, tt.wantErr) {
t.Errorf("%s: Source() = reader %v, format %q, error %v; "+
"want no reader and %v", tt.name, reader, format, err, tt.wantErr)
}
}
}
internal/imgcache/storage.go
-326
View File
@@ -1,326 +0,0 @@
package imgcache
import (
"errors"
"fmt"
"io"
"os"
"path/filepath"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic"
)
// Storage constants.
const (
// StorageDirPerm is the permission mode for storage directories.
StorageDirPerm = 0750
// StorageFilePerm is the permission mode for storage files.
StorageFilePerm = 0600
// MinHashLength is the minimum hash length for path splitting.
MinHashLength = 4
)
// Storage errors.
var (
ErrNotFound = errors.New("content not found")
)
// ContentStorage handles source image file storage, one file per source
// URL. Files are stored at: <basedir>/<ab>/<cd>/<source key>
type ContentStorage struct {
baseDir string
}
// NewContentStorage creates a new content storage at the given base directory.
func NewContentStorage(baseDir string) (*ContentStorage, error) {
err := os.MkdirAll(baseDir, StorageDirPerm)
if err != nil {
return nil, fmt.Errorf("failed to create storage directory: %w", err)
}
return &ContentStorage{baseDir: baseDir}, nil
}
// Store writes data with writeFileSynced to the path derived from key,
// replacing any file already there, and returns its size.
func (s *ContentStorage) Store(
key imagerequest.SourceKey, data []byte,
) (int64, error) {
path := s.keyToPath(key)
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
if err != nil {
return 0, fmt.Errorf("failed to create directory: %w", err)
}
err = writeFileSynced(path, data)
if err != nil {
return 0, err
}
return int64(len(data)), nil
}
// LoadWithSize returns a reader and file size for the content with the given key.
func (s *ContentStorage) LoadWithSize(
key imagerequest.SourceKey,
) (io.ReadCloser, int64, error) {
path := s.keyToPath(key)
f, err := os.Open(path) //nolint:gosec // path derived from source key
if err != nil {
if os.IsNotExist(err) {
return nil, 0, ErrNotFound
}
return nil, 0, fmt.Errorf("failed to open content: %w", err)
}
stat, err := f.Stat()
if err != nil {
_ = f.Close()
return nil, 0, fmt.Errorf("failed to stat content: %w", err)
}
return f, stat.Size(), nil
}
// Delete removes content with the given key.
func (s *ContentStorage) Delete(key imagerequest.SourceKey) error {
path := s.keyToPath(key)
err := os.Remove(path)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("failed to delete content: %w", err)
}
return nil
}
// Exists checks if content with the given key exists.
func (s *ContentStorage) Exists(key imagerequest.SourceKey) bool {
path := s.keyToPath(key)
_, err := os.Stat(path)
return err == nil
}
// writeFileSynced writes data to path through a temp file in the same
// directory: the temp file is synced to disk, renamed to path, and the
// directory synced after, so that neither a concurrent reader nor a
// crash finds a partial file under path. When the directory sync fails,
// the file is removed from path, so that a failed write leaves no file
// there.
func writeFileSynced(path string, data []byte) error {
dir := filepath.Dir(path)
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
if err != nil {
return fmt.Errorf("failed to create temp file: %w", err)
}
tmpPath := tmpFile.Name()
_, err = tmpFile.Write(data)
if err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to write temp file: %w", err)
}
err = tmpFile.Sync()
if err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to sync temp file: %w", err)
}
err = tmpFile.Close()
if err != nil {
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to close temp file: %w", err)
}
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
if err != nil {
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to rename temp file: %w", err)
}
err = syncDirectory(dir)
if err != nil {
_ = os.Remove(path)
return err
}
return nil
}
// syncDirectory syncs the directory dir to disk, so that a file just
// renamed into it is still there after a crash.
func syncDirectory(dir string) error {
d, err := os.Open(filepath.Clean(dir))
if err != nil {
return fmt.Errorf("failed to open directory: %w", err)
}
err = d.Sync()
_ = d.Close()
if err != nil {
return fmt.Errorf("failed to sync directory: %w", err)
}
return nil
}
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
func (s *ContentStorage) keyToPath(key imagerequest.SourceKey) string {
k := string(key)
if len(k) < MinHashLength {
return filepath.Clean(filepath.Join(s.baseDir, k))
}
return filepath.Clean(filepath.Join(s.baseDir, k[0:2], k[2:4], k))
}
// VariantStorage handles key-based file storage for processed image variants.
// Files are stored at: <basedir>/<ab>/<cd>/<cache_key>
type VariantStorage struct {
baseDir string
}
// errNotAnOutputFormat is returned for a variant file whose first bytes are
// those of a format pixa reads but never writes, such as SVG.
var errNotAnOutputFormat = errors.New("variant file is not in an output format")
// NewVariantStorage creates a new variant storage at the given base directory.
func NewVariantStorage(baseDir string) (*VariantStorage, error) {
err := os.MkdirAll(baseDir, StorageDirPerm)
if err != nil {
return nil, fmt.Errorf("failed to create variant storage directory: %w", err)
}
return &VariantStorage{baseDir: baseDir}, nil
}
// Store writes content to storage at the given key with writeFileSynced.
func (s *VariantStorage) Store(
key imagerequest.VariantKey, r io.Reader,
) (int64, error) {
data, err := io.ReadAll(r)
if err != nil {
return 0, fmt.Errorf("failed to read content: %w", err)
}
path := s.keyToPath(key)
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
if err != nil {
return 0, fmt.Errorf("failed to create directory: %w", err)
}
err = writeFileSynced(path, data)
if err != nil {
return 0, err
}
return int64(len(data)), nil
}
// LoadWithContentType opens the content at the given key and returns a
// reader, the file size and the content type, which is read from the
// file's first bytes. A missing file is ErrNotFound. A file whose first
// bytes are not those of an output format, an empty one included, is
// closed again and the reason returned.
func (s *VariantStorage) LoadWithContentType(
key imagerequest.VariantKey,
) (io.ReadCloser, int64, string, error) {
path := s.keyToPath(key)
f, err := os.Open(path) //nolint:gosec // path derived from cache key
if err != nil {
if os.IsNotExist(err) {
return nil, 0, "", ErrNotFound
}
return nil, 0, "", fmt.Errorf("failed to open content: %w", err)
}
stat, err := f.Stat()
if err != nil {
_ = f.Close()
return nil, 0, "", fmt.Errorf("failed to stat content: %w", err)
}
contentType, err := outputContentType(f)
if err != nil {
_ = f.Close()
return nil, 0, "", err
}
return f, stat.Size(), contentType, nil
}
// outputContentType reads the first bytes of content, without moving its
// offset, and returns the content type of the output format they show.
func outputContentType(content io.ReaderAt) (string, error) {
firstBytes := make([]byte, magic.MinMagicBytes)
_, err := content.ReadAt(firstBytes, 0)
if err != nil {
return "", fmt.Errorf("failed to read the first bytes: %w", err)
}
detected, err := magic.DetectFormat(firstBytes)
if err != nil {
return "", err
}
_, isOutputFormat := magic.MIMEToImageFormat(string(detected))
if !isOutputFormat {
return "", fmt.Errorf("%w: %s", errNotAnOutputFormat, detected)
}
return string(detected), nil
}
// Exists checks if content exists at the given key.
func (s *VariantStorage) Exists(key imagerequest.VariantKey) bool {
path := s.keyToPath(key)
_, err := os.Stat(path)
return err == nil
}
// Delete removes the content at the given key. A missing file is not an
// error.
func (s *VariantStorage) Delete(key imagerequest.VariantKey) error {
path := s.keyToPath(key)
err := os.Remove(path)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("failed to delete content: %w", err)
}
return nil
}
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
func (s *VariantStorage) keyToPath(key imagerequest.VariantKey) string {
k := string(key)
if len(k) < MinHashLength {
return filepath.Join(s.baseDir, k)
}
return filepath.Join(s.baseDir, k[0:2], k[2:4], k)
}
internal/imgcache/storage_internal_test.go
-156
View File
@@ -1,156 +0,0 @@
package imgcache
import (
"bytes"
"errors"
"io"
"os"
"path/filepath"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
func TestContentStorage_StoreAndLoadWithSize(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
content := []byte("hello world")
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
size, err := storage.Store(key, content)
if err != nil {
t.Fatalf("Store() error = %v", err)
}
if size != int64(len(content)) {
t.Errorf("Store() size = %d, want %d", size, len(content))
}
// Verify file exists at expected path
keyStr := string(key)
expectedPath := filepath.Join(tmpDir, keyStr[0:2], keyStr[2:4], keyStr)
_, err = os.Stat(expectedPath)
if err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
// Load and verify content
r, _, err := storage.LoadWithSize(key)
if err != nil {
t.Fatalf("LoadWithSize() error = %v", err)
}
defer func() { _ = r.Close() }()
loaded, err := io.ReadAll(r)
if err != nil {
t.Fatalf("ReadAll() error = %v", err)
}
if !bytes.Equal(loaded, content) {
t.Errorf("LoadWithSize() content = %q, want %q", loaded, content)
}
}
func TestContentStorage_LoadWithSizeNotFound(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
_, _, err = storage.LoadWithSize(imagerequest.SourceKey("nonexistent"))
if !errors.Is(err, ErrNotFound) {
t.Errorf("LoadWithSize() error = %v, want ErrNotFound", err)
}
}
func TestContentStorage_Delete(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
content := []byte("to be deleted")
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
_, err = storage.Store(key, content)
if err != nil {
t.Fatalf("Store() error = %v", err)
}
if !storage.Exists(key) {
t.Error("Exists() = false, want true")
}
err = storage.Delete(key)
if err != nil {
t.Fatalf("Delete() error = %v", err)
}
if storage.Exists(key) {
t.Error("Exists() = true after delete, want false")
}
}
func TestContentStorage_DeleteNonexistent(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
// Should not error
err = storage.Delete(imagerequest.SourceKey("nonexistent"))
if err != nil {
t.Errorf("Delete() error = %v, want nil", err)
}
}
func TestContentStorage_KeyToPath(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
// Test by storing and verifying the resulting path structure
content := []byte("test content for path verification")
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
_, err = storage.Store(key, content)
if err != nil {
t.Fatalf("Store() error = %v", err)
}
keyStr := string(key)
expectedPath := filepath.Join(tmpDir, keyStr[0:2], keyStr[2:4], keyStr)
_, err = os.Stat(expectedPath)
if err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
}
internal/imgcache/testdata/red.avif
Vendored
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 281 B

internal/imgcache/testutil_internal_test.go
+3 -183
View File
@@ -2,98 +2,19 @@ package imgcache
import ( import (
"bytes" "bytes"
"context"
"database/sql"
"image" "image"
"image/color" "image/color"
"image/gif"
"image/jpeg" "image/jpeg"
"image/png"
"io/fs"
"testing" "testing"
"testing/fstest"
"time"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
) )
// Shared test data literals, extracted as constants for goconst. // Shared test data literals, extracted as constants for goconst.
const ( const (
testHostCDN = "cdn.example.com" testHostCDN = "cdn.example.com"
testHostExample = "example.com" testHostExample = "example.com"
testPathCat = "/photos/cat.jpg" testPathCat = "/photos/cat.jpg"
testContentTypeJPEG = "image/jpeg"
) )
// TestFixtures contains paths to test files in the mock filesystem.
type TestFixtures struct {
// Valid image files
GoodHostJPEG string // allowlisted host, valid JPEG
GoodHostPNG string // allowlisted host, valid PNG
GoodHostGIF string // allowlisted host, valid GIF
OtherHostJPEG string // non-allowlisted host, valid JPEG
OtherHostPNG string // non-allowlisted host, valid PNG
// Invalid/edge case files
InvalidFile string // file with wrong magic bytes
EmptyFile string // zero-byte file
TextFile string // text file masquerading as image
// Hostnames
GoodHost string // allowlisted hostname
OtherHost string // non-allowlisted hostname
}
// DefaultFixtures returns the standard test fixture paths.
func DefaultFixtures() *TestFixtures {
return &TestFixtures{
GoodHostJPEG: "goodhost.example.com/images/photo.jpg",
GoodHostPNG: "goodhost.example.com/images/logo.png",
GoodHostGIF: "goodhost.example.com/images/animation.gif",
OtherHostJPEG: "otherhost.example.com/uploads/image.jpg",
OtherHostPNG: "otherhost.example.com/uploads/icon.png",
InvalidFile: "goodhost.example.com/images/fake.jpg",
EmptyFile: "goodhost.example.com/images/empty.jpg",
TextFile: "goodhost.example.com/images/text.png",
GoodHost: "goodhost.example.com",
OtherHost: "otherhost.example.com",
}
}
// NewTestFS creates a mock filesystem with test images.
func NewTestFS(t *testing.T) (fs.FS, *TestFixtures) {
t.Helper()
fixtures := DefaultFixtures()
// Generate test images
jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
pngData := generateTestPNG(t, 100, 100, color.RGBA{0, 255, 0, 255})
gifData := generateTestGIF(t, 100, 100, color.RGBA{0, 0, 255, 255})
// Create the mock filesystem
mockFS := fstest.MapFS{
// Good host files
fixtures.GoodHostJPEG: &fstest.MapFile{Data: jpegData},
fixtures.GoodHostPNG: &fstest.MapFile{Data: pngData},
fixtures.GoodHostGIF: &fstest.MapFile{Data: gifData},
// Other host files
fixtures.OtherHostJPEG: &fstest.MapFile{Data: jpegData},
fixtures.OtherHostPNG: &fstest.MapFile{Data: pngData},
// Invalid files
fixtures.InvalidFile: &fstest.MapFile{Data: []byte("not a real image file content")},
fixtures.EmptyFile: &fstest.MapFile{Data: []byte{}},
fixtures.TextFile: &fstest.MapFile{Data: []byte("This is just text, not a PNG")},
}
return mockFS, fixtures
}
// generateTestJPEG creates a minimal valid JPEG image. // generateTestJPEG creates a minimal valid JPEG image.
func generateTestJPEG(t *testing.T, width, height int, c color.Color) []byte { func generateTestJPEG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper() t.Helper()
@@ -114,104 +35,3 @@ func generateTestJPEG(t *testing.T, width, height int, c color.Color) []byte {
return buf.Bytes() return buf.Bytes()
} }
// generateTestPNG creates a minimal valid PNG image.
func generateTestPNG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
err := png.Encode(&buf, img)
if err != nil {
t.Fatalf("failed to encode test PNG: %v", err)
}
return buf.Bytes()
}
// generateTestGIF creates a minimal valid GIF image.
func generateTestGIF(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewPaletted(
image.Rect(0, 0, width, height),
[]color.Color{c, color.White},
)
for y := range height {
for x := range width {
img.SetColorIndex(x, y, 0)
}
}
var buf bytes.Buffer
err := gif.Encode(&buf, img, nil)
if err != nil {
t.Fatalf("failed to encode test GIF: %v", err)
}
return buf.Bytes()
}
// SetupTestService creates a Service with mock fetcher for testing.
func SetupTestService(t *testing.T) (*Service, *TestFixtures) {
t.Helper()
mockFS, fixtures := NewTestFS(t)
// Create temp directory for cache
tmpDir := t.TempDir()
// Create in-memory database
db := setupServiceTestDB(t)
cache, err := NewCache(db, CacheConfig{
StateDir: tmpDir,
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
if err != nil {
t.Fatalf("failed to create cache: %v", err)
}
svc, err := NewService(&ServiceConfig{
Cache: cache,
Fetcher: httpfetcher.NewMock(mockFS),
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
})
if err != nil {
t.Fatalf("failed to create service: %v", err)
}
return svc, fixtures
}
// setupServiceTestDB creates an in-memory SQLite database for testing
// using the production schema.
func setupServiceTestDB(t *testing.T) *sql.DB {
t.Helper()
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatalf("failed to open test db: %v", err)
}
// Use the real production schema via migrations
err = database.ApplyMigrations(context.Background(), db, nil)
if err != nil {
t.Fatalf("failed to apply migrations: %v", err)
}
return db
}
internal/magic/jpegxl_internal_test.go
+11 -16
View File
@@ -1,12 +1,17 @@
package magic package magic
import "testing" import (
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// testMIMEJXL is the MIME type of JPEG XL. // testMIMEJXL is the MIME type of JPEG XL.
const testMIMEJXL = "image/jxl" const testMIMEJXL = "image/jxl"
// TestDetectFormatJPEGXL verifies that both JPEG XL signatures, that of a // TestDetectFormatJPEGXL verifies that both JPEG XL signatures, that of a
// bare codestream and that of the container, are detected as image/jxl. // bare codestream and that of the container, are detected as JPEG XL and
// match the MIME type image/jxl.
func TestDetectFormatJPEGXL(t *testing.T) { func TestDetectFormatJPEGXL(t *testing.T) {
t.Parallel() t.Parallel()
@@ -25,25 +30,15 @@ func TestDetectFormatJPEGXL(t *testing.T) {
t.Parallel() t.Parallel()
got, err := DetectFormat(tt.data) got, err := DetectFormat(tt.data)
if err != nil || got != MIMETypeJXL { if err != nil || got != imagerequest.FormatJXL {
t.Errorf("DetectFormat() = %q, %v, want %q", got, err, testMIMEJXL) t.Errorf("DetectFormat() = %q, %v, want %q",
got, err, imagerequest.FormatJXL)
} }
err = ValidateMagicBytes(tt.data, testMIMEJXL) _, err = ValidateMagicBytes(tt.data, testMIMEJXL)
if err != nil { if err != nil {
t.Errorf("ValidateMagicBytes(%q) error = %v", testMIMEJXL, err) t.Errorf("ValidateMagicBytes(%q) error = %v", testMIMEJXL, err)
} }
}) })
} }
} }
// TestJPEGXLMIMEType verifies that image/jxl maps to the format jxl.
func TestJPEGXLMIMEType(t *testing.T) {
t.Parallel()
format, ok := MIMEToImageFormat(testMIMEJXL)
if format != FormatJXL || !ok {
t.Errorf("MIMEToImageFormat(%q) = %q, %v, want %q, true",
testMIMEJXL, format, ok, FormatJXL)
}
}
internal/magic/magic.go
+21 -98
View File
@@ -6,6 +6,8 @@ import (
"bytes" "bytes"
"errors" "errors"
"strings" "strings"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// Magic byte errors. // Magic byte errors.
@@ -15,34 +17,6 @@ var (
ErrNotEnoughData = errors.New("not enough data to detect format") ErrNotEnoughData = errors.New("not enough data to detect format")
) )
// MIMEType represents a supported MIME type for input images.
type MIMEType string
// Supported input MIME types.
const (
MIMETypeJPEG = MIMEType("image/jpeg")
MIMETypePNG = MIMEType("image/png")
MIMETypeWebP = MIMEType("image/webp")
MIMETypeGIF = MIMEType("image/gif")
MIMETypeAVIF = MIMEType("image/avif")
MIMETypeJXL = MIMEType("image/jxl")
MIMETypeSVG = MIMEType("image/svg+xml")
)
// ImageFormat represents supported output image formats.
// This mirrors the type in imgcache to avoid circular imports.
type ImageFormat string
// Supported image output formats.
const (
FormatJPEG ImageFormat = "jpeg"
FormatPNG ImageFormat = "png"
FormatWebP ImageFormat = "webp"
FormatAVIF ImageFormat = "avif"
FormatJXL ImageFormat = "jxl"
FormatGIF ImageFormat = "gif"
)
// MinMagicBytes is the minimum number of bytes needed to detect format. // MinMagicBytes is the minimum number of bytes needed to detect format.
const MinMagicBytes = 12 const MinMagicBytes = 12
@@ -78,32 +52,32 @@ var (
avisBrand = []byte{0x61, 0x76, 0x69, 0x73} // "avis" (AVIF sequence) avisBrand = []byte{0x61, 0x76, 0x69, 0x73} // "avis" (AVIF sequence)
) )
// DetectFormat detects the image format from magic bytes. // DetectFormat detects the image format from magic bytes. Data in none of
// Returns the MIME type and nil error on success. // the formats pixa reads, SVG included, is ErrUnknownFormat.
func DetectFormat(data []byte) (MIMEType, error) { func DetectFormat(data []byte) (imagerequest.Format, error) {
if len(data) < MinMagicBytes { if len(data) < MinMagicBytes {
return "", ErrNotEnoughData return "", ErrNotEnoughData
} }
// Check JPEG (FFD8FF) // Check JPEG (FFD8FF)
if bytes.HasPrefix(data, magicJPEG) { if bytes.HasPrefix(data, magicJPEG) {
return MIMETypeJPEG, nil return imagerequest.FormatJPEG, nil
} }
// Check PNG (89504E47 0D0A1A0A) // Check PNG (89504E47 0D0A1A0A)
if bytes.HasPrefix(data, magicPNG) { if bytes.HasPrefix(data, magicPNG) {
return MIMETypePNG, nil return imagerequest.FormatPNG, nil
} }
// Check GIF (GIF87a or GIF89a) // Check GIF (GIF87a or GIF89a)
if bytes.HasPrefix(data, magicGIF) { if bytes.HasPrefix(data, magicGIF) {
return MIMETypeGIF, nil return imagerequest.FormatGIF, nil
} }
// Check WebP (RIFF....WEBP) // Check WebP (RIFF....WEBP)
if bytes.HasPrefix(data, magicWebP) && len(data) >= 12 { if bytes.HasPrefix(data, magicWebP) && len(data) >= 12 {
if bytes.Equal(data[8:12], webpIdent) { if bytes.Equal(data[8:12], webpIdent) {
return MIMETypeWebP, nil return imagerequest.FormatWebP, nil
} }
} }
@@ -112,66 +86,39 @@ func DetectFormat(data []byte) (MIMEType, error) {
if len(data) >= 12 && bytes.Equal(data[4:8], magicFtyp) { if len(data) >= 12 && bytes.Equal(data[4:8], magicFtyp) {
brand := data[8:12] brand := data[8:12]
if bytes.Equal(brand, avifBrand) || bytes.Equal(brand, avisBrand) { if bytes.Equal(brand, avifBrand) || bytes.Equal(brand, avisBrand) {
return MIMETypeAVIF, nil return imagerequest.FormatAVIF, nil
} }
} }
// Check JPEG XL (FF0A, or the 12-byte container signature) // Check JPEG XL (FF0A, or the 12-byte container signature)
if bytes.HasPrefix(data, magicJXLCodestream) || if bytes.HasPrefix(data, magicJXLCodestream) ||
bytes.HasPrefix(data, magicJXLContainer) { bytes.HasPrefix(data, magicJXLContainer) {
return MIMETypeJXL, nil return imagerequest.FormatJXL, nil
}
// Check SVG - look for XML declaration or SVG tag
if detectSVG(data) {
return MIMETypeSVG, nil
} }
return "", ErrUnknownFormat return "", ErrUnknownFormat
} }
// detectSVG checks if data appears to be SVG content. // ValidateMagicBytes detects the format of data from its magic bytes, and
func detectSVG(data []byte) bool { // returns it when it is the format the declared MIME type names. Data in
// Skip BOM if present // another format is ErrMagicByteMismatch.
content := skipBOM(data) func ValidateMagicBytes(
data []byte, declaredType string,
// Convert to string for easier pattern matching ) (imagerequest.Format, error) {
s := strings.ToLower(string(content))
// Skip leading whitespace
s = strings.TrimSpace(s)
// Check for XML declaration or SVG element
return strings.HasPrefix(s, "<?xml") ||
strings.HasPrefix(s, "<svg") ||
strings.HasPrefix(s, "<!doctype svg")
}
// skipBOM removes UTF-8 BOM if present.
func skipBOM(data []byte) []byte {
if len(data) >= 3 && data[0] == 0xEF && data[1] == 0xBB && data[2] == 0xBF {
return data[3:]
}
return data
}
// ValidateMagicBytes validates that the content matches the declared MIME type.
func ValidateMagicBytes(data []byte, declaredType string) error {
detected, err := DetectFormat(data) detected, err := DetectFormat(data)
if err != nil { if err != nil {
return err return "", err
} }
// Normalize the declared type (remove parameters like charset) // Normalize the declared type (remove parameters like charset)
normalizedDeclared := normalizeMIMEType(declaredType) normalizedDeclared := normalizeMIMEType(declaredType)
// Check if they match // Check if they match
if string(detected) != normalizedDeclared { if detected.MIMEType() != normalizedDeclared {
return ErrMagicByteMismatch return "", ErrMagicByteMismatch
} }
return nil return detected, nil
} }
// normalizeMIMEType extracts just the media type, removing parameters. // normalizeMIMEType extracts just the media type, removing parameters.
@@ -183,27 +130,3 @@ func normalizeMIMEType(mimeType string) string {
return strings.TrimSpace(strings.ToLower(mimeType)) return strings.TrimSpace(strings.ToLower(mimeType))
} }
// MIMEToImageFormat converts a MIME type to an ImageFormat.
func MIMEToImageFormat(mimeType string) (ImageFormat, bool) {
normalized := normalizeMIMEType(mimeType)
switch MIMEType(normalized) {
case MIMETypeJPEG:
return FormatJPEG, true
case MIMETypePNG:
return FormatPNG, true
case MIMETypeWebP:
return FormatWebP, true
case MIMETypeGIF:
return FormatGIF, true
case MIMETypeAVIF:
return FormatAVIF, true
case MIMETypeJXL:
return FormatJXL, true
case MIMETypeSVG:
// SVG has no corresponding output format.
return "", false
default:
return "", false
}
}
internal/magic/magic_internal_test.go
+24 -171
View File
@@ -1,9 +1,10 @@
package magic package magic
import ( import (
"bytes"
"errors" "errors"
"testing" "testing"
"sneak.berlin/go/pixa/internal/imagerequest"
) )
// Shared test fixture strings. // Shared test fixture strings.
@@ -12,9 +13,6 @@ const (
testMIMEJPEG = "image/jpeg" testMIMEJPEG = "image/jpeg"
testMIMEJPEGParams = "image/jpeg; charset=utf-8" testMIMEJPEGParams = "image/jpeg; charset=utf-8"
testMIMEPNG = "image/png" testMIMEPNG = "image/png"
testMIMEWebP = "image/webp"
testMIMEGIF = "image/gif"
testMIMEAVIF = "image/avif"
) )
// pad appends zero bytes so data is comfortably above MinMagicBytes. // pad appends zero bytes so data is comfortably above MinMagicBytes.
@@ -36,37 +34,22 @@ func TestDetectFormat(t *testing.T) {
avis := pad(0x00, 0x00, 0x00, 0x1C, 0x66, 0x74, 0x79, 0x70, 0x61, 0x76, 0x69, 0x73) avis := pad(0x00, 0x00, 0x00, 0x1C, 0x66, 0x74, 0x79, 0x70, 0x61, 0x76, 0x69, 0x73)
tests := []struct { tests := []struct {
name string name string
data []byte data []byte
wantMIME MIMEType wantFormat imagerequest.Format
wantErr error wantErr error
}{ }{
{name: "JPEG", data: jpeg, wantMIME: MIMETypeJPEG}, {name: "JPEG", data: jpeg, wantFormat: imagerequest.FormatJPEG},
{name: "PNG", data: png, wantMIME: MIMETypePNG}, {name: "PNG", data: png, wantFormat: imagerequest.FormatPNG},
{name: "GIF87a", data: gif87a, wantMIME: MIMETypeGIF}, {name: "GIF87a", data: gif87a, wantFormat: imagerequest.FormatGIF},
{name: "GIF89a", data: gif89a, wantMIME: MIMETypeGIF}, {name: "GIF89a", data: gif89a, wantFormat: imagerequest.FormatGIF},
{name: "WebP", data: webp, wantMIME: MIMETypeWebP}, {name: "WebP", data: webp, wantFormat: imagerequest.FormatWebP},
{name: "AVIF", data: avif, wantMIME: MIMETypeAVIF}, {name: "AVIF", data: avif, wantFormat: imagerequest.FormatAVIF},
{name: "AVIF sequence", data: avis, wantMIME: MIMETypeAVIF}, {name: "AVIF sequence", data: avis, wantFormat: imagerequest.FormatAVIF},
{ {
name: "SVG with XML declaration", name: "SVG",
data: []byte(`<?xml version="1.0"?><svg></svg>`), data: []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`),
wantMIME: MIMETypeSVG, wantErr: ErrUnknownFormat,
},
{
name: "SVG without declaration",
data: []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`),
wantMIME: MIMETypeSVG,
},
{
name: "SVG with whitespace",
data: []byte(` <?xml version="1.0"?><svg></svg>`),
wantMIME: MIMETypeSVG,
},
{
name: "SVG with BOM",
data: append([]byte{0xEF, 0xBB, 0xBF}, []byte(`<svg></svg>`)...),
wantMIME: MIMETypeSVG,
}, },
{ {
name: "unknown format", name: "unknown format",
@@ -88,8 +71,8 @@ func TestDetectFormat(t *testing.T) {
return return
} }
if got != tt.wantMIME { if got != tt.wantFormat {
t.Errorf("DetectFormat() = %v, want %v", got, tt.wantMIME) t.Errorf("DetectFormat() = %v, want %v", got, tt.wantFormat)
} }
}) })
} }
@@ -105,24 +88,28 @@ func TestValidateMagicBytes(t *testing.T) {
name string name string
data []byte data []byte
declaredType string declaredType string
wantFormat imagerequest.Format
wantErr error wantErr error
}{ }{
{ {
name: "matching JPEG", name: "matching JPEG",
data: jpegData, data: jpegData,
declaredType: testMIMEJPEG, declaredType: testMIMEJPEG,
wantFormat: imagerequest.FormatJPEG,
wantErr: nil, wantErr: nil,
}, },
{ {
name: "matching JPEG with params", name: "matching JPEG with params",
data: jpegData, data: jpegData,
declaredType: testMIMEJPEGParams, declaredType: testMIMEJPEGParams,
wantFormat: imagerequest.FormatJPEG,
wantErr: nil, wantErr: nil,
}, },
{ {
name: "matching PNG", name: "matching PNG",
data: pngData, data: pngData,
declaredType: testMIMEPNG, declaredType: testMIMEPNG,
wantFormat: imagerequest.FormatPNG,
wantErr: nil, wantErr: nil,
}, },
{ {
@@ -143,45 +130,14 @@ func TestValidateMagicBytes(t *testing.T) {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
err := ValidateMagicBytes(tt.data, tt.declaredType) got, err := ValidateMagicBytes(tt.data, tt.declaredType)
if !errors.Is(err, tt.wantErr) { if !errors.Is(err, tt.wantErr) {
t.Errorf("ValidateMagicBytes() error = %v, wantErr %v", err, tt.wantErr) t.Errorf("ValidateMagicBytes() error = %v, wantErr %v", err, tt.wantErr)
} }
})
}
}
func TestMIMEToImageFormat(t *testing.T) {
t.Parallel()
tests := []struct {
mimeType string
wantFormat ImageFormat
wantOk bool
}{
{testMIMEJPEG, FormatJPEG, true},
{testMIMEPNG, FormatPNG, true},
{testMIMEWebP, FormatWebP, true},
{testMIMEGIF, FormatGIF, true},
{testMIMEAVIF, FormatAVIF, true},
{"image/svg+xml", "", false}, // SVG doesn't convert to ImageFormat
{"image/tiff", "", false},
{"text/plain", "", false},
}
for _, tt := range tests {
t.Run(tt.mimeType, func(t *testing.T) {
t.Parallel()
got, ok := MIMEToImageFormat(tt.mimeType)
if ok != tt.wantOk {
t.Errorf("MIMEToImageFormat(%q) ok = %v, want %v", tt.mimeType, ok, tt.wantOk)
}
if got != tt.wantFormat { if got != tt.wantFormat {
t.Errorf("MIMEToImageFormat(%q) = %v, want %v", tt.mimeType, got, tt.wantFormat) t.Errorf("ValidateMagicBytes() = %q, want %q", got, tt.wantFormat)
} }
}) })
} }
@@ -214,109 +170,6 @@ func TestNormalizeMIMEType(t *testing.T) {
} }
} }
func TestDetectSVG(t *testing.T) {
t.Parallel()
tests := []struct {
name string
data string
want bool
}{
{"xml declaration", `<?xml version="1.0"?><svg></svg>`, true},
{"svg element", `<svg xmlns="http://www.w3.org/2000/svg"></svg>`, true},
{
"doctype",
`<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" ` +
`"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">`,
true,
},
{"with whitespace", `
<?xml version="1.0"?><svg></svg>`, true},
{"uppercase", `<SVG></SVG>`, true},
{"not svg", `<html></html>`, false},
{"random text", `hello world`, false},
{testNameEmpty, ``, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got := detectSVG([]byte(tt.data))
if got != tt.want {
t.Errorf("detectSVG(%q) = %v, want %v", tt.data, got, tt.want)
}
})
}
}
func TestSkipBOM(t *testing.T) {
t.Parallel()
tests := []struct {
name string
data []byte
want []byte
}{
{"with BOM", []byte{0xEF, 0xBB, 0xBF, 'h', 'e', 'l', 'l', 'o'}, []byte("hello")},
{"without BOM", []byte("hello"), []byte("hello")},
{testNameEmpty, []byte{}, []byte{}},
{"only BOM", []byte{0xEF, 0xBB, 0xBF}, []byte{}},
{"partial BOM", []byte{0xEF, 0xBB}, []byte{0xEF, 0xBB}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got := skipBOM(tt.data)
if !bytes.Equal(got, tt.want) {
t.Errorf("skipBOM() = %v, want %v", got, tt.want)
}
})
}
}
func TestRealWorldSVGPatterns(t *testing.T) {
t.Parallel()
// Test various real-world SVG patterns
svgPatterns := []string{
`<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100">
<circle cx="50" cy="50" r="40"/>
</svg>`,
`<svg viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M12 2L2 7l10 5 10-5-10-5z"/>
</svg>`,
`<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" ` +
`"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">` + `
<svg xmlns="http://www.w3.org/2000/svg">
</svg>`,
}
for i, pattern := range svgPatterns {
data := []byte(pattern)
if len(data) < MinMagicBytes {
// Pad short SVGs for detection
data = append(data, make([]byte, MinMagicBytes-len(data))...)
}
got, err := DetectFormat(data)
if err != nil {
t.Errorf("Pattern %d: DetectFormat() error = %v", i, err)
continue
}
if got != MIMETypeSVG {
t.Errorf("Pattern %d: DetectFormat() = %v, want %v", i, got, MIMETypeSVG)
}
}
}
func TestDetectFormatRIFFNotWebP(t *testing.T) { func TestDetectFormatRIFFNotWebP(t *testing.T) {
t.Parallel() t.Parallel()
internal/providers/providers.go
+32 -20
View File
@@ -1,5 +1,6 @@
// Package providers builds from the config the parts pixad serves images // Package providers builds from the config the parts pixad serves images
// and logins with, and provides them to the fx graph: // and logins with, and provides them to the fx graph:
// - the keys derived from the signing key
// - the image cache, with its default limit // - the image cache, with its default limit
// - the fetcher // - the fetcher
// - the processor // - the processor
@@ -25,16 +26,19 @@ import (
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor" "sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/logger" "sneak.berlin/go/pixa/internal/logger"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
// Constructors returns the constructors of the parts this package builds, // Constructors returns the constructors of the parts this package builds,
// for fx.Provide. The fx graph must also provide the prometheus.Registerer // for fx.Provide. The fx graph must also provide the prometheus.Registerer
// the cache registers its counters on. // the cache and the image service register their counters on.
func Constructors() []any { func Constructors() []any {
return []any{ return []any{
newKeys,
newCache, newCache,
newFetcherConfig, newFetcherConfig,
httpfetcher.New, httpfetcher.New,
@@ -45,6 +49,12 @@ func Constructors() []any {
} }
} }
// newKeys derives the keys of seal.Keys from the signing key. fx calls it
// once, at startup.
func newKeys(cfg *config.Config) (seal.Keys, error) {
return seal.DeriveKeys(cfg.SigningKey)
}
// cacheParams defines dependencies for newCache. // cacheParams defines dependencies for newCache.
type cacheParams struct { type cacheParams struct {
fx.In fx.In
@@ -70,7 +80,7 @@ func newCache(lc fx.Lifecycle, params cacheParams) (*imgcache.Cache, error) {
return nil, err return nil, err
} }
cache, err := imgcache.NewCache(db, imgcache.CacheConfig{ cache, err := imgcache.NewCache(db, imgcache.Config{
StateDir: cfg.StateDir, StateDir: cfg.StateDir,
NegativeTTL: imgcache.DefaultNegativeTTL, NegativeTTL: imgcache.DefaultNegativeTTL,
MaxBytes: maxBytes, MaxBytes: maxBytes,
@@ -204,47 +214,49 @@ func newProcessor(cfg *config.Config) *imageprocessor.ImageProcessor {
type imageServiceParams struct { type imageServiceParams struct {
fx.In fx.In
Config *config.Config Config *config.Config
Cache *imgcache.Cache Cache *imgcache.Cache
Fetcher *httpfetcher.HTTPFetcher Fetcher *httpfetcher.HTTPFetcher
Processor *imageprocessor.ImageProcessor Processor *imageprocessor.ImageProcessor
Logger *logger.Logger Logger *logger.Logger
MetricsRegistry prometheus.Registerer
} }
// newImageService builds the image service from the cache, the fetcher // newImageService builds the image service from the cache, the fetcher
// and the processor, with the largest upstream response the config allows. // and the processor, with the largest upstream response the config allows.
func newImageService( func newImageService(
_ fx.Lifecycle, params imageServiceParams, _ fx.Lifecycle, params imageServiceParams,
) (*imgcache.Service, error) { ) (*imageproxy.Proxy, error) {
return imgcache.NewService(&imgcache.ServiceConfig{ return imageproxy.New(&imageproxy.Config{
Cache: params.Cache, Cache: params.Cache,
Fetcher: params.Fetcher, Fetcher: params.Fetcher,
Processor: params.Processor, Processor: params.Processor,
MaxResponseSize: params.Config.UpstreamMaxResponseSize, MaxResponseSize: params.Config.UpstreamMaxResponseSize,
Logger: params.Logger.Get(), Logger: params.Logger.Get(),
}) }, params.MetricsRegistry)
} }
// sessionManagerParams defines dependencies for newSessionManager. // sessionManagerParams defines dependencies for newSessionManager.
type sessionManagerParams struct { type sessionManagerParams struct {
fx.In fx.In
Config *config.Config Keys seal.Keys
Database *database.Database Database *database.Database
} }
// newSessionManager builds the session manager. Session cookies are // newSessionManager builds the session manager with its two keys. Session
// always Secure, HttpOnly and SameSite=Strict. The random value that each // cookies are always Secure, HttpOnly and SameSite=Strict. The random value
// logout replaces is saved in the database, so it outlasts a restart. // that each logout replaces is saved in the database, so it outlasts a
// restart.
func newSessionManager( func newSessionManager(
_ fx.Lifecycle, params sessionManagerParams, _ fx.Lifecycle, params sessionManagerParams,
) (*session.Manager, error) { ) (*session.Manager, error) {
return session.NewManagerWithDatabase( return session.NewManagerWithDatabase(context.Background(),
context.Background(), params.Config.SigningKey, params.Database.DB()) params.Keys.SessionHashKey, params.Keys.SessionBlockKey,
params.Database.DB())
} }
// newEncryptedURLGenerator builds the encrypted URL generator, with a key // newEncryptedURLGenerator builds the encrypted URL generator with its key.
// derived from the signing key. func newEncryptedURLGenerator(keys seal.Keys) *encurl.Generator {
func newEncryptedURLGenerator(cfg *config.Config) (*encurl.Generator, error) { return encurl.NewGenerator(keys.EncryptedURLKey)
return encurl.NewGenerator(cfg.SigningKey)
} }
internal/seal/keys.go
+57
View File
@@ -0,0 +1,57 @@
package seal
// The salt of each key that DeriveKeys derives. Each key has a salt of its
// own, so no two uses share a key.
const (
sessionHashKeySalt = "pixa-session-hash-v1"
sessionBlockKeySalt = "pixa-session-block-v1"
encryptedURLKeySalt = "pixa-urlenc-v1"
csrfKeySalt = "pixa-csrf-v1"
)
// Keys holds the keys derived from the signing key, one for each use. The
// signing key itself is used for two things only: it is the HMAC key of signed
// URLs and the password of the login page.
type Keys struct {
// SessionHashKey signs the session cookie.
SessionHashKey [KeySize]byte
// SessionBlockKey encrypts the session cookie.
SessionBlockKey [KeySize]byte
// EncryptedURLKey encrypts the token of an encrypted URL.
EncryptedURLKey [KeySize]byte
// CSRFKey signs the CSRF token cookie of the login and generator pages.
CSRFKey [KeySize]byte
}
// DeriveKeys derives each key of Keys from signingKey, with DeriveKey and the
// salt of that key.
func DeriveKeys(signingKey string) (Keys, error) {
masterKey := []byte(signingKey)
sessionHashKey, err := DeriveKey(masterKey, sessionHashKeySalt)
if err != nil {
return Keys{}, err
}
sessionBlockKey, err := DeriveKey(masterKey, sessionBlockKeySalt)
if err != nil {
return Keys{}, err
}
encryptedURLKey, err := DeriveKey(masterKey, encryptedURLKeySalt)
if err != nil {
return Keys{}, err
}
csrfKey, err := DeriveKey(masterKey, csrfKeySalt)
if err != nil {
return Keys{}, err
}
return Keys{
SessionHashKey: sessionHashKey,
SessionBlockKey: sessionBlockKey,
EncryptedURLKey: encryptedURLKey,
CSRFKey: csrfKey,
}, nil
}
internal/seal/keys_test.go
+42
View File
@@ -0,0 +1,42 @@
package seal_test
import (
"bytes"
"testing"
"sneak.berlin/go/pixa/internal/seal"
)
// TestDeriveKeys_KeysDiffer verifies that the keys DeriveKeys derives differ
// from each other and from the signing key.
func TestDeriveKeys_KeysDiffer(t *testing.T) {
t.Parallel()
// 32 bytes, as long as a derived key, so that no key differs from it by
// its length alone
const signingKey = "0123456789abcdef0123456789abcdef"
keys, err := seal.DeriveKeys(signingKey)
if err != nil {
t.Fatalf("DeriveKeys() error = %v", err)
}
named := []struct {
name string
key []byte
}{
{"the signing key", []byte(signingKey)},
{"SessionHashKey", keys.SessionHashKey[:]},
{"SessionBlockKey", keys.SessionBlockKey[:]},
{"EncryptedURLKey", keys.EncryptedURLKey[:]},
{"CSRFKey", keys.CSRFKey[:]},
}
for i, a := range named {
for _, b := range named[i+1:] {
if bytes.Equal(a.key, b.key) {
t.Errorf("%s equals %s", a.name, b.name)
}
}
}
}
internal/server/computed_side_internal_test.go
+5 -2
View File
@@ -11,6 +11,7 @@ import (
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
) )
// TestImageProxyRefusesComputedSideAboveMaximum requests sizes of a 3000x600 // TestImageProxyRefusesComputedSideAboveMaximum requests sizes of a 3000x600
@@ -48,11 +49,13 @@ func TestImageProxyRefusesComputedSideAboveMaximum(t *testing.T) {
s, _, _ := startImageProxy(t, upstream) s, _, _ := startImageProxy(t, upstream)
generator, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator: %v", err) t.Fatalf("seal.DeriveKeys: %v", err)
} }
generator := encurl.NewGenerator(keys.EncryptedURLKey)
tests := []struct { tests := []struct {
path string path string
// size is the size of the /v1/image/ URL, and width and height are // size is the size of the /v1/image/ URL, and width and height are
internal/server/fit_outside_internal_test.go
+5 -2
View File
@@ -9,6 +9,7 @@ import (
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
) )
// TestImageProxyRefusesFitOutsideAboveMaximum requests sizes with fit=outside // TestImageProxyRefusesFitOutsideAboveMaximum requests sizes with fit=outside
@@ -42,11 +43,13 @@ func TestImageProxyRefusesFitOutsideAboveMaximum(t *testing.T) {
s, _, _ := startImageProxy(t, upstream) s, _, _ := startImageProxy(t, upstream)
generator, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator: %v", err) t.Fatalf("seal.DeriveKeys: %v", err)
} }
generator := encurl.NewGenerator(keys.EncryptedURLKey)
tests := []struct { tests := []struct {
path string path string
width, height int width, height int
internal/server/source_url_internal_test.go
+9 -6
View File
@@ -14,8 +14,9 @@ import (
"time" "time"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache" "sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
"sneak.berlin/go/pixa/internal/signature" "sneak.berlin/go/pixa/internal/signature"
) )
@@ -226,7 +227,7 @@ func TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart(t *testing.T) {
for _, imageURL := range imageURLs { for _, imageURL := range imageURLs {
cacheStatus := getImage(t, s, imageURL).Header().Get("X-Pixa-Cache") cacheStatus := getImage(t, s, imageURL).Header().Get("X-Pixa-Cache")
if cacheStatus != string(imgcache.CacheHit) { if cacheStatus != string(imageproxy.CacheHit) {
t.Errorf("second GET %s: X-Pixa-Cache = %q, want HIT", t.Errorf("second GET %s: X-Pixa-Cache = %q, want HIT",
imageURL, cacheStatus) imageURL, cacheStatus)
} }
@@ -255,7 +256,7 @@ func TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest(t *testing.T) {
second := getImage(t, s, second := getImage(t, s,
"/v1/image/"+upstreamHost+"/photo.png%3Fname=my%20cat/orig.png") "/v1/image/"+upstreamHost+"/photo.png%3Fname=my%20cat/orig.png")
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) { if got := second.Header().Get("X-Pixa-Cache"); got != string(imageproxy.CacheHit) {
t.Errorf("second X-Pixa-Cache = %q, want HIT", got) t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
} }
@@ -364,7 +365,7 @@ func TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL(t *testing.T) {
getImage(t, s, imageURLs[0]) getImage(t, s, imageURLs[0])
second := getImage(t, s, imageURLs[1]) second := getImage(t, s, imageURLs[1])
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) { if got := second.Header().Get("X-Pixa-Cache"); got != string(imageproxy.CacheHit) {
t.Errorf("second X-Pixa-Cache = %q, want HIT", got) t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
} }
@@ -540,11 +541,13 @@ func signedImageURL(t *testing.T, sourceHost string) string {
func encryptedImageURL(t *testing.T, sourceHost string) string { func encryptedImageURL(t *testing.T, sourceHost string) string {
t.Helper() t.Helper()
generator, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator: %v", err) t.Fatalf("seal.DeriveKeys: %v", err)
} }
generator := encurl.NewGenerator(keys.EncryptedURLKey)
token, err := generator.Generate(&imagerequest.Request{ token, err := generator.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: sourceHost, Path: ipv6SourcePath}, Source: imagerequest.Source{Host: sourceHost, Path: ipv6SourcePath},
Transform: imagerequest.Transform{Format: imagerequest.FormatPNG}, Transform: imagerequest.Transform{Format: imagerequest.FormatPNG},
internal/server/svg_as_avif_internal_test.go
+5 -2
View File
@@ -10,6 +10,7 @@ import (
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest" "sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
) )
// svgServedAsAVIF is an SVG whose first bytes pass the first-bytes check for // svgServedAsAVIF is an SVG whose first bytes pass the first-bytes check for
@@ -34,11 +35,13 @@ func TestImageProxyRefusesSVGServedAsAVIF(t *testing.T) {
s, _, _ := startImageProxy(t, upstream) s, _, _ := startImageProxy(t, upstream)
generator, err := encurl.NewGenerator(testSigningKey) keys, err := seal.DeriveKeys(testSigningKey)
if err != nil { if err != nil {
t.Fatalf("encurl.NewGenerator: %v", err) t.Fatalf("seal.DeriveKeys: %v", err)
} }
generator := encurl.NewGenerator(keys.EncryptedURLKey)
encryptedURL := func(format imagerequest.Format) string { encryptedURL := func(format imagerequest.Format) string {
token, err := generator.Generate(&imagerequest.Request{ token, err := generator.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: upstreamHost, Path: "/drawing.avif"}, Source: imagerequest.Source{Host: upstreamHost, Path: "/drawing.avif"},
internal/session/session.go
+9 -28
View File
@@ -22,10 +22,6 @@ import (
const ( const (
CookieName = "pixa_session" CookieName = "pixa_session"
SessionTTL = 30 * 24 * time.Hour // 30 days SessionTTL = 30 * 24 * time.Hour // 30 days
// HKDF salts for key derivation
hashKeySalt = "pixa-session-hash-v1"
blockKeySalt = "pixa-session-block-v1"
) )
// Errors returned by session operations. // Errors returned by session operations.
@@ -66,35 +62,23 @@ type Manager struct {
logoutValue string logoutValue string
} }
// NewManager creates a session manager with keys derived from the signing key. // NewManager creates a session manager whose session cookies hashKey signs
// It makes a new random value and keeps it in memory only, so a restart ends // and blockKey encrypts. It makes a new random value and keeps it in memory
// every session; NewManagerWithDatabase keeps the value in the database. // only, so a restart ends every session; NewManagerWithDatabase keeps the
// value in the database.
// //
// Session cookies always carry the Secure, HttpOnly, and SameSite=Strict // Session cookies always carry the Secure, HttpOnly, and SameSite=Strict
// attributes; this cannot be configured. Browsers treat http://localhost as a // attributes; this cannot be configured. Browsers treat http://localhost as a
// trustworthy origin and accept Secure cookies there, so local development // trustworthy origin and accept Secure cookies there, so local development
// keeps working. // keeps working.
func NewManager(signingKey string) (*Manager, error) { func NewManager(hashKey, blockKey [seal.KeySize]byte) *Manager {
masterKey := []byte(signingKey)
// Derive separate keys for HMAC (hash) and encryption (block)
hashKey, err := seal.DeriveKey(masterKey, hashKeySalt)
if err != nil {
return nil, err
}
blockKey, err := seal.DeriveKey(masterKey, blockKeySalt)
if err != nil {
return nil, err
}
sc := securecookie.New(hashKey[:], blockKey[:]) sc := securecookie.New(hashKey[:], blockKey[:])
sc.MaxAge(int(SessionTTL.Seconds())) sc.MaxAge(int(SessionTTL.Seconds()))
return &Manager{ return &Manager{
sc: sc, sc: sc,
logoutValue: rand.Text(), logoutValue: rand.Text(),
}, nil }
} }
// NewManagerWithDatabase creates a session manager like NewManager that keeps // NewManagerWithDatabase creates a session manager like NewManager that keeps
@@ -102,14 +86,11 @@ func NewManager(signingKey string) (*Manager, error) {
// NewManager made when db has none, and otherwise starts from the one saved // NewManager made when db has none, and otherwise starts from the one saved
// there. // there.
func NewManagerWithDatabase( func NewManagerWithDatabase(
ctx context.Context, signingKey string, db *sql.DB, ctx context.Context, hashKey, blockKey [seal.KeySize]byte, db *sql.DB,
) (*Manager, error) { ) (*Manager, error) {
m, err := NewManager(signingKey) m := NewManager(hashKey, blockKey)
if err != nil {
return nil, err
}
_, err = db.ExecContext(ctx, _, err := db.ExecContext(ctx,
"INSERT OR IGNORE INTO logout (id, logout_value) VALUES (1, ?)", "INSERT OR IGNORE INTO logout (id, logout_value) VALUES (1, ?)",
m.logoutValue) m.logoutValue)
if err != nil { if err != nil {
internal/session/session_cookie_attributes_test.go
+5 -2
View File
@@ -5,6 +5,7 @@ import (
"net/http/httptest" "net/http/httptest"
"testing" "testing"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
@@ -20,11 +21,13 @@ import (
func TestSessionCookieAttributesAlwaysSecure(t *testing.T) { func TestSessionCookieAttributesAlwaysSecure(t *testing.T) {
t.Parallel() t.Parallel()
mgr, err := session.NewManager("test-signing-key-12345") keys, err := seal.DeriveKeys("test-signing-key-12345")
if err != nil { if err != nil {
t.Fatalf("NewManager() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
writePaths := []struct { writePaths := []struct {
name string name string
setCookie func(t *testing.T, w http.ResponseWriter) setCookie func(t *testing.T, w http.ResponseWriter)
internal/session/session_logout_test.go
+15 -4
View File
@@ -8,6 +8,7 @@ import (
"testing" "testing"
"sneak.berlin/go/pixa/internal/database" "sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
@@ -80,8 +81,13 @@ func TestLogout_EndsEverySessionMadeBeforeIt(t *testing.T) {
db := newTestDatabase(t) db := newTestDatabase(t)
keys, err := seal.DeriveKeys(logoutTestSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
mgr, err := session.NewManagerWithDatabase( mgr, err := session.NewManagerWithDatabase(
t.Context(), logoutTestSigningKey, db) t.Context(), keys.SessionHashKey, keys.SessionBlockKey, db)
if err != nil { if err != nil {
t.Fatalf("NewManagerWithDatabase() error = %v", err) t.Fatalf("NewManagerWithDatabase() error = %v", err)
} }
@@ -109,7 +115,7 @@ func TestLogout_EndsEverySessionMadeBeforeIt(t *testing.T) {
} }
restarted, err := session.NewManagerWithDatabase( restarted, err := session.NewManagerWithDatabase(
t.Context(), logoutTestSigningKey, db) t.Context(), keys.SessionHashKey, keys.SessionBlockKey, db)
if err != nil { if err != nil {
t.Fatalf("NewManagerWithDatabase() after restart error = %v", err) t.Fatalf("NewManagerWithDatabase() after restart error = %v", err)
} }
@@ -129,8 +135,13 @@ func TestLogout_EndsEverySessionMadeBeforeIt(t *testing.T) {
func TestLogout_WithoutSession_EndsNoSession(t *testing.T) { func TestLogout_WithoutSession_EndsNoSession(t *testing.T) {
t.Parallel() t.Parallel()
mgr, err := session.NewManagerWithDatabase( keys, err := seal.DeriveKeys(logoutTestSigningKey)
t.Context(), logoutTestSigningKey, newTestDatabase(t)) if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
mgr, err := session.NewManagerWithDatabase(t.Context(),
keys.SessionHashKey, keys.SessionBlockKey, newTestDatabase(t))
if err != nil { if err != nil {
t.Fatalf("NewManagerWithDatabase() error = %v", err) t.Fatalf("NewManagerWithDatabase() error = %v", err)
} }
internal/session/session_test.go
+19 -9
View File
@@ -7,17 +7,20 @@ import (
"testing" "testing"
"time" "time"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session" "sneak.berlin/go/pixa/internal/session"
) )
func TestManager_CreateAndValidate(t *testing.T) { func TestManager_CreateAndValidate(t *testing.T) {
t.Parallel() t.Parallel()
mgr, err := session.NewManager("test-signing-key-12345") keys, err := seal.DeriveKeys("test-signing-key-12345")
if err != nil { if err != nil {
t.Fatalf("NewManager() error = %v", err) t.Fatalf("seal.DeriveKeys() error = %v", err)
} }
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
// Create a session // Create a session
w := httptest.NewRecorder() w := httptest.NewRecorder()
@@ -69,7 +72,8 @@ func TestManager_CreateAndValidate(t *testing.T) {
func TestManager_ValidateSession_NoCookie(t *testing.T) { func TestManager_ValidateSession_NoCookie(t *testing.T) {
t.Parallel() t.Parallel()
mgr, _ := session.NewManager("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil) req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
@@ -86,7 +90,8 @@ func TestManager_ValidateSession_NoCookie(t *testing.T) {
func TestManager_ValidateSession_TamperedCookie(t *testing.T) { func TestManager_ValidateSession_TamperedCookie(t *testing.T) {
t.Parallel() t.Parallel()
mgr, _ := session.NewManager("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil) req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req.AddCookie(&http.Cookie{ req.AddCookie(&http.Cookie{
@@ -110,8 +115,10 @@ func TestManager_ValidateSession_TamperedCookie(t *testing.T) {
func TestManager_ValidateSession_WrongKey(t *testing.T) { func TestManager_ValidateSession_WrongKey(t *testing.T) {
t.Parallel() t.Parallel()
mgr1, _ := session.NewManager("signing-key-1") keys1, _ := seal.DeriveKeys("signing-key-1")
mgr2, _ := session.NewManager("signing-key-2") keys2, _ := seal.DeriveKeys("signing-key-2")
mgr1 := session.NewManager(keys1.SessionHashKey, keys1.SessionBlockKey)
mgr2 := session.NewManager(keys2.SessionHashKey, keys2.SessionBlockKey)
// Create session with mgr1 // Create session with mgr1
w := httptest.NewRecorder() w := httptest.NewRecorder()
@@ -142,7 +149,8 @@ func TestManager_ValidateSession_WrongKey(t *testing.T) {
func TestManager_ClearSession(t *testing.T) { func TestManager_ClearSession(t *testing.T) {
t.Parallel() t.Parallel()
mgr, _ := session.NewManager("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
w := httptest.NewRecorder() w := httptest.NewRecorder()
mgr.ClearSession(w) mgr.ClearSession(w)
@@ -172,7 +180,8 @@ func TestManager_ClearSession(t *testing.T) {
func TestManager_IsAuthenticated(t *testing.T) { func TestManager_IsAuthenticated(t *testing.T) {
t.Parallel() t.Parallel()
mgr, _ := session.NewManager("test-signing-key-12345") keys, _ := seal.DeriveKeys("test-signing-key-12345")
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
// No session - should return false // No session - should return false
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil) req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
@@ -208,7 +217,8 @@ func TestManager_IsAuthenticated(t *testing.T) {
func TestManager_CookieAttributes(t *testing.T) { func TestManager_CookieAttributes(t *testing.T) {
t.Parallel() t.Parallel()
mgr, _ := session.NewManager("test-key") keys, _ := seal.DeriveKeys("test-key")
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
w := httptest.NewRecorder() w := httptest.NewRecorder()
_ = mgr.CreateSession(w) _ = mgr.CreateSession(w)