Author SHA1 Message Date
clawbot 7cbcd3957f Eviction no longer reads a whole table while requests wait on the database (closes #227)
check / check (push) Canceled after 0s
UsageBytes now reads the new cache_usage row, which triggers on
source_content and variant_content keep up to date in the statement
that adds, removes or resizes a row. The reconciliation pass reads both
tables 1000 rows per query, sums them, and corrects the total when it
differs, unless a row changed while it summed. Source rows now get
last_accessed_at when added, so choosing source images to evict reads
that column's index instead of sorting the whole table. Stats still
sums the tables, now in pages: an existing test drops both tables and
expects that sum to fail.

Model: opus-5-5
2026-10-08 04:28:31 +00:00
31 changed files with 153 additions and 1757 deletions
+10 -15
View File
@@ -17,8 +17,8 @@ COPY . .
RUN golangci-lint run --config .golangci.yml ./... RUN golangci-lint run --config .golangci.yml ./...
# Test phase. script/test builds it alone. # Test phase. script/test builds it alone.
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too # golang:1.25.4-alpine, 2026-02-25
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS test FROM golang:1.25.4-alpine@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS test
WORKDIR /src WORKDIR /src
@@ -32,19 +32,16 @@ RUN script/bootstrap --cgo
COPY . . COPY . .
# Without -v first; on a failure, again with -v for the details, and # Without -v first; on a failure, again with -v for the details, and
# the step fails even if the second run passes. -parallel 4: by default # the step fails even if the second run passes.
# a package runs as many of its tests at once as the host has CPUs, and RUN go test -count=1 -timeout 90s -race -cover ./... || \
# on a busy host they then wait so long to be scheduled that a test
# that times a request can see it return a second late.
RUN go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \ { echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; } go test -count=1 -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from the two phases above: these copies # Build stage. Nothing is wanted from the two phases above: these copies
# make BuildKit build them first, so this stage runs only when lint and # make BuildKit build them first, so this stage runs only when lint and
# test passed. # test passed.
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too # golang:1.25.4-alpine, 2026-02-25
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder FROM golang:1.25.4-alpine@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
@@ -80,11 +77,9 @@ RUN version="${VERSION:-$(git describe --tags --always)}"; \
-o /pixad ./cmd/pixad -o /pixad ./cmd/pixad
# Runtime stage, and the last one: a plain `docker build .` builds this # Runtime stage, and the last one: a plain `docker build .` builds this
# stage and what it depends on, and nothing else. It must use the Alpine # stage and what it depends on, and nothing else.
# release the golang image above is based on, so that pixad runs against # alpine:3.21, 2026-02-25
# the libvips and musl it was built with. FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
# alpine:3.22, 2026-10-08
FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8
# Install runtime dependencies only. vips-jxl is libvips' JPEG XL # Install runtime dependencies only. vips-jxl is libvips' JPEG XL
# support, without which pixad does not start. # support, without which pixad does not start.
+38 -57
View File
@@ -83,14 +83,12 @@ which answers 200 whenever pixa is running, in maintenance mode too (see
`maintenance_mode`). `maintenance_mode`).
On SIGTERM or SIGINT pixa stops accepting connections, gives the requests in On SIGTERM or SIGINT pixa stops accepting connections, gives the requests in
progress and the images being processed 5 seconds to finish, then writes to the progress and the images being processed 5 seconds to finish, and exits: with 0,
database the counts of cache hits, misses, fetches and conversions that requests or with 1 when images were still being processed after those 5 seconds or
could not write by their deadline, and exits: with 0, or with 1 when images were another part of pixa failed to stop. A request not finished by then is cut off.
still being processed after those 5 seconds, some of those counts could not be `docker stop` waits 10 seconds before it kills the container.
written, or another part of pixa failed to stop. A request not finished by then
is cut off. `docker stop` waits 10 seconds before it kills the container.
Outside Docker, pixa needs libvips (the image has 8.16) and libheif to run, as Outside Docker, pixa needs libvips (the image has 8.15) and libheif to run, as
it uses libvips through CGO. pixad does not start unless libvips has its JPEG XL it uses libvips through CGO. pixad does not start unless libvips has its JPEG XL
support, which on Alpine is the `vips-jxl` package and which the nix and brew support, which on Alpine is the `vips-jxl` package and which the nix and brew
packages of libvips include, as do the apt ones from Debian 12 and Ubuntu 24.04 packages of libvips include, as do the apt ones from Debian 12 and Ubuntu 24.04
@@ -177,21 +175,20 @@ path under `/v1/` answers 200, in maintenance mode too.
with the page naming a field that is not valid; 500 when the URL cannot be with the page naming a field that is not valid; 500 when the URL cannot be
made. made.
- `GET /logout` — end the login session. Needs: nothing. Answers: 303 to `/`. - `GET /logout` — end the login session. Needs: nothing. Answers: 303 to `/`.
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or - `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>` — an image, fetched,
`/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized resized and converted (below). Needs: a signature, unless the host is
and converted (below). Needs: a signature, unless the host is allowlisted (see allowlisted (see Source Hosts). Answers: 200; 304 when `If-None-Match` matches
Source Hosts). Answers: 200; 304 when `If-None-Match` matches the image's the image's `ETag`; 400 for a URL or parameter that is not valid, or for the
`ETag`; 400 for a URL or parameter that is not valid, or for the format `auto` format `auto` an `Accept` header that is not valid; 406 for the format `auto`
an `Accept` header that is not valid; 406 for the format `auto` when `Accept` when `Accept` allows none of the formats it chooses from; 401 for a missing or
allows none of the formats it chooses from; 401 for a missing or wrong wrong signature, a missing `exp` or an `exp` in the past; 403 when the
signature, a missing `exp` or an `exp` in the past; 403 when the request's request's `Referer` names a host in `referer_blocklist`, checked before the
`Referer` names a host in `referer_blocklist`, checked before the signature, signature, the cache and the upstream fetch; 403 when the upstream host, or a
the cache and the upstream fetch; 403 when the upstream host, or a host it host it redirects to, is `localhost`, ends in `.localhost` or `.local`, or has
redirects to, is `localhost`, ends in `.localhost` or `.local`, or has an an address in a blocked network (see `blocked_networks`); 502 when the
address in a blocked network (see `blocked_networks`); 502 when the upstream upstream answered with an error status, and for 5 minutes after that for the
answered with an error status, and for 5 minutes after that for the same same source URL; 503 when pixa is busy or in maintenance mode; 500 for any
source URL; 503 when pixa is busy or in maintenance mode; 500 for any other other failure.
failure.
- `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL - `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL
(see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when (see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when
`If-None-Match` matches the image's `ETag`; 400 for a token that does not `If-None-Match` matches the image's `ETag`; 400 for a token that does not
@@ -234,11 +231,10 @@ proxy in front of pixa must pass that header on unchanged. A form body over 1
MiB is refused with 413. The image routes answer the errors listed for them with MiB is refused with 413. The image routes answer the errors listed for them with
JSON holding `error`, `status` and `timestamp`. JSON holding `error`, `status` and `timestamp`.
An image URL has one of these forms, the second with no format: An image URL has this form:
``` ```
/v1/image/<host>/<path>/<size>.<format>?sig=<signature>&exp=<expiration>&q=<quality>&fit=<fit> /v1/image/<host>/<path>/<size>.<format>?sig=<signature>&exp=<expiration>&q=<quality>&fit=<fit>
/v1/image/<host>/<path>/<size>?sig=<signature>&exp=<expiration>&q=<quality>&fit=<fit>
``` ```
Images are only fetched from origins using TLS with valid certificates, unless Images are only fetched from origins using TLS with valid certificates, unless
@@ -249,9 +245,7 @@ A request whose query string cannot be decoded, or gives any parameter more than
once, is refused with 400. once, is refused with 400.
- `<format>`: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`, - `<format>`: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`,
`avif`, `jxl` (JPEG XL), `gif`, or `auto` (below). A URL with no format (the `avif`, `gif`, or `auto` (below)
second form, with no dot after the size) is served as JPEG XL, the default, as
with `jxl`
- `<size>`: `orig` or `<width>x<height>` (e.g. `800x600`) - `<size>`: `orig` or `<width>x<height>` (e.g. `800x600`)
- `sig` and `exp`: the signature and its expiry, needed unless the host is - `sig` and `exp`: the signature and its expiry, needed unless the host is
allowlisted (see Signature Specification) allowlisted (see Signature Specification)
@@ -259,29 +253,23 @@ once, is refused with 400.
both optional (values under Signature Specification). Both are part of what is both optional (values under Signature Specification). Both are part of what is
cached, so each value of either is a separate cached image. cached, so each value of either is a separate cached image.
The source image may be JPEG, PNG, GIF, WebP, AVIF, JPEG XL or SVG. The
upstream's `Content-Type` must name its format, and the image's first bytes must
match it.
With the format `auto`, pixa chooses the format for each request from its With the format `auto`, pixa chooses the format for each request from its
`Accept` header, in this order: `Accept` header, in this order:
1. JPEG XL, when the header names `image/jxl`; 1. AVIF, when the header names `image/avif`;
2. AVIF, when it names `image/avif`; 2. WebP, when it names `image/webp`;
3. WebP, when it names `image/webp`; 3. JPEG, when the first of `image/jpeg`, `image/*` and `*/*` that it names
4. JPEG, when the first of `image/jpeg`, `image/*` and `*/*` that it names
allows it, or when there is no `Accept` header or it is empty. allows it, or when there is no `Accept` header or it is empty.
An entry with `q=0` refuses its format; other `q` values do not change the An entry with `q=0` refuses its format; other `q` values do not change the
order. JPEG XL, AVIF and WebP must be named, as clients that cannot show them order. AVIF and WebP must be named, as clients that cannot show them also send
also send `image/*` and `*/*`. pixa never sends a format the client refused: `image/*` and `*/*`. pixa never sends a format the client refused: when the
when the header allows none of the four, the answer is 406, and a header that header allows none of the three, the answer is 406, and a header that does not
does not parse, or has a `q` that is not a number from 0 to 1, is refused parse, or has a `q` that is not a number from 0 to 1, is refused with 400. The
with 400. The signature, or the token of an encrypted URL, covers `auto` itself, signature, or the token of an encrypted URL, covers `auto` itself, so one URL
so one URL serves every client. Each format chosen is cached as a separate serves every client. Each format chosen is cached as a separate image, and every
image, and every answer that depends on `Accept` (the image, a 304, and the 400 answer that depends on `Accept` (the image, a 304, and the 400 and 406 above)
and 406 above) carries `Vary: Accept`, so a shared cache keeps the formats apart carries `Vary: Accept`, so a shared cache keeps the formats apart too.
too.
An image is served with `Cache-Control: public, max-age=<seconds>, immutable`. An image is served with `Cache-Control: public, max-age=<seconds>, immutable`.
When the URL has an expiry (an `exp`, or the TTL of an encrypted URL), `max-age` When the URL has an expiry (an `exp`, or the TTL of an encrypted URL), `max-age`
@@ -327,15 +315,13 @@ nor change what it asks for.
lasts 30 days, or until `/logout`. lasts 30 days, or until `/logout`.
2. On the generator page, give the source image's URL, the width and height, the 2. On the generator page, give the source image's URL, the width and height, the
format, quality and fit, and how long the URL lasts, then submit the form format, quality and fit, and how long the URL lasts, then submit the form
(`POST /generate`). The format is JPEG XL unless another is chosen; a form (`POST /generate`). Width and height both empty or `0` keep the original
sent with an empty format, or none, also makes a JPEG XL URL. Width and size; if only one of them is empty or `0`, that side is scaled to keep the
height both empty or `0` keep the original size; if only one of them is empty image's proportions.
or `0`, that side is scaled to keep the image's proportions.
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when 3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
it expires. `<host>` is the host the page was opened on, and the URL starts it expires. `<host>` is the host the page was opened on, and the URL starts
with `http` instead while `debug` is on. The name after the token is ignored with `http` instead while `debug` is on. The name after the token is ignored
and only gives the URL a file extension, `jpg` for `orig` and `auto`, and and only gives the URL a file extension, `jpg` for `orig` and `auto`.
`jxl` for a form with no format.
The token holds the source's host, path and query and the size, format, quality, The token holds the source's host, path and query and the size, format, quality,
fit and expiry, encrypted with a key derived from `signing_key`. The source fit and expiry, encrypted with a key derived from `signing_key`. The source
@@ -364,10 +350,6 @@ turned off.
- An image with an ICC profile is converted to sRGB first, since clients show an - An image with an ICC profile is converted to sRGB first, since clients show an
image with no profile as sRGB. Colours outside sRGB, such as the most image with no profile as sRGB. Colours outside sRGB, such as the most
saturated ones in a Display P3 photo, are clipped. saturated ones in a Display P3 photo, are clipped.
- The one exception is JPEG XL with libvips 8.16 and later: libvips then writes
an EXIF block of its own into the image, as govips cannot ask libvips to leave
it out. It holds the image's size and otherwise fixed values, such as an
orientation of 1 and a resolution of 72 dpi, and nothing from the source.
### Source Hosts ### Source Hosts
@@ -395,9 +377,8 @@ Where:
- `width` — requested width in pixels, `0` for original - `width` — requested width in pixels, `0` for original
- `height` — requested height in pixels, `0` for original - `height` — requested height in pixels, `0` for original
- `format` — output format, one of those listed under Routes, with `original` - `format` — output format, one of those listed under Routes, with `original`
signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no signed as `orig` and `jpg` as `jpeg`; `auto` is signed as `auto`, not as the
format has the signature of the same URL ending in `.jxl`; `auto` is signed as format chosen for the request
`auto`, not as the format chosen for the request
- `expiration` — the URL's `exp` query parameter, the Unix timestamp when the - `expiration` — the URL's `exp` query parameter, the Unix timestamp when the
signature expires; a request whose `exp` is not a whole number, an empty signature expires; a request whose `exp` is not a whole number, an empty
`exp=` included, is refused with 400 `exp=` included, is refused with 400
+8 -66
View File
@@ -30,72 +30,6 @@ P2: security: per-IP rate limiting on the image routes
# Completed Steps # Completed Steps
- 2026-10-08 a request past its deadline no longer waits for the database to
count it (closes #224). On a busy host, `TestService_Get_ReturnsByItsDeadline`
failed because its request, past its deadline, still waited for its miss count
to be written, a write with no deadline, which in pixad waits for the one
database connection every request shares. Each count write now keeps the
request's deadline but not its cancellation. A count not written by then is
kept in memory, where `Stats` sees it, and one goroutine of the cache writes
those counts, one write at a time, and once more at shutdown before the
database closes. The test phase of the `Dockerfile` also runs at most 4 tests
of a package at once (`go test -parallel 4`), where it ran as many as the host
has CPUs: on a busy host they then wait so long to be scheduled that a test
that times a request can fail. `-p`, how many packages are tested at once, is
unchanged, as capping it also slows compiling.
- 2026-10-08 every output format is saved with settings pixa sets on purpose
(closes #232): each format has its own govips export, as JPEG XL does, in
place of govips' generic `Export`, which sent libvips a zero for some settings
it was not given. PNG gets libvips' default compression, 6, where it had none,
and WebP libvips' default effort, 4, where it had 0. GIF was already at
libvips' default effort, 7, and JPEG output is unchanged. AVIF is saved at
effort 1, the lowest govips can set, where it had libvips' default, 4. With
one libvips thread, an 8192x8192 image of random pixels, the worst case, takes
about 51 seconds as AVIF at effort 1 and 43 as WebP at effort 4, against the
default `downstream_timeout` of 60 seconds. On an image of milder noise, which
AVIF at effort 1 saves in about 12 seconds, effort 4 takes minutes. AVIF is
also saved with 8 bits per sample from a 16-bit source, which libvips would
save with 12: a 16-bit 8192x8192 image of milder noise takes about 54 seconds
at effort 1 with 12 bits, nearly all of the default `downstream_timeout`, and
about 12 with 8.
- 2026-10-08 JPEG XL is the default output (closes #222): a `/v1/image/` URL
whose last segment is a size with no format, such as `800x600` or `orig`, is
served as JPEG XL and signed as `jxl`, so it has the signature of the same URL
ending in `.jxl`. An encrypted URL whose token holds no format is served as
JPEG XL (`encurl.DefaultFormat`). The generator page selects JPEG XL until
another format is chosen, and a form with an empty format, or none, makes a
JPEG XL URL whose name ends in `.jxl`. The image processor no longer takes an
empty format as `orig`: both routes give every request a format, and it
refuses a request with none. `auto` still ends with JPEG.
- 2026-10-08 JPEG XL as an input and output format (part of #222): a source
whose bytes start with either JPEG XL signature, the bare codestream's `FF 0A`
or the container's, is detected as `image/jxl`, which the upstream fetch
accepts; `orig` of such a source is JPEG XL. The format `jxl` works in plain
and encrypted URLs and on the generator page, served as `image/jxl` and cached
like the other formats. `auto` chooses JPEG XL first when `Accept` names
`image/jxl`. govips sends libvips a JPEG XL distance with every save, so
libvips ignores the quality: pixa turns `q` into a distance with libvips' own
formula, keeping 100 lossy, and removes the metadata from the image before
saving, as govips cannot have libvips strip it from JPEG XL. libvips 8.16 and
later still write an EXIF block of their own into JPEG XL, from the image's
size, orientation and resolution, and fixed values; the image is upright and
is given 72 dpi before the save, so the block holds nothing from the source.
An image with an ICC profile is converted to sRGB before the JPEG XL save too,
and a CMYK image with none is converted to sRGB, as libvips cannot save CMYK
as JPEG XL. libvips' default effort, 7, is kept. JPEG XL is not yet the
default output.
- 2026-10-08 pixad runs on the Alpine release it is built on (closes #229): the
runtime stage of the `Dockerfile` uses `alpine:3.22`, the release of the
`golang:1.25.4-alpine3.22` image that the test phase and the build stage use,
so all three have libvips 8.16, where the runtime image had 8.15.
- 2026-10-08 requests no longer wait behind eviction queries that read a whole
table (closes #227): the new `cache_usage` table holds the total cache usage,
kept up to date by triggers on `source_content` and `variant_content` in the
statement that adds, removes or resizes a row, and `UsageBytes` reads it. The
reconciliation pass reads the content tables 1000 rows per query, sums them
and corrects the total when it differs, unless a row changed while it summed.
Source rows get `last_accessed_at` when added, so choosing source images to
evict reads that column's index instead of sorting the whole table.
- 2026-10-08 libvips' JPEG XL support is installed and required (part of #222): - 2026-10-08 libvips' JPEG XL support is installed and required (part of #222):
`script/bootstrap --cgo` installs `vips-jxl` when its package manager is apk, `script/bootstrap --cgo` installs `vips-jxl` when its package manager is apk,
as Alpine's `vips` package lacks the support, and the runtime stage of the as Alpine's `vips` package lacks the support, and the runtime stage of the
@@ -104,6 +38,14 @@ P2: security: per-IP rate limiting on the image routes
save JPEG XL, so pixad does not start without it. A test saves an image as save JPEG XL, so pixad does not start without it. A test saves an image as
JPEG XL with govips and loads it back. JPEG XL is not yet a format pixa JPEG XL with govips and loads it back. JPEG XL is not yet a format pixa
serves. serves.
- 2026-10-08 requests no longer wait behind eviction queries that read a whole
table (closes #227): the new `cache_usage` table holds the total cache usage,
kept up to date by triggers on `source_content` and `variant_content` in the
statement that adds, removes or resizes a row, and `UsageBytes` reads it. The
reconciliation pass reads the content tables 1000 rows per query, sums them
and corrects the total when it differs, unless a row changed while it summed.
Source rows get `last_accessed_at` when added, so choosing source images to
evict reads that column's index instead of sorting the whole table.
- 2026-10-08 SQLite writes no longer fail with "database is locked" under load - 2026-10-08 SQLite writes no longer fail with "database is locked" under load
(closes #223): `internal/database` opens the database with one connection, so (closes #223): `internal/database` opens the database with one connection, so
pixa's own reads and writes run on it one at a time instead of competing for pixa's own reads and writes run on it one at a time instead of competing for
+2 -2
View File
@@ -14,7 +14,7 @@ import (
// Default values for optional fields. // Default values for optional fields.
const ( const (
DefaultQuality = 85 DefaultQuality = 85
DefaultFormat = imgcache.FormatJXL DefaultFormat = imgcache.FormatOriginal
DefaultFitMode = imgcache.FitCover DefaultFitMode = imgcache.FitCover
// HKDF salt for URL encryption key derivation // HKDF salt for URL encryption key derivation
@@ -37,7 +37,7 @@ type Payload struct {
SourceQuery string `cbor:"q,omitempty"` // optional SourceQuery string `cbor:"q,omitempty"` // optional
Width int `cbor:"w,omitempty"` // 0 = original Width int `cbor:"w,omitempty"` // 0 = original
Height int `cbor:"ht,omitempty"` // 0 = original Height int `cbor:"ht,omitempty"` // 0 = original
Format imgcache.ImageFormat `cbor:"f,omitempty"` // default: jxl Format imgcache.ImageFormat `cbor:"f,omitempty"` // default: orig
Quality int `cbor:"ql,omitempty"` // default: 85 Quality int `cbor:"ql,omitempty"` // default: 85
FitMode imgcache.FitMode `cbor:"fm,omitempty"` // default: cover FitMode imgcache.FitMode `cbor:"fm,omitempty"` // default: cover
ExpiresAt int64 `cbor:"e,omitempty"` // 0 = never expires ExpiresAt int64 `cbor:"e,omitempty"` // 0 = never expires
+3 -8
View File
@@ -369,15 +369,10 @@ func (s *Handlers) buildGeneratedURL(r *http.Request, token, format string) stri
scheme = "http" scheme = "http"
} }
// Determine file extension for the trailing filename. A form with no // Determine file extension for the trailing filename
// format makes a token with none, which is served as encurl.DefaultFormat.
ext := format ext := format
if ext == "" || ext == "orig" || ext == "auto" {
switch format { ext = "jpg" // Default extension
case "":
ext = string(encurl.DefaultFormat)
case "orig", "auto":
ext = "jpg"
} }
return scheme + "://" + r.Host + "/v1/e/" + url.PathEscape(token) + "/img." + ext return scheme + "://" + r.Host + "/v1/e/" + url.PathEscape(token) + "/img." + ext
+7 -11
View File
@@ -14,8 +14,8 @@ import (
// Errors for an Accept header that an auto URL cannot be served for. // Errors for an Accept header that an auto URL cannot be served for.
var ( var (
errInvalidAccept = errors.New("invalid Accept header") errInvalidAccept = errors.New("invalid Accept header")
errNotAcceptable = errors.New("not acceptable: auto serves " + errNotAcceptable = errors.New(
"image/jxl, image/avif, image/webp or image/jpeg") "not acceptable: auto serves image/avif, image/webp or image/jpeg")
) )
// chooseAutoFormat replaces the format auto in req with the format // chooseAutoFormat replaces the format auto in req with the format
@@ -51,11 +51,11 @@ func (s *Handlers) chooseAutoFormat(
} }
// formatForAccept returns the format an auto URL is served in for the Accept // formatForAccept returns the format an auto URL is served in for the Accept
// header accept: JPEG XL when it names image/jxl, else AVIF when it names // header accept: AVIF when it names image/avif, else WebP when it names
// image/avif, else WebP when it names image/webp, else JPEG when its most // image/webp, else JPEG when its most specific entry of image/jpeg, image/*
// specific entry of image/jpeg, image/* and */* allows it, or when it names // and */* allows it, or when it names nothing. A q of 0 refuses a format.
// nothing. A q of 0 refuses a format. JPEG XL, AVIF and WebP must be named, as // AVIF and WebP must be named, as clients that cannot show them send image/*
// clients that cannot show them send image/* and */* too. // and */* too.
func formatForAccept(accept string) (imgcache.ImageFormat, error) { func formatForAccept(accept string) (imgcache.ImageFormat, error) {
qualities, err := parseAccept(accept) qualities, err := parseAccept(accept)
if err != nil { if err != nil {
@@ -66,10 +66,6 @@ func formatForAccept(accept string) (imgcache.ImageFormat, error) {
return imgcache.FormatJPEG, nil return imgcache.FormatJPEG, nil
} }
if qualities["image/jxl"] > 0 {
return imgcache.FormatJXL, nil
}
if qualities["image/avif"] > 0 { if qualities["image/avif"] > 0 {
return imgcache.FormatAVIF, nil return imgcache.FormatAVIF, nil
} }
+2 -11
View File
@@ -4,7 +4,6 @@ package handlers
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"errors"
"log/slog" "log/slog"
"net/http" "net/http"
"time" "time"
@@ -72,21 +71,16 @@ func New(lc fx.Lifecycle, params Params) (*Handlers, error) {
} }
lc.Append(fx.Hook{ lc.Append(fx.Hook{
//nolint:contextcheck // the cache's goroutines outlive OnStart; OnStop stops them //nolint:contextcheck // the eviction loop outlives OnStart; OnStop cancels it
OnStart: func(_ context.Context) error { OnStart: func(_ context.Context) error {
return s.initImageService() return s.initImageService()
}, },
// The pending counts are written here, before the database's stop
// hook closes it.
OnStop: func(ctx context.Context) error { OnStop: func(ctx context.Context) error {
if s.imgCache == nil { if s.imgCache == nil {
return nil return nil
} }
return errors.Join( return s.imgCache.StopEviction(ctx)
s.imgCache.StopEviction(ctx),
s.imgCache.StopPendingCountWrites(ctx),
)
}, },
}) })
@@ -128,9 +122,6 @@ func (s *Handlers) initImageService() error {
// write-pressure passes. No-op when the disk cache is disabled. // write-pressure passes. No-op when the disk cache is disabled.
cache.StartEviction(imgcache.DefaultEvictionInterval) cache.StartEviction(imgcache.DefaultEvictionInterval)
// Writes the counts requests could not write by their deadline
cache.StartPendingCountWrites()
// Create the fetcher config // Create the fetcher config
fetcherCfg := httpfetcher.DefaultConfig() fetcherCfg := httpfetcher.DefaultConfig()
fetcherCfg.AllowHTTP = s.config.AllowHTTP fetcherCfg.AllowHTTP = s.config.AllowHTTP
+1 -2
View File
@@ -18,8 +18,7 @@ import (
) )
// HandleImage handles the main image proxy route: // HandleImage handles the main image proxy route:
// /v1/image/<host>/<path>/<width>x<height>.<format>, or with no format // /v1/image/<host>/<path>/<width>x<height>.<format>
// /v1/image/<host>/<path>/<width>x<height>
func (s *Handlers) HandleImage() http.HandlerFunc { func (s *Handlers) HandleImage() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
if s.refuseBlockedReferer(w, r) { if s.refuseBlockedReferer(w, r) {
@@ -1,162 +0,0 @@
package handlers
import (
"fmt"
"log/slog"
"maps"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
"github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/signature"
)
// requireJPEGXL requires that rec answers 200 with a JPEG XL image.
func requireJPEGXL(t *testing.T, rec *httptest.ResponseRecorder) {
t.Helper()
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want %d; body %q",
rec.Code, http.StatusOK, rec.Body.String())
}
if got := rec.Header().Get("Content-Type"); got != jxlType {
t.Errorf("Content-Type = %q, want %s", got, jxlType)
}
if got := vips.DetermineImageType(rec.Body.Bytes()); got != vips.ImageTypeJXL {
t.Errorf("body is %s, want jxl", vips.ImageTypes[got])
}
}
// TestImageWithoutFormat_ServesJPEGXL verifies that a /v1/image/ URL whose
// last segment is a size with no format, 50x50 or orig, answers JPEG XL.
func TestImageWithoutFormat_ServesJPEGXL(t *testing.T) {
t.Parallel()
route := newImageRoute(t, newPhotoFetcher(t, allowlistedHost))
for _, size := range []string{"50x50", "orig"} {
target := "/v1/image/" + allowlistedHost + photoPath + "/" + size
requireJPEGXL(t, sendGet(t, route, target))
}
}
// TestImageWithoutFormat_SignedAsJXL verifies that a /v1/image/ URL with no
// format is signed as jxl: the signature made for the URL ending in .jxl is
// accepted for the same URL without .jxl.
func TestImageWithoutFormat_SignedAsJXL(t *testing.T) {
t.Parallel()
route := newImageRoute(t, newPhotoFetcher(t, signedHost))
expires := time.Now().Add(time.Hour)
sig := signature.New(testSigningKey).Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: photoPath,
Width: 50,
Height: 50,
Format: string(imgcache.FormatJXL),
Quality: encurl.DefaultQuality,
FitMode: string(imgcache.FitCover),
Expires: expires,
})
query := fmt.Sprintf("?sig=%s&exp=%d", sig, expires.Unix())
for _, size := range []string{"50x50.jxl", "50x50"} {
target := "/v1/image/" + signedHost + photoPath + "/" + size + query
requireJPEGXL(t, sendGet(t, route, target))
}
}
// TestImageEncWithoutFormat_ServesJPEGXL verifies that an encrypted URL whose
// token holds no format answers JPEG XL.
func TestImageEncWithoutFormat_ServesJPEGXL(t *testing.T) {
t.Parallel()
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
token, err := h.encGen.Generate(&encurl.Payload{
SourceHost: signedHost,
SourcePath: photoPath,
Width: 50,
Height: 50,
})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
requireJPEGXL(t, getEncToken(srv, token))
}
// TestGeneratorPage_SelectsJPEGXL verifies that the generator page's format
// choice is JPEG XL until another is chosen.
func TestGeneratorPage_SelectsJPEGXL(t *testing.T) {
t.Parallel()
h, srv := newCSRFTestRouter(t)
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req.AddCookie(newSessionCookie(t, h))
rec := httptest.NewRecorder()
srv.ServeHTTP(rec, req)
if !strings.Contains(rec.Body.String(), `<option value="jxl" selected>`) {
t.Errorf("generator page does not select JPEG XL: %s", rec.Body.String())
}
}
// TestGeneratePost_NoFormat_MakesJPEGXLURL verifies that the generator form
// sent with an empty format field, or with none, makes a URL whose name ends
// in .jxl and which answers JPEG XL.
func TestGeneratePost_NoFormat_MakesJPEGXLURL(t *testing.T) {
t.Parallel()
photo := url.Values{
sourceURLField: {"https://" + signedHost + photoPath},
widthField: {"50"},
heightField: {"50"},
}
emptyFormat := maps.Clone(photo)
emptyFormat.Set(formatField, "")
for name, form := range map[string]url.Values{
"empty format field": emptyFormat,
"no format field": photo,
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
_, imageSrv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
rec := generatePost(t, form)
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
if match == nil {
t.Fatalf("generator page shows no URL: %d %s",
rec.Code, rec.Body.String())
}
t.Logf("generated URL path: %s", match[1])
if !strings.HasSuffix(match[1], "/img.jxl") {
t.Errorf("generated URL %s does not end in /img.jxl", match[1])
}
imageRec := httptest.NewRecorder()
imageSrv.ServeHTTP(imageRec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, match[1], nil))
requireJPEGXL(t, imageRec)
})
}
}
-73
View File
@@ -1,73 +0,0 @@
package handlers
import (
"log/slog"
"net/http"
"testing"
"github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imgcache"
)
// jxlType is the content type of JPEG XL.
const jxlType = "image/jxl"
// TestFormatForAccept_JPEGXL verifies that the format auto chooses JPEG XL
// when Accept names image/jxl, ahead of AVIF whatever their q, and AVIF when
// Accept refuses JPEG XL with q=0.
func TestFormatForAccept_JPEGXL(t *testing.T) {
t.Parallel()
tests := []struct {
name string
accept string
want imgcache.ImageFormat
}{
{"JPEG XL-capable browser",
"image/jxl,image/avif,image/webp,image/*,*/*;q=0.8", imgcache.FormatJXL},
{"JPEG XL only", jxlType, imgcache.FormatJXL},
{"JPEG XL with a lower q than AVIF", "image/avif,image/jxl;q=0.5",
imgcache.FormatJXL},
{"q=0 on JPEG XL", "image/jxl;q=0,image/avif,*/*", imgcache.FormatAVIF},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := formatForAccept(tt.accept)
if got != tt.want || err != nil {
t.Errorf("formatForAccept(%q) = %q, %v, want %q",
tt.accept, got, err, tt.want)
}
})
}
}
// TestFormatAuto_JPEGXL requests an auto URL on each image route with an
// Accept header that names image/jxl, and checks that the answer is a JPEG XL
// image.
func TestFormatAuto_JPEGXL(t *testing.T) {
t.Parallel()
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
signedURL, encryptedURL := autoPhotoURLs(t, h)
for _, target := range []string{signedURL, encryptedURL} {
rec := requestImage(t, srv, http.MethodGet, target,
"image/jxl,image/avif,image/webp,*/*;q=0.8")
gotType := rec.Header().Get("Content-Type")
if rec.Code != http.StatusOK || gotType != jxlType {
t.Errorf("%s: %d %s, want 200 %s; body %s",
target, rec.Code, gotType, jxlType, rec.Body)
continue
}
if got := vips.DetermineImageType(rec.Body.Bytes()); got != vips.ImageTypeJXL {
t.Errorf("%s: body is %s, want jxl", target, vips.ImageTypes[got])
}
}
}
-2
View File
@@ -45,7 +45,6 @@ const (
contentTypeGIF = "image/gif" contentTypeGIF = "image/gif"
contentTypeWebP = "image/webp" contentTypeWebP = "image/webp"
contentTypeAVIF = "image/avif" contentTypeAVIF = "image/avif"
contentTypeJXL = "image/jxl"
contentTypeSVG = "image/svg+xml" contentTypeSVG = "image/svg+xml"
contentTypeOctetStream = "application/octet-stream" contentTypeOctetStream = "application/octet-stream"
) )
@@ -157,7 +156,6 @@ func DefaultConfig() *Config {
contentTypeGIF, contentTypeGIF,
contentTypeWebP, contentTypeWebP,
contentTypeAVIF, contentTypeAVIF,
contentTypeJXL,
contentTypeSVG, contentTypeSVG,
}, },
AllowHTTP: false, AllowHTTP: false,
@@ -1,20 +0,0 @@
package httpfetcher
import "testing"
// TestJPEGXLContentType verifies that the fetcher accepts an upstream answer
// of type image/jxl by default, and that the mock fetcher serves a .jxl file
// as image/jxl.
func TestJPEGXLContentType(t *testing.T) {
t.Parallel()
const jxlType = "image/jxl"
if !New(DefaultConfig()).isAllowedContentType(jxlType) {
t.Errorf("isAllowedContentType(%q) = false, want true", jxlType)
}
if got := detectContentTypeFromPath("images/photo.jxl"); got != jxlType {
t.Errorf("detectContentTypeFromPath(photo.jxl) = %q, want %q", got, jxlType)
}
}
-2
View File
@@ -109,8 +109,6 @@ func detectContentTypeFromPath(path string) string {
return contentTypeWebP return contentTypeWebP
case strings.HasSuffix(path, ".avif"): case strings.HasSuffix(path, ".avif"):
return contentTypeAVIF return contentTypeAVIF
case strings.HasSuffix(path, ".jxl"):
return contentTypeJXL
case strings.HasSuffix(path, ".svg"): case strings.HasSuffix(path, ".svg"):
return contentTypeSVG return contentTypeSVG
default: default:
@@ -1,21 +0,0 @@
package imageprocessor
import (
"bytes"
"errors"
"testing"
)
// TestImageProcessor_EmptyFormatRefused verifies that a request with no format
// is refused, as both image routes give every request a format before it is
// processed.
func TestImageProcessor_EmptyFormatRefused(t *testing.T) {
t.Parallel()
_, err := New(Params{}).Process(
t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)), &Request{},
)
if !errors.Is(err, ErrUnsupportedOutputFormat) {
t.Errorf("Process() error = %v, want %v", err, ErrUnsupportedOutputFormat)
}
}
@@ -1,145 +0,0 @@
package imageprocessor
import (
"bytes"
"image"
"image/color"
"image/png"
"testing"
"github.com/davidbyttow/govips/v2/vips"
)
// processedSize runs input through Process and returns the output's size in
// bytes.
func processedSize(t *testing.T, input []byte, req *Request) int64 {
t.Helper()
result, err := New(Params{}).Process(t.Context(), bytes.NewReader(input), req)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
_ = result.Content.Close()
return result.ContentLength
}
// encodePNG encodes img as PNG with Go's encoder.
func encodePNG(t *testing.T, img image.Image) []byte {
t.Helper()
var buf bytes.Buffer
err := png.Encode(&buf, img)
if err != nil {
t.Fatalf("failed to encode test PNG: %v", err)
}
return buf.Bytes()
}
// decode decodes input with vips, as Process does.
func decode(t *testing.T, input []byte) *vips.ImageRef {
t.Helper()
img, err := vips.NewImageFromBuffer(input)
if err != nil {
t.Fatalf("failed to decode input: %v", err)
}
t.Cleanup(img.Close)
return img
}
// TestImageProcessor_PNGIsCompressed verifies that a PNG output is
// compressed: a flat 200x150 image, 90,000 bytes of raw pixels, comes out at
// a small fraction of that.
func TestImageProcessor_PNGIsCompressed(t *testing.T) {
t.Parallel()
const width, height = 200, 150
flat := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
flat.Set(x, y, color.RGBA{R: 40, G: 120, B: 200, A: 255})
}
}
size := processedSize(t, encodePNG(t, flat), &Request{Format: FormatPNG})
const rawBytes = width * height * 3
if size > rawBytes/10 {
t.Errorf("PNG output is %d bytes, want under a tenth of its %d raw",
size, rawBytes)
}
}
// TestImageProcessor_WebPAtDefaultEffort verifies that WebP is saved at
// libvips' default effort, 4: the output is the size of the same image saved
// at that effort.
func TestImageProcessor_WebPAtDefaultEffort(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, 200, 150)
size := processedSize(t, input, &Request{Format: FormatWebP, Quality: 85})
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
StripMetadata: true,
Quality: 85,
ReductionEffort: 4,
})
if err != nil {
t.Fatalf("ExportWebp() error = %v", err)
}
if size != int64(len(want)) {
t.Errorf("WebP output is %d bytes, want %d, the size at effort 4",
size, len(want))
}
}
// TestImageProcessor_AVIFAtEffort1 verifies that AVIF is saved at effort 1
// with 8 bits per sample: the output is the size of the same image saved
// with those settings. The source is 16-bit, which libvips saves with 12
// bits when it is not given a bit depth, and 640x480: on a small image,
// such as 64x48, efforts 1 and 2 give the same output.
func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
t.Parallel()
const width, height = 640, 480
source := image.NewRGBA64(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
source.Set(x, y, color.RGBA64{
R: uint16((x * 65535 / width) & 0xffff),
G: uint16((y * 65535 / height) & 0xffff),
B: 32768,
A: 65535,
})
}
}
input := encodePNG(t, source)
size := processedSize(t, input, &Request{Format: FormatAVIF, Quality: 85})
want, _, err := decode(t, input).ExportAvif(&vips.AvifExportParams{
StripMetadata: true,
Quality: 85,
Effort: 1,
Bitdepth: 8,
})
if err != nil {
t.Fatalf("ExportAvif() error = %v", err)
}
if size != int64(len(want)) {
t.Errorf("AVIF output is %d bytes, want %d, the size at effort 1 "+
"and 8 bits", size, len(want))
}
}
+45 -188
View File
@@ -65,7 +65,6 @@ const (
FormatPNG Format = "png" FormatPNG Format = "png"
FormatWebP Format = "webp" FormatWebP Format = "webp"
FormatAVIF Format = "avif" FormatAVIF Format = "avif"
FormatJXL Format = "jxl"
FormatGIF Format = "gif" FormatGIF Format = "gif"
) )
@@ -256,9 +255,9 @@ func (p *ImageProcessor) Process(
} }
} }
// orig is the source's own format; encode refuses an empty format // Determine output format
outputFormat := req.Format outputFormat := req.Format
if outputFormat == FormatOriginal { if outputFormat == FormatOriginal || outputFormat == "" {
outputFormat = p.formatFromString(inputFormat) outputFormat = p.formatFromString(inputFormat)
} }
@@ -306,7 +305,6 @@ const (
mimeGIF = "image/gif" mimeGIF = "image/gif"
mimeWebP = "image/webp" mimeWebP = "image/webp"
mimeAVIF = "image/avif" mimeAVIF = "image/avif"
mimeJXL = "image/jxl"
) )
// SupportedInputFormats returns MIME types this processor can read. // SupportedInputFormats returns MIME types this processor can read.
@@ -317,7 +315,6 @@ func (p *ImageProcessor) SupportedInputFormats() []string {
mimeGIF, mimeGIF,
mimeWebP, mimeWebP,
mimeAVIF, mimeAVIF,
mimeJXL,
} }
} }
@@ -329,7 +326,6 @@ func (p *ImageProcessor) SupportedOutputFormats() []Format {
FormatGIF, FormatGIF,
FormatWebP, FormatWebP,
FormatAVIF, FormatAVIF,
FormatJXL,
} }
} }
@@ -346,8 +342,6 @@ func FormatToMIME(format Format) string {
return mimeGIF return mimeGIF
case FormatAVIF: case FormatAVIF:
return mimeAVIF return mimeAVIF
case FormatJXL:
return mimeJXL
case FormatOriginal: case FormatOriginal:
return "application/octet-stream" return "application/octet-stream"
default: default:
@@ -417,11 +411,9 @@ func (p *ImageProcessor) detectFormat(img *vips.ImageRef) string {
return "webp" return "webp"
case vips.ImageTypeAVIF, vips.ImageTypeHEIF: case vips.ImageTypeAVIF, vips.ImageTypeHEIF:
return string(FormatAVIF) return string(FormatAVIF)
case vips.ImageTypeJXL:
return string(FormatJXL)
case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF, case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF,
vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP, vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP,
vips.ImageTypeJP2K: vips.ImageTypeJP2K, vips.ImageTypeJXL:
return "unknown" return "unknown"
default: default:
return "unknown" return "unknown"
@@ -493,6 +485,44 @@ func (p *ImageProcessor) encode(
quality = defaultQuality quality = defaultQuality
} }
var params vips.ExportParams
switch format {
case FormatJPEG:
params = vips.ExportParams{
Format: vips.ImageTypeJPEG,
Quality: quality,
}
case FormatPNG:
params = vips.ExportParams{
Format: vips.ImageTypePNG,
}
case FormatGIF:
params = vips.ExportParams{
Format: vips.ImageTypeGIF,
}
case FormatWebP:
params = vips.ExportParams{
Format: vips.ImageTypeWEBP,
Quality: quality,
}
case FormatAVIF:
params = vips.ExportParams{
Format: vips.ImageTypeAVIF,
Quality: quality,
}
case FormatOriginal:
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
default:
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
}
// Stripping drops the ICC profile as well, and clients show an image // Stripping drops the ICC profile as well, and clients show an image
// with no profile as sRGB, so convert to sRGB first. "srgb" names // with no profile as sRGB, so convert to sRGB first. "srgb" names
// libvips' built-in profile; govips' own sRGB path variable is set on // libvips' built-in profile; govips' own sRGB path variable is set on
@@ -504,166 +534,11 @@ func (p *ImageProcessor) encode(
} }
} }
switch format { // Drop EXIF, XMP, IPTC and the ICC profile. govips ignores this for
case FormatJPEG: // GIF, which carries none of them.
return exportJPEG(img, quality) params.StripMetadata = true
case FormatPNG: output, _, err := img.Export(&params)
return exportPNG(img)
case FormatGIF:
return exportGIF(img)
case FormatWebP:
return exportWebP(img, quality)
case FormatAVIF:
return exportAVIF(img, quality)
case FormatJXL:
return exportJXL(img, quality)
case FormatOriginal:
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
default:
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
}
}
// govips sends libvips Go's zero value for some settings an export leaves
// out, such as no compression at all for PNG, so each export below sets
// every setting whose zero value is not what pixa wants. Stripping metadata
// drops EXIF, XMP, IPTC and the ICC profile.
// exportJPEG encodes img as JPEG at quality, without metadata. The settings
// it leaves out are at libvips' defaults.
func exportJPEG(img *vips.ImageRef, quality int) ([]byte, error) {
output, _, err := img.ExportJpeg(&vips.JpegExportParams{
StripMetadata: true,
Quality: quality,
})
return output, err
}
// pngCompression is libvips' default PNG compression, from 0 (none) to 9.
const pngCompression = 6
// exportPNG encodes img as PNG at libvips' default compression and row
// filter, without metadata.
func exportPNG(img *vips.ImageRef) ([]byte, error) {
output, _, err := img.ExportPng(&vips.PngExportParams{
StripMetadata: true,
Compression: pngCompression,
Filter: vips.PngFilterNone,
})
return output, err
}
// gifEffort is libvips' default GIF effort, from 1 to 10.
const gifEffort = 7
// exportGIF encodes img as GIF at libvips' default effort. govips cannot
// have libvips strip metadata from GIF, which carries none.
func exportGIF(img *vips.ImageRef) ([]byte, error) {
output, _, err := img.ExportGIF(&vips.GifExportParams{Effort: gifEffort})
return output, err
}
// webpEffort is libvips' default WebP effort, from 0 (fastest) to 6.
const webpEffort = 4
// exportWebP encodes img as lossy WebP at quality and libvips' default
// effort, without metadata.
func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
output, _, err := img.ExportWebp(&vips.WebpExportParams{
StripMetadata: true,
Quality: quality,
ReductionEffort: webpEffort,
})
return output, err
}
// avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest
// govips can set. With one thread, as pixad runs libvips, 1 takes about 51
// seconds to save an 8192x8192 image of random pixels, the worst case,
// against the default downstream_timeout of 60 seconds. On an image of
// milder noise, which 1 saves in about 12 seconds, 2 takes nearly a minute
// and libvips' default, 4, takes minutes.
const avifEffort = 1
// avifBitdepth is the AVIF bit depth, 8 bits per sample for every image.
// libvips would save a 16-bit image with 12, but at avifEffort that takes
// about 54 seconds for a 16-bit 8192x8192 image of milder noise, nearly all
// of the default downstream_timeout, and about 12 seconds with 8.
const avifBitdepth = 8
// exportAVIF encodes img as lossy AVIF at quality, avifEffort and
// avifBitdepth, without metadata.
func exportAVIF(img *vips.ImageRef, quality int) ([]byte, error) {
output, _, err := img.ExportAvif(&vips.AvifExportParams{
StripMetadata: true,
Quality: quality,
Effort: avifEffort,
Bitdepth: avifBitdepth,
})
return output, err
}
// jxlResolution is the resolution every JPEG XL image is saved with, in
// pixels per millimetre as libvips counts it: 72 dpi, what libvips gives a
// JPEG that names none.
const jxlResolution = 72 / 25.4
// exportJXL encodes img as JPEG XL at quality, with libvips' default effort
// and without metadata. govips sends libvips a distance, the JPEG XL
// encoder's own measure of quality, along with the quality, and libvips then
// uses the distance alone, so the quality is also given as a distance.
func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
// libvips converts a CMYK image to sRGB before it saves WebP, AVIF or
// PNG, but cannot save one as JPEG XL. encode has already converted
// any image with an ICC profile to sRGB, so this is CMYK with none.
if img.Interpretation() == vips.InterpretationCMYK {
err := img.ToColorSpace(vips.InterpretationSRGB)
if err != nil {
return nil, fmt.Errorf("failed to convert CMYK to sRGB: %w", err)
}
}
// govips cannot make libvips strip metadata from JPEG XL, so it is
// removed from the image itself. RemoveMetadata removes EXIF, XMP and
// IPTC but keeps the ICC profile.
err := img.RemoveMetadata()
if err != nil {
return nil, err
}
err = img.RemoveICCProfile()
if err != nil {
return nil, err
}
// libvips 8.16 and later still write an EXIF block of their own, from
// the image's size, orientation and resolution, and fixed values. The
// image is upright, so its orientation is 1, but its resolution is still
// the source's.
toSave, err := img.CopyChangingResolution(jxlResolution, jxlResolution)
if err != nil {
return nil, err
}
defer toSave.Close()
params := vips.NewJxlExportParams()
params.Quality = quality
params.Distance = jxlDistance(quality)
output, _, err := toSave.ExportJxl(params)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -671,22 +546,6 @@ func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
return output, nil return output, nil
} }
// jxlDistance turns a quality from 1 to 100 into the JPEG XL encoder's
// distance, with the formula libvips and libjxl use for their own quality
// setting, except that 100 stays lossy (distance 0.1) where libjxl makes it
// lossless.
//
//nolint:mnd // the constants of that formula
func jxlDistance(quality int) float64 {
q := float64(quality)
if quality >= 30 {
return 0.1 + (100-q)*0.09
}
return 53.0/3000.0*q*q - 23.0/20.0*q + 25.0
}
// formatFromString converts a format string to Format. // formatFromString converts a format string to Format.
func (p *ImageProcessor) formatFromString(format string) Format { func (p *ImageProcessor) formatFromString(format string) Format {
switch format { switch format {
@@ -700,8 +559,6 @@ func (p *ImageProcessor) formatFromString(format string) Format {
return FormatWebP return FormatWebP
case string(FormatAVIF): case string(FormatAVIF):
return FormatAVIF return FormatAVIF
case string(FormatJXL):
return FormatJXL
default: default:
return FormatJPEG return FormatJPEG
} }
@@ -1,244 +0,0 @@
package imageprocessor
import (
"bytes"
"io"
"math"
"os"
"testing"
"github.com/davidbyttow/govips/v2/vips"
)
// TestImageProcessor_EncodeJPEGXL converts a JPEG to a smaller JPEG XL and
// checks the content type, and the format and size the output loads as.
func TestImageProcessor_EncodeJPEGXL(t *testing.T) {
t.Parallel()
req := &Request{Size: Size{Width: 100, Height: 75}, Format: FormatJXL}
result, err := New(Params{}).Process(
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)), req,
)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
if result.ContentType != "image/jxl" {
t.Errorf("ContentType = %q, want image/jxl", result.ContentType)
}
data, err := io.ReadAll(result.Content)
if err != nil {
t.Fatalf("failed to read result: %v", err)
}
output, err := vips.NewImageFromBuffer(data)
if err != nil {
t.Fatalf("failed to load the output: %v", err)
}
defer output.Close()
if output.Format() != vips.ImageTypeJXL {
t.Errorf("output format = %s, want jxl", vips.ImageTypes[output.Format()])
}
if output.Width() != 100 || output.Height() != 75 {
t.Errorf("output size = %dx%d, want 100x75", output.Width(), output.Height())
}
}
// TestImageProcessor_JPEGXLQuality verifies that the quality reaches the JPEG
// XL encoder: the same image comes out smaller at quality 30 than at 90.
func TestImageProcessor_JPEGXLQuality(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, 400, 300)
outputBytes := make(map[int]int64)
for _, quality := range []int{30, 90} {
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input), &Request{Format: FormatJXL, Quality: quality})
if err != nil {
t.Fatalf("Process() at quality %d error = %v", quality, err)
}
_ = result.Content.Close()
outputBytes[quality] = result.ContentLength
}
if outputBytes[30] >= outputBytes[90] {
t.Errorf("%d bytes at quality 30, %d at 90, want fewer at 30",
outputBytes[30], outputBytes[90])
}
}
// TestImageProcessor_JPEGXLDropsSourceEXIF verifies that none of the source's
// EXIF reaches a JPEG XL output. libvips 8.16 and later write an EXIF block of
// their own into JPEG XL (orientation, resolution, size, colour space and
// fixed defaults), and govips cannot ask them to leave it out, so the test
// looks for the source's fields rather than for no EXIF at all.
func TestImageProcessor_JPEGXLDropsSourceEXIF(t *testing.T) {
t.Parallel()
input, err := os.ReadFile("testdata/gps-exif.jpg")
if err != nil {
t.Fatalf("failed to read test JPEG: %v", err)
}
exif := processAndDecode(t, input, &Request{Format: FormatJXL}).GetExif()
for _, field := range []string{
"exif-ifd0-Make", "exif-ifd0-Model", "exif-ifd2-BodySerialNumber",
"exif-ifd2-DateTimeOriginal", "exif-ifd3-GPSLatitude",
"exif-ifd3-GPSLongitude",
} {
if value, found := exif[field]; found {
t.Errorf("output has %s: %s", field, value)
}
}
}
// TestImageProcessor_JPEGXLDropsSourceResolution verifies that the source's
// resolution does not reach the EXIF block libvips 8.16 and later write into
// JPEG XL. A JPEG XL image holds its resolution in that block alone, so the
// resolution the output loads with is the block's.
func TestImageProcessor_JPEGXLDropsSourceResolution(t *testing.T) {
t.Parallel()
// dpi-300.jpg is a flat 8x8 grey image with a resolution of 300 dpi.
input, err := os.ReadFile("testdata/dpi-300.jpg")
if err != nil {
t.Fatalf("failed to read test JPEG: %v", err)
}
output := processAndDecode(t, input, &Request{Format: FormatJXL})
// libvips gives the resolution in pixels per millimetre.
xDPI := math.Round(output.ResX() * 25.4)
yDPI := math.Round(output.ResY() * 25.4)
if xDPI == 300 || yDPI == 300 {
t.Errorf("output resolution = %vx%v dpi, the source's", xDPI, yDPI)
}
}
// TestImageProcessor_JPEGXLAppliesEXIFOrientation verifies that a JPEG XL
// output is turned upright, as TestImageProcessor_AppliesEXIFOrientation does
// for PNG.
func TestImageProcessor_JPEGXLAppliesEXIFOrientation(t *testing.T) {
t.Parallel()
// orientation-6.jpg is stored 16x8, red on the left and blue on the
// right, with EXIF orientation 6 (turn 90 degrees clockwise to view).
// Upright it is 8x16, red on top and blue below.
input, err := os.ReadFile("testdata/orientation-6.jpg")
if err != nil {
t.Fatalf("failed to read test JPEG: %v", err)
}
output := processAndDecode(t, input, &Request{Format: FormatJXL})
if output.Width() != 8 || output.Height() != 16 {
t.Fatalf("output is %dx%d, want 8x16", output.Width(), output.Height())
}
top, err := output.GetPoint(4, 0)
if err != nil {
t.Fatalf("GetPoint() error = %v", err)
}
bottom, err := output.GetPoint(4, 15)
if err != nil {
t.Fatalf("GetPoint() error = %v", err)
}
if top[0] <= top[2] || bottom[2] <= bottom[0] {
t.Errorf("top pixel = %v, bottom pixel = %v, want red above blue",
top, bottom)
}
}
// TestImageProcessor_JPEGXLConvertsWideGamutToSRGB verifies that a JPEG XL
// output is converted to sRGB, as TestImageProcessor_ConvertsWideGamutToSRGB
// does for PNG. It does not check for an ICC profile, as libvips reports one
// for every JPEG XL image it loads.
func TestImageProcessor_JPEGXLConvertsWideGamutToSRGB(t *testing.T) {
t.Parallel()
// display-p3.jpg is a flat 8x8 image with the Display P3 profile
// embedded, filled with Display P3 (234, 51, 35), which is sRGB red.
input, err := os.ReadFile("testdata/display-p3.jpg")
if err != nil {
t.Fatalf("failed to read test JPEG: %v", err)
}
output := processAndDecode(t, input, &Request{Format: FormatJXL})
pixel, err := output.GetPoint(4, 4)
if err != nil {
t.Fatalf("GetPoint() error = %v", err)
}
want := []float64{255, 0, 0}
for i := range want {
if math.Abs(pixel[i]-want[i]) > 5 {
t.Fatalf("pixel = %v, want within 5 of %v", pixel, want)
}
}
}
// TestImageProcessor_JPEGXLFromCMYK verifies that a CMYK JPEG with no ICC
// profile can be served as JPEG XL, in sRGB.
func TestImageProcessor_JPEGXLFromCMYK(t *testing.T) {
t.Parallel()
// cmyk.jpg is a flat 8x8 CMYK image with no ICC profile, filled with
// full cyan and no magenta, yellow or black.
input, err := os.ReadFile("testdata/cmyk.jpg")
if err != nil {
t.Fatalf("failed to read test JPEG: %v", err)
}
output := processAndDecode(t, input, &Request{Format: FormatJXL})
if output.Bands() != 3 {
t.Fatalf("output has %d bands, want 3", output.Bands())
}
pixel, err := output.GetPoint(4, 4)
if err != nil {
t.Fatalf("GetPoint() error = %v", err)
}
// Cyan in sRGB: little red, much green and blue.
if pixel[0] > 50 || pixel[1] < 100 || pixel[2] < 200 {
t.Errorf("pixel = %v, want cyan", pixel)
}
}
// TestJXLDistance verifies the JPEG XL distance for a few qualities: 90 is
// distance 1, the encoder's own default, and 100 stays lossy.
func TestJXLDistance(t *testing.T) {
t.Parallel()
tests := []struct {
quality int
want float64
}{
{quality: 100, want: 0.1},
{quality: 90, want: 1},
{quality: 30, want: 6.4},
{quality: 20, want: 9.0667},
}
for _, tt := range tests {
got := jxlDistance(tt.quality)
if math.Abs(got-tt.want) > 0.0001 {
t.Errorf("jxlDistance(%d) = %v, want %v", tt.quality, got, tt.want)
}
}
}
Binary file not shown.

Before

Width:  |  Height:  |  Size: 352 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 799 B

+8 -71
View File
@@ -11,8 +11,6 @@ import (
"io" "io"
"log/slog" "log/slog"
"path/filepath" "path/filepath"
"sync"
"sync/atomic"
"time" "time"
lru "github.com/hashicorp/golang-lru/v2" lru "github.com/hashicorp/golang-lru/v2"
@@ -81,26 +79,6 @@ type Cache struct {
evictionDone chan struct{} evictionDone chan struct{}
evictionCancel context.CancelFunc evictionCancel context.CancelFunc
// The pending counts: hits, misses, upstream fetches and transforms
// whose write to the database missed its request's deadline, kept
// here until they are written by the goroutine that
// StartPendingCountWrites starts. As for eviction, the channels are
// created in NewCache: pendingCountsAdded wakes that goroutine, and
// pendingCountsDone is closed when it returns. pendingCountsCancel,
// set by StartPendingCountWrites, cancels its context, which tells it
// to finish.
pendingHits atomic.Int64
pendingMisses atomic.Int64
pendingUpstreamFetches atomic.Int64
pendingUpstreamFetchBytes atomic.Int64
pendingTransforms atomic.Int64
pendingCountsAdded chan struct{}
pendingCountsDone chan struct{}
pendingCountsCancel context.CancelFunc
// Held through each write of the pending counts, and by Stats to read them
pendingCountsWriteMutex sync.Mutex
// metaCache holds the content types of the variants most recently // metaCache holds the content types of the variants most recently
// stored or served, so a hit does not read the variant's .meta file. // stored or served, so a hit does not read the variant's .meta file.
// It never stands in for the variant file, which is always opened. // It never stands in for the variant file, which is always opened.
@@ -161,9 +139,6 @@ func newCache(
metaCache: metaCache, metaCache: metaCache,
contentLocks: newContentLock(), contentLocks: newContentLock(),
pendingCountsAdded: make(chan struct{}, 1),
pendingCountsDone: make(chan struct{}),
reconciliationPageSize: defaultReconciliationPageSize, reconciliationPageSize: defaultReconciliationPageSize,
} }
@@ -499,21 +474,12 @@ func (c *Cache) CleanExpired(ctx context.Context) error {
func (c *Cache) Stats(ctx context.Context) (*CacheStats, error) { func (c *Cache) Stats(ctx context.Context) (*CacheStats, error) {
var stats CacheStats var stats CacheStats
// So that no write of the pending counts falls between the two reads
c.pendingCountsWriteMutex.Lock()
// Fetch hit/miss counts from the stats table // Fetch hit/miss counts from the stats table
err := c.db.QueryRowContext(ctx, ` err := c.db.QueryRowContext(ctx, `
SELECT hit_count, miss_count SELECT hit_count, miss_count
FROM cache_stats WHERE id = 1 FROM cache_stats WHERE id = 1
`).Scan(&stats.HitCount, &stats.MissCount) `).Scan(&stats.HitCount, &stats.MissCount)
// Hits and misses not yet written count too (see StartPendingCountWrites)
stats.HitCount += c.pendingHits.Load()
stats.MissCount += c.pendingMisses.Load()
c.pendingCountsWriteMutex.Unlock()
if err != nil && !errors.Is(err, sql.ErrNoRows) { if err != nil && !errors.Is(err, sql.ErrNoRows) {
return nil, fmt.Errorf("failed to get cache stats: %w", err) return nil, fmt.Errorf("failed to get cache stats: %w", err)
} }
@@ -545,30 +511,19 @@ func (c *Cache) Stats(ctx context.Context) (*CacheStats, error) {
} }
// IncrementStats counts a cache hit or miss, and an upstream fetch that read // IncrementStats counts a cache hit or miss, and an upstream fetch that read
// fetchBytes bytes, as IncrementUpstreamFetch does. Like the other Increment // fetchBytes bytes, as IncrementUpstreamFetch does.
// methods, it writes to the database with ctx's deadline but not its
// cancellation, so an ended request is still counted but does not wait for
// the database past its deadline; a count not written by then becomes a
// pending count (see StartPendingCountWrites).
func (c *Cache) IncrementStats(ctx context.Context, hit bool, fetchBytes int64) { func (c *Cache) IncrementStats(ctx context.Context, hit bool, fetchBytes int64) {
countCtx, cancel := withoutCancelKeepingDeadline(ctx)
defer cancel()
var err error var err error
pendingCount := &c.pendingMisses
if hit { if hit {
pendingCount = &c.pendingHits _, err = c.db.ExecContext(ctx, `
_, err = c.db.ExecContext(countCtx, `
UPDATE cache_stats UPDATE cache_stats
SET hit_count = hit_count + 1, SET hit_count = hit_count + 1,
last_updated_at = CURRENT_TIMESTAMP last_updated_at = CURRENT_TIMESTAMP
WHERE id = 1 WHERE id = 1
`) `)
} else { } else {
_, err = c.db.ExecContext(countCtx, ` _, err = c.db.ExecContext(ctx, `
UPDATE cache_stats UPDATE cache_stats
SET miss_count = miss_count + 1, SET miss_count = miss_count + 1,
last_updated_at = CURRENT_TIMESTAMP last_updated_at = CURRENT_TIMESTAMP
@@ -576,10 +531,7 @@ func (c *Cache) IncrementStats(ctx context.Context, hit bool, fetchBytes int64)
`) `)
} }
switch { if err != nil {
case errors.Is(err, context.DeadlineExceeded):
c.addPendingCount(pendingCount, 1)
case err != nil:
c.log.Warn("failed to count cache hit or miss", "hit", hit, "error", err) c.log.Warn("failed to count cache hit or miss", "hit", hit, "error", err)
} }
@@ -593,22 +545,14 @@ func (c *Cache) IncrementUpstreamFetch(ctx context.Context, fetchBytes int64) {
return return
} }
countCtx, cancel := withoutCancelKeepingDeadline(ctx) _, err := c.db.ExecContext(ctx, `
defer cancel()
_, err := c.db.ExecContext(countCtx, `
UPDATE cache_stats UPDATE cache_stats
SET upstream_fetch_count = upstream_fetch_count + 1, SET upstream_fetch_count = upstream_fetch_count + 1,
upstream_fetch_bytes = upstream_fetch_bytes + ?, upstream_fetch_bytes = upstream_fetch_bytes + ?,
last_updated_at = CURRENT_TIMESTAMP last_updated_at = CURRENT_TIMESTAMP
WHERE id = 1 WHERE id = 1
`, fetchBytes) `, fetchBytes)
if err != nil {
switch {
case errors.Is(err, context.DeadlineExceeded):
c.addPendingCount(&c.pendingUpstreamFetches, 1)
c.addPendingCount(&c.pendingUpstreamFetchBytes, fetchBytes)
case err != nil:
c.log.Warn("failed to count upstream fetch", c.log.Warn("failed to count upstream fetch",
"fetch_bytes", fetchBytes, "error", err) "fetch_bytes", fetchBytes, "error", err)
} }
@@ -616,20 +560,13 @@ func (c *Cache) IncrementUpstreamFetch(ctx context.Context, fetchBytes int64) {
// IncrementTransformCount counts one image transcoded by the image processor. // IncrementTransformCount counts one image transcoded by the image processor.
func (c *Cache) IncrementTransformCount(ctx context.Context) { func (c *Cache) IncrementTransformCount(ctx context.Context) {
countCtx, cancel := withoutCancelKeepingDeadline(ctx) _, err := c.db.ExecContext(ctx, `
defer cancel()
_, err := c.db.ExecContext(countCtx, `
UPDATE cache_stats UPDATE cache_stats
SET transform_count = transform_count + 1, SET transform_count = transform_count + 1,
last_updated_at = CURRENT_TIMESTAMP last_updated_at = CURRENT_TIMESTAMP
WHERE id = 1 WHERE id = 1
`) `)
if err != nil {
switch {
case errors.Is(err, context.DeadlineExceeded):
c.addPendingCount(&c.pendingTransforms, 1)
case err != nil:
c.log.Warn("failed to count transform", "error", err) c.log.Warn("failed to count transform", "error", err)
} }
} }
+3 -4
View File
@@ -21,12 +21,11 @@ const (
FormatPNG ImageFormat = "png" FormatPNG ImageFormat = "png"
FormatWebP ImageFormat = "webp" FormatWebP ImageFormat = "webp"
FormatAVIF ImageFormat = "avif" FormatAVIF ImageFormat = "avif"
FormatJXL ImageFormat = "jxl"
FormatGIF ImageFormat = "gif" FormatGIF ImageFormat = "gif"
// FormatAuto stands for JPEG XL, AVIF, WebP or JPEG, chosen for each // FormatAuto stands for AVIF, WebP or JPEG, chosen for each request
// request from its Accept header once the URL's signature or token has // from its Accept header once the URL's signature or token has been
// been checked; it is never processed or cached as itself. // checked; it is never processed or cached as itself.
FormatAuto ImageFormat = "auto" FormatAuto ImageFormat = "auto"
) )
-141
View File
@@ -1,141 +0,0 @@
package imgcache
import (
"context"
"fmt"
"sync/atomic"
)
// StartPendingCountWrites starts the goroutine that writes the pending
// counts to the database whenever there are some, one UPDATE at a time:
// however many requests pass their deadline before their counts are
// written, at most this one write waits for the database. It is a no-op
// when already started. The goroutine outlives the caller, so it runs with
// its own context, which StopPendingCountWrites cancels to tell it to
// finish.
func (c *Cache) StartPendingCountWrites() {
if c.pendingCountsCancel != nil {
return
}
ctx, cancel := context.WithCancel(context.Background())
c.pendingCountsCancel = cancel
go func() {
c.pendingCountWriteLoop(ctx)
}()
}
// StopPendingCountWrites tells the goroutine StartPendingCountWrites
// started, if it did, to finish, and waits for its write under way to end
// and for it to return. It then writes the pending counts left, so that
// they are written before the database closes. It waits for all of this at
// most until ctx ends, then logs the counts not written and returns an
// error.
func (c *Cache) StopPendingCountWrites(ctx context.Context) error {
var err error
if c.pendingCountsCancel != nil {
c.pendingCountsCancel()
select {
case <-c.pendingCountsDone:
case <-ctx.Done():
// The write under way is left to finish: the database's
// close waits for a query under way.
err = fmt.Errorf("pending counts still being written: %w", ctx.Err())
}
}
if err == nil {
err = c.writePendingCounts(ctx)
}
if err != nil {
c.log.Error("counts not written at shutdown",
"hits", c.pendingHits.Load(),
"misses", c.pendingMisses.Load(),
"upstream_fetches", c.pendingUpstreamFetches.Load(),
"upstream_fetch_bytes", c.pendingUpstreamFetchBytes.Load(),
"transforms", c.pendingTransforms.Load(),
"error", err,
)
return err
}
return nil
}
// pendingCountWriteLoop is the body of the goroutine
// StartPendingCountWrites starts. It returns when ctx is cancelled, but
// does not cut off a write under way then. A write that fails leaves the
// counts pending, for the next write.
func (c *Cache) pendingCountWriteLoop(ctx context.Context) {
defer close(c.pendingCountsDone)
for {
select {
case <-ctx.Done():
return
case <-c.pendingCountsAdded:
}
err := c.writePendingCounts(context.WithoutCancel(ctx))
if err != nil {
c.log.Warn("failed to write pending counts", "error", err)
}
}
}
// addPendingCount adds n to pendingCount, one of the pending counts, and
// wakes the goroutine that writes them. pendingCountsAdded has capacity one,
// so a wakeup already waiting is enough.
func (c *Cache) addPendingCount(pendingCount *atomic.Int64, n int64) {
pendingCount.Add(n)
select {
case c.pendingCountsAdded <- struct{}{}:
default:
}
}
// writePendingCounts adds the pending counts to the cache_stats row in one
// UPDATE, then takes what it wrote out of them, leaving any added
// meanwhile. When the UPDATE fails, they all stay pending.
func (c *Cache) writePendingCounts(ctx context.Context) error {
c.pendingCountsWriteMutex.Lock()
defer c.pendingCountsWriteMutex.Unlock()
hits := c.pendingHits.Load()
misses := c.pendingMisses.Load()
upstreamFetches := c.pendingUpstreamFetches.Load()
upstreamFetchBytes := c.pendingUpstreamFetchBytes.Load()
transforms := c.pendingTransforms.Load()
if hits+misses+upstreamFetches+upstreamFetchBytes+transforms == 0 {
return nil
}
_, err := c.db.ExecContext(ctx, `
UPDATE cache_stats
SET hit_count = hit_count + ?,
miss_count = miss_count + ?,
upstream_fetch_count = upstream_fetch_count + ?,
upstream_fetch_bytes = upstream_fetch_bytes + ?,
transform_count = transform_count + ?,
last_updated_at = CURRENT_TIMESTAMP
WHERE id = 1
`, hits, misses, upstreamFetches, upstreamFetchBytes, transforms)
if err != nil {
return fmt.Errorf("failed to write pending counts: %w", err)
}
c.pendingHits.Add(-hits)
c.pendingMisses.Add(-misses)
c.pendingUpstreamFetches.Add(-upstreamFetches)
c.pendingUpstreamFetchBytes.Add(-upstreamFetchBytes)
c.pendingTransforms.Add(-transforms)
return nil
}
@@ -1,316 +0,0 @@
package imgcache
import (
"context"
"database/sql"
"errors"
"testing"
"time"
)
// pendingCountsWait is how long a test waits for the database's connection,
// for a count to reach the database, for a write to start waiting for the
// connection, or for StopPendingCountWrites to return.
const pendingCountsWait = 5 * time.Second
// holdDatabase takes the one connection of cache's database, so that every
// other query waits for it, and returns the func that frees it.
func holdDatabase(t *testing.T, cache *Cache) func() {
t.Helper()
ctx, cancel := context.WithTimeout(t.Context(), pendingCountsWait)
defer cancel()
conn, err := cache.db.Conn(ctx)
if err != nil {
t.Fatalf("failed to take the database connection: %v", err)
}
return func() { _ = conn.Close() }
}
// waitForConnectionWaits waits until db.Stats().WaitCount, the number of
// times a caller has waited for a connection, reaches waits.
func waitForConnectionWaits(t *testing.T, db *sql.DB, waits int64) {
t.Helper()
deadline := time.Now().Add(pendingCountsWait)
for db.Stats().WaitCount < waits {
if time.Now().After(deadline) {
t.Fatalf("callers waited for the database connection %d times, want %d",
db.Stats().WaitCount, waits)
}
time.Sleep(10 * time.Millisecond)
}
}
// waitForCounters waits until the cache_stats row holds want.
func waitForCounters(t *testing.T, cache *Cache, want cacheStatsCounters) {
t.Helper()
deadline := time.Now().Add(pendingCountsWait)
for {
got := readCacheStatsCounters(t, cache)
if got == want {
return
}
if time.Now().After(deadline) {
t.Fatalf("counters = %+v, want %+v", got, want)
}
time.Sleep(10 * time.Millisecond)
}
}
// TestService_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy holds the
// database's one connection while a request whose fetch is held reaches its
// deadline. The request must still return by its deadline with the
// deadline's error, and its miss must reach the database once the
// connection is free.
func TestService_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t)
svc.cache.StartPendingCountWrites()
defer func() { _ = svc.cache.StopPendingCountWrites(t.Context()) }()
// Room for the request to reach its fetch on a busy host
const timeout = 2 * time.Second
ctx, cancel := context.WithTimeout(t.Context(), timeout)
defer cancel()
deadline, _ := ctx.Deadline()
results := startGet(ctx, svc, photoVariant(fixtures, 85, FitCover))
// The request has made its database reads by the time it fetches.
select {
case <-fetcher.started:
case <-time.After(timeout):
t.Fatal("request did not reach its fetch by its deadline")
}
releaseDatabase := holdDatabase(t, svc.cache)
defer releaseDatabase()
select {
case got := <-results:
t.Logf("Get() returned %v after its deadline, error = %v",
time.Since(deadline), got.err)
if !errors.Is(got.err, context.DeadlineExceeded) {
t.Errorf("Get() error = %v, want %v", got.err, context.DeadlineExceeded)
}
case <-time.After(time.Until(deadline) + time.Second):
t.Fatal("request did not return by its deadline while the database was busy")
}
releaseDatabase()
waitForCounters(t, svc.cache, cacheStatsCounters{missCount: 1})
}
// TestIncrementStats_ManyCountsPastTheirDeadlineLeaveOneWriteWaiting holds the
// database's one connection while many misses are counted past their
// deadline. At most one write may then wait for the connection, and every
// miss must reach the database once it is free.
func TestIncrementStats_ManyCountsPastTheirDeadlineLeaveOneWriteWaiting(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
cache.StartPendingCountWrites()
defer func() { _ = cache.StopPendingCountWrites(t.Context()) }()
releaseDatabase := holdDatabase(t, cache)
defer releaseDatabase()
waitsBefore := cache.db.Stats().WaitCount
// Writes given a context past its deadline fail at once.
ended, cancel := context.WithDeadline(t.Context(), time.Now())
defer cancel()
const misses = 20
for range misses {
cache.IncrementStats(ended, false, 0)
}
// Once one write waits for the connection, any others start waiting too.
waitForConnectionWaits(t, cache.db, waitsBefore+1)
time.Sleep(arrivalWait)
waits := cache.db.Stats().WaitCount - waitsBefore
t.Logf("%d writes waited for the database", waits)
if waits > 1 {
t.Errorf("%d writes waited for the database, want at most 1", waits)
}
releaseDatabase()
waitForCounters(t, cache, cacheStatsCounters{missCount: misses})
}
// TestStats_IncludesPendingCounts counts hits and misses whose writes miss
// their deadline, and checks that Stats adds them to the database's counts
// before they are written.
func TestStats_IncludesPendingCounts(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
_, err := cache.db.ExecContext(t.Context(),
`UPDATE cache_stats SET hit_count = 75, miss_count = 25 WHERE id = 1`)
if err != nil {
t.Fatal(err)
}
// Writes given a context past its deadline fail at once.
ended, cancel := context.WithDeadline(t.Context(), time.Now())
defer cancel()
cache.IncrementStats(ended, true, 0)
cache.IncrementStats(ended, true, 0)
cache.IncrementStats(ended, false, 0)
stats, err := cache.Stats(t.Context())
if err != nil {
t.Fatalf("Stats() error = %v", err)
}
if stats.HitCount != 77 || stats.MissCount != 26 {
t.Errorf("HitCount = %d, MissCount = %d, want 77 and 26",
stats.HitCount, stats.MissCount)
}
want := cacheStatsCounters{hitCount: 75, missCount: 25}
if got := readCacheStatsCounters(t, cache); got != want {
t.Errorf("counters in the database = %+v, want %+v", got, want)
}
}
// TestStopPendingCountWrites_WritesPendingCounts holds the database's one
// connection while the goroutine that writes the pending counts waits for it,
// counts more, then stops that goroutine. The stop must leave the write under
// way to finish rather than cut it off, and every count must reach the
// database once the connection is free.
func TestStopPendingCountWrites_WritesPendingCounts(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
cache.StartPendingCountWrites()
defer func() { _ = cache.StopPendingCountWrites(t.Context()) }()
releaseDatabase := holdDatabase(t, cache)
defer releaseDatabase()
waitsBefore := cache.db.Stats().WaitCount
// Writes given a context past its deadline fail at once.
ended, cancel := context.WithDeadline(t.Context(), time.Now())
defer cancel()
cache.IncrementStats(ended, true, 0)
// The goroutine's write waits for the connection.
waitForConnectionWaits(t, cache.db, waitsBefore+1)
// Counted after that write read the pending counts
cache.IncrementStats(ended, false, 0)
cache.IncrementUpstreamFetch(ended, 1024)
cache.IncrementTransformCount(ended)
stopped := make(chan error, 1)
go func() {
stopped <- cache.StopPendingCountWrites(t.Context())
}()
// Had the stop cut off the write under way, its own write would wait for
// the connection too.
time.Sleep(arrivalWait)
waits := cache.db.Stats().WaitCount - waitsBefore
t.Logf("%d writes waited for the database", waits)
if waits > 1 {
t.Errorf("%d writes waited for the database, want 1: "+
"the stop cut off the write under way", waits)
}
releaseDatabase()
select {
case err := <-stopped:
if err != nil {
t.Fatalf("StopPendingCountWrites() error = %v", err)
}
case <-time.After(pendingCountsWait):
t.Fatal("StopPendingCountWrites() did not return once the database was free")
}
want := cacheStatsCounters{1, 1, 1, 1024, 1}
if got := readCacheStatsCounters(t, cache); got != want {
t.Errorf("counters = %+v, want %+v", got, want)
}
}
// TestStopPendingCountWrites_ReturnsWhenItsContextEnds holds the database's
// one connection while the goroutine that writes the pending counts waits for
// it, then stops that goroutine with a context that ends first. The stop must
// return the context's error, and the write under way must still reach the
// database once the connection is free.
func TestStopPendingCountWrites_ReturnsWhenItsContextEnds(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
cache.StartPendingCountWrites()
defer func() { _ = cache.StopPendingCountWrites(t.Context()) }()
releaseDatabase := holdDatabase(t, cache)
defer releaseDatabase()
waitsBefore := cache.db.Stats().WaitCount
// Writes given a context past its deadline fail at once.
ended, cancel := context.WithDeadline(t.Context(), time.Now())
defer cancel()
cache.IncrementStats(ended, true, 0)
// The goroutine's write waits for the connection.
waitForConnectionWaits(t, cache.db, waitsBefore+1)
stopCtx, cancelStop := context.WithCancel(t.Context())
cancelStop()
stopped := make(chan error, 1)
go func() {
stopped <- cache.StopPendingCountWrites(stopCtx)
}()
select {
case err := <-stopped:
if !errors.Is(err, context.Canceled) {
t.Errorf("StopPendingCountWrites() error = %v, want %v",
err, context.Canceled)
}
case <-time.After(pendingCountsWait):
t.Fatal("StopPendingCountWrites() did not return once its context ended")
}
releaseDatabase()
waitForCounters(t, cache, cacheStatsCounters{hitCount: 1})
}
+14 -22
View File
@@ -100,8 +100,8 @@ func NewService(cfg *ServiceConfig) (*Service, error) {
allowHTTP = cfg.FetcherConfig.AllowHTTP allowHTTP = cfg.FetcherConfig.AllowHTTP
} }
// JPEG XL is the default output format, so pixad does not start // JPEG XL is to become the default output format, so pixad does not
// without it. // start without it.
err := imageprocessor.CheckJPEGXLSupport() err := imageprocessor.CheckJPEGXLSupport()
if err != nil { if err != nil {
return nil, err return nil, err
@@ -162,7 +162,7 @@ func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, e
// Fall through to re-process // Fall through to re-process
} else { } else {
// Counted also when the request context has ended meanwhile // Counted also when the request context has ended meanwhile
s.cache.IncrementStats(ctx, true, 0) s.cache.IncrementStats(context.WithoutCancel(ctx), true, 0)
return &ImageResponse{ return &ImageResponse{
Content: reader, Content: reader,
@@ -179,7 +179,7 @@ func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, e
// failed or the request context has ended meanwhile // failed or the request context has ended meanwhile
response, err := s.processOrWait(ctx, req) response, err := s.processOrWait(ctx, req)
s.cache.IncrementStats(ctx, false, 0) s.cache.IncrementStats(context.WithoutCancel(ctx), false, 0)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -300,8 +300,14 @@ func (s *Service) processOrWait(
} }
}() }()
processingCtx, cancel := withoutCancelKeepingDeadline(ctx) processingCtx := context.WithoutCancel(ctx)
defer cancel()
if deadline, ok := ctx.Deadline(); ok {
var cancel context.CancelFunc
processingCtx, cancel = context.WithDeadline(processingCtx, deadline)
defer cancel()
}
return s.processFromSourceOrFetch(processingCtx, req, cacheKey) return s.processFromSourceOrFetch(processingCtx, req, cacheKey)
}) })
@@ -338,20 +344,6 @@ func (s *Service) processOrWait(
}, nil }, nil
} }
// withoutCancelKeepingDeadline returns ctx without its cancellation but with
// its deadline, if it has one, and the func that releases the returned
// context.
func withoutCancelKeepingDeadline(
ctx context.Context,
) (context.Context, context.CancelFunc) {
deadline, hasDeadline := ctx.Deadline()
if !hasDeadline {
return context.WithoutCancel(ctx), func() {}
}
return context.WithDeadline(context.WithoutCancel(ctx), deadline)
}
// loadCachedSource opens source content from cache, without reading it, and // loadCachedSource opens source content from cache, without reading it, and
// returns it with its size; nil if the cached data is unavailable, empty or // returns it with its size; nil if the cached data is unavailable, empty or
// exceeds maxResponseSize. // exceeds maxResponseSize.
@@ -461,7 +453,7 @@ func (s *Service) fetchAndProcess(
fetchBytes := int64(len(sourceData)) fetchBytes := int64(len(sourceData))
// Counted also when the request context has ended meanwhile // Counted also when the request context has ended meanwhile
s.cache.IncrementUpstreamFetch(ctx, fetchBytes) s.cache.IncrementUpstreamFetch(context.WithoutCancel(ctx), fetchBytes)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read upstream response: %w", err) return nil, fmt.Errorf("failed to read upstream response: %w", err)
@@ -543,7 +535,7 @@ func (s *Service) processAndStore(
processDuration := time.Since(processStart) processDuration := time.Since(processStart)
// Counted also when the request context has ended meanwhile // Counted also when the request context has ended meanwhile
s.cache.IncrementTransformCount(ctx) s.cache.IncrementTransformCount(context.WithoutCancel(ctx))
// Read processed content // Read processed content
processedData, err := io.ReadAll(processResult.Content) processedData, err := io.ReadAll(processResult.Content)
+9 -21
View File
@@ -38,10 +38,8 @@ func ValidateDimension(name string, value int) error {
return nil return nil
} }
// sizeFormatRegex matches patterns like "800x600.webp", "0x0.jpeg", "orig.png", // sizeFormatRegex matches patterns like "800x600.webp", "0x0.jpeg", "orig.png"
// and a size with no format, such as "800x600" or "orig" var sizeFormatRegex = regexp.MustCompile(`^(\d+)x(\d+)\.(\w+)$|^(orig)\.(\w+)$`)
var sizeFormatRegex = regexp.MustCompile(
`^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`)
// ParsedURL contains the parsed components of an image proxy URL. // ParsedURL contains the parsed components of an image proxy URL.
type ParsedURL struct { type ParsedURL struct {
@@ -58,13 +56,12 @@ type ParsedURL struct {
} }
// ParseImagePath parses the path captured by chi's wildcard: // ParseImagePath parses the path captured by chi's wildcard:
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL // <host>/<path>/<size>.<format>
// This is the primary entry point when using chi routing. // This is the primary entry point when using chi routing.
// Examples: // Examples:
// - cdn.example.com/photos/cat.jpg/800x600.webp // - cdn.example.com/photos/cat.jpg/800x600.webp
// - cdn.example.com/photos/cat.jpg/0x0.jpeg // - cdn.example.com/photos/cat.jpg/0x0.jpeg
// - cdn.example.com/photos/cat.jpg/orig.png // - cdn.example.com/photos/cat.jpg/orig.png
// - cdn.example.com/photos/cat.jpg/800x600
func ParseImagePath(path string) (*ParsedURL, error) { func ParseImagePath(path string) (*ParsedURL, error) {
// Strip leading slash if present (chi may include it) // Strip leading slash if present (chi may include it)
path = strings.TrimPrefix(path, "/") path = strings.TrimPrefix(path, "/")
@@ -75,8 +72,7 @@ func ParseImagePath(path string) (*ParsedURL, error) {
return parseImageComponents(path) return parseImageComponents(path)
} }
// ParseImageURL parses a full URL path like /v1/image/<host>/<path>/<size>.<format>, // ParseImageURL parses a full URL path like /v1/image/<host>/<path>/<size>.<format>
// or /v1/image/<host>/<path>/<size> for JPEG XL
// Use ParseImagePath instead when working with chi's wildcard capture. // Use ParseImagePath instead when working with chi's wildcard capture.
func ParseImageURL(urlPath string) (*ParsedURL, error) { func ParseImageURL(urlPath string) (*ParsedURL, error) {
// Remove the /v1/image/ prefix // Remove the /v1/image/ prefix
@@ -93,8 +89,7 @@ func ParseImageURL(urlPath string) (*ParsedURL, error) {
return parseImageComponents(remainder) return parseImageComponents(remainder)
} }
// parseImageComponents parses <host>/<path>/<size>.<format>, or // parseImageComponents parses <host>/<path>/<size>.<format> structure.
// <host>/<path>/<size> for JPEG XL.
func parseImageComponents(remainder string) (*ParsedURL, error) { func parseImageComponents(remainder string) (*ParsedURL, error) {
// Check for path traversal before any other processing // Check for path traversal before any other processing
err := checkPathTraversal(remainder) err := checkPathTraversal(remainder)
@@ -102,7 +97,7 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
return nil, err return nil, err
} }
// Find the last path segment, which holds "size" or "size.format" // Find the last path segment which contains size.format
lastSlash := strings.LastIndex(remainder, "/") lastSlash := strings.LastIndex(remainder, "/")
if lastSlash == -1 { if lastSlash == -1 {
return nil, ErrMissingSize return nil, ErrMissingSize
@@ -217,7 +212,7 @@ func checkPathTraversal(path string) error {
return nil return nil
} }
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600" // parseSizeFormat parses strings like "800x600.webp" or "orig.png"
func parseSizeFormat(s string) (Size, ImageFormat, error) { func parseSizeFormat(s string) (Size, ImageFormat, error) {
matches := sizeFormatRegex.FindStringSubmatch(s) matches := sizeFormatRegex.FindStringSubmatch(s)
if matches == nil { if matches == nil {
@@ -230,11 +225,11 @@ func parseSizeFormat(s string) (Size, ImageFormat, error) {
) )
if matches[4] == "orig" { if matches[4] == "orig" {
// "orig" or "orig.format" pattern // "orig.format" pattern
size = Size{Width: 0, Height: 0} size = Size{Width: 0, Height: 0}
formatStr = matches[5] formatStr = matches[5]
} else { } else {
// "WxH" or "WxH.format" pattern // "WxH.format" pattern
width, err := strconv.Atoi(matches[1]) width, err := strconv.Atoi(matches[1])
if err != nil { if err != nil {
return Size{}, "", ErrInvalidSize return Size{}, "", ErrInvalidSize
@@ -259,11 +254,6 @@ func parseSizeFormat(s string) (Size, ImageFormat, error) {
return Size{}, "", err return Size{}, "", err
} }
// A URL that names no format is served, and signed, as JPEG XL
if formatStr == "" {
return size, FormatJXL, nil
}
format, err := parseFormat(formatStr) format, err := parseFormat(formatStr)
if err != nil { if err != nil {
return Size{}, "", err return Size{}, "", err
@@ -285,8 +275,6 @@ func parseFormat(s string) (ImageFormat, error) {
return FormatWebP, nil return FormatWebP, nil
case "avif": case "avif":
return FormatAVIF, nil return FormatAVIF, nil
case "jxl":
return FormatJXL, nil
case "gif": case "gif":
return FormatGIF, nil return FormatGIF, nil
case "auto": case "auto":
-58
View File
@@ -1,58 +0,0 @@
package magic
import "testing"
// testMIMEJXL is the MIME type of JPEG XL.
const testMIMEJXL = "image/jxl"
// TestDetectFormatJPEGXL verifies that both JPEG XL signatures, that of a
// bare codestream and that of the container, are detected as image/jxl.
func TestDetectFormatJPEGXL(t *testing.T) {
t.Parallel()
tests := []struct {
name string
data []byte
}{
{"codestream", pad(0xFF, 0x0A)},
{"container", pad(
0x00, 0x00, 0x00, 0x0C, 0x4A, 0x58, 0x4C, 0x20, 0x0D, 0x0A, 0x87, 0x0A,
)},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := DetectFormat(tt.data)
if err != nil || got != MIMETypeJXL {
t.Errorf("DetectFormat() = %q, %v, want %q", got, err, testMIMEJXL)
}
err = ValidateMagicBytes(tt.data, testMIMEJXL)
if err != nil {
t.Errorf("ValidateMagicBytes(%q) error = %v", testMIMEJXL, err)
}
})
}
}
// TestJPEGXLMIMEType verifies that image/jxl is a supported input type and
// maps to and from the format jxl.
func TestJPEGXLMIMEType(t *testing.T) {
t.Parallel()
if !IsSupportedMIMEType(testMIMEJXL) {
t.Errorf("IsSupportedMIMEType(%q) = false", testMIMEJXL)
}
format, ok := MIMEToImageFormat(testMIMEJXL)
if format != FormatJXL || !ok {
t.Errorf("MIMEToImageFormat(%q) = %q, %v, want %q, true",
testMIMEJXL, format, ok, FormatJXL)
}
if got := ImageFormatToMIME(FormatJXL); got != testMIMEJXL {
t.Errorf("ImageFormatToMIME(%q) = %q, want %q", FormatJXL, got, testMIMEJXL)
}
}
+1 -19
View File
@@ -26,7 +26,6 @@ const (
MIMETypeWebP = MIMEType("image/webp") MIMETypeWebP = MIMEType("image/webp")
MIMETypeGIF = MIMEType("image/gif") MIMETypeGIF = MIMEType("image/gif")
MIMETypeAVIF = MIMEType("image/avif") MIMETypeAVIF = MIMEType("image/avif")
MIMETypeJXL = MIMEType("image/jxl")
MIMETypeSVG = MIMEType("image/svg+xml") MIMETypeSVG = MIMEType("image/svg+xml")
) )
@@ -41,7 +40,6 @@ const (
FormatPNG ImageFormat = "png" FormatPNG ImageFormat = "png"
FormatWebP ImageFormat = "webp" FormatWebP ImageFormat = "webp"
FormatAVIF ImageFormat = "avif" FormatAVIF ImageFormat = "avif"
FormatJXL ImageFormat = "jxl"
FormatGIF ImageFormat = "gif" FormatGIF ImageFormat = "gif"
) )
@@ -64,11 +62,6 @@ var (
// AVIF uses the ftyp box with brand "avif" or "avis" // AVIF uses the ftyp box with brand "avif" or "avis"
// Format: size(4 bytes) + "ftyp" + brand(4 bytes) // Format: size(4 bytes) + "ftyp" + brand(4 bytes)
magicFtyp = []byte{0x66, 0x74, 0x79, 0x70} // "ftyp" magicFtyp = []byte{0x66, 0x74, 0x79, 0x70} // "ftyp"
// JPEG XL is a bare codestream or a codestream in a container
magicJXLCodestream = []byte{0xFF, 0x0A}
magicJXLContainer = []byte{
0x00, 0x00, 0x00, 0x0C, 0x4A, 0x58, 0x4C, 0x20, 0x0D, 0x0A, 0x87, 0x0A,
}
) )
// WebP identifier appears at offset 8 after RIFF header. // WebP identifier appears at offset 8 after RIFF header.
@@ -122,12 +115,6 @@ func DetectFormat(data []byte) (MIMEType, error) {
} }
} }
// Check JPEG XL (FF0A, or the 12-byte container signature)
if bytes.HasPrefix(data, magicJXLCodestream) ||
bytes.HasPrefix(data, magicJXLContainer) {
return MIMETypeJXL, nil
}
// Check SVG - look for XML declaration or SVG tag // Check SVG - look for XML declaration or SVG tag
if detectSVG(data) { if detectSVG(data) {
return MIMETypeSVG, nil return MIMETypeSVG, nil
@@ -194,8 +181,7 @@ func normalizeMIMEType(mimeType string) string {
func IsSupportedMIMEType(mimeType string) bool { func IsSupportedMIMEType(mimeType string) bool {
normalized := normalizeMIMEType(mimeType) normalized := normalizeMIMEType(mimeType)
switch MIMEType(normalized) { switch MIMEType(normalized) {
case MIMETypeJPEG, MIMETypePNG, MIMETypeWebP, MIMETypeGIF, MIMETypeAVIF, case MIMETypeJPEG, MIMETypePNG, MIMETypeWebP, MIMETypeGIF, MIMETypeAVIF, MIMETypeSVG:
MIMETypeJXL, MIMETypeSVG:
return true return true
default: default:
return false return false
@@ -240,8 +226,6 @@ func MIMEToImageFormat(mimeType string) (ImageFormat, bool) {
return FormatGIF, true return FormatGIF, true
case MIMETypeAVIF: case MIMETypeAVIF:
return FormatAVIF, true return FormatAVIF, true
case MIMETypeJXL:
return FormatJXL, true
case MIMETypeSVG: case MIMETypeSVG:
// SVG has no corresponding output format. // SVG has no corresponding output format.
return "", false return "", false
@@ -263,8 +247,6 @@ func ImageFormatToMIME(format ImageFormat) string {
return string(MIMETypeGIF) return string(MIMETypeGIF)
case FormatAVIF: case FormatAVIF:
return string(MIMETypeAVIF) return string(MIMETypeAVIF)
case FormatJXL:
return string(MIMETypeJXL)
case FormatOriginal: case FormatOriginal:
// Original format passes content through unchanged. // Original format passes content through unchanged.
return mimeOctetStream return mimeOctetStream
-72
View File
@@ -1,72 +0,0 @@
package server
import (
"net/http"
"net/http/httptest"
"os"
"testing"
"github.com/davidbyttow/govips/v2/vips"
)
// TestImageProxyFlowJPEGXL requests the JPEG XL testdata/red.jxl, 64x48 and
// made with libvips, through pixa's router, upstream fetcher, image processor
// and disk cache: resized as PNG, resized as JPEG XL, and at its own size and
// format, which is JPEG XL. The second request for each URL is a cache hit.
func TestImageProxyFlowJPEGXL(t *testing.T) {
t.Parallel()
source, err := os.ReadFile("testdata/red.jxl")
if err != nil {
t.Fatalf("reading the test image: %v", err)
}
upstream := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "image/jxl")
_, _ = w.Write(source)
}))
t.Cleanup(upstream.Close)
s, _, _ := startImageProxy(t, upstream)
tests := []struct {
sizeFormat string // the <size>.<format> part of the image URL
contentType string
imageType vips.ImageType
width, height int
}{
{"32x24.png", "image/png", vips.ImageTypePNG, 32, 24},
{"32x24.jxl", "image/jxl", vips.ImageTypeJXL, 32, 24},
{"orig.orig", "image/jxl", vips.ImageTypeJXL, 64, 48},
}
for _, tt := range tests {
target := "/v1/image/" + upstreamHost + "/red.jxl/" + tt.sizeFormat
for _, wantCache := range []string{"MISS", "HIT"} {
rec := getImage(t, s, target)
gotType := rec.Header().Get("Content-Type")
gotCache := rec.Header().Get("X-Pixa-Cache")
if gotType != tt.contentType || gotCache != wantCache {
t.Errorf("%s: %s, X-Pixa-Cache %s, want %s, %s",
target, gotType, gotCache, tt.contentType, wantCache)
}
img, err := vips.NewImageFromBuffer(rec.Body.Bytes())
if err != nil {
t.Fatalf("%s: loading the image: %v", target, err)
}
if img.Format() != tt.imageType ||
img.Width() != tt.width || img.Height() != tt.height {
t.Errorf("%s: %s %dx%d, want %s %dx%d", target,
vips.ImageTypes[img.Format()], img.Width(), img.Height(),
vips.ImageTypes[tt.imageType], tt.width, tt.height)
}
img.Close()
}
}
}
+1 -2
View File
@@ -89,8 +89,7 @@ func (s *Server) SetupRoutes() {
r.Use(s.refuseDuringMaintenance) r.Use(s.refuseDuringMaintenance)
// Main image proxy route // Main image proxy route
// /v1/image/<host>/<path>/<width>x<height>.<format>, or with no // /v1/image/<host>/<path>/<width>x<height>.<format>
// format /v1/image/<host>/<path>/<width>x<height>
r.Get("/image/*", s.h.HandleImage()) r.Get("/image/*", s.h.HandleImage())
r.Head("/image/*", s.h.HandleImage()) r.Head("/image/*", s.h.HandleImage())
Binary file not shown.
+1 -2
View File
@@ -95,12 +95,11 @@
</label> </label>
<select id="format" name="format"> <select id="format" name="format">
<option value="orig" {{if eq .FormFormat "orig"}}selected{{end}}>Original</option> <option value="orig" {{if eq .FormFormat "orig"}}selected{{end}}>Original</option>
<option value="auto" {{if eq .FormFormat "auto"}}selected{{end}}>Auto (JPEG XL, AVIF, WebP or JPEG)</option> <option value="auto" {{if eq .FormFormat "auto"}}selected{{end}}>Auto (AVIF, WebP or JPEG)</option>
<option value="jpeg" {{if eq .FormFormat "jpeg"}}selected{{end}}>JPEG</option> <option value="jpeg" {{if eq .FormFormat "jpeg"}}selected{{end}}>JPEG</option>
<option value="png" {{if eq .FormFormat "png"}}selected{{end}}>PNG</option> <option value="png" {{if eq .FormFormat "png"}}selected{{end}}>PNG</option>
<option value="webp" {{if eq .FormFormat "webp"}}selected{{end}}>WebP</option> <option value="webp" {{if eq .FormFormat "webp"}}selected{{end}}>WebP</option>
<option value="avif" {{if eq .FormFormat "avif"}}selected{{end}}>AVIF</option> <option value="avif" {{if eq .FormFormat "avif"}}selected{{end}}>AVIF</option>
<option value="jxl" {{if or (eq .FormFormat "jxl") (eq .FormFormat "")}}selected{{end}}>JPEG XL</option>
<option value="gif" {{if eq .FormFormat "gif"}}selected{{end}}>GIF</option> <option value="gif" {{if eq .FormFormat "gif"}}selected{{end}}>GIF</option>
</select> </select>
</div> </div>