Compare commits
2
Commits
ff17f5cca8
...
c465049e67
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c465049e67 | ||
|
|
d2944aa891 |
No files matched your search
@@ -245,7 +245,7 @@ A request whose query string cannot be decoded, or gives any parameter more than
|
|||||||
once, is refused with 400.
|
once, is refused with 400.
|
||||||
|
|
||||||
- `<format>`: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`,
|
- `<format>`: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`,
|
||||||
`avif`, `gif`, or `auto` (below)
|
`avif`, `jxl` (JPEG XL), `gif`, or `auto` (below)
|
||||||
- `<size>`: `orig` or `<width>x<height>` (e.g. `800x600`)
|
- `<size>`: `orig` or `<width>x<height>` (e.g. `800x600`)
|
||||||
- `sig` and `exp`: the signature and its expiry, needed unless the host is
|
- `sig` and `exp`: the signature and its expiry, needed unless the host is
|
||||||
allowlisted (see Signature Specification)
|
allowlisted (see Signature Specification)
|
||||||
@@ -253,23 +253,28 @@ once, is refused with 400.
|
|||||||
both optional (values under Signature Specification). Both are part of what is
|
both optional (values under Signature Specification). Both are part of what is
|
||||||
cached, so each value of either is a separate cached image.
|
cached, so each value of either is a separate cached image.
|
||||||
|
|
||||||
|
The source image may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The upstream's
|
||||||
|
`Content-Type` must name its format, and the image's first bytes must match it.
|
||||||
|
|
||||||
With the format `auto`, pixa chooses the format for each request from its
|
With the format `auto`, pixa chooses the format for each request from its
|
||||||
`Accept` header, in this order:
|
`Accept` header, in this order:
|
||||||
|
|
||||||
1. AVIF, when the header names `image/avif`;
|
1. JPEG XL, when the header names `image/jxl`;
|
||||||
2. WebP, when it names `image/webp`;
|
2. AVIF, when it names `image/avif`;
|
||||||
3. JPEG, when the first of `image/jpeg`, `image/*` and `*/*` that it names
|
3. WebP, when it names `image/webp`;
|
||||||
|
4. JPEG, when the first of `image/jpeg`, `image/*` and `*/*` that it names
|
||||||
allows it, or when there is no `Accept` header or it is empty.
|
allows it, or when there is no `Accept` header or it is empty.
|
||||||
|
|
||||||
An entry with `q=0` refuses its format; other `q` values do not change the
|
An entry with `q=0` refuses its format; other `q` values do not change the
|
||||||
order. AVIF and WebP must be named, as clients that cannot show them also send
|
order. JPEG XL, AVIF and WebP must be named, as clients that cannot show them
|
||||||
`image/*` and `*/*`. pixa never sends a format the client refused: when the
|
also send `image/*` and `*/*`. pixa never sends a format the client refused:
|
||||||
header allows none of the three, the answer is 406, and a header that does not
|
when the header allows none of the four, the answer is 406, and a header that
|
||||||
parse, or has a `q` that is not a number from 0 to 1, is refused with 400. The
|
does not parse, or has a `q` that is not a number from 0 to 1, is refused
|
||||||
signature, or the token of an encrypted URL, covers `auto` itself, so one URL
|
with 400. The signature, or the token of an encrypted URL, covers `auto` itself,
|
||||||
serves every client. Each format chosen is cached as a separate image, and every
|
so one URL serves every client. Each format chosen is cached as a separate
|
||||||
answer that depends on `Accept` (the image, a 304, and the 400 and 406 above)
|
image, and every answer that depends on `Accept` (the image, a 304, and the 400
|
||||||
carries `Vary: Accept`, so a shared cache keeps the formats apart too.
|
and 406 above) carries `Vary: Accept`, so a shared cache keeps the formats apart
|
||||||
|
too.
|
||||||
|
|
||||||
An image is served with `Cache-Control: public, max-age=<seconds>, immutable`.
|
An image is served with `Cache-Control: public, max-age=<seconds>, immutable`.
|
||||||
When the URL has an expiry (an `exp`, or the TTL of an encrypted URL), `max-age`
|
When the URL has an expiry (an `exp`, or the TTL of an encrypted URL), `max-age`
|
||||||
@@ -350,6 +355,10 @@ turned off.
|
|||||||
- An image with an ICC profile is converted to sRGB first, since clients show an
|
- An image with an ICC profile is converted to sRGB first, since clients show an
|
||||||
image with no profile as sRGB. Colours outside sRGB, such as the most
|
image with no profile as sRGB. Colours outside sRGB, such as the most
|
||||||
saturated ones in a Display P3 photo, are clipped.
|
saturated ones in a Display P3 photo, are clipped.
|
||||||
|
- The one exception is JPEG XL: a JPEG XL image carries an EXIF block that
|
||||||
|
libvips writes itself, with the image's orientation, resolution, size and
|
||||||
|
colour space and fixed defaults, but nothing from the source's EXIF, as govips
|
||||||
|
cannot ask libvips to leave it out.
|
||||||
|
|
||||||
### Source Hosts
|
### Source Hosts
|
||||||
|
|
||||||
@@ -533,12 +542,10 @@ Key settings in more detail:
|
|||||||
- `db_url` — the SQLite database to open; omitted, it is
|
- `db_url` — the SQLite database to open; omitted, it is
|
||||||
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
|
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
|
||||||
database in WAL mode. pixa opens one connection to it, so its own reads and
|
database in WAL mode. pixa opens one connection to it, so its own reads and
|
||||||
writes run one at a time. Requests wait while eviction runs one of its
|
writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to any
|
||||||
queries, some of which read a whole table. pixa adds
|
`db_url`, so a write that finds another program writing to the file waits up
|
||||||
`_pragma=busy_timeout(5000)` to any `db_url`, so a write that finds another
|
to five seconds for it instead of failing. WAL mode comes only from the URL:
|
||||||
program writing to the file waits up to five seconds for it instead of
|
keep `_pragma=journal_mode(WAL)` in one you set
|
||||||
failing. WAL mode comes only from the URL: keep `_pragma=journal_mode(WAL)` in
|
|
||||||
one you set
|
|
||||||
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
|
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
|
||||||
cache entirely; omitted defaults to 75% of the sum of the free space on the
|
cache entirely; omitted defaults to 75% of the sum of the free space on the
|
||||||
filesystem containing `<state_dir>/cache/` and the bytes of source and
|
filesystem containing `<state_dir>/cache/` and the bytes of source and
|
||||||
|
|||||||
@@ -30,6 +30,27 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-08 JPEG XL as an input and output format (part of #222): a source
|
||||||
|
whose bytes start with either JPEG XL signature, the bare codestream's `FF 0A`
|
||||||
|
or the container's, is detected as `image/jxl`, which the upstream fetch
|
||||||
|
accepts; `orig` of such a source is JPEG XL. The format `jxl` works in plain
|
||||||
|
and encrypted URLs and on the generator page, served as `image/jxl` and cached
|
||||||
|
like the other formats. `auto` chooses JPEG XL first when `Accept` names
|
||||||
|
`image/jxl`. govips sends libvips a JPEG XL distance with every save, so
|
||||||
|
libvips ignores the quality: pixa turns `q` into a distance with libvips' own
|
||||||
|
formula, keeping 100 lossy, and removes the metadata from the image before
|
||||||
|
saving, as govips cannot have libvips strip it from JPEG XL. libvips still
|
||||||
|
writes an EXIF block of its own into JPEG XL (orientation, resolution, size,
|
||||||
|
colour space, fixed defaults), with nothing from the source. libvips' default
|
||||||
|
effort, 7, is kept. JPEG XL is not yet the default output.
|
||||||
|
- 2026-10-08 requests no longer wait behind eviction queries that read a whole
|
||||||
|
table (closes #227): the new `cache_usage` table holds the total cache usage,
|
||||||
|
kept up to date by triggers on `source_content` and `variant_content` in the
|
||||||
|
statement that adds, removes or resizes a row, and `UsageBytes` reads it. The
|
||||||
|
reconciliation pass reads the content tables 1000 rows per query, sums them
|
||||||
|
and corrects the total when it differs, unless a row changed while it summed.
|
||||||
|
Source rows get `last_accessed_at` when added, so choosing source images to
|
||||||
|
evict reads that column's index instead of sorting the whole table.
|
||||||
- 2026-10-08 libvips' JPEG XL support is installed and required (part of #222):
|
- 2026-10-08 libvips' JPEG XL support is installed and required (part of #222):
|
||||||
`script/bootstrap --cgo` installs `vips-jxl` when its package manager is apk,
|
`script/bootstrap --cgo` installs `vips-jxl` when its package manager is apk,
|
||||||
as Alpine's `vips` package lacks the support, and the runtime stage of the
|
as Alpine's `vips` package lacks the support, and the runtime stage of the
|
||||||
|
|||||||
@@ -3,14 +3,12 @@
|
|||||||
|
|
||||||
-- Source content blobs
|
-- Source content blobs
|
||||||
-- Files stored at: cache/sources/<ab>/<cd>/<sha256>
|
-- Files stored at: cache/sources/<ab>/<cd>/<sha256>
|
||||||
-- last_accessed_at is NULL until the first LRU touch; eviction falls
|
|
||||||
-- back to fetched_at for rows that have never been touched.
|
|
||||||
CREATE TABLE IF NOT EXISTS source_content (
|
CREATE TABLE IF NOT EXISTS source_content (
|
||||||
content_hash TEXT PRIMARY KEY,
|
content_hash TEXT PRIMARY KEY,
|
||||||
content_type TEXT NOT NULL,
|
content_type TEXT NOT NULL,
|
||||||
size_bytes INTEGER NOT NULL,
|
size_bytes INTEGER NOT NULL,
|
||||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
last_accessed_at DATETIME
|
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||||
);
|
);
|
||||||
CREATE INDEX IF NOT EXISTS idx_source_content_last_accessed
|
CREATE INDEX IF NOT EXISTS idx_source_content_last_accessed
|
||||||
ON source_content(last_accessed_at);
|
ON source_content(last_accessed_at);
|
||||||
@@ -42,9 +40,8 @@ CREATE INDEX IF NOT EXISTS idx_source_meta_content_hash ON source_metadata(conte
|
|||||||
-- Processed variant blobs
|
-- Processed variant blobs
|
||||||
-- Files stored at: cache/variants/<ab>/<cd>/<cache_key> (plus a .meta
|
-- Files stored at: cache/variants/<ab>/<cd>/<cache_key> (plus a .meta
|
||||||
-- sidecar with the content type). Tracked here (like source content
|
-- sidecar with the content type). Tracked here (like source content
|
||||||
-- blobs above) so total cache usage can be computed with a SUM query,
|
-- blobs above) so total cache usage is known without a directory scan,
|
||||||
-- never a directory scan, and so LRU eviction has a timestamp to order
|
-- and so LRU eviction has a timestamp to order on.
|
||||||
-- on.
|
|
||||||
CREATE TABLE IF NOT EXISTS variant_content (
|
CREATE TABLE IF NOT EXISTS variant_content (
|
||||||
cache_key TEXT PRIMARY KEY,
|
cache_key TEXT PRIMARY KEY,
|
||||||
size_bytes INTEGER NOT NULL,
|
size_bytes INTEGER NOT NULL,
|
||||||
@@ -55,6 +52,74 @@ CREATE TABLE IF NOT EXISTS variant_content (
|
|||||||
CREATE INDEX IF NOT EXISTS idx_variant_content_last_accessed
|
CREATE INDEX IF NOT EXISTS idx_variant_content_last_accessed
|
||||||
ON variant_content(last_accessed_at);
|
ON variant_content(last_accessed_at);
|
||||||
|
|
||||||
|
-- Total cache usage: the sum of size_bytes over source_content and
|
||||||
|
-- variant_content, kept by the triggers below in the same statement
|
||||||
|
-- that adds, removes or resizes a row, so eviction reads this one row
|
||||||
|
-- instead of summing both tables. Each trigger also adds one to
|
||||||
|
-- change_count; the reconciliation pass, which sums both tables a page
|
||||||
|
-- at a time, corrects total_size_bytes only if change_count did not
|
||||||
|
-- move while it summed.
|
||||||
|
CREATE TABLE IF NOT EXISTS cache_usage (
|
||||||
|
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||||
|
total_size_bytes INTEGER NOT NULL DEFAULT 0,
|
||||||
|
change_count INTEGER NOT NULL DEFAULT 0
|
||||||
|
);
|
||||||
|
INSERT OR IGNORE INTO cache_usage (id) VALUES (1);
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS source_content_usage_insert
|
||||||
|
AFTER INSERT ON source_content
|
||||||
|
BEGIN
|
||||||
|
UPDATE cache_usage
|
||||||
|
SET total_size_bytes = total_size_bytes + NEW.size_bytes,
|
||||||
|
change_count = change_count + 1
|
||||||
|
WHERE id = 1;
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS source_content_usage_delete
|
||||||
|
AFTER DELETE ON source_content
|
||||||
|
BEGIN
|
||||||
|
UPDATE cache_usage
|
||||||
|
SET total_size_bytes = total_size_bytes - OLD.size_bytes,
|
||||||
|
change_count = change_count + 1
|
||||||
|
WHERE id = 1;
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS source_content_usage_update
|
||||||
|
AFTER UPDATE OF size_bytes ON source_content
|
||||||
|
BEGIN
|
||||||
|
UPDATE cache_usage
|
||||||
|
SET total_size_bytes = total_size_bytes - OLD.size_bytes + NEW.size_bytes,
|
||||||
|
change_count = change_count + 1
|
||||||
|
WHERE id = 1;
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS variant_content_usage_insert
|
||||||
|
AFTER INSERT ON variant_content
|
||||||
|
BEGIN
|
||||||
|
UPDATE cache_usage
|
||||||
|
SET total_size_bytes = total_size_bytes + NEW.size_bytes,
|
||||||
|
change_count = change_count + 1
|
||||||
|
WHERE id = 1;
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS variant_content_usage_delete
|
||||||
|
AFTER DELETE ON variant_content
|
||||||
|
BEGIN
|
||||||
|
UPDATE cache_usage
|
||||||
|
SET total_size_bytes = total_size_bytes - OLD.size_bytes,
|
||||||
|
change_count = change_count + 1
|
||||||
|
WHERE id = 1;
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS variant_content_usage_update
|
||||||
|
AFTER UPDATE OF size_bytes ON variant_content
|
||||||
|
BEGIN
|
||||||
|
UPDATE cache_usage
|
||||||
|
SET total_size_bytes = total_size_bytes - OLD.size_bytes + NEW.size_bytes,
|
||||||
|
change_count = change_count + 1
|
||||||
|
WHERE id = 1;
|
||||||
|
END;
|
||||||
|
|
||||||
-- Output/transformed content blobs
|
-- Output/transformed content blobs
|
||||||
-- Not written: transformed images are stored in cache/variants and
|
-- Not written: transformed images are stored in cache/variants and
|
||||||
-- tracked in variant_content above.
|
-- tracked in variant_content above.
|
||||||
|
|||||||
@@ -14,8 +14,8 @@ import (
|
|||||||
// Errors for an Accept header that an auto URL cannot be served for.
|
// Errors for an Accept header that an auto URL cannot be served for.
|
||||||
var (
|
var (
|
||||||
errInvalidAccept = errors.New("invalid Accept header")
|
errInvalidAccept = errors.New("invalid Accept header")
|
||||||
errNotAcceptable = errors.New(
|
errNotAcceptable = errors.New("not acceptable: auto serves " +
|
||||||
"not acceptable: auto serves image/avif, image/webp or image/jpeg")
|
"image/jxl, image/avif, image/webp or image/jpeg")
|
||||||
)
|
)
|
||||||
|
|
||||||
// chooseAutoFormat replaces the format auto in req with the format
|
// chooseAutoFormat replaces the format auto in req with the format
|
||||||
@@ -51,11 +51,11 @@ func (s *Handlers) chooseAutoFormat(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// formatForAccept returns the format an auto URL is served in for the Accept
|
// formatForAccept returns the format an auto URL is served in for the Accept
|
||||||
// header accept: AVIF when it names image/avif, else WebP when it names
|
// header accept: JPEG XL when it names image/jxl, else AVIF when it names
|
||||||
// image/webp, else JPEG when its most specific entry of image/jpeg, image/*
|
// image/avif, else WebP when it names image/webp, else JPEG when its most
|
||||||
// and */* allows it, or when it names nothing. A q of 0 refuses a format.
|
// specific entry of image/jpeg, image/* and */* allows it, or when it names
|
||||||
// AVIF and WebP must be named, as clients that cannot show them send image/*
|
// nothing. A q of 0 refuses a format. JPEG XL, AVIF and WebP must be named, as
|
||||||
// and */* too.
|
// clients that cannot show them send image/* and */* too.
|
||||||
func formatForAccept(accept string) (imgcache.ImageFormat, error) {
|
func formatForAccept(accept string) (imgcache.ImageFormat, error) {
|
||||||
qualities, err := parseAccept(accept)
|
qualities, err := parseAccept(accept)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -66,6 +66,10 @@ func formatForAccept(accept string) (imgcache.ImageFormat, error) {
|
|||||||
return imgcache.FormatJPEG, nil
|
return imgcache.FormatJPEG, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if qualities["image/jxl"] > 0 {
|
||||||
|
return imgcache.FormatJXL, nil
|
||||||
|
}
|
||||||
|
|
||||||
if qualities["image/avif"] > 0 {
|
if qualities["image/avif"] > 0 {
|
||||||
return imgcache.FormatAVIF, nil
|
return imgcache.FormatAVIF, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/davidbyttow/govips/v2/vips"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/imgcache"
|
||||||
|
)
|
||||||
|
|
||||||
|
// jxlType is the content type of JPEG XL.
|
||||||
|
const jxlType = "image/jxl"
|
||||||
|
|
||||||
|
// TestFormatForAccept_JPEGXL verifies that the format auto chooses JPEG XL
|
||||||
|
// when Accept names image/jxl, ahead of AVIF whatever their q, and AVIF when
|
||||||
|
// Accept refuses JPEG XL with q=0.
|
||||||
|
func TestFormatForAccept_JPEGXL(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
accept string
|
||||||
|
want imgcache.ImageFormat
|
||||||
|
}{
|
||||||
|
{"JPEG XL-capable browser",
|
||||||
|
"image/jxl,image/avif,image/webp,image/*,*/*;q=0.8", imgcache.FormatJXL},
|
||||||
|
{"JPEG XL only", jxlType, imgcache.FormatJXL},
|
||||||
|
{"JPEG XL with a lower q than AVIF", "image/avif,image/jxl;q=0.5",
|
||||||
|
imgcache.FormatJXL},
|
||||||
|
{"q=0 on JPEG XL", "image/jxl;q=0,image/avif,*/*", imgcache.FormatAVIF},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
got, err := formatForAccept(tt.accept)
|
||||||
|
if got != tt.want || err != nil {
|
||||||
|
t.Errorf("formatForAccept(%q) = %q, %v, want %q",
|
||||||
|
tt.accept, got, err, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFormatAuto_JPEGXL requests an auto URL on each image route with an
|
||||||
|
// Accept header that names image/jxl, and checks that the answer is a JPEG XL
|
||||||
|
// image.
|
||||||
|
func TestFormatAuto_JPEGXL(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||||
|
signedURL, encryptedURL := autoPhotoURLs(t, h)
|
||||||
|
|
||||||
|
for _, target := range []string{signedURL, encryptedURL} {
|
||||||
|
rec := requestImage(t, srv, http.MethodGet, target,
|
||||||
|
"image/jxl,image/avif,image/webp,*/*;q=0.8")
|
||||||
|
gotType := rec.Header().Get("Content-Type")
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK || gotType != jxlType {
|
||||||
|
t.Errorf("%s: %d %s, want 200 %s; body %s",
|
||||||
|
target, rec.Code, gotType, jxlType, rec.Body)
|
||||||
|
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := vips.DetermineImageType(rec.Body.Bytes()); got != vips.ImageTypeJXL {
|
||||||
|
t.Errorf("%s: body is %s, want jxl", target, vips.ImageTypes[got])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -45,6 +45,7 @@ const (
|
|||||||
contentTypeGIF = "image/gif"
|
contentTypeGIF = "image/gif"
|
||||||
contentTypeWebP = "image/webp"
|
contentTypeWebP = "image/webp"
|
||||||
contentTypeAVIF = "image/avif"
|
contentTypeAVIF = "image/avif"
|
||||||
|
contentTypeJXL = "image/jxl"
|
||||||
contentTypeSVG = "image/svg+xml"
|
contentTypeSVG = "image/svg+xml"
|
||||||
contentTypeOctetStream = "application/octet-stream"
|
contentTypeOctetStream = "application/octet-stream"
|
||||||
)
|
)
|
||||||
@@ -156,6 +157,7 @@ func DefaultConfig() *Config {
|
|||||||
contentTypeGIF,
|
contentTypeGIF,
|
||||||
contentTypeWebP,
|
contentTypeWebP,
|
||||||
contentTypeAVIF,
|
contentTypeAVIF,
|
||||||
|
contentTypeJXL,
|
||||||
contentTypeSVG,
|
contentTypeSVG,
|
||||||
},
|
},
|
||||||
AllowHTTP: false,
|
AllowHTTP: false,
|
||||||
|
|||||||
@@ -0,0 +1,20 @@
|
|||||||
|
package httpfetcher
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
// TestJPEGXLContentType verifies that the fetcher accepts an upstream answer
|
||||||
|
// of type image/jxl by default, and that the mock fetcher serves a .jxl file
|
||||||
|
// as image/jxl.
|
||||||
|
func TestJPEGXLContentType(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const jxlType = "image/jxl"
|
||||||
|
|
||||||
|
if !New(DefaultConfig()).isAllowedContentType(jxlType) {
|
||||||
|
t.Errorf("isAllowedContentType(%q) = false, want true", jxlType)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := detectContentTypeFromPath("images/photo.jxl"); got != jxlType {
|
||||||
|
t.Errorf("detectContentTypeFromPath(photo.jxl) = %q, want %q", got, jxlType)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -109,6 +109,8 @@ func detectContentTypeFromPath(path string) string {
|
|||||||
return contentTypeWebP
|
return contentTypeWebP
|
||||||
case strings.HasSuffix(path, ".avif"):
|
case strings.HasSuffix(path, ".avif"):
|
||||||
return contentTypeAVIF
|
return contentTypeAVIF
|
||||||
|
case strings.HasSuffix(path, ".jxl"):
|
||||||
|
return contentTypeJXL
|
||||||
case strings.HasSuffix(path, ".svg"):
|
case strings.HasSuffix(path, ".svg"):
|
||||||
return contentTypeSVG
|
return contentTypeSVG
|
||||||
default:
|
default:
|
||||||
|
|||||||
@@ -65,6 +65,7 @@ const (
|
|||||||
FormatPNG Format = "png"
|
FormatPNG Format = "png"
|
||||||
FormatWebP Format = "webp"
|
FormatWebP Format = "webp"
|
||||||
FormatAVIF Format = "avif"
|
FormatAVIF Format = "avif"
|
||||||
|
FormatJXL Format = "jxl"
|
||||||
FormatGIF Format = "gif"
|
FormatGIF Format = "gif"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -305,6 +306,7 @@ const (
|
|||||||
mimeGIF = "image/gif"
|
mimeGIF = "image/gif"
|
||||||
mimeWebP = "image/webp"
|
mimeWebP = "image/webp"
|
||||||
mimeAVIF = "image/avif"
|
mimeAVIF = "image/avif"
|
||||||
|
mimeJXL = "image/jxl"
|
||||||
)
|
)
|
||||||
|
|
||||||
// SupportedInputFormats returns MIME types this processor can read.
|
// SupportedInputFormats returns MIME types this processor can read.
|
||||||
@@ -315,6 +317,7 @@ func (p *ImageProcessor) SupportedInputFormats() []string {
|
|||||||
mimeGIF,
|
mimeGIF,
|
||||||
mimeWebP,
|
mimeWebP,
|
||||||
mimeAVIF,
|
mimeAVIF,
|
||||||
|
mimeJXL,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -326,6 +329,7 @@ func (p *ImageProcessor) SupportedOutputFormats() []Format {
|
|||||||
FormatGIF,
|
FormatGIF,
|
||||||
FormatWebP,
|
FormatWebP,
|
||||||
FormatAVIF,
|
FormatAVIF,
|
||||||
|
FormatJXL,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -342,6 +346,8 @@ func FormatToMIME(format Format) string {
|
|||||||
return mimeGIF
|
return mimeGIF
|
||||||
case FormatAVIF:
|
case FormatAVIF:
|
||||||
return mimeAVIF
|
return mimeAVIF
|
||||||
|
case FormatJXL:
|
||||||
|
return mimeJXL
|
||||||
case FormatOriginal:
|
case FormatOriginal:
|
||||||
return "application/octet-stream"
|
return "application/octet-stream"
|
||||||
default:
|
default:
|
||||||
@@ -411,9 +417,11 @@ func (p *ImageProcessor) detectFormat(img *vips.ImageRef) string {
|
|||||||
return "webp"
|
return "webp"
|
||||||
case vips.ImageTypeAVIF, vips.ImageTypeHEIF:
|
case vips.ImageTypeAVIF, vips.ImageTypeHEIF:
|
||||||
return string(FormatAVIF)
|
return string(FormatAVIF)
|
||||||
|
case vips.ImageTypeJXL:
|
||||||
|
return string(FormatJXL)
|
||||||
case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF,
|
case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF,
|
||||||
vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP,
|
vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP,
|
||||||
vips.ImageTypeJP2K, vips.ImageTypeJXL:
|
vips.ImageTypeJP2K:
|
||||||
return "unknown"
|
return "unknown"
|
||||||
default:
|
default:
|
||||||
return "unknown"
|
return "unknown"
|
||||||
@@ -485,6 +493,17 @@ func (p *ImageProcessor) encode(
|
|||||||
quality = defaultQuality
|
quality = defaultQuality
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Stripping drops the ICC profile as well, and clients show an image
|
||||||
|
// with no profile as sRGB, so convert to sRGB first. "srgb" names
|
||||||
|
// libvips' built-in profile; govips' own sRGB path variable is set on
|
||||||
|
// first use but read without a lock, so concurrent requests race on it.
|
||||||
|
if img.HasICCProfile() {
|
||||||
|
err := img.TransformICCProfileWithFallback("srgb", "srgb")
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to convert to sRGB: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
var params vips.ExportParams
|
var params vips.ExportParams
|
||||||
|
|
||||||
switch format {
|
switch format {
|
||||||
@@ -516,6 +535,9 @@ func (p *ImageProcessor) encode(
|
|||||||
Quality: quality,
|
Quality: quality,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
case FormatJXL:
|
||||||
|
return exportJXL(img, quality)
|
||||||
|
|
||||||
case FormatOriginal:
|
case FormatOriginal:
|
||||||
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
|
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
|
||||||
|
|
||||||
@@ -523,17 +545,6 @@ func (p *ImageProcessor) encode(
|
|||||||
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
|
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stripping drops the ICC profile as well, and clients show an image
|
|
||||||
// with no profile as sRGB, so convert to sRGB first. "srgb" names
|
|
||||||
// libvips' built-in profile; govips' own sRGB path variable is set on
|
|
||||||
// first use but read without a lock, so concurrent requests race on it.
|
|
||||||
if img.HasICCProfile() {
|
|
||||||
err := img.TransformICCProfileWithFallback("srgb", "srgb")
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to convert to sRGB: %w", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Drop EXIF, XMP, IPTC and the ICC profile. govips ignores this for
|
// Drop EXIF, XMP, IPTC and the ICC profile. govips ignores this for
|
||||||
// GIF, which carries none of them.
|
// GIF, which carries none of them.
|
||||||
params.StripMetadata = true
|
params.StripMetadata = true
|
||||||
@@ -546,6 +557,54 @@ func (p *ImageProcessor) encode(
|
|||||||
return output, nil
|
return output, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// exportJXL encodes img as JPEG XL at quality, with libvips' default effort
|
||||||
|
// and without metadata. govips sends libvips a distance, the JPEG XL
|
||||||
|
// encoder's own measure of quality, along with the quality, and libvips then
|
||||||
|
// uses the distance alone, so the quality is also given as a distance.
|
||||||
|
func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||||
|
// govips cannot make libvips strip metadata from JPEG XL, so it is
|
||||||
|
// removed from the image itself. RemoveMetadata removes EXIF, XMP and
|
||||||
|
// IPTC but keeps the ICC profile. libvips 8.16 and later still write an
|
||||||
|
// EXIF block of their own: orientation, resolution, size, colour space
|
||||||
|
// and fixed defaults.
|
||||||
|
err := img.RemoveMetadata()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
err = img.RemoveICCProfile()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
params := vips.NewJxlExportParams()
|
||||||
|
params.Quality = quality
|
||||||
|
params.Distance = jxlDistance(quality)
|
||||||
|
|
||||||
|
output, _, err := img.ExportJxl(params)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return output, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// jxlDistance turns a quality from 1 to 100 into the JPEG XL encoder's
|
||||||
|
// distance, with the formula libvips and libjxl use for their own quality
|
||||||
|
// setting, except that 100 stays lossy (distance 0.1) where libjxl makes it
|
||||||
|
// lossless.
|
||||||
|
//
|
||||||
|
//nolint:mnd // the constants of that formula
|
||||||
|
func jxlDistance(quality int) float64 {
|
||||||
|
q := float64(quality)
|
||||||
|
|
||||||
|
if quality >= 30 {
|
||||||
|
return 0.1 + (100-q)*0.09
|
||||||
|
}
|
||||||
|
|
||||||
|
return 53.0/3000.0*q*q - 23.0/20.0*q + 25.0
|
||||||
|
}
|
||||||
|
|
||||||
// formatFromString converts a format string to Format.
|
// formatFromString converts a format string to Format.
|
||||||
func (p *ImageProcessor) formatFromString(format string) Format {
|
func (p *ImageProcessor) formatFromString(format string) Format {
|
||||||
switch format {
|
switch format {
|
||||||
@@ -559,6 +618,8 @@ func (p *ImageProcessor) formatFromString(format string) Format {
|
|||||||
return FormatWebP
|
return FormatWebP
|
||||||
case string(FormatAVIF):
|
case string(FormatAVIF):
|
||||||
return FormatAVIF
|
return FormatAVIF
|
||||||
|
case string(FormatJXL):
|
||||||
|
return FormatJXL
|
||||||
default:
|
default:
|
||||||
return FormatJPEG
|
return FormatJPEG
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,10 @@
|
|||||||
package imageprocessor
|
package imageprocessor
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
|
"io"
|
||||||
|
"math"
|
||||||
|
"os"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/davidbyttow/govips/v2/vips"
|
"github.com/davidbyttow/govips/v2/vips"
|
||||||
@@ -50,3 +54,118 @@ func TestLibvipsSavesAndLoadsJPEGXL(t *testing.T) {
|
|||||||
t.Errorf("loaded size = %dx%d, want 64x48", loaded.Width(), loaded.Height())
|
t.Errorf("loaded size = %dx%d, want 64x48", loaded.Width(), loaded.Height())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestImageProcessor_EncodeJPEGXL converts a JPEG to a smaller JPEG XL and
|
||||||
|
// checks the content type, and the format and size the output loads as.
|
||||||
|
func TestImageProcessor_EncodeJPEGXL(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
req := &Request{Size: Size{Width: 100, Height: 75}, Format: FormatJXL}
|
||||||
|
|
||||||
|
result, err := New(Params{}).Process(
|
||||||
|
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)), req,
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Process() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = result.Content.Close() }()
|
||||||
|
|
||||||
|
if result.ContentType != "image/jxl" {
|
||||||
|
t.Errorf("ContentType = %q, want image/jxl", result.ContentType)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := io.ReadAll(result.Content)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to read result: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
output, err := vips.NewImageFromBuffer(data)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to load the output: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer output.Close()
|
||||||
|
|
||||||
|
if output.Format() != vips.ImageTypeJXL {
|
||||||
|
t.Errorf("output format = %s, want jxl", vips.ImageTypes[output.Format()])
|
||||||
|
}
|
||||||
|
|
||||||
|
if output.Width() != 100 || output.Height() != 75 {
|
||||||
|
t.Errorf("output size = %dx%d, want 100x75", output.Width(), output.Height())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestImageProcessor_JPEGXLQuality verifies that the quality reaches the JPEG
|
||||||
|
// XL encoder: the same image comes out smaller at quality 30 than at 90.
|
||||||
|
func TestImageProcessor_JPEGXLQuality(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
input := createTestJPEG(t, 400, 300)
|
||||||
|
outputBytes := make(map[int]int64)
|
||||||
|
|
||||||
|
for _, quality := range []int{30, 90} {
|
||||||
|
result, err := New(Params{}).Process(t.Context(),
|
||||||
|
bytes.NewReader(input), &Request{Format: FormatJXL, Quality: quality})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Process() at quality %d error = %v", quality, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_ = result.Content.Close()
|
||||||
|
outputBytes[quality] = result.ContentLength
|
||||||
|
}
|
||||||
|
|
||||||
|
if outputBytes[30] >= outputBytes[90] {
|
||||||
|
t.Errorf("%d bytes at quality 30, %d at 90, want fewer at 30",
|
||||||
|
outputBytes[30], outputBytes[90])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestImageProcessor_JPEGXLDropsSourceEXIF verifies that none of the source's
|
||||||
|
// EXIF reaches a JPEG XL output. libvips 8.16 and later write an EXIF block of
|
||||||
|
// their own into JPEG XL (orientation, resolution, size, colour space and
|
||||||
|
// fixed defaults), and govips cannot ask them to leave it out, so the test
|
||||||
|
// looks for the source's fields rather than for no EXIF at all.
|
||||||
|
func TestImageProcessor_JPEGXLDropsSourceEXIF(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
input, err := os.ReadFile("testdata/gps-exif.jpg")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to read test JPEG: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
exif := processAndDecode(t, input, &Request{Format: FormatJXL}).GetExif()
|
||||||
|
|
||||||
|
for _, field := range []string{
|
||||||
|
"exif-ifd0-Make", "exif-ifd0-Model", "exif-ifd2-BodySerialNumber",
|
||||||
|
"exif-ifd2-DateTimeOriginal", "exif-ifd3-GPSLatitude",
|
||||||
|
"exif-ifd3-GPSLongitude",
|
||||||
|
} {
|
||||||
|
if value, found := exif[field]; found {
|
||||||
|
t.Errorf("output has %s: %s", field, value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestJXLDistance verifies the JPEG XL distance for a few qualities: 90 is
|
||||||
|
// distance 1, the encoder's own default, and 100 stays lossy.
|
||||||
|
func TestJXLDistance(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
quality int
|
||||||
|
want float64
|
||||||
|
}{
|
||||||
|
{quality: 100, want: 0.1},
|
||||||
|
{quality: 90, want: 1},
|
||||||
|
{quality: 30, want: 6.4},
|
||||||
|
{quality: 20, want: 9.0667},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
got := jxlDistance(tt.quality)
|
||||||
|
if math.Abs(got-tt.want) > 0.0001 {
|
||||||
|
t.Errorf("jxlDistance(%d) = %v, want %v", tt.quality, got, tt.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -96,6 +96,17 @@ type Cache struct {
|
|||||||
// deterministically pause inside that window to exercise
|
// deterministically pause inside that window to exercise
|
||||||
// concurrent stores against it; production code leaves it nil.
|
// concurrent stores against it; production code leaves it nil.
|
||||||
evictSourceBlobTestHook func(ContentHash)
|
evictSourceBlobTestHook func(ContentHash)
|
||||||
|
|
||||||
|
// reconciliationPageSize is the most rows one read of a content table
|
||||||
|
// returns in the reconciliation pass and in Stats. newCache sets it to
|
||||||
|
// defaultReconciliationPageSize; tests set it smaller.
|
||||||
|
reconciliationPageSize int
|
||||||
|
|
||||||
|
// reconciliationReadTestHook, when set, is called after each of those
|
||||||
|
// reads with the number of rows the read covered, so tests can check
|
||||||
|
// that no read covers more than one page; production code leaves it
|
||||||
|
// nil.
|
||||||
|
reconciliationReadTestHook func(rows int)
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewCache creates a new cache instance.
|
// NewCache creates a new cache instance.
|
||||||
@@ -127,6 +138,8 @@ func newCache(
|
|||||||
evictionDone: make(chan struct{}),
|
evictionDone: make(chan struct{}),
|
||||||
metaCache: metaCache,
|
metaCache: metaCache,
|
||||||
contentLocks: newContentLock(),
|
contentLocks: newContentLock(),
|
||||||
|
|
||||||
|
reconciliationPageSize: defaultReconciliationPageSize,
|
||||||
}
|
}
|
||||||
|
|
||||||
if c.disabled {
|
if c.disabled {
|
||||||
@@ -471,8 +484,9 @@ func (c *Cache) Stats(ctx context.Context) (*CacheStats, error) {
|
|||||||
return nil, fmt.Errorf("failed to get cache stats: %w", err)
|
return nil, fmt.Errorf("failed to get cache stats: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Count and size the cached source images and processed variants. A
|
// Count and size the cached source images and processed variants from
|
||||||
// disabled cache holds none, whatever rows an earlier run left.
|
// their tables. A disabled cache holds none, whatever rows an earlier
|
||||||
|
// run left.
|
||||||
if !c.disabled {
|
if !c.disabled {
|
||||||
err = c.db.QueryRowContext(ctx, `
|
err = c.db.QueryRowContext(ctx, `
|
||||||
SELECT (SELECT COUNT(*) FROM source_content)
|
SELECT (SELECT COUNT(*) FROM source_content)
|
||||||
@@ -482,7 +496,7 @@ func (c *Cache) Stats(ctx context.Context) (*CacheStats, error) {
|
|||||||
c.log.Warn("failed to count cache items for stats", "error", err)
|
c.log.Warn("failed to count cache items for stats", "error", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
stats.TotalSizeBytes, err = c.UsageBytes(ctx)
|
stats.TotalSizeBytes, err = c.sumContentSizeBytes(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
c.log.Warn("failed to sum cache size for stats", "error", err)
|
c.log.Warn("failed to sum cache size for stats", "error", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,265 @@
|
|||||||
|
package imgcache
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestEvictionReadsTheUsageTotal adds, stores again, resizes and evicts
|
||||||
|
// cache content, then drops both content tables. UsageBytes must still
|
||||||
|
// report what the tables held, and an eviction pass under the limit must
|
||||||
|
// still succeed: neither may sum the tables.
|
||||||
|
func TestEvictionReadsTheUsageTotal(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||||
|
ctx := t.Context()
|
||||||
|
|
||||||
|
kept := storeEvictionTestSource(t, cache, "total.example.com", "/kept.jpg",
|
||||||
|
bytes.Repeat([]byte{0x71}, 1000))
|
||||||
|
evicted := storeEvictionTestSource(t, cache, "total.example.com", "/evicted.jpg",
|
||||||
|
bytes.Repeat([]byte{0x72}, 700))
|
||||||
|
|
||||||
|
storeEvictionTestVariant(t, cache, testVariantKeyOne,
|
||||||
|
bytes.Repeat([]byte{0x73}, 500))
|
||||||
|
storeEvictionTestVariant(t, cache, testVariantKeyOne,
|
||||||
|
bytes.Repeat([]byte{0x74}, 300))
|
||||||
|
storeEvictionTestVariant(t, cache, testVariantKeyTwo,
|
||||||
|
bytes.Repeat([]byte{0x75}, 200))
|
||||||
|
|
||||||
|
// pixa never changes a source's size, but a statement that does must
|
||||||
|
// change the total too.
|
||||||
|
_, err := cache.db.ExecContext(ctx,
|
||||||
|
`UPDATE source_content SET size_bytes = 900 WHERE content_hash = ?`,
|
||||||
|
string(kept))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to change the source's size: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = cache.evictSourceBlob(ctx, evicted)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("evictSourceBlob failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = cache.evictVariant(ctx, testVariantKeyTwo)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("evictVariant failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = cache.db.ExecContext(ctx,
|
||||||
|
`DROP TABLE source_content; DROP TABLE variant_content`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to drop the content tables: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
usage, err := cache.UsageBytes(ctx)
|
||||||
|
t.Logf("UsageBytes() = %d, error = %v", usage, err)
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UsageBytes() error = %v, want nil: it read a content table", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if usage != 1200 {
|
||||||
|
t.Errorf("UsageBytes() = %d, want 1200 (900 + 300)", usage)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = cache.EvictToLimit(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("EvictToLimit() error = %v, want nil: it read a content table", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReconciliationCorrectsTheUsageTotal sets the total cache usage to a
|
||||||
|
// wrong value and checks that a reconciliation pass, which sums both
|
||||||
|
// content tables, puts it right.
|
||||||
|
func TestReconciliationCorrectsTheUsageTotal(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||||
|
ctx := t.Context()
|
||||||
|
|
||||||
|
storeEvictionTestSource(t, cache, "total.example.com", "/a.jpg",
|
||||||
|
bytes.Repeat([]byte{0x76}, 1000))
|
||||||
|
storeEvictionTestVariant(t, cache, testVariantKeyOne,
|
||||||
|
bytes.Repeat([]byte{0x77}, 500))
|
||||||
|
|
||||||
|
_, err := cache.db.ExecContext(ctx,
|
||||||
|
`UPDATE cache_usage SET total_size_bytes = 1 WHERE id = 1`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to set a wrong total: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = cache.reconcileAccounting(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("reconcileAccounting failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
usage, err := cache.UsageBytes(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UsageBytes failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if usage != 1500 {
|
||||||
|
t.Errorf("UsageBytes() after reconciliation = %d, want 1500 (1000 + 500)",
|
||||||
|
usage)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUsageTotalNotCorrectedFromAnOlderSum stores a variant after the
|
||||||
|
// change count was read, then offers a sum taken before the store as the
|
||||||
|
// correction: the total must keep the stored variant, as that sum may
|
||||||
|
// have missed it.
|
||||||
|
func TestUsageTotalNotCorrectedFromAnOlderSum(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||||
|
ctx := t.Context()
|
||||||
|
|
||||||
|
var changeCount int64
|
||||||
|
|
||||||
|
err := cache.db.QueryRowContext(ctx,
|
||||||
|
`SELECT change_count FROM cache_usage WHERE id = 1`,
|
||||||
|
).Scan(&changeCount)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to read the change count: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
storeEvictionTestVariant(t, cache, testVariantKeyOne,
|
||||||
|
bytes.Repeat([]byte{0x78}, 500))
|
||||||
|
|
||||||
|
corrected, err := cache.correctUsageTotal(ctx, 0, changeCount)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("correctUsageTotal failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if corrected {
|
||||||
|
t.Error("correctUsageTotal() = true, want false: content changed since the sum")
|
||||||
|
}
|
||||||
|
|
||||||
|
usage, err := cache.UsageBytes(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UsageBytes failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if usage != 500 {
|
||||||
|
t.Errorf("UsageBytes() = %d, want 500", usage)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReconciliationReadsAPageAtATime stores five source images and five
|
||||||
|
// variants, sets the page size to two rows and runs a reconciliation
|
||||||
|
// pass. No read the pass makes of a content table, to check its rows or
|
||||||
|
// to sum them, may cover more than two rows, so a request's query waits
|
||||||
|
// for one page at most. Between them the reads must still cover every
|
||||||
|
// row of both tables twice, once to check it and once to sum it, and the
|
||||||
|
// sum must put a wrong total right.
|
||||||
|
func TestReconciliationReadsAPageAtATime(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||||
|
ctx := t.Context()
|
||||||
|
|
||||||
|
const pageSize = 2
|
||||||
|
|
||||||
|
cache.reconciliationPageSize = pageSize
|
||||||
|
|
||||||
|
// Sources of 100 bytes and variants of 10, five of each.
|
||||||
|
for i := range 5 {
|
||||||
|
content := []byte(strconv.Itoa(i))
|
||||||
|
|
||||||
|
storeEvictionTestSource(t, cache, "pages.example.com",
|
||||||
|
"/"+strconv.Itoa(i)+".jpg", bytes.Repeat(content, 100))
|
||||||
|
storeEvictionTestVariant(t, cache, VariantKey("aabbccdd000"+strconv.Itoa(i)),
|
||||||
|
bytes.Repeat(content, 10))
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := cache.db.ExecContext(ctx,
|
||||||
|
`UPDATE cache_usage SET total_size_bytes = 1 WHERE id = 1`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to set a wrong total: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var rowsPerRead []int
|
||||||
|
|
||||||
|
cache.reconciliationReadTestHook = func(rows int) {
|
||||||
|
rowsPerRead = append(rowsPerRead, rows)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = cache.reconcileAccounting(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("reconcileAccounting failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("rows covered by each read, in order: %v", rowsPerRead)
|
||||||
|
|
||||||
|
coveredRows := 0
|
||||||
|
|
||||||
|
for _, rows := range rowsPerRead {
|
||||||
|
if rows > pageSize {
|
||||||
|
t.Errorf("a read covered %d rows, want at most %d", rows, pageSize)
|
||||||
|
}
|
||||||
|
|
||||||
|
coveredRows += rows
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ten rows, each read once to check it and once to sum it.
|
||||||
|
if coveredRows != 20 {
|
||||||
|
t.Errorf("the reads covered %d rows in all, want 20", coveredRows)
|
||||||
|
}
|
||||||
|
|
||||||
|
usage, err := cache.UsageBytes(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UsageBytes failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if usage != 550 {
|
||||||
|
t.Errorf("UsageBytes() after reconciliation = %d, want 550 (5*100 + 5*10)",
|
||||||
|
usage)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSourceEvictionCandidatesComeFromTheIndex checks that the query
|
||||||
|
// choosing source images to evict reads source_content in last access
|
||||||
|
// order from its index, instead of reading and sorting the whole table.
|
||||||
|
func TestSourceEvictionCandidatesComeFromTheIndex(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||||
|
|
||||||
|
rows, err := cache.db.QueryContext(t.Context(),
|
||||||
|
`EXPLAIN QUERY PLAN `+sourceCandidatesQuery, evictionBatchSize)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("EXPLAIN QUERY PLAN failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = rows.Close() }()
|
||||||
|
|
||||||
|
var plan []string
|
||||||
|
|
||||||
|
for rows.Next() {
|
||||||
|
var id, parent, unused int
|
||||||
|
|
||||||
|
var detail string
|
||||||
|
|
||||||
|
err := rows.Scan(&id, &parent, &unused, &detail)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("failed to scan the query plan: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
plan = append(plan, detail)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = rows.Err()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("query plan iteration failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("query plan: %q", plan)
|
||||||
|
|
||||||
|
if !strings.Contains(strings.Join(plan, "\n"), "idx_source_content_last_accessed") {
|
||||||
|
t.Errorf("the source candidate query does not use "+
|
||||||
|
"idx_source_content_last_accessed; plan: %q", plan)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -72,7 +72,7 @@ func (c *Cache) computeDefaultMaxBytes(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Both terms are at most math.MaxInt64, so the sum cannot overflow.
|
// Both terms are at most math.MaxInt64, so the sum cannot overflow.
|
||||||
//nolint:gosec // G115: UsageBytes sums file sizes, never negative
|
//nolint:gosec // G115: UsageBytes returns the total cache usage, never negative
|
||||||
spaceBytes := min(freeBytes, math.MaxInt64) + uint64(usedBytes)
|
spaceBytes := min(freeBytes, math.MaxInt64) + uint64(usedBytes)
|
||||||
|
|
||||||
computed := spaceBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
|
computed := spaceBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
|
||||||
|
|||||||
+246
-72
@@ -21,6 +21,12 @@ const DefaultEvictionInterval = 5 * time.Minute
|
|||||||
// and source blobs) one eviction pass fetches from the database.
|
// and source blobs) one eviction pass fetches from the database.
|
||||||
const evictionBatchSize = 100
|
const evictionBatchSize = 100
|
||||||
|
|
||||||
|
// defaultReconciliationPageSize is the most rows one read of the
|
||||||
|
// reconciliation pass returns, unless a test sets
|
||||||
|
// Cache.reconciliationPageSize smaller. Each read is a query of its own,
|
||||||
|
// so a request waits for one page at most, however large the cache is.
|
||||||
|
const defaultReconciliationPageSize = 1000
|
||||||
|
|
||||||
// staleTempFileAge is how old an orphaned temp file (left behind by a
|
// staleTempFileAge is how old an orphaned temp file (left behind by a
|
||||||
// crashed write) must be before reconciliation removes it. Fresh temp
|
// crashed write) must be before reconciliation removes it. Fresh temp
|
||||||
// files may still belong to an in-flight store.
|
// files may still belong to an in-flight store.
|
||||||
@@ -45,7 +51,9 @@ const fallbackContentType = "application/octet-stream"
|
|||||||
|
|
||||||
// UsageBytes returns the total number of bytes of cache content
|
// UsageBytes returns the total number of bytes of cache content
|
||||||
// tracked in the database (source content blobs plus processed
|
// tracked in the database (source content blobs plus processed
|
||||||
// variants). It never scans the cache directories.
|
// variants). It reads the total the database keeps up to date as rows
|
||||||
|
// are added and removed, so it neither scans the cache directories nor
|
||||||
|
// sums the tables.
|
||||||
func (c *Cache) UsageBytes(ctx context.Context) (int64, error) {
|
func (c *Cache) UsageBytes(ctx context.Context) (int64, error) {
|
||||||
if c.disabled {
|
if c.disabled {
|
||||||
return 0, nil
|
return 0, nil
|
||||||
@@ -53,12 +61,11 @@ func (c *Cache) UsageBytes(ctx context.Context) (int64, error) {
|
|||||||
|
|
||||||
var total int64
|
var total int64
|
||||||
|
|
||||||
err := c.db.QueryRowContext(ctx, `
|
err := c.db.QueryRowContext(ctx,
|
||||||
SELECT (SELECT COALESCE(SUM(size_bytes), 0) FROM source_content)
|
`SELECT total_size_bytes FROM cache_usage WHERE id = 1`,
|
||||||
+ (SELECT COALESCE(SUM(size_bytes), 0) FROM variant_content)
|
).Scan(&total)
|
||||||
`).Scan(&total)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, fmt.Errorf("failed to compute cache usage: %w", err)
|
return 0, fmt.Errorf("failed to read cache usage: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
return total, nil
|
return total, nil
|
||||||
@@ -236,16 +243,19 @@ func (c *Cache) variantCandidates(ctx context.Context) ([]evictionCandidate, err
|
|||||||
return candidates, nil
|
return candidates, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// sourceCandidates returns the least recently used source blobs. Rows
|
// sourceCandidatesQuery selects the least recently used source blobs. It
|
||||||
// written before the LRU column existed fall back to fetched_at.
|
// orders by the last_accessed_at column itself, not by an expression, so
|
||||||
|
// SQLite reads the rows in order from that column's index instead of
|
||||||
|
// sorting the whole table.
|
||||||
|
const sourceCandidatesQuery = `
|
||||||
|
SELECT content_hash, size_bytes, last_accessed_at
|
||||||
|
FROM source_content
|
||||||
|
ORDER BY last_accessed_at ASC, content_hash ASC
|
||||||
|
LIMIT ?`
|
||||||
|
|
||||||
|
// sourceCandidates returns the least recently used source blobs.
|
||||||
func (c *Cache) sourceCandidates(ctx context.Context) ([]evictionCandidate, error) {
|
func (c *Cache) sourceCandidates(ctx context.Context) ([]evictionCandidate, error) {
|
||||||
rows, err := c.db.QueryContext(ctx, `
|
rows, err := c.db.QueryContext(ctx, sourceCandidatesQuery, evictionBatchSize)
|
||||||
SELECT content_hash, size_bytes,
|
|
||||||
COALESCE(last_accessed_at, fetched_at, '1970-01-01 00:00:00') AS lru
|
|
||||||
FROM source_content
|
|
||||||
ORDER BY lru ASC, content_hash ASC
|
|
||||||
LIMIT ?
|
|
||||||
`, evictionBatchSize)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to query source eviction candidates: %w", err)
|
return nil, fmt.Errorf("failed to query source eviction candidates: %w", err)
|
||||||
}
|
}
|
||||||
@@ -532,11 +542,13 @@ func (c *Cache) runReconciliationPass(ctx context.Context) {
|
|||||||
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
|
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
|
||||||
// insert under transient DB contention), drops accounting rows whose
|
// insert under transient DB contention), drops accounting rows whose
|
||||||
// files are missing, removes source blob files the database does not
|
// files are missing, removes source blob files the database does not
|
||||||
// know (and rows whose files are gone), and sweeps stale temp files
|
// know (and rows whose files are gone), sweeps stale temp files left
|
||||||
// left behind by crashed writes. Running it periodically, not just
|
// behind by crashed writes, and last checks the total cache usage
|
||||||
// once, bounds how long such drift can accumulate unaccounted for on a
|
// against the tables. It reads the tables a page at a time. Running it
|
||||||
// long-running process to one eviction interval. Once ctx is cancelled,
|
// periodically, not just once, bounds how long such drift can
|
||||||
// it stops at the next file or row and returns ctx's error.
|
// accumulate unaccounted for on a long-running process to one eviction
|
||||||
|
// interval. Once ctx is cancelled, it stops at the next file or row and
|
||||||
|
// returns ctx's error.
|
||||||
func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
||||||
if c.disabled {
|
if c.disabled {
|
||||||
return nil
|
return nil
|
||||||
@@ -562,7 +574,7 @@ func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return c.reconcileUsageTotal(ctx)
|
||||||
}
|
}
|
||||||
|
|
||||||
// reconcileVariantFiles walks the variant storage directory, adopting
|
// reconcileVariantFiles walks the variant storage directory, adopting
|
||||||
@@ -657,46 +669,63 @@ func (c *Cache) variantContentTypeFromSidecar(variantPath string) string {
|
|||||||
// reconcileVariantRows drops accounting rows whose variant files are
|
// reconcileVariantRows drops accounting rows whose variant files are
|
||||||
// missing, so the database never references deleted content.
|
// missing, so the database never references deleted content.
|
||||||
func (c *Cache) reconcileVariantRows(ctx context.Context) error {
|
func (c *Cache) reconcileVariantRows(ctx context.Context) error {
|
||||||
keys, err := c.allVariantKeys(ctx)
|
var after VariantKey
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, key := range keys {
|
for {
|
||||||
if ctx.Err() != nil {
|
keys, err := c.variantKeysAfter(ctx, after)
|
||||||
return ctx.Err()
|
|
||||||
}
|
|
||||||
|
|
||||||
if c.variants.Exists(key) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err := c.db.ExecContext(ctx,
|
|
||||||
`DELETE FROM variant_content WHERE cache_key = ?`, string(key))
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to drop stale variant accounting row: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
c.log.Info("dropped accounting row for missing variant file", "cache_key", key)
|
if c.reconciliationReadTestHook != nil {
|
||||||
|
c.reconciliationReadTestHook(len(keys))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(keys) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, key := range keys {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
if c.variants.Exists(key) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := c.db.ExecContext(ctx,
|
||||||
|
`DELETE FROM variant_content WHERE cache_key = ?`, string(key))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to drop stale variant accounting row: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
c.log.Info("dropped accounting row for missing variant file", "cache_key", key)
|
||||||
|
}
|
||||||
|
|
||||||
|
after = keys[len(keys)-1]
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// allVariantKeys returns every tracked variant cache key.
|
// variantKeysAfter returns, in order, up to c.reconciliationPageSize
|
||||||
func (c *Cache) allVariantKeys(ctx context.Context) ([]VariantKey, error) {
|
// tracked variant cache keys that sort after the given one.
|
||||||
return queryStringColumn[VariantKey](ctx, c.db,
|
func (c *Cache) variantKeysAfter(
|
||||||
`SELECT cache_key FROM variant_content`, "variant keys", "variant key")
|
ctx context.Context, after VariantKey,
|
||||||
|
) ([]VariantKey, error) {
|
||||||
|
return queryStringColumn[VariantKey](ctx, c.db, `
|
||||||
|
SELECT cache_key FROM variant_content
|
||||||
|
WHERE cache_key > ? ORDER BY cache_key LIMIT ?
|
||||||
|
`, "variant keys", "variant key", string(after), c.reconciliationPageSize)
|
||||||
}
|
}
|
||||||
|
|
||||||
// queryStringColumn runs a single-column query and returns the column
|
// queryStringColumn runs a single-column query with args and returns the
|
||||||
// values as T. plural names the set for the query and scan failure
|
// column values as T. plural names the set for the query and scan
|
||||||
// messages; singular names one row for the scan and iteration failure
|
// failure messages; singular names one row for the scan and iteration
|
||||||
// messages.
|
// failure messages.
|
||||||
func queryStringColumn[T ~string](
|
func queryStringColumn[T ~string](
|
||||||
ctx context.Context, db *sql.DB, query, plural, singular string,
|
ctx context.Context, db *sql.DB, query, plural, singular string, args ...any,
|
||||||
) ([]T, error) {
|
) ([]T, error) {
|
||||||
rows, err := db.QueryContext(ctx, query)
|
rows, err := db.QueryContext(ctx, query, args...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to query %s: %w", plural, err)
|
return nil, fmt.Errorf("failed to query %s: %w", plural, err)
|
||||||
}
|
}
|
||||||
@@ -791,38 +820,183 @@ func (c *Cache) removeUntrackedSourceFile(
|
|||||||
// reconcileSourceRows removes source_content rows (and their metadata
|
// reconcileSourceRows removes source_content rows (and their metadata
|
||||||
// references and sidecars) whose blob files are missing on disk.
|
// references and sidecars) whose blob files are missing on disk.
|
||||||
func (c *Cache) reconcileSourceRows(ctx context.Context) error {
|
func (c *Cache) reconcileSourceRows(ctx context.Context) error {
|
||||||
hashes, err := c.allSourceContentHashes(ctx)
|
var after ContentHash
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, hash := range hashes {
|
for {
|
||||||
if ctx.Err() != nil {
|
hashes, err := c.sourceContentHashesAfter(ctx, after)
|
||||||
return ctx.Err()
|
|
||||||
}
|
|
||||||
|
|
||||||
if c.srcContent.Exists(hash) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
// The blob file is already gone; evictSourceBlob removes the
|
|
||||||
// rows and sidecars and tolerates the missing file.
|
|
||||||
err := c.evictSourceBlob(ctx, hash)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
c.log.Info("dropped rows for missing source content file", "content_hash", hash)
|
if c.reconciliationReadTestHook != nil {
|
||||||
|
c.reconciliationReadTestHook(len(hashes))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(hashes) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, hash := range hashes {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
if c.srcContent.Exists(hash) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// The blob file is already gone; evictSourceBlob removes the
|
||||||
|
// rows and sidecars and tolerates the missing file.
|
||||||
|
err := c.evictSourceBlob(ctx, hash)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
c.log.Info("dropped rows for missing source content file", "content_hash", hash)
|
||||||
|
}
|
||||||
|
|
||||||
|
after = hashes[len(hashes)-1]
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// sourceContentHashesAfter returns, in order, up to
|
||||||
|
// c.reconciliationPageSize tracked source content hashes that sort after
|
||||||
|
// the given one.
|
||||||
|
func (c *Cache) sourceContentHashesAfter(
|
||||||
|
ctx context.Context, after ContentHash,
|
||||||
|
) ([]ContentHash, error) {
|
||||||
|
return queryStringColumn[ContentHash](ctx, c.db, `
|
||||||
|
SELECT content_hash FROM source_content
|
||||||
|
WHERE content_hash > ? ORDER BY content_hash LIMIT ?
|
||||||
|
`, "source content hashes", "content hash", string(after),
|
||||||
|
c.reconciliationPageSize)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Each of these queries sums size_bytes over the next page of rows of
|
||||||
|
// one content table, the rows that sort after a key, and returns the
|
||||||
|
// page's last key, the sum and the number of rows in the page. Past the
|
||||||
|
// last row the page has no rows and the key is NULL.
|
||||||
|
const (
|
||||||
|
sourceSizePageQuery = `
|
||||||
|
SELECT MAX(content_hash), COALESCE(SUM(size_bytes), 0), COUNT(*)
|
||||||
|
FROM (
|
||||||
|
SELECT content_hash, size_bytes FROM source_content
|
||||||
|
WHERE content_hash > ? ORDER BY content_hash LIMIT ?
|
||||||
|
)`
|
||||||
|
variantSizePageQuery = `
|
||||||
|
SELECT MAX(cache_key), COALESCE(SUM(size_bytes), 0), COUNT(*)
|
||||||
|
FROM (
|
||||||
|
SELECT cache_key, size_bytes FROM variant_content
|
||||||
|
WHERE cache_key > ? ORDER BY cache_key LIMIT ?
|
||||||
|
)`
|
||||||
|
)
|
||||||
|
|
||||||
|
// sumContentSizeBytes sums size_bytes over both content tables, a page
|
||||||
|
// of rows per query.
|
||||||
|
func (c *Cache) sumContentSizeBytes(ctx context.Context) (int64, error) {
|
||||||
|
sourceBytes, err := c.sumSizeBytesInPages(ctx, sourceSizePageQuery)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
|
||||||
|
variantBytes, err := c.sumSizeBytesInPages(ctx, variantSizePageQuery)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return sourceBytes + variantBytes, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// sumSizeBytesInPages runs pageQuery, one of the size page queries
|
||||||
|
// above, from the first page to the last and adds up the page sums.
|
||||||
|
func (c *Cache) sumSizeBytesInPages(
|
||||||
|
ctx context.Context, pageQuery string,
|
||||||
|
) (int64, error) {
|
||||||
|
var total int64
|
||||||
|
|
||||||
|
after := ""
|
||||||
|
|
||||||
|
for {
|
||||||
|
var lastKey sql.NullString
|
||||||
|
|
||||||
|
var pageBytes int64
|
||||||
|
|
||||||
|
var pageRows int
|
||||||
|
|
||||||
|
err := c.db.QueryRowContext(ctx, pageQuery, after, c.reconciliationPageSize).
|
||||||
|
Scan(&lastKey, &pageBytes, &pageRows)
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf("failed to sum cache content sizes: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if c.reconciliationReadTestHook != nil {
|
||||||
|
c.reconciliationReadTestHook(pageRows)
|
||||||
|
}
|
||||||
|
|
||||||
|
if pageRows == 0 {
|
||||||
|
return total, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
total += pageBytes
|
||||||
|
after = lastKey.String
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// reconcileUsageTotal checks the total cache usage the database keeps
|
||||||
|
// against size_bytes summed over both content tables, and corrects the
|
||||||
|
// total when they differ.
|
||||||
|
func (c *Cache) reconcileUsageTotal(ctx context.Context) error {
|
||||||
|
var totalBytes, changeCount int64
|
||||||
|
|
||||||
|
err := c.db.QueryRowContext(ctx,
|
||||||
|
`SELECT total_size_bytes, change_count FROM cache_usage WHERE id = 1`,
|
||||||
|
).Scan(&totalBytes, &changeCount)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to read cache usage: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
sumBytes, err := c.sumContentSizeBytes(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
if sumBytes == totalBytes {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
corrected, err := c.correctUsageTotal(ctx, sumBytes, changeCount)
|
||||||
|
if err != nil || !corrected {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
c.log.Warn("corrected total cache usage to the sum of the content tables",
|
||||||
|
"previous_usage_bytes", totalBytes, "usage_bytes", sumBytes)
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// allSourceContentHashes returns every tracked source content hash.
|
// correctUsageTotal sets the total cache usage to sumBytes, a sum of the
|
||||||
func (c *Cache) allSourceContentHashes(ctx context.Context) ([]ContentHash, error) {
|
// content tables taken when the change count was changeCount, and
|
||||||
return queryStringColumn[ContentHash](ctx, c.db,
|
// reports whether it did. If a row was added, removed or resized since,
|
||||||
`SELECT content_hash FROM source_content`,
|
// the count has moved and the total is left alone: the sum may have
|
||||||
"source content hashes", "content hash")
|
// missed that change, and the next pass checks again.
|
||||||
|
func (c *Cache) correctUsageTotal(
|
||||||
|
ctx context.Context, sumBytes, changeCount int64,
|
||||||
|
) (bool, error) {
|
||||||
|
result, err := c.db.ExecContext(ctx, `
|
||||||
|
UPDATE cache_usage SET total_size_bytes = ?
|
||||||
|
WHERE id = 1 AND change_count = ?
|
||||||
|
`, sumBytes, changeCount)
|
||||||
|
if err != nil {
|
||||||
|
return false, fmt.Errorf("failed to correct cache usage: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
affected, err := result.RowsAffected()
|
||||||
|
if err != nil {
|
||||||
|
return false, fmt.Errorf("failed to read cache usage correction: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return affected > 0, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// sweepStaleTempFile removes a temp file left behind by a crashed
|
// sweepStaleTempFile removes a temp file left behind by a crashed
|
||||||
|
|||||||
@@ -21,11 +21,12 @@ const (
|
|||||||
FormatPNG ImageFormat = "png"
|
FormatPNG ImageFormat = "png"
|
||||||
FormatWebP ImageFormat = "webp"
|
FormatWebP ImageFormat = "webp"
|
||||||
FormatAVIF ImageFormat = "avif"
|
FormatAVIF ImageFormat = "avif"
|
||||||
|
FormatJXL ImageFormat = "jxl"
|
||||||
FormatGIF ImageFormat = "gif"
|
FormatGIF ImageFormat = "gif"
|
||||||
|
|
||||||
// FormatAuto stands for AVIF, WebP or JPEG, chosen for each request
|
// FormatAuto stands for JPEG XL, AVIF, WebP or JPEG, chosen for each
|
||||||
// from its Accept header once the URL's signature or token has been
|
// request from its Accept header once the URL's signature or token has
|
||||||
// checked; it is never processed or cached as itself.
|
// been checked; it is never processed or cached as itself.
|
||||||
FormatAuto ImageFormat = "auto"
|
FormatAuto ImageFormat = "auto"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -275,6 +275,8 @@ func parseFormat(s string) (ImageFormat, error) {
|
|||||||
return FormatWebP, nil
|
return FormatWebP, nil
|
||||||
case "avif":
|
case "avif":
|
||||||
return FormatAVIF, nil
|
return FormatAVIF, nil
|
||||||
|
case "jxl":
|
||||||
|
return FormatJXL, nil
|
||||||
case "gif":
|
case "gif":
|
||||||
return FormatGIF, nil
|
return FormatGIF, nil
|
||||||
case "auto":
|
case "auto":
|
||||||
|
|||||||
@@ -0,0 +1,58 @@
|
|||||||
|
package magic
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
// testMIMEJXL is the MIME type of JPEG XL.
|
||||||
|
const testMIMEJXL = "image/jxl"
|
||||||
|
|
||||||
|
// TestDetectFormatJPEGXL verifies that both JPEG XL signatures, that of a
|
||||||
|
// bare codestream and that of the container, are detected as image/jxl.
|
||||||
|
func TestDetectFormatJPEGXL(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
data []byte
|
||||||
|
}{
|
||||||
|
{"codestream", pad(0xFF, 0x0A)},
|
||||||
|
{"container", pad(
|
||||||
|
0x00, 0x00, 0x00, 0x0C, 0x4A, 0x58, 0x4C, 0x20, 0x0D, 0x0A, 0x87, 0x0A,
|
||||||
|
)},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
got, err := DetectFormat(tt.data)
|
||||||
|
if err != nil || got != MIMETypeJXL {
|
||||||
|
t.Errorf("DetectFormat() = %q, %v, want %q", got, err, testMIMEJXL)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = ValidateMagicBytes(tt.data, testMIMEJXL)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("ValidateMagicBytes(%q) error = %v", testMIMEJXL, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestJPEGXLMIMEType verifies that image/jxl is a supported input type and
|
||||||
|
// maps to and from the format jxl.
|
||||||
|
func TestJPEGXLMIMEType(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
if !IsSupportedMIMEType(testMIMEJXL) {
|
||||||
|
t.Errorf("IsSupportedMIMEType(%q) = false", testMIMEJXL)
|
||||||
|
}
|
||||||
|
|
||||||
|
format, ok := MIMEToImageFormat(testMIMEJXL)
|
||||||
|
if format != FormatJXL || !ok {
|
||||||
|
t.Errorf("MIMEToImageFormat(%q) = %q, %v, want %q, true",
|
||||||
|
testMIMEJXL, format, ok, FormatJXL)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := ImageFormatToMIME(FormatJXL); got != testMIMEJXL {
|
||||||
|
t.Errorf("ImageFormatToMIME(%q) = %q, want %q", FormatJXL, got, testMIMEJXL)
|
||||||
|
}
|
||||||
|
}
|
||||||
+19
-1
@@ -26,6 +26,7 @@ const (
|
|||||||
MIMETypeWebP = MIMEType("image/webp")
|
MIMETypeWebP = MIMEType("image/webp")
|
||||||
MIMETypeGIF = MIMEType("image/gif")
|
MIMETypeGIF = MIMEType("image/gif")
|
||||||
MIMETypeAVIF = MIMEType("image/avif")
|
MIMETypeAVIF = MIMEType("image/avif")
|
||||||
|
MIMETypeJXL = MIMEType("image/jxl")
|
||||||
MIMETypeSVG = MIMEType("image/svg+xml")
|
MIMETypeSVG = MIMEType("image/svg+xml")
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -40,6 +41,7 @@ const (
|
|||||||
FormatPNG ImageFormat = "png"
|
FormatPNG ImageFormat = "png"
|
||||||
FormatWebP ImageFormat = "webp"
|
FormatWebP ImageFormat = "webp"
|
||||||
FormatAVIF ImageFormat = "avif"
|
FormatAVIF ImageFormat = "avif"
|
||||||
|
FormatJXL ImageFormat = "jxl"
|
||||||
FormatGIF ImageFormat = "gif"
|
FormatGIF ImageFormat = "gif"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -62,6 +64,11 @@ var (
|
|||||||
// AVIF uses the ftyp box with brand "avif" or "avis"
|
// AVIF uses the ftyp box with brand "avif" or "avis"
|
||||||
// Format: size(4 bytes) + "ftyp" + brand(4 bytes)
|
// Format: size(4 bytes) + "ftyp" + brand(4 bytes)
|
||||||
magicFtyp = []byte{0x66, 0x74, 0x79, 0x70} // "ftyp"
|
magicFtyp = []byte{0x66, 0x74, 0x79, 0x70} // "ftyp"
|
||||||
|
// JPEG XL is a bare codestream or a codestream in a container
|
||||||
|
magicJXLCodestream = []byte{0xFF, 0x0A}
|
||||||
|
magicJXLContainer = []byte{
|
||||||
|
0x00, 0x00, 0x00, 0x0C, 0x4A, 0x58, 0x4C, 0x20, 0x0D, 0x0A, 0x87, 0x0A,
|
||||||
|
}
|
||||||
)
|
)
|
||||||
|
|
||||||
// WebP identifier appears at offset 8 after RIFF header.
|
// WebP identifier appears at offset 8 after RIFF header.
|
||||||
@@ -115,6 +122,12 @@ func DetectFormat(data []byte) (MIMEType, error) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Check JPEG XL (FF0A, or the 12-byte container signature)
|
||||||
|
if bytes.HasPrefix(data, magicJXLCodestream) ||
|
||||||
|
bytes.HasPrefix(data, magicJXLContainer) {
|
||||||
|
return MIMETypeJXL, nil
|
||||||
|
}
|
||||||
|
|
||||||
// Check SVG - look for XML declaration or SVG tag
|
// Check SVG - look for XML declaration or SVG tag
|
||||||
if detectSVG(data) {
|
if detectSVG(data) {
|
||||||
return MIMETypeSVG, nil
|
return MIMETypeSVG, nil
|
||||||
@@ -181,7 +194,8 @@ func normalizeMIMEType(mimeType string) string {
|
|||||||
func IsSupportedMIMEType(mimeType string) bool {
|
func IsSupportedMIMEType(mimeType string) bool {
|
||||||
normalized := normalizeMIMEType(mimeType)
|
normalized := normalizeMIMEType(mimeType)
|
||||||
switch MIMEType(normalized) {
|
switch MIMEType(normalized) {
|
||||||
case MIMETypeJPEG, MIMETypePNG, MIMETypeWebP, MIMETypeGIF, MIMETypeAVIF, MIMETypeSVG:
|
case MIMETypeJPEG, MIMETypePNG, MIMETypeWebP, MIMETypeGIF, MIMETypeAVIF,
|
||||||
|
MIMETypeJXL, MIMETypeSVG:
|
||||||
return true
|
return true
|
||||||
default:
|
default:
|
||||||
return false
|
return false
|
||||||
@@ -226,6 +240,8 @@ func MIMEToImageFormat(mimeType string) (ImageFormat, bool) {
|
|||||||
return FormatGIF, true
|
return FormatGIF, true
|
||||||
case MIMETypeAVIF:
|
case MIMETypeAVIF:
|
||||||
return FormatAVIF, true
|
return FormatAVIF, true
|
||||||
|
case MIMETypeJXL:
|
||||||
|
return FormatJXL, true
|
||||||
case MIMETypeSVG:
|
case MIMETypeSVG:
|
||||||
// SVG has no corresponding output format.
|
// SVG has no corresponding output format.
|
||||||
return "", false
|
return "", false
|
||||||
@@ -247,6 +263,8 @@ func ImageFormatToMIME(format ImageFormat) string {
|
|||||||
return string(MIMETypeGIF)
|
return string(MIMETypeGIF)
|
||||||
case FormatAVIF:
|
case FormatAVIF:
|
||||||
return string(MIMETypeAVIF)
|
return string(MIMETypeAVIF)
|
||||||
|
case FormatJXL:
|
||||||
|
return string(MIMETypeJXL)
|
||||||
case FormatOriginal:
|
case FormatOriginal:
|
||||||
// Original format passes content through unchanged.
|
// Original format passes content through unchanged.
|
||||||
return mimeOctetStream
|
return mimeOctetStream
|
||||||
|
|||||||
@@ -0,0 +1,72 @@
|
|||||||
|
package server
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/davidbyttow/govips/v2/vips"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestImageProxyFlowJPEGXL requests the JPEG XL testdata/red.jxl, 64x48 and
|
||||||
|
// made with libvips, through pixa's router, upstream fetcher, image processor
|
||||||
|
// and disk cache: resized as PNG, resized as JPEG XL, and at its own size and
|
||||||
|
// format, which is JPEG XL. The second request for each URL is a cache hit.
|
||||||
|
func TestImageProxyFlowJPEGXL(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
source, err := os.ReadFile("testdata/red.jxl")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("reading the test image: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
upstream := httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "image/jxl")
|
||||||
|
_, _ = w.Write(source)
|
||||||
|
}))
|
||||||
|
t.Cleanup(upstream.Close)
|
||||||
|
|
||||||
|
s, _, _ := startImageProxy(t, upstream)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
sizeFormat string // the <size>.<format> part of the image URL
|
||||||
|
contentType string
|
||||||
|
imageType vips.ImageType
|
||||||
|
width, height int
|
||||||
|
}{
|
||||||
|
{"32x24.png", "image/png", vips.ImageTypePNG, 32, 24},
|
||||||
|
{"32x24.jxl", "image/jxl", vips.ImageTypeJXL, 32, 24},
|
||||||
|
{"orig.orig", "image/jxl", vips.ImageTypeJXL, 64, 48},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
target := "/v1/image/" + upstreamHost + "/red.jxl/" + tt.sizeFormat
|
||||||
|
|
||||||
|
for _, wantCache := range []string{"MISS", "HIT"} {
|
||||||
|
rec := getImage(t, s, target)
|
||||||
|
gotType := rec.Header().Get("Content-Type")
|
||||||
|
gotCache := rec.Header().Get("X-Pixa-Cache")
|
||||||
|
|
||||||
|
if gotType != tt.contentType || gotCache != wantCache {
|
||||||
|
t.Errorf("%s: %s, X-Pixa-Cache %s, want %s, %s",
|
||||||
|
target, gotType, gotCache, tt.contentType, wantCache)
|
||||||
|
}
|
||||||
|
|
||||||
|
img, err := vips.NewImageFromBuffer(rec.Body.Bytes())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%s: loading the image: %v", target, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if img.Format() != tt.imageType ||
|
||||||
|
img.Width() != tt.width || img.Height() != tt.height {
|
||||||
|
t.Errorf("%s: %s %dx%d, want %s %dx%d", target,
|
||||||
|
vips.ImageTypes[img.Format()], img.Width(), img.Height(),
|
||||||
|
vips.ImageTypes[tt.imageType], tt.width, tt.height)
|
||||||
|
}
|
||||||
|
|
||||||
|
img.Close()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Vendored
BIN
Binary file not shown.
@@ -95,11 +95,12 @@
|
|||||||
</label>
|
</label>
|
||||||
<select id="format" name="format">
|
<select id="format" name="format">
|
||||||
<option value="orig" {{if eq .FormFormat "orig"}}selected{{end}}>Original</option>
|
<option value="orig" {{if eq .FormFormat "orig"}}selected{{end}}>Original</option>
|
||||||
<option value="auto" {{if eq .FormFormat "auto"}}selected{{end}}>Auto (AVIF, WebP or JPEG)</option>
|
<option value="auto" {{if eq .FormFormat "auto"}}selected{{end}}>Auto (JPEG XL, AVIF, WebP or JPEG)</option>
|
||||||
<option value="jpeg" {{if eq .FormFormat "jpeg"}}selected{{end}}>JPEG</option>
|
<option value="jpeg" {{if eq .FormFormat "jpeg"}}selected{{end}}>JPEG</option>
|
||||||
<option value="png" {{if eq .FormFormat "png"}}selected{{end}}>PNG</option>
|
<option value="png" {{if eq .FormFormat "png"}}selected{{end}}>PNG</option>
|
||||||
<option value="webp" {{if eq .FormFormat "webp"}}selected{{end}}>WebP</option>
|
<option value="webp" {{if eq .FormFormat "webp"}}selected{{end}}>WebP</option>
|
||||||
<option value="avif" {{if eq .FormFormat "avif"}}selected{{end}}>AVIF</option>
|
<option value="avif" {{if eq .FormFormat "avif"}}selected{{end}}>AVIF</option>
|
||||||
|
<option value="jxl" {{if eq .FormFormat "jxl"}}selected{{end}}>JPEG XL</option>
|
||||||
<option value="gif" {{if eq .FormFormat "gif"}}selected{{end}}>GIF</option>
|
<option value="gif" {{if eq .FormFormat "gif"}}selected{{end}}>GIF</option>
|
||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
Reference in new issue
Block a user