3 Commits
Author SHA1 Message Date
sneak e06192b7a5 Store a source the way a variant is stored, one row and one file per source URL (closes #308)
check / check (push) Canceled after 0s
A source is now one source_content row and one file named by its
SourceKey, the SHA-256 of its host, path and query, made as the variant
key is. A store writes the file, then upserts the row, and removes the
file when the row cannot be written; an eviction deletes the row, then
the file, with no lock. source_metadata, the JSON files under
cache/metadata/, MetadataStorage, the content lock, the eviction
transaction and its test hook are gone, as is the reconciliation pass
that dropped a source row whose file is missing, so
TestReconciliationReadsAPageAtATime now counts 15 rows read, not 20.
Two source URLs serving the same bytes store them twice. The tests of
the removed sharing and lock go with them; the two eviction
cancellation tests no longer use the hook.

Model: opus-5-5
2026-10-21 13:50:43 +00:00
clawbot 00d8ba26c5 One image request type instead of five (closes #307)
check / check (push) Canceled after 0s
The new package internal/imagerequest holds the request type, its two
parts Source and Transform, the Format and Fit types, the dimension and
fit checks, the /v1/image/ path parsing and the variant key; it imports
no other package of pixa. imgcache.ParsedURL, imgcache.ImageRequest,
signature.Request, imageprocessor.Request, the second Size, format and
fit types and the conversions between them are gone. The image
processor takes a Transform only. encurl keeps a private struct with the
token's short CBOR field names. VariantKey moves with the function that
makes it. Tests follow the moved and renamed code; the tests of the
removed conversions go with them.

Model: opus-5-5
2026-10-21 14:41:07 +02:00
clawbot 385c0ca76f Run only script/cibuild in the Gitea workflow (closes #203)
check / check (push) Canceled after 0s
The workflow's script/docker-smoke step is removed, so the workflow
checks out the repo and runs script/cibuild and nothing else, as the
canonical REPO_POLICIES.md says. The owner ruled on the issue that
make check covers what needs covering.

CI no longer starts the image and waits for its healthcheck.
script/docker-smoke and make docker-smoke stay, to run by hand, and
the script's header comment no longer says the workflow runs it. The
dated TODO.md entries that say the workflow runs it are history and
stay as written.

Model: opus-5-5
2026-10-21 13:22:59 +02:00
80 changed files with 2790 additions and 3509 deletions
-1
View File
@@ -18,4 +18,3 @@ jobs:
# All history and tags, so git describe finds the version tag.
fetch-depth: 0
- run: script/cibuild
- run: script/docker-smoke
+11 -11
View File
@@ -76,10 +76,9 @@ database and the disk cache:
pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
must be able to write the directory.
- `cache_max_bytes` limits the source and transformed images together. The
database, the metadata files, the `.meta` file beside each transformed image
and files still being written come on top, and eviction runs in the
background, so the cache can pass the limit for a while: leave room on the
volume beyond it.
database, the `.meta` file beside each transformed image and files still being
written come on top, and eviction runs in the background, so the cache can
pass the limit for a while: leave room on the volume beyond it.
- Set `cache_max_bytes` for a lasting deployment. Its default, worked out each
time pixa starts, is 75% of the sum of the space free on the volume and the
space the cached images already take, so a restart keeps the limit the cache
@@ -177,11 +176,9 @@ allowlisted source hosts for open access.
### Storage
- **Source content**:
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of source content>`
- **Source metadata**:
`<state_dir>/cache/metadata/<hostname>/<sha256 of path and query>.json` (host,
path and query, content hash, upstream status and headers, fetch time)
- **Source images**:
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of host, path and query>`, one
file for each source URL
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
- **Transformed images**:
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`,
@@ -190,8 +187,11 @@ allowlisted source hosts for open access.
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
name.
Multiple source paths may reference the same content blob; the database tracks
references rather than using filesystem refcounting.
Each source image and each transformed image has one row in the database,
holding its size and when it was last used. Two source URLs that serve the same
bytes are stored twice, once under each URL. A source image is stored by writing
its file and then its row; when the row cannot be written, the file is removed
again.
Source and transformed image files, and the `.meta` file beside each transformed
image, are written under a temporary name, synced to disk, and only then renamed
+24
View File
@@ -30,6 +30,30 @@ P2: security: per-origin rate limiting
# Completed Steps
- 2026-10-21 a source is stored the way a variant is (closes #308): one
`source_content` row and one file per source URL, the file named by the
source's `SourceKey`, the SHA-256 of its host, path and query made as the
variant key is, under `cache/sources/`. A store writes the file, then the row,
and removes the file when the row cannot be written; an eviction deletes the
row, then the file, with no lock and no transaction. `source_metadata`, the
JSON files under `cache/metadata/`, `MetadataStorage` and the content lock are
gone. Two source URLs that serve the same bytes now store them twice.
- 2026-10-21 one image request type instead of five (closes #307): the new
package `internal/imagerequest`, which imports no other package of pixa, holds
`imagerequest.Request` with its two parts, `Source` (host, path and query) and
`Transform` (size, format, quality and fit), the `Format` and `Fit` types, the
checks, the parsing of the `/v1/image/` path and the variant key.
`imgcache.ParsedURL`, `imgcache.ImageRequest`, `signature.Request`,
`imageprocessor.Request`, the second `Size`, format and fit types and the
conversions between them are gone. The image processor takes a `Transform`
only, and `encurl` keeps a private struct with the token's short CBOR field
names, so tokens made before still decode. Every URL of both image routes
answers as before.
- 2026-10-21 the Gitea workflow checks out the repo and runs `script/cibuild`,
nothing else (closes #203): its `script/docker-smoke` step is removed, as the
canonical `REPO_POLICIES.md` says, since `make check` covers what CI needs to
check. CI no longer starts the image and waits for its healthcheck.
`script/docker-smoke` and `make docker-smoke` stay, to run by hand.
- 2026-10-20 a WebP, AVIF or JPEG XL encode at the limits takes at most about 34
seconds (closes #272): with one libvips thread, 8192x8192 random pixels with
an alpha channel took about 61 seconds to encode as WebP, 81 as JPEG XL and,
@@ -131,7 +131,8 @@ func writeLikeOneRequest(ctx context.Context, db *sql.DB, key string) error {
}
_, err = db.ExecContext(ctx, `INSERT INTO source_content
(content_hash, content_type, size_bytes) VALUES (?, 'image/png', 1)`, key)
(source_key, source_host, source_path, content_type, size_bytes)
VALUES (?, 'example.com', '/image.png', 'image/png', 1)`, key)
if err != nil {
return fmt.Errorf("storing a source: %w", err)
}
+1 -1
View File
@@ -160,7 +160,7 @@ func TestApplyMigrations_CreatesSchemaAndTables(t *testing.T) {
// Verify that the application tables created by 001.sql exist.
tables := []string{
"source_content", "source_metadata", "output_content",
"source_content", "output_content",
"request_cache", "negative_cache", "cache_stats",
}
for _, table := range tables {
+7 -27
View File
@@ -1,10 +1,14 @@
-- Migration 001: Initial schema
-- Creates all tables for the pixa caching image proxy
-- Source content blobs
-- Files stored at: cache/sources/<ab>/<cd>/<sha256>
-- Source images, one row per source URL
-- Files stored at: cache/sources/<ab>/<cd>/<source_key>, where source_key
-- is the SHA-256 of the source's host, path and query
CREATE TABLE IF NOT EXISTS source_content (
content_hash TEXT PRIMARY KEY,
source_key TEXT PRIMARY KEY,
source_host TEXT NOT NULL,
source_path TEXT NOT NULL,
source_query TEXT NOT NULL DEFAULT '',
content_type TEXT NOT NULL,
size_bytes INTEGER NOT NULL,
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
@@ -13,30 +17,6 @@ CREATE TABLE IF NOT EXISTS source_content (
CREATE INDEX IF NOT EXISTS idx_source_content_last_accessed
ON source_content(last_accessed_at);
-- Source URL metadata - maps URLs to content hashes
-- JSON stored at: cache/metadata/<hostname>/<path_hash>.json
CREATE TABLE IF NOT EXISTS source_metadata (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_host TEXT NOT NULL,
source_path TEXT NOT NULL,
source_query TEXT NOT NULL DEFAULT '',
path_hash TEXT NOT NULL,
content_hash TEXT,
status_code INTEGER NOT NULL,
content_type TEXT,
response_headers TEXT,
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
expires_at DATETIME,
etag TEXT,
last_modified TEXT,
UNIQUE(source_host, source_path, source_query),
FOREIGN KEY (content_hash) REFERENCES source_content(content_hash)
);
CREATE INDEX IF NOT EXISTS idx_source_meta_host ON source_metadata(source_host);
CREATE INDEX IF NOT EXISTS idx_source_meta_path_hash ON source_metadata(path_hash);
CREATE INDEX IF NOT EXISTS idx_source_meta_expires ON source_metadata(expires_at);
CREATE INDEX IF NOT EXISTS idx_source_meta_content_hash ON source_metadata(content_hash);
-- Processed variant blobs
-- Files stored at: cache/variants/<ab>/<cd>/<cache_key> (plus a .meta
-- sidecar with the content type). Tracked here (like source content
+52 -65
View File
@@ -7,15 +7,15 @@ import (
"github.com/fxamacker/cbor/v2"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
)
// Default values for optional fields.
const (
DefaultQuality = 85
DefaultFormat = imgcache.FormatJXL
DefaultFitMode = imgcache.FitCover
DefaultFormat = imagerequest.FormatJXL
DefaultFitMode = imagerequest.FitCover
// HKDF salt for URL encryption key derivation
urlKeySalt = "pixa-urlenc-v1"
@@ -28,19 +28,19 @@ var (
ErrDecryptFailed = errors.New("failed to decrypt URL")
)
// Payload contains all image request parameters in a compact format.
// payload contains all image request parameters in a compact format.
// Field names are short to minimize encoded size.
// Fields with omitempty use sane defaults when absent.
type Payload struct {
SourceHost string `cbor:"h"` // required
SourcePath string `cbor:"p"` // required
SourceQuery string `cbor:"q,omitempty"` // optional
Width int `cbor:"w,omitempty"` // 0 = original
Height int `cbor:"ht,omitempty"` // 0 = original
Format imgcache.ImageFormat `cbor:"f,omitempty"` // default: jxl
Quality int `cbor:"ql,omitempty"` // default: 85
FitMode imgcache.FitMode `cbor:"fm,omitempty"` // default: cover
ExpiresAt int64 `cbor:"e,omitempty"` // 0 = never expires
type payload struct {
SourceHost string `cbor:"h"` // required
SourcePath string `cbor:"p"` // required
SourceQuery string `cbor:"q,omitempty"` // optional
Width int `cbor:"w,omitempty"` // 0 = original
Height int `cbor:"ht,omitempty"` // 0 = original
Format imagerequest.Format `cbor:"f,omitempty"` // default: jxl
Quality int `cbor:"ql,omitempty"` // default: 85
FitMode imagerequest.Fit `cbor:"fm,omitempty"` // default: cover
ExpiresAt int64 `cbor:"e,omitempty"` // 0 = never expires
}
// Generator creates and parses encrypted URL tokens.
@@ -59,9 +59,27 @@ func NewGenerator(signingKey string) (*Generator, error) {
return &Generator{key: key}, nil
}
// Generate creates an encrypted URL token from the payload.
// The token is CBOR-encoded, encrypted with NaCl secretbox, and base64url-encoded.
func (g *Generator) Generate(p *Payload) (string, error) {
// Generate creates an encrypted URL token from the request's source,
// transform and expiry, each as the request holds it; an empty format, a
// quality of 0 or an empty fit mode is left out of the token, and a zero
// Expires makes a token that never expires. The token is CBOR-encoded,
// encrypted with NaCl secretbox, and base64url-encoded.
func (g *Generator) Generate(req *imagerequest.Request) (string, error) {
p := &payload{
SourceHost: req.Source.Host,
SourcePath: req.Source.Path,
SourceQuery: req.Source.Query,
Width: req.Transform.Size.Width,
Height: req.Transform.Size.Height,
Format: req.Transform.Format,
Quality: req.Transform.Quality,
FitMode: req.Transform.FitMode,
}
if !req.Expires.IsZero() {
p.ExpiresAt = req.Expires.Unix()
}
// CBOR encode the payload
data, err := cbor.Marshal(p)
if err != nil {
@@ -72,9 +90,11 @@ func (g *Generator) Generate(p *Payload) (string, error) {
return seal.Encrypt(g.key, data)
}
// Parse decrypts and validates an encrypted URL token.
// Parse decrypts and validates an encrypted URL token and returns the
// request it holds, with default values for omitted optional fields. A token
// that never expires gives the zero Expires.
// Returns ErrExpired if the token has expired, or other errors for invalid tokens.
func (g *Generator) Parse(token string) (*Payload, error) {
func (g *Generator) Parse(token string) (*imagerequest.Request, error) {
// Decrypt
data, err := seal.Decrypt(g.key, token)
if err != nil {
@@ -87,7 +107,7 @@ func (g *Generator) Parse(token string) (*Payload, error) {
}
// CBOR decode
var p Payload
var p payload
err = cbor.Unmarshal(data, &p)
if err != nil {
@@ -99,13 +119,6 @@ func (g *Generator) Parse(token string) (*Payload, error) {
return nil, ErrExpired
}
return &p, nil
}
// ToImageRequest converts the payload to an ImageRequest.
// Applies default values for omitted optional fields. An ExpiresAt of 0, a URL
// that never expires, gives the zero Expires.
func (p *Payload) ToImageRequest() *imgcache.ImageRequest {
format := p.Format
if format == "" {
format = DefaultFormat
@@ -126,44 +139,18 @@ func (p *Payload) ToImageRequest() *imgcache.ImageRequest {
expires = time.Unix(p.ExpiresAt, 0)
}
return &imgcache.ImageRequest{
SourceHost: p.SourceHost,
SourcePath: p.SourcePath,
SourceQuery: p.SourceQuery,
Size: imgcache.Size{
Width: p.Width,
Height: p.Height,
return &imagerequest.Request{
Source: imagerequest.Source{
Host: p.SourceHost,
Path: p.SourcePath,
Query: p.SourceQuery,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: p.Width, Height: p.Height},
Format: format,
Quality: quality,
FitMode: fitMode,
},
Format: format,
Quality: quality,
FitMode: fitMode,
Expires: expires,
}
}
// FromImageRequest creates a Payload from an ImageRequest with the given expiration.
func FromImageRequest(req *imgcache.ImageRequest, expiresAt time.Time) *Payload {
p := &Payload{
SourceHost: req.SourceHost,
SourcePath: req.SourcePath,
SourceQuery: req.SourceQuery,
Width: req.Size.Width,
Height: req.Size.Height,
ExpiresAt: expiresAt.Unix(),
}
// Only set non-default values to benefit from omitempty
if req.Format != "" && req.Format != DefaultFormat {
p.Format = req.Format
}
if req.Quality != 0 && req.Quality != DefaultQuality {
p.Quality = req.Quality
}
if req.FitMode != "" && req.FitMode != DefaultFitMode {
p.FitMode = req.FitMode
}
return p
}, nil
}
+91 -174
View File
@@ -6,7 +6,7 @@ import (
"time"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// Shared test fixture strings.
@@ -24,19 +24,22 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
t.Fatalf("NewGenerator() error = %v", err)
}
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
SourceQuery: testSourceQuery,
Width: 800,
Height: 600,
Format: imgcache.FormatWebP,
Quality: 90,
FitMode: imgcache.FitContain,
ExpiresAt: time.Now().Add(time.Hour).Unix(),
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
Path: testSourcePath,
Query: testSourceQuery,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: imagerequest.FormatWebP,
Quality: 90,
FitMode: imagerequest.FitContain,
},
Expires: time.Now().Add(time.Hour),
}
token, err := gen.Generate(payload)
token, err := gen.Generate(req)
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
@@ -51,40 +54,45 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
}
// Verify all fields
if parsed.SourceHost != payload.SourceHost {
t.Errorf("SourceHost = %q, want %q", parsed.SourceHost, payload.SourceHost)
if parsed.Source.Host != req.Source.Host {
t.Errorf("SourceHost = %q, want %q", parsed.Source.Host, req.Source.Host)
}
if parsed.SourcePath != payload.SourcePath {
t.Errorf("SourcePath = %q, want %q", parsed.SourcePath, payload.SourcePath)
if parsed.Source.Path != req.Source.Path {
t.Errorf("SourcePath = %q, want %q", parsed.Source.Path, req.Source.Path)
}
if parsed.SourceQuery != payload.SourceQuery {
t.Errorf("SourceQuery = %q, want %q", parsed.SourceQuery, payload.SourceQuery)
if parsed.Source.Query != req.Source.Query {
t.Errorf("SourceQuery = %q, want %q", parsed.Source.Query, req.Source.Query)
}
if parsed.Width != payload.Width {
t.Errorf("Width = %d, want %d", parsed.Width, payload.Width)
if parsed.Transform.Size.Width != req.Transform.Size.Width {
t.Errorf("Width = %d, want %d",
parsed.Transform.Size.Width, req.Transform.Size.Width)
}
if parsed.Height != payload.Height {
t.Errorf("Height = %d, want %d", parsed.Height, payload.Height)
if parsed.Transform.Size.Height != req.Transform.Size.Height {
t.Errorf("Height = %d, want %d",
parsed.Transform.Size.Height, req.Transform.Size.Height)
}
if parsed.Format != payload.Format {
t.Errorf("Format = %q, want %q", parsed.Format, payload.Format)
if parsed.Transform.Format != req.Transform.Format {
t.Errorf("Format = %q, want %q",
parsed.Transform.Format, req.Transform.Format)
}
if parsed.Quality != payload.Quality {
t.Errorf("Quality = %d, want %d", parsed.Quality, payload.Quality)
if parsed.Transform.Quality != req.Transform.Quality {
t.Errorf("Quality = %d, want %d",
parsed.Transform.Quality, req.Transform.Quality)
}
if parsed.FitMode != payload.FitMode {
t.Errorf("FitMode = %q, want %q", parsed.FitMode, payload.FitMode)
if parsed.Transform.FitMode != req.Transform.FitMode {
t.Errorf("FitMode = %q, want %q",
parsed.Transform.FitMode, req.Transform.FitMode)
}
if parsed.ExpiresAt != payload.ExpiresAt {
t.Errorf("ExpiresAt = %d, want %d", parsed.ExpiresAt, payload.ExpiresAt)
if parsed.Expires.Unix() != req.Expires.Unix() {
t.Errorf("ExpiresAt = %d, want %d", parsed.Expires.Unix(), req.Expires.Unix())
}
}
@@ -93,13 +101,15 @@ func TestGenerator_Parse_Expired(t *testing.T) {
gen, _ := encurl.NewGenerator("test-signing-key-12345")
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
ExpiresAt: time.Now().Add(-time.Hour).Unix(), // Already expired
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
Path: testSourcePath,
},
Expires: time.Now().Add(-time.Hour), // Already expired
}
token, err := gen.Generate(payload)
token, err := gen.Generate(req)
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
@@ -130,13 +140,15 @@ func TestGenerator_Parse_TamperedToken(t *testing.T) {
gen, _ := encurl.NewGenerator("test-signing-key-12345")
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
ExpiresAt: time.Now().Add(time.Hour).Unix(),
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
Path: testSourcePath,
},
Expires: time.Now().Add(time.Hour),
}
token, _ := gen.Generate(payload)
token, _ := gen.Generate(req)
// Tamper with the token
tampered := []byte(token)
@@ -156,13 +168,15 @@ func TestGenerator_Parse_WrongKey(t *testing.T) {
gen1, _ := encurl.NewGenerator("signing-key-1")
gen2, _ := encurl.NewGenerator("signing-key-2")
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
ExpiresAt: time.Now().Add(time.Hour).Unix(),
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
Path: testSourcePath,
},
Expires: time.Now().Add(time.Hour),
}
token, _ := gen1.Generate(payload)
token, _ := gen1.Generate(req)
_, err := gen2.Parse(token)
if err == nil {
@@ -170,140 +184,41 @@ func TestGenerator_Parse_WrongKey(t *testing.T) {
}
}
func TestPayload_ToImageRequest(t *testing.T) {
func TestGenerator_Parse_Defaults(t *testing.T) {
t.Parallel()
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
SourceQuery: testSourceQuery,
Width: 800,
Height: 600,
Format: imgcache.FormatWebP,
Quality: 90,
FitMode: imgcache.FitContain,
ExpiresAt: time.Now().Add(time.Hour).Unix(),
gen, _ := encurl.NewGenerator("test-signing-key-12345")
// Request with only required fields - should get defaults
token, err := gen.Generate(&imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
Path: testSourcePath,
},
Expires: time.Now().Add(time.Hour),
})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
req := payload.ToImageRequest()
if req.SourceHost != payload.SourceHost {
t.Errorf("SourceHost = %q, want %q", req.SourceHost, payload.SourceHost)
req, err := gen.Parse(token)
if err != nil {
t.Fatalf("Parse() error = %v", err)
}
if req.SourcePath != payload.SourcePath {
t.Errorf("SourcePath = %q, want %q", req.SourcePath, payload.SourcePath)
if req.Transform.Format != encurl.DefaultFormat {
t.Errorf("Format = %q, want default %q",
req.Transform.Format, encurl.DefaultFormat)
}
if req.SourceQuery != payload.SourceQuery {
t.Errorf("SourceQuery = %q, want %q", req.SourceQuery, payload.SourceQuery)
if req.Transform.Quality != encurl.DefaultQuality {
t.Errorf("Quality = %d, want default %d",
req.Transform.Quality, encurl.DefaultQuality)
}
if req.Size.Width != payload.Width {
t.Errorf("Width = %d, want %d", req.Size.Width, payload.Width)
}
if req.Size.Height != payload.Height {
t.Errorf("Height = %d, want %d", req.Size.Height, payload.Height)
}
if req.Format != payload.Format {
t.Errorf("Format = %q, want %q", req.Format, payload.Format)
}
if req.Quality != payload.Quality {
t.Errorf("Quality = %d, want %d", req.Quality, payload.Quality)
}
if req.FitMode != payload.FitMode {
t.Errorf("FitMode = %q, want %q", req.FitMode, payload.FitMode)
}
}
func TestPayload_ToImageRequest_Defaults(t *testing.T) {
t.Parallel()
// Payload with only required fields - should get defaults
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
ExpiresAt: time.Now().Add(time.Hour).Unix(),
}
req := payload.ToImageRequest()
if req.Format != encurl.DefaultFormat {
t.Errorf("Format = %q, want default %q", req.Format, encurl.DefaultFormat)
}
if req.Quality != encurl.DefaultQuality {
t.Errorf("Quality = %d, want default %d", req.Quality, encurl.DefaultQuality)
}
if req.FitMode != encurl.DefaultFitMode {
t.Errorf("FitMode = %q, want default %q", req.FitMode, encurl.DefaultFitMode)
}
}
func TestFromImageRequest(t *testing.T) {
t.Parallel()
req := &imgcache.ImageRequest{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
SourceQuery: testSourceQuery,
Size: imgcache.Size{Width: 800, Height: 600},
Format: imgcache.FormatWebP,
Quality: 90,
FitMode: imgcache.FitContain,
}
expiresAt := time.Now().Add(time.Hour)
payload := encurl.FromImageRequest(req, expiresAt)
if payload.SourceHost != req.SourceHost {
t.Errorf("SourceHost = %q, want %q", payload.SourceHost, req.SourceHost)
}
if payload.SourcePath != req.SourcePath {
t.Errorf("SourcePath = %q, want %q", payload.SourcePath, req.SourcePath)
}
if payload.Width != req.Size.Width {
t.Errorf("Width = %d, want %d", payload.Width, req.Size.Width)
}
if payload.ExpiresAt != expiresAt.Unix() {
t.Errorf("ExpiresAt = %d, want %d", payload.ExpiresAt, expiresAt.Unix())
}
}
func TestFromImageRequest_OmitsDefaults(t *testing.T) {
t.Parallel()
// Request with default values - payload should omit them for
// smaller encoding
req := &imgcache.ImageRequest{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
Format: encurl.DefaultFormat,
Quality: encurl.DefaultQuality,
FitMode: encurl.DefaultFitMode,
}
payload := encurl.FromImageRequest(req, time.Now().Add(time.Hour))
// These should be zero/empty because they match defaults
if payload.Format != "" {
t.Errorf("Format should be empty for default, got %q", payload.Format)
}
if payload.Quality != 0 {
t.Errorf("Quality should be 0 for default, got %d", payload.Quality)
}
if payload.FitMode != "" {
t.Errorf("FitMode should be empty for default, got %q", payload.FitMode)
if req.Transform.FitMode != encurl.DefaultFitMode {
t.Errorf("FitMode = %q, want default %q",
req.Transform.FitMode, encurl.DefaultFitMode)
}
}
@@ -312,13 +227,15 @@ func TestGenerator_TokenIsURLSafe(t *testing.T) {
gen, _ := encurl.NewGenerator("test-signing-key-12345")
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
ExpiresAt: time.Now().Add(time.Hour).Unix(),
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
Path: testSourcePath,
},
Expires: time.Now().Add(time.Hour),
}
token, _ := gen.Generate(payload)
token, _ := gen.Generate(req)
// Check that token only contains URL-safe characters
for _, c := range token {
+33 -35
View File
@@ -16,7 +16,7 @@ import (
"sneak.berlin/go/pixa/internal/clientip"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/templates"
)
@@ -145,7 +145,7 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
return
}
payload, expiresAt, ttl, err := buildGeneratePayload(parsed, r.Form)
req, expiresAt, ttl, err := buildGenerateRequest(parsed, r.Form)
if err != nil {
s.renderGeneratorWithForm(w, r, err.Error(), r.Form,
http.StatusBadRequest)
@@ -154,9 +154,8 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
}
// The image URL must fetch the source URL as it was written
sourceRequest := payload.ToImageRequest()
sourceRequest.AllowHTTP = s.config.AllowHTTP
fetchURL := sourceRequest.SourceURL()
req.AllowHTTP = s.config.AllowHTTP
fetchURL := req.SourceURL()
if !fetchedAsWritten(parsed, fetchURL) {
s.renderGeneratorWithForm(w, r, "Source URL cannot be fetched as "+
@@ -167,7 +166,7 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
}
// Generate encrypted token
token, err := s.encGen.Generate(payload)
token, err := s.encGen.Generate(req)
if err != nil {
s.log.Error("failed to generate encrypted URL", "error", err)
s.renderGeneratorWithForm(w, r, "Failed to generate URL", r.Form,
@@ -240,14 +239,15 @@ func fetchedAsWritten(source *url.URL, fetchURL string) bool {
fetched.RawQuery == source.RawQuery
}
// buildGeneratePayload parses the numeric form fields and assembles the
// encrypted URL payload. ttl=0 means never expires (ExpiresAt stays 0). A
// non-numeric or out-of-range width, height, quality or ttl, or an
// unrecognized fit mode, is a client error naming the offending field. The
// format field is passed through unchecked.
func buildGeneratePayload(
// buildGenerateRequest parses the numeric form fields and assembles the
// image request the encrypted URL's token is made from. ttl=0 means never
// expires (Expires stays the zero time). A non-numeric or out-of-range
// width, height, quality or ttl, or an unrecognized fit mode, is a client
// error naming the offending field. The format field is passed through
// unchecked.
func buildGenerateRequest(
parsed *url.URL, form url.Values,
) (*encurl.Payload, time.Time, int, error) {
) (*imagerequest.Request, time.Time, int, error) {
width, err := parseFormDimension(form, "width")
if err != nil {
return nil, time.Time{}, 0, err
@@ -269,42 +269,40 @@ func buildGeneratePayload(
return nil, time.Time{}, 0, err
}
fitMode := imgcache.FitMode(form.Get("fit"))
fitMode := imagerequest.Fit(form.Get("fit"))
err = imgcache.ValidateFitMode(fitMode)
err = imagerequest.ValidateFitMode(fitMode)
if err != nil {
return nil, time.Time{}, 0,
fmt.Errorf("%w: %s", imgcache.ErrInvalidFitMode, form.Get("fit"))
fmt.Errorf("%w: %s", imagerequest.ErrInvalidFitMode, form.Get("fit"))
}
var (
expiresAt time.Time
expiresAtUnix int64
)
var expiresAt time.Time
if ttl > 0 {
expiresAt = time.Now().Add(time.Duration(ttl) * time.Second)
expiresAtUnix = expiresAt.Unix()
}
payload := &encurl.Payload{
SourceHost: parsed.Host,
SourcePath: parsed.Path,
SourceQuery: parsed.RawQuery,
Width: width,
Height: height,
Format: imgcache.ImageFormat(form.Get("format")),
Quality: quality,
FitMode: fitMode,
ExpiresAt: expiresAtUnix,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: parsed.Host,
Path: parsed.Path,
Query: parsed.RawQuery,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: width, Height: height},
Format: imagerequest.Format(form.Get("format")),
Quality: quality,
FitMode: fitMode,
},
Expires: expiresAt,
}
return payload, expiresAt, ttl, nil
return req, expiresAt, ttl, nil
}
// parseFormDimension reads an optional width or height form field. An empty
// value means "original size" (0). A non-numeric value, or one
// imgcache.ValidateDimension rejects, is an error naming the field.
// imagerequest.ValidateDimension rejects, is an error naming the field.
func parseFormDimension(form url.Values, field string) (int, error) {
raw := form.Get(field)
if raw == "" {
@@ -316,7 +314,7 @@ func parseFormDimension(form url.Values, field string) (int, error) {
return 0, fmt.Errorf("%w %s: not a number", errInvalidFormField, field)
}
err = imgcache.ValidateDimension(field, value)
err = imagerequest.ValidateDimension(field, value)
if err != nil {
return 0, err
}
@@ -11,7 +11,7 @@ import (
"time"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// Generator form field names, and a value that is not a number.
@@ -125,10 +125,10 @@ func TestGeneratePostRejectsBadField(t *testing.T) {
}
}
// TestBuildGeneratePayloadDefaultAndLimits verifies that an empty quality
// TestBuildGenerateRequestDefaultAndLimits verifies that an empty quality
// takes the default, and that the largest accepted width, height, quality and
// ttl are accepted with an expiry still in the future.
func TestBuildGeneratePayloadDefaultAndLimits(t *testing.T) {
func TestBuildGenerateRequestDefaultAndLimits(t *testing.T) {
t.Parallel()
parsed, err := url.Parse(testSourceURL)
@@ -136,19 +136,19 @@ func TestBuildGeneratePayloadDefaultAndLimits(t *testing.T) {
t.Fatalf("url.Parse() error = %v", err)
}
payload, _, _, err := buildGeneratePayload(parsed, url.Values{})
req, _, _, err := buildGenerateRequest(parsed, url.Values{})
if err != nil {
t.Fatalf("empty form: error = %v", err)
}
if payload.Quality != encurl.DefaultQuality {
if req.Transform.Quality != encurl.DefaultQuality {
t.Errorf("empty quality gave %d, want %d",
payload.Quality, encurl.DefaultQuality)
req.Transform.Quality, encurl.DefaultQuality)
}
_, expiresAt, _, err := buildGeneratePayload(parsed, url.Values{
widthField: {strconv.Itoa(imgcache.MaxDimension)},
heightField: {strconv.Itoa(imgcache.MaxDimension)},
_, expiresAt, _, err := buildGenerateRequest(parsed, url.Values{
widthField: {strconv.Itoa(imagerequest.MaxDimension)},
heightField: {strconv.Itoa(imagerequest.MaxDimension)},
qualityField: {strconv.Itoa(maxQuality)},
ttlField: {strconv.Itoa(maxTTL)},
})
@@ -10,7 +10,7 @@ import (
"testing"
"time"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/session"
)
@@ -314,7 +314,7 @@ func TestGeneratePost_URLServesImage(t *testing.T) {
sourceURLField: {"https://" + signedHost + photoPath},
widthField: {"50"},
heightField: {"50"},
formatField: {string(imgcache.FormatJPEG)},
formatField: {string(imagerequest.FormatJPEG)},
})
if rec.Code != http.StatusOK {
@@ -347,7 +347,7 @@ func TestGeneratePost_URLWithTTLExpires(t *testing.T) {
sourceURLField: {"https://" + signedHost + photoPath},
widthField: {"50"},
heightField: {"50"},
formatField: {string(imgcache.FormatJPEG)},
formatField: {string(imagerequest.FormatJPEG)},
ttlField: {"1"},
})
@@ -9,7 +9,7 @@ import (
"testing"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError and
@@ -26,7 +26,7 @@ func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
plain := httptest.NewRecorder()
h.respondImageError(plain, &imgcache.ImageRequest{}, err)
h.respondImageError(plain, &imagerequest.Request{}, err)
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
+10 -6
View File
@@ -14,6 +14,7 @@ import (
"github.com/go-chi/chi/v5"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
@@ -75,12 +76,15 @@ func mismatchedSourcePlainURL(*testing.T, *Handlers) string {
func mismatchedSourceEncURL(t *testing.T, h *Handlers) string {
t.Helper()
token, err := h.encGen.Generate(&encurl.Payload{
SourceHost: allowlistedHost,
SourcePath: mismatchedSourcePath,
Width: 50,
Height: 50,
Format: imgcache.FormatJPEG,
token, err := h.encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{
Host: allowlistedHost,
Path: mismatchedSourcePath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
})
if err != nil {
t.Fatalf("Generate() error = %v", err)
+10 -10
View File
@@ -8,7 +8,7 @@ import (
"strconv"
"strings"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// Errors for an Accept header that an auto URL cannot be served for.
@@ -24,9 +24,9 @@ var (
// header that is not valid and 406 for one that allows none of the formats,
// and reports whether req can be served. Any other format is left as it is.
func (s *Handlers) chooseAutoFormat(
w http.ResponseWriter, r *http.Request, req *imgcache.ImageRequest,
w http.ResponseWriter, r *http.Request, req *imagerequest.Request,
) bool {
if req.Format != imgcache.FormatAuto {
if req.Transform.Format != imagerequest.FormatAuto {
return true
}
@@ -45,7 +45,7 @@ func (s *Handlers) chooseAutoFormat(
return false
}
req.Format = format
req.Transform.Format = format
return true
}
@@ -56,26 +56,26 @@ func (s *Handlers) chooseAutoFormat(
// specific entry of image/jpeg, image/* and */* allows it, or when it names
// nothing. A q of 0 refuses a format. JPEG XL, AVIF and WebP must be named, as
// clients that cannot show them send image/* and */* too.
func formatForAccept(accept string) (imgcache.ImageFormat, error) {
func formatForAccept(accept string) (imagerequest.Format, error) {
qualities, err := parseAccept(accept)
if err != nil {
return "", err
}
if len(qualities) == 0 {
return imgcache.FormatJPEG, nil
return imagerequest.FormatJPEG, nil
}
if qualities["image/jxl"] > 0 {
return imgcache.FormatJXL, nil
return imagerequest.FormatJXL, nil
}
if qualities["image/avif"] > 0 {
return imgcache.FormatAVIF, nil
return imagerequest.FormatAVIF, nil
}
if qualities["image/webp"] > 0 {
return imgcache.FormatWebP, nil
return imagerequest.FormatWebP, nil
}
// For JPEG, the most specific entry the header has decides
@@ -89,7 +89,7 @@ func formatForAccept(accept string) (imgcache.ImageFormat, error) {
}
if quality > 0 {
return imgcache.FormatJPEG, nil
return imagerequest.FormatJPEG, nil
}
return "", errNotAcceptable
+40 -30
View File
@@ -12,8 +12,7 @@ import (
"github.com/go-chi/chi/v5"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// The content types the tests below expect.
@@ -33,24 +32,24 @@ func TestFormatForAccept(t *testing.T) {
tests := []struct {
name string
accept string
want imgcache.ImageFormat
want imagerequest.Format
wantErr error
}{
{"AVIF-capable browser",
"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
imgcache.FormatAVIF, nil},
imagerequest.FormatAVIF, nil},
{"WebP-capable browser",
"image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5",
imgcache.FormatWebP, nil},
{"WebP only", webpType, imgcache.FormatWebP, nil},
{"neither", "image/png,image/*;q=0.8,*/*;q=0.5", imgcache.FormatJPEG, nil},
{"wildcard only", "*/*", imgcache.FormatJPEG, nil},
{"image wildcard only", "image/*", imgcache.FormatJPEG, nil},
{"absent", "", imgcache.FormatJPEG, nil},
{"q=0 on AVIF", "image/avif;q=0,image/webp,*/*", imgcache.FormatWebP, nil},
imagerequest.FormatWebP, nil},
{"WebP only", webpType, imagerequest.FormatWebP, nil},
{"neither", "image/png,image/*;q=0.8,*/*;q=0.5", imagerequest.FormatJPEG, nil},
{"wildcard only", "*/*", imagerequest.FormatJPEG, nil},
{"image wildcard only", "image/*", imagerequest.FormatJPEG, nil},
{"absent", "", imagerequest.FormatJPEG, nil},
{"q=0 on AVIF", "image/avif;q=0,image/webp,*/*", imagerequest.FormatWebP, nil},
{"AVIF named twice, once with q=0", "image/avif,image/avif;q=0.0,*/*",
imgcache.FormatJPEG, nil},
{"upper case and spaces", " Image/AVIF ; Q=0.5 ", imgcache.FormatAVIF, nil},
imagerequest.FormatJPEG, nil},
{"upper case and spaces", " Image/AVIF ; Q=0.5 ", imagerequest.FormatAVIF, nil},
{"q=0 on JPEG", "image/jpeg;q=0,image/*", "", errNotAcceptable},
{"q=0 on everything", "*/*;q=0", "", errNotAcceptable},
{"PNG only", "image/png", "", errNotAcceptable},
@@ -80,23 +79,30 @@ func TestFormatForAccept(t *testing.T) {
func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) {
t.Helper()
signedURL, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: photoPath,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatAuto,
signedURL, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatAuto,
},
}, time.Minute)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
token, err := h.encGen.Generate(&encurl.Payload{
SourceHost: signedHost,
SourcePath: photoPath,
Width: 50,
Height: 50,
Format: imgcache.FormatAuto,
ExpiresAt: time.Now().Add(time.Minute).Unix(),
token, err := h.encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatAuto,
},
Expires: time.Now().Add(time.Minute),
})
if err != nil {
t.Fatalf("Generate() error = %v", err)
@@ -184,11 +190,15 @@ func TestFormatAuto_SignatureCoversAuto(t *testing.T) {
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
signedForAVIF, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: photoPath,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatAVIF,
signedForAVIF, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatAVIF,
},
}, time.Minute)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
+19 -21
View File
@@ -13,6 +13,7 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic"
)
@@ -35,8 +36,8 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
err := s.imgSvc.ValidateRequest(req)
if err != nil {
s.log.Warn("signature validation failed",
"host", req.SourceHost,
"path", req.SourcePath,
"host", req.Source.Host,
"path", req.Source.Path,
"error", err,
)
s.respondError(w, "unauthorized", http.StatusUnauthorized)
@@ -50,7 +51,7 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
}
// Get cache key for logging
cacheKey := imgcache.CacheKey(req)
cacheKey := imagerequest.CacheKey(req)
// Get the image (from cache or fetch/process)
startTime := time.Now()
@@ -81,17 +82,17 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
}
// parseImageRequest parses the image URL's path and query parameters into
// an ImageRequest. On invalid input it writes an error response and
// an image request. On invalid input it writes an error response and
// returns false.
func (s *Handlers) parseImageRequest(
w http.ResponseWriter, r *http.Request,
) (*imgcache.ImageRequest, bool) {
) (*imagerequest.Request, bool) {
// The path is parsed decoded once, as r.URL.Path holds it, and not from
// chi's wildcard: chi routes on the path as the client wrote it whenever
// that is not Go's own encoding of it, so its wildcard would still hold
// the %3F before a source query when, say, a "/" in the query is written
// as %2F
parsed, err := imgcache.ParseImageURL(r.URL.Path)
req, err := imagerequest.ParseImageURL(r.URL.Path)
if err != nil {
s.log.Warn("failed to parse image URL",
"path", r.URL.Path,
@@ -102,9 +103,6 @@ func (s *Handlers) parseImageRequest(
return nil, false
}
// Convert to ImageRequest
req := parsed.ToImageRequest()
// Parse signature params from query string. r.URL.Query() would silently
// drop a pair it cannot decode, such as q=80%, so that q would be served
// at 85; a query string that cannot be decoded is refused instead. A
@@ -148,7 +146,7 @@ func (s *Handlers) parseImageRequest(
return nil, false
}
req.Quality, err = parseFormInt(query, "q",
req.Transform.Quality, err = parseFormInt(query, "q",
encurl.DefaultQuality, minQuality, maxQuality)
if err != nil {
s.respondError(w, fmt.Sprintf("%v, got %q", err, qStr),
@@ -160,23 +158,23 @@ func (s *Handlers) parseImageRequest(
// Only a fit missing from the URL is cover. A fit in the URL that is not a
// fit mode is refused by the fit-mode check below; that check would take an
// empty fit as missing, so an empty one is refused here.
req.FitMode = imgcache.FitMode(query.Get("fit"))
if query.Has("fit") && req.FitMode == "" {
req.Transform.FitMode = imagerequest.Fit(query.Get("fit"))
if query.Has("fit") && req.Transform.FitMode == "" {
s.respondError(w, `invalid fit: not a fit mode, got ""`, http.StatusBadRequest)
return nil, false
}
// Default fit mode if not set
if req.FitMode == "" {
req.FitMode = imgcache.FitCover
if req.Transform.FitMode == "" {
req.Transform.FitMode = imagerequest.FitCover
}
// Enforce dimension and fit-mode bounds, shared with the encrypted-URL
// route. Dimensions are already bounded by the path parser above; this
// also rejects an unrecognized fit mode with 400 instead of letting it
// reach the processor as a 500.
err = imgcache.ValidateImageRequest(req)
err = imagerequest.ValidateImageRequest(req)
if err != nil {
s.respondError(w, "invalid image request: "+err.Error(),
http.StatusBadRequest)
@@ -219,7 +217,7 @@ func (s *Handlers) HandleTooManyRequests() http.HandlerFunc {
// respondImageError maps image retrieval errors to HTTP responses.
func (s *Handlers) respondImageError(
w http.ResponseWriter, req *imgcache.ImageRequest, err error,
w http.ResponseWriter, req *imagerequest.Request, err error,
) {
// An upstream body whose first bytes are not those of the format its
// Content-Type names is the upstream's error, not pixa's: the image
@@ -234,8 +232,8 @@ func (s *Handlers) respondImageError(
}
s.log.Error("failed to get image",
"host", req.SourceHost,
"path", req.SourcePath,
"host", req.Source.Host,
"path", req.Source.Path,
"error", err,
)
@@ -346,8 +344,8 @@ func notModified(w http.ResponseWriter, r *http.Request, etag string) bool {
// handling conditional and HEAD requests.
func (s *Handlers) writeImageResponse(
w http.ResponseWriter, r *http.Request,
req *imgcache.ImageRequest, resp *imgcache.ImageResponse,
cacheKey imgcache.VariantKey, startTime time.Time,
req *imagerequest.Request, resp *imgcache.ImageResponse,
cacheKey imagerequest.VariantKey, startTime time.Time,
) {
// Set response headers
w.Header().Set("Content-Type", resp.ContentType)
@@ -388,7 +386,7 @@ func (s *Handlers) writeImageResponse(
"cache_key", cacheKey,
"cache_status", resp.CacheStatus,
"duration_ms", duration.Milliseconds(),
"format", req.Format,
"format", req.Transform.Format,
"served_bytes", servedBytes,
"fetched_bytes", resp.FetchedBytes,
)
@@ -14,6 +14,7 @@ import (
"github.com/go-chi/chi/v5"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
@@ -107,11 +108,15 @@ func TestHandleImage_SignedURL_MaxAgeEndsAtExp(t *testing.T) {
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
signedURL, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: photoPath,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
signedURL, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, time.Minute)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
@@ -168,13 +173,13 @@ func TestHandleImageEnc_MaxAge(t *testing.T) {
tests := []struct {
name string
expiresAt int64
expires time.Time
wantAtLeast int
wantAtMost int
}{
{"60 second TTL", time.Now().Add(time.Minute).Unix(), 50, 60},
{"two-year TTL", time.Now().Add(2 * 365 * 24 * time.Hour).Unix(), 31536000, 31536000},
{"no TTL", 0, 31536000, 31536000},
{"60 second TTL", time.Now().Add(time.Minute), 50, 60},
{"two-year TTL", time.Now().Add(2 * 365 * 24 * time.Hour), 31536000, 31536000},
{"no TTL", time.Time{}, 31536000, 31536000},
}
for _, tt := range tests {
@@ -183,13 +188,16 @@ func TestHandleImageEnc_MaxAge(t *testing.T) {
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
token, err := h.encGen.Generate(&encurl.Payload{
SourceHost: signedHost,
SourcePath: photoPath,
Width: 50,
Height: 50,
Format: imgcache.FormatJPEG,
ExpiresAt: tt.expiresAt,
token, err := h.encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
Expires: tt.expires,
})
if err != nil {
t.Fatalf("Generate() error = %v", err)
+25 -18
View File
@@ -14,6 +14,7 @@ import (
"github.com/go-chi/chi/v5"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -84,15 +85,18 @@ func photoURLWithSig(host, sig string, expires time.Time) string {
// photoSignature returns the signature of photoURL(host) at the default
// quality and fit, made with key and expiring at expires.
func photoSignature(key, host string, expires time.Time) string {
return signature.New(key).Sign(&signature.Request{
SourceHost: host,
SourcePath: photoPath,
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: expires,
return signature.New(key).Sign(&imagerequest.Request{
Source: imagerequest.Source{
Host: host,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: expires,
})
}
@@ -353,15 +357,18 @@ func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
photos := newPhotoFetcher(t, allowlistedHost, signedHost)
exp := time.Now().Add(time.Hour)
sig := signature.New(testSigningKey).Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: photoPath + "#x",
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: exp,
sig := signature.New(testSigningKey).Sign(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath + "#x",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: exp,
})
tests := []struct {
@@ -9,7 +9,7 @@ import (
"time"
"github.com/go-chi/chi/v5"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -44,15 +44,18 @@ func TestHandleImage_SignatureCoversQualityAndFit(t *testing.T) {
expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: "/images/photo.jpg",
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: expires,
sig := signer.Sign(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: "/images/photo.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: expires,
})
signedURL := fmt.Sprintf("/v1/image/%s/images/photo.jpg/50x50.jpeg?sig=%s&exp=%d",
signedHost, sig, expires.Unix())
@@ -88,9 +91,9 @@ func TestHandleImage_GeneratedSignedURLVerifies(t *testing.T) {
tests := []struct {
name string
quality int
fitMode imgcache.FitMode
fitMode imagerequest.Fit
}{
{"quality 40 and fit contain", 40, imgcache.FitContain},
{"quality 40 and fit contain", 40, imagerequest.FitContain},
{"quality and fit unset", 0, ""},
}
@@ -100,13 +103,17 @@ func TestHandleImage_GeneratedSignedURLVerifies(t *testing.T) {
fix := setupTestHandler(t)
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: "/images/photo.jpg",
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
Quality: tt.quality,
FitMode: tt.fitMode,
signedURL, err := fix.service.GenerateSignedURL("", &imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: "/images/photo.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: tt.quality,
FitMode: tt.fitMode,
},
}, time.Hour)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
@@ -196,12 +203,16 @@ func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
fix := setupTestHandler(t)
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: photoPath,
SourceQuery: tt.query,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
signedURL, err := fix.service.GenerateSignedURL("", &imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
Query: tt.query,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, time.Hour)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
@@ -224,16 +235,19 @@ func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: photoPath,
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: expires,
sig := signer.Sign(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
Query: "url=https://origin.example.com/my%20cat.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: expires,
})
tests := []struct {
@@ -288,16 +302,19 @@ func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: tt.sourcePath,
SourceQuery: tt.sourceQuery,
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: expires,
sig := signer.Sign(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: tt.sourcePath,
Query: tt.sourceQuery,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: expires,
})
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
@@ -314,12 +331,16 @@ func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
fix := setupTestHandler(t)
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: tt.sourcePath,
SourceQuery: tt.sourceQuery,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
signedURL, err := fix.service.GenerateSignedURL("", &imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: tt.sourcePath,
Query: tt.sourceQuery,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, time.Hour)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
+14 -15
View File
@@ -14,6 +14,7 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic"
)
@@ -37,10 +38,11 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
// Log the request
s.log.Debug("encrypted image request",
"host", req.SourceHost,
"path", req.SourcePath,
"dimensions", fmt.Sprintf("%dx%d", req.Size.Width, req.Size.Height),
"format", req.Format,
"host", req.Source.Host,
"path", req.Source.Path,
"dimensions", fmt.Sprintf("%dx%d",
req.Transform.Size.Width, req.Transform.Size.Height),
"format", req.Transform.Format,
)
// Fetch and process the image (no signature validation
@@ -88,10 +90,10 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
duration := time.Since(start)
s.log.Info("image served",
"request_id", middleware.GetReqID(ctx),
"cache_key", imgcache.CacheKey(req),
"host", req.SourceHost,
"path", req.SourcePath,
"format", req.Format,
"cache_key", imagerequest.CacheKey(req),
"host", req.Source.Host,
"path", req.Source.Path,
"format", req.Transform.Format,
"cache_status", resp.CacheStatus,
"served_bytes", written,
"duration_ms", duration.Milliseconds(),
@@ -100,12 +102,12 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
}
// parseImageEncRequest decrypts the token of an encrypted image URL into an
// ImageRequest and checks it. On a token that is missing, does not decrypt,
// image request and checks it. On a token that is missing, does not decrypt,
// has expired or asks for something not valid, it writes an error response
// and returns false.
func (s *Handlers) parseImageEncRequest(
w http.ResponseWriter, r *http.Request,
) (*imgcache.ImageRequest, bool) {
) (*imagerequest.Request, bool) {
// Extract token from URL
token := chi.URLParam(r, "token")
if token == "" {
@@ -115,7 +117,7 @@ func (s *Handlers) parseImageEncRequest(
}
// Decrypt and validate the payload
payload, err := s.encGen.Parse(token)
req, err := s.encGen.Parse(token)
if err != nil {
if errors.Is(err, encurl.ErrExpired) {
s.log.Debug("encrypted URL expired", "error", err)
@@ -130,14 +132,11 @@ func (s *Handlers) parseImageEncRequest(
return nil, false
}
// Convert payload to ImageRequest
req := payload.ToImageRequest()
// Apply the same dimension and fit-mode bounds as the plain image
// route: a sealed payload is trusted for its origin, not for staying
// within limits, so an over-limit size or unknown fit mode is a 400
// here rather than an out-of-memory or a 500 from the processor.
err = imgcache.ValidateImageRequest(req)
err = imagerequest.ValidateImageRequest(req)
if err != nil {
s.log.Debug("encrypted URL failed validation", "error", err)
s.respondError(w, "invalid encrypted URL: "+err.Error(),
+18 -12
View File
@@ -10,7 +10,7 @@ import (
"github.com/go-chi/chi/v5"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// newEncTestServer builds a router serving the encrypted-URL route with a
@@ -54,11 +54,14 @@ func TestHandleImageEnc_OverLimitDimension_Returns400(t *testing.T) {
encGen, srv := newEncTestServer(t)
token, err := encGen.Generate(&encurl.Payload{
SourceHost: "cdn.example.com",
SourcePath: "/photo.jpg",
Width: 100000,
Height: 100000,
token, err := encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{
Host: "cdn.example.com",
Path: "/photo.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 100000, Height: 100000},
},
})
if err != nil {
t.Fatalf("Generate() error = %v", err)
@@ -79,12 +82,15 @@ func TestHandleImageEnc_InvalidFitMode_Returns400(t *testing.T) {
encGen, srv := newEncTestServer(t)
token, err := encGen.Generate(&encurl.Payload{
SourceHost: "cdn.example.com",
SourcePath: "/photo.jpg",
Width: 800,
Height: 600,
FitMode: imgcache.FitMode("bogus"),
token, err := encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{
Host: "cdn.example.com",
Path: "/photo.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
FitMode: imagerequest.Fit("bogus"),
},
})
if err != nil {
t.Fatalf("Generate() error = %v", err)
@@ -9,6 +9,7 @@ import (
"time"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// requireServedPhoto requires that rec answers 200 with the JPEG at photoPath
@@ -59,11 +60,14 @@ func TestHandleImageEnc_RejectedToken(t *testing.T) {
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
photo := encurl.Payload{
SourceHost: signedHost,
SourcePath: photoPath,
Width: 50,
Height: 50,
photo := imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
},
}
valid, err := h.encGen.Generate(&photo)
@@ -72,7 +76,7 @@ func TestHandleImageEnc_RejectedToken(t *testing.T) {
}
expiredPhoto := photo
expiredPhoto.ExpiresAt = time.Now().Add(-time.Minute).Unix()
expiredPhoto.Expires = time.Now().Add(-time.Minute)
expired, err := h.encGen.Generate(&expiredPhoto)
if err != nil {
@@ -14,7 +14,7 @@ import (
"github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -58,15 +58,18 @@ func TestImageWithoutFormat_SignedAsJXL(t *testing.T) {
route := newImageRoute(t, newPhotoFetcher(t, signedHost))
expires := time.Now().Add(time.Hour)
sig := signature.New(testSigningKey).Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: photoPath,
Width: 50,
Height: 50,
Format: string(imgcache.FormatJXL),
Quality: encurl.DefaultQuality,
FitMode: string(imgcache.FitCover),
Expires: expires,
sig := signature.New(testSigningKey).Sign(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJXL,
Quality: encurl.DefaultQuality,
FitMode: imagerequest.FitCover,
},
Expires: expires,
})
query := fmt.Sprintf("?sig=%s&exp=%d", sig, expires.Unix())
@@ -83,11 +86,14 @@ func TestImageEncWithoutFormat_ServesJPEGXL(t *testing.T) {
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
token, err := h.encGen.Generate(&encurl.Payload{
SourceHost: signedHost,
SourcePath: photoPath,
Width: 50,
Height: 50,
token, err := h.encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
},
})
if err != nil {
t.Fatalf("Generate() error = %v", err)
+6 -6
View File
@@ -7,7 +7,7 @@ import (
"github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// jxlType is the content type of JPEG XL.
@@ -22,14 +22,14 @@ func TestFormatForAccept_JPEGXL(t *testing.T) {
tests := []struct {
name string
accept string
want imgcache.ImageFormat
want imagerequest.Format
}{
{"JPEG XL-capable browser",
"image/jxl,image/avif,image/webp,image/*,*/*;q=0.8", imgcache.FormatJXL},
{"JPEG XL only", jxlType, imgcache.FormatJXL},
"image/jxl,image/avif,image/webp,image/*,*/*;q=0.8", imagerequest.FormatJXL},
{"JPEG XL only", jxlType, imagerequest.FormatJXL},
{"JPEG XL with a lower q than AVIF", "image/avif,image/jxl;q=0.5",
imgcache.FormatJXL},
{"q=0 on JPEG XL", "image/jxl;q=0,image/avif,*/*", imgcache.FormatAVIF},
imagerequest.FormatJXL},
{"q=0 on JPEG XL", "image/jxl;q=0,image/avif,*/*", imagerequest.FormatAVIF},
}
for _, tt := range tests {
+19 -13
View File
@@ -13,8 +13,7 @@ import (
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// signedPhotoURL returns a signed /v1/image/ URL, valid for a minute, for the
@@ -22,11 +21,15 @@ import (
func signedPhotoURL(t *testing.T, h *Handlers) string {
t.Helper()
signedURL, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: photoPath,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
signedURL, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, time.Minute)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
@@ -40,12 +43,15 @@ func signedPhotoURL(t *testing.T, h *Handlers) string {
func encPhotoURL(t *testing.T, h *Handlers) string {
t.Helper()
token, err := h.encGen.Generate(&encurl.Payload{
SourceHost: signedHost,
SourcePath: photoPath,
Width: 50,
Height: 50,
Format: imgcache.FormatJPEG,
token, err := h.encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
Path: photoPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
})
if err != nil {
t.Fatalf("Generate() error = %v", err)
@@ -11,6 +11,7 @@ import (
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
@@ -21,7 +22,9 @@ func TestServerBusyAnswers503(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
req := &imgcache.ImageRequest{SourceHost: "img.example.com", SourcePath: "/a.jpg"}
req := &imagerequest.Request{
Source: imagerequest.Source{Host: "img.example.com", Path: "/a.jpg"},
}
for _, err := range []error{
fmt.Errorf("upstream fetch failed: %w", httpfetcher.ErrTooManyConnections),
@@ -64,7 +67,7 @@ func TestProcessingTimedOutAnswers503(t *testing.T) {
imgcache.ErrProcessingTimedOut, context.DeadlineExceeded)
plain := httptest.NewRecorder()
h.respondImageError(plain, &imgcache.ImageRequest{}, err)
h.respondImageError(plain, &imagerequest.Request{}, err)
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
@@ -10,7 +10,7 @@ import (
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestUnsupportedSourceFormatAnswers415 checks that both image routes answer
@@ -21,7 +21,9 @@ func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
req := &imgcache.ImageRequest{SourceHost: "img.example.com", SourcePath: "/a.svg"}
req := &imagerequest.Request{
Source: imagerequest.Source{Host: "img.example.com", Path: "/a.svg"},
}
for _, err := range []error{
fmt.Errorf("upstream fetch failed: %w: image/svg+xml",
@@ -6,6 +6,8 @@ import (
"errors"
"os"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_RefusesAVIFAboveMaxPixels verifies that an AVIF output
@@ -37,22 +39,30 @@ func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
tests := []struct {
name string
input []byte
format Format
format imagerequest.Format
width, height int
fit FitMode
fit imagerequest.Fit
refused bool
}{
{"AVIF at the limit", cutOff, FormatAVIF, 4096, 4096, FitCover, false},
{"AVIF above the limit", cutOff, FormatAVIF, 4096, 4097, FitCover, true},
{"WebP above the AVIF limit", cutOff, FormatWebP, 4096, 4097, FitCover, false},
{"AVIF at the limit", cutOff, imagerequest.FormatAVIF,
4096, 4096, imagerequest.FitCover, false},
{"AVIF above the limit", cutOff, imagerequest.FormatAVIF,
4096, 4097, imagerequest.FitCover, true},
{"WebP above the AVIF limit", cutOff, imagerequest.FormatWebP,
4096, 4097, imagerequest.FitCover, false},
// FitInside does not enlarge, so the output stays 64x32
{"AVIF inside a larger size", cutOff, FormatAVIF, 8192, 8192, FitInside, false},
{"AVIF inside a larger size", cutOff, imagerequest.FormatAVIF,
8192, 8192, imagerequest.FitInside, false},
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
{"AVIF outside a smaller size", cutOff, FormatAVIF, 8192, 1, FitOutside, true},
{"AVIF outside a smaller size", cutOff, imagerequest.FormatAVIF,
8192, 1, imagerequest.FitOutside, true},
// With no size, the output keeps the source's 4097x4096
{"AVIF of the source's size", largeAVIF, FormatAVIF, 0, 0, FitCover, true},
{"orig of an AVIF source", largeAVIF, FormatOriginal, 0, 0, FitCover, true},
{"JPEG of the source's size", largeAVIF, FormatJPEG, 0, 0, FitCover, false},
{"AVIF of the source's size", largeAVIF, imagerequest.FormatAVIF,
0, 0, imagerequest.FitCover, true},
{"orig of an AVIF source", largeAVIF, imagerequest.FormatOriginal,
0, 0, imagerequest.FitCover, true},
{"JPEG of the source's size", largeAVIF, imagerequest.FormatJPEG,
0, 0, imagerequest.FitCover, false},
}
for _, tt := range tests {
@@ -60,8 +70,8 @@ func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
t.Parallel()
_, err := New(Params{}).Process(t.Context(), bytes.NewReader(tt.input),
&Request{
Size: Size{Width: tt.width, Height: tt.height},
&imagerequest.Transform{
Size: imagerequest.Size{Width: tt.width, Height: tt.height},
Format: tt.format,
FitMode: tt.fit,
})
@@ -4,6 +4,8 @@ import (
"bytes"
"errors"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_RefusesComputedSideAboveMaximum verifies that a size
@@ -17,13 +19,17 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
tests := []struct {
name string
inputWidth, inputHeight int
size Size
size imagerequest.Size
wantErr error
}{
{"width at the maximum", 4096, 1, Size{Height: 2}, nil},
{"width above the maximum", 4096, 1, Size{Height: 3}, ErrOutputTooLarge},
{"height at the maximum", 1, 4096, Size{Width: 2}, nil},
{"height above the maximum", 1, 4096, Size{Width: 3}, ErrOutputTooLarge},
{"width at the maximum", 4096, 1,
imagerequest.Size{Height: 2}, nil},
{"width above the maximum", 4096, 1,
imagerequest.Size{Height: 3}, ErrOutputTooLarge},
{"height at the maximum", 1, 4096,
imagerequest.Size{Width: 2}, nil},
{"height above the maximum", 1, 4096,
imagerequest.Size{Width: 3}, ErrOutputTooLarge},
}
for _, tt := range tests {
@@ -34,7 +40,11 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitCover})
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
FitMode: imagerequest.FitCover,
})
if result != nil {
t.Logf("Process() made %dx%d", result.Width, result.Height)
@@ -64,18 +74,21 @@ func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
tests := []struct {
name string
inputWidth, inputHeight int
size Size
size imagerequest.Size
wantWidth, wantHeight int
}{
{"8192x1 at 100x0", 8192, 1, Size{Width: 100}, 100, 1},
{"8192x1 at 1300x0", 8192, 1, Size{Width: 1300}, 1300, 1},
{"8192x2 at 100x0", 8192, 2, Size{Width: 100}, 100, 1},
{"1x8192 at 0x100", 1, 8192, Size{Height: 100}, 1, 100},
{"1x8192 at 0x1300", 1, 8192, Size{Height: 1300}, 1, 1300},
{"2x8192 at 0x100", 2, 8192, Size{Height: 100}, 1, 100},
{"8192x1 at 100x0", 8192, 1, imagerequest.Size{Width: 100}, 100, 1},
{"8192x1 at 1300x0", 8192, 1, imagerequest.Size{Width: 1300}, 1300, 1},
{"8192x2 at 100x0", 8192, 2, imagerequest.Size{Width: 100}, 100, 1},
{"1x8192 at 0x100", 1, 8192, imagerequest.Size{Height: 100}, 1, 100},
{"1x8192 at 0x1300", 1, 8192, imagerequest.Size{Height: 1300}, 1, 1300},
{"2x8192 at 0x100", 2, 8192, imagerequest.Size{Height: 100}, 1, 100},
}
fits := []FitMode{FitCover, FitContain, FitFill, FitInside, FitOutside}
fits := []imagerequest.Fit{
imagerequest.FitCover, imagerequest.FitContain, imagerequest.FitFill,
imagerequest.FitInside, imagerequest.FitOutside,
}
for _, tt := range tests {
for _, fit := range fits {
@@ -93,7 +106,11 @@ func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatJPEG, FitMode: fit})
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
FitMode: fit,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -103,14 +120,14 @@ func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
t.Logf("Process() made %dx%d", result.Width, result.Height)
ok := result.Width == tt.wantWidth && result.Height == tt.wantHeight
if fit == FitOutside {
if fit == imagerequest.FitOutside {
ok = result.Width >= tt.wantWidth && result.Height >= tt.wantHeight
}
if !ok {
t.Errorf("Process() made %dx%d, want %dx%d (at least, for %s)",
result.Width, result.Height, tt.wantWidth, tt.wantHeight,
FitOutside)
imagerequest.FitOutside)
}
})
}
@@ -86,7 +86,7 @@ func checkProcessStopsAtDeadline(
proc := New(Params{MaxConcurrentProcessing: 1})
result, err := proc.Process(ctx, input, smallJPEGRequest())
result, err := proc.Process(ctx, input, smallJPEGTransform())
if result != nil {
t.Logf("Process() made a %dx%d image after its deadline passed",
result.Width, result.Height)
@@ -4,6 +4,8 @@ import (
"bytes"
"errors"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_EmptyFormatRefused verifies that a request with no format
@@ -13,7 +15,8 @@ func TestImageProcessor_EmptyFormatRefused(t *testing.T) {
t.Parallel()
_, err := New(Params{}).Process(
t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)), &Request{},
t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)),
&imagerequest.Transform{},
)
if !errors.Is(err, ErrUnsupportedOutputFormat) {
t.Errorf("Process() error = %v, want %v", err, ErrUnsupportedOutputFormat)
@@ -8,14 +8,17 @@ import (
"testing"
"github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// processedSize runs input through Process and returns the output's size in
// bytes.
func processedSize(t *testing.T, input []byte, req *Request) int64 {
func processedSize(
t *testing.T, input []byte, transform *imagerequest.Transform,
) int64 {
t.Helper()
result, err := New(Params{}).Process(t.Context(), bytes.NewReader(input), req)
result, err := New(Params{}).Process(t.Context(), bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -68,7 +71,8 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
}
}
size := processedSize(t, encodePNG(t, flat), &Request{Format: FormatPNG})
size := processedSize(t, encodePNG(t, flat),
&imagerequest.Transform{Format: imagerequest.FormatPNG})
const rawBytes = width * height * 3
if size > rawBytes/10 {
@@ -85,7 +89,8 @@ func TestImageProcessor_WebPAtEffort2(t *testing.T) {
input := createTestJPEG(t, 200, 150)
size := processedSize(t, input, &Request{Format: FormatWebP, Quality: 85})
size := processedSize(t, input,
&imagerequest.Transform{Format: imagerequest.FormatWebP, Quality: 85})
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
StripMetadata: true,
@@ -126,7 +131,8 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
input := encodePNG(t, source)
size := processedSize(t, input, &Request{Format: FormatAVIF, Quality: 85})
size := processedSize(t, input,
&imagerequest.Transform{Format: imagerequest.FormatAVIF, Quality: 85})
want, _, err := decode(t, input).ExportAvif(&vips.AvifExportParams{
StripMetadata: true,
@@ -152,7 +158,8 @@ func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
input := createTestJPEG(t, 200, 150)
size := processedSize(t, input, &Request{Format: FormatJXL, Quality: 85})
size := processedSize(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: 85})
params := vips.NewJxlExportParams()
params.Quality = 85
@@ -7,6 +7,8 @@ import (
"image/png"
"math"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_FitCoverServesThinSource verifies that FitCover serves
@@ -20,12 +22,12 @@ func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
tests := []struct {
name string
inputWidth, inputHeight int
size Size
size imagerequest.Size
}{
{"8192x1 at 8192x8192", 8192, 1, Size{Width: 8192, Height: 8192}},
{"8192x1 at 1x8192", 8192, 1, Size{Width: 1, Height: 8192}},
{"1x8192 at 8192x8192", 1, 8192, Size{Width: 8192, Height: 8192}},
{"1x8192 at 8192x1", 1, 8192, Size{Width: 8192, Height: 1}},
{"8192x1 at 8192x8192", 8192, 1, imagerequest.Size{Width: 8192, Height: 8192}},
{"8192x1 at 1x8192", 8192, 1, imagerequest.Size{Width: 1, Height: 8192}},
{"1x8192 at 8192x8192", 1, 8192, imagerequest.Size{Width: 8192, Height: 8192}},
{"1x8192 at 8192x1", 1, 8192, imagerequest.Size{Width: 8192, Height: 1}},
}
for _, tt := range tests {
@@ -36,7 +38,11 @@ func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitCover})
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
FitMode: imagerequest.FitCover,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -102,10 +108,10 @@ func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{
Size: Size{Width: 50, Height: 50},
Format: FormatPNG,
FitMode: FitCover,
&imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatPNG,
FitMode: imagerequest.FitCover,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
@@ -208,14 +214,14 @@ func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
tests := []struct {
name string
side, squareSide int
size Size
size imagerequest.Size
}{
{"16x16 at 100x60", 16, 8, Size{Width: 100, Height: 60}},
{"16x16 at 60x100", 16, 8, Size{Width: 60, Height: 100}},
{"10x10 at 100x35", 10, 2, Size{Width: 100, Height: 35}},
{"10x10 at 35x100", 10, 2, Size{Width: 35, Height: 100}},
{"64x64 at 300x100", 64, 16, Size{Width: 300, Height: 100}},
{"64x64 at 100x300", 64, 16, Size{Width: 100, Height: 300}},
{"16x16 at 100x60", 16, 8, imagerequest.Size{Width: 100, Height: 60}},
{"16x16 at 60x100", 16, 8, imagerequest.Size{Width: 60, Height: 100}},
{"10x10 at 100x35", 10, 2, imagerequest.Size{Width: 100, Height: 35}},
{"10x10 at 35x100", 10, 2, imagerequest.Size{Width: 35, Height: 100}},
{"64x64 at 300x100", 64, 16, imagerequest.Size{Width: 300, Height: 100}},
{"64x64 at 100x300", 64, 16, imagerequest.Size{Width: 100, Height: 300}},
}
for _, tt := range tests {
@@ -226,7 +232,11 @@ func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatPNG, FitMode: FitCover})
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatPNG,
FitMode: imagerequest.FitCover,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -4,6 +4,8 @@ import (
"bytes"
"errors"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_RefusesFitOutsideAboveMaximum verifies that FitOutside,
@@ -17,15 +19,20 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
tests := []struct {
name string
inputWidth, inputHeight int
size Size
size imagerequest.Size
// wantWidth and wantHeight are the output's, 0 for a refusal
wantWidth, wantHeight int
}{
{"width above the maximum", 3000, 600, Size{Width: 1, Height: 8192}, 0, 0},
{"height above the maximum", 600, 3000, Size{Width: 8192, Height: 1}, 0, 0},
{"width at the maximum", 1024, 256, Size{Width: 1, Height: 2048}, 8192, 2048},
{"height at the maximum", 256, 1024, Size{Width: 2048, Height: 1}, 2048, 8192},
{"within the maximum", 3000, 600, Size{Width: 1, Height: 1200}, 6000, 1200},
{"width above the maximum", 3000, 600,
imagerequest.Size{Width: 1, Height: 8192}, 0, 0},
{"height above the maximum", 600, 3000,
imagerequest.Size{Width: 8192, Height: 1}, 0, 0},
{"width at the maximum", 1024, 256,
imagerequest.Size{Width: 1, Height: 2048}, 8192, 2048},
{"height at the maximum", 256, 1024,
imagerequest.Size{Width: 2048, Height: 1}, 2048, 8192},
{"within the maximum", 3000, 600,
imagerequest.Size{Width: 1, Height: 1200}, 6000, 1200},
}
for _, tt := range tests {
@@ -36,7 +43,11 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitOutside})
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
FitMode: imagerequest.FitOutside,
})
if result != nil {
t.Logf("Process() made %dx%d", result.Width, result.Height)
}
@@ -71,11 +82,11 @@ func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
tests := []struct {
name string
inputWidth, inputHeight int
size Size
size imagerequest.Size
wantWidth, wantHeight int
}{
{"49x8192 at 0x100", 49, 8192, Size{Height: 100}, 1, 167},
{"49x49 at 1x1", 49, 49, Size{Width: 1, Height: 1}, 1, 1},
{"49x8192 at 0x100", 49, 8192, imagerequest.Size{Height: 100}, 1, 167},
{"49x49 at 1x1", 49, 49, imagerequest.Size{Width: 1, Height: 1}, 1, 1},
}
for _, tt := range tests {
@@ -86,7 +97,11 @@ func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitOutside})
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
FitMode: imagerequest.FitOutside,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
+68 -102
View File
@@ -12,6 +12,7 @@ import (
"github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// vipsOnce ensures vips is initialized exactly once.
@@ -54,49 +55,6 @@ func CheckJPEGXLSupport() error {
return nil
}
// Format represents supported output image formats.
type Format string
// Supported image output formats.
const (
FormatOriginal Format = "orig"
FormatJPEG Format = "jpeg"
FormatPNG Format = "png"
FormatWebP Format = "webp"
FormatAVIF Format = "avif"
FormatJXL Format = "jxl"
FormatGIF Format = "gif"
)
// FitMode represents how to fit an image into requested dimensions.
type FitMode string
// Supported image fit modes.
const (
FitCover FitMode = "cover"
FitContain FitMode = "contain"
FitFill FitMode = "fill"
FitInside FitMode = "inside"
FitOutside FitMode = "outside"
)
// ErrInvalidFitMode is returned when an invalid fit mode is provided.
var ErrInvalidFitMode = errors.New("invalid fit mode")
// Size represents requested image dimensions.
type Size struct {
Width int
Height int
}
// Request holds the parameters for image processing.
type Request struct {
Size Size
Format Format
Quality int
FitMode FitMode
}
// Result contains the output of image processing.
type Result struct {
// Content is the processed image data.
@@ -133,8 +91,8 @@ var ErrInputTooLarge = errors.New("input image dimensions exceed maximum")
var ErrInputDataTooLarge = errors.New("input data exceeds maximum allowed size")
// MaxOutputDimension is the maximum width or height of an output image, and
// so of a requested size (imgcache.MaxDimension).
const MaxOutputDimension = 8192
// so of a requested size (imagerequest.MaxDimension).
const MaxOutputDimension = imagerequest.MaxDimension
// ErrOutputTooLarge is returned, before resizing, when the output would have a
// side above MaxOutputDimension: the side of a size with one side 0, computed
@@ -217,7 +175,7 @@ func New(params Params) *ImageProcessor {
}
}
// Process transforms an image according to the request. When
// Process transforms an image according to transform. When
// MaxConcurrentProcessing images are already being processed, it waits up
// to ProcessingWaitTimeout for one to finish, then fails with
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
@@ -228,7 +186,7 @@ func New(params Params) *ImageProcessor {
func (p *ImageProcessor) Process(
ctx context.Context,
input io.Reader,
req *Request,
transform *imagerequest.Transform,
) (*Result, error) {
release, err := p.acquireSlot(ctx)
if err != nil {
@@ -277,35 +235,36 @@ func (p *ImageProcessor) Process(
}
// Determine target dimensions; too large ones are refused before resizing
targetWidth, targetHeight, err := targetDimensions(req.Size, origWidth, origHeight)
targetWidth, targetHeight, err := targetDimensions(transform.Size,
origWidth, origHeight)
if err != nil {
return nil, err
}
// orig is the source's own format; encode refuses an empty format
outputFormat := req.Format
if outputFormat == FormatOriginal {
outputFormat := transform.Format
if outputFormat == imagerequest.FormatOriginal {
outputFormat = sourceFormat
}
// Determine output dimensions; too large ones are refused before
// resizing, which with FitCover already decodes the input
outputWidth, outputHeight, err := outputDimensions(origWidth, origHeight,
targetWidth, targetHeight, req.FitMode, outputFormat)
targetWidth, targetHeight, transform.FitMode, outputFormat)
if err != nil {
return nil, err
}
// Resize if needed
if targetWidth != origWidth || targetHeight != origHeight {
err := p.resize(img, outputWidth, outputHeight, req.FitMode)
err := p.resize(img, outputWidth, outputHeight, transform.FitMode)
if err != nil {
return nil, fmt.Errorf("failed to resize: %w", err)
}
}
// Encode to target format
output, err := p.encode(ctx, img, outputFormat, req.Quality)
output, err := p.encode(ctx, img, outputFormat, transform.Quality)
if err != nil {
return nil, fmt.Errorf("failed to encode: %w", err)
}
@@ -328,7 +287,9 @@ func (p *ImageProcessor) Process(
// rounded down but kept at 1 or more, as for a thin image it would round
// down to 0, a side of no pixels. It can come out above MaxOutputDimension,
// which is ErrOutputTooLarge.
func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
func targetDimensions(
size imagerequest.Size, origWidth, origHeight int,
) (int, int, error) {
width, height := size.Width, size.Height
switch {
@@ -374,33 +335,33 @@ func (p *ImageProcessor) SupportedInputFormats() []string {
}
// SupportedOutputFormats returns formats this processor can write.
func (p *ImageProcessor) SupportedOutputFormats() []Format {
return []Format{
FormatJPEG,
FormatPNG,
FormatGIF,
FormatWebP,
FormatAVIF,
FormatJXL,
func (p *ImageProcessor) SupportedOutputFormats() []imagerequest.Format {
return []imagerequest.Format{
imagerequest.FormatJPEG,
imagerequest.FormatPNG,
imagerequest.FormatGIF,
imagerequest.FormatWebP,
imagerequest.FormatAVIF,
imagerequest.FormatJXL,
}
}
// FormatToMIME converts a Format to its MIME type string.
func FormatToMIME(format Format) string {
func FormatToMIME(format imagerequest.Format) string {
switch format {
case FormatJPEG:
case imagerequest.FormatJPEG:
return mimeJPEG
case FormatPNG:
case imagerequest.FormatPNG:
return mimePNG
case FormatWebP:
case imagerequest.FormatWebP:
return mimeWebP
case FormatGIF:
case imagerequest.FormatGIF:
return mimeGIF
case FormatAVIF:
case imagerequest.FormatAVIF:
return mimeAVIF
case FormatJXL:
case imagerequest.FormatJXL:
return mimeJXL
case FormatOriginal:
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
return "application/octet-stream"
default:
return "application/octet-stream"
@@ -487,9 +448,9 @@ func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
case vips.ImageTypeWEBP:
return "webp"
case vips.ImageTypeAVIF, vips.ImageTypeHEIF:
return string(FormatAVIF)
return string(imagerequest.FormatAVIF)
case vips.ImageTypeJXL:
return string(FormatJXL)
return string(imagerequest.FormatJXL)
case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF,
vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP,
vips.ImageTypeJP2K:
@@ -506,17 +467,18 @@ func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
// outputDimensions returns ErrOutputTooLarge. An AVIF output of more than
// MaxAVIFOutputPixels is ErrAVIFOutputTooLarge.
func outputDimensions(
imgWidth, imgHeight, width, height int, fit FitMode, format Format,
imgWidth, imgHeight, width, height int,
fit imagerequest.Fit, format imagerequest.Format,
) (int, int, error) {
scaleW := float64(width) / float64(imgWidth)
scaleH := float64(height) / float64(imgHeight)
outputWidth, outputHeight := width, height
switch fit {
case FitCover, FitFill, "":
case imagerequest.FitCover, imagerequest.FitFill, "":
// Cover crops, and fill stretches, to exactly the target dimensions
case FitContain:
case imagerequest.FitContain:
// Fit within the target dimensions, maintaining aspect ratio. Scaled
// by the smaller scale, the short side of a thin image can round
// down to 0, which libvips refuses as a width, so each side is at
@@ -525,13 +487,13 @@ func outputDimensions(
outputWidth = max(1, int(float64(imgWidth)*scale))
outputHeight = max(1, int(float64(imgHeight)*scale))
case FitInside:
case imagerequest.FitInside:
// Same as contain, but only shrink
scale := min(scaleW, scaleH, 1)
outputWidth = max(1, int(float64(imgWidth)*scale))
outputHeight = max(1, int(float64(imgHeight)*scale))
case FitOutside:
case imagerequest.FitOutside:
// The smallest dimension fits, the other may exceed the target. A
// side scaled to 1 can come out just under 1 in floating point, as
// 49 times 1/49 does, and round down to 0, which libvips refuses as
@@ -541,7 +503,7 @@ func outputDimensions(
outputHeight = max(1, int(float64(imgHeight)*scale))
default:
return 0, 0, fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
return 0, 0, fmt.Errorf("%w: %s", imagerequest.ErrInvalidFitMode, fit)
}
// The side FitOutside scales past the target is held to the maximum too
@@ -550,7 +512,8 @@ func outputDimensions(
outputWidth, outputHeight, MaxOutputDimension)
}
if format == FormatAVIF && outputWidth*outputHeight > MaxAVIFOutputPixels {
if format == imagerequest.FormatAVIF &&
outputWidth*outputHeight > MaxAVIFOutputPixels {
return 0, 0, fmt.Errorf("%w: %dx%d is above %d pixels",
ErrAVIFOutputTooLarge, outputWidth, outputHeight, MaxAVIFOutputPixels)
}
@@ -561,18 +524,18 @@ func outputDimensions(
// resize resizes the image to width x height, the dimensions
// outputDimensions gives for the fit mode.
func (p *ImageProcessor) resize(
img *vips.ImageRef, width, height int, fit FitMode,
img *vips.ImageRef, width, height int, fit imagerequest.Fit,
) error {
switch fit {
case FitCover, "":
case imagerequest.FitCover, "":
// Crop the centre and resize to fill exact dimensions (default)
return cropCentreAndResize(img, width, height)
case FitFill:
case imagerequest.FitFill:
// Resize to exact dimensions (may distort)
return img.ThumbnailWithSize(width, height, vips.InterestingNone, vips.SizeForce)
case FitInside:
case imagerequest.FitInside:
// outputDimensions keeps the dimensions of an image that already fits
if img.Width() == width && img.Height() == height {
return nil
@@ -580,11 +543,11 @@ func (p *ImageProcessor) resize(
return img.Thumbnail(width, height, vips.InterestingNone)
case FitContain, FitOutside:
case imagerequest.FitContain, imagerequest.FitOutside:
return img.Thumbnail(width, height, vips.InterestingNone)
default:
return fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
return fmt.Errorf("%w: %s", imagerequest.ErrInvalidFitMode, fit)
}
}
@@ -626,7 +589,8 @@ const defaultQuality = 85
// to stop libvips once an encode has begun, so when ctx has ended encode
// returns ctx's error without beginning one.
func (p *ImageProcessor) encode(
ctx context.Context, img *vips.ImageRef, format Format, quality int,
ctx context.Context, img *vips.ImageRef, format imagerequest.Format,
quality int,
) ([]byte, error) {
err := ctx.Err()
if err != nil {
@@ -649,25 +613,25 @@ func (p *ImageProcessor) encode(
}
switch format {
case FormatJPEG:
case imagerequest.FormatJPEG:
return exportJPEG(img, quality)
case FormatPNG:
case imagerequest.FormatPNG:
return exportPNG(img)
case FormatGIF:
case imagerequest.FormatGIF:
return exportGIF(img)
case FormatWebP:
case imagerequest.FormatWebP:
return exportWebP(img, quality)
case FormatAVIF:
case imagerequest.FormatAVIF:
return exportAVIF(img, quality)
case FormatJXL:
case imagerequest.FormatJXL:
return exportJXL(img, quality)
case FormatOriginal:
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
default:
@@ -849,20 +813,22 @@ func jxlDistance(quality int) float64 {
// formatFromString converts a format string from detectFormat to Format. A
// format detectFormat could not name is ErrUnsupportedInputFormat.
func (p *ImageProcessor) formatFromString(format string) (Format, error) {
func (p *ImageProcessor) formatFromString(
format string,
) (imagerequest.Format, error) {
switch format {
case "jpeg":
return FormatJPEG, nil
return imagerequest.FormatJPEG, nil
case "png":
return FormatPNG, nil
return imagerequest.FormatPNG, nil
case "gif":
return FormatGIF, nil
return imagerequest.FormatGIF, nil
case "webp":
return FormatWebP, nil
case string(FormatAVIF):
return FormatAVIF, nil
case string(FormatJXL):
return FormatJXL, nil
return imagerequest.FormatWebP, nil
case string(imagerequest.FormatAVIF):
return imagerequest.FormatAVIF, nil
case string(imagerequest.FormatJXL):
return imagerequest.FormatJXL, nil
default:
return "", fmt.Errorf("%w: %s", ErrUnsupportedInputFormat, format)
}
@@ -15,6 +15,7 @@ import (
"testing"
"github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imagerequest"
)
func TestMain(m *testing.M) {
@@ -87,7 +88,7 @@ func isAVIF(data []byte) bool {
brand := string(data[8:12])
return brand == string(FormatAVIF) || brand == "avis"
return brand == string(imagerequest.FormatAVIF) || brand == "avis"
}
// detectMIME is a minimal magic-byte detector for test assertions.
@@ -123,14 +124,14 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
input := createTestJPEG(t, 800, 600)
req := &Request{
Size: Size{Width: 400, Height: 300},
Format: FormatJPEG,
transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 400, Height: 300},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: FitCover,
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -169,13 +170,13 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
input := createTestJPEG(t, 200, 150)
req := &Request{
Size: Size{Width: 200, Height: 150},
Format: FormatPNG,
FitMode: FitCover,
transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 200, Height: 150},
Format: imagerequest.FormatPNG,
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -196,7 +197,7 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
// processAndCheckSize processes a test JPEG of the given input dimensions
// with the requested size and asserts the resulting dimensions.
func processAndCheckSize(
t *testing.T, inputW, inputH int, size Size, wantW, wantH int,
t *testing.T, inputW, inputH int, size imagerequest.Size, wantW, wantH int,
) {
t.Helper()
@@ -205,14 +206,14 @@ func processAndCheckSize(
input := createTestJPEG(t, inputW, inputH)
req := &Request{
transform := &imagerequest.Transform{
Size: size,
Format: FormatJPEG,
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: FitCover,
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -232,7 +233,7 @@ func TestImageProcessor_OriginalSize(t *testing.T) {
t.Parallel()
// Width and height 0: keep original size
processAndCheckSize(t, 640, 480, Size{Width: 0, Height: 0}, 640, 480)
processAndCheckSize(t, 640, 480, imagerequest.Size{Width: 0, Height: 0}, 640, 480)
}
func TestImageProcessor_FitContain(t *testing.T) {
@@ -245,14 +246,14 @@ func TestImageProcessor_FitContain(t *testing.T) {
// Should result in 400x200 (maintaining aspect ratio)
input := createTestJPEG(t, 800, 400)
req := &Request{
Size: Size{Width: 400, Height: 400},
Format: FormatJPEG,
transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 400, Height: 400},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: FitContain,
FitMode: imagerequest.FitContain,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -270,7 +271,7 @@ func TestImageProcessor_ProportionalScale_WidthOnly(t *testing.T) {
// 800x600 image, request width=400 height=0
// Should scale proportionally to 400x300
processAndCheckSize(t, 800, 600, Size{Width: 400, Height: 0}, 400, 300)
processAndCheckSize(t, 800, 600, imagerequest.Size{Width: 400, Height: 0}, 400, 300)
}
func TestImageProcessor_ProportionalScale_HeightOnly(t *testing.T) {
@@ -278,7 +279,7 @@ func TestImageProcessor_ProportionalScale_HeightOnly(t *testing.T) {
// 800x600 image, request width=0 height=300
// Should scale proportionally to 400x300
processAndCheckSize(t, 800, 600, Size{Width: 0, Height: 300}, 400, 300)
processAndCheckSize(t, 800, 600, imagerequest.Size{Width: 0, Height: 300}, 400, 300)
}
func TestImageProcessor_ProcessPNG(t *testing.T) {
@@ -289,13 +290,13 @@ func TestImageProcessor_ProcessPNG(t *testing.T) {
input := createTestPNG(t, 400, 300)
req := &Request{
Size: Size{Width: 200, Height: 150},
Format: FormatPNG,
FitMode: FitCover,
transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 200, Height: 150},
Format: imagerequest.FormatPNG,
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -349,14 +350,14 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
ctx := context.Background()
input := createTestJPEG(t, tt.width, tt.height)
req := &Request{
Size: Size{Width: 100, Height: 100},
Format: FormatJPEG,
transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 100, Height: 100},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: FitCover,
FitMode: imagerequest.FitCover,
}
_, err := proc.Process(ctx, bytes.NewReader(input), req)
_, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err == nil {
t.Error("Process() should reject oversized input images")
}
@@ -377,14 +378,14 @@ func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
// Create an image at exactly MaxInputDimension - should be accepted
input := createTestJPEG(t, MaxInputDimension, 100)
req := &Request{
Size: Size{Width: 100, Height: 100},
Format: FormatJPEG,
transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 100, Height: 100},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: FitCover,
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf(
"Process() should accept images at MaxInputDimension, got error: %v",
@@ -397,7 +398,9 @@ func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
// encodeAndCheck processes a 200x150 test JPEG into a 100x75 output of the
// given format and asserts the output MIME type and dimensions.
func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) {
func encodeAndCheck(
t *testing.T, format imagerequest.Format, quality int, wantMIME string,
) {
t.Helper()
proc := New(Params{})
@@ -405,14 +408,14 @@ func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) {
input := createTestJPEG(t, 200, 150)
req := &Request{
Size: Size{Width: 100, Height: 75},
transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 100, Height: 75},
Format: format,
Quality: quality,
FitMode: FitCover,
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v, want nil", err)
}
@@ -443,7 +446,7 @@ func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) {
func TestImageProcessor_EncodeWebP(t *testing.T) {
t.Parallel()
encodeAndCheck(t, FormatWebP, 80, mimeWebP)
encodeAndCheck(t, imagerequest.FormatWebP, 80, mimeWebP)
}
func TestImageProcessor_DecodeAVIF(t *testing.T) {
@@ -459,14 +462,14 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
}
// Request resize and convert to JPEG
req := &Request{
Size: Size{Width: 2, Height: 2},
Format: FormatJPEG,
transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 2, Height: 2},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: FitCover,
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v, want nil (AVIF decoding should work)", err)
}
@@ -500,14 +503,14 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
t.Fatalf("test JPEG must exceed %d bytes, got %d", limit, len(input))
}
req := &Request{
Size: Size{Width: 100, Height: 75},
Format: FormatJPEG,
transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 100, Height: 75},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: FitCover,
FitMode: imagerequest.FitCover,
}
_, err := proc.Process(ctx, bytes.NewReader(input), req)
_, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err == nil {
t.Fatal("Process() should reject input exceeding maxInputBytes")
}
@@ -527,14 +530,14 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
proc := New(Params{MaxInputBytes: limit})
ctx := context.Background()
req := &Request{
Size: Size{Width: 10, Height: 10},
Format: FormatJPEG,
transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 10, Height: 10},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: FitCover,
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v, want nil", err)
}
@@ -561,16 +564,18 @@ func TestImageProcessor_DefaultMaxInputBytes(t *testing.T) {
func TestImageProcessor_EncodeAVIF(t *testing.T) {
t.Parallel()
encodeAndCheck(t, FormatAVIF, 85, mimeAVIF)
encodeAndCheck(t, imagerequest.FormatAVIF, 85, mimeAVIF)
}
// processAndDecode runs input through Process and decodes the output with
// vips, so a test can inspect the image a client would receive.
func processAndDecode(t *testing.T, input []byte, req *Request) *vips.ImageRef {
func processAndDecode(
t *testing.T, input []byte, transform *imagerequest.Transform,
) *vips.ImageRef {
t.Helper()
result, err := New(Params{}).Process(
context.Background(), bytes.NewReader(input), req,
context.Background(), bytes.NewReader(input), transform,
)
if err != nil {
t.Fatalf("Process() error = %v", err)
@@ -614,15 +619,17 @@ func TestImageProcessor_StripsEXIF(t *testing.T) {
t.Fatal("testdata/gps-exif.jpg has no GPS latitude")
}
formats := []Format{
FormatJPEG, FormatPNG, FormatWebP, FormatAVIF, FormatGIF, FormatOriginal,
formats := []imagerequest.Format{
imagerequest.FormatJPEG, imagerequest.FormatPNG, imagerequest.FormatWebP,
imagerequest.FormatAVIF, imagerequest.FormatGIF, imagerequest.FormatOriginal,
}
for _, format := range formats {
t.Run(string(format), func(t *testing.T) {
t.Parallel()
output := processAndDecode(t, input, &Request{Format: format})
output := processAndDecode(t, input,
&imagerequest.Transform{Format: format})
if output.HasExif() {
t.Errorf("output has EXIF: %v", output.GetExif())
@@ -644,21 +651,21 @@ func TestImageProcessor_AppliesEXIFOrientation(t *testing.T) {
tests := []struct {
name string
size Size
size imagerequest.Size
wantW int
wantH int
}{
{name: "original size", size: Size{}, wantW: 8, wantH: 16},
{name: "width only", size: Size{Width: 4}, wantW: 4, wantH: 8},
{name: "original size", size: imagerequest.Size{}, wantW: 8, wantH: 16},
{name: "width only", size: imagerequest.Size{Width: 4}, wantW: 4, wantH: 8},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
output := processAndDecode(t, input, &Request{
output := processAndDecode(t, input, &imagerequest.Transform{
Size: tt.size,
Format: FormatPNG,
Format: imagerequest.FormatPNG,
})
if output.Width() != tt.wantW || output.Height() != tt.wantH {
@@ -694,7 +701,8 @@ func TestImageProcessor_ConvertsWideGamutToSRGB(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err)
}
output := processAndDecode(t, input, &Request{Format: FormatPNG})
output := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatPNG})
if output.HasICCProfile() {
t.Error("output has an ICC profile")
@@ -8,6 +8,7 @@ import (
"testing"
"github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_EncodeJPEGXL converts a JPEG to a smaller JPEG XL and
@@ -15,10 +16,13 @@ import (
func TestImageProcessor_EncodeJPEGXL(t *testing.T) {
t.Parallel()
req := &Request{Size: Size{Width: 100, Height: 75}, Format: FormatJXL}
transform := &imagerequest.Transform{
Size: imagerequest.Size{Width: 100, Height: 75},
Format: imagerequest.FormatJXL,
}
result, err := New(Params{}).Process(
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)), req,
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)), transform,
)
if err != nil {
t.Fatalf("Process() error = %v", err)
@@ -61,7 +65,8 @@ func TestImageProcessor_JPEGXLQuality(t *testing.T) {
for _, quality := range []int{30, 90} {
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input), &Request{Format: FormatJXL, Quality: quality})
bytes.NewReader(input),
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: quality})
if err != nil {
t.Fatalf("Process() at quality %d error = %v", quality, err)
}
@@ -89,7 +94,8 @@ func TestImageProcessor_JPEGXLDropsSourceEXIF(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err)
}
exif := processAndDecode(t, input, &Request{Format: FormatJXL}).GetExif()
exif := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL}).GetExif()
for _, field := range []string{
"exif-ifd0-Make", "exif-ifd0-Model", "exif-ifd2-BodySerialNumber",
@@ -115,7 +121,8 @@ func TestImageProcessor_JPEGXLDropsSourceResolution(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err)
}
output := processAndDecode(t, input, &Request{Format: FormatJXL})
output := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL})
// libvips gives the resolution in pixels per millimetre.
xDPI := math.Round(output.ResX() * 25.4)
@@ -140,7 +147,8 @@ func TestImageProcessor_JPEGXLAppliesEXIFOrientation(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err)
}
output := processAndDecode(t, input, &Request{Format: FormatJXL})
output := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL})
if output.Width() != 8 || output.Height() != 16 {
t.Fatalf("output is %dx%d, want 8x16", output.Width(), output.Height())
@@ -176,7 +184,8 @@ func TestImageProcessor_JPEGXLConvertsWideGamutToSRGB(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err)
}
output := processAndDecode(t, input, &Request{Format: FormatJXL})
output := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL})
pixel, err := output.GetPoint(4, 4)
if err != nil {
@@ -203,7 +212,8 @@ func TestImageProcessor_JPEGXLFromCMYK(t *testing.T) {
t.Fatalf("failed to read test JPEG: %v", err)
}
output := processAndDecode(t, input, &Request{Format: FormatJXL})
output := processAndDecode(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL})
if output.Bands() != 3 {
t.Fatalf("output has %d bands, want 3", output.Bands())
@@ -11,6 +11,8 @@ import (
"testing"
"testing/iotest"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// errTestReadFailed is the error the unreadable test input returns.
@@ -76,13 +78,13 @@ func (r *gatedReader) Read(p []byte) (int, error) {
return n, err
}
// smallJPEGRequest asks for a 5x5 JPEG.
func smallJPEGRequest() *Request {
return &Request{
Size: Size{Width: 5, Height: 5},
Format: FormatJPEG,
// smallJPEGTransform asks for a 5x5 JPEG.
func smallJPEGTransform() *imagerequest.Transform {
return &imagerequest.Transform{
Size: imagerequest.Size{Width: 5, Height: 5},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: FitCover,
FitMode: imagerequest.FitCover,
}
}
@@ -92,7 +94,7 @@ func processInBackground(
proc *ImageProcessor, reader *gatedReader, results chan<- error,
) {
go func() {
result, err := proc.Process(context.Background(), reader, smallJPEGRequest())
result, err := proc.Process(context.Background(), reader, smallJPEGTransform())
if err == nil {
_ = result.Content.Close()
}
@@ -237,7 +239,7 @@ func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) {
start := time.Now()
_, err := proc.Process(context.Background(), bytes.NewReader(input),
smallJPEGRequest())
smallJPEGTransform())
if !errors.Is(err, ErrTooManyImages) {
t.Fatalf("Process() error = %v, want ErrTooManyImages", err)
}
@@ -263,38 +265,38 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
valid := createTestJPEG(t, 10, 10)
unsupported := smallJPEGRequest()
unsupported := smallJPEGTransform()
unsupported.Format = "bmp"
cases := []struct {
name string
input io.Reader
req *Request
name string
input io.Reader
transform *imagerequest.Transform
// want is the error Process must return; nil means any error.
want error
}{
{
name: "input cannot be read",
input: iotest.ErrReader(errTestReadFailed),
req: smallJPEGRequest(),
want: errTestReadFailed,
name: "input cannot be read",
input: iotest.ErrReader(errTestReadFailed),
transform: smallJPEGTransform(),
want: errTestReadFailed,
},
{
name: "input over the byte limit",
input: bytes.NewReader(createTestJPEG(t, 800, 600)),
req: smallJPEGRequest(),
want: ErrInputDataTooLarge,
name: "input over the byte limit",
input: bytes.NewReader(createTestJPEG(t, 800, 600)),
transform: smallJPEGTransform(),
want: ErrInputDataTooLarge,
},
{
name: "input not an image",
input: strings.NewReader("not an image"),
req: smallJPEGRequest(),
name: "input not an image",
input: strings.NewReader("not an image"),
transform: smallJPEGTransform(),
},
{
name: "output format not supported",
input: bytes.NewReader(valid),
req: unsupported,
want: ErrUnsupportedOutputFormat,
name: "output format not supported",
input: bytes.NewReader(valid),
transform: unsupported,
want: ErrUnsupportedOutputFormat,
},
}
@@ -305,7 +307,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
proc := New(Params{MaxInputBytes: 4096, MaxConcurrentProcessing: 1})
proc.processingWaitTimeout = 100 * time.Millisecond
_, err := proc.Process(context.Background(), tc.input, tc.req)
_, err := proc.Process(context.Background(), tc.input, tc.transform)
if err == nil || (tc.want != nil && !errors.Is(err, tc.want)) {
t.Fatalf("Process() error = %v, want %v", err, tc.want)
}
@@ -315,7 +317,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
}
result, err := proc.Process(context.Background(), bytes.NewReader(valid),
smallJPEGRequest())
smallJPEGTransform())
if err != nil {
t.Fatalf("Process() after the error = %v, want nil", err)
}
@@ -6,6 +6,7 @@ import (
"testing"
"github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_OrigOfUnnamedFormatRefused verifies that orig of a TIFF
@@ -27,7 +28,8 @@ func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
}
result, err := New(Params{}).Process(
t.Context(), bytes.NewReader(tiff), &Request{Format: FormatOriginal},
t.Context(), bytes.NewReader(tiff),
&imagerequest.Transform{Format: imagerequest.FormatOriginal},
)
if result != nil {
t.Logf("Process() returned %s", result.ContentType)
@@ -4,6 +4,8 @@ import (
"bytes"
"io"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProcessor_LoadsEachSourceFormat verifies that an image in each of
@@ -14,11 +16,13 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
proc := New(Params{})
for _, format := range []Format{
FormatJPEG, FormatPNG, FormatGIF, FormatWebP, FormatAVIF, FormatJXL,
for _, format := range []imagerequest.Format{
imagerequest.FormatJPEG, imagerequest.FormatPNG, imagerequest.FormatGIF,
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
} {
saved, err := proc.Process(t.Context(),
bytes.NewReader(createTestPNG(t, 20, 20)), &Request{Format: format})
bytes.NewReader(createTestPNG(t, 20, 20)),
&imagerequest.Transform{Format: format})
if err != nil {
t.Fatalf("saving the test image as %s: %v", format, err)
}
@@ -29,7 +33,7 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
}
result, err := proc.Process(t.Context(), bytes.NewReader(source),
&Request{Format: FormatOriginal})
&imagerequest.Transform{Format: imagerequest.FormatOriginal})
if err != nil {
t.Errorf("orig of the %s image: %v", format, err)
+147
View File
@@ -0,0 +1,147 @@
// Package imagerequest holds the image request type, its parts, its checks,
// the parsing of the /v1/image/ path, the variant key and the source key. It
// imports no other package of this repo, so every package that handles an
// image request can import it.
package imagerequest
import (
"errors"
"net/url"
"time"
)
// ErrInvalidFitMode is returned when an invalid fit mode is provided.
var ErrInvalidFitMode = errors.New("invalid fit mode")
// Format represents supported output image formats.
type Format string
// Supported image output formats.
const (
FormatOriginal Format = "orig"
FormatJPEG Format = "jpeg"
FormatPNG Format = "png"
FormatWebP Format = "webp"
FormatAVIF Format = "avif"
FormatJXL Format = "jxl"
FormatGIF Format = "gif"
// FormatAuto stands for JPEG XL, AVIF, WebP or JPEG, chosen for each
// request from its Accept header once the URL's signature or token has
// been checked; it is never processed or cached as itself.
FormatAuto Format = "auto"
)
// Size represents requested image dimensions
type Size struct {
Width int
Height int
}
// OriginalSize returns true if this represents "keep original size"
func (s Size) OriginalSize() bool {
return s.Width == 0 && s.Height == 0
}
// Fit represents how to fit image into requested dimensions.
type Fit string
// Supported image fit modes.
const (
FitCover Fit = "cover"
FitContain Fit = "contain"
FitFill Fit = "fill"
FitInside Fit = "inside"
FitOutside Fit = "outside"
)
// ValidateFitMode checks if the given fit mode is valid.
// Returns ErrInvalidFitMode for unrecognized fit modes.
func ValidateFitMode(fit Fit) error {
switch fit {
case FitCover, FitContain, FitFill, FitInside, FitOutside, "":
return nil
default:
return ErrInvalidFitMode
}
}
// ValidateImageRequest checks a request's width and height with
// ValidateDimension and its fit mode with ValidateFitMode. Both the plain
// /v1/image/ route and the encrypted /v1/e/ route validate through this
// function so a request from either source enforces identical bounds.
func ValidateImageRequest(req *Request) error {
err := ValidateDimension("width", req.Transform.Size.Width)
if err != nil {
return err
}
err = ValidateDimension("height", req.Transform.Size.Height)
if err != nil {
return err
}
return ValidateFitMode(req.Transform.FitMode)
}
// Request represents a request for a processed image: the source to fetch
// and the transform to apply to it.
type Request struct {
// Source is what to fetch
Source Source
// Transform is what to do to it
Transform Transform
// Signature is the HMAC signature for non-allowlisted hosts
Signature string
// Expires is when the URL expires: the exp of a signed URL, or the expiry
// of an encrypted URL; the zero time if it has none
Expires time.Time
// AllowHTTP indicates whether HTTP (non-TLS) is allowed for this request
AllowHTTP bool
}
// Source is the image a request fetches from the origin.
type Source struct {
// Host is the origin host (e.g., "cdn.example.com")
Host string
// Path is the path on the origin with its percent-escapes decoded, as
// url.URL's Path holds it (e.g., "/photos/my cat.jpg")
Path string
// Query is the optional query string for the origin URL as the URL
// writes it, percent-escapes included, as url.URL's RawQuery holds it
Query string
}
// Transform is what a request does to its source; the image processor
// takes it alone.
type Transform struct {
// Size is the requested output dimensions
Size Size
// Format is the requested output format
Format Format
// Quality is the output quality (1-100) for lossy formats
Quality int
// FitMode is how to fit the image into requested dimensions
FitMode Fit
}
// SourceURL returns the full upstream URL to fetch: the source path
// percent-encoded again, as net/url writes a path, so each source path is one
// upstream path, and the source query as it is, with the characters
// escapeQuery encodes percent-encoded. Uses http:// scheme when AllowHTTP is
// true, otherwise https://.
func (r *Request) SourceURL() string {
scheme := "https"
if r.AllowHTTP {
scheme = "http"
}
sourceURL := url.URL{
Scheme: scheme,
Host: r.Source.Host,
Path: r.Source.Path,
RawQuery: escapeQuery(r.Source.Query),
}
return sourceURL.String()
}
+26
View File
@@ -0,0 +1,26 @@
package imagerequest
import (
"crypto/sha256"
"encoding/hex"
"fmt"
)
// SourceKey is a SHA256 hash identifying a cached source image (hex-encoded).
type SourceKey string
// Key returns the key the source is cached under, made as CacheKey makes a
// variant key.
// Format: sha256(host:path:query)
// The path and query go through escapePercentAndColon, so that a ":" in
// either cannot be taken for the one between them.
func (s Source) Key() SourceKey {
data := fmt.Sprintf("%s:%s:%s",
s.Host,
escapePercentAndColon(s.Path),
escapePercentAndColon(s.Query),
)
h := sha256.Sum256([]byte(data))
return SourceKey(hex.EncodeToString(h[:]))
}
@@ -0,0 +1,40 @@
package imagerequest
import "testing"
// TestSourceKey_DiffersForEachSource checks that each source gets a key of
// 64 hex characters, the same each time, and that no two of these sources
// share one, also when they differ only by a ":" or "%" in the path or query.
func TestSourceKey_DiffersForEachSource(t *testing.T) {
t.Parallel()
sources := []Source{
{Host: testHostCDN, Path: testPathCat},
{Host: testHostCDN, Path: testPathCat, Query: "v=2"},
{Host: "other.example.com", Path: testPathCat},
{Host: testHostCDN, Path: "/c:d"},
{Host: testHostCDN, Path: "/c", Query: "d:"},
{Host: testHostCDN, Path: "/c%3Ad"},
}
seen := make(map[SourceKey]Source)
for _, source := range sources {
key := source.Key()
t.Logf("Key() of %+v = %s", source, key)
if len(key) != 64 {
t.Errorf("Key() of %+v length = %d, want 64", source, len(key))
}
if again := source.Key(); again != key {
t.Errorf("Key() of %+v not deterministic: %s vs %s", source, key, again)
}
if earlier, found := seen[key]; found {
t.Errorf("Key() gave %+v and %+v the same key", earlier, source)
}
seen[key] = source
}
}
@@ -1,14 +1,16 @@
package imgcache
package imagerequest
import "testing"
func TestImageRequest_SourceURL_DefaultHTTPS(t *testing.T) {
func TestRequest_SourceURL_DefaultHTTPS(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceQuery: "v=2",
req := &Request{
Source: Source{
Host: testHostCDN,
Path: testPathCat,
Query: "v=2",
},
}
got := req.SourceURL()
@@ -19,13 +21,15 @@ func TestImageRequest_SourceURL_DefaultHTTPS(t *testing.T) {
}
}
func TestImageRequest_SourceURL_AllowHTTP(t *testing.T) {
func TestRequest_SourceURL_AllowHTTP(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: "localhost:8080",
SourcePath: testPathCat,
AllowHTTP: true,
req := &Request{
Source: Source{
Host: "localhost:8080",
Path: testPathCat,
},
AllowHTTP: true,
}
got := req.SourceURL()
@@ -36,13 +40,15 @@ func TestImageRequest_SourceURL_AllowHTTP(t *testing.T) {
}
}
func TestImageRequest_SourceURL_AllowHTTPFalse(t *testing.T) {
func TestRequest_SourceURL_AllowHTTPFalse(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: "/img.jpg",
AllowHTTP: false,
req := &Request{
Source: Source{
Host: testHostCDN,
Path: "/img.jpg",
},
AllowHTTP: false,
}
got := req.SourceURL()
@@ -1,4 +1,4 @@
package imgcache
package imagerequest
import (
"errors"
@@ -8,8 +8,6 @@ import (
"regexp"
"strconv"
"strings"
"sneak.berlin/go/pixa/internal/imageprocessor"
)
// URL parsing errors.
@@ -25,8 +23,8 @@ var (
)
// MaxDimension is the maximum allowed width or height, the same as the image
// processor's maximum for an output image.
const MaxDimension = imageprocessor.MaxOutputDimension
// processor's maximum for an output image (imageprocessor.MaxOutputDimension).
const MaxDimension = 8192
// ValidateDimension checks one requested width or height; name ("width" or
// "height") appears in the error. 0 is valid: that side is then worked out from
@@ -68,20 +66,6 @@ var queryEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}|[^!-~]|["#'<>%]`)
var hostRegex = regexp.MustCompile(
`^(?:[A-Za-z0-9_.-]+|\[([0-9A-Fa-f:.]+)\])(?::[0-9]*)?$`)
// ParsedURL contains the parsed components of an image proxy URL.
type ParsedURL struct {
// Host is the source origin host (e.g., "cdn.example.com")
Host string
// Path is the path on the origin (e.g., "/photos/cat.jpg")
Path string
// Query is the optional query string for the origin
Query string
// Size is the requested output dimensions
Size Size
// Format is the requested output format
Format ImageFormat
}
// ParseImagePath parses an image URL's path after /v1/image/, decoded once:
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL.
// Examples:
@@ -89,7 +73,7 @@ type ParsedURL struct {
// - cdn.example.com/photos/cat.jpg/0x0.jpeg
// - cdn.example.com/photos/cat.jpg/orig.png
// - cdn.example.com/photos/cat.jpg/800x600
func ParseImagePath(path string) (*ParsedURL, error) {
func ParseImagePath(path string) (*Request, error) {
// Strip leading slash if present (chi may include it)
path = strings.TrimPrefix(path, "/")
if path == "" {
@@ -103,7 +87,7 @@ func ParseImagePath(path string) (*ParsedURL, error) {
// it: /v1/image/<host>/<path>/<size>.<format>, or /v1/image/<host>/<path>/<size>
// for JPEG XL. A source query follows the source path after a "?", which the
// URL writes as %3F.
func ParseImageURL(urlPath string) (*ParsedURL, error) {
func ParseImageURL(urlPath string) (*Request, error) {
// Remove the /v1/image/ prefix
const prefix = "/v1/image/"
if !strings.HasPrefix(urlPath, prefix) {
@@ -120,7 +104,7 @@ func ParseImageURL(urlPath string) (*ParsedURL, error) {
// parseImageComponents parses <host>/<path>/<size>.<format>, or
// <host>/<path>/<size> for JPEG XL.
func parseImageComponents(remainder string) (*ParsedURL, error) {
func parseImageComponents(remainder string) (*Request, error) {
// Find the last path segment, which holds "size" or "size.format"
lastSlash := strings.LastIndex(remainder, "/")
if lastSlash == -1 {
@@ -209,12 +193,9 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
path = "/" + path
}
return &ParsedURL{
Host: host,
Path: path,
Query: query,
Size: size,
Format: format,
return &Request{
Source: Source{Host: host, Path: path, Query: query},
Transform: Transform{Size: size, Format: format},
}, nil
}
@@ -297,7 +278,7 @@ func escapeQuery(query string) string {
}
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
func parseSizeFormat(s string) (Size, ImageFormat, error) {
func parseSizeFormat(s string) (Size, Format, error) {
matches := sizeFormatRegex.FindStringSubmatch(s)
if matches == nil {
return Size{}, "", ErrInvalidSize
@@ -351,8 +332,8 @@ func parseSizeFormat(s string) (Size, ImageFormat, error) {
return size, format, nil
}
// parseFormat converts a format string to ImageFormat.
func parseFormat(s string) (ImageFormat, error) {
// parseFormat converts a format string to Format.
func parseFormat(s string) (Format, error) {
switch strings.ToLower(s) {
case "orig", "original":
return FormatOriginal, nil
@@ -374,14 +355,3 @@ func parseFormat(s string) (ImageFormat, error) {
return "", fmt.Errorf("%w: %s", ErrInvalidFormat, s)
}
}
// ToImageRequest converts a ParsedURL to an ImageRequest.
func (p *ParsedURL) ToImageRequest() *ImageRequest {
return &ImageRequest{
SourceHost: p.Host,
SourcePath: p.Path,
SourceQuery: p.Query,
Size: p.Size,
Format: p.Format,
}
}
@@ -1,33 +1,41 @@
package imgcache
package imagerequest
import (
"errors"
"testing"
)
// assertParsedURL compares all fields of a parsed URL against the
// expected value.
func assertParsedURL(t *testing.T, got, want *ParsedURL) {
// Shared test fixture strings.
const (
testHostCDN = "cdn.example.com"
testHostExample = "example.com"
testPathCat = "/photos/cat.jpg"
)
// assertRequest compares the source host, path and query and the size and
// format of got against want.
func assertRequest(t *testing.T, got, want *Request) {
t.Helper()
if got.Host != want.Host {
t.Errorf("Host = %q, want %q", got.Host, want.Host)
if got.Source.Host != want.Source.Host {
t.Errorf("Host = %q, want %q", got.Source.Host, want.Source.Host)
}
if got.Path != want.Path {
t.Errorf("Path = %q, want %q", got.Path, want.Path)
if got.Source.Path != want.Source.Path {
t.Errorf("Path = %q, want %q", got.Source.Path, want.Source.Path)
}
if got.Query != want.Query {
t.Errorf("Query = %q, want %q", got.Query, want.Query)
if got.Source.Query != want.Source.Query {
t.Errorf("Query = %q, want %q", got.Source.Query, want.Source.Query)
}
if got.Size != want.Size {
t.Errorf("Size = %v, want %v", got.Size, want.Size)
if got.Transform.Size != want.Transform.Size {
t.Errorf("Size = %v, want %v", got.Transform.Size, want.Transform.Size)
}
if got.Format != want.Format {
t.Errorf("Format = %q, want %q", got.Format, want.Format)
if got.Transform.Format != want.Transform.Format {
t.Errorf("Format = %q, want %q",
got.Transform.Format, want.Transform.Format)
}
}
@@ -37,62 +45,64 @@ func TestParseImageURL(t *testing.T) {
tests := []struct {
name string
input string
want *ParsedURL
want *Request
}{
{
name: "basic path with size",
input: "/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat,
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
want: &Request{
Source: Source{Host: testHostCDN, Path: testPathCat},
Transform: Transform{Size: Size{Width: 800, Height: 600}, Format: FormatWebP},
},
},
{
name: "original size with 0x0",
input: "/v1/image/cdn.example.com/photos/cat.jpg/0x0.jpeg",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat,
Size: Size{Width: 0, Height: 0}, Format: FormatJPEG,
want: &Request{
Source: Source{Host: testHostCDN, Path: testPathCat},
Transform: Transform{Size: Size{Width: 0, Height: 0}, Format: FormatJPEG},
},
},
{
name: "original size with orig keyword",
input: "/v1/image/cdn.example.com/photos/cat.jpg/orig.png",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat,
Size: Size{Width: 0, Height: 0}, Format: FormatPNG,
want: &Request{
Source: Source{Host: testHostCDN, Path: testPathCat},
Transform: Transform{Size: Size{Width: 0, Height: 0}, Format: FormatPNG},
},
},
{
name: "path with query string",
input: "/v1/image/cdn.example.com/photos/cat.jpg?arg1=val1&arg2=val2/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat, Query: "arg1=val1&arg2=val2",
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
want: &Request{
Source: Source{
Host: testHostCDN, Path: testPathCat, Query: "arg1=val1&arg2=val2",
},
Transform: Transform{Size: Size{Width: 800, Height: 600}, Format: FormatWebP},
},
},
{
name: "deep nested path",
input: "/v1/image/cdn.example.com/a/b/c/d/image.jpg/1920x1080.avif",
want: &ParsedURL{
Host: testHostCDN, Path: "/a/b/c/d/image.jpg",
Size: Size{Width: 1920, Height: 1080}, Format: FormatAVIF,
want: &Request{
Source: Source{Host: testHostCDN, Path: "/a/b/c/d/image.jpg"},
Transform: Transform{Size: Size{Width: 1920, Height: 1080}, Format: FormatAVIF},
},
},
{
name: "jpg alias for jpeg",
input: "/v1/image/example.com/img.png/100x100.jpg",
want: &ParsedURL{
Host: testHostExample, Path: "/img.png",
Size: Size{Width: 100, Height: 100}, Format: FormatJPEG,
want: &Request{
Source: Source{Host: testHostExample, Path: "/img.png"},
Transform: Transform{Size: Size{Width: 100, Height: 100}, Format: FormatJPEG},
},
},
{
name: "gif format",
input: "/v1/image/example.com/animated.gif/200x200.gif",
want: &ParsedURL{
Host: testHostExample, Path: "/animated.gif",
Size: Size{Width: 200, Height: 200}, Format: FormatGIF,
want: &Request{
Source: Source{Host: testHostExample, Path: "/animated.gif"},
Transform: Transform{Size: Size{Width: 200, Height: 200}, Format: FormatGIF},
},
},
}
@@ -106,7 +116,7 @@ func TestParseImageURL(t *testing.T) {
t.Fatalf("ParseImageURL() unexpected error = %v", err)
}
assertParsedURL(t, got, tt.want)
assertRequest(t, got, tt.want)
})
}
}
@@ -121,8 +131,8 @@ func TestParseImageURL_AutoFormat(t *testing.T) {
t.Fatalf("ParseImageURL() error = %v", err)
}
if got.Format != FormatAuto {
t.Errorf("Format = %q, want %q", got.Format, FormatAuto)
if got.Transform.Format != FormatAuto {
t.Errorf("Format = %q, want %q", got.Transform.Format, FormatAuto)
}
}
@@ -189,27 +199,35 @@ func TestParseImagePath(t *testing.T) {
tests := []struct {
name string
input string
want *ParsedURL
want *Request
wantErr bool
}{
{
name: "chi wildcard capture",
input: "cdn.example.com/photos/cat.jpg/800x600.webp",
want: &ParsedURL{
Host: testHostCDN,
Path: testPathCat,
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
want: &Request{
Source: Source{
Host: testHostCDN,
Path: testPathCat,
},
Transform: Transform{
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
},
},
},
{
name: "with leading slash from chi",
input: "/cdn.example.com/photos/cat.jpg/800x600.webp",
want: &ParsedURL{
Host: testHostCDN,
Path: testPathCat,
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
want: &Request{
Source: Source{
Host: testHostCDN,
Path: testPathCat,
},
Transform: Transform{
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
},
},
},
}
@@ -229,7 +247,7 @@ func TestParseImagePath(t *testing.T) {
return
}
assertParsedURL(t, got, tt.want)
assertRequest(t, got, tt.want)
})
}
}
@@ -246,14 +264,14 @@ func TestParseImagePath_LowerCasesHost(t *testing.T) {
t.Fatalf("ParseImagePath() error = %v", err)
}
t.Logf("parsed host %q, path %q", got.Host, got.Path)
t.Logf("parsed host %q, path %q", got.Source.Host, got.Source.Path)
if got.Host != testHostCDN {
t.Errorf("Host = %q, want %q", got.Host, testHostCDN)
if got.Source.Host != testHostCDN {
t.Errorf("Host = %q, want %q", got.Source.Host, testHostCDN)
}
if got.Path != "/Photos/Cat.jpg" {
t.Errorf("Path = %q, want %q", got.Path, "/Photos/Cat.jpg")
if got.Source.Path != "/Photos/Cat.jpg" {
t.Errorf("Path = %q, want %q", got.Source.Path, "/Photos/Cat.jpg")
}
}
@@ -277,40 +295,6 @@ func TestParseImagePath_RefusesUserinfo(t *testing.T) {
}
}
func TestParsedURL_ToImageRequest(t *testing.T) {
t.Parallel()
parsed := &ParsedURL{
Host: testHostCDN,
Path: testPathCat,
Query: "version=2",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
}
req := parsed.ToImageRequest()
if req.SourceHost != parsed.Host {
t.Errorf("SourceHost = %q, want %q", req.SourceHost, parsed.Host)
}
if req.SourcePath != parsed.Path {
t.Errorf("SourcePath = %q, want %q", req.SourcePath, parsed.Path)
}
if req.SourceQuery != parsed.Query {
t.Errorf("SourceQuery = %q, want %q", req.SourceQuery, parsed.Query)
}
if req.Size != parsed.Size {
t.Errorf("Size = %v, want %v", req.Size, parsed.Size)
}
if req.Format != parsed.Format {
t.Errorf("Format = %q, want %q", req.Format, parsed.Format)
}
}
func TestParseImageURL_PathTraversal(t *testing.T) {
t.Parallel()
@@ -417,22 +401,24 @@ func TestParseImageURL_SourceURLHoldingPercent25(t *testing.T) {
tests := []struct {
name string
input string
want *ParsedURL
want *Request
}{
{
name: "path",
input: "/v1/image/cdn.example.com/sale/50%25off.jpg/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: "/sale/50%25off.jpg",
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
want: &Request{
Source: Source{Host: testHostCDN, Path: "/sale/50%25off.jpg"},
Transform: Transform{Size: Size{Width: 800, Height: 600}, Format: FormatWebP},
},
},
{
name: "query",
input: "/v1/image/cdn.example.com/photos/cat.jpg?discount=50%25/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat, Query: "discount=50%25",
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
want: &Request{
Source: Source{
Host: testHostCDN, Path: testPathCat, Query: "discount=50%25",
},
Transform: Transform{Size: Size{Width: 800, Height: 600}, Format: FormatWebP},
},
},
}
@@ -446,7 +432,7 @@ func TestParseImageURL_SourceURLHoldingPercent25(t *testing.T) {
t.Fatalf("ParseImageURL() unexpected error = %v", err)
}
assertParsedURL(t, got, tt.want)
assertRequest(t, got, tt.want)
})
}
}
@@ -1,4 +1,4 @@
package imgcache
package imagerequest
import (
"errors"
@@ -10,35 +10,43 @@ func TestValidateImageRequest(t *testing.T) {
tests := []struct {
name string
req ImageRequest
req Request
wantErr error
}{
{
name: "within bounds",
req: ImageRequest{Size: Size{Width: 800, Height: 600}, FitMode: FitCover},
req: Request{Transform: Transform{
Size: Size{Width: 800, Height: 600}, FitMode: FitCover,
}},
},
{
name: "original size and empty fit",
req: ImageRequest{Size: Size{Width: 0, Height: 0}},
req: Request{Transform: Transform{Size: Size{Width: 0, Height: 0}}},
},
{
name: "width over limit",
req: ImageRequest{Size: Size{Width: MaxDimension + 1, Height: 600}},
name: "width over limit",
req: Request{Transform: Transform{
Size: Size{Width: MaxDimension + 1, Height: 600},
}},
wantErr: ErrDimensionTooLarge,
},
{
name: "height over limit",
req: ImageRequest{Size: Size{Width: 800, Height: MaxDimension + 1}},
name: "height over limit",
req: Request{Transform: Transform{
Size: Size{Width: 800, Height: MaxDimension + 1},
}},
wantErr: ErrDimensionTooLarge,
},
{
name: "negative width",
req: ImageRequest{Size: Size{Width: -1, Height: 600}},
req: Request{Transform: Transform{Size: Size{Width: -1, Height: 600}}},
wantErr: ErrInvalidSize,
},
{
name: "invalid fit mode",
req: ImageRequest{Size: Size{Width: 800, Height: 600}, FitMode: "bogus"},
name: "invalid fit mode",
req: Request{Transform: Transform{
Size: Size{Width: 800, Height: 600}, FitMode: "bogus",
}},
wantErr: ErrInvalidFitMode,
},
}
+39
View File
@@ -0,0 +1,39 @@
package imagerequest
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"strings"
)
// VariantKey is a SHA256 hash identifying a specific image variant (hex-encoded).
type VariantKey string
// CacheKey generates a unique key for a request variant.
// Format: sha256(host:path:query:width:height:format:quality:fit_mode)
// The path and query go through escapePercentAndColon, so that a ":" in
// either cannot be taken for the one between them.
func CacheKey(req *Request) VariantKey {
data := fmt.Sprintf("%s:%s:%s:%d:%d:%s:%d:%s",
req.Source.Host,
escapePercentAndColon(req.Source.Path),
escapePercentAndColon(req.Source.Query),
req.Transform.Size.Width,
req.Transform.Size.Height,
req.Transform.Format,
req.Transform.Quality,
req.Transform.FitMode,
)
h := sha256.Sum256([]byte(data))
return VariantKey(hex.EncodeToString(h[:]))
}
// escapePercentAndColon returns s with each "%" written as "%25" and each
// ":" as "%3A", so that the result holds no ":".
func escapePercentAndColon(s string) string {
s = strings.ReplaceAll(s, "%", "%25")
return strings.ReplaceAll(s, ":", "%3A")
}
@@ -0,0 +1,142 @@
package imagerequest
import "testing"
func TestCacheKey(t *testing.T) {
t.Parallel()
req1 := &Request{
Source: Source{Host: testHostCDN, Path: testPathCat, Query: ""},
Transform: Transform{
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
Quality: 85, FitMode: FitCover,
},
}
req2 := &Request{
Source: Source{Host: testHostCDN, Path: testPathCat, Query: ""},
Transform: Transform{
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
Quality: 85, FitMode: FitCover,
},
}
// Same request should produce same key
key1 := CacheKey(req1)
key2 := CacheKey(req2)
if key1 != key2 {
t.Errorf("CacheKey() not deterministic: %s vs %s", key1, key2)
}
// Key should be 64 hex chars
if len(string(key1)) != 64 {
t.Errorf("CacheKey() length = %d, want 64", len(string(key1)))
}
// Different size should produce different key
req3 := &Request{
Source: Source{Host: testHostCDN, Path: testPathCat, Query: ""},
Transform: Transform{
Size: Size{Width: 400, Height: 300}, Format: FormatWebP, // Different size
Quality: 85, FitMode: FitCover,
},
}
key3 := CacheKey(req3)
if key1 == key3 {
t.Error("CacheKey() produced same key for different sizes")
}
// Different format should produce different key
req4 := &Request{
Source: Source{Host: testHostCDN, Path: testPathCat, Query: ""},
Transform: Transform{
Size: Size{Width: 800, Height: 600}, Format: FormatPNG, // Different format
Quality: 85, FitMode: FitCover,
},
}
key4 := CacheKey(req4)
if key1 == key4 {
t.Error("CacheKey() produced same key for different formats")
}
// Different quality should produce different key
req5 := &Request{
Source: Source{Host: testHostCDN, Path: testPathCat, Query: ""},
Transform: Transform{
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
Quality: 50, FitMode: FitCover, // Different quality
},
}
key5 := CacheKey(req5)
if key1 == key5 {
t.Error("CacheKey() produced same key for different quality")
}
}
// TestCacheKey_ColonOrPercentInPathOrQuery checks that two different source
// URLs whose path or query holds ":" or "%" get different keys.
func TestCacheKey_ColonOrPercentInPathOrQuery(t *testing.T) {
t.Parallel()
cases := []struct {
name string
path1, query1 string
path2, query2 string
}{
{
name: "colon moved from path to query",
path1: "/a:b", query1: "",
path2: "/a", query2: "b:",
},
{
name: "colon in path and its percent-encoding",
path1: "/a:b", query1: "",
path2: "/a%3Ab", query2: "",
},
}
for _, tt := range cases {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
key1 := CacheKey(&Request{
Source: Source{
Host: testHostCDN,
Path: tt.path1,
Query: tt.query1,
},
Transform: Transform{
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
},
})
key2 := CacheKey(&Request{
Source: Source{
Host: testHostCDN,
Path: tt.path2,
Query: tt.query2,
},
Transform: Transform{
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
},
})
t.Logf("CacheKey() of path %q query %q = %s", tt.path1, tt.query1, key1)
t.Logf("CacheKey() of path %q query %q = %s", tt.path2, tt.query2, key2)
if key1 == key2 {
t.Errorf("CacheKey() gave path %q query %q and path %q query %q "+
"the same key", tt.path1, tt.query1, tt.path2, tt.query2)
}
})
}
}
+77 -176
View File
@@ -2,10 +2,7 @@ package imgcache
import (
"context"
"crypto/sha256"
"database/sql"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
@@ -17,6 +14,7 @@ import (
lru "github.com/hashicorp/golang-lru/v2"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// Cache errors.
@@ -25,9 +23,6 @@ var (
ErrNegativeCache = errors.New("negative cache hit")
)
// HTTP status code for successful fetch.
const httpStatusOK = 200
// metaCacheSize is how many variants' content types metaCache holds. A
// variant not among them is served as before, reading its .meta file.
const metaCacheSize = 10000
@@ -68,12 +63,11 @@ type CacheConfig struct {
// Cache implements the caching layer for the image proxy.
type Cache struct {
db *sql.DB
srcContent *ContentStorage // source images by content hash
variants *VariantStorage // processed variants by cache key
srcMetadata *MetadataStorage // source metadata by host/path
config CacheConfig
log *slog.Logger
db *sql.DB
srcContent *ContentStorage // source images by source key
variants *VariantStorage // processed variants by cache key
config CacheConfig
log *slog.Logger
// disabled means the disk cache is turned off entirely: lookups
// always miss, stores are no-ops, and no eviction runs.
@@ -111,20 +105,7 @@ type Cache struct {
// metaCache holds the content types of the variants most recently
// stored or served, so a hit does not read the variant's .meta file.
// It never stands in for the variant file, which is always opened.
metaCache *lru.Cache[VariantKey, string]
// contentLocks serializes StoreSource and evictSourceBlob per
// content hash, closing the race window between an eviction's row
// deletion and its file unlink against a concurrent store of
// identical content.
contentLocks *contentLock
// evictSourceBlobTestHook, when set, is invoked by evictSourceBlob
// after its row-deletion transaction commits and before the
// content file is unlinked. It exists solely so tests can
// deterministically pause inside that window to exercise
// concurrent stores against it; production code leaves it nil.
evictSourceBlobTestHook func(ContentHash)
metaCache *lru.Cache[imagerequest.VariantKey, string]
// reconciliationPageSize is the most rows one read of a content table
// returns in the reconciliation pass and in Stats. newCache sets it to
@@ -153,7 +134,7 @@ func newCache(
log = slog.Default()
}
metaCache, err := lru.New[VariantKey, string](metaCacheSize)
metaCache, err := lru.New[imagerequest.VariantKey, string](metaCacheSize)
if err != nil {
return nil, fmt.Errorf("failed to create variant content type cache: %w", err)
}
@@ -166,7 +147,6 @@ func newCache(
evictionPressure: make(chan struct{}, 1),
evictionDone: make(chan struct{}),
metaCache: metaCache,
contentLocks: newContentLock(),
pendingCountsAdded: make(chan struct{}, 1),
pendingCountsDone: make(chan struct{}),
@@ -192,16 +172,8 @@ func newCache(
return nil, fmt.Errorf("failed to create variant storage: %w", err)
}
srcMetadata, err := NewMetadataStorage(
filepath.Join(config.StateDir, "cache", "metadata"),
)
if err != nil {
return nil, fmt.Errorf("failed to create source metadata storage: %w", err)
}
c.srcContent = srcContent
c.variants = variants
c.srcMetadata = srcMetadata
if config.UseDefaultMaxBytes {
limit, err := c.computeDefaultMaxBytes(context.Background(), probe)
@@ -223,7 +195,7 @@ func newCache(
// LookupResult contains the result of a cache lookup.
type LookupResult struct {
Hit bool
CacheKey VariantKey
CacheKey imagerequest.VariantKey
ContentType string
SizeBytes int64
CacheStatus CacheStatus
@@ -231,8 +203,10 @@ type LookupResult struct {
// Lookup checks if a processed variant exists on disk. Hits touch the
// variant's LRU timestamp; a disabled cache always misses.
func (c *Cache) Lookup(ctx context.Context, req *ImageRequest) (*LookupResult, error) {
cacheKey := CacheKey(req)
func (c *Cache) Lookup(
ctx context.Context, req *imagerequest.Request,
) (*LookupResult, error) {
cacheKey := imagerequest.CacheKey(req)
// Check variant storage directly - no DB needed for cache hits
if !c.disabled && c.variants.Exists(cacheKey) {
@@ -257,7 +231,9 @@ func (c *Cache) Lookup(ctx context.Context, req *ImageRequest) (*LookupResult, e
// variant's .meta file and is then kept in metaCache. A variant with
// no .meta file is served as application/octet-stream, which is not
// kept.
func (c *Cache) GetVariant(cacheKey VariantKey) (io.ReadCloser, int64, string, error) {
func (c *Cache) GetVariant(
cacheKey imagerequest.VariantKey,
) (io.ReadCloser, int64, string, error) {
if c.disabled {
return nil, 0, "", ErrNotFound
}
@@ -278,88 +254,59 @@ func (c *Cache) GetVariant(cacheKey VariantKey) (io.ReadCloser, int64, string, e
return reader, size, contentType, nil
}
// StoreSource stores fetched source content and metadata. On a
// disabled cache it is a no-op returning an empty hash.
// StoreSource stores fetched source content: it writes the file named by
// the source's key, then the source's row. When the row cannot be written
// the file is removed and the store fails, as a file with no row is never
// served or evicted; if that removal fails too, the next reconciliation
// pass removes the file. On a disabled cache it is a no-op returning an
// empty key.
func (c *Cache) StoreSource(
ctx context.Context,
req *ImageRequest,
req *imagerequest.Request,
content io.Reader,
result *httpfetcher.FetchResult,
) (ContentHash, error) {
) (imagerequest.SourceKey, error) {
if c.disabled {
return "", nil
}
// Hash the content ourselves (rather than via srcContent.Store,
// which would hash internally) so the content hash is known before
// any file or database work happens: that lets the entire store be
// serialized, per hash, against a concurrent eviction of the same
// content below.
data, err := io.ReadAll(content)
if err != nil {
return "", fmt.Errorf("failed to read source content: %w", err)
}
sum := sha256.Sum256(data)
contentHash := ContentHash(hex.EncodeToString(sum[:]))
key := req.Source.Key()
// Hold the content hash's lock for the whole store operation. A
// concurrent eviction of this exact hash (the real SHA-256 dedup
// case: a different source path whose bytes hash identically)
// deletes the accounting rows and unlinks the file inside the same
// lock, so the two can never interleave: either this store
// completes first (and a subsequent eviction removes it together
// with its rows and file, correctly), or eviction completes first
// (and this store finds the file already gone and recreates it
// fresh) — never a fresh row left pointing at a file eviction is
// mid-unlink on.
unlock := c.contentLocks.Lock(string(contentHash))
defer unlock()
size, err := c.srcContent.StoreHashed(contentHash, data)
size, err := c.srcContent.Store(key, data)
if err != nil {
return "", fmt.Errorf("failed to store source content: %w", err)
}
// Store in database
pathHash := HashPath(req.SourcePath + "?" + req.SourceQuery)
headersJSON, err := json.Marshal(result.Headers)
if err != nil {
return "", fmt.Errorf("failed to marshal response headers: %w", err)
}
_, err = c.db.ExecContext(ctx, `
INSERT INTO source_content (content_hash, content_type, size_bytes)
VALUES (?, ?, ?)
ON CONFLICT(content_hash) DO NOTHING
`, contentHash, result.ContentType, size)
INSERT INTO source_content
(source_key, source_host, source_path, source_query,
content_type, size_bytes)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(source_key) DO UPDATE SET
content_type = excluded.content_type,
size_bytes = excluded.size_bytes,
fetched_at = CURRENT_TIMESTAMP,
last_accessed_at = CURRENT_TIMESTAMP
`, string(key), req.Source.Host, req.Source.Path, req.Source.Query,
result.ContentType, size)
if err != nil {
deleteErr := c.srcContent.Delete(key)
if deleteErr != nil {
c.log.Warn("failed to remove source file whose row was not written",
"source_key", key, "error", deleteErr)
}
return "", fmt.Errorf("failed to insert source content: %w", err)
}
_, err = c.db.ExecContext(ctx, `
INSERT INTO source_metadata
(source_host, source_path, source_query, path_hash,
content_hash, status_code, content_type, response_headers)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(source_host, source_path, source_query) DO UPDATE SET
content_hash = excluded.content_hash,
status_code = excluded.status_code,
content_type = excluded.content_type,
response_headers = excluded.response_headers,
fetched_at = CURRENT_TIMESTAMP
`, req.SourceHost, req.SourcePath, req.SourceQuery, pathHash,
contentHash, httpStatusOK, result.ContentType, string(headersJSON))
if err != nil {
return "", fmt.Errorf("failed to insert source metadata: %w", err)
}
c.writeMetadataSidecar(req, pathHash, contentHash, result)
c.notifyWritePressure()
return contentHash, nil
return key, nil
}
// StoreVariant stores a processed variant by its cache key and records
@@ -367,7 +314,8 @@ func (c *Cache) StoreSource(
// accounting insert is best-effort (the startup reconciliation pass
// adopts any variant file that misses its accounting row).
func (c *Cache) StoreVariant(
ctx context.Context, cacheKey VariantKey, content io.Reader, contentType string,
ctx context.Context, cacheKey imagerequest.VariantKey,
content io.Reader, contentType string,
) error {
if c.disabled {
return nil
@@ -399,22 +347,24 @@ func (c *Cache) StoreVariant(
}
// LookupSource checks if we have cached source content for a request.
// Returns the content hash and content type if found, or empty values
// if not. Hits touch the blob's LRU timestamp; a disabled cache always
// reports no cached source.
// Returns the source key and content type if found, or empty values
// if not; a row whose file is missing is not found. Hits touch the
// source's LRU timestamp; a disabled cache always reports no cached
// source.
func (c *Cache) LookupSource(
ctx context.Context, req *ImageRequest,
) (ContentHash, string, error) {
ctx context.Context, req *imagerequest.Request,
) (imagerequest.SourceKey, string, error) {
if c.disabled {
return "", "", nil
}
var hashStr, contentType string
key := req.Source.Key()
var contentType string
err := c.db.QueryRowContext(ctx, `
SELECT content_hash, content_type FROM source_metadata
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.SourceHost, req.SourcePath, req.SourceQuery).Scan(&hashStr, &contentType)
SELECT content_type FROM source_content WHERE source_key = ?
`, string(key)).Scan(&contentType)
if errors.Is(err, sql.ErrNoRows) {
return "", "", nil
@@ -424,21 +374,19 @@ func (c *Cache) LookupSource(
return "", "", fmt.Errorf("failed to lookup source: %w", err)
}
contentHash := ContentHash(hashStr)
// Verify the content file exists
if !c.srcContent.Exists(contentHash) {
if !c.srcContent.Exists(key) {
return "", "", nil
}
c.touchSourceContent(ctx, contentHash)
c.touchSourceContent(ctx, key)
return contentHash, contentType, nil
return key, contentType, nil
}
// StoreNegative stores a negative cache entry for a failed fetch.
func (c *Cache) StoreNegative(
ctx context.Context, req *ImageRequest, statusCode int, errMsg string,
ctx context.Context, req *imagerequest.Request, statusCode int, errMsg string,
) error {
expiresAt := time.Now().UTC().Add(c.config.NegativeTTL)
@@ -452,7 +400,7 @@ func (c *Cache) StoreNegative(
error_message = excluded.error_message,
fetched_at = CURRENT_TIMESTAMP,
expires_at = excluded.expires_at
`, req.SourceHost, req.SourcePath, req.SourceQuery, statusCode, errMsg, expiresAt)
`, req.Source.Host, req.Source.Path, req.Source.Query, statusCode, errMsg, expiresAt)
if err != nil {
return fmt.Errorf("failed to insert negative cache: %w", err)
}
@@ -460,33 +408,16 @@ func (c *Cache) StoreNegative(
return nil
}
// GetSourceMetadataID returns the source metadata ID for a request.
func (c *Cache) GetSourceMetadataID(
ctx context.Context, req *ImageRequest,
) (int64, error) {
var id int64
err := c.db.QueryRowContext(ctx, `
SELECT id FROM source_metadata
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.SourceHost, req.SourcePath, req.SourceQuery).Scan(&id)
if err != nil {
return 0, fmt.Errorf("failed to get source metadata ID: %w", err)
}
return id, nil
}
// GetSourceContent returns a reader for cached source content by its hash,
// GetSourceContent returns a reader for cached source content by its key,
// and the content's size in bytes.
func (c *Cache) GetSourceContent(
contentHash ContentHash,
key imagerequest.SourceKey,
) (io.ReadCloser, int64, error) {
if c.disabled {
return nil, 0, ErrNotFound
}
return c.srcContent.LoadWithSize(contentHash)
return c.srcContent.LoadWithSize(key)
}
// CleanExpired removes the negative cache rows past their expiry, one
@@ -665,7 +596,7 @@ func (c *Cache) readHitAndMissCounts(ctx context.Context, stats *CacheStats) err
// application/octet-stream and keeps nothing, so metaCache only ever
// holds a type read from a .meta file or passed to StoreVariant.
func (c *Cache) loadVariantWithMeta(
cacheKey VariantKey,
cacheKey imagerequest.VariantKey,
) (io.ReadCloser, int64, string, error) {
reader, size, contentType, err := c.variants.LoadWithMeta(cacheKey)
if err != nil {
@@ -681,39 +612,9 @@ func (c *Cache) loadVariantWithMeta(
return reader, size, contentType, nil
}
// writeMetadataSidecar writes the JSON metadata sidecar of a stored source.
// A failure is logged and is otherwise non-fatal; the metadata is in the
// database.
func (c *Cache) writeMetadataSidecar(
req *ImageRequest,
pathHash PathHash,
contentHash ContentHash,
result *httpfetcher.FetchResult,
) {
meta := &SourceMetadata{
Host: req.SourceHost,
Path: req.SourcePath,
Query: req.SourceQuery,
ContentHash: string(contentHash),
StatusCode: result.StatusCode,
ContentType: result.ContentType,
ContentLength: result.ContentLength,
ResponseHeaders: result.Headers,
FetchedAt: time.Now().UTC().Unix(),
FetchDurationMs: result.FetchDurationMs,
RemoteAddr: result.RemoteAddr,
}
err := c.srcMetadata.Store(req.SourceHost, pathHash, meta)
if err != nil {
c.log.Warn("failed to write metadata sidecar",
"host", req.SourceHost, "path_hash", pathHash, "error", err)
}
}
// touchVariant updates the LRU timestamp of a variant, best-effort:
// a failed touch only makes the entry look colder to eviction.
func (c *Cache) touchVariant(ctx context.Context, cacheKey VariantKey) {
func (c *Cache) touchVariant(ctx context.Context, cacheKey imagerequest.VariantKey) {
_, err := c.db.ExecContext(ctx, `
UPDATE variant_content SET last_accessed_at = CURRENT_TIMESTAMP
WHERE cache_key = ?
@@ -724,29 +625,29 @@ func (c *Cache) touchVariant(ctx context.Context, cacheKey VariantKey) {
}
}
// touchSourceContent updates the LRU timestamp of a source content
// blob, best-effort: a failed touch only makes the blob look colder.
func (c *Cache) touchSourceContent(ctx context.Context, contentHash ContentHash) {
// touchSourceContent updates the LRU timestamp of a cached source,
// best-effort: a failed touch only makes the source look colder.
func (c *Cache) touchSourceContent(ctx context.Context, key imagerequest.SourceKey) {
_, err := c.db.ExecContext(ctx, `
UPDATE source_content SET last_accessed_at = CURRENT_TIMESTAMP
WHERE content_hash = ?
`, string(contentHash))
WHERE source_key = ?
`, string(key))
if err != nil {
c.log.Debug("failed to touch source content LRU timestamp",
"content_hash", contentHash, "error", err)
"source_key", key, "error", err)
}
}
// checkNegativeCache checks if a request is in the negative cache.
func (c *Cache) checkNegativeCache(
ctx context.Context, req *ImageRequest,
ctx context.Context, req *imagerequest.Request,
) (bool, error) {
var expiresAt time.Time
err := c.db.QueryRowContext(ctx, `
SELECT expires_at FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.SourceHost, req.SourcePath, req.SourceQuery).Scan(&expiresAt)
`, req.Source.Host, req.Source.Path, req.Source.Query).Scan(&expiresAt)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
@@ -762,10 +663,10 @@ func (c *Cache) checkNegativeCache(
_, err = c.db.ExecContext(ctx, `
DELETE FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.SourceHost, req.SourcePath, req.SourceQuery)
`, req.Source.Host, req.Source.Path, req.Source.Query)
if err != nil {
c.log.Warn("failed to delete expired negative cache entry",
"host", req.SourceHost, "path", req.SourcePath, "error", err)
"host", req.Source.Host, "path", req.Source.Path, "error", err)
}
return false, nil
+85 -69
View File
@@ -10,6 +10,7 @@ import (
_ "modernc.org/sqlite"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
)
func setupTestDB(t *testing.T) *sql.DB {
@@ -23,26 +24,14 @@ func setupTestDB(t *testing.T) *sql.DB {
// Create schema
schema := `
CREATE TABLE source_content (
content_hash TEXT PRIMARY KEY,
content_type TEXT NOT NULL,
size_bytes INTEGER NOT NULL,
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE source_metadata (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_key TEXT PRIMARY KEY,
source_host TEXT NOT NULL,
source_path TEXT NOT NULL,
source_query TEXT NOT NULL DEFAULT '',
path_hash TEXT NOT NULL,
content_hash TEXT,
status_code INTEGER NOT NULL,
content_type TEXT,
response_headers TEXT,
content_type TEXT NOT NULL,
size_bytes INTEGER NOT NULL,
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
expires_at DATETIME,
etag TEXT,
last_modified TEXT,
UNIQUE(source_host, source_path, source_query)
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE output_content (
content_hash TEXT PRIMARY KEY,
@@ -117,13 +106,17 @@ func TestCache_LookupMiss(t *testing.T) {
cache := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostCDN,
Path: testPathCat,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: imagerequest.FormatWebP,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
result, err := cache.Lookup(ctx, req)
@@ -146,13 +139,17 @@ func TestCache_StoreAndLookup(t *testing.T) {
cache := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostCDN,
Path: testPathCat,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: imagerequest.FormatWebP,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Store source content
@@ -162,18 +159,18 @@ func TestCache_StoreAndLookup(t *testing.T) {
Headers: map[string][]string{testHeaderContentType: {testContentTypeJPEG}},
}
contentHash, err := cache.StoreSource(
sourceKey, err := cache.StoreSource(
ctx, req, bytes.NewReader(sourceContent), fetchResult)
if err != nil {
t.Fatalf("StoreSource() error = %v", err)
}
if contentHash == "" {
t.Error("StoreSource() returned empty hash")
if sourceKey == "" {
t.Error("StoreSource() returned empty key")
}
// Store variant
cacheKey := CacheKey(req)
cacheKey := imagerequest.CacheKey(req)
outputContent := []byte("fake webp data")
err = cache.StoreVariant(
@@ -207,11 +204,15 @@ func TestCache_NegativeCache(t *testing.T) {
cache := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: "/photos/notfound.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostCDN,
Path: "/photos/notfound.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: imagerequest.FormatWebP,
},
}
// Store negative cache entry
@@ -248,11 +249,15 @@ func TestCache_NegativeCacheExpiry(t *testing.T) {
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: "/photos/expired.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostCDN,
Path: "/photos/expired.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: imagerequest.FormatWebP,
},
}
// Store negative cache entry
@@ -281,17 +286,21 @@ func TestCache_VariantLookup(t *testing.T) {
cache := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: "/photos/variant.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostCDN,
Path: "/photos/variant.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: imagerequest.FormatWebP,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Store variant
cacheKey := CacheKey(req)
cacheKey := imagerequest.CacheKey(req)
outputContent := []byte("output data")
err := cache.StoreVariant(
@@ -331,17 +340,21 @@ func TestCache_GetVariant_ReturnsContentType(t *testing.T) {
cache := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: "/photos/variantct.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostCDN,
Path: "/photos/variantct.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: imagerequest.FormatWebP,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Store variant
cacheKey := CacheKey(req)
cacheKey := imagerequest.CacheKey(req)
outputContent := []byte("output webp data")
err := cache.StoreVariant(
@@ -383,17 +396,21 @@ func TestCache_GetVariant(t *testing.T) {
cache := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: "/photos/output.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostCDN,
Path: "/photos/output.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: imagerequest.FormatWebP,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Store variant
cacheKey := CacheKey(req)
cacheKey := imagerequest.CacheKey(req)
outputContent := []byte("the actual output content")
err := cache.StoreVariant(
@@ -518,7 +535,6 @@ func TestCache_StorageDirectoriesCreated(t *testing.T) {
dirs := []string{
"cache/sources",
"cache/variants",
"cache/metadata",
}
for _, dir := range dirs {
+12 -7
View File
@@ -5,6 +5,8 @@ import (
"strconv"
"strings"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestEvictionReadsTheUsageTotal adds, stores again, resizes and evicts
@@ -32,7 +34,7 @@ func TestEvictionReadsTheUsageTotal(t *testing.T) {
// pixa never changes a source's size, but a statement that does must
// change the total too.
_, err := cache.db.ExecContext(ctx,
`UPDATE source_content SET size_bytes = 900 WHERE content_hash = ?`,
`UPDATE source_content SET size_bytes = 900 WHERE source_key = ?`,
string(kept))
if err != nil {
t.Fatalf("failed to change the source's size: %v", err)
@@ -153,8 +155,9 @@ func TestUsageTotalNotCorrectedFromAnOlderSum(t *testing.T) {
// pass. No read the pass makes of a content table, to check its rows or
// to sum them, may cover more than two rows, so a request's query waits
// for one page at most. Between them the reads must still cover every
// row of both tables twice, once to check it and once to sum it, and the
// sum must put a wrong total right.
// variant row twice, once to check it and once to sum it, and every
// source row once, as source rows are read only to sum them, and the sum
// must put a wrong total right.
func TestReconciliationReadsAPageAtATime(t *testing.T) {
t.Parallel()
@@ -171,7 +174,8 @@ func TestReconciliationReadsAPageAtATime(t *testing.T) {
storeEvictionTestSource(t, cache, "pages.example.com",
"/"+strconv.Itoa(i)+".jpg", bytes.Repeat(content, 100))
storeEvictionTestVariant(t, cache, VariantKey("aabbccdd000"+strconv.Itoa(i)),
storeEvictionTestVariant(t, cache,
imagerequest.VariantKey("aabbccdd000"+strconv.Itoa(i)),
bytes.Repeat(content, 10))
}
@@ -204,9 +208,10 @@ func TestReconciliationReadsAPageAtATime(t *testing.T) {
coveredRows += rows
}
// Ten rows, each read once to check it and once to sum it.
if coveredRows != 20 {
t.Errorf("the reads covered %d rows in all, want 20", coveredRows)
// Five variant rows, each read once to check it and once to sum it,
// and five source rows, read only to sum them.
if coveredRows != 15 {
t.Errorf("the reads covered %d rows in all, want 15", coveredRows)
}
usage, err := cache.UsageBytes(ctx)
@@ -16,6 +16,7 @@ import (
"github.com/getsentry/sentry-go"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic"
)
@@ -80,14 +81,20 @@ func setupHeldFetchService(t *testing.T) (*Service, *TestFixtures, *heldFetcher)
// photoVariant asks for the test photo, 100x100, at 50x25 as a JPEG of the
// given quality and fit mode. Each call returns a new request, as Get writes
// to the request it is given.
func photoVariant(fixtures *TestFixtures, quality int, fit FitMode) *ImageRequest {
return &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 50, Height: 25},
Format: FormatJPEG,
Quality: quality,
FitMode: fit,
func photoVariant(
fixtures *TestFixtures, quality int, fit imagerequest.Fit,
) *imagerequest.Request {
return &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 25},
Format: imagerequest.FormatJPEG,
Quality: quality,
FitMode: fit,
},
}
}
@@ -100,7 +107,7 @@ type getResult struct {
// startGet calls Get in a goroutine of its own and delivers what it returned
// on the channel.
func startGet(
ctx context.Context, svc *Service, req *ImageRequest,
ctx context.Context, svc *Service, req *imagerequest.Request,
) <-chan getResult {
results := make(chan getResult, 1)
@@ -146,7 +153,7 @@ func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
pending := make([]<-chan getResult, 0, requests)
for range requests {
pending = append(pending,
startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover)))
startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover)))
}
<-fetcher.started
@@ -199,9 +206,11 @@ func TestService_Get_ConcurrentVariantsStayApart(t *testing.T) {
svc, fixtures, fetcher := setupHeldFetchService(t)
cover := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover))
lowQuality := startGet(t.Context(), svc, photoVariant(fixtures, 40, FitCover))
contain := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitContain))
cover := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
lowQuality := startGet(t.Context(), svc,
photoVariant(fixtures, 40, imagerequest.FitCover))
contain := startGet(t.Context(), svc,
photoVariant(fixtures, 85, imagerequest.FitContain))
for range 3 {
select {
@@ -253,12 +262,12 @@ func TestService_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
svc, fixtures, fetcher := setupHeldFetchService(t)
first := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover))
first := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
<-fetcher.started
waiterCtx, cancelWaiter := context.WithCancel(t.Context())
waiter := startGet(waiterCtx, svc, photoVariant(fixtures, 85, FitCover))
waiter := startGet(waiterCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
time.Sleep(arrivalWait)
cancelWaiter()
@@ -300,11 +309,11 @@ func TestService_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
svc, fixtures, fetcher := setupHeldFetchService(t)
firstCtx, cancelFirst := context.WithCancel(t.Context())
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, FitCover))
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
<-fetcher.started
second := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover))
second := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
time.Sleep(arrivalWait)
cancelFirst()
@@ -355,9 +364,9 @@ func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
svc, fixtures, fetcher := setupHeldFetchService(t)
request := func() *ImageRequest {
req := photoVariant(fixtures, 85, FitCover)
req.SourcePath = tc.path
request := func() *imagerequest.Request {
req := photoVariant(fixtures, 85, imagerequest.FitCover)
req.Source.Path = tc.path
return req
}
@@ -399,7 +408,7 @@ func TestService_Get_EndedRequestFetchesNothing(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
cancel()
_, err := svc.Get(ctx, photoVariant(fixtures, 85, FitCover))
_, err := svc.Get(ctx, photoVariant(fixtures, 85, imagerequest.FitCover))
if !errors.Is(err, context.Canceled) {
t.Errorf("Get() error = %v, want %v", err, context.Canceled)
}
@@ -423,7 +432,7 @@ func TestService_Get_ReturnsByItsDeadline(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), timeout)
defer cancel()
results := startGet(ctx, svc, photoVariant(fixtures, 85, FitCover))
results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
select {
case got := <-results:
@@ -456,11 +465,11 @@ func TestService_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
firstCtx, cancelFirst := context.WithTimeout(t.Context(), firstTimeout)
defer cancelFirst()
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, FitCover))
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
<-fetcher.started
waiter := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover))
waiter := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
time.Sleep(arrivalWait)
fetcher.releaseFetches()
@@ -503,7 +512,7 @@ func TestService_Get_PanicBecomesAnError(t *testing.T) {
}
}()
_, err = svc.Get(t.Context(), photoVariant(fixtures, 85, FitCover))
_, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
}()
if err == nil || !strings.Contains(err.Error(), "upstream fetcher panicked") {
@@ -540,7 +549,7 @@ func TestService_Get_PanicIsReportedToSentry(t *testing.T) {
}
}()
_, _ = svc.Get(ctx, photoVariant(fixtures, 85, FitCover))
_, _ = svc.Get(ctx, photoVariant(fixtures, 85, imagerequest.FitCover))
}()
events := transport.Events()
-64
View File
@@ -1,64 +0,0 @@
package imgcache
import "sync"
// contentLock provides per-key mutual exclusion for content-hash keyed
// operations. StoreSource and evictSourceBlob each hold a content
// hash's lock for the full duration of their file-plus-accounting-row
// work, so a store and an eviction racing on identical content bytes
// (the real SHA-256 content-addressed dedup case, not a contrived one)
// can never interleave: the unlink of an evicted blob's file can never
// race the creation of a fresh database row for a concurrently
// re-stored copy of the same content. Entries are removed once no
// goroutine holds or is waiting for them, so a long-running process
// does not accumulate memory proportional to the number of distinct
// content hashes it has ever seen.
type contentLock struct {
mu sync.Mutex
entries map[string]*contentLockEntry
}
// contentLockEntry is one key's exclusion lock plus a count of
// goroutines currently holding or waiting to acquire it, used to know
// when it is safe to remove the entry from the map.
type contentLockEntry struct {
mu sync.Mutex
count int
}
// newContentLock creates an empty contentLock.
func newContentLock() *contentLock {
return &contentLock{entries: make(map[string]*contentLockEntry)}
}
// Lock acquires exclusive access for key, blocking until it is
// available, and returns a function that releases it. The caller must
// invoke the returned function exactly once to release the lock.
func (c *contentLock) Lock(key string) func() {
c.mu.Lock()
entry, ok := c.entries[key]
if !ok {
entry = &contentLockEntry{}
c.entries[key] = entry
}
entry.count++
c.mu.Unlock()
entry.mu.Lock()
return func() {
entry.mu.Unlock()
c.mu.Lock()
entry.count--
if entry.count == 0 {
delete(c.entries, key)
}
c.mu.Unlock()
}
}
@@ -1,142 +0,0 @@
package imgcache
import (
"sync"
"sync/atomic"
"testing"
"time"
)
// TestContentLockExcludesSameKey verifies that two goroutines locking
// the same key never run their critical sections concurrently.
func TestContentLockExcludesSameKey(t *testing.T) {
t.Parallel()
lock := newContentLock()
var (
active atomic.Int32
maxSeen int32
wg sync.WaitGroup
)
const goroutines = 20
wg.Add(goroutines)
for range goroutines {
go func() {
defer wg.Done()
unlock := lock.Lock("same-key")
defer unlock()
n := active.Add(1)
for {
seen := atomic.LoadInt32(&maxSeen)
if n <= seen || atomic.CompareAndSwapInt32(&maxSeen, seen, n) {
break
}
}
time.Sleep(time.Millisecond)
active.Add(-1)
}()
}
wg.Wait()
if maxSeen != 1 {
t.Errorf("max concurrent holders of the same key = %d, want 1", maxSeen)
}
}
// TestContentLockAllowsDifferentKeys verifies that locking distinct
// keys does not serialize unrelated work: all goroutines must be able
// to enter their critical sections at once, proven by every one of
// them reaching the rendezvous point before any is allowed to
// proceed.
func TestContentLockAllowsDifferentKeys(t *testing.T) {
t.Parallel()
lock := newContentLock()
const goroutines = 20
var (
wg sync.WaitGroup
inside atomic.Int32
reached = make(chan struct{}, goroutines)
)
wg.Add(goroutines)
release := make(chan struct{})
for i := range goroutines {
key := string(rune('a' + i))
go func() {
defer wg.Done()
unlock := lock.Lock(key)
defer unlock()
inside.Add(1)
reached <- struct{}{}
<-release
}()
}
// Every goroutine must reach the rendezvous point (i.e. acquire its
// own key's lock) without needing any other to release first. If
// keys were incorrectly serialized onto one underlying lock, only
// one would get here and this would time out.
for i := range goroutines {
select {
case <-reached:
case <-time.After(2 * time.Second):
t.Fatalf("only %d/%d goroutines locking distinct keys made progress; "+
"keys may be incorrectly serialized", i, goroutines)
}
}
if n := inside.Load(); n != goroutines {
t.Errorf("goroutines inside their critical section = %d, want %d",
n, goroutines)
}
close(release)
wg.Wait()
}
// TestContentLockRemovesEntryAfterUnlock verifies that the internal
// entries map does not grow without bound: once no goroutine holds or
// awaits a key, its entry is removed.
func TestContentLockRemovesEntryAfterUnlock(t *testing.T) {
t.Parallel()
lock := newContentLock()
unlock := lock.Lock("k")
lock.mu.Lock()
if _, ok := lock.entries["k"]; !ok {
lock.mu.Unlock()
t.Fatal("entry missing while lock is held")
}
lock.mu.Unlock()
unlock()
lock.mu.Lock()
defer lock.mu.Unlock()
if _, ok := lock.entries["k"]; ok {
t.Error("entry for key still present after the last holder unlocked")
}
}
+55 -217
View File
@@ -11,6 +11,8 @@ import (
"path/filepath"
"strings"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// DefaultEvictionInterval is how often the background evictor checks
@@ -82,11 +84,11 @@ func (c *Cache) UsageBytes(ctx context.Context) (int64, error) {
// evictionCandidate is one LRU eviction victim candidate: either a
// processed variant (isVariant true, identified by cacheKey) or a
// source content blob (identified by contentHash).
// source content blob (identified by sourceKey).
type evictionCandidate struct {
isVariant bool
cacheKey VariantKey
contentHash ContentHash
cacheKey imagerequest.VariantKey
sourceKey imagerequest.SourceKey
sizeBytes int64
lastAccessedAt string
}
@@ -158,7 +160,7 @@ func (c *Cache) evictBatch(ctx context.Context, excessBytes int64) (int64, error
c.log.Warn("failed to evict cache entry",
"cache_key", candidate.cacheKey,
"content_hash", candidate.contentHash,
"source_key", candidate.sourceKey,
"error", err,
)
@@ -177,7 +179,7 @@ func (c *Cache) evictCandidate(ctx context.Context, candidate evictionCandidate)
return c.evictVariant(ctx, candidate.cacheKey)
}
return c.evictSourceBlob(ctx, candidate.contentHash)
return c.evictSourceBlob(ctx, candidate.sourceKey)
}
// evictionCandidates returns up to evictionBatchSize variants and
@@ -240,7 +242,7 @@ func (c *Cache) variantCandidates(ctx context.Context) ([]evictionCandidate, err
return nil, fmt.Errorf("failed to scan variant candidate: %w", err)
}
candidate.cacheKey = VariantKey(key)
candidate.cacheKey = imagerequest.VariantKey(key)
candidates = append(candidates, candidate)
}
@@ -257,9 +259,9 @@ func (c *Cache) variantCandidates(ctx context.Context) ([]evictionCandidate, err
// SQLite reads the rows in order from that column's index instead of
// sorting the whole table.
const sourceCandidatesQuery = `
SELECT content_hash, size_bytes, last_accessed_at
SELECT source_key, size_bytes, last_accessed_at
FROM source_content
ORDER BY last_accessed_at ASC, content_hash ASC
ORDER BY last_accessed_at ASC, source_key ASC
LIMIT ?`
// sourceCandidates returns the least recently used source blobs.
@@ -276,14 +278,14 @@ func (c *Cache) sourceCandidates(ctx context.Context) ([]evictionCandidate, erro
for rows.Next() {
var candidate evictionCandidate
var hash string
var key string
err := rows.Scan(&hash, &candidate.sizeBytes, &candidate.lastAccessedAt)
err := rows.Scan(&key, &candidate.sizeBytes, &candidate.lastAccessedAt)
if err != nil {
return nil, fmt.Errorf("failed to scan source candidate: %w", err)
}
candidate.contentHash = ContentHash(hash)
candidate.sourceKey = imagerequest.SourceKey(key)
candidates = append(candidates, candidate)
}
@@ -300,7 +302,9 @@ func (c *Cache) sourceCandidates(ctx context.Context) ([]evictionCandidate, erro
// file. The metaCache entry goes before the files; a GetVariant that
// read them just before may put it back, and the next GetVariant then
// fails to open the file and removes it again.
func (c *Cache) evictVariant(ctx context.Context, cacheKey VariantKey) error {
func (c *Cache) evictVariant(
ctx context.Context, cacheKey imagerequest.VariantKey,
) error {
_, err := c.db.ExecContext(ctx,
`DELETE FROM variant_content WHERE cache_key = ?`, string(cacheKey))
if err != nil {
@@ -317,117 +321,18 @@ func (c *Cache) evictVariant(ctx context.Context, cacheKey VariantKey) error {
return nil
}
// sourceReference identifies one source_metadata row's JSON sidecar.
type sourceReference struct {
host string
pathHash PathHash
}
// evictSourceBlob removes one source content blob. All source_metadata
// rows referencing the blob are deleted together with its
// source_content row in a single transaction BEFORE the file is
// unlinked: a blob referenced by multiple source paths is only ever
// removed together with all of its references, and database rows never
// point at deleted files. The JSON metadata sidecars for the removed
// rows are deleted afterwards.
//
// The whole operation holds the content hash's lock (the same one
// StoreSource holds for its full store), so a concurrent store of
// identical content bytes can never observe the file gone but a row
// still present, or insert a fresh row between this transaction's
// commit and the file unlink below: it either runs entirely before
// this eviction starts, or is blocked until this eviction (row
// deletion and unlink together) has fully completed.
func (c *Cache) evictSourceBlob(ctx context.Context, contentHash ContentHash) error {
unlock := c.contentLocks.Lock(string(contentHash))
defer unlock()
references, err := c.sourceReferences(ctx, contentHash)
if err != nil {
return err
}
tx, err := c.db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("failed to begin eviction transaction: %w", err)
}
defer func() { _ = tx.Rollback() }()
_, err = tx.ExecContext(ctx,
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash))
if err != nil {
return fmt.Errorf("failed to delete source metadata rows: %w", err)
}
_, err = tx.ExecContext(ctx,
`DELETE FROM source_content WHERE content_hash = ?`, string(contentHash))
// evictSourceBlob removes one source: its row first, then its file, as
// evictVariant does for a variant. It holds no lock: a store of the same
// source in between can leave the source's row with no file, which
// LookupSource treats as a miss.
func (c *Cache) evictSourceBlob(ctx context.Context, key imagerequest.SourceKey) error {
_, err := c.db.ExecContext(ctx,
`DELETE FROM source_content WHERE source_key = ?`, string(key))
if err != nil {
return fmt.Errorf("failed to delete source content row: %w", err)
}
err = tx.Commit()
if err != nil {
return fmt.Errorf("failed to commit eviction transaction: %w", err)
}
if c.evictSourceBlobTestHook != nil {
c.evictSourceBlobTestHook(contentHash)
}
// Only after the rows are gone may the files be removed.
for _, reference := range references {
err := c.srcMetadata.Delete(reference.host, reference.pathHash)
if err != nil {
c.log.Warn("failed to delete metadata sidecar",
"host", reference.host, "path_hash", reference.pathHash, "error", err)
}
}
err = c.srcContent.Delete(contentHash)
if err != nil {
return err
}
return nil
}
// sourceReferences lists the metadata sidecar locations of every
// source_metadata row referencing the given blob.
func (c *Cache) sourceReferences(
ctx context.Context, contentHash ContentHash,
) ([]sourceReference, error) {
rows, err := c.db.QueryContext(ctx, `
SELECT source_host, path_hash FROM source_metadata WHERE content_hash = ?
`, string(contentHash))
if err != nil {
return nil, fmt.Errorf("failed to query source references: %w", err)
}
defer func() { _ = rows.Close() }()
var references []sourceReference
for rows.Next() {
var reference sourceReference
var pathHash string
err := rows.Scan(&reference.host, &pathHash)
if err != nil {
return nil, fmt.Errorf("failed to scan source reference: %w", err)
}
reference.pathHash = PathHash(pathHash)
references = append(references, reference)
}
err = rows.Err()
if err != nil {
return nil, fmt.Errorf("source reference iteration failed: %w", err)
}
return references, nil
return c.srcContent.Delete(key)
}
// notifyWritePressure wakes the background evictor after a store, so
@@ -582,12 +487,12 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
// as long after the last pass ended as that pass took, off the request
// hot path: it adopts variant files that predate the accounting table
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
// insert under transient DB contention), drops accounting rows whose
// files are missing, removes source blob files the database does not
// know (and rows whose files are gone), removes source and variant files
// that are empty or not the size their rows record, sweeps stale temp
// files left behind by crashed writes, and last checks the total cache
// usage against the tables. It reads the tables a page at a time.
// insert under transient DB contention), drops variant accounting rows
// whose files are missing, removes source blob files the database does
// not know, removes source and variant files that are empty or not the
// size their rows record, sweeps stale temp files left behind by crashed
// writes, and last checks the total cache usage against the tables. It
// reads the tables a page at a time.
// Running it periodically, not just once, bounds how long such drift can
// accumulate unaccounted for on a long-running process to about one
// eviction interval plus ten times the last pass's time.
@@ -613,11 +518,6 @@ func (c *Cache) reconcileAccounting(ctx context.Context) error {
return err
}
err = c.reconcileSourceRows(ctx)
if err != nil {
return err
}
return c.reconcileUsageTotal(ctx)
}
@@ -648,7 +548,7 @@ func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
return nil
}
return c.reconcileVariantFile(ctx, path, entry, VariantKey(name))
return c.reconcileVariantFile(ctx, path, entry, imagerequest.VariantKey(name))
},
)
}
@@ -659,7 +559,7 @@ func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
// crash or a restored snapshot can leave it, is removed with its row and
// a warning naming it, so the variant is converted again.
func (c *Cache) reconcileVariantFile(
ctx context.Context, path string, entry fs.DirEntry, cacheKey VariantKey,
ctx context.Context, path string, entry fs.DirEntry, cacheKey imagerequest.VariantKey,
) error {
info, err := entry.Info()
if err != nil {
@@ -691,7 +591,7 @@ func (c *Cache) reconcileVariantFile(
// adoptVariantFile inserts an accounting row for a variant file that
// has none, using the file's size and modification time.
func (c *Cache) adoptVariantFile(
ctx context.Context, path string, info fs.FileInfo, cacheKey VariantKey,
ctx context.Context, path string, info fs.FileInfo, cacheKey imagerequest.VariantKey,
) error {
modTime := info.ModTime().UTC().Format(sqliteTimestampLayout)
contentType := c.variantContentTypeFromSidecar(path)
@@ -731,7 +631,7 @@ func (c *Cache) variantContentTypeFromSidecar(variantPath string) string {
// reconcileVariantRows drops accounting rows whose variant files are
// missing, so the database never references deleted content.
func (c *Cache) reconcileVariantRows(ctx context.Context) error {
var after VariantKey
var after imagerequest.VariantKey
for {
keys, err := c.variantKeysAfter(ctx, after)
@@ -772,9 +672,9 @@ func (c *Cache) reconcileVariantRows(ctx context.Context) error {
// variantKeysAfter returns, in order, up to c.reconciliationPageSize
// tracked variant cache keys that sort after the given one.
func (c *Cache) variantKeysAfter(
ctx context.Context, after VariantKey,
) ([]VariantKey, error) {
return queryStringColumn[VariantKey](ctx, c.db, `
ctx context.Context, after imagerequest.VariantKey,
) ([]imagerequest.VariantKey, error) {
return queryStringColumn[imagerequest.VariantKey](ctx, c.db, `
SELECT cache_key FROM variant_content
WHERE cache_key > ? ORDER BY cache_key LIMIT ?
`, "variant keys", "variant key", string(after), c.reconciliationPageSize)
@@ -838,19 +738,19 @@ func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
return nil
}
return c.reconcileSourceFile(ctx, path, entry, ContentHash(name))
return c.reconcileSourceFile(ctx, path, entry, imagerequest.SourceKey(name))
},
)
}
// reconcileSourceFile checks one source blob file against its
// source_content row. A file without a row is removed: it is
// unreachable, as source lookups always go through source_metadata. A
// file that is empty, or not the size its row records, as a crash or a
// restored snapshot can leave it, is removed with its rows and a warning
// naming it, so the source is fetched again.
// source_content row. A file without a row is removed, as no lookup
// serves it and no eviction removes it. A file that is empty, or not the
// size its row records, as a crash or a restored snapshot can leave it,
// is removed with its row and a warning naming it, so the source is
// fetched again.
func (c *Cache) reconcileSourceFile(
ctx context.Context, path string, entry fs.DirEntry, contentHash ContentHash,
ctx context.Context, path string, entry fs.DirEntry, key imagerequest.SourceKey,
) error {
info, err := entry.Info()
if err != nil {
@@ -860,13 +760,13 @@ func (c *Cache) reconcileSourceFile(
var recordedSize int64
err = c.db.QueryRowContext(ctx,
`SELECT size_bytes FROM source_content WHERE content_hash = ?`,
string(contentHash),
`SELECT size_bytes FROM source_content WHERE source_key = ?`,
string(key),
).Scan(&recordedSize)
switch {
case errors.Is(err, sql.ErrNoRows):
return c.removeUntrackedSourceFile(ctx, path, contentHash)
return c.removeUntrackedSourceFile(path, key)
case err != nil:
return fmt.Errorf("failed to check source content row: %w", err)
case info.Size() > 0 && info.Size() == recordedSize:
@@ -877,96 +777,34 @@ func (c *Cache) reconcileSourceFile(
c.log.Warn("removing damaged cached source file", "path", path,
"file_bytes", info.Size(), "recorded_bytes", recordedSize)
return c.evictSourceBlob(ctx, contentHash)
return c.evictSourceBlob(ctx, key)
}
// removeUntrackedSourceFile deletes a source blob file that has no
// source_content row. Any source_metadata rows referencing the hash
// are removed first so no row ever points at a deleted file.
// source_content row.
func (c *Cache) removeUntrackedSourceFile(
ctx context.Context, path string, contentHash ContentHash,
path string, key imagerequest.SourceKey,
) error {
_, err := c.db.ExecContext(ctx,
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash))
if err != nil {
return fmt.Errorf("failed to delete metadata rows for untracked blob: %w", err)
}
err = os.Remove(path)
err := os.Remove(path)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("failed to remove untracked source file: %w", err)
}
c.log.Info("removed untracked source content file", "content_hash", contentHash)
c.log.Info("removed untracked source content file", "source_key", key)
return nil
}
// reconcileSourceRows removes source_content rows (and their metadata
// references and sidecars) whose blob files are missing on disk.
func (c *Cache) reconcileSourceRows(ctx context.Context) error {
var after ContentHash
for {
hashes, err := c.sourceContentHashesAfter(ctx, after)
if err != nil {
return err
}
if c.reconciliationReadTestHook != nil {
c.reconciliationReadTestHook(len(hashes))
}
if len(hashes) == 0 {
return nil
}
for _, hash := range hashes {
if ctx.Err() != nil {
return ctx.Err()
}
if c.srcContent.Exists(hash) {
continue
}
// The blob file is already gone; evictSourceBlob removes the
// rows and sidecars and tolerates the missing file.
err := c.evictSourceBlob(ctx, hash)
if err != nil {
return err
}
c.log.Info("dropped rows for missing source content file", "content_hash", hash)
}
after = hashes[len(hashes)-1]
}
}
// sourceContentHashesAfter returns, in order, up to
// c.reconciliationPageSize tracked source content hashes that sort after
// the given one.
func (c *Cache) sourceContentHashesAfter(
ctx context.Context, after ContentHash,
) ([]ContentHash, error) {
return queryStringColumn[ContentHash](ctx, c.db, `
SELECT content_hash FROM source_content
WHERE content_hash > ? ORDER BY content_hash LIMIT ?
`, "source content hashes", "content hash", string(after),
c.reconciliationPageSize)
}
// Each of these queries sums size_bytes over the next page of rows of
// one content table, the rows that sort after a key, and returns the
// page's last key, the sum and the number of rows in the page. Past the
// last row the page has no rows and the key is NULL.
const (
sourceSizePageQuery = `
SELECT MAX(content_hash), COALESCE(SUM(size_bytes), 0), COUNT(*)
SELECT MAX(source_key), COALESCE(SUM(size_bytes), 0), COUNT(*)
FROM (
SELECT content_hash, size_bytes FROM source_content
WHERE content_hash > ? ORDER BY content_hash LIMIT ?
SELECT source_key, size_bytes FROM source_content
WHERE source_key > ? ORDER BY source_key LIMIT ?
)`
variantSizePageQuery = `
SELECT MAX(cache_key), COALESCE(SUM(size_bytes), 0), COUNT(*)
+192 -338
View File
@@ -10,12 +10,14 @@ import (
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
_ "modernc.org/sqlite"
"modernc.org/sqlite"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// sqliteTimestampFormat matches the format SQLite's CURRENT_TIMESTAMP
@@ -27,10 +29,10 @@ const sqliteTimestampFormat = "2006-01-02 15:04:05"
// keys reused across the eviction tests, in the order the tests store
// them.
const (
testVariantKeyOne VariantKey = "aabbccdd0001"
testVariantKeyTwo VariantKey = "aabbccdd0002"
testVariantKeyThree VariantKey = "aabbccdd0003"
testVariantKeyFour VariantKey = "aabbccdd0004"
testVariantKeyOne imagerequest.VariantKey = "aabbccdd0001"
testVariantKeyTwo imagerequest.VariantKey = "aabbccdd0002"
testVariantKeyThree imagerequest.VariantKey = "aabbccdd0003"
testVariantKeyFour imagerequest.VariantKey = "aabbccdd0004"
)
// evictionTestDB creates an in-memory SQLite database with the real
@@ -82,18 +84,22 @@ func newEvictionTestCache(t *testing.T, maxBytes int64) (*Cache, string) {
}
// storeEvictionTestSource stores content as a fetched source for
// host/path and returns the resulting content hash.
// host/path and returns the resulting source key.
func storeEvictionTestSource(
t *testing.T, cache *Cache, host, path string, content []byte,
) ContentHash {
) imagerequest.SourceKey {
t.Helper()
req := &ImageRequest{
SourceHost: host,
SourcePath: path,
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: host,
Path: path,
},
Transform: imagerequest.Transform{
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
result := &httpfetcher.FetchResult{
@@ -105,20 +111,20 @@ func storeEvictionTestSource(
},
}
hash, err := cache.StoreSource(
key, err := cache.StoreSource(
context.Background(), req, bytes.NewReader(content), result,
)
if err != nil {
t.Fatalf("StoreSource(%s%s) failed: %v", host, path, err)
}
return hash
return key
}
// storeEvictionTestVariant stores content as a processed variant under
// the given cache key.
func storeEvictionTestVariant(
t *testing.T, cache *Cache, key VariantKey, content []byte,
t *testing.T, cache *Cache, key imagerequest.VariantKey, content []byte,
) {
t.Helper()
@@ -131,7 +137,7 @@ func storeEvictionTestVariant(
// setVariantLastAccessed backdates the last access time of a tracked
// variant, to make LRU ordering deterministic in tests.
func setVariantLastAccessed(
t *testing.T, cache *Cache, key VariantKey, when time.Time,
t *testing.T, cache *Cache, key imagerequest.VariantKey, when time.Time,
) {
t.Helper()
@@ -157,13 +163,13 @@ func setVariantLastAccessed(
// setSourceLastAccessed backdates the last access time of a tracked
// source content blob.
func setSourceLastAccessed(
t *testing.T, cache *Cache, hash ContentHash, when time.Time,
t *testing.T, cache *Cache, key imagerequest.SourceKey, when time.Time,
) {
t.Helper()
res, err := cache.db.ExecContext(t.Context(),
`UPDATE source_content SET last_accessed_at = ? WHERE content_hash = ?`,
when.UTC().Format(sqliteTimestampFormat), string(hash),
`UPDATE source_content SET last_accessed_at = ? WHERE source_key = ?`,
when.UTC().Format(sqliteTimestampFormat), string(key),
)
if err != nil {
t.Fatalf("failed to set source last_accessed_at: %v", err)
@@ -175,7 +181,7 @@ func setSourceLastAccessed(
}
if affected != 1 {
t.Fatalf("source %s has no accounting row (affected=%d)", hash, affected)
t.Fatalf("source %s has no accounting row (affected=%d)", key, affected)
}
}
@@ -200,32 +206,31 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
t.Helper()
rows, err := cache.db.QueryContext(t.Context(),
`SELECT content_hash FROM source_metadata
WHERE content_hash IS NOT NULL AND content_hash != ''`,
`SELECT source_key FROM source_content`,
)
if err != nil {
t.Fatalf("failed to query source_metadata: %v", err)
t.Fatalf("failed to query source_content: %v", err)
}
defer func() { _ = rows.Close() }()
for rows.Next() {
var hash string
var key string
err := rows.Scan(&hash)
err := rows.Scan(&key)
if err != nil {
t.Fatalf("failed to scan content_hash: %v", err)
t.Fatalf("failed to scan source_key: %v", err)
}
if !cache.srcContent.Exists(ContentHash(hash)) {
t.Errorf("source_metadata references content %s but the file is missing",
hash)
if !cache.srcContent.Exists(imagerequest.SourceKey(key)) {
t.Errorf("source_content references content %s but the file is missing",
key)
}
}
err = rows.Err()
if err != nil {
t.Fatalf("source_metadata iteration failed: %v", err)
t.Fatalf("source_content iteration failed: %v", err)
}
variantRows, err := cache.db.QueryContext(t.Context(),
@@ -244,7 +249,7 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
t.Fatalf("failed to scan cache_key: %v", err)
}
if !cache.variants.Exists(VariantKey(key)) {
if !cache.variants.Exists(imagerequest.VariantKey(key)) {
t.Errorf("variant_content references key %s but the file is missing", key)
}
}
@@ -308,33 +313,6 @@ func TestUsageBytesAccountsSourceAndVariantBytes(t *testing.T) {
}
}
func TestUsageBytesCountsMultiReferencedBlobOnce(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
content := bytes.Repeat([]byte{0xCC}, 1200)
hashOne := storeEvictionTestSource(t, cache,
"src.example.com", "/one.jpg", content)
hashTwo := storeEvictionTestSource(t, cache,
"src.example.com", "/two.jpg", content)
if hashOne != hashTwo {
t.Fatalf("identical content produced different hashes: %s vs %s",
hashOne, hashTwo)
}
usage, err := cache.UsageBytes(context.Background())
if err != nil {
t.Fatalf("UsageBytes failed: %v", err)
}
if usage != 1200 {
t.Errorf("UsageBytes = %d, want 1200 (deduplicated blob counted once)", usage)
}
}
func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
t.Parallel()
@@ -343,7 +321,7 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
cache, _ := newEvictionTestCache(t, limit)
now := time.Now()
keys := []VariantKey{
keys := []imagerequest.VariantKey{
testVariantKeyOne, testVariantKeyTwo,
testVariantKeyThree, testVariantKeyFour,
}
@@ -388,102 +366,15 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
assertNoDanglingReferences(t, cache)
}
func TestEvictionRemovesMultiReferencedBlobTogetherWithAllReferences(t *testing.T) {
t.Parallel()
const limit = 1000
cache, _ := newEvictionTestCache(t, limit)
now := time.Now()
// One 800-byte blob referenced by two source paths.
sharedContent := bytes.Repeat([]byte{0xDD}, 800)
sharedHash := storeEvictionTestSource(t, cache,
"src.example.com", "/a.jpg", sharedContent)
h := storeEvictionTestSource(t, cache,
"src.example.com", "/b.jpg", sharedContent)
if h != sharedHash {
t.Fatalf("identical content produced different hashes: %s vs %s",
h, sharedHash)
}
// A newer 600-byte blob referenced by one source path.
recentHash := storeEvictionTestSource(t, cache, "src.example.com", "/c.jpg",
bytes.Repeat([]byte{0xEE}, 600))
setSourceLastAccessed(t, cache, sharedHash, now.Add(-2*time.Hour))
setSourceLastAccessed(t, cache, recentHash, now.Add(-time.Minute))
err := cache.EvictToLimit(context.Background())
if err != nil {
t.Fatalf("EvictToLimit failed: %v", err)
}
usage, err := cache.UsageBytes(context.Background())
if err != nil {
t.Fatalf("UsageBytes failed: %v", err)
}
if usage > limit {
t.Errorf("usage after eviction = %d, want <= %d", usage, limit)
}
// The multi-referenced blob must be gone from disk, from
// source_content, and from BOTH source_metadata rows: references
// are removed together with the blob, never left dangling.
if cache.srcContent.Exists(sharedHash) {
t.Errorf("evicted blob %s still exists on disk", sharedHash)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM source_content WHERE content_hash = ?`, string(sharedHash),
); n != 0 {
t.Errorf("evicted blob %s still has %d source_content rows", sharedHash, n)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(sharedHash),
); n != 0 {
t.Errorf("evicted blob %s still has %d source_metadata references", sharedHash, n)
}
// The JSON metadata sidecars for both referencing paths must be
// removed along with the rows.
for _, path := range []string{"/a.jpg", "/b.jpg"} {
pathHash := HashPath(path + "?")
if cache.srcMetadata.Exists("src.example.com", pathHash) {
t.Errorf("metadata sidecar for %s must be removed with its row", path)
}
}
// The more recently used blob survives fully intact.
if !cache.srcContent.Exists(recentHash) {
t.Errorf("recently used blob %s must survive eviction", recentHash)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(recentHash),
); n != 1 {
t.Errorf("recently used blob %s has %d source_metadata rows, want 1",
recentHash, n)
}
assertNoDanglingReferences(t, cache)
}
func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
content := bytes.Repeat([]byte{0xDF}, 800)
hash := storeEvictionTestSource(t, cache, "src.example.com", "/a.jpg", content)
h := storeEvictionTestSource(t, cache, "src.example.com", "/b.jpg", content)
if h != hash {
t.Fatalf("identical content produced different hashes: %s vs %s", h, hash)
keys := []imagerequest.SourceKey{
storeEvictionTestSource(t, cache, "src.example.com", "/a.jpg", content),
storeEvictionTestSource(t, cache, "src.example.com", "/b.jpg", content),
}
storeEvictionTestVariant(t, cache, testVariantKeyOne,
@@ -494,14 +385,16 @@ func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
t.Fatalf("EvictToLimit failed: %v", err)
}
if !cache.srcContent.Exists(hash) {
t.Errorf("blob %s must not be evicted while usage is under the limit", hash)
}
for _, key := range keys {
if !cache.srcContent.Exists(key) {
t.Errorf("source %s must not be evicted while usage is under the limit", key)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(hash),
); n != 2 {
t.Errorf("blob %s has %d source_metadata rows, want 2", hash, n)
if n := countRows(t, cache,
`SELECT COUNT(*) FROM source_content WHERE source_key = ?`, string(key),
); n != 1 {
t.Errorf("source %s has %d source_content rows, want 1", key, n)
}
}
if !cache.variants.Exists(testVariantKeyOne) {
@@ -516,12 +409,16 @@ func TestZeroMaxBytesDisablesDiskCache(t *testing.T) {
cache, tmpDir := newEvictionTestCache(t, 0)
req := &ImageRequest{
SourceHost: "src.example.com",
SourcePath: "/a.jpg",
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: "src.example.com",
Path: "/a.jpg",
},
Transform: imagerequest.Transform{
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
assertDisabledCacheWritesAreNoOps(t, cache, req)
@@ -533,14 +430,14 @@ func TestZeroMaxBytesDisablesDiskCache(t *testing.T) {
// assertDisabledCacheWritesAreNoOps verifies that stores against a
// disabled cache report success without recording anything.
func assertDisabledCacheWritesAreNoOps(
t *testing.T, cache *Cache, req *ImageRequest,
t *testing.T, cache *Cache, req *imagerequest.Request,
) {
t.Helper()
ctx := t.Context()
err := cache.StoreVariant(
ctx, CacheKey(req), bytes.NewReader([]byte("data")), "image/webp",
ctx, imagerequest.CacheKey(req), bytes.NewReader([]byte("data")), "image/webp",
)
if err != nil {
t.Fatalf("StoreVariant on disabled cache must be a no-op, got error: %v", err)
@@ -553,20 +450,20 @@ func assertDisabledCacheWritesAreNoOps(
Headers: map[string][]string{},
}
hash, err := cache.StoreSource(ctx, req, bytes.NewReader([]byte("data")), result)
key, err := cache.StoreSource(ctx, req, bytes.NewReader([]byte("data")), result)
if err != nil {
t.Fatalf("StoreSource on disabled cache must be a no-op, got error: %v", err)
}
if hash != "" {
t.Errorf("StoreSource on disabled cache returned hash %q, want empty", hash)
if key != "" {
t.Errorf("StoreSource on disabled cache returned key %q, want empty", key)
}
}
// assertDisabledCacheReadsAlwaysMiss verifies that lookups against a
// disabled cache never report a hit.
func assertDisabledCacheReadsAlwaysMiss(
t *testing.T, cache *Cache, req *ImageRequest,
t *testing.T, cache *Cache, req *imagerequest.Request,
) {
t.Helper()
@@ -581,14 +478,14 @@ func assertDisabledCacheReadsAlwaysMiss(
t.Error("Lookup on disabled cache must always miss")
}
srcHash, srcType, err := cache.LookupSource(ctx, req)
key, srcType, err := cache.LookupSource(ctx, req)
if err != nil {
t.Fatalf("LookupSource on disabled cache failed: %v", err)
}
if srcHash != "" || srcType != "" {
if key != "" || srcType != "" {
t.Errorf("LookupSource on disabled cache = (%q, %q), want empty",
srcHash, srcType)
key, srcType)
}
}
@@ -609,10 +506,6 @@ func assertDisabledCacheTracksNothing(t *testing.T, cache *Cache) {
if n := countRows(t, cache, `SELECT COUNT(*) FROM source_content`); n != 0 {
t.Errorf("disabled cache wrote %d source_content rows, want 0", n)
}
if n := countRows(t, cache, `SELECT COUNT(*) FROM source_metadata`); n != 0 {
t.Errorf("disabled cache wrote %d source_metadata rows, want 0", n)
}
}
// assertDisabledCacheWroteNothingToDisk verifies that a disabled cache
@@ -661,7 +554,7 @@ func TestEvictionRunsUnderWritePressure(t *testing.T) {
cache.StartEviction(time.Hour)
defer func() { _ = cache.StopEviction(t.Context()) }()
keys := []VariantKey{
keys := []imagerequest.VariantKey{
testVariantKeyOne, testVariantKeyTwo, testVariantKeyThree,
}
fills := []byte{0x11, 0x12, 0x13}
@@ -715,7 +608,7 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
time.Sleep(10 * time.Millisecond)
}
keys := []VariantKey{
keys := []imagerequest.VariantKey{
testVariantKeyOne, testVariantKeyTwo, testVariantKeyThree,
}
fills := []byte{0x21, 0x22, 0x23}
@@ -1055,9 +948,14 @@ func TestStopEvictionInterruptsPassInProgress(t *testing.T) {
}
// TestEvictToLimitStopsAtNextCandidateOnceCancelled cancels the context
// while the oldest of three source blobs is being evicted, and checks that
// EvictToLimit then returns context.Canceled without evicting the other
// two or logging a warning for either of them.
// while the oldest of three sources is being evicted, and checks that
// EvictToLimit then returns context.Canceled, with that source gone and
// the other two kept, and logs no warning. The test database has one
// connection. When the deletion of the oldest source's row commits, a
// commit hook holds the deletion until the test waits for the
// connection, so the test takes it as soon as the deletion is done. The
// eviction of the next source then waits for the connection, and the
// test cancels the context while it waits.
func TestEvictToLimitStopsAtNextCandidateOnceCancelled(t *testing.T) {
t.Parallel()
@@ -1067,7 +965,7 @@ func TestEvictToLimitStopsAtNextCandidateOnceCancelled(t *testing.T) {
cache.log = slog.New(slog.NewJSONHandler(&logBuf, nil))
hashes := []ContentHash{
keys := []imagerequest.SourceKey{
storeEvictionTestSource(t, cache, "cancel.example.com", "/a.jpg",
bytes.Repeat([]byte{0x61}, 1000)),
storeEvictionTestSource(t, cache, "cancel.example.com", "/b.jpg",
@@ -1078,69 +976,153 @@ func TestEvictToLimitStopsAtNextCandidateOnceCancelled(t *testing.T) {
base := time.Now().Add(-time.Hour)
for i, hash := range hashes {
setSourceLastAccessed(t, cache, hash, base.Add(time.Duration(i)*time.Minute))
for i, key := range keys {
setSourceLastAccessed(t, cache, key, base.Add(time.Duration(i)*time.Minute))
}
waitsBefore := cache.db.Stats().WaitCount
oldestRowDeleted := make(chan struct{})
var oldestRowDeletedOnce sync.Once
setCommitHook(t, cache, func() int32 {
oldestRowDeletedOnce.Do(func() {
close(oldestRowDeleted)
deadline := time.Now().Add(5 * time.Second)
for cache.db.Stats().WaitCount == waitsBefore && time.Now().Before(deadline) {
time.Sleep(time.Millisecond)
}
})
return 0
})
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
cache.evictSourceBlobTestHook = func(ContentHash) { cancel() }
evicted := make(chan error, 1)
err := cache.EvictToLimit(ctx)
go func() { evicted <- cache.EvictToLimit(ctx) }()
<-oldestRowDeleted
conn, err := cache.db.Conn(t.Context())
if err != nil {
t.Fatalf("failed to take the database connection: %v", err)
}
defer func() { _ = conn.Close() }()
waitForConnectionWaits(t, cache.db, waitsBefore+2)
cancel()
err = <-evicted
t.Logf("EvictToLimit() error = %v", err)
t.Logf("log output: %s", logBuf.String())
// The checks below read the database, so the connection goes back.
_ = conn.Close()
setCommitHook(t, cache, nil)
if !errors.Is(err, context.Canceled) {
t.Errorf("EvictToLimit() error = %v, want context.Canceled", err)
}
if cache.srcContent.Exists(hashes[0]) {
t.Errorf("source blob %s, evicted when the context was cancelled, "+
"is still on disk", hashes[0])
}
for _, hash := range hashes[1:] {
if !cache.srcContent.Exists(hash) {
t.Errorf("source blob %s was evicted after the context was cancelled", hash)
}
}
assertOnlyOldestSourceEvicted(t, cache, keys)
if strings.Contains(logBuf.String(), `"level":"WARN"`) {
t.Errorf("EvictToLimit logged a warning after the context was cancelled")
}
}
// setCommitHook makes the SQLite driver call hook each time a write on
// the test database's one connection commits, or stops it when hook is
// nil.
func setCommitHook(t *testing.T, cache *Cache, hook sqlite.CommitHookFn) {
t.Helper()
conn, err := cache.db.Conn(t.Context())
if err != nil {
t.Fatalf("failed to take the database connection: %v", err)
}
defer func() { _ = conn.Close() }()
err = conn.Raw(func(driverConn any) error {
hooks, ok := driverConn.(sqlite.HookRegisterer)
if !ok {
t.Fatalf("the database connection is a %T, which takes no hooks",
driverConn)
}
hooks.RegisterCommitHook(hook)
return nil
})
if err != nil {
t.Fatalf("failed to set the commit hook: %v", err)
}
}
// assertOnlyOldestSourceEvicted checks that the first of keys has neither
// its file nor its row, and that every other one has both.
func assertOnlyOldestSourceEvicted(
t *testing.T, cache *Cache, keys []imagerequest.SourceKey,
) {
t.Helper()
if cache.srcContent.Exists(keys[0]) {
t.Errorf("source %s, evicted when the context was cancelled, "+
"is still on disk", keys[0])
}
for _, key := range keys[1:] {
if !cache.srcContent.Exists(key) {
t.Errorf("source %s was evicted after the context was cancelled", key)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM source_content WHERE source_key = ?`, string(key),
); n != 1 {
t.Errorf("source %s, kept, has %d source_content rows, want 1", key, n)
}
}
assertNoDanglingReferences(t, cache)
}
// TestStopEvictionReturnsWhenItsContextEnds pauses an eviction pass where
// cancellation cannot reach it, after a source blob's rows are deleted and
// before its file is removed, and checks that StopEviction returns its
// context's error when that context ends instead of waiting for the pass.
// Once the pass goes on, the goroutine exits and no row points at a
// missing file.
// TestStopEvictionReturnsWhenItsContextEnds pauses the startup
// reconciliation pass in the hook called after each of its reads, where
// cancellation cannot reach it, and checks that StopEviction returns an
// error wrapping its context's error when that context ends, instead of
// waiting for the pass. Once the pass goes on, the goroutine exits, and a
// second StopEviction returns nil.
func TestStopEvictionReturnsWhenItsContextEnds(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1)
cache, _ := newEvictionTestCache(t, 1<<30)
paused := make(chan struct{})
resume := make(chan struct{})
cache.evictSourceBlobTestHook = func(ContentHash) {
close(paused)
<-resume
}
var pauseOnce sync.Once
hash := storeEvictionTestSource(t, cache, "stop.example.com", "/a.jpg",
bytes.Repeat([]byte{0x61}, 1000))
cache.reconciliationReadTestHook = func(int) {
pauseOnce.Do(func() {
close(paused)
<-resume
})
}
cache.StartEviction(time.Hour)
select {
case <-paused:
case <-time.After(5 * time.Second):
t.Fatal("the eviction pass never reached the source blob")
t.Fatal("the reconciliation pass never made a read")
}
ctx, cancel := context.WithTimeout(t.Context(), 50*time.Millisecond)
@@ -1159,12 +1141,6 @@ func TestStopEvictionReturnsWhenItsContextEnds(t *testing.T) {
if err != nil {
t.Fatalf("second StopEviction() error = %v, want nil", err)
}
assertNoDanglingReferences(t, cache)
if cache.srcContent.Exists(hash) {
t.Errorf("source blob %s is still on disk after its rows were deleted", hash)
}
}
// TestReconciliationWalksStopOnceCancelled checks that both directory
@@ -1236,125 +1212,3 @@ func TestReconciliationPassLogsNoWarningOnceCancelled(t *testing.T) {
t.Errorf("runReconciliationPass logged a warning with a cancelled context")
}
}
// TestEvictSourceBlobExcludesConcurrentStoreOfIdenticalContent exercises
// the exact TOCTOU window between evictSourceBlob's row-deletion
// transaction commit and its content file unlink: a concurrent
// StoreSource for a different source path whose content hashes to the
// same value (real SHA-256 dedup, not a contrived case) must not be
// able to insert a fresh row referencing the file while eviction is
// mid-unlink, and must not lose its own store once eviction has fully
// released the content hash.
func TestEvictSourceBlobExcludesConcurrentStoreOfIdenticalContent(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
ctx := context.Background()
content := bytes.Repeat([]byte{0x55}, 400)
hash := storeEvictionTestSource(t, cache,
"race.example.com", "/first.jpg", content)
proceed := make(chan struct{})
storeAttempted := make(chan struct{})
cache.evictSourceBlobTestHook = func(gotHash ContentHash) {
if gotHash != hash {
t.Errorf("test hook invoked for hash %s, want %s", gotHash, hash)
}
close(storeAttempted)
<-proceed
}
evictDone := make(chan error, 1)
go func() {
evictDone <- cache.evictSourceBlob(ctx, hash)
}()
// Wait until eviction has committed its delete transaction and is
// paused (inside the test hook) immediately before unlinking the
// content file: exactly the window the review flagged.
<-storeAttempted
storeDone := storeIdenticalContentConcurrently(ctx, cache, content)
// The concurrent store must not be able to complete while eviction
// still holds the content hash (i.e. before the file is unlinked):
// if it could, it would insert a row referencing a file about to be
// removed out from under it.
select {
case err := <-storeDone:
t.Fatalf("StoreSource for identical content completed (err=%v) while eviction "+
"still held the content hash open between commit and unlink; the store and "+
"the evict of identical content are not mutually exclusive", err)
case <-time.After(200 * time.Millisecond):
// Expected: the store is blocked behind eviction's exclusion.
}
close(proceed)
err := <-evictDone
if err != nil {
t.Fatalf("evictSourceBlob failed: %v", err)
}
err = <-storeDone
if err != nil {
t.Fatalf("StoreSource failed: %v", err)
}
assertNoDanglingReferences(t, cache)
dupHash, _, err := cache.LookupSource(ctx, &ImageRequest{
SourceHost: "race.example.com",
SourcePath: "/dup.jpg",
})
if err != nil {
t.Fatalf("LookupSource failed: %v", err)
}
if dupHash == "" {
t.Fatal("re-stored blob was lost: the store legitimately ran after eviction " +
"released the content hash and must have recreated the file and row")
}
if !cache.srcContent.Exists(dupHash) {
t.Errorf("source_content/source_metadata references %s but its file is missing",
dupHash)
}
}
// storeIdenticalContentConcurrently starts a StoreSource for a second
// source path whose body hashes to the same content hash, returning the
// channel its error is delivered on.
func storeIdenticalContentConcurrently(
ctx context.Context, cache *Cache, content []byte,
) chan error {
storeDone := make(chan error, 1)
go func() {
req := &ImageRequest{
SourceHost: "race.example.com",
SourcePath: "/dup.jpg",
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
result := &httpfetcher.FetchResult{
StatusCode: 200,
ContentType: testContentTypeJPEG,
ContentLength: int64(len(content)),
Headers: map[string][]string{
testHeaderContentType: {testContentTypeJPEG},
},
}
_, err := cache.StoreSource(ctx, req, bytes.NewReader(content), result)
storeDone <- err
}()
return storeDone
}
+4 -126
View File
@@ -3,134 +3,12 @@ package imgcache
import (
"context"
"errors"
"io"
"net/url"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// ErrInvalidFitMode is returned when an invalid fit mode is provided.
var ErrInvalidFitMode = errors.New("invalid fit mode")
// ImageFormat represents supported output image formats.
type ImageFormat string
// Supported image output formats.
const (
FormatOriginal ImageFormat = "orig"
FormatJPEG ImageFormat = "jpeg"
FormatPNG ImageFormat = "png"
FormatWebP ImageFormat = "webp"
FormatAVIF ImageFormat = "avif"
FormatJXL ImageFormat = "jxl"
FormatGIF ImageFormat = "gif"
// FormatAuto stands for JPEG XL, AVIF, WebP or JPEG, chosen for each
// request from its Accept header once the URL's signature or token has
// been checked; it is never processed or cached as itself.
FormatAuto ImageFormat = "auto"
)
// Size represents requested image dimensions
type Size struct {
Width int
Height int
}
// OriginalSize returns true if this represents "keep original size"
func (s Size) OriginalSize() bool {
return s.Width == 0 && s.Height == 0
}
// FitMode represents how to fit image into requested dimensions.
type FitMode string
// Supported image fit modes.
const (
FitCover FitMode = "cover"
FitContain FitMode = "contain"
FitFill FitMode = "fill"
FitInside FitMode = "inside"
FitOutside FitMode = "outside"
)
// ValidateFitMode checks if the given fit mode is valid.
// Returns ErrInvalidFitMode for unrecognized fit modes.
func ValidateFitMode(fit FitMode) error {
switch fit {
case FitCover, FitContain, FitFill, FitInside, FitOutside, "":
return nil
default:
return ErrInvalidFitMode
}
}
// ValidateImageRequest checks a request's width and height with
// ValidateDimension and its fit mode with ValidateFitMode. Both the plain
// /v1/image/ route and the encrypted /v1/e/ route validate through this
// function so a request from either source enforces identical bounds.
func ValidateImageRequest(req *ImageRequest) error {
err := ValidateDimension("width", req.Size.Width)
if err != nil {
return err
}
err = ValidateDimension("height", req.Size.Height)
if err != nil {
return err
}
return ValidateFitMode(req.FitMode)
}
// ImageRequest represents a request for a processed image
type ImageRequest struct {
// SourceHost is the origin host (e.g., "cdn.example.com")
SourceHost string
// SourcePath is the path on the origin with its percent-escapes decoded,
// as url.URL's Path holds it (e.g., "/photos/my cat.jpg")
SourcePath string
// SourceQuery is the optional query string for the origin URL as the URL
// writes it, percent-escapes included, as url.URL's RawQuery holds it
SourceQuery string
// Size is the requested output dimensions
Size Size
// Format is the requested output format
Format ImageFormat
// Quality is the output quality (1-100) for lossy formats
Quality int
// FitMode is how to fit the image into requested dimensions
FitMode FitMode
// Signature is the HMAC signature for non-allowlisted hosts
Signature string
// Expires is when the URL expires: the exp of a signed URL, or the expiry
// of an encrypted URL; the zero time if it has none
Expires time.Time
// AllowHTTP indicates whether HTTP (non-TLS) is allowed for this request
AllowHTTP bool
}
// SourceURL returns the full upstream URL to fetch: the source path
// percent-encoded again, as net/url writes a path, so each source path is one
// upstream path, and the source query as it is, with the characters
// escapeQuery encodes percent-encoded. Uses http:// scheme when AllowHTTP is
// true, otherwise https://.
func (r *ImageRequest) SourceURL() string {
scheme := "https"
if r.AllowHTTP {
scheme = "http"
}
sourceURL := url.URL{
Scheme: scheme,
Host: r.SourceHost,
Path: r.SourcePath,
RawQuery: escapeQuery(r.SourceQuery),
}
return sourceURL.String()
}
// ImageResponse represents a processed image ready to serve
type ImageResponse struct {
// Content is the image data reader
@@ -162,10 +40,10 @@ const (
// ImageCache is the main interface for the image caching proxy
type ImageCache interface {
// Get retrieves a processed image, fetching and processing if necessary
Get(ctx context.Context, req *ImageRequest) (*ImageResponse, error)
Get(ctx context.Context, req *imagerequest.Request) (*ImageResponse, error)
// Warm pre-fetches and caches an image without returning it
Warm(ctx context.Context, req *ImageRequest) error
Warm(ctx context.Context, req *imagerequest.Request) error
// Stats returns cache statistics
Stats(ctx context.Context) (*CacheStats, error)
@@ -9,18 +9,23 @@ import (
"time"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// widthOnlyRequest asks for the test photo at width, its height scaled to
// keep the photo's aspect ratio.
func widthOnlyRequest(fixtures *TestFixtures, width int) *ImageRequest {
return &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: width},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
func widthOnlyRequest(fixtures *TestFixtures, width int) *imagerequest.Request {
return &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: width},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
}
@@ -36,7 +41,7 @@ func holdProcessingSlot(
input, feed := io.Pipe()
go func() {
_, _ = proc.Process(t.Context(), input, &imageprocessor.Request{})
_, _ = proc.Process(t.Context(), input, &imagerequest.Transform{})
}()
_, err := feed.Write([]byte{0})
@@ -72,11 +77,11 @@ func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
_ = resp.Content.Close()
contentHash, _, err := svc.cache.LookupSource(t.Context(),
sourceKey, _, err := svc.cache.LookupSource(t.Context(),
widthOnlyRequest(fixtures, 50))
if err != nil || contentHash == "" {
if err != nil || sourceKey == "" {
t.Fatalf("LookupSource() = %q, %v; want the cached source",
contentHash, err)
sourceKey, err)
}
release := holdProcessingSlot(t, svc.processor)
@@ -98,7 +103,7 @@ func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
// before waiting, it would have read it by now.
time.Sleep(100 * time.Millisecond)
err = os.WriteFile(svc.cache.srcContent.hashToPath(contentHash),
err = os.WriteFile(svc.cache.srcContent.hashToPath(sourceKey),
generateTestJPEG(t, 100, 50, color.RGBA{0, 0, 255, 255}), 0o600)
if err != nil {
t.Fatalf("failed to rewrite the cached source: %v", err)
+29 -17
View File
@@ -10,23 +10,31 @@ import (
"sync"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// webpRequest returns a request for a 100x100 WebP variant of path.
func webpRequest(path string) *ImageRequest {
return &ImageRequest{
SourceHost: testHostCDN,
SourcePath: path,
Size: Size{Width: 100, Height: 100},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
func webpRequest(path string) *imagerequest.Request {
return &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostCDN,
Path: path,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 100, Height: 100},
Format: imagerequest.FormatWebP,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
}
// assertVariantServed checks that GetVariant serves key with the given
// content and the image/webp content type storeEvictionTestVariant stores.
func assertVariantServed(t *testing.T, cache *Cache, key VariantKey, content []byte) {
func assertVariantServed(
t *testing.T, cache *Cache, key imagerequest.VariantKey, content []byte,
) {
t.Helper()
reader, size, contentType, err := cache.GetVariant(key)
@@ -57,7 +65,9 @@ func assertVariantServed(t *testing.T, cache *Cache, key VariantKey, content []b
// assertVariantNotFound checks that GetVariant refuses key with
// ErrNotFound.
func assertVariantNotFound(t *testing.T, cache *Cache, key VariantKey) {
func assertVariantNotFound(
t *testing.T, cache *Cache, key imagerequest.VariantKey,
) {
t.Helper()
reader, _, _, err := cache.GetVariant(key)
@@ -71,16 +81,18 @@ func assertVariantNotFound(t *testing.T, cache *Cache, key VariantKey) {
}
// assertLookupMisses checks that Lookup reports request as a miss.
func assertLookupMisses(t *testing.T, cache *Cache, request *ImageRequest) {
func assertLookupMisses(
t *testing.T, cache *Cache, request *imagerequest.Request,
) {
t.Helper()
lookup, err := cache.Lookup(t.Context(), request)
if err != nil {
t.Fatalf("Lookup(%s) error = %v", request.SourcePath, err)
t.Fatalf("Lookup(%s) error = %v", request.Source.Path, err)
}
if lookup.Hit {
t.Errorf("Lookup(%s) is a hit, want a miss", request.SourcePath)
t.Errorf("Lookup(%s) is a hit, want a miss", request.Source.Path)
}
}
@@ -251,8 +263,8 @@ func TestEvictedVariantIsNotServed(t *testing.T) {
oldRequest := webpRequest("/old.jpg")
newRequest := webpRequest("/new.jpg")
oldKey := CacheKey(oldRequest)
newKey := CacheKey(newRequest)
oldKey := imagerequest.CacheKey(oldRequest)
newKey := imagerequest.CacheKey(newRequest)
oldContent := bytes.Repeat([]byte{0x01}, 1000)
newContent := bytes.Repeat([]byte{0x02}, 1000)
@@ -280,7 +292,7 @@ func TestVariantDeletedFromDiskIsNotServed(t *testing.T) {
cache, _ := newEvictionTestCache(t, 1<<20)
request := webpRequest("/deleted.jpg")
key := CacheKey(request)
key := imagerequest.CacheKey(request)
content := []byte("webp variant bytes")
storeEvictionTestVariant(t, cache, key, content)
@@ -307,7 +319,7 @@ func TestConcurrentVariantStoreReadAndEvict(t *testing.T) {
for goroutine := range 8 {
wg.Go(func() {
key := VariantKey(fmt.Sprintf("aabbccdd01%02d", goroutine))
key := imagerequest.VariantKey(fmt.Sprintf("aabbccdd01%02d", goroutine))
content := []byte(key)
for range 20 {
@@ -6,6 +6,8 @@ import (
"strconv"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
func TestNegativeCache_StoreAndCheck(t *testing.T) {
@@ -23,9 +25,11 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
}
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostExample,
SourcePath: "/missing.jpg",
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostExample,
Path: "/missing.jpg",
},
}
// Initially should not be in negative cache
@@ -70,9 +74,11 @@ func TestNegativeCache_Expired(t *testing.T) {
}
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostExample,
SourcePath: "/expired.jpg",
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostExample,
Path: "/expired.jpg",
},
}
// Store a negative entry with very short TTL
@@ -106,20 +112,21 @@ func TestService_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
svc, fixtures := SetupTestService(t)
resp, err := svc.Get(t.Context(), photoVariant(fixtures, 85, FitCover))
resp, err := svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
_ = resp.Content.Close()
err = svc.cache.StoreNegative(t.Context(), photoVariant(fixtures, 85, FitCover),
err = svc.cache.StoreNegative(t.Context(),
photoVariant(fixtures, 85, imagerequest.FitCover),
httpStatusBadGateway, "upstream answered 503")
if err != nil {
t.Fatalf("StoreNegative() error = %v", err)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 85, FitCover))
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() of the cached variant error = %v", err)
}
@@ -131,7 +138,7 @@ func TestService_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
resp.CacheStatus, CacheHit)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, FitCover))
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
t.Logf("Get() of a variant made from the cached source: error = %v", err)
if err != nil {
@@ -140,18 +147,18 @@ func TestService_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
_ = resp.Content.Close()
contentHash, _, err := svc.cache.LookupSource(t.Context(),
photoVariant(fixtures, 85, FitCover))
if err != nil || contentHash == "" {
t.Fatalf("LookupSource() = %q, %v, want the cached source", contentHash, err)
sourceKey, _, err := svc.cache.LookupSource(t.Context(),
photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil || sourceKey == "" {
t.Fatalf("LookupSource() = %q, %v, want the cached source", sourceKey, err)
}
err = svc.cache.evictSourceBlob(t.Context(), contentHash)
err = svc.cache.evictSourceBlob(t.Context(), sourceKey)
if err != nil {
t.Fatalf("evictSourceBlob() error = %v", err)
}
_, err = svc.Get(t.Context(), photoVariant(fixtures, 60, FitCover))
_, err = svc.Get(t.Context(), photoVariant(fixtures, 60, imagerequest.FitCover))
t.Logf("Get() of a variant whose source was evicted: error = %v", err)
if !errors.Is(err, ErrNegativeCached) {
@@ -190,8 +197,8 @@ func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
t.Fatalf("failed to create cache: %v", err)
}
expired := &ImageRequest{
SourceHost: testHostExample, SourcePath: "/gone.jpg",
expired := &imagerequest.Request{
Source: imagerequest.Source{Host: testHostExample, Path: "/gone.jpg"},
}
err = cache.StoreNegative(t.Context(), expired, 404, "not found")
@@ -215,7 +222,7 @@ func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
cache.StartEviction(100 * time.Millisecond)
defer func() { _ = cache.StopEviction(t.Context()) }()
for negativeCacheRowCount(t, cache, expired.SourcePath) > 0 {
for negativeCacheRowCount(t, cache, expired.Source.Path) > 0 {
if time.Since(started) > 10*time.Second {
t.Fatal("the expired row is still in negative_cache")
}
@@ -2,31 +2,33 @@ package imgcache
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// getImage calls Get for req and returns the image it answered with and
// its cache status.
func getImage(t *testing.T, svc *Service, req *ImageRequest) ([]byte, CacheStatus) {
func getImage(
t *testing.T, svc *Service, req *imagerequest.Request,
) ([]byte, CacheStatus) {
t.Helper()
resp, err := svc.Get(t.Context(), req)
if err != nil {
t.Fatalf("Get() of quality %d error = %v", req.Quality, err)
t.Fatalf("Get() of quality %d error = %v", req.Transform.Quality, err)
}
defer func() { _ = resp.Content.Close() }()
body, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("reading the image of quality %d: %v", req.Quality, err)
t.Fatalf("reading the image of quality %d: %v", req.Transform.Quality, err)
}
return body, resp.CacheStatus
@@ -58,15 +60,17 @@ func truncateFile(t *testing.T, path string, size int64) {
}
// cachedSourcePath returns the path of the cached source file req uses.
func cachedSourcePath(t *testing.T, svc *Service, req *ImageRequest) string {
func cachedSourcePath(
t *testing.T, svc *Service, req *imagerequest.Request,
) string {
t.Helper()
contentHash, _, err := svc.cache.LookupSource(t.Context(), req)
if err != nil || contentHash == "" {
t.Fatalf("LookupSource() = %q, %v; want the cached source", contentHash, err)
sourceKey, _, err := svc.cache.LookupSource(t.Context(), req)
if err != nil || sourceKey == "" {
t.Fatalf("LookupSource() = %q, %v; want the cached source", sourceKey, err)
}
return svc.cache.srcContent.hashToPath(contentHash)
return svc.cache.srcContent.hashToPath(sourceKey)
}
// reconcileAsAtStartup runs the cache's reconciliation pass, as pixa does
@@ -113,16 +117,16 @@ func TestService_Get_FetchesAgainForEmptyCachedSource(t *testing.T) {
svc, fixtures := SetupTestService(t)
getImage(t, svc, photoVariant(fixtures, 85, FitCover))
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
truncateFile(t, path, 0)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
getImage(t, svc, photoVariant(fixtures, 40, FitCover))
getImage(t, svc, photoVariant(fixtures, 30, FitCover))
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
fetches := readCacheStatsCounters(t, svc.cache).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
@@ -149,16 +153,16 @@ func TestService_Get_FetchesAgainForPartialCachedSource(t *testing.T) {
svc, fixtures := SetupTestService(t)
getImage(t, svc, photoVariant(fixtures, 85, FitCover))
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
truncateFile(t, path, wholeSize/2)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
getImage(t, svc, photoVariant(fixtures, 40, FitCover))
getImage(t, svc, photoVariant(fixtures, 30, FitCover))
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
fetches := readCacheStatsCounters(t, svc.cache).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
@@ -182,11 +186,11 @@ func assertConvertedAgain(t *testing.T, size int64) {
t.Helper()
svc, fixtures := SetupTestService(t)
req := photoVariant(fixtures, 85, FitCover)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := svc.cache.variants.keyToPath(CacheKey(req))
path := svc.cache.variants.keyToPath(imagerequest.CacheKey(req))
truncateFile(t, path, size)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
@@ -310,10 +314,10 @@ func TestVariantStorage_StoreReplacesMetaFile(t *testing.T) {
}
}
// TestContentStorage_StoreHashedReplacesFileOfAnotherSize checks that a
// store over a file of another size at the content's path, such as a
// TestContentStorage_StoreReplacesFileOfAnotherSize checks that a
// store over a file of another size at the source's path, such as a
// partial file a crash left, replaces it with the content.
func TestContentStorage_StoreHashedReplacesFileOfAnotherSize(t *testing.T) {
func TestContentStorage_StoreReplacesFileOfAnotherSize(t *testing.T) {
t.Parallel()
storage, err := NewContentStorage(t.TempDir())
@@ -322,13 +326,12 @@ func TestContentStorage_StoreHashedReplacesFileOfAnotherSize(t *testing.T) {
}
content := []byte("the whole source image")
sum := sha256.Sum256(content)
hash := ContentHash(hex.EncodeToString(sum[:]))
path := storage.hashToPath(hash)
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
path := storage.hashToPath(key)
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
if err != nil {
t.Fatalf("creating the content's directory: %v", err)
t.Fatalf("creating the source's directory: %v", err)
}
err = os.WriteFile(path, content[:5], StorageFilePerm)
@@ -336,12 +339,12 @@ func TestContentStorage_StoreHashedReplacesFileOfAnotherSize(t *testing.T) {
t.Fatalf("writing a partial file: %v", err)
}
_, err = storage.StoreHashed(hash, content)
_, err = storage.Store(key, content)
if err != nil {
t.Fatalf("StoreHashed() error = %v", err)
t.Fatalf("Store() error = %v", err)
}
stored, err := storage.Load(hash)
stored, err := storage.Load(key)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
@@ -6,6 +6,8 @@ import (
"errors"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// pendingCountsWait is how long a test waits for the database's connection,
@@ -92,7 +94,7 @@ func TestService_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy(t *testing.T) {
deadline, _ := ctx.Deadline()
results := startGet(ctx, svc, photoVariant(fixtures, 85, FitCover))
results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
// The request has made its database reads by the time it fetches.
select {
+45 -71
View File
@@ -19,6 +19,7 @@ import (
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -139,7 +140,9 @@ var ErrProcessingTimedOut = errors.New(
// While its source URL is in the negative cache, a cached variant is still
// served and a variant not in the cache is still made from a cached source;
// only a request that needs the source fetched is refused.
func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, error) {
func (s *Service) Get(
ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) {
// Propagate AllowHTTP setting to the request
req.AllowHTTP = s.allowHTTP
@@ -186,7 +189,7 @@ func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, e
}
// Warm pre-fetches and caches an image without returning it.
func (s *Service) Warm(ctx context.Context, req *ImageRequest) error {
func (s *Service) Warm(ctx context.Context, req *imagerequest.Request) error {
_, err := s.Get(ctx, req)
return err
@@ -204,7 +207,7 @@ func (s *Service) WaitForProcessing(ctx context.Context) int {
}
// ValidateRequest validates the request signature if required.
func (s *Service) ValidateRequest(req *ImageRequest) error {
func (s *Service) ValidateRequest(req *imagerequest.Request) error {
// Check if host is allowed (no signature required)
sourceURL := req.SourceURL()
@@ -227,7 +230,7 @@ func (s *Service) ValidateRequest(req *ImageRequest) error {
}
// Signature required for non-allowed hosts
return s.signer.Verify(signatureRequest(req))
return s.signer.Verify(req)
}
// GenerateSignedURL generates a signed URL for the given request. The URL
@@ -236,26 +239,22 @@ func (s *Service) ValidateRequest(req *ImageRequest) error {
// set to 85 or cover, the values the route uses when a URL has no q or fit.
func (s *Service) GenerateSignedURL(
baseURL string,
req *ImageRequest,
req *imagerequest.Request,
ttl time.Duration,
) (string, error) {
if req.Quality == 0 {
req.Quality = 85
if req.Transform.Quality == 0 {
req.Transform.Quality = 85
}
if req.FitMode == "" {
req.FitMode = FitCover
if req.Transform.FitMode == "" {
req.Transform.FitMode = imagerequest.FitCover
}
sigReq := signatureRequest(req)
path, sig, exp := s.signer.GenerateSignedURL(sigReq, ttl)
// Propagate the generated signature and expiration back onto the request.
req.Expires = sigReq.Expires
req.Signature = sigReq.Signature
// Sets the request's Expires and Signature
path, sig, exp := s.signer.GenerateSignedURL(req, ttl)
return fmt.Sprintf("%s%s?sig=%s&exp=%d&q=%d&fit=%s",
baseURL, path, sig, exp, req.Quality, req.FitMode), nil
baseURL, path, sig, exp, req.Transform.Quality, req.Transform.FitMode), nil
}
// errPanicked is returned when processing a variant panicked.
@@ -268,14 +267,14 @@ var errPanicked = errors.New("panic while processing image")
// the first request's cancellation, so the others are still served if that
// client goes away, but keeps its deadline.
func (s *Service) processOrWait(
ctx context.Context, req *ImageRequest,
ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) {
// A request that has already ended starts no processing
if ctx.Err() != nil {
return nil, ctx.Err()
}
cacheKey := CacheKey(req)
cacheKey := imagerequest.CacheKey(req)
// Closed when this request's own function runs, which singleflight does
// only when no other request is processing the variant
@@ -293,7 +292,7 @@ func (s *Service) processOrWait(
recovered := recover()
if recovered != nil {
s.log.Error("panic while processing image",
"host", req.SourceHost, "path", req.SourcePath,
"host", req.Source.Host, "path", req.Source.Path,
"panic", recovered, "stack", string(debug.Stack()))
if hub := sentry.GetHubFromContext(ctx); hub != nil {
@@ -360,9 +359,9 @@ func withoutCancelKeepingDeadline(
// returns it with its size; nil if the cached data is unavailable, empty or
// exceeds maxResponseSize.
func (s *Service) loadCachedSource(
contentHash ContentHash,
sourceKey imagerequest.SourceKey,
) (io.ReadCloser, int64) {
reader, size, err := s.cache.GetSourceContent(contentHash)
reader, size, err := s.cache.GetSourceContent(sourceKey)
if err != nil {
s.log.Warn("failed to load cached source, fetching", "error", err)
@@ -373,7 +372,7 @@ func (s *Service) loadCachedSource(
_ = reader.Close()
s.log.Warn("cached source exceeds max response size, discarding",
"hash", contentHash,
"source_key", sourceKey,
"max_bytes", s.maxResponseSize,
)
@@ -393,11 +392,11 @@ func (s *Service) loadCachedSource(
// if available.
func (s *Service) processFromSourceOrFetch(
ctx context.Context,
req *ImageRequest,
cacheKey VariantKey,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
) (*processedVariant, error) {
// Check if we have cached source content
contentHash, _, err := s.cache.LookupSource(ctx, req)
sourceKey, _, err := s.cache.LookupSource(ctx, req)
if err != nil {
s.log.Warn("source lookup failed", "error", err)
}
@@ -407,9 +406,9 @@ func (s *Service) processFromSourceOrFetch(
sourceSize int64
)
if contentHash != "" {
s.log.Debug("using cached source", "hash", contentHash)
source, sourceSize = s.loadCachedSource(contentHash)
if sourceKey != "" {
s.log.Debug("using cached source", "source_key", sourceKey)
source, sourceSize = s.loadCachedSource(sourceKey)
}
// Fetch from upstream if we don't have source data or it's empty
@@ -431,8 +430,8 @@ func (s *Service) processFromSourceOrFetch(
// source URL is in the negative cache.
func (s *Service) fetchAndProcess(
ctx context.Context,
req *ImageRequest,
cacheKey VariantKey,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
) (*processedVariant, error) {
// Check negative cache - skip fetching for recently-failed URLs
negHit, err := s.cache.checkNegativeCache(ctx, req)
@@ -442,8 +441,8 @@ func (s *Service) fetchAndProcess(
if negHit {
s.log.Debug("negative cache hit",
"host", req.SourceHost,
"path", req.SourcePath,
"host", req.Source.Host,
"path", req.Source.Path,
)
return nil, fmt.Errorf("%w: %w", httpfetcher.ErrUpstreamError, ErrNegativeCached)
@@ -493,8 +492,8 @@ func (s *Service) fetchAndProcess(
// Log upstream fetch details
s.log.Info("upstream fetched",
"request_id", middleware.GetReqID(ctx),
"host", req.SourceHost,
"path", req.SourcePath,
"host", req.Source.Host,
"path", req.Source.Path,
"bytes", fetchBytes,
"fetch_ms", fetchResult.FetchDurationMs,
"rate", downloadRate,
@@ -510,7 +509,7 @@ func (s *Service) fetchAndProcess(
err = magic.ValidateMagicBytes(sourceData, fetchResult.ContentType)
if err != nil {
s.log.Warn("source refused", "request_id", middleware.GetReqID(ctx),
"host", req.SourceHost, "path", req.SourcePath,
"host", req.Source.Host, "path", req.Source.Path,
"content_type", fetchResult.ContentType, "error", err)
s.storeNegative(ctx, req, httpStatusUnsupportedMediaType, err)
@@ -533,12 +532,12 @@ func (s *Service) fetchAndProcess(
// failed with err, so the source is not fetched again until the entry
// expires. A failure to store the entry is only logged.
func (s *Service) storeNegative(
ctx context.Context, req *ImageRequest, statusCode int, err error,
ctx context.Context, req *imagerequest.Request, statusCode int, err error,
) {
storeErr := s.cache.StoreNegative(ctx, req, statusCode, err.Error())
if storeErr != nil {
s.log.Warn("failed to store negative cache entry",
"host", req.SourceHost, "path", req.SourcePath, "error", storeErr)
"host", req.Source.Host, "path", req.Source.Path, "error", storeErr)
}
}
@@ -554,24 +553,17 @@ type processedVariant struct {
// result.
func (s *Service) processAndStore(
ctx context.Context,
req *ImageRequest,
cacheKey VariantKey,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
source io.Reader,
fetchBytes int64,
) (*processedVariant, error) {
// Process the image
processStart := time.Now()
processReq := &imageprocessor.Request{
Size: imageprocessor.Size{Width: req.Size.Width, Height: req.Size.Height},
Format: imageprocessor.Format(req.Format),
Quality: req.Quality,
FitMode: imageprocessor.FitMode(req.FitMode),
}
// ctx has the deadline of the request doing the processing, which can
// end while a request waiting for its image has time left
processResult, err := s.processor.Process(ctx, source, processReq)
processResult, err := s.processor.Process(ctx, source, &req.Transform)
if errors.Is(err, context.DeadlineExceeded) {
return nil, fmt.Errorf("%w: %w", ErrProcessingTimedOut, err)
}
@@ -604,10 +596,10 @@ func (s *Service) processAndStore(
s.log.Info("image converted",
"request_id", middleware.GetReqID(ctx),
"host", req.SourceHost,
"path", req.SourcePath,
"host", req.Source.Host,
"path", req.Source.Path,
"src_format", processResult.InputFormat,
"dst_format", req.Format,
"dst_format", req.Transform.Format,
"src_bytes", fetchBytes,
"dst_bytes", outputSize,
"src_dimensions", fmt.Sprintf("%dx%d",
@@ -616,8 +608,8 @@ func (s *Service) processAndStore(
processResult.Width, processResult.Height),
"size_ratio", fmt.Sprintf("%.1f%%", sizePercent),
"convert_ms", processDuration.Milliseconds(),
"quality", req.Quality,
"fit", req.FitMode,
"quality", req.Transform.Quality,
"fit", req.Transform.FitMode,
)
// Store variant to cache
@@ -636,24 +628,6 @@ func (s *Service) processAndStore(
}, nil
}
// signatureRequest projects an ImageRequest onto the standalone
// signature.Request type used by the signature package. This keeps the
// import edge one-way: imgcache depends on signature, never the reverse.
func signatureRequest(req *ImageRequest) *signature.Request {
return &signature.Request{
SourceHost: req.SourceHost,
SourcePath: req.SourcePath,
SourceQuery: req.SourceQuery,
Width: req.Size.Width,
Height: req.Size.Height,
Format: string(req.Format),
Quality: req.Quality,
FitMode: string(req.FitMode),
Signature: req.Signature,
Expires: req.Expires,
}
}
// HTTP status codes for error responses.
const (
httpStatusBadGateway = 502
@@ -680,7 +654,7 @@ func extractStatusCode(err error) int {
const etagHashLength = 16
// formatETag formats a VariantKey as a quoted ETag value.
func formatETag(key VariantKey) string {
func formatETag(key imagerequest.VariantKey) string {
hash := string(key)
// Use first 16 characters of hash for a shorter but still unique ETag
if len(hash) > etagHashLength {
+230 -201
View File
@@ -8,6 +8,7 @@ import (
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -25,13 +26,17 @@ func TestService_Get_AllowlistedHost(t *testing.T) {
svc, fixtures := SetupTestService(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
resp, err := svc.Get(ctx, req)
@@ -61,13 +66,17 @@ func TestService_Get_NonAllowlistedHost_NoSignature(t *testing.T) {
svc, fixtures := SetupTestService(t, WithSigningKey("test-key"))
req := &ImageRequest{
SourceHost: fixtures.OtherHost,
SourcePath: testPathUpload,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.OtherHost,
Path: testPathUpload,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Should fail validation - not allowlisted and no signature
@@ -84,19 +93,23 @@ func TestService_Get_NonAllowlistedHost_ValidSignature(t *testing.T) {
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.OtherHost,
SourcePath: testPathUpload,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.OtherHost,
Path: testPathUpload,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Generate a valid signature
signer := signature.New(signingKey)
req.Expires = time.Now().Add(time.Hour)
req.Signature = signer.Sign(signatureRequest(req))
req.Signature = signer.Sign(req)
// Should pass validation
err := svc.ValidateRequest(req)
@@ -128,19 +141,23 @@ func TestService_Get_NonAllowlistedHost_ExpiredSignature(t *testing.T) {
signingKey := testSigningKey
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
req := &ImageRequest{
SourceHost: fixtures.OtherHost,
SourcePath: testPathUpload,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.OtherHost,
Path: testPathUpload,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Generate an expired signature
signer := signature.New(signingKey)
req.Expires = time.Now().Add(-time.Hour) // Already expired
req.Signature = signer.Sign(signatureRequest(req))
req.Signature = signer.Sign(req)
// Should fail validation
err := svc.ValidateRequest(req)
@@ -155,13 +172,17 @@ func TestService_Get_NonAllowlistedHost_InvalidSignature(t *testing.T) {
signingKey := testSigningKey
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
req := &ImageRequest{
SourceHost: fixtures.OtherHost,
SourcePath: testPathUpload,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.OtherHost,
Path: testPathUpload,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Set an invalid signature
@@ -191,16 +212,20 @@ func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
signer := signature.New(signingKey)
// Sign a request for "cdn.example.com"
signedReq := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
Expires: time.Now().Add(time.Hour),
signedReq := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostCDN,
Path: testPathCat,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: time.Now().Add(time.Hour),
}
signedReq.Signature = signer.Sign(signatureRequest(signedReq))
signedReq.Signature = signer.Sign(signedReq)
// The original request should pass validation
t.Run("exact host passes", func(t *testing.T) {
@@ -228,23 +253,22 @@ func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
t.Run(tt.name+" rejected", func(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: tt.host,
SourcePath: signedReq.SourcePath,
SourceQuery: signedReq.SourceQuery,
Size: signedReq.Size,
Format: signedReq.Format,
Quality: signedReq.Quality,
FitMode: signedReq.FitMode,
Expires: signedReq.Expires,
Signature: signedReq.Signature,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: tt.host,
Path: signedReq.Source.Path,
Query: signedReq.Source.Query,
},
Transform: signedReq.Transform,
Expires: signedReq.Expires,
Signature: signedReq.Signature,
}
err := svc.ValidateRequest(req)
if err == nil {
t.Errorf(
"ValidateRequest() should reject signature for host %q (signed for %q)",
tt.host, signedReq.SourceHost)
tt.host, signedReq.Source.Host)
}
})
}
@@ -280,13 +304,17 @@ func TestService_ValidateRequest_HostWithPortNeedsSignature(t *testing.T) {
t.Run(tt.host, func(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: tt.host,
SourcePath: testPathCat,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: tt.host,
Path: testPathCat,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
err := svc.ValidateRequest(req)
@@ -320,13 +348,17 @@ func TestService_ValidateRequest_UnbracketedIPv6PortNeedsSignature(
t.Run(host, func(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: host,
SourcePath: testPathCat,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: host,
Path: testPathCat,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
err := svc.ValidateRequest(req)
@@ -347,14 +379,9 @@ func TestService_Get_InvalidFile(t *testing.T) {
svc, fixtures := SetupTestService(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: "/images/fake.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
req := photoVariant(fixtures, 85, imagerequest.FitCover)
req.Source.Path = "/images/fake.jpg"
req.Transform.Size = imagerequest.Size{Width: 50, Height: 50}
// Should fail because magic bytes don't match
_, err := svc.Get(ctx, req)
@@ -369,14 +396,9 @@ func TestService_Get_NotFound(t *testing.T) {
svc, fixtures := SetupTestService(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: "/images/nonexistent.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
req := photoVariant(fixtures, 85, imagerequest.FitCover)
req.Source.Path = "/images/nonexistent.jpg"
req.Transform.Size = imagerequest.Size{Width: 50, Height: 50}
_, err := svc.Get(ctx, req)
if err == nil {
@@ -393,25 +415,25 @@ func TestService_Get_FormatConversion(t *testing.T) {
tests := []struct {
name string
sourcePath string
outFormat ImageFormat
outFormat imagerequest.Format
wantMIME string
}{
{
name: "JPEG to PNG",
sourcePath: testPathPhoto,
outFormat: FormatPNG,
outFormat: imagerequest.FormatPNG,
wantMIME: "image/png",
},
{
name: "PNG to JPEG",
sourcePath: "/images/logo.png",
outFormat: FormatJPEG,
outFormat: imagerequest.FormatJPEG,
wantMIME: testContentTypeJPEG,
},
{
name: "GIF to PNG",
sourcePath: "/images/animation.gif",
outFormat: FormatPNG,
outFormat: imagerequest.FormatPNG,
wantMIME: "image/png",
},
}
@@ -420,13 +442,12 @@ func TestService_Get_FormatConversion(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: tt.sourcePath,
Size: Size{Width: 50, Height: 50},
Format: tt.outFormat,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{Host: fixtures.GoodHost, Path: tt.sourcePath},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: tt.outFormat, Quality: 85, FitMode: imagerequest.FitCover,
},
}
resp, err := svc.Get(ctx, req)
@@ -474,13 +495,17 @@ func TestService_Get_Caching(t *testing.T) {
svc, fixtures := SetupTestService(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// First request - should be a cache miss
@@ -525,26 +550,22 @@ func TestService_Get_Caching(t *testing.T) {
// TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo caches a source and
// a variant of it and sets back two days every time pixa stored for them.
// For the source: fetched_at in source_metadata, fetched_at and
// last_accessed_at in source_content, the fetched_at in its metadata file,
// and the access and modification times of its file and of its metadata
// file. For the variant: created_at and last_accessed_at in variant_content,
// the created_at in its .meta file, and the access and modification times of
// its file and of its .meta file. It then checks that the variant is still
// served from the cache and that a new variant is made from the cached source
// without fetching it again.
// For the source: fetched_at and last_accessed_at in source_content, and the
// access and modification times of its file. For the variant: created_at and
// last_accessed_at in variant_content, the created_at in its .meta file, and
// the access and modification times of its file and of its .meta file. It
// then checks that the variant is still served from the cache and that a new
// variant is made from the cached source without fetching it again.
//
// Left at the present: the files' status change and birth times, which a
// program cannot set, and the times of the directories holding the files,
// which other entries share. The source's stored response headers are empty,
// as the mock fetcher sends none. expires_at, etag and last_modified in
// source_metadata and in the metadata file are never written, nor are the
// request_cache and output_content tables.
// which other entries share. The request_cache and output_content tables are
// never written.
func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
variant := photoVariant(fixtures, 85, FitCover)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
resp, err := svc.Get(t.Context(), variant)
if err != nil {
@@ -570,7 +591,7 @@ func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
}
// Another quality is a new variant, made from the cached source
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, FitCover))
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() of a new variant error = %v", err)
}
@@ -586,43 +607,23 @@ func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
// backdateSource sets every time pixa stored for req's cached source to when,
// as listed on TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo.
func backdateSource(
t *testing.T, cache *Cache, req *ImageRequest, when time.Time,
t *testing.T, cache *Cache, req *imagerequest.Request, when time.Time,
) {
t.Helper()
pathHash := HashPath(req.SourcePath + "?" + req.SourceQuery)
meta, err := cache.srcMetadata.Load(req.SourceHost, pathHash)
if err != nil {
t.Fatalf("reading the source's metadata file: %v", err)
}
meta.FetchedAt = when.Unix()
err = cache.srcMetadata.Store(req.SourceHost, pathHash, meta)
if err != nil {
t.Fatalf("writing the source's metadata file: %v", err)
}
key := req.Source.Key()
dbTime := when.UTC().Format(sqliteTimestampFormat)
updateOneRow(t, cache, `
UPDATE source_metadata SET fetched_at = ?
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, dbTime, req.SourceHost, req.SourcePath, req.SourceQuery)
updateOneRow(t, cache,
`UPDATE source_content SET fetched_at = ? WHERE content_hash = ?`,
dbTime, meta.ContentHash)
setSourceLastAccessed(t, cache, ContentHash(meta.ContentHash), when)
`UPDATE source_content SET fetched_at = ? WHERE source_key = ?`,
dbTime, string(key))
setSourceLastAccessed(t, cache, key, when)
for _, path := range []string{
cache.srcContent.hashToPath(ContentHash(meta.ContentHash)),
cache.srcMetadata.metaPath(req.SourceHost, pathHash),
} {
err = os.Chtimes(path, when, when)
if err != nil {
t.Fatalf("setting back the times of %s: %v", path, err)
}
path := cache.srcContent.hashToPath(key)
err := os.Chtimes(path, when, when)
if err != nil {
t.Fatalf("setting back the times of %s: %v", path, err)
}
}
@@ -630,11 +631,11 @@ func backdateSource(
// for to when, as listed on
// TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo.
func backdateVariant(
t *testing.T, cache *Cache, req *ImageRequest, when time.Time,
t *testing.T, cache *Cache, req *imagerequest.Request, when time.Time,
) {
t.Helper()
key := CacheKey(req)
key := imagerequest.CacheKey(req)
content, size, contentType, err := cache.variants.LoadWithMeta(key)
if err != nil || contentType == "" {
@@ -696,7 +697,7 @@ func TestService_Get_DifferentSizes(t *testing.T) {
ctx := context.Background()
// Request same image at different sizes
sizes := []Size{
sizes := []imagerequest.Size{
{Width: 25, Height: 25},
{Width: 50, Height: 50},
{Width: 75, Height: 75},
@@ -705,13 +706,17 @@ func TestService_Get_DifferentSizes(t *testing.T) {
responses := make([][]byte, 0, len(sizes))
for _, size := range sizes {
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: size,
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: size,
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
resp, err := svc.Get(ctx, req)
@@ -745,13 +750,17 @@ func TestService_ValidateRequest_NoSigningKey(t *testing.T) {
// Service with no signing key - all non-allowlisted requests should fail
svc, fixtures := SetupTestService(t, WithNoAllowlist())
req := &ImageRequest{
SourceHost: fixtures.OtherHost,
SourcePath: testPathUpload,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.OtherHost,
Path: testPathUpload,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
err := svc.ValidateRequest(req)
@@ -770,13 +779,17 @@ func TestService_Get_ContextCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel() // Cancel immediately
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
_, err := svc.Get(ctx, req)
@@ -791,13 +804,17 @@ func TestService_Get_ReturnsETag(t *testing.T) {
svc, fixtures := SetupTestService(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
resp, err := svc.Get(ctx, req)
@@ -824,13 +841,17 @@ func TestService_Get_ETagConsistency(t *testing.T) {
svc, fixtures := SetupTestService(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// First request
@@ -866,22 +887,30 @@ func TestService_Get_DifferentETagsForDifferentContent(t *testing.T) {
ctx := context.Background()
// Request same image at different sizes - should get different ETags
req1 := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 25, Height: 25},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req1 := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 25, Height: 25},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
req2 := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req2 := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
resp1, err := svc.Get(ctx, req1)
@@ -0,0 +1,146 @@
package imgcache
import (
"bytes"
"io"
"os"
"path/filepath"
"testing"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestStoreSource_StoresIdenticalBytesOfTwoURLsTwice stores the same bytes
// for two source URLs and checks that each URL has a source_content row of
// its own and a file of its own, named by its source key under
// cache/sources/<ab>/<cd>/.
func TestStoreSource_StoresIdenticalBytesOfTwoURLsTwice(t *testing.T) {
t.Parallel()
cache, stateDir := newEvictionTestCache(t, 1<<30)
content := bytes.Repeat([]byte{0x5A}, 400)
keys := map[string]imagerequest.SourceKey{
"/a.jpg": storeEvictionTestSource(t, cache, testHostCDN, "/a.jpg", content),
"/b.jpg": storeEvictionTestSource(t, cache, testHostCDN, "/b.jpg", content),
}
t.Logf("source keys: %v", keys)
if keys["/a.jpg"] == keys["/b.jpg"] {
t.Fatalf("both source URLs got the key %s", keys["/a.jpg"])
}
for sourcePath, key := range keys {
rows := countRows(t, cache, `
SELECT COUNT(*) FROM source_content
WHERE source_key = ? AND source_host = ? AND source_path = ?
`, string(key), testHostCDN, sourcePath)
if rows != 1 {
t.Errorf("%s has %d source_content rows, want 1", sourcePath, rows)
}
name := string(key)
file := filepath.Join(stateDir, "cache", "sources", name[0:2], name[2:4], name)
stored, err := os.ReadFile(file) //nolint:gosec // path in the test's state dir
if err != nil || !bytes.Equal(stored, content) {
t.Errorf("file of %s at %s holds %d bytes (error %v), want the %d stored",
sourcePath, file, len(stored), err, len(content))
}
}
usage, err := cache.UsageBytes(t.Context())
if err != nil {
t.Fatalf("UsageBytes() error = %v", err)
}
if usage != 800 {
t.Errorf("UsageBytes() = %d, want 800, the 400 bytes once for each URL", usage)
}
}
// TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed stores the same
// bytes for two source URLs and evicts the first. Its row and file must be
// gone, and the second must still be found and read from the cache.
func TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
content := bytes.Repeat([]byte{0x5B}, 400)
evicted := storeEvictionTestSource(t, cache, testHostCDN, "/evicted.jpg", content)
storeEvictionTestSource(t, cache, testHostCDN, "/kept.jpg", content)
err := cache.evictSourceBlob(t.Context(), evicted)
if err != nil {
t.Fatalf("evictSourceBlob() error = %v", err)
}
if cache.srcContent.Exists(evicted) {
t.Error("the evicted source's file is still on disk")
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM source_content WHERE source_key = ?`, string(evicted),
); n != 0 {
t.Errorf("the evicted source still has %d source_content rows", n)
}
kept, contentType, err := cache.LookupSource(t.Context(), &imagerequest.Request{
Source: imagerequest.Source{Host: testHostCDN, Path: "/kept.jpg"},
})
t.Logf("LookupSource() of the kept source = %q, %q, error %v", kept, contentType, err)
if err != nil || kept == "" {
t.Fatalf("LookupSource() of the kept source = %q, %v; want it cached", kept, err)
}
reader, _, err := cache.GetSourceContent(kept)
if err != nil {
t.Fatalf("GetSourceContent() of the kept source error = %v", err)
}
defer func() { _ = reader.Close() }()
served, err := io.ReadAll(reader)
if err != nil {
t.Fatalf("reading the kept source: %v", err)
}
if !bytes.Equal(served, content) {
t.Errorf("the kept source reads as %d bytes, want the %d stored",
len(served), len(content))
}
}
// TestStoreSource_FailedRowWriteLeavesNoFile drops source_content, so that
// StoreSource cannot write the source's row, and checks that the store fails
// and leaves no file under the source's key.
func TestStoreSource_FailedRowWriteLeavesNoFile(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
_, err := cache.db.ExecContext(t.Context(), `DROP TABLE source_content`)
if err != nil {
t.Fatalf("failed to drop source_content: %v", err)
}
req := &imagerequest.Request{
Source: imagerequest.Source{Host: testHostCDN, Path: testPathCat},
}
result := &httpfetcher.FetchResult{ContentType: testContentTypeJPEG}
key, err := cache.StoreSource(t.Context(), req,
bytes.NewReader([]byte("source bytes")), result)
t.Logf("StoreSource() = %q, error = %v", key, err)
if err == nil {
t.Error("StoreSource() error = nil, want the failed row write")
}
if cache.srcContent.Exists(req.Source.Key()) {
t.Error("the source's file is on disk although its row was not written")
}
}
+34 -21
View File
@@ -15,6 +15,7 @@ import (
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
)
func setupStatsTestDB(t *testing.T) *sql.DB {
@@ -319,13 +320,17 @@ func TestService_Get_CountsStats(t *testing.T) {
}
for _, step := range steps {
resp, err := svc.Get(t.Context(), &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: step.path,
Size: Size{Width: step.size, Height: step.size},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
resp, err := svc.Get(t.Context(), &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: step.path,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: step.size, Height: step.size},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
})
if (err != nil) != step.wantErr {
t.Fatalf("%s: Get() error = %v, want error %t", step.name, err, step.wantErr)
@@ -419,13 +424,17 @@ func TestService_Get_CountsInterruptedMisses(t *testing.T) {
svc.fetcher = upstream
resp, err := svc.Get(ctx, &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
resp, err := svc.Get(ctx, &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
})
t.Logf("Get() error = %v", err)
@@ -453,13 +462,17 @@ func TestService_Get_CountsHitAfterRequestEnds(t *testing.T) {
svc, fixtures := SetupTestService(t)
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: testPathPhoto,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// A first request caches the variant.
+44 -265
View File
@@ -1,8 +1,6 @@
package imgcache
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
@@ -10,8 +8,9 @@ import (
"log/slog"
"os"
"path/filepath"
"strings"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// Storage constants.
@@ -29,17 +28,8 @@ var (
ErrNotFound = errors.New("content not found")
)
// ContentHash is a SHA256 hash of file content (hex-encoded).
type ContentHash string
// VariantKey is a SHA256 hash identifying a specific image variant (hex-encoded).
type VariantKey string
// PathHash is a SHA256 hash of a URL path (hex-encoded).
type PathHash string
// ContentStorage handles content-addressable file storage.
// Files are stored at: <basedir>/<ab>/<cd>/<abcdef...sha256>
// ContentStorage handles source image file storage, one file per source
// URL. Files are stored at: <basedir>/<ab>/<cd>/<source key>
type ContentStorage struct {
baseDir string
}
@@ -54,38 +44,19 @@ func NewContentStorage(baseDir string) (*ContentStorage, error) {
return &ContentStorage{baseDir: baseDir}, nil
}
// Store writes content to storage and returns its SHA256 hash.
// The content is read fully into memory to compute the hash before writing.
func (s *ContentStorage) Store(r io.Reader) (ContentHash, int64, error) {
// Read all content to compute hash
data, err := io.ReadAll(r)
if err != nil {
return "", 0, fmt.Errorf("failed to read content: %w", err)
}
// Compute hash
h := sha256.Sum256(data)
hash := ContentHash(hex.EncodeToString(h[:]))
size := int64(len(data))
err = s.writeIfAbsent(hash, data)
if err != nil {
return "", 0, err
}
return hash, size, nil
}
// StoreHashed writes pre-hashed content to storage at the path derived
// from hash, without recomputing it. Callers that already know the
// hash before writing (e.g. because they must hold a hash-keyed lock
// across the whole store operation) use this instead of Store. Like
// Store, it is idempotent: a file of the content's size already at that
// path is left untouched, and a file of another size is replaced.
func (s *ContentStorage) StoreHashed(
hash ContentHash, data []byte,
// Store writes data with writeFileSynced to the path derived from key,
// replacing any file already there, and returns its size.
func (s *ContentStorage) Store(
key imagerequest.SourceKey, data []byte,
) (int64, error) {
err := s.writeIfAbsent(hash, data)
path := s.hashToPath(key)
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
if err != nil {
return 0, fmt.Errorf("failed to create directory: %w", err)
}
err = writeFileSynced(path, data)
if err != nil {
return 0, err
}
@@ -93,11 +64,11 @@ func (s *ContentStorage) StoreHashed(
return int64(len(data)), nil
}
// Load returns a reader for the content with the given hash.
func (s *ContentStorage) Load(hash ContentHash) (io.ReadCloser, error) {
path := s.hashToPath(hash)
// Load returns a reader for the content with the given key.
func (s *ContentStorage) Load(key imagerequest.SourceKey) (io.ReadCloser, error) {
path := s.hashToPath(key)
f, err := os.Open(path) //nolint:gosec // path derived from content hash
f, err := os.Open(path) //nolint:gosec // path derived from source key
if err != nil {
if os.IsNotExist(err) {
return nil, ErrNotFound
@@ -109,11 +80,13 @@ func (s *ContentStorage) Load(hash ContentHash) (io.ReadCloser, error) {
return f, nil
}
// LoadWithSize returns a reader and file size for the content with the given hash.
func (s *ContentStorage) LoadWithSize(hash ContentHash) (io.ReadCloser, int64, error) {
path := s.hashToPath(hash)
// LoadWithSize returns a reader and file size for the content with the given key.
func (s *ContentStorage) LoadWithSize(
key imagerequest.SourceKey,
) (io.ReadCloser, int64, error) {
path := s.hashToPath(key)
f, err := os.Open(path) //nolint:gosec // path derived from content hash
f, err := os.Open(path) //nolint:gosec // path derived from source key
if err != nil {
if os.IsNotExist(err) {
return nil, 0, ErrNotFound
@@ -132,9 +105,9 @@ func (s *ContentStorage) LoadWithSize(hash ContentHash) (io.ReadCloser, int64, e
return f, stat.Size(), nil
}
// Delete removes content with the given hash.
func (s *ContentStorage) Delete(hash ContentHash) error {
path := s.hashToPath(hash)
// Delete removes content with the given key.
func (s *ContentStorage) Delete(key imagerequest.SourceKey) error {
path := s.hashToPath(key)
err := os.Remove(path)
if err != nil && !os.IsNotExist(err) {
@@ -144,34 +117,14 @@ func (s *ContentStorage) Delete(hash ContentHash) error {
return nil
}
// Exists checks if content with the given hash exists.
func (s *ContentStorage) Exists(hash ContentHash) bool {
path := s.hashToPath(hash)
// Exists checks if content with the given key exists.
func (s *ContentStorage) Exists(key imagerequest.SourceKey) bool {
path := s.hashToPath(key)
_, err := os.Stat(path)
return err == nil
}
// writeIfAbsent writes data with writeFileSynced to the path derived
// from hash, unless a file of the data's size is already there.
func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error {
// Build path: <basedir>/<ab>/<cd>/<hash>
path := s.hashToPath(hash)
// A file of another size there is damaged, and is replaced
info, statErr := os.Stat(path)
if statErr == nil && info.Size() == int64(len(data)) {
return nil
}
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
if err != nil {
return fmt.Errorf("failed to create directory: %w", err)
}
return writeFileSynced(path, data)
}
// writeFileSynced writes data to path through a temp file in the same
// directory: the temp file is synced to disk, renamed to path, and the
// directory synced after, so that neither a concurrent reader nor a
@@ -246,9 +199,9 @@ func syncDirectory(dir string) error {
return nil
}
// hashToPath converts a hash to a file path: <basedir>/<ab>/<cd>/<hash>
func (s *ContentStorage) hashToPath(hash ContentHash) string {
h := string(hash)
// hashToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
func (s *ContentStorage) hashToPath(key imagerequest.SourceKey) string {
h := string(key)
if len(h) < MinHashLength {
return filepath.Clean(filepath.Join(s.baseDir, h))
}
@@ -256,185 +209,9 @@ func (s *ContentStorage) hashToPath(hash ContentHash) string {
return filepath.Clean(filepath.Join(s.baseDir, h[0:2], h[2:4], h))
}
// MetadataStorage handles JSON metadata file storage.
// Files are stored at: <basedir>/<hostname>/<path_hash>.json
type MetadataStorage struct {
baseDir string
}
// NewMetadataStorage creates a new metadata storage at the given base directory.
func NewMetadataStorage(baseDir string) (*MetadataStorage, error) {
err := os.MkdirAll(baseDir, StorageDirPerm)
if err != nil {
return nil, fmt.Errorf("failed to create metadata directory: %w", err)
}
return &MetadataStorage{baseDir: baseDir}, nil
}
// SourceMetadata represents cached metadata about a source URL.
//
//nolint:tagliatelle // stored metadata format uses snake_case
type SourceMetadata struct {
Host string `json:"host"`
Path string `json:"path"`
Query string `json:"query,omitempty"`
ContentHash string `json:"content_hash,omitempty"`
StatusCode int `json:"status_code"`
ContentType string `json:"content_type,omitempty"`
ContentLength int64 `json:"content_length,omitempty"`
ResponseHeaders map[string][]string `json:"response_headers,omitempty"`
FetchedAt int64 `json:"fetched_at"`
FetchDurationMs int64 `json:"fetch_duration_ms,omitempty"`
ExpiresAt int64 `json:"expires_at,omitempty"`
ETag string `json:"etag,omitempty"`
LastModified string `json:"last_modified,omitempty"`
RemoteAddr string `json:"remote_addr,omitempty"`
}
// Store writes metadata to storage.
func (s *MetadataStorage) Store(
host string, pathHash PathHash, meta *SourceMetadata,
) error {
path := s.metaPath(host, pathHash)
// Create directory structure
dir := filepath.Dir(path)
err := os.MkdirAll(dir, StorageDirPerm)
if err != nil {
return fmt.Errorf("failed to create directory: %w", err)
}
// Marshal to JSON
data, err := json.MarshalIndent(meta, "", " ")
if err != nil {
return fmt.Errorf("failed to marshal metadata: %w", err)
}
// Write to temp file first, then rename for atomicity
tmpFile, err := os.CreateTemp(dir, ".tmp-*.json")
if err != nil {
return fmt.Errorf("failed to create temp file: %w", err)
}
tmpPath := tmpFile.Name()
_, err = tmpFile.Write(data)
if err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to write metadata: %w", err)
}
err = tmpFile.Close()
if err != nil {
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to close temp file: %w", err)
}
// Atomic rename
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
if err != nil {
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to rename temp file: %w", err)
}
return nil
}
// Load reads metadata from storage.
func (s *MetadataStorage) Load(
host string, pathHash PathHash,
) (*SourceMetadata, error) {
path := s.metaPath(host, pathHash)
data, err := os.ReadFile(path) //nolint:gosec // path derived from host+hash
if err != nil {
if os.IsNotExist(err) {
return nil, ErrNotFound
}
return nil, fmt.Errorf("failed to read metadata: %w", err)
}
var meta SourceMetadata
err = json.Unmarshal(data, &meta)
if err != nil {
return nil, fmt.Errorf("failed to unmarshal metadata: %w", err)
}
return &meta, nil
}
// Delete removes metadata for the given host and path hash.
func (s *MetadataStorage) Delete(host string, pathHash PathHash) error {
path := s.metaPath(host, pathHash)
err := os.Remove(path)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("failed to delete metadata: %w", err)
}
return nil
}
// Exists checks if metadata exists for the given host and path hash.
func (s *MetadataStorage) Exists(host string, pathHash PathHash) bool {
path := s.metaPath(host, pathHash)
_, err := os.Stat(path)
return err == nil
}
// metaPath returns the file path for metadata: <basedir>/<host>/<path_hash>.json
func (s *MetadataStorage) metaPath(host string, pathHash PathHash) string {
return filepath.Clean(filepath.Join(s.baseDir, host, string(pathHash)+".json"))
}
// HashPath computes the SHA256 hash of a path string.
func HashPath(path string) PathHash {
h := sha256.Sum256([]byte(path))
return PathHash(hex.EncodeToString(h[:]))
}
// CacheKey generates a unique key for a request variant.
// Format: sha256(host:path:query:width:height:format:quality:fit_mode)
// The path and query go through escapePercentAndColon, so that a ":" in
// either cannot be taken for the one between them.
func CacheKey(req *ImageRequest) VariantKey {
data := fmt.Sprintf("%s:%s:%s:%d:%d:%s:%d:%s",
req.SourceHost,
escapePercentAndColon(req.SourcePath),
escapePercentAndColon(req.SourceQuery),
req.Size.Width,
req.Size.Height,
req.Format,
req.Quality,
req.FitMode,
)
h := sha256.Sum256([]byte(data))
return VariantKey(hex.EncodeToString(h[:]))
}
// escapePercentAndColon returns s with each "%" written as "%25" and each
// ":" as "%3A", so that the result holds no ":".
func escapePercentAndColon(s string) string {
s = strings.ReplaceAll(s, "%", "%25")
return strings.ReplaceAll(s, ":", "%3A")
}
// VariantStorage handles key-based file storage for processed image variants.
// Files are stored at: <basedir>/<ab>/<cd>/<cache_key>
// Metadata is stored at: <basedir>/<ab>/<cd>/<cache_key>.meta
// Unlike ContentStorage, the key is provided by the caller (not computed from content).
type VariantStorage struct {
baseDir string
log *slog.Logger
@@ -464,7 +241,7 @@ func NewVariantStorage(baseDir string, log *slog.Logger) (*VariantStorage, error
// with writeFileSynced. The .meta file is written first, so that a crash
// cannot leave the content in place without it.
func (s *VariantStorage) Store(
key VariantKey, r io.Reader, contentType string,
key imagerequest.VariantKey, r io.Reader, contentType string,
) (int64, error) {
data, err := io.ReadAll(r)
if err != nil {
@@ -508,7 +285,7 @@ func (s *VariantStorage) Store(
}
// Load returns a reader for the content at the given key.
func (s *VariantStorage) Load(key VariantKey) (io.ReadCloser, error) {
func (s *VariantStorage) Load(key imagerequest.VariantKey) (io.ReadCloser, error) {
path := s.keyToPath(key)
f, err := os.Open(path) //nolint:gosec // path derived from cache key
@@ -525,7 +302,9 @@ func (s *VariantStorage) Load(key VariantKey) (io.ReadCloser, error) {
// LoadWithSize returns a reader and file size for the content at the
// given key.
func (s *VariantStorage) LoadWithSize(key VariantKey) (io.ReadCloser, int64, error) {
func (s *VariantStorage) LoadWithSize(
key imagerequest.VariantKey,
) (io.ReadCloser, int64, error) {
path := s.keyToPath(key)
f, err := os.Open(path) //nolint:gosec // path derived from cache key
@@ -551,7 +330,7 @@ func (s *VariantStorage) LoadWithSize(key VariantKey) (io.ReadCloser, int64, err
// the given key. The content type is read from the .meta file, and is
// empty when that file is missing or unreadable.
func (s *VariantStorage) LoadWithMeta(
key VariantKey,
key imagerequest.VariantKey,
) (io.ReadCloser, int64, string, error) {
f, size, err := s.LoadWithSize(key)
if err != nil {
@@ -574,7 +353,7 @@ func (s *VariantStorage) LoadWithMeta(
}
// Exists checks if content exists at the given key.
func (s *VariantStorage) Exists(key VariantKey) bool {
func (s *VariantStorage) Exists(key imagerequest.VariantKey) bool {
path := s.keyToPath(key)
_, err := os.Stat(path)
@@ -583,7 +362,7 @@ func (s *VariantStorage) Exists(key VariantKey) bool {
// Delete removes the content at the given key together with its .meta
// sidecar file. A missing file is not an error.
func (s *VariantStorage) Delete(key VariantKey) error {
func (s *VariantStorage) Delete(key imagerequest.VariantKey) error {
path := s.keyToPath(key)
err := os.Remove(path)
@@ -602,7 +381,7 @@ func (s *VariantStorage) Delete(key VariantKey) error {
}
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
func (s *VariantStorage) keyToPath(key VariantKey) string {
func (s *VariantStorage) keyToPath(key imagerequest.VariantKey) string {
k := string(key)
if len(k) < MinHashLength {
return filepath.Join(s.baseDir, k)
+11 -426
View File
@@ -3,94 +3,14 @@ package imgcache
import (
"bytes"
"errors"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
func TestContentStorage_StoreAndLoad(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
content := []byte("hello world")
hash, size, err := storage.Store(bytes.NewReader(content))
if err != nil {
t.Fatalf("Store() error = %v", err)
}
if size != int64(len(content)) {
t.Errorf("Store() size = %d, want %d", size, len(content))
}
if hash == "" {
t.Error("Store() returned empty hash")
}
// Verify file exists at expected path
hashStr := string(hash)
expectedPath := filepath.Join(tmpDir, hashStr[0:2], hashStr[2:4], hashStr)
_, err = os.Stat(expectedPath)
if err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
// Load and verify content
r, err := storage.Load(hash)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
defer func() { _ = r.Close() }()
loaded, err := io.ReadAll(r)
if err != nil {
t.Fatalf("ReadAll() error = %v", err)
}
if !bytes.Equal(loaded, content) {
t.Errorf("Load() content = %q, want %q", loaded, content)
}
}
func TestContentStorage_StoreIdempotent(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
content := []byte("same content")
hash1, _, err := storage.Store(bytes.NewReader(content))
if err != nil {
t.Fatalf("Store() first error = %v", err)
}
hash2, _, err := storage.Store(bytes.NewReader(content))
if err != nil {
t.Fatalf("Store() second error = %v", err)
}
if hash1 != hash2 {
t.Errorf("Store() hashes differ: %s vs %s", hash1, hash2)
}
}
func TestContentStorage_LoadNotFound(t *testing.T) {
t.Parallel()
@@ -101,43 +21,12 @@ func TestContentStorage_LoadNotFound(t *testing.T) {
t.Fatalf("NewContentStorage() error = %v", err)
}
_, err = storage.Load(ContentHash("nonexistent"))
_, err = storage.Load(imagerequest.SourceKey("nonexistent"))
if !errors.Is(err, ErrNotFound) {
t.Errorf("Load() error = %v, want ErrNotFound", err)
}
}
func TestContentStorage_Delete(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
content := []byte("to be deleted")
hash, _, err := storage.Store(bytes.NewReader(content))
if err != nil {
t.Fatalf("Store() error = %v", err)
}
if !storage.Exists(hash) {
t.Error("Exists() = false, want true")
}
err = storage.Delete(hash)
if err != nil {
t.Fatalf("Delete() error = %v", err)
}
if storage.Exists(hash) {
t.Error("Exists() = true after delete, want false")
}
}
func TestContentStorage_DeleteNonexistent(t *testing.T) {
t.Parallel()
@@ -149,320 +38,12 @@ func TestContentStorage_DeleteNonexistent(t *testing.T) {
}
// Should not error
err = storage.Delete(ContentHash("nonexistent"))
err = storage.Delete(imagerequest.SourceKey("nonexistent"))
if err != nil {
t.Errorf("Delete() error = %v, want nil", err)
}
}
func TestContentStorage_HashToPath(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
// Test by storing and verifying the resulting path structure
content := []byte("test content for path verification")
hash, _, err := storage.Store(bytes.NewReader(content))
if err != nil {
t.Fatalf("Store() error = %v", err)
}
hashStr := string(hash)
expectedPath := filepath.Join(tmpDir, hashStr[0:2], hashStr[2:4], hashStr)
_, err = os.Stat(expectedPath)
if err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
}
func TestMetadataStorage_StoreAndLoad(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewMetadataStorage(tmpDir)
if err != nil {
t.Fatalf("NewMetadataStorage() error = %v", err)
}
meta := &SourceMetadata{
Host: testHostCDN,
Path: testPathCat,
ContentHash: "abc123",
StatusCode: 200,
ContentType: testContentTypeJPEG,
FetchedAt: 1704067200,
ETag: `"etag123"`,
}
pathHash := HashPath(testPathCat)
err = storage.Store(testHostCDN, pathHash, meta)
if err != nil {
t.Fatalf("Store() error = %v", err)
}
// Verify file exists at expected path
expectedPath := filepath.Join(tmpDir, testHostCDN, string(pathHash)+".json")
_, err = os.Stat(expectedPath)
if err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
// Load and verify
loaded, err := storage.Load(testHostCDN, pathHash)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if loaded.Host != meta.Host {
t.Errorf("Host = %q, want %q", loaded.Host, meta.Host)
}
if loaded.Path != meta.Path {
t.Errorf("Path = %q, want %q", loaded.Path, meta.Path)
}
if loaded.ContentHash != meta.ContentHash {
t.Errorf("ContentHash = %q, want %q", loaded.ContentHash, meta.ContentHash)
}
if loaded.StatusCode != meta.StatusCode {
t.Errorf("StatusCode = %d, want %d", loaded.StatusCode, meta.StatusCode)
}
if loaded.ETag != meta.ETag {
t.Errorf("ETag = %q, want %q", loaded.ETag, meta.ETag)
}
}
func TestMetadataStorage_LoadNotFound(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewMetadataStorage(tmpDir)
if err != nil {
t.Fatalf("NewMetadataStorage() error = %v", err)
}
_, err = storage.Load(testHostExample, PathHash("nonexistent"))
if !errors.Is(err, ErrNotFound) {
t.Errorf("Load() error = %v, want ErrNotFound", err)
}
}
func TestMetadataStorage_Delete(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewMetadataStorage(tmpDir)
if err != nil {
t.Fatalf("NewMetadataStorage() error = %v", err)
}
meta := &SourceMetadata{
Host: testHostExample,
Path: "/test.jpg",
StatusCode: 200,
}
pathHash := HashPath("/test.jpg")
err = storage.Store(testHostExample, pathHash, meta)
if err != nil {
t.Fatalf("Store() error = %v", err)
}
if !storage.Exists(testHostExample, pathHash) {
t.Error("Exists() = false, want true")
}
err = storage.Delete(testHostExample, pathHash)
if err != nil {
t.Fatalf("Delete() error = %v", err)
}
if storage.Exists(testHostExample, pathHash) {
t.Error("Exists() = true after delete, want false")
}
}
func TestHashPath(t *testing.T) {
t.Parallel()
// Same input should produce same hash
hash1 := HashPath(testPathCat)
hash2 := HashPath(testPathCat)
if hash1 != hash2 {
t.Errorf("HashPath() not deterministic: %s vs %s", hash1, hash2)
}
// Different input should produce different hash
hash3 := HashPath("/photos/dog.jpg")
if hash1 == hash3 {
t.Error("HashPath() produced same hash for different inputs")
}
// Hash should be 64 hex chars (256 bits)
if len(string(hash1)) != 64 {
t.Errorf("HashPath() length = %d, want 64", len(string(hash1)))
}
}
func TestCacheKey(t *testing.T) {
t.Parallel()
req1 := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceQuery: "",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
}
req2 := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceQuery: "",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
}
// Same request should produce same key
key1 := CacheKey(req1)
key2 := CacheKey(req2)
if key1 != key2 {
t.Errorf("CacheKey() not deterministic: %s vs %s", key1, key2)
}
// Key should be 64 hex chars
if len(string(key1)) != 64 {
t.Errorf("CacheKey() length = %d, want 64", len(string(key1)))
}
// Different size should produce different key
req3 := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceQuery: "",
Size: Size{Width: 400, Height: 300}, // Different size
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
}
key3 := CacheKey(req3)
if key1 == key3 {
t.Error("CacheKey() produced same key for different sizes")
}
// Different format should produce different key
req4 := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceQuery: "",
Size: Size{Width: 800, Height: 600},
Format: FormatPNG, // Different format
Quality: 85,
FitMode: FitCover,
}
key4 := CacheKey(req4)
if key1 == key4 {
t.Error("CacheKey() produced same key for different formats")
}
// Different quality should produce different key
req5 := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceQuery: "",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 50, // Different quality
FitMode: FitCover,
}
key5 := CacheKey(req5)
if key1 == key5 {
t.Error("CacheKey() produced same key for different quality")
}
}
// TestCacheKey_ColonOrPercentInPathOrQuery checks that two different source
// URLs whose path or query holds ":" or "%" get different keys.
func TestCacheKey_ColonOrPercentInPathOrQuery(t *testing.T) {
t.Parallel()
cases := []struct {
name string
path1, query1 string
path2, query2 string
}{
{
name: "colon moved from path to query",
path1: "/a:b", query1: "",
path2: "/a", query2: "b:",
},
{
name: "colon in path and its percent-encoding",
path1: "/a:b", query1: "",
path2: "/a%3Ab", query2: "",
},
}
for _, tt := range cases {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
key1 := CacheKey(&ImageRequest{
SourceHost: testHostCDN,
SourcePath: tt.path1,
SourceQuery: tt.query1,
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
})
key2 := CacheKey(&ImageRequest{
SourceHost: testHostCDN,
SourcePath: tt.path2,
SourceQuery: tt.query2,
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
})
t.Logf("CacheKey() of path %q query %q = %s", tt.path1, tt.query1, key1)
t.Logf("CacheKey() of path %q query %q = %s", tt.path2, tt.query2, key2)
if key1 == key2 {
t.Errorf("CacheKey() gave path %q query %q and path %q query %q "+
"the same key", tt.path1, tt.query1, tt.path2, tt.query2)
}
})
}
}
// TestVariantStorage_StoreLogsFailedMetaWrite verifies that a .meta write
// that fails is logged at warn and the store still succeeds.
func TestVariantStorage_StoreLogsFailedMetaWrite(t *testing.T) {
@@ -476,7 +57,9 @@ func TestVariantStorage_StoreLogsFailedMetaWrite(t *testing.T) {
t.Fatalf("NewVariantStorage() error = %v", err)
}
key := CacheKey(&ImageRequest{SourceHost: testHostCDN, SourcePath: testPathCat})
key := imagerequest.CacheKey(&imagerequest.Request{
Source: imagerequest.Source{Host: testHostCDN, Path: testPathCat},
})
// A directory where the .meta file goes makes the .meta write fail.
err = os.MkdirAll(storage.keyToPath(key)+".meta", StorageDirPerm)
@@ -497,7 +80,7 @@ func TestVariantStorage_StoreLogsFailedMetaWrite(t *testing.T) {
// storeTestVariant stores one variant, with its .meta file, in a new
// VariantStorage and returns the storage and the variant's key.
func storeTestVariant(t *testing.T) (*VariantStorage, VariantKey) {
func storeTestVariant(t *testing.T) (*VariantStorage, imagerequest.VariantKey) {
t.Helper()
storage, err := NewVariantStorage(t.TempDir(), slog.New(slog.DiscardHandler))
@@ -505,7 +88,9 @@ func storeTestVariant(t *testing.T) (*VariantStorage, VariantKey) {
t.Fatalf("NewVariantStorage() error = %v", err)
}
key := CacheKey(&ImageRequest{SourceHost: testHostCDN, SourcePath: testPathCat})
key := imagerequest.CacheKey(&imagerequest.Request{
Source: imagerequest.Source{Host: testHostCDN, Path: testPathCat},
})
_, err = storage.Store(key, bytes.NewReader([]byte("variant data")), "image/webp")
if err != nil {
@@ -9,7 +9,7 @@ import (
"testing"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProxyRefusesComputedSideAboveMaximum requests sizes of a 3000x600
@@ -70,9 +70,12 @@ func TestImageProxyRefusesComputedSideAboveMaximum(t *testing.T) {
}
for _, tt := range tests {
token, err := generator.Generate(&encurl.Payload{
SourceHost: upstreamHost, SourcePath: tt.path,
Width: tt.width, Height: tt.height, Format: imgcache.FormatJPEG,
token, err := generator.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: upstreamHost, Path: tt.path},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: tt.width, Height: tt.height},
Format: imagerequest.FormatJPEG,
},
})
if err != nil {
t.Fatalf("encurl Generate: %v", err)
+8 -5
View File
@@ -7,7 +7,7 @@ import (
"testing"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestImageProxyRefusesFitOutsideAboveMaximum requests sizes with fit=outside
@@ -59,10 +59,13 @@ func TestImageProxyRefusesFitOutsideAboveMaximum(t *testing.T) {
}
for _, tt := range tests {
token, err := generator.Generate(&encurl.Payload{
SourceHost: upstreamHost, SourcePath: tt.path,
Width: tt.width, Height: tt.height, Format: imgcache.FormatJPEG,
FitMode: imgcache.FitOutside,
token, err := generator.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: upstreamHost, Path: tt.path},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: tt.width, Height: tt.height},
Format: imagerequest.FormatJPEG,
FitMode: imagerequest.FitOutside,
},
})
if err != nil {
t.Fatalf("encurl Generate: %v", err)
+7 -21
View File
@@ -3,9 +3,7 @@ package server
import (
"bytes"
"context"
"crypto/sha256"
"database/sql"
"encoding/hex"
"image"
"image/color"
"image/jpeg"
@@ -30,6 +28,7 @@ import (
"sneak.berlin/go/pixa/internal/handlers"
"sneak.berlin/go/pixa/internal/healthcheck"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/logger"
"sneak.berlin/go/pixa/internal/middleware"
)
@@ -254,39 +253,26 @@ func getImage(t *testing.T, s *Server, target string) *httptest.ResponseRecorder
return rec
}
// checkSourceCached checks that source is stored under its SHA-256 in
// cache/sources, recorded in source_content, and that the source URL's row in
// source_metadata points at it.
// checkSourceCached checks that source is stored in cache/sources under the
// source key of upstreamHost and /photo.png, and recorded under that key in
// source_content.
func checkSourceCached(t *testing.T, db *sql.DB, stateDir string, source []byte) {
t.Helper()
sum := sha256.Sum256(source)
hash := hex.EncodeToString(sum[:])
key := string(imagerequest.Source{Host: upstreamHost, Path: "/photo.png"}.Key())
checkFile(t, filepath.Join(stateDir, "cache", "sources", hash[0:2], hash[2:4], hash),
checkFile(t, filepath.Join(stateDir, "cache", "sources", key[0:2], key[2:4], key),
source)
var rows int
err := db.QueryRowContext(t.Context(),
"SELECT COUNT(*) FROM source_content WHERE content_hash = ?", hash,
"SELECT COUNT(*) FROM source_content WHERE source_key = ?", key,
).Scan(&rows)
if err != nil || rows != 1 {
t.Errorf("source_content rows for the source = %d (error %v), want 1",
rows, err)
}
var metadataHash string
err = db.QueryRowContext(t.Context(),
`SELECT content_hash FROM source_metadata
WHERE source_host = ? AND source_path = ?`,
upstreamHost, "/photo.png",
).Scan(&metadataHash)
if err != nil || metadataHash != hash {
t.Errorf("source_metadata content_hash = %q (error %v), want %q",
metadataHash, err, hash)
}
}
// checkVariantCached checks that the converted image served is recorded in
+14 -9
View File
@@ -14,6 +14,7 @@ import (
"time"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/session"
"sneak.berlin/go/pixa/internal/signature"
@@ -513,12 +514,16 @@ func signedImageURL(t *testing.T, sourceHost string) string {
t.Helper()
path, sig, exp := signature.New(testSigningKey).GenerateSignedURL(
&signature.Request{
SourceHost: sourceHost,
SourcePath: ipv6SourcePath,
Format: string(imgcache.FormatPNG),
Quality: 85,
FitMode: string(imgcache.FitCover),
&imagerequest.Request{
Source: imagerequest.Source{
Host: sourceHost,
Path: ipv6SourcePath,
},
Transform: imagerequest.Transform{
Format: imagerequest.FormatPNG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}, time.Hour)
return path + "?sig=" + sig + "&exp=" + strconv.FormatInt(exp, 10)
@@ -534,9 +539,9 @@ func encryptedImageURL(t *testing.T, sourceHost string) string {
t.Fatalf("encurl.NewGenerator: %v", err)
}
token, err := generator.Generate(&encurl.Payload{
SourceHost: sourceHost, SourcePath: ipv6SourcePath,
Format: imgcache.FormatPNG,
token, err := generator.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: sourceHost, Path: ipv6SourcePath},
Transform: imagerequest.Transform{Format: imagerequest.FormatPNG},
})
if err != nil {
t.Fatalf("encurl Generate: %v", err)
+10 -7
View File
@@ -8,7 +8,7 @@ import (
"testing"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// svgServedAsAVIF is an SVG whose first bytes pass the first-bytes check for
@@ -38,10 +38,13 @@ func TestImageProxyRefusesSVGServedAsAVIF(t *testing.T) {
t.Fatalf("encurl.NewGenerator: %v", err)
}
encryptedURL := func(format imgcache.ImageFormat) string {
token, err := generator.Generate(&encurl.Payload{
SourceHost: upstreamHost, SourcePath: "/drawing.avif",
Width: 32, Height: 24, Format: format,
encryptedURL := func(format imagerequest.Format) string {
token, err := generator.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: upstreamHost, Path: "/drawing.avif"},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 32, Height: 24},
Format: format,
},
})
if err != nil {
t.Fatalf("encurl Generate: %v", err)
@@ -53,8 +56,8 @@ func TestImageProxyRefusesSVGServedAsAVIF(t *testing.T) {
for _, target := range []string{
"/v1/image/" + upstreamHost + "/drawing.avif/32x24.png",
"/v1/image/" + upstreamHost + "/drawing.avif/64x48.orig",
encryptedURL(imgcache.FormatPNG),
encryptedURL(imgcache.FormatOriginal),
encryptedURL(imagerequest.FormatPNG),
encryptedURL(imagerequest.FormatOriginal),
} {
rec := httptest.NewRecorder()
s.ServeHTTP(rec, httptest.NewRequestWithContext(
+31 -24
View File
@@ -5,6 +5,7 @@ import (
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -16,32 +17,35 @@ func TestSigner_Verify_ColonCannotMovePathQueryBoundary(t *testing.T) {
signer := signature.New(goldenSigningKey)
signed := &signature.Request{
SourceHost: testHost,
SourcePath: "/a:b",
Width: 800,
Height: 600,
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
Expires: time.Now().Add(1 * time.Hour),
signed := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: "/a:b",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
},
Expires: time.Now().Add(1 * time.Hour),
}
signed.Signature = signer.Sign(signed)
replayed := *signed
replayed.SourcePath = "/a"
replayed.SourceQuery = "b:"
replayed.Source.Path = "/a"
replayed.Source.Query = "b:"
t.Logf("signature for path %q query %q: %s",
signed.SourcePath, signed.SourceQuery, signed.Signature)
signed.Source.Path, signed.Source.Query, signed.Signature)
t.Logf("signature for path %q query %q: %s",
replayed.SourcePath, replayed.SourceQuery, signer.Sign(&replayed))
replayed.Source.Path, replayed.Source.Query, signer.Sign(&replayed))
err := signer.Verify(&replayed)
if !errors.Is(err, signature.ErrInvalid) {
t.Errorf("Verify() of path %q query %q with the signature for "+
"path %q = %v, want %v",
replayed.SourcePath, replayed.SourceQuery, signed.SourcePath,
replayed.Source.Path, replayed.Source.Query, signed.Source.Path,
err, signature.ErrInvalid)
}
}
@@ -90,16 +94,19 @@ func TestSigner_EscapedGoldenVectors(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
req := &signature.Request{
SourceHost: testHost,
SourcePath: tt.path,
SourceQuery: tt.query,
Width: 800,
Height: 600,
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
Expires: time.Unix(goldenExpiresUnix, 0),
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: tt.path,
Query: tt.query,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
},
Expires: time.Unix(goldenExpiresUnix, 0),
}
gotSignature := signer.Sign(req)
+50 -37
View File
@@ -4,6 +4,7 @@ import (
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -16,7 +17,7 @@ const goldenSigningKey = "golden-test-key"
type goldenVector struct {
name string
req signature.Request
req imagerequest.Request
// wantSignature is the exact base64url (RFC 4648 URL-safe,
// padded) HMAC-SHA256 signature for the request with Expires
// set to goldenExpiresUnix.
@@ -34,15 +35,18 @@ func goldenVectors() []goldenVector {
return []goldenVector{
{
name: "resized without query",
req: signature.Request{
SourceHost: testHost,
SourcePath: testPath,
SourceQuery: "",
Width: 800,
Height: 600,
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
req: imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
Query: "",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
},
},
// Signed data:
// "cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200:85:cover"
@@ -51,15 +55,18 @@ func goldenVectors() []goldenVector {
},
{
name: "resized with query string",
req: signature.Request{
SourceHost: testHost,
SourcePath: testPath,
SourceQuery: "token=abc&v=2",
Width: 800,
Height: 600,
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
req: imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
Query: "token=abc&v=2",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
},
},
// Signed data:
// "cdn.example.com:/photos/cat.jpg:token=abc&v=2:800:600:webp:1704067200:85:cover"
@@ -69,15 +76,18 @@ func goldenVectors() []goldenVector {
},
{
name: "original size without query",
req: signature.Request{
SourceHost: testHost,
SourcePath: testPath,
SourceQuery: "",
Width: 0,
Height: 0,
Format: testFormatPNG,
Quality: 85,
FitMode: testFitCover,
req: imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
Query: "",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 0, Height: 0},
Format: testFormatPNG,
Quality: 85,
FitMode: testFitCover,
},
},
// Signed data:
// "cdn.example.com:/photos/cat.jpg::0:0:png:1704067200:85:cover"
@@ -86,15 +96,18 @@ func goldenVectors() []goldenVector {
},
{
name: "non-default quality and fit",
req: signature.Request{
SourceHost: testHost,
SourcePath: testPath,
SourceQuery: "",
Width: 800,
Height: 600,
Format: testFormatWebP,
Quality: 40,
FitMode: testFitContain,
req: imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
Query: "",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
Quality: 40,
FitMode: testFitContain,
},
},
// Signed data:
// "cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200:40:contain"
+17 -13
View File
@@ -5,22 +5,26 @@ import (
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/signature"
)
// signedQualityFitRequest returns a request signed for quality 85 and fit
// mode "cover", the effective defaults the handler applies before
// verification.
func signedQualityFitRequest(signer *signature.Signer) *signature.Request {
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
Width: 800,
Height: 600,
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
Expires: time.Now().Add(1 * time.Hour),
func signedQualityFitRequest(signer *signature.Signer) *imagerequest.Request {
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
},
Expires: time.Now().Add(1 * time.Hour),
}
req.Signature = signer.Sign(req)
@@ -40,15 +44,15 @@ func TestSigner_Verify_QualityAndFitAreSigned(t *testing.T) {
cases := []struct {
name string
tamper func(r *signature.Request)
tamper func(r *imagerequest.Request)
}{
{
name: "replayed with different quality",
tamper: func(r *signature.Request) { r.Quality = 40 },
tamper: func(r *imagerequest.Request) { r.Transform.Quality = 40 },
},
{
name: "replayed with different fit mode",
tamper: func(r *signature.Request) { r.FitMode = testFitContain },
tamper: func(r *imagerequest.Request) { r.Transform.FitMode = testFitContain },
},
}
+25 -50
View File
@@ -12,6 +12,8 @@ import (
"strconv"
"strings"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// Signature errors.
@@ -22,37 +24,6 @@ var (
ErrMissingExpiration = errors.New("signature expiration is required")
)
// Request carries the components an image request signature covers. It is a
// standalone type so that this package does not depend on imgcache, keeping
// the import edge one-way (imgcache depends on signature, never the reverse).
type Request struct {
// SourceHost is the origin host (e.g. "cdn.example.com").
SourceHost string
// SourcePath is the path on the origin (e.g. "/photos/cat.jpg").
SourcePath string
// SourceQuery is the optional query string for the origin URL.
SourceQuery string
// Width is the requested output width in pixels.
Width int
// Height is the requested output height in pixels.
Height int
// Format is the requested output format (e.g. "webp").
Format string
// Quality is the requested output quality (1-100) for lossy formats.
// It is the effective value the request resolves to: callers pass the
// default quality when the request omits the parameter, so an omitted
// quality signs identically to that same value stated explicitly.
Quality int
// FitMode is how the image is fit into the requested dimensions
// (e.g. "cover"). Like Quality it is the effective value: callers pass
// the default fit mode when the request omits the parameter.
FitMode string
// Signature is the HMAC signature to verify.
Signature string
// Expires is the signature expiration timestamp.
Expires time.Time
}
// Signer handles HMAC-SHA256 signature generation and verification.
type Signer struct {
secretKey []byte
@@ -67,8 +38,11 @@ func New(secretKey string) *Signer {
// Sign generates an HMAC-SHA256 signature for the given request.
// The signature covers: host + path + query + width + height + format +
// expiration + quality + fit.
func (s *Signer) Sign(req *Request) string {
// expiration + quality + fit. The quality and fit mode are the effective
// values the request resolves to: callers pass the defaults when the request
// omits those parameters, so an omitted quality or fit mode signs
// identically to that same value stated explicitly.
func (s *Signer) Sign(req *imagerequest.Request) string {
data := s.buildSignatureData(req)
mac := hmac.New(sha256.New, s.secretKey)
mac.Write([]byte(data))
@@ -84,7 +58,7 @@ func (s *Signer) Sign(req *Request) string {
// No suffix matching, wildcard matching, or partial matching is supported.
// A signature for "cdn.example.com" will NOT verify for "example.com" or
// "other.cdn.example.com", and vice versa.
func (s *Signer) Verify(req *Request) error {
func (s *Signer) Verify(req *imagerequest.Request) error {
// Check expiration first
if req.Expires.IsZero() {
return ErrMissingExpiration
@@ -108,7 +82,7 @@ func (s *Signer) Verify(req *Request) error {
// GenerateSignedURL creates a complete URL with signature and expiration.
// Returns the path portion that should be appended to the base URL.
func (s *Signer) GenerateSignedURL(
req *Request, ttl time.Duration,
req *imagerequest.Request, ttl time.Duration,
) (string, string, int64) {
// Set expiration
req.Expires = time.Now().Add(ttl)
@@ -120,10 +94,11 @@ func (s *Signer) GenerateSignedURL(
// Build the size component
var sizeStr string
if req.Width == 0 && req.Height == 0 {
if req.Transform.Size.Width == 0 && req.Transform.Size.Height == 0 {
sizeStr = "orig"
} else {
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
sizeStr = fmt.Sprintf("%dx%d",
req.Transform.Size.Width, req.Transform.Size.Height)
}
// Build the path. A source query follows the source path after a "?".
@@ -131,13 +106,13 @@ func (s *Signer) GenerateSignedURL(
// written in Go's own percent-encoding of a URL path, which writes "?"
// as %3F and "%" as %25, and leaves "/", ";", ",", "=" and "&" as they
// are.
sourcePathAndQuery := req.SourcePath
if req.SourceQuery != "" {
sourcePathAndQuery += "?" + req.SourceQuery
sourcePathAndQuery := req.Source.Path
if req.Source.Query != "" {
sourcePathAndQuery += "?" + req.Source.Query
}
unescapedPath := fmt.Sprintf("/v1/image/%s%s/%s.%s",
req.SourceHost, sourcePathAndQuery, sizeStr, req.Format)
req.Source.Host, sourcePathAndQuery, sizeStr, req.Transform.Format)
return (&url.URL{Path: unescapedPath}).EscapedPath(), sig, exp
}
@@ -150,17 +125,17 @@ func (s *Signer) GenerateSignedURL(
// expansion is performed. Quality and fit are the effective transform
// values, so replaying a signed URL with a different quality or fit mode
// fails verification.
func (s *Signer) buildSignatureData(req *Request) string {
func (s *Signer) buildSignatureData(req *imagerequest.Request) string {
return fmt.Sprintf("%s:%s:%s:%d:%d:%s:%d:%d:%s",
req.SourceHost,
escapePercentAndColon(req.SourcePath),
escapePercentAndColon(req.SourceQuery),
req.Width,
req.Height,
req.Format,
req.Source.Host,
escapePercentAndColon(req.Source.Path),
escapePercentAndColon(req.Source.Query),
req.Transform.Size.Width,
req.Transform.Size.Height,
req.Transform.Format,
req.Expires.Unix(),
req.Quality,
req.FitMode,
req.Transform.Quality,
req.Transform.FitMode,
)
}
+140 -106
View File
@@ -6,6 +6,7 @@ import (
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -26,14 +27,17 @@ func TestSigner_Sign(t *testing.T) {
signer := signature.New("test-secret-key")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
SourceQuery: "",
Width: 800,
Height: 600,
Format: testFormatWebP,
Expires: time.Unix(1704067200, 0), // Fixed timestamp for reproducibility
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
Query: "",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Unix(1704067200, 0), // Fixed timestamp for reproducibility
}
sig1 := signer.Sign(req)
@@ -51,14 +55,17 @@ func TestSigner_Sign(t *testing.T) {
}
// Different input should produce different signature
req2 := &signature.Request{
SourceHost: testHost,
SourcePath: "/photos/dog.jpg", // Different path
SourceQuery: "",
Width: 800,
Height: 600,
Format: testFormatWebP,
Expires: time.Unix(1704067200, 0),
req2 := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: "/photos/dog.jpg", // Different path
Query: "",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Unix(1704067200, 0),
}
sig3 := signer.Sign(req2)
@@ -69,20 +76,23 @@ func TestSigner_Sign(t *testing.T) {
// validVerifyRequest returns a fully-populated request that verifies
// successfully once signed.
func validVerifyRequest() *signature.Request {
return &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
Width: 800,
Height: 600,
Format: testFormatWebP,
Expires: time.Now().Add(1 * time.Hour),
func validVerifyRequest() *imagerequest.Request {
return &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Now().Add(1 * time.Hour),
}
}
type verifyCase struct {
name string
setup func() *signature.Request
setup func() *imagerequest.Request
wantErr error
}
@@ -90,7 +100,7 @@ func verifyCases(signer *signature.Signer) []verifyCase {
return []verifyCase{
{
name: "valid signature",
setup: func() *signature.Request {
setup: func() *imagerequest.Request {
req := validVerifyRequest()
req.Signature = signer.Sign(req)
@@ -100,7 +110,7 @@ func verifyCases(signer *signature.Signer) []verifyCase {
},
{
name: "expired signature",
setup: func() *signature.Request {
setup: func() *imagerequest.Request {
req := validVerifyRequest()
req.Expires = time.Now().Add(-1 * time.Hour)
req.Signature = signer.Sign(req)
@@ -111,7 +121,7 @@ func verifyCases(signer *signature.Signer) []verifyCase {
},
{
name: "invalid signature",
setup: func() *signature.Request {
setup: func() *imagerequest.Request {
req := validVerifyRequest()
req.Signature = "invalid-signature"
@@ -121,7 +131,7 @@ func verifyCases(signer *signature.Signer) []verifyCase {
},
{
name: "missing expiration",
setup: func() *signature.Request {
setup: func() *imagerequest.Request {
req := validVerifyRequest()
req.Expires = time.Time{}
req.Signature = "some-signature"
@@ -132,10 +142,10 @@ func verifyCases(signer *signature.Signer) []verifyCase {
},
{
name: "tampered request",
setup: func() *signature.Request {
setup: func() *imagerequest.Request {
req := validVerifyRequest()
req.Signature = signer.Sign(req)
req.SourcePath = "/photos/secret.jpg"
req.Source.Path = "/photos/secret.jpg"
return req
},
@@ -173,7 +183,7 @@ func TestSigner_Verify(t *testing.T) {
type tamperCase struct {
name string
tamper func(r *signature.Request)
tamper func(r *imagerequest.Request)
}
// exactMatchTamperCases mutates one signed component per case; every
@@ -182,59 +192,59 @@ func exactMatchTamperCases() []tamperCase {
return []tamperCase{
{
name: "parent domain does not match subdomain",
tamper: func(r *signature.Request) { r.SourceHost = "example.com" },
tamper: func(r *imagerequest.Request) { r.Source.Host = "example.com" },
},
{
name: "subdomain does not match parent domain",
tamper: func(r *signature.Request) { r.SourceHost = "images.cdn.example.com" },
tamper: func(r *imagerequest.Request) { r.Source.Host = "images.cdn.example.com" },
},
{
name: "sibling subdomain does not match",
tamper: func(r *signature.Request) { r.SourceHost = "images.example.com" },
tamper: func(r *imagerequest.Request) { r.Source.Host = "images.example.com" },
},
{
name: "host with suffix appended does not match",
tamper: func(r *signature.Request) { r.SourceHost = testHost + ".evil.com" },
tamper: func(r *imagerequest.Request) { r.Source.Host = testHost + ".evil.com" },
},
{
name: "host with prefix does not match",
tamper: func(r *signature.Request) { r.SourceHost = "evilcdn.example.com" },
tamper: func(r *imagerequest.Request) { r.Source.Host = "evilcdn.example.com" },
},
{
name: "different path does not match",
tamper: func(r *signature.Request) { r.SourcePath = "/photos/dog.jpg" },
tamper: func(r *imagerequest.Request) { r.Source.Path = "/photos/dog.jpg" },
},
{
name: "path suffix does not match",
tamper: func(r *signature.Request) { r.SourcePath = testPath + "/extra" },
tamper: func(r *imagerequest.Request) { r.Source.Path = testPath + "/extra" },
},
{
name: "path prefix does not match",
tamper: func(r *signature.Request) { r.SourcePath = "/other" + testPath },
tamper: func(r *imagerequest.Request) { r.Source.Path = "/other" + testPath },
},
{
name: "different query does not match",
tamper: func(r *signature.Request) { r.SourceQuery = "token=xyz" },
tamper: func(r *imagerequest.Request) { r.Source.Query = "token=xyz" },
},
{
name: "added query does not match empty query",
tamper: func(r *signature.Request) { r.SourceQuery = "extra=1" },
tamper: func(r *imagerequest.Request) { r.Source.Query = "extra=1" },
},
{
name: "removed query does not match",
tamper: func(r *signature.Request) { r.SourceQuery = "" },
tamper: func(r *imagerequest.Request) { r.Source.Query = "" },
},
{
name: "different width does not match",
tamper: func(r *signature.Request) { r.Width = 801 },
tamper: func(r *imagerequest.Request) { r.Transform.Size.Width = 801 },
},
{
name: "different height does not match",
tamper: func(r *signature.Request) { r.Height = 601 },
tamper: func(r *imagerequest.Request) { r.Transform.Size.Height = 601 },
},
{
name: "different format does not match",
tamper: func(r *signature.Request) { r.Format = testFormatPNG },
tamper: func(r *imagerequest.Request) { r.Transform.Format = testFormatPNG },
},
}
}
@@ -248,15 +258,18 @@ func TestSigner_Verify_ExactMatchOnly(t *testing.T) {
signer := signature.New("test-secret-key")
// Base request that we'll sign, then tamper with individual fields.
baseReq := func() *signature.Request {
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
SourceQuery: "token=abc",
Width: 800,
Height: 600,
Format: testFormatWebP,
Expires: time.Now().Add(1 * time.Hour),
baseReq := func() *imagerequest.Request {
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
Query: "token=abc",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Now().Add(1 * time.Hour),
}
req.Signature = signer.Sign(req)
@@ -309,14 +322,17 @@ func TestSigner_Sign_ExactHostInData(t *testing.T) {
sigs := make(map[string]string)
for _, host := range hosts {
req := &signature.Request{
SourceHost: host,
SourcePath: testPath,
SourceQuery: "",
Width: 800,
Height: 600,
Format: testFormatWebP,
Expires: time.Unix(1704067200, 0),
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: host,
Path: testPath,
Query: "",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Unix(1704067200, 0),
}
sig := signer.Sign(req)
@@ -334,13 +350,16 @@ func TestSigner_DifferentKeys(t *testing.T) {
signer1 := signature.New("secret-key-1")
signer2 := signature.New("secret-key-2")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
Width: 800,
Height: 600,
Format: testFormatWebP,
Expires: time.Now().Add(1 * time.Hour),
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Now().Add(1 * time.Hour),
}
// Sign with key 1
@@ -364,13 +383,16 @@ func TestGenerateSignedURL(t *testing.T) {
signer := signature.New("test-secret-key")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
SourceQuery: "",
Width: 800,
Height: 600,
Format: testFormatWebP,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
Query: "",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
}
ttl := 1 * time.Hour
@@ -405,12 +427,15 @@ func TestGenerateSignedURL_OrigSize(t *testing.T) {
signer := signature.New("test-secret-key")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
Width: 0, // Original size
Height: 0,
Format: testFormatPNG,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 0, Height: 0}, // Original size
Format: testFormatPNG,
},
}
path, _, _ := signer.GenerateSignedURL(req, time.Hour)
@@ -426,13 +451,16 @@ func TestGenerateSignedURL_WithQueryString(t *testing.T) {
signer := signature.New("test-secret-key-for-testing!")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
SourceQuery: "token=abc&v=2",
Width: 800,
Height: 600,
Format: testFormatWebP,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
Query: "token=abc&v=2",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
}
path, _, _ := signer.GenerateSignedURL(req, time.Hour)
@@ -460,12 +488,15 @@ func TestGenerateSignedURL_WithoutQueryString(t *testing.T) {
signer := signature.New("test-secret-key-for-testing!")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
Width: 800,
Height: 600,
Format: testFormatWebP,
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: testPath,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
}
path, _, _ := signer.GenerateSignedURL(req, time.Hour)
@@ -485,16 +516,19 @@ func TestGenerateSignedURL_ReadmeQueryExample(t *testing.T) {
signer := signature.New("example-signing-key-for-documentation")
req := signature.Request{
SourceHost: testHost,
SourcePath: "/img",
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
Width: 800,
Height: 600,
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
Expires: time.Unix(1704067200, 0),
req := imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
Path: "/img",
Query: "url=https://origin.example.com/my%20cat.jpg",
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
},
Expires: time.Unix(1704067200, 0),
}
const wantSig = "37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA="
+3 -3
View File
@@ -1,8 +1,8 @@
#!/bin/sh
# script/docker-smoke: build the Docker image, start it, and wait up to
# 30 seconds for its HEALTHCHECK to report healthy. Needs a Docker
# daemon, so it is not part of script/check; the Gitea workflow runs it
# after script/cibuild.
# 30 seconds for its HEALTHCHECK to report healthy. It is not part of
# script/check, which never starts the image, and the Gitea workflow
# does not run it. Run it by hand.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"