Compare commits
3
Commits
v1.0.0-rc3
...
2dd2134f7f
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2dd2134f7f | ||
|
|
00d8ba26c5 | ||
|
|
385c0ca76f |
@@ -18,4 +18,3 @@ jobs:
|
||||
# All history and tags, so git describe finds the version tag.
|
||||
fetch-depth: 0
|
||||
- run: script/cibuild
|
||||
- run: script/docker-smoke
|
||||
|
||||
@@ -76,10 +76,9 @@ database and the disk cache:
|
||||
pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
|
||||
must be able to write the directory.
|
||||
- `cache_max_bytes` limits the source and transformed images together. The
|
||||
database, the metadata files, the `.meta` file beside each transformed image
|
||||
and files still being written come on top, and eviction runs in the
|
||||
background, so the cache can pass the limit for a while: leave room on the
|
||||
volume beyond it.
|
||||
database, the `.meta` file beside each transformed image and files still being
|
||||
written come on top, and eviction runs in the background, so the cache can
|
||||
pass the limit for a while: leave room on the volume beyond it.
|
||||
- Set `cache_max_bytes` for a lasting deployment. Its default, worked out each
|
||||
time pixa starts, is 75% of the sum of the space free on the volume and the
|
||||
space the cached images already take, so a restart keeps the limit the cache
|
||||
@@ -177,11 +176,9 @@ allowlisted source hosts for open access.
|
||||
|
||||
### Storage
|
||||
|
||||
- **Source content**:
|
||||
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of source content>`
|
||||
- **Source metadata**:
|
||||
`<state_dir>/cache/metadata/<hostname>/<sha256 of path and query>.json` (host,
|
||||
path and query, content hash, upstream status and headers, fetch time)
|
||||
- **Source images**:
|
||||
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of host, path and query>`, one
|
||||
file for each source URL
|
||||
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
|
||||
- **Transformed images**:
|
||||
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`,
|
||||
@@ -190,8 +187,11 @@ allowlisted source hosts for open access.
|
||||
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
|
||||
name.
|
||||
|
||||
Multiple source paths may reference the same content blob; the database tracks
|
||||
references rather than using filesystem refcounting.
|
||||
Each source image and each transformed image has one row in the database,
|
||||
holding its size and when it was last used. Two source URLs that serve the same
|
||||
bytes are stored twice, once under each URL. A source image is stored by writing
|
||||
its file and then its row; when the row cannot be written, the file is removed
|
||||
again.
|
||||
|
||||
Source and transformed image files, and the `.meta` file beside each transformed
|
||||
image, are written under a temporary name, synced to disk, and only then renamed
|
||||
|
||||
@@ -30,6 +30,30 @@ P2: security: per-origin rate limiting
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-21 a source is stored the way a variant is (closes #308): one
|
||||
`source_content` row and one file per source URL, the file named by the
|
||||
source's `SourceKey`, the SHA-256 of its host, path and query made as the
|
||||
variant key is, under `cache/sources/`. A store writes the file, then the row,
|
||||
and removes the file when the row cannot be written; an eviction deletes the
|
||||
row, then the file, with no lock and no transaction. `source_metadata`, the
|
||||
JSON files under `cache/metadata/`, `MetadataStorage` and the content lock are
|
||||
gone. Two source URLs that serve the same bytes now store them twice.
|
||||
- 2026-10-21 one image request type instead of five (closes #307): the new
|
||||
package `internal/imagerequest`, which imports no other package of pixa, holds
|
||||
`imagerequest.Request` with its two parts, `Source` (host, path and query) and
|
||||
`Transform` (size, format, quality and fit), the `Format` and `Fit` types, the
|
||||
checks, the parsing of the `/v1/image/` path and the variant key.
|
||||
`imgcache.ParsedURL`, `imgcache.ImageRequest`, `signature.Request`,
|
||||
`imageprocessor.Request`, the second `Size`, format and fit types and the
|
||||
conversions between them are gone. The image processor takes a `Transform`
|
||||
only, and `encurl` keeps a private struct with the token's short CBOR field
|
||||
names, so tokens made before still decode. Every URL of both image routes
|
||||
answers as before.
|
||||
- 2026-10-21 the Gitea workflow checks out the repo and runs `script/cibuild`,
|
||||
nothing else (closes #203): its `script/docker-smoke` step is removed, as the
|
||||
canonical `REPO_POLICIES.md` says, since `make check` covers what CI needs to
|
||||
check. CI no longer starts the image and waits for its healthcheck.
|
||||
`script/docker-smoke` and `make docker-smoke` stay, to run by hand.
|
||||
- 2026-10-20 a WebP, AVIF or JPEG XL encode at the limits takes at most about 34
|
||||
seconds (closes #272): with one libvips thread, 8192x8192 random pixels with
|
||||
an alpha channel took about 61 seconds to encode as WebP, 81 as JPEG XL and,
|
||||
|
||||
@@ -131,7 +131,8 @@ func writeLikeOneRequest(ctx context.Context, db *sql.DB, key string) error {
|
||||
}
|
||||
|
||||
_, err = db.ExecContext(ctx, `INSERT INTO source_content
|
||||
(content_hash, content_type, size_bytes) VALUES (?, 'image/png', 1)`, key)
|
||||
(source_key, source_host, source_path, content_type, size_bytes)
|
||||
VALUES (?, 'example.com', '/image.png', 'image/png', 1)`, key)
|
||||
if err != nil {
|
||||
return fmt.Errorf("storing a source: %w", err)
|
||||
}
|
||||
|
||||
@@ -160,7 +160,7 @@ func TestApplyMigrations_CreatesSchemaAndTables(t *testing.T) {
|
||||
|
||||
// Verify that the application tables created by 001.sql exist.
|
||||
tables := []string{
|
||||
"source_content", "source_metadata", "output_content",
|
||||
"source_content", "output_content",
|
||||
"request_cache", "negative_cache", "cache_stats",
|
||||
}
|
||||
for _, table := range tables {
|
||||
|
||||
@@ -1,10 +1,14 @@
|
||||
-- Migration 001: Initial schema
|
||||
-- Creates all tables for the pixa caching image proxy
|
||||
|
||||
-- Source content blobs
|
||||
-- Files stored at: cache/sources/<ab>/<cd>/<sha256>
|
||||
-- Source images, one row per source URL
|
||||
-- Files stored at: cache/sources/<ab>/<cd>/<source_key>, where source_key
|
||||
-- is the SHA-256 of the source's host, path and query
|
||||
CREATE TABLE IF NOT EXISTS source_content (
|
||||
content_hash TEXT PRIMARY KEY,
|
||||
source_key TEXT PRIMARY KEY,
|
||||
source_host TEXT NOT NULL,
|
||||
source_path TEXT NOT NULL,
|
||||
source_query TEXT NOT NULL DEFAULT '',
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes INTEGER NOT NULL,
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
@@ -13,30 +17,6 @@ CREATE TABLE IF NOT EXISTS source_content (
|
||||
CREATE INDEX IF NOT EXISTS idx_source_content_last_accessed
|
||||
ON source_content(last_accessed_at);
|
||||
|
||||
-- Source URL metadata - maps URLs to content hashes
|
||||
-- JSON stored at: cache/metadata/<hostname>/<path_hash>.json
|
||||
CREATE TABLE IF NOT EXISTS source_metadata (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_host TEXT NOT NULL,
|
||||
source_path TEXT NOT NULL,
|
||||
source_query TEXT NOT NULL DEFAULT '',
|
||||
path_hash TEXT NOT NULL,
|
||||
content_hash TEXT,
|
||||
status_code INTEGER NOT NULL,
|
||||
content_type TEXT,
|
||||
response_headers TEXT,
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
expires_at DATETIME,
|
||||
etag TEXT,
|
||||
last_modified TEXT,
|
||||
UNIQUE(source_host, source_path, source_query),
|
||||
FOREIGN KEY (content_hash) REFERENCES source_content(content_hash)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_source_meta_host ON source_metadata(source_host);
|
||||
CREATE INDEX IF NOT EXISTS idx_source_meta_path_hash ON source_metadata(path_hash);
|
||||
CREATE INDEX IF NOT EXISTS idx_source_meta_expires ON source_metadata(expires_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_source_meta_content_hash ON source_metadata(content_hash);
|
||||
|
||||
-- Processed variant blobs
|
||||
-- Files stored at: cache/variants/<ab>/<cd>/<cache_key> (plus a .meta
|
||||
-- sidecar with the content type). Tracked here (like source content
|
||||
|
||||
+52
-65
@@ -7,15 +7,15 @@ import (
|
||||
|
||||
"github.com/fxamacker/cbor/v2"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/seal"
|
||||
)
|
||||
|
||||
// Default values for optional fields.
|
||||
const (
|
||||
DefaultQuality = 85
|
||||
DefaultFormat = imgcache.FormatJXL
|
||||
DefaultFitMode = imgcache.FitCover
|
||||
DefaultFormat = imagerequest.FormatJXL
|
||||
DefaultFitMode = imagerequest.FitCover
|
||||
|
||||
// HKDF salt for URL encryption key derivation
|
||||
urlKeySalt = "pixa-urlenc-v1"
|
||||
@@ -28,19 +28,19 @@ var (
|
||||
ErrDecryptFailed = errors.New("failed to decrypt URL")
|
||||
)
|
||||
|
||||
// Payload contains all image request parameters in a compact format.
|
||||
// payload contains all image request parameters in a compact format.
|
||||
// Field names are short to minimize encoded size.
|
||||
// Fields with omitempty use sane defaults when absent.
|
||||
type Payload struct {
|
||||
SourceHost string `cbor:"h"` // required
|
||||
SourcePath string `cbor:"p"` // required
|
||||
SourceQuery string `cbor:"q,omitempty"` // optional
|
||||
Width int `cbor:"w,omitempty"` // 0 = original
|
||||
Height int `cbor:"ht,omitempty"` // 0 = original
|
||||
Format imgcache.ImageFormat `cbor:"f,omitempty"` // default: jxl
|
||||
Quality int `cbor:"ql,omitempty"` // default: 85
|
||||
FitMode imgcache.FitMode `cbor:"fm,omitempty"` // default: cover
|
||||
ExpiresAt int64 `cbor:"e,omitempty"` // 0 = never expires
|
||||
type payload struct {
|
||||
SourceHost string `cbor:"h"` // required
|
||||
SourcePath string `cbor:"p"` // required
|
||||
SourceQuery string `cbor:"q,omitempty"` // optional
|
||||
Width int `cbor:"w,omitempty"` // 0 = original
|
||||
Height int `cbor:"ht,omitempty"` // 0 = original
|
||||
Format imagerequest.Format `cbor:"f,omitempty"` // default: jxl
|
||||
Quality int `cbor:"ql,omitempty"` // default: 85
|
||||
FitMode imagerequest.Fit `cbor:"fm,omitempty"` // default: cover
|
||||
ExpiresAt int64 `cbor:"e,omitempty"` // 0 = never expires
|
||||
}
|
||||
|
||||
// Generator creates and parses encrypted URL tokens.
|
||||
@@ -59,9 +59,27 @@ func NewGenerator(signingKey string) (*Generator, error) {
|
||||
return &Generator{key: key}, nil
|
||||
}
|
||||
|
||||
// Generate creates an encrypted URL token from the payload.
|
||||
// The token is CBOR-encoded, encrypted with NaCl secretbox, and base64url-encoded.
|
||||
func (g *Generator) Generate(p *Payload) (string, error) {
|
||||
// Generate creates an encrypted URL token from the request's source,
|
||||
// transform and expiry, each as the request holds it; an empty format, a
|
||||
// quality of 0 or an empty fit mode is left out of the token, and a zero
|
||||
// Expires makes a token that never expires. The token is CBOR-encoded,
|
||||
// encrypted with NaCl secretbox, and base64url-encoded.
|
||||
func (g *Generator) Generate(req *imagerequest.Request) (string, error) {
|
||||
p := &payload{
|
||||
SourceHost: req.Source.Host,
|
||||
SourcePath: req.Source.Path,
|
||||
SourceQuery: req.Source.Query,
|
||||
Width: req.Transform.Size.Width,
|
||||
Height: req.Transform.Size.Height,
|
||||
Format: req.Transform.Format,
|
||||
Quality: req.Transform.Quality,
|
||||
FitMode: req.Transform.FitMode,
|
||||
}
|
||||
|
||||
if !req.Expires.IsZero() {
|
||||
p.ExpiresAt = req.Expires.Unix()
|
||||
}
|
||||
|
||||
// CBOR encode the payload
|
||||
data, err := cbor.Marshal(p)
|
||||
if err != nil {
|
||||
@@ -72,9 +90,11 @@ func (g *Generator) Generate(p *Payload) (string, error) {
|
||||
return seal.Encrypt(g.key, data)
|
||||
}
|
||||
|
||||
// Parse decrypts and validates an encrypted URL token.
|
||||
// Parse decrypts and validates an encrypted URL token and returns the
|
||||
// request it holds, with default values for omitted optional fields. A token
|
||||
// that never expires gives the zero Expires.
|
||||
// Returns ErrExpired if the token has expired, or other errors for invalid tokens.
|
||||
func (g *Generator) Parse(token string) (*Payload, error) {
|
||||
func (g *Generator) Parse(token string) (*imagerequest.Request, error) {
|
||||
// Decrypt
|
||||
data, err := seal.Decrypt(g.key, token)
|
||||
if err != nil {
|
||||
@@ -87,7 +107,7 @@ func (g *Generator) Parse(token string) (*Payload, error) {
|
||||
}
|
||||
|
||||
// CBOR decode
|
||||
var p Payload
|
||||
var p payload
|
||||
|
||||
err = cbor.Unmarshal(data, &p)
|
||||
if err != nil {
|
||||
@@ -99,13 +119,6 @@ func (g *Generator) Parse(token string) (*Payload, error) {
|
||||
return nil, ErrExpired
|
||||
}
|
||||
|
||||
return &p, nil
|
||||
}
|
||||
|
||||
// ToImageRequest converts the payload to an ImageRequest.
|
||||
// Applies default values for omitted optional fields. An ExpiresAt of 0, a URL
|
||||
// that never expires, gives the zero Expires.
|
||||
func (p *Payload) ToImageRequest() *imgcache.ImageRequest {
|
||||
format := p.Format
|
||||
if format == "" {
|
||||
format = DefaultFormat
|
||||
@@ -126,44 +139,18 @@ func (p *Payload) ToImageRequest() *imgcache.ImageRequest {
|
||||
expires = time.Unix(p.ExpiresAt, 0)
|
||||
}
|
||||
|
||||
return &imgcache.ImageRequest{
|
||||
SourceHost: p.SourceHost,
|
||||
SourcePath: p.SourcePath,
|
||||
SourceQuery: p.SourceQuery,
|
||||
Size: imgcache.Size{
|
||||
Width: p.Width,
|
||||
Height: p.Height,
|
||||
return &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: p.SourceHost,
|
||||
Path: p.SourcePath,
|
||||
Query: p.SourceQuery,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: p.Width, Height: p.Height},
|
||||
Format: format,
|
||||
Quality: quality,
|
||||
FitMode: fitMode,
|
||||
},
|
||||
Format: format,
|
||||
Quality: quality,
|
||||
FitMode: fitMode,
|
||||
Expires: expires,
|
||||
}
|
||||
}
|
||||
|
||||
// FromImageRequest creates a Payload from an ImageRequest with the given expiration.
|
||||
func FromImageRequest(req *imgcache.ImageRequest, expiresAt time.Time) *Payload {
|
||||
p := &Payload{
|
||||
SourceHost: req.SourceHost,
|
||||
SourcePath: req.SourcePath,
|
||||
SourceQuery: req.SourceQuery,
|
||||
Width: req.Size.Width,
|
||||
Height: req.Size.Height,
|
||||
ExpiresAt: expiresAt.Unix(),
|
||||
}
|
||||
|
||||
// Only set non-default values to benefit from omitempty
|
||||
if req.Format != "" && req.Format != DefaultFormat {
|
||||
p.Format = req.Format
|
||||
}
|
||||
|
||||
if req.Quality != 0 && req.Quality != DefaultQuality {
|
||||
p.Quality = req.Quality
|
||||
}
|
||||
|
||||
if req.FitMode != "" && req.FitMode != DefaultFitMode {
|
||||
p.FitMode = req.FitMode
|
||||
}
|
||||
|
||||
return p
|
||||
}, nil
|
||||
}
|
||||
|
||||
+91
-174
@@ -6,7 +6,7 @@ import (
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// Shared test fixture strings.
|
||||
@@ -24,19 +24,22 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
|
||||
t.Fatalf("NewGenerator() error = %v", err)
|
||||
}
|
||||
|
||||
payload := &encurl.Payload{
|
||||
SourceHost: testSourceHost,
|
||||
SourcePath: testSourcePath,
|
||||
SourceQuery: testSourceQuery,
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: imgcache.FormatWebP,
|
||||
Quality: 90,
|
||||
FitMode: imgcache.FitContain,
|
||||
ExpiresAt: time.Now().Add(time.Hour).Unix(),
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testSourceHost,
|
||||
Path: testSourcePath,
|
||||
Query: testSourceQuery,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 90,
|
||||
FitMode: imagerequest.FitContain,
|
||||
},
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
}
|
||||
|
||||
token, err := gen.Generate(payload)
|
||||
token, err := gen.Generate(req)
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
@@ -51,40 +54,45 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
|
||||
}
|
||||
|
||||
// Verify all fields
|
||||
if parsed.SourceHost != payload.SourceHost {
|
||||
t.Errorf("SourceHost = %q, want %q", parsed.SourceHost, payload.SourceHost)
|
||||
if parsed.Source.Host != req.Source.Host {
|
||||
t.Errorf("SourceHost = %q, want %q", parsed.Source.Host, req.Source.Host)
|
||||
}
|
||||
|
||||
if parsed.SourcePath != payload.SourcePath {
|
||||
t.Errorf("SourcePath = %q, want %q", parsed.SourcePath, payload.SourcePath)
|
||||
if parsed.Source.Path != req.Source.Path {
|
||||
t.Errorf("SourcePath = %q, want %q", parsed.Source.Path, req.Source.Path)
|
||||
}
|
||||
|
||||
if parsed.SourceQuery != payload.SourceQuery {
|
||||
t.Errorf("SourceQuery = %q, want %q", parsed.SourceQuery, payload.SourceQuery)
|
||||
if parsed.Source.Query != req.Source.Query {
|
||||
t.Errorf("SourceQuery = %q, want %q", parsed.Source.Query, req.Source.Query)
|
||||
}
|
||||
|
||||
if parsed.Width != payload.Width {
|
||||
t.Errorf("Width = %d, want %d", parsed.Width, payload.Width)
|
||||
if parsed.Transform.Size.Width != req.Transform.Size.Width {
|
||||
t.Errorf("Width = %d, want %d",
|
||||
parsed.Transform.Size.Width, req.Transform.Size.Width)
|
||||
}
|
||||
|
||||
if parsed.Height != payload.Height {
|
||||
t.Errorf("Height = %d, want %d", parsed.Height, payload.Height)
|
||||
if parsed.Transform.Size.Height != req.Transform.Size.Height {
|
||||
t.Errorf("Height = %d, want %d",
|
||||
parsed.Transform.Size.Height, req.Transform.Size.Height)
|
||||
}
|
||||
|
||||
if parsed.Format != payload.Format {
|
||||
t.Errorf("Format = %q, want %q", parsed.Format, payload.Format)
|
||||
if parsed.Transform.Format != req.Transform.Format {
|
||||
t.Errorf("Format = %q, want %q",
|
||||
parsed.Transform.Format, req.Transform.Format)
|
||||
}
|
||||
|
||||
if parsed.Quality != payload.Quality {
|
||||
t.Errorf("Quality = %d, want %d", parsed.Quality, payload.Quality)
|
||||
if parsed.Transform.Quality != req.Transform.Quality {
|
||||
t.Errorf("Quality = %d, want %d",
|
||||
parsed.Transform.Quality, req.Transform.Quality)
|
||||
}
|
||||
|
||||
if parsed.FitMode != payload.FitMode {
|
||||
t.Errorf("FitMode = %q, want %q", parsed.FitMode, payload.FitMode)
|
||||
if parsed.Transform.FitMode != req.Transform.FitMode {
|
||||
t.Errorf("FitMode = %q, want %q",
|
||||
parsed.Transform.FitMode, req.Transform.FitMode)
|
||||
}
|
||||
|
||||
if parsed.ExpiresAt != payload.ExpiresAt {
|
||||
t.Errorf("ExpiresAt = %d, want %d", parsed.ExpiresAt, payload.ExpiresAt)
|
||||
if parsed.Expires.Unix() != req.Expires.Unix() {
|
||||
t.Errorf("ExpiresAt = %d, want %d", parsed.Expires.Unix(), req.Expires.Unix())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -93,13 +101,15 @@ func TestGenerator_Parse_Expired(t *testing.T) {
|
||||
|
||||
gen, _ := encurl.NewGenerator("test-signing-key-12345")
|
||||
|
||||
payload := &encurl.Payload{
|
||||
SourceHost: testSourceHost,
|
||||
SourcePath: testSourcePath,
|
||||
ExpiresAt: time.Now().Add(-time.Hour).Unix(), // Already expired
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testSourceHost,
|
||||
Path: testSourcePath,
|
||||
},
|
||||
Expires: time.Now().Add(-time.Hour), // Already expired
|
||||
}
|
||||
|
||||
token, err := gen.Generate(payload)
|
||||
token, err := gen.Generate(req)
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
@@ -130,13 +140,15 @@ func TestGenerator_Parse_TamperedToken(t *testing.T) {
|
||||
|
||||
gen, _ := encurl.NewGenerator("test-signing-key-12345")
|
||||
|
||||
payload := &encurl.Payload{
|
||||
SourceHost: testSourceHost,
|
||||
SourcePath: testSourcePath,
|
||||
ExpiresAt: time.Now().Add(time.Hour).Unix(),
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testSourceHost,
|
||||
Path: testSourcePath,
|
||||
},
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
}
|
||||
|
||||
token, _ := gen.Generate(payload)
|
||||
token, _ := gen.Generate(req)
|
||||
|
||||
// Tamper with the token
|
||||
tampered := []byte(token)
|
||||
@@ -156,13 +168,15 @@ func TestGenerator_Parse_WrongKey(t *testing.T) {
|
||||
gen1, _ := encurl.NewGenerator("signing-key-1")
|
||||
gen2, _ := encurl.NewGenerator("signing-key-2")
|
||||
|
||||
payload := &encurl.Payload{
|
||||
SourceHost: testSourceHost,
|
||||
SourcePath: testSourcePath,
|
||||
ExpiresAt: time.Now().Add(time.Hour).Unix(),
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testSourceHost,
|
||||
Path: testSourcePath,
|
||||
},
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
}
|
||||
|
||||
token, _ := gen1.Generate(payload)
|
||||
token, _ := gen1.Generate(req)
|
||||
|
||||
_, err := gen2.Parse(token)
|
||||
if err == nil {
|
||||
@@ -170,140 +184,41 @@ func TestGenerator_Parse_WrongKey(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPayload_ToImageRequest(t *testing.T) {
|
||||
func TestGenerator_Parse_Defaults(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
payload := &encurl.Payload{
|
||||
SourceHost: testSourceHost,
|
||||
SourcePath: testSourcePath,
|
||||
SourceQuery: testSourceQuery,
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: imgcache.FormatWebP,
|
||||
Quality: 90,
|
||||
FitMode: imgcache.FitContain,
|
||||
ExpiresAt: time.Now().Add(time.Hour).Unix(),
|
||||
gen, _ := encurl.NewGenerator("test-signing-key-12345")
|
||||
|
||||
// Request with only required fields - should get defaults
|
||||
token, err := gen.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testSourceHost,
|
||||
Path: testSourcePath,
|
||||
},
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
}
|
||||
|
||||
req := payload.ToImageRequest()
|
||||
|
||||
if req.SourceHost != payload.SourceHost {
|
||||
t.Errorf("SourceHost = %q, want %q", req.SourceHost, payload.SourceHost)
|
||||
req, err := gen.Parse(token)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse() error = %v", err)
|
||||
}
|
||||
|
||||
if req.SourcePath != payload.SourcePath {
|
||||
t.Errorf("SourcePath = %q, want %q", req.SourcePath, payload.SourcePath)
|
||||
if req.Transform.Format != encurl.DefaultFormat {
|
||||
t.Errorf("Format = %q, want default %q",
|
||||
req.Transform.Format, encurl.DefaultFormat)
|
||||
}
|
||||
|
||||
if req.SourceQuery != payload.SourceQuery {
|
||||
t.Errorf("SourceQuery = %q, want %q", req.SourceQuery, payload.SourceQuery)
|
||||
if req.Transform.Quality != encurl.DefaultQuality {
|
||||
t.Errorf("Quality = %d, want default %d",
|
||||
req.Transform.Quality, encurl.DefaultQuality)
|
||||
}
|
||||
|
||||
if req.Size.Width != payload.Width {
|
||||
t.Errorf("Width = %d, want %d", req.Size.Width, payload.Width)
|
||||
}
|
||||
|
||||
if req.Size.Height != payload.Height {
|
||||
t.Errorf("Height = %d, want %d", req.Size.Height, payload.Height)
|
||||
}
|
||||
|
||||
if req.Format != payload.Format {
|
||||
t.Errorf("Format = %q, want %q", req.Format, payload.Format)
|
||||
}
|
||||
|
||||
if req.Quality != payload.Quality {
|
||||
t.Errorf("Quality = %d, want %d", req.Quality, payload.Quality)
|
||||
}
|
||||
|
||||
if req.FitMode != payload.FitMode {
|
||||
t.Errorf("FitMode = %q, want %q", req.FitMode, payload.FitMode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPayload_ToImageRequest_Defaults(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Payload with only required fields - should get defaults
|
||||
payload := &encurl.Payload{
|
||||
SourceHost: testSourceHost,
|
||||
SourcePath: testSourcePath,
|
||||
ExpiresAt: time.Now().Add(time.Hour).Unix(),
|
||||
}
|
||||
|
||||
req := payload.ToImageRequest()
|
||||
|
||||
if req.Format != encurl.DefaultFormat {
|
||||
t.Errorf("Format = %q, want default %q", req.Format, encurl.DefaultFormat)
|
||||
}
|
||||
|
||||
if req.Quality != encurl.DefaultQuality {
|
||||
t.Errorf("Quality = %d, want default %d", req.Quality, encurl.DefaultQuality)
|
||||
}
|
||||
|
||||
if req.FitMode != encurl.DefaultFitMode {
|
||||
t.Errorf("FitMode = %q, want default %q", req.FitMode, encurl.DefaultFitMode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFromImageRequest(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &imgcache.ImageRequest{
|
||||
SourceHost: testSourceHost,
|
||||
SourcePath: testSourcePath,
|
||||
SourceQuery: testSourceQuery,
|
||||
Size: imgcache.Size{Width: 800, Height: 600},
|
||||
Format: imgcache.FormatWebP,
|
||||
Quality: 90,
|
||||
FitMode: imgcache.FitContain,
|
||||
}
|
||||
|
||||
expiresAt := time.Now().Add(time.Hour)
|
||||
payload := encurl.FromImageRequest(req, expiresAt)
|
||||
|
||||
if payload.SourceHost != req.SourceHost {
|
||||
t.Errorf("SourceHost = %q, want %q", payload.SourceHost, req.SourceHost)
|
||||
}
|
||||
|
||||
if payload.SourcePath != req.SourcePath {
|
||||
t.Errorf("SourcePath = %q, want %q", payload.SourcePath, req.SourcePath)
|
||||
}
|
||||
|
||||
if payload.Width != req.Size.Width {
|
||||
t.Errorf("Width = %d, want %d", payload.Width, req.Size.Width)
|
||||
}
|
||||
|
||||
if payload.ExpiresAt != expiresAt.Unix() {
|
||||
t.Errorf("ExpiresAt = %d, want %d", payload.ExpiresAt, expiresAt.Unix())
|
||||
}
|
||||
}
|
||||
|
||||
func TestFromImageRequest_OmitsDefaults(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Request with default values - payload should omit them for
|
||||
// smaller encoding
|
||||
req := &imgcache.ImageRequest{
|
||||
SourceHost: testSourceHost,
|
||||
SourcePath: testSourcePath,
|
||||
Format: encurl.DefaultFormat,
|
||||
Quality: encurl.DefaultQuality,
|
||||
FitMode: encurl.DefaultFitMode,
|
||||
}
|
||||
|
||||
payload := encurl.FromImageRequest(req, time.Now().Add(time.Hour))
|
||||
|
||||
// These should be zero/empty because they match defaults
|
||||
if payload.Format != "" {
|
||||
t.Errorf("Format should be empty for default, got %q", payload.Format)
|
||||
}
|
||||
|
||||
if payload.Quality != 0 {
|
||||
t.Errorf("Quality should be 0 for default, got %d", payload.Quality)
|
||||
}
|
||||
|
||||
if payload.FitMode != "" {
|
||||
t.Errorf("FitMode should be empty for default, got %q", payload.FitMode)
|
||||
if req.Transform.FitMode != encurl.DefaultFitMode {
|
||||
t.Errorf("FitMode = %q, want default %q",
|
||||
req.Transform.FitMode, encurl.DefaultFitMode)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -312,13 +227,15 @@ func TestGenerator_TokenIsURLSafe(t *testing.T) {
|
||||
|
||||
gen, _ := encurl.NewGenerator("test-signing-key-12345")
|
||||
|
||||
payload := &encurl.Payload{
|
||||
SourceHost: testSourceHost,
|
||||
SourcePath: testSourcePath,
|
||||
ExpiresAt: time.Now().Add(time.Hour).Unix(),
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testSourceHost,
|
||||
Path: testSourcePath,
|
||||
},
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
}
|
||||
|
||||
token, _ := gen.Generate(payload)
|
||||
token, _ := gen.Generate(req)
|
||||
|
||||
// Check that token only contains URL-safe characters
|
||||
for _, c := range token {
|
||||
|
||||
+33
-35
@@ -16,7 +16,7 @@ import (
|
||||
|
||||
"sneak.berlin/go/pixa/internal/clientip"
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/templates"
|
||||
)
|
||||
|
||||
@@ -145,7 +145,7 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
payload, expiresAt, ttl, err := buildGeneratePayload(parsed, r.Form)
|
||||
req, expiresAt, ttl, err := buildGenerateRequest(parsed, r.Form)
|
||||
if err != nil {
|
||||
s.renderGeneratorWithForm(w, r, err.Error(), r.Form,
|
||||
http.StatusBadRequest)
|
||||
@@ -154,9 +154,8 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
||||
}
|
||||
|
||||
// The image URL must fetch the source URL as it was written
|
||||
sourceRequest := payload.ToImageRequest()
|
||||
sourceRequest.AllowHTTP = s.config.AllowHTTP
|
||||
fetchURL := sourceRequest.SourceURL()
|
||||
req.AllowHTTP = s.config.AllowHTTP
|
||||
fetchURL := req.SourceURL()
|
||||
|
||||
if !fetchedAsWritten(parsed, fetchURL) {
|
||||
s.renderGeneratorWithForm(w, r, "Source URL cannot be fetched as "+
|
||||
@@ -167,7 +166,7 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
||||
}
|
||||
|
||||
// Generate encrypted token
|
||||
token, err := s.encGen.Generate(payload)
|
||||
token, err := s.encGen.Generate(req)
|
||||
if err != nil {
|
||||
s.log.Error("failed to generate encrypted URL", "error", err)
|
||||
s.renderGeneratorWithForm(w, r, "Failed to generate URL", r.Form,
|
||||
@@ -240,14 +239,15 @@ func fetchedAsWritten(source *url.URL, fetchURL string) bool {
|
||||
fetched.RawQuery == source.RawQuery
|
||||
}
|
||||
|
||||
// buildGeneratePayload parses the numeric form fields and assembles the
|
||||
// encrypted URL payload. ttl=0 means never expires (ExpiresAt stays 0). A
|
||||
// non-numeric or out-of-range width, height, quality or ttl, or an
|
||||
// unrecognized fit mode, is a client error naming the offending field. The
|
||||
// format field is passed through unchecked.
|
||||
func buildGeneratePayload(
|
||||
// buildGenerateRequest parses the numeric form fields and assembles the
|
||||
// image request the encrypted URL's token is made from. ttl=0 means never
|
||||
// expires (Expires stays the zero time). A non-numeric or out-of-range
|
||||
// width, height, quality or ttl, or an unrecognized fit mode, is a client
|
||||
// error naming the offending field. The format field is passed through
|
||||
// unchecked.
|
||||
func buildGenerateRequest(
|
||||
parsed *url.URL, form url.Values,
|
||||
) (*encurl.Payload, time.Time, int, error) {
|
||||
) (*imagerequest.Request, time.Time, int, error) {
|
||||
width, err := parseFormDimension(form, "width")
|
||||
if err != nil {
|
||||
return nil, time.Time{}, 0, err
|
||||
@@ -269,42 +269,40 @@ func buildGeneratePayload(
|
||||
return nil, time.Time{}, 0, err
|
||||
}
|
||||
|
||||
fitMode := imgcache.FitMode(form.Get("fit"))
|
||||
fitMode := imagerequest.Fit(form.Get("fit"))
|
||||
|
||||
err = imgcache.ValidateFitMode(fitMode)
|
||||
err = imagerequest.ValidateFitMode(fitMode)
|
||||
if err != nil {
|
||||
return nil, time.Time{}, 0,
|
||||
fmt.Errorf("%w: %s", imgcache.ErrInvalidFitMode, form.Get("fit"))
|
||||
fmt.Errorf("%w: %s", imagerequest.ErrInvalidFitMode, form.Get("fit"))
|
||||
}
|
||||
|
||||
var (
|
||||
expiresAt time.Time
|
||||
expiresAtUnix int64
|
||||
)
|
||||
|
||||
var expiresAt time.Time
|
||||
if ttl > 0 {
|
||||
expiresAt = time.Now().Add(time.Duration(ttl) * time.Second)
|
||||
expiresAtUnix = expiresAt.Unix()
|
||||
}
|
||||
|
||||
payload := &encurl.Payload{
|
||||
SourceHost: parsed.Host,
|
||||
SourcePath: parsed.Path,
|
||||
SourceQuery: parsed.RawQuery,
|
||||
Width: width,
|
||||
Height: height,
|
||||
Format: imgcache.ImageFormat(form.Get("format")),
|
||||
Quality: quality,
|
||||
FitMode: fitMode,
|
||||
ExpiresAt: expiresAtUnix,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: parsed.Host,
|
||||
Path: parsed.Path,
|
||||
Query: parsed.RawQuery,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: width, Height: height},
|
||||
Format: imagerequest.Format(form.Get("format")),
|
||||
Quality: quality,
|
||||
FitMode: fitMode,
|
||||
},
|
||||
Expires: expiresAt,
|
||||
}
|
||||
|
||||
return payload, expiresAt, ttl, nil
|
||||
return req, expiresAt, ttl, nil
|
||||
}
|
||||
|
||||
// parseFormDimension reads an optional width or height form field. An empty
|
||||
// value means "original size" (0). A non-numeric value, or one
|
||||
// imgcache.ValidateDimension rejects, is an error naming the field.
|
||||
// imagerequest.ValidateDimension rejects, is an error naming the field.
|
||||
func parseFormDimension(form url.Values, field string) (int, error) {
|
||||
raw := form.Get(field)
|
||||
if raw == "" {
|
||||
@@ -316,7 +314,7 @@ func parseFormDimension(form url.Values, field string) (int, error) {
|
||||
return 0, fmt.Errorf("%w %s: not a number", errInvalidFormField, field)
|
||||
}
|
||||
|
||||
err = imgcache.ValidateDimension(field, value)
|
||||
err = imagerequest.ValidateDimension(field, value)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
@@ -11,7 +11,7 @@ import (
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// Generator form field names, and a value that is not a number.
|
||||
@@ -125,10 +125,10 @@ func TestGeneratePostRejectsBadField(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestBuildGeneratePayloadDefaultAndLimits verifies that an empty quality
|
||||
// TestBuildGenerateRequestDefaultAndLimits verifies that an empty quality
|
||||
// takes the default, and that the largest accepted width, height, quality and
|
||||
// ttl are accepted with an expiry still in the future.
|
||||
func TestBuildGeneratePayloadDefaultAndLimits(t *testing.T) {
|
||||
func TestBuildGenerateRequestDefaultAndLimits(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
parsed, err := url.Parse(testSourceURL)
|
||||
@@ -136,19 +136,19 @@ func TestBuildGeneratePayloadDefaultAndLimits(t *testing.T) {
|
||||
t.Fatalf("url.Parse() error = %v", err)
|
||||
}
|
||||
|
||||
payload, _, _, err := buildGeneratePayload(parsed, url.Values{})
|
||||
req, _, _, err := buildGenerateRequest(parsed, url.Values{})
|
||||
if err != nil {
|
||||
t.Fatalf("empty form: error = %v", err)
|
||||
}
|
||||
|
||||
if payload.Quality != encurl.DefaultQuality {
|
||||
if req.Transform.Quality != encurl.DefaultQuality {
|
||||
t.Errorf("empty quality gave %d, want %d",
|
||||
payload.Quality, encurl.DefaultQuality)
|
||||
req.Transform.Quality, encurl.DefaultQuality)
|
||||
}
|
||||
|
||||
_, expiresAt, _, err := buildGeneratePayload(parsed, url.Values{
|
||||
widthField: {strconv.Itoa(imgcache.MaxDimension)},
|
||||
heightField: {strconv.Itoa(imgcache.MaxDimension)},
|
||||
_, expiresAt, _, err := buildGenerateRequest(parsed, url.Values{
|
||||
widthField: {strconv.Itoa(imagerequest.MaxDimension)},
|
||||
heightField: {strconv.Itoa(imagerequest.MaxDimension)},
|
||||
qualityField: {strconv.Itoa(maxQuality)},
|
||||
ttlField: {strconv.Itoa(maxTTL)},
|
||||
})
|
||||
|
||||
@@ -10,7 +10,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/session"
|
||||
)
|
||||
|
||||
@@ -314,7 +314,7 @@ func TestGeneratePost_URLServesImage(t *testing.T) {
|
||||
sourceURLField: {"https://" + signedHost + photoPath},
|
||||
widthField: {"50"},
|
||||
heightField: {"50"},
|
||||
formatField: {string(imgcache.FormatJPEG)},
|
||||
formatField: {string(imagerequest.FormatJPEG)},
|
||||
})
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
@@ -347,7 +347,7 @@ func TestGeneratePost_URLWithTTLExpires(t *testing.T) {
|
||||
sourceURLField: {"https://" + signedHost + photoPath},
|
||||
widthField: {"50"},
|
||||
heightField: {"50"},
|
||||
formatField: {string(imgcache.FormatJPEG)},
|
||||
formatField: {string(imagerequest.FormatJPEG)},
|
||||
ttlField: {"1"},
|
||||
})
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError and
|
||||
@@ -26,7 +26,7 @@ func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
|
||||
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
|
||||
|
||||
plain := httptest.NewRecorder()
|
||||
h.respondImageError(plain, &imgcache.ImageRequest{}, err)
|
||||
h.respondImageError(plain, &imagerequest.Request{}, err)
|
||||
|
||||
encrypted := httptest.NewRecorder()
|
||||
h.handleImageError(encrypted, err)
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"github.com/go-chi/chi/v5"
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
)
|
||||
|
||||
@@ -75,12 +76,15 @@ func mismatchedSourcePlainURL(*testing.T, *Handlers) string {
|
||||
func mismatchedSourceEncURL(t *testing.T, h *Handlers) string {
|
||||
t.Helper()
|
||||
|
||||
token, err := h.encGen.Generate(&encurl.Payload{
|
||||
SourceHost: allowlistedHost,
|
||||
SourcePath: mismatchedSourcePath,
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: imgcache.FormatJPEG,
|
||||
token, err := h.encGen.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: allowlistedHost,
|
||||
Path: mismatchedSourcePath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// Errors for an Accept header that an auto URL cannot be served for.
|
||||
@@ -24,9 +24,9 @@ var (
|
||||
// header that is not valid and 406 for one that allows none of the formats,
|
||||
// and reports whether req can be served. Any other format is left as it is.
|
||||
func (s *Handlers) chooseAutoFormat(
|
||||
w http.ResponseWriter, r *http.Request, req *imgcache.ImageRequest,
|
||||
w http.ResponseWriter, r *http.Request, req *imagerequest.Request,
|
||||
) bool {
|
||||
if req.Format != imgcache.FormatAuto {
|
||||
if req.Transform.Format != imagerequest.FormatAuto {
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -45,7 +45,7 @@ func (s *Handlers) chooseAutoFormat(
|
||||
return false
|
||||
}
|
||||
|
||||
req.Format = format
|
||||
req.Transform.Format = format
|
||||
|
||||
return true
|
||||
}
|
||||
@@ -56,26 +56,26 @@ func (s *Handlers) chooseAutoFormat(
|
||||
// specific entry of image/jpeg, image/* and */* allows it, or when it names
|
||||
// nothing. A q of 0 refuses a format. JPEG XL, AVIF and WebP must be named, as
|
||||
// clients that cannot show them send image/* and */* too.
|
||||
func formatForAccept(accept string) (imgcache.ImageFormat, error) {
|
||||
func formatForAccept(accept string) (imagerequest.Format, error) {
|
||||
qualities, err := parseAccept(accept)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
if len(qualities) == 0 {
|
||||
return imgcache.FormatJPEG, nil
|
||||
return imagerequest.FormatJPEG, nil
|
||||
}
|
||||
|
||||
if qualities["image/jxl"] > 0 {
|
||||
return imgcache.FormatJXL, nil
|
||||
return imagerequest.FormatJXL, nil
|
||||
}
|
||||
|
||||
if qualities["image/avif"] > 0 {
|
||||
return imgcache.FormatAVIF, nil
|
||||
return imagerequest.FormatAVIF, nil
|
||||
}
|
||||
|
||||
if qualities["image/webp"] > 0 {
|
||||
return imgcache.FormatWebP, nil
|
||||
return imagerequest.FormatWebP, nil
|
||||
}
|
||||
|
||||
// For JPEG, the most specific entry the header has decides
|
||||
@@ -89,7 +89,7 @@ func formatForAccept(accept string) (imgcache.ImageFormat, error) {
|
||||
}
|
||||
|
||||
if quality > 0 {
|
||||
return imgcache.FormatJPEG, nil
|
||||
return imagerequest.FormatJPEG, nil
|
||||
}
|
||||
|
||||
return "", errNotAcceptable
|
||||
|
||||
@@ -12,8 +12,7 @@ import (
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// The content types the tests below expect.
|
||||
@@ -33,24 +32,24 @@ func TestFormatForAccept(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
accept string
|
||||
want imgcache.ImageFormat
|
||||
want imagerequest.Format
|
||||
wantErr error
|
||||
}{
|
||||
{"AVIF-capable browser",
|
||||
"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
|
||||
imgcache.FormatAVIF, nil},
|
||||
imagerequest.FormatAVIF, nil},
|
||||
{"WebP-capable browser",
|
||||
"image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5",
|
||||
imgcache.FormatWebP, nil},
|
||||
{"WebP only", webpType, imgcache.FormatWebP, nil},
|
||||
{"neither", "image/png,image/*;q=0.8,*/*;q=0.5", imgcache.FormatJPEG, nil},
|
||||
{"wildcard only", "*/*", imgcache.FormatJPEG, nil},
|
||||
{"image wildcard only", "image/*", imgcache.FormatJPEG, nil},
|
||||
{"absent", "", imgcache.FormatJPEG, nil},
|
||||
{"q=0 on AVIF", "image/avif;q=0,image/webp,*/*", imgcache.FormatWebP, nil},
|
||||
imagerequest.FormatWebP, nil},
|
||||
{"WebP only", webpType, imagerequest.FormatWebP, nil},
|
||||
{"neither", "image/png,image/*;q=0.8,*/*;q=0.5", imagerequest.FormatJPEG, nil},
|
||||
{"wildcard only", "*/*", imagerequest.FormatJPEG, nil},
|
||||
{"image wildcard only", "image/*", imagerequest.FormatJPEG, nil},
|
||||
{"absent", "", imagerequest.FormatJPEG, nil},
|
||||
{"q=0 on AVIF", "image/avif;q=0,image/webp,*/*", imagerequest.FormatWebP, nil},
|
||||
{"AVIF named twice, once with q=0", "image/avif,image/avif;q=0.0,*/*",
|
||||
imgcache.FormatJPEG, nil},
|
||||
{"upper case and spaces", " Image/AVIF ; Q=0.5 ", imgcache.FormatAVIF, nil},
|
||||
imagerequest.FormatJPEG, nil},
|
||||
{"upper case and spaces", " Image/AVIF ; Q=0.5 ", imagerequest.FormatAVIF, nil},
|
||||
{"q=0 on JPEG", "image/jpeg;q=0,image/*", "", errNotAcceptable},
|
||||
{"q=0 on everything", "*/*;q=0", "", errNotAcceptable},
|
||||
{"PNG only", "image/png", "", errNotAcceptable},
|
||||
@@ -80,23 +79,30 @@ func TestFormatForAccept(t *testing.T) {
|
||||
func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) {
|
||||
t.Helper()
|
||||
|
||||
signedURL, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
Size: imgcache.Size{Width: 50, Height: 50},
|
||||
Format: imgcache.FormatAuto,
|
||||
signedURL, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatAuto,
|
||||
},
|
||||
}, time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
}
|
||||
|
||||
token, err := h.encGen.Generate(&encurl.Payload{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: imgcache.FormatAuto,
|
||||
ExpiresAt: time.Now().Add(time.Minute).Unix(),
|
||||
token, err := h.encGen.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatAuto,
|
||||
},
|
||||
Expires: time.Now().Add(time.Minute),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
@@ -184,11 +190,15 @@ func TestFormatAuto_SignatureCoversAuto(t *testing.T) {
|
||||
|
||||
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||
|
||||
signedForAVIF, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
Size: imgcache.Size{Width: 50, Height: 50},
|
||||
Format: imgcache.FormatAVIF,
|
||||
signedForAVIF, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatAVIF,
|
||||
},
|
||||
}, time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
|
||||
+19
-21
@@ -13,6 +13,7 @@ import (
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
@@ -35,8 +36,8 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
|
||||
err := s.imgSvc.ValidateRequest(req)
|
||||
if err != nil {
|
||||
s.log.Warn("signature validation failed",
|
||||
"host", req.SourceHost,
|
||||
"path", req.SourcePath,
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"error", err,
|
||||
)
|
||||
s.respondError(w, "unauthorized", http.StatusUnauthorized)
|
||||
@@ -50,7 +51,7 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
|
||||
}
|
||||
|
||||
// Get cache key for logging
|
||||
cacheKey := imgcache.CacheKey(req)
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
|
||||
// Get the image (from cache or fetch/process)
|
||||
startTime := time.Now()
|
||||
@@ -81,17 +82,17 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
|
||||
}
|
||||
|
||||
// parseImageRequest parses the image URL's path and query parameters into
|
||||
// an ImageRequest. On invalid input it writes an error response and
|
||||
// an image request. On invalid input it writes an error response and
|
||||
// returns false.
|
||||
func (s *Handlers) parseImageRequest(
|
||||
w http.ResponseWriter, r *http.Request,
|
||||
) (*imgcache.ImageRequest, bool) {
|
||||
) (*imagerequest.Request, bool) {
|
||||
// The path is parsed decoded once, as r.URL.Path holds it, and not from
|
||||
// chi's wildcard: chi routes on the path as the client wrote it whenever
|
||||
// that is not Go's own encoding of it, so its wildcard would still hold
|
||||
// the %3F before a source query when, say, a "/" in the query is written
|
||||
// as %2F
|
||||
parsed, err := imgcache.ParseImageURL(r.URL.Path)
|
||||
req, err := imagerequest.ParseImageURL(r.URL.Path)
|
||||
if err != nil {
|
||||
s.log.Warn("failed to parse image URL",
|
||||
"path", r.URL.Path,
|
||||
@@ -102,9 +103,6 @@ func (s *Handlers) parseImageRequest(
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Convert to ImageRequest
|
||||
req := parsed.ToImageRequest()
|
||||
|
||||
// Parse signature params from query string. r.URL.Query() would silently
|
||||
// drop a pair it cannot decode, such as q=80%, so that q would be served
|
||||
// at 85; a query string that cannot be decoded is refused instead. A
|
||||
@@ -148,7 +146,7 @@ func (s *Handlers) parseImageRequest(
|
||||
return nil, false
|
||||
}
|
||||
|
||||
req.Quality, err = parseFormInt(query, "q",
|
||||
req.Transform.Quality, err = parseFormInt(query, "q",
|
||||
encurl.DefaultQuality, minQuality, maxQuality)
|
||||
if err != nil {
|
||||
s.respondError(w, fmt.Sprintf("%v, got %q", err, qStr),
|
||||
@@ -160,23 +158,23 @@ func (s *Handlers) parseImageRequest(
|
||||
// Only a fit missing from the URL is cover. A fit in the URL that is not a
|
||||
// fit mode is refused by the fit-mode check below; that check would take an
|
||||
// empty fit as missing, so an empty one is refused here.
|
||||
req.FitMode = imgcache.FitMode(query.Get("fit"))
|
||||
if query.Has("fit") && req.FitMode == "" {
|
||||
req.Transform.FitMode = imagerequest.Fit(query.Get("fit"))
|
||||
if query.Has("fit") && req.Transform.FitMode == "" {
|
||||
s.respondError(w, `invalid fit: not a fit mode, got ""`, http.StatusBadRequest)
|
||||
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Default fit mode if not set
|
||||
if req.FitMode == "" {
|
||||
req.FitMode = imgcache.FitCover
|
||||
if req.Transform.FitMode == "" {
|
||||
req.Transform.FitMode = imagerequest.FitCover
|
||||
}
|
||||
|
||||
// Enforce dimension and fit-mode bounds, shared with the encrypted-URL
|
||||
// route. Dimensions are already bounded by the path parser above; this
|
||||
// also rejects an unrecognized fit mode with 400 instead of letting it
|
||||
// reach the processor as a 500.
|
||||
err = imgcache.ValidateImageRequest(req)
|
||||
err = imagerequest.ValidateImageRequest(req)
|
||||
if err != nil {
|
||||
s.respondError(w, "invalid image request: "+err.Error(),
|
||||
http.StatusBadRequest)
|
||||
@@ -219,7 +217,7 @@ func (s *Handlers) HandleTooManyRequests() http.HandlerFunc {
|
||||
|
||||
// respondImageError maps image retrieval errors to HTTP responses.
|
||||
func (s *Handlers) respondImageError(
|
||||
w http.ResponseWriter, req *imgcache.ImageRequest, err error,
|
||||
w http.ResponseWriter, req *imagerequest.Request, err error,
|
||||
) {
|
||||
// An upstream body whose first bytes are not those of the format its
|
||||
// Content-Type names is the upstream's error, not pixa's: the image
|
||||
@@ -234,8 +232,8 @@ func (s *Handlers) respondImageError(
|
||||
}
|
||||
|
||||
s.log.Error("failed to get image",
|
||||
"host", req.SourceHost,
|
||||
"path", req.SourcePath,
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
@@ -346,8 +344,8 @@ func notModified(w http.ResponseWriter, r *http.Request, etag string) bool {
|
||||
// handling conditional and HEAD requests.
|
||||
func (s *Handlers) writeImageResponse(
|
||||
w http.ResponseWriter, r *http.Request,
|
||||
req *imgcache.ImageRequest, resp *imgcache.ImageResponse,
|
||||
cacheKey imgcache.VariantKey, startTime time.Time,
|
||||
req *imagerequest.Request, resp *imgcache.ImageResponse,
|
||||
cacheKey imagerequest.VariantKey, startTime time.Time,
|
||||
) {
|
||||
// Set response headers
|
||||
w.Header().Set("Content-Type", resp.ContentType)
|
||||
@@ -388,7 +386,7 @@ func (s *Handlers) writeImageResponse(
|
||||
"cache_key", cacheKey,
|
||||
"cache_status", resp.CacheStatus,
|
||||
"duration_ms", duration.Milliseconds(),
|
||||
"format", req.Format,
|
||||
"format", req.Transform.Format,
|
||||
"served_bytes", servedBytes,
|
||||
"fetched_bytes", resp.FetchedBytes,
|
||||
)
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
)
|
||||
|
||||
@@ -107,11 +108,15 @@ func TestHandleImage_SignedURL_MaxAgeEndsAtExp(t *testing.T) {
|
||||
|
||||
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||
|
||||
signedURL, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
Size: imgcache.Size{Width: 50, Height: 50},
|
||||
Format: imgcache.FormatJPEG,
|
||||
signedURL, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
@@ -168,13 +173,13 @@ func TestHandleImageEnc_MaxAge(t *testing.T) {
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
expiresAt int64
|
||||
expires time.Time
|
||||
wantAtLeast int
|
||||
wantAtMost int
|
||||
}{
|
||||
{"60 second TTL", time.Now().Add(time.Minute).Unix(), 50, 60},
|
||||
{"two-year TTL", time.Now().Add(2 * 365 * 24 * time.Hour).Unix(), 31536000, 31536000},
|
||||
{"no TTL", 0, 31536000, 31536000},
|
||||
{"60 second TTL", time.Now().Add(time.Minute), 50, 60},
|
||||
{"two-year TTL", time.Now().Add(2 * 365 * 24 * time.Hour), 31536000, 31536000},
|
||||
{"no TTL", time.Time{}, 31536000, 31536000},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -183,13 +188,16 @@ func TestHandleImageEnc_MaxAge(t *testing.T) {
|
||||
|
||||
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||
|
||||
token, err := h.encGen.Generate(&encurl.Payload{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: imgcache.FormatJPEG,
|
||||
ExpiresAt: tt.expiresAt,
|
||||
token, err := h.encGen.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
Expires: tt.expires,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
@@ -84,15 +85,18 @@ func photoURLWithSig(host, sig string, expires time.Time) string {
|
||||
// photoSignature returns the signature of photoURL(host) at the default
|
||||
// quality and fit, made with key and expiring at expires.
|
||||
func photoSignature(key, host string, expires time.Time) string {
|
||||
return signature.New(key).Sign(&signature.Request{
|
||||
SourceHost: host,
|
||||
SourcePath: photoPath,
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: string(imgcache.FormatJPEG),
|
||||
Quality: 85,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
Expires: expires,
|
||||
return signature.New(key).Sign(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: host,
|
||||
Path: photoPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
Expires: expires,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -353,15 +357,18 @@ func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
|
||||
photos := newPhotoFetcher(t, allowlistedHost, signedHost)
|
||||
exp := time.Now().Add(time.Hour)
|
||||
|
||||
sig := signature.New(testSigningKey).Sign(&signature.Request{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath + "#x",
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: string(imgcache.FormatJPEG),
|
||||
Quality: 85,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
Expires: exp,
|
||||
sig := signature.New(testSigningKey).Sign(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath + "#x",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
Expires: exp,
|
||||
})
|
||||
|
||||
tests := []struct {
|
||||
|
||||
@@ -9,7 +9,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
@@ -44,15 +44,18 @@ func TestHandleImage_SignatureCoversQualityAndFit(t *testing.T) {
|
||||
|
||||
expires := time.Now().Add(time.Hour)
|
||||
signer := signature.New("test-signing-key-must-be-32-chars")
|
||||
sig := signer.Sign(&signature.Request{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: "/images/photo.jpg",
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: string(imgcache.FormatJPEG),
|
||||
Quality: 85,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
Expires: expires,
|
||||
sig := signer.Sign(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: "/images/photo.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
Expires: expires,
|
||||
})
|
||||
signedURL := fmt.Sprintf("/v1/image/%s/images/photo.jpg/50x50.jpeg?sig=%s&exp=%d",
|
||||
signedHost, sig, expires.Unix())
|
||||
@@ -88,9 +91,9 @@ func TestHandleImage_GeneratedSignedURLVerifies(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
quality int
|
||||
fitMode imgcache.FitMode
|
||||
fitMode imagerequest.Fit
|
||||
}{
|
||||
{"quality 40 and fit contain", 40, imgcache.FitContain},
|
||||
{"quality 40 and fit contain", 40, imagerequest.FitContain},
|
||||
{"quality and fit unset", 0, ""},
|
||||
}
|
||||
|
||||
@@ -100,13 +103,17 @@ func TestHandleImage_GeneratedSignedURLVerifies(t *testing.T) {
|
||||
|
||||
fix := setupTestHandler(t)
|
||||
|
||||
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: "/images/photo.jpg",
|
||||
Size: imgcache.Size{Width: 50, Height: 50},
|
||||
Format: imgcache.FormatJPEG,
|
||||
Quality: tt.quality,
|
||||
FitMode: tt.fitMode,
|
||||
signedURL, err := fix.service.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: "/images/photo.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: tt.quality,
|
||||
FitMode: tt.fitMode,
|
||||
},
|
||||
}, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
@@ -196,12 +203,16 @@ func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
|
||||
|
||||
fix := setupTestHandler(t)
|
||||
|
||||
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
SourceQuery: tt.query,
|
||||
Size: imgcache.Size{Width: 50, Height: 50},
|
||||
Format: imgcache.FormatJPEG,
|
||||
signedURL, err := fix.service.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
Query: tt.query,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
@@ -224,16 +235,19 @@ func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
|
||||
|
||||
expires := time.Now().Add(time.Hour)
|
||||
signer := signature.New("test-signing-key-must-be-32-chars")
|
||||
sig := signer.Sign(&signature.Request{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: string(imgcache.FormatJPEG),
|
||||
Quality: 85,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
Expires: expires,
|
||||
sig := signer.Sign(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
Query: "url=https://origin.example.com/my%20cat.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
Expires: expires,
|
||||
})
|
||||
|
||||
tests := []struct {
|
||||
@@ -288,16 +302,19 @@ func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
|
||||
|
||||
expires := time.Now().Add(time.Hour)
|
||||
signer := signature.New("test-signing-key-must-be-32-chars")
|
||||
sig := signer.Sign(&signature.Request{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: tt.sourcePath,
|
||||
SourceQuery: tt.sourceQuery,
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: string(imgcache.FormatJPEG),
|
||||
Quality: 85,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
Expires: expires,
|
||||
sig := signer.Sign(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: tt.sourcePath,
|
||||
Query: tt.sourceQuery,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
Expires: expires,
|
||||
})
|
||||
|
||||
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
|
||||
@@ -314,12 +331,16 @@ func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
|
||||
|
||||
fix := setupTestHandler(t)
|
||||
|
||||
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: tt.sourcePath,
|
||||
SourceQuery: tt.sourceQuery,
|
||||
Size: imgcache.Size{Width: 50, Height: 50},
|
||||
Format: imgcache.FormatJPEG,
|
||||
signedURL, err := fix.service.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: tt.sourcePath,
|
||||
Query: tt.sourceQuery,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
@@ -37,10 +38,11 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
|
||||
|
||||
// Log the request
|
||||
s.log.Debug("encrypted image request",
|
||||
"host", req.SourceHost,
|
||||
"path", req.SourcePath,
|
||||
"dimensions", fmt.Sprintf("%dx%d", req.Size.Width, req.Size.Height),
|
||||
"format", req.Format,
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"dimensions", fmt.Sprintf("%dx%d",
|
||||
req.Transform.Size.Width, req.Transform.Size.Height),
|
||||
"format", req.Transform.Format,
|
||||
)
|
||||
|
||||
// Fetch and process the image (no signature validation
|
||||
@@ -88,10 +90,10 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
|
||||
duration := time.Since(start)
|
||||
s.log.Info("image served",
|
||||
"request_id", middleware.GetReqID(ctx),
|
||||
"cache_key", imgcache.CacheKey(req),
|
||||
"host", req.SourceHost,
|
||||
"path", req.SourcePath,
|
||||
"format", req.Format,
|
||||
"cache_key", imagerequest.CacheKey(req),
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"format", req.Transform.Format,
|
||||
"cache_status", resp.CacheStatus,
|
||||
"served_bytes", written,
|
||||
"duration_ms", duration.Milliseconds(),
|
||||
@@ -100,12 +102,12 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
|
||||
}
|
||||
|
||||
// parseImageEncRequest decrypts the token of an encrypted image URL into an
|
||||
// ImageRequest and checks it. On a token that is missing, does not decrypt,
|
||||
// image request and checks it. On a token that is missing, does not decrypt,
|
||||
// has expired or asks for something not valid, it writes an error response
|
||||
// and returns false.
|
||||
func (s *Handlers) parseImageEncRequest(
|
||||
w http.ResponseWriter, r *http.Request,
|
||||
) (*imgcache.ImageRequest, bool) {
|
||||
) (*imagerequest.Request, bool) {
|
||||
// Extract token from URL
|
||||
token := chi.URLParam(r, "token")
|
||||
if token == "" {
|
||||
@@ -115,7 +117,7 @@ func (s *Handlers) parseImageEncRequest(
|
||||
}
|
||||
|
||||
// Decrypt and validate the payload
|
||||
payload, err := s.encGen.Parse(token)
|
||||
req, err := s.encGen.Parse(token)
|
||||
if err != nil {
|
||||
if errors.Is(err, encurl.ErrExpired) {
|
||||
s.log.Debug("encrypted URL expired", "error", err)
|
||||
@@ -130,14 +132,11 @@ func (s *Handlers) parseImageEncRequest(
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Convert payload to ImageRequest
|
||||
req := payload.ToImageRequest()
|
||||
|
||||
// Apply the same dimension and fit-mode bounds as the plain image
|
||||
// route: a sealed payload is trusted for its origin, not for staying
|
||||
// within limits, so an over-limit size or unknown fit mode is a 400
|
||||
// here rather than an out-of-memory or a 500 from the processor.
|
||||
err = imgcache.ValidateImageRequest(req)
|
||||
err = imagerequest.ValidateImageRequest(req)
|
||||
if err != nil {
|
||||
s.log.Debug("encrypted URL failed validation", "error", err)
|
||||
s.respondError(w, "invalid encrypted URL: "+err.Error(),
|
||||
|
||||
@@ -10,7 +10,7 @@ import (
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// newEncTestServer builds a router serving the encrypted-URL route with a
|
||||
@@ -54,11 +54,14 @@ func TestHandleImageEnc_OverLimitDimension_Returns400(t *testing.T) {
|
||||
|
||||
encGen, srv := newEncTestServer(t)
|
||||
|
||||
token, err := encGen.Generate(&encurl.Payload{
|
||||
SourceHost: "cdn.example.com",
|
||||
SourcePath: "/photo.jpg",
|
||||
Width: 100000,
|
||||
Height: 100000,
|
||||
token, err := encGen.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: "cdn.example.com",
|
||||
Path: "/photo.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 100000, Height: 100000},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
@@ -79,12 +82,15 @@ func TestHandleImageEnc_InvalidFitMode_Returns400(t *testing.T) {
|
||||
|
||||
encGen, srv := newEncTestServer(t)
|
||||
|
||||
token, err := encGen.Generate(&encurl.Payload{
|
||||
SourceHost: "cdn.example.com",
|
||||
SourcePath: "/photo.jpg",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
FitMode: imgcache.FitMode("bogus"),
|
||||
token, err := encGen.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: "cdn.example.com",
|
||||
Path: "/photo.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
FitMode: imagerequest.Fit("bogus"),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// requireServedPhoto requires that rec answers 200 with the JPEG at photoPath
|
||||
@@ -59,11 +60,14 @@ func TestHandleImageEnc_RejectedToken(t *testing.T) {
|
||||
|
||||
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||
|
||||
photo := encurl.Payload{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
photo := imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
},
|
||||
}
|
||||
|
||||
valid, err := h.encGen.Generate(&photo)
|
||||
@@ -72,7 +76,7 @@ func TestHandleImageEnc_RejectedToken(t *testing.T) {
|
||||
}
|
||||
|
||||
expiredPhoto := photo
|
||||
expiredPhoto.ExpiresAt = time.Now().Add(-time.Minute).Unix()
|
||||
expiredPhoto.Expires = time.Now().Add(-time.Minute)
|
||||
|
||||
expired, err := h.encGen.Generate(&expiredPhoto)
|
||||
if err != nil {
|
||||
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
"github.com/davidbyttow/govips/v2/vips"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
@@ -58,15 +58,18 @@ func TestImageWithoutFormat_SignedAsJXL(t *testing.T) {
|
||||
route := newImageRoute(t, newPhotoFetcher(t, signedHost))
|
||||
expires := time.Now().Add(time.Hour)
|
||||
|
||||
sig := signature.New(testSigningKey).Sign(&signature.Request{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: string(imgcache.FormatJXL),
|
||||
Quality: encurl.DefaultQuality,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
Expires: expires,
|
||||
sig := signature.New(testSigningKey).Sign(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJXL,
|
||||
Quality: encurl.DefaultQuality,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
Expires: expires,
|
||||
})
|
||||
query := fmt.Sprintf("?sig=%s&exp=%d", sig, expires.Unix())
|
||||
|
||||
@@ -83,11 +86,14 @@ func TestImageEncWithoutFormat_ServesJPEGXL(t *testing.T) {
|
||||
|
||||
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
|
||||
|
||||
token, err := h.encGen.Generate(&encurl.Payload{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
token, err := h.encGen.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
|
||||
"github.com/davidbyttow/govips/v2/vips"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// jxlType is the content type of JPEG XL.
|
||||
@@ -22,14 +22,14 @@ func TestFormatForAccept_JPEGXL(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
accept string
|
||||
want imgcache.ImageFormat
|
||||
want imagerequest.Format
|
||||
}{
|
||||
{"JPEG XL-capable browser",
|
||||
"image/jxl,image/avif,image/webp,image/*,*/*;q=0.8", imgcache.FormatJXL},
|
||||
{"JPEG XL only", jxlType, imgcache.FormatJXL},
|
||||
"image/jxl,image/avif,image/webp,image/*,*/*;q=0.8", imagerequest.FormatJXL},
|
||||
{"JPEG XL only", jxlType, imagerequest.FormatJXL},
|
||||
{"JPEG XL with a lower q than AVIF", "image/avif,image/jxl;q=0.5",
|
||||
imgcache.FormatJXL},
|
||||
{"q=0 on JPEG XL", "image/jxl;q=0,image/avif,*/*", imgcache.FormatAVIF},
|
||||
imagerequest.FormatJXL},
|
||||
{"q=0 on JPEG XL", "image/jxl;q=0,image/avif,*/*", imagerequest.FormatAVIF},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
|
||||
@@ -13,8 +13,7 @@ import (
|
||||
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// signedPhotoURL returns a signed /v1/image/ URL, valid for a minute, for the
|
||||
@@ -22,11 +21,15 @@ import (
|
||||
func signedPhotoURL(t *testing.T, h *Handlers) string {
|
||||
t.Helper()
|
||||
|
||||
signedURL, err := h.imgSvc.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
Size: imgcache.Size{Width: 50, Height: 50},
|
||||
Format: imgcache.FormatJPEG,
|
||||
signedURL, err := h.imgSvc.GenerateSignedURL("", &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
}, time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
@@ -40,12 +43,15 @@ func signedPhotoURL(t *testing.T, h *Handlers) string {
|
||||
func encPhotoURL(t *testing.T, h *Handlers) string {
|
||||
t.Helper()
|
||||
|
||||
token, err := h.encGen.Generate(&encurl.Payload{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: imgcache.FormatJPEG,
|
||||
token, err := h.encGen.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: signedHost,
|
||||
Path: photoPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate() error = %v", err)
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
)
|
||||
|
||||
@@ -21,7 +22,9 @@ func TestServerBusyAnswers503(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := &Handlers{log: slog.New(slog.DiscardHandler)}
|
||||
req := &imgcache.ImageRequest{SourceHost: "img.example.com", SourcePath: "/a.jpg"}
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: "img.example.com", Path: "/a.jpg"},
|
||||
}
|
||||
|
||||
for _, err := range []error{
|
||||
fmt.Errorf("upstream fetch failed: %w", httpfetcher.ErrTooManyConnections),
|
||||
@@ -64,7 +67,7 @@ func TestProcessingTimedOutAnswers503(t *testing.T) {
|
||||
imgcache.ErrProcessingTimedOut, context.DeadlineExceeded)
|
||||
|
||||
plain := httptest.NewRecorder()
|
||||
h.respondImageError(plain, &imgcache.ImageRequest{}, err)
|
||||
h.respondImageError(plain, &imagerequest.Request{}, err)
|
||||
|
||||
encrypted := httptest.NewRecorder()
|
||||
h.handleImageError(encrypted, err)
|
||||
|
||||
@@ -10,7 +10,7 @@ import (
|
||||
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestUnsupportedSourceFormatAnswers415 checks that both image routes answer
|
||||
@@ -21,7 +21,9 @@ func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := &Handlers{log: slog.New(slog.DiscardHandler)}
|
||||
req := &imgcache.ImageRequest{SourceHost: "img.example.com", SourcePath: "/a.svg"}
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: "img.example.com", Path: "/a.svg"},
|
||||
}
|
||||
|
||||
for _, err := range []error{
|
||||
fmt.Errorf("upstream fetch failed: %w: image/svg+xml",
|
||||
|
||||
@@ -6,6 +6,8 @@ import (
|
||||
"errors"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestImageProcessor_RefusesAVIFAboveMaxPixels verifies that an AVIF output
|
||||
@@ -37,22 +39,30 @@ func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input []byte
|
||||
format Format
|
||||
format imagerequest.Format
|
||||
width, height int
|
||||
fit FitMode
|
||||
fit imagerequest.Fit
|
||||
refused bool
|
||||
}{
|
||||
{"AVIF at the limit", cutOff, FormatAVIF, 4096, 4096, FitCover, false},
|
||||
{"AVIF above the limit", cutOff, FormatAVIF, 4096, 4097, FitCover, true},
|
||||
{"WebP above the AVIF limit", cutOff, FormatWebP, 4096, 4097, FitCover, false},
|
||||
{"AVIF at the limit", cutOff, imagerequest.FormatAVIF,
|
||||
4096, 4096, imagerequest.FitCover, false},
|
||||
{"AVIF above the limit", cutOff, imagerequest.FormatAVIF,
|
||||
4096, 4097, imagerequest.FitCover, true},
|
||||
{"WebP above the AVIF limit", cutOff, imagerequest.FormatWebP,
|
||||
4096, 4097, imagerequest.FitCover, false},
|
||||
// FitInside does not enlarge, so the output stays 64x32
|
||||
{"AVIF inside a larger size", cutOff, FormatAVIF, 8192, 8192, FitInside, false},
|
||||
{"AVIF inside a larger size", cutOff, imagerequest.FormatAVIF,
|
||||
8192, 8192, imagerequest.FitInside, false},
|
||||
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
|
||||
{"AVIF outside a smaller size", cutOff, FormatAVIF, 8192, 1, FitOutside, true},
|
||||
{"AVIF outside a smaller size", cutOff, imagerequest.FormatAVIF,
|
||||
8192, 1, imagerequest.FitOutside, true},
|
||||
// With no size, the output keeps the source's 4097x4096
|
||||
{"AVIF of the source's size", largeAVIF, FormatAVIF, 0, 0, FitCover, true},
|
||||
{"orig of an AVIF source", largeAVIF, FormatOriginal, 0, 0, FitCover, true},
|
||||
{"JPEG of the source's size", largeAVIF, FormatJPEG, 0, 0, FitCover, false},
|
||||
{"AVIF of the source's size", largeAVIF, imagerequest.FormatAVIF,
|
||||
0, 0, imagerequest.FitCover, true},
|
||||
{"orig of an AVIF source", largeAVIF, imagerequest.FormatOriginal,
|
||||
0, 0, imagerequest.FitCover, true},
|
||||
{"JPEG of the source's size", largeAVIF, imagerequest.FormatJPEG,
|
||||
0, 0, imagerequest.FitCover, false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -60,8 +70,8 @@ func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := New(Params{}).Process(t.Context(), bytes.NewReader(tt.input),
|
||||
&Request{
|
||||
Size: Size{Width: tt.width, Height: tt.height},
|
||||
&imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: tt.width, Height: tt.height},
|
||||
Format: tt.format,
|
||||
FitMode: tt.fit,
|
||||
})
|
||||
|
||||
@@ -4,6 +4,8 @@ import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestImageProcessor_RefusesComputedSideAboveMaximum verifies that a size
|
||||
@@ -17,13 +19,17 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
inputWidth, inputHeight int
|
||||
size Size
|
||||
size imagerequest.Size
|
||||
wantErr error
|
||||
}{
|
||||
{"width at the maximum", 4096, 1, Size{Height: 2}, nil},
|
||||
{"width above the maximum", 4096, 1, Size{Height: 3}, ErrOutputTooLarge},
|
||||
{"height at the maximum", 1, 4096, Size{Width: 2}, nil},
|
||||
{"height above the maximum", 1, 4096, Size{Width: 3}, ErrOutputTooLarge},
|
||||
{"width at the maximum", 4096, 1,
|
||||
imagerequest.Size{Height: 2}, nil},
|
||||
{"width above the maximum", 4096, 1,
|
||||
imagerequest.Size{Height: 3}, ErrOutputTooLarge},
|
||||
{"height at the maximum", 1, 4096,
|
||||
imagerequest.Size{Width: 2}, nil},
|
||||
{"height above the maximum", 1, 4096,
|
||||
imagerequest.Size{Width: 3}, ErrOutputTooLarge},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -34,7 +40,11 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitCover})
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
FitMode: imagerequest.FitCover,
|
||||
})
|
||||
if result != nil {
|
||||
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
||||
|
||||
@@ -64,18 +74,21 @@ func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
inputWidth, inputHeight int
|
||||
size Size
|
||||
size imagerequest.Size
|
||||
wantWidth, wantHeight int
|
||||
}{
|
||||
{"8192x1 at 100x0", 8192, 1, Size{Width: 100}, 100, 1},
|
||||
{"8192x1 at 1300x0", 8192, 1, Size{Width: 1300}, 1300, 1},
|
||||
{"8192x2 at 100x0", 8192, 2, Size{Width: 100}, 100, 1},
|
||||
{"1x8192 at 0x100", 1, 8192, Size{Height: 100}, 1, 100},
|
||||
{"1x8192 at 0x1300", 1, 8192, Size{Height: 1300}, 1, 1300},
|
||||
{"2x8192 at 0x100", 2, 8192, Size{Height: 100}, 1, 100},
|
||||
{"8192x1 at 100x0", 8192, 1, imagerequest.Size{Width: 100}, 100, 1},
|
||||
{"8192x1 at 1300x0", 8192, 1, imagerequest.Size{Width: 1300}, 1300, 1},
|
||||
{"8192x2 at 100x0", 8192, 2, imagerequest.Size{Width: 100}, 100, 1},
|
||||
{"1x8192 at 0x100", 1, 8192, imagerequest.Size{Height: 100}, 1, 100},
|
||||
{"1x8192 at 0x1300", 1, 8192, imagerequest.Size{Height: 1300}, 1, 1300},
|
||||
{"2x8192 at 0x100", 2, 8192, imagerequest.Size{Height: 100}, 1, 100},
|
||||
}
|
||||
|
||||
fits := []FitMode{FitCover, FitContain, FitFill, FitInside, FitOutside}
|
||||
fits := []imagerequest.Fit{
|
||||
imagerequest.FitCover, imagerequest.FitContain, imagerequest.FitFill,
|
||||
imagerequest.FitInside, imagerequest.FitOutside,
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
for _, fit := range fits {
|
||||
@@ -93,7 +106,11 @@ func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: fit})
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
FitMode: fit,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -103,14 +120,14 @@ func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
|
||||
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
||||
|
||||
ok := result.Width == tt.wantWidth && result.Height == tt.wantHeight
|
||||
if fit == FitOutside {
|
||||
if fit == imagerequest.FitOutside {
|
||||
ok = result.Width >= tt.wantWidth && result.Height >= tt.wantHeight
|
||||
}
|
||||
|
||||
if !ok {
|
||||
t.Errorf("Process() made %dx%d, want %dx%d (at least, for %s)",
|
||||
result.Width, result.Height, tt.wantWidth, tt.wantHeight,
|
||||
FitOutside)
|
||||
imagerequest.FitOutside)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -86,7 +86,7 @@ func checkProcessStopsAtDeadline(
|
||||
|
||||
proc := New(Params{MaxConcurrentProcessing: 1})
|
||||
|
||||
result, err := proc.Process(ctx, input, smallJPEGRequest())
|
||||
result, err := proc.Process(ctx, input, smallJPEGTransform())
|
||||
if result != nil {
|
||||
t.Logf("Process() made a %dx%d image after its deadline passed",
|
||||
result.Width, result.Height)
|
||||
|
||||
@@ -4,6 +4,8 @@ import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestImageProcessor_EmptyFormatRefused verifies that a request with no format
|
||||
@@ -13,7 +15,8 @@ func TestImageProcessor_EmptyFormatRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := New(Params{}).Process(
|
||||
t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)), &Request{},
|
||||
t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)),
|
||||
&imagerequest.Transform{},
|
||||
)
|
||||
if !errors.Is(err, ErrUnsupportedOutputFormat) {
|
||||
t.Errorf("Process() error = %v, want %v", err, ErrUnsupportedOutputFormat)
|
||||
|
||||
@@ -8,14 +8,17 @@ import (
|
||||
"testing"
|
||||
|
||||
"github.com/davidbyttow/govips/v2/vips"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// processedSize runs input through Process and returns the output's size in
|
||||
// bytes.
|
||||
func processedSize(t *testing.T, input []byte, req *Request) int64 {
|
||||
func processedSize(
|
||||
t *testing.T, input []byte, transform *imagerequest.Transform,
|
||||
) int64 {
|
||||
t.Helper()
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(), bytes.NewReader(input), req)
|
||||
result, err := New(Params{}).Process(t.Context(), bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -68,7 +71,8 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
size := processedSize(t, encodePNG(t, flat), &Request{Format: FormatPNG})
|
||||
size := processedSize(t, encodePNG(t, flat),
|
||||
&imagerequest.Transform{Format: imagerequest.FormatPNG})
|
||||
|
||||
const rawBytes = width * height * 3
|
||||
if size > rawBytes/10 {
|
||||
@@ -85,7 +89,8 @@ func TestImageProcessor_WebPAtEffort2(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
|
||||
size := processedSize(t, input, &Request{Format: FormatWebP, Quality: 85})
|
||||
size := processedSize(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatWebP, Quality: 85})
|
||||
|
||||
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
|
||||
StripMetadata: true,
|
||||
@@ -126,7 +131,8 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
|
||||
|
||||
input := encodePNG(t, source)
|
||||
|
||||
size := processedSize(t, input, &Request{Format: FormatAVIF, Quality: 85})
|
||||
size := processedSize(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatAVIF, Quality: 85})
|
||||
|
||||
want, _, err := decode(t, input).ExportAvif(&vips.AvifExportParams{
|
||||
StripMetadata: true,
|
||||
@@ -152,7 +158,8 @@ func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
|
||||
size := processedSize(t, input, &Request{Format: FormatJXL, Quality: 85})
|
||||
size := processedSize(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: 85})
|
||||
|
||||
params := vips.NewJxlExportParams()
|
||||
params.Quality = 85
|
||||
|
||||
@@ -7,6 +7,8 @@ import (
|
||||
"image/png"
|
||||
"math"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestImageProcessor_FitCoverServesThinSource verifies that FitCover serves
|
||||
@@ -20,12 +22,12 @@ func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
inputWidth, inputHeight int
|
||||
size Size
|
||||
size imagerequest.Size
|
||||
}{
|
||||
{"8192x1 at 8192x8192", 8192, 1, Size{Width: 8192, Height: 8192}},
|
||||
{"8192x1 at 1x8192", 8192, 1, Size{Width: 1, Height: 8192}},
|
||||
{"1x8192 at 8192x8192", 1, 8192, Size{Width: 8192, Height: 8192}},
|
||||
{"1x8192 at 8192x1", 1, 8192, Size{Width: 8192, Height: 1}},
|
||||
{"8192x1 at 8192x8192", 8192, 1, imagerequest.Size{Width: 8192, Height: 8192}},
|
||||
{"8192x1 at 1x8192", 8192, 1, imagerequest.Size{Width: 1, Height: 8192}},
|
||||
{"1x8192 at 8192x8192", 1, 8192, imagerequest.Size{Width: 8192, Height: 8192}},
|
||||
{"1x8192 at 8192x1", 1, 8192, imagerequest.Size{Width: 8192, Height: 1}},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -36,7 +38,11 @@ func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitCover})
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
FitMode: imagerequest.FitCover,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -102,10 +108,10 @@ func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatPNG,
|
||||
FitMode: FitCover,
|
||||
&imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatPNG,
|
||||
FitMode: imagerequest.FitCover,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
@@ -208,14 +214,14 @@ func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
side, squareSide int
|
||||
size Size
|
||||
size imagerequest.Size
|
||||
}{
|
||||
{"16x16 at 100x60", 16, 8, Size{Width: 100, Height: 60}},
|
||||
{"16x16 at 60x100", 16, 8, Size{Width: 60, Height: 100}},
|
||||
{"10x10 at 100x35", 10, 2, Size{Width: 100, Height: 35}},
|
||||
{"10x10 at 35x100", 10, 2, Size{Width: 35, Height: 100}},
|
||||
{"64x64 at 300x100", 64, 16, Size{Width: 300, Height: 100}},
|
||||
{"64x64 at 100x300", 64, 16, Size{Width: 100, Height: 300}},
|
||||
{"16x16 at 100x60", 16, 8, imagerequest.Size{Width: 100, Height: 60}},
|
||||
{"16x16 at 60x100", 16, 8, imagerequest.Size{Width: 60, Height: 100}},
|
||||
{"10x10 at 100x35", 10, 2, imagerequest.Size{Width: 100, Height: 35}},
|
||||
{"10x10 at 35x100", 10, 2, imagerequest.Size{Width: 35, Height: 100}},
|
||||
{"64x64 at 300x100", 64, 16, imagerequest.Size{Width: 300, Height: 100}},
|
||||
{"64x64 at 100x300", 64, 16, imagerequest.Size{Width: 100, Height: 300}},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -226,7 +232,11 @@ func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatPNG, FitMode: FitCover})
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatPNG,
|
||||
FitMode: imagerequest.FitCover,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -4,6 +4,8 @@ import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestImageProcessor_RefusesFitOutsideAboveMaximum verifies that FitOutside,
|
||||
@@ -17,15 +19,20 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
inputWidth, inputHeight int
|
||||
size Size
|
||||
size imagerequest.Size
|
||||
// wantWidth and wantHeight are the output's, 0 for a refusal
|
||||
wantWidth, wantHeight int
|
||||
}{
|
||||
{"width above the maximum", 3000, 600, Size{Width: 1, Height: 8192}, 0, 0},
|
||||
{"height above the maximum", 600, 3000, Size{Width: 8192, Height: 1}, 0, 0},
|
||||
{"width at the maximum", 1024, 256, Size{Width: 1, Height: 2048}, 8192, 2048},
|
||||
{"height at the maximum", 256, 1024, Size{Width: 2048, Height: 1}, 2048, 8192},
|
||||
{"within the maximum", 3000, 600, Size{Width: 1, Height: 1200}, 6000, 1200},
|
||||
{"width above the maximum", 3000, 600,
|
||||
imagerequest.Size{Width: 1, Height: 8192}, 0, 0},
|
||||
{"height above the maximum", 600, 3000,
|
||||
imagerequest.Size{Width: 8192, Height: 1}, 0, 0},
|
||||
{"width at the maximum", 1024, 256,
|
||||
imagerequest.Size{Width: 1, Height: 2048}, 8192, 2048},
|
||||
{"height at the maximum", 256, 1024,
|
||||
imagerequest.Size{Width: 2048, Height: 1}, 2048, 8192},
|
||||
{"within the maximum", 3000, 600,
|
||||
imagerequest.Size{Width: 1, Height: 1200}, 6000, 1200},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -36,7 +43,11 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitOutside})
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
FitMode: imagerequest.FitOutside,
|
||||
})
|
||||
if result != nil {
|
||||
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
||||
}
|
||||
@@ -71,11 +82,11 @@ func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
inputWidth, inputHeight int
|
||||
size Size
|
||||
size imagerequest.Size
|
||||
wantWidth, wantHeight int
|
||||
}{
|
||||
{"49x8192 at 0x100", 49, 8192, Size{Height: 100}, 1, 167},
|
||||
{"49x49 at 1x1", 49, 49, Size{Width: 1, Height: 1}, 1, 1},
|
||||
{"49x8192 at 0x100", 49, 8192, imagerequest.Size{Height: 100}, 1, 167},
|
||||
{"49x49 at 1x1", 49, 49, imagerequest.Size{Width: 1, Height: 1}, 1, 1},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -86,7 +97,11 @@ func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitOutside})
|
||||
&imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
FitMode: imagerequest.FitOutside,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
|
||||
"github.com/davidbyttow/govips/v2/vips"
|
||||
"sneak.berlin/go/pixa/internal/config"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// vipsOnce ensures vips is initialized exactly once.
|
||||
@@ -54,49 +55,6 @@ func CheckJPEGXLSupport() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Format represents supported output image formats.
|
||||
type Format string
|
||||
|
||||
// Supported image output formats.
|
||||
const (
|
||||
FormatOriginal Format = "orig"
|
||||
FormatJPEG Format = "jpeg"
|
||||
FormatPNG Format = "png"
|
||||
FormatWebP Format = "webp"
|
||||
FormatAVIF Format = "avif"
|
||||
FormatJXL Format = "jxl"
|
||||
FormatGIF Format = "gif"
|
||||
)
|
||||
|
||||
// FitMode represents how to fit an image into requested dimensions.
|
||||
type FitMode string
|
||||
|
||||
// Supported image fit modes.
|
||||
const (
|
||||
FitCover FitMode = "cover"
|
||||
FitContain FitMode = "contain"
|
||||
FitFill FitMode = "fill"
|
||||
FitInside FitMode = "inside"
|
||||
FitOutside FitMode = "outside"
|
||||
)
|
||||
|
||||
// ErrInvalidFitMode is returned when an invalid fit mode is provided.
|
||||
var ErrInvalidFitMode = errors.New("invalid fit mode")
|
||||
|
||||
// Size represents requested image dimensions.
|
||||
type Size struct {
|
||||
Width int
|
||||
Height int
|
||||
}
|
||||
|
||||
// Request holds the parameters for image processing.
|
||||
type Request struct {
|
||||
Size Size
|
||||
Format Format
|
||||
Quality int
|
||||
FitMode FitMode
|
||||
}
|
||||
|
||||
// Result contains the output of image processing.
|
||||
type Result struct {
|
||||
// Content is the processed image data.
|
||||
@@ -133,8 +91,8 @@ var ErrInputTooLarge = errors.New("input image dimensions exceed maximum")
|
||||
var ErrInputDataTooLarge = errors.New("input data exceeds maximum allowed size")
|
||||
|
||||
// MaxOutputDimension is the maximum width or height of an output image, and
|
||||
// so of a requested size (imgcache.MaxDimension).
|
||||
const MaxOutputDimension = 8192
|
||||
// so of a requested size (imagerequest.MaxDimension).
|
||||
const MaxOutputDimension = imagerequest.MaxDimension
|
||||
|
||||
// ErrOutputTooLarge is returned, before resizing, when the output would have a
|
||||
// side above MaxOutputDimension: the side of a size with one side 0, computed
|
||||
@@ -217,7 +175,7 @@ func New(params Params) *ImageProcessor {
|
||||
}
|
||||
}
|
||||
|
||||
// Process transforms an image according to the request. When
|
||||
// Process transforms an image according to transform. When
|
||||
// MaxConcurrentProcessing images are already being processed, it waits up
|
||||
// to ProcessingWaitTimeout for one to finish, then fails with
|
||||
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
|
||||
@@ -228,7 +186,7 @@ func New(params Params) *ImageProcessor {
|
||||
func (p *ImageProcessor) Process(
|
||||
ctx context.Context,
|
||||
input io.Reader,
|
||||
req *Request,
|
||||
transform *imagerequest.Transform,
|
||||
) (*Result, error) {
|
||||
release, err := p.acquireSlot(ctx)
|
||||
if err != nil {
|
||||
@@ -277,35 +235,36 @@ func (p *ImageProcessor) Process(
|
||||
}
|
||||
|
||||
// Determine target dimensions; too large ones are refused before resizing
|
||||
targetWidth, targetHeight, err := targetDimensions(req.Size, origWidth, origHeight)
|
||||
targetWidth, targetHeight, err := targetDimensions(transform.Size,
|
||||
origWidth, origHeight)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// orig is the source's own format; encode refuses an empty format
|
||||
outputFormat := req.Format
|
||||
if outputFormat == FormatOriginal {
|
||||
outputFormat := transform.Format
|
||||
if outputFormat == imagerequest.FormatOriginal {
|
||||
outputFormat = sourceFormat
|
||||
}
|
||||
|
||||
// Determine output dimensions; too large ones are refused before
|
||||
// resizing, which with FitCover already decodes the input
|
||||
outputWidth, outputHeight, err := outputDimensions(origWidth, origHeight,
|
||||
targetWidth, targetHeight, req.FitMode, outputFormat)
|
||||
targetWidth, targetHeight, transform.FitMode, outputFormat)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Resize if needed
|
||||
if targetWidth != origWidth || targetHeight != origHeight {
|
||||
err := p.resize(img, outputWidth, outputHeight, req.FitMode)
|
||||
err := p.resize(img, outputWidth, outputHeight, transform.FitMode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to resize: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Encode to target format
|
||||
output, err := p.encode(ctx, img, outputFormat, req.Quality)
|
||||
output, err := p.encode(ctx, img, outputFormat, transform.Quality)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to encode: %w", err)
|
||||
}
|
||||
@@ -328,7 +287,9 @@ func (p *ImageProcessor) Process(
|
||||
// rounded down but kept at 1 or more, as for a thin image it would round
|
||||
// down to 0, a side of no pixels. It can come out above MaxOutputDimension,
|
||||
// which is ErrOutputTooLarge.
|
||||
func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
|
||||
func targetDimensions(
|
||||
size imagerequest.Size, origWidth, origHeight int,
|
||||
) (int, int, error) {
|
||||
width, height := size.Width, size.Height
|
||||
|
||||
switch {
|
||||
@@ -374,33 +335,33 @@ func (p *ImageProcessor) SupportedInputFormats() []string {
|
||||
}
|
||||
|
||||
// SupportedOutputFormats returns formats this processor can write.
|
||||
func (p *ImageProcessor) SupportedOutputFormats() []Format {
|
||||
return []Format{
|
||||
FormatJPEG,
|
||||
FormatPNG,
|
||||
FormatGIF,
|
||||
FormatWebP,
|
||||
FormatAVIF,
|
||||
FormatJXL,
|
||||
func (p *ImageProcessor) SupportedOutputFormats() []imagerequest.Format {
|
||||
return []imagerequest.Format{
|
||||
imagerequest.FormatJPEG,
|
||||
imagerequest.FormatPNG,
|
||||
imagerequest.FormatGIF,
|
||||
imagerequest.FormatWebP,
|
||||
imagerequest.FormatAVIF,
|
||||
imagerequest.FormatJXL,
|
||||
}
|
||||
}
|
||||
|
||||
// FormatToMIME converts a Format to its MIME type string.
|
||||
func FormatToMIME(format Format) string {
|
||||
func FormatToMIME(format imagerequest.Format) string {
|
||||
switch format {
|
||||
case FormatJPEG:
|
||||
case imagerequest.FormatJPEG:
|
||||
return mimeJPEG
|
||||
case FormatPNG:
|
||||
case imagerequest.FormatPNG:
|
||||
return mimePNG
|
||||
case FormatWebP:
|
||||
case imagerequest.FormatWebP:
|
||||
return mimeWebP
|
||||
case FormatGIF:
|
||||
case imagerequest.FormatGIF:
|
||||
return mimeGIF
|
||||
case FormatAVIF:
|
||||
case imagerequest.FormatAVIF:
|
||||
return mimeAVIF
|
||||
case FormatJXL:
|
||||
case imagerequest.FormatJXL:
|
||||
return mimeJXL
|
||||
case FormatOriginal:
|
||||
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
|
||||
return "application/octet-stream"
|
||||
default:
|
||||
return "application/octet-stream"
|
||||
@@ -487,9 +448,9 @@ func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
|
||||
case vips.ImageTypeWEBP:
|
||||
return "webp"
|
||||
case vips.ImageTypeAVIF, vips.ImageTypeHEIF:
|
||||
return string(FormatAVIF)
|
||||
return string(imagerequest.FormatAVIF)
|
||||
case vips.ImageTypeJXL:
|
||||
return string(FormatJXL)
|
||||
return string(imagerequest.FormatJXL)
|
||||
case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF,
|
||||
vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP,
|
||||
vips.ImageTypeJP2K:
|
||||
@@ -506,17 +467,18 @@ func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
|
||||
// outputDimensions returns ErrOutputTooLarge. An AVIF output of more than
|
||||
// MaxAVIFOutputPixels is ErrAVIFOutputTooLarge.
|
||||
func outputDimensions(
|
||||
imgWidth, imgHeight, width, height int, fit FitMode, format Format,
|
||||
imgWidth, imgHeight, width, height int,
|
||||
fit imagerequest.Fit, format imagerequest.Format,
|
||||
) (int, int, error) {
|
||||
scaleW := float64(width) / float64(imgWidth)
|
||||
scaleH := float64(height) / float64(imgHeight)
|
||||
outputWidth, outputHeight := width, height
|
||||
|
||||
switch fit {
|
||||
case FitCover, FitFill, "":
|
||||
case imagerequest.FitCover, imagerequest.FitFill, "":
|
||||
// Cover crops, and fill stretches, to exactly the target dimensions
|
||||
|
||||
case FitContain:
|
||||
case imagerequest.FitContain:
|
||||
// Fit within the target dimensions, maintaining aspect ratio. Scaled
|
||||
// by the smaller scale, the short side of a thin image can round
|
||||
// down to 0, which libvips refuses as a width, so each side is at
|
||||
@@ -525,13 +487,13 @@ func outputDimensions(
|
||||
outputWidth = max(1, int(float64(imgWidth)*scale))
|
||||
outputHeight = max(1, int(float64(imgHeight)*scale))
|
||||
|
||||
case FitInside:
|
||||
case imagerequest.FitInside:
|
||||
// Same as contain, but only shrink
|
||||
scale := min(scaleW, scaleH, 1)
|
||||
outputWidth = max(1, int(float64(imgWidth)*scale))
|
||||
outputHeight = max(1, int(float64(imgHeight)*scale))
|
||||
|
||||
case FitOutside:
|
||||
case imagerequest.FitOutside:
|
||||
// The smallest dimension fits, the other may exceed the target. A
|
||||
// side scaled to 1 can come out just under 1 in floating point, as
|
||||
// 49 times 1/49 does, and round down to 0, which libvips refuses as
|
||||
@@ -541,7 +503,7 @@ func outputDimensions(
|
||||
outputHeight = max(1, int(float64(imgHeight)*scale))
|
||||
|
||||
default:
|
||||
return 0, 0, fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
|
||||
return 0, 0, fmt.Errorf("%w: %s", imagerequest.ErrInvalidFitMode, fit)
|
||||
}
|
||||
|
||||
// The side FitOutside scales past the target is held to the maximum too
|
||||
@@ -550,7 +512,8 @@ func outputDimensions(
|
||||
outputWidth, outputHeight, MaxOutputDimension)
|
||||
}
|
||||
|
||||
if format == FormatAVIF && outputWidth*outputHeight > MaxAVIFOutputPixels {
|
||||
if format == imagerequest.FormatAVIF &&
|
||||
outputWidth*outputHeight > MaxAVIFOutputPixels {
|
||||
return 0, 0, fmt.Errorf("%w: %dx%d is above %d pixels",
|
||||
ErrAVIFOutputTooLarge, outputWidth, outputHeight, MaxAVIFOutputPixels)
|
||||
}
|
||||
@@ -561,18 +524,18 @@ func outputDimensions(
|
||||
// resize resizes the image to width x height, the dimensions
|
||||
// outputDimensions gives for the fit mode.
|
||||
func (p *ImageProcessor) resize(
|
||||
img *vips.ImageRef, width, height int, fit FitMode,
|
||||
img *vips.ImageRef, width, height int, fit imagerequest.Fit,
|
||||
) error {
|
||||
switch fit {
|
||||
case FitCover, "":
|
||||
case imagerequest.FitCover, "":
|
||||
// Crop the centre and resize to fill exact dimensions (default)
|
||||
return cropCentreAndResize(img, width, height)
|
||||
|
||||
case FitFill:
|
||||
case imagerequest.FitFill:
|
||||
// Resize to exact dimensions (may distort)
|
||||
return img.ThumbnailWithSize(width, height, vips.InterestingNone, vips.SizeForce)
|
||||
|
||||
case FitInside:
|
||||
case imagerequest.FitInside:
|
||||
// outputDimensions keeps the dimensions of an image that already fits
|
||||
if img.Width() == width && img.Height() == height {
|
||||
return nil
|
||||
@@ -580,11 +543,11 @@ func (p *ImageProcessor) resize(
|
||||
|
||||
return img.Thumbnail(width, height, vips.InterestingNone)
|
||||
|
||||
case FitContain, FitOutside:
|
||||
case imagerequest.FitContain, imagerequest.FitOutside:
|
||||
return img.Thumbnail(width, height, vips.InterestingNone)
|
||||
|
||||
default:
|
||||
return fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
|
||||
return fmt.Errorf("%w: %s", imagerequest.ErrInvalidFitMode, fit)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -626,7 +589,8 @@ const defaultQuality = 85
|
||||
// to stop libvips once an encode has begun, so when ctx has ended encode
|
||||
// returns ctx's error without beginning one.
|
||||
func (p *ImageProcessor) encode(
|
||||
ctx context.Context, img *vips.ImageRef, format Format, quality int,
|
||||
ctx context.Context, img *vips.ImageRef, format imagerequest.Format,
|
||||
quality int,
|
||||
) ([]byte, error) {
|
||||
err := ctx.Err()
|
||||
if err != nil {
|
||||
@@ -649,25 +613,25 @@ func (p *ImageProcessor) encode(
|
||||
}
|
||||
|
||||
switch format {
|
||||
case FormatJPEG:
|
||||
case imagerequest.FormatJPEG:
|
||||
return exportJPEG(img, quality)
|
||||
|
||||
case FormatPNG:
|
||||
case imagerequest.FormatPNG:
|
||||
return exportPNG(img)
|
||||
|
||||
case FormatGIF:
|
||||
case imagerequest.FormatGIF:
|
||||
return exportGIF(img)
|
||||
|
||||
case FormatWebP:
|
||||
case imagerequest.FormatWebP:
|
||||
return exportWebP(img, quality)
|
||||
|
||||
case FormatAVIF:
|
||||
case imagerequest.FormatAVIF:
|
||||
return exportAVIF(img, quality)
|
||||
|
||||
case FormatJXL:
|
||||
case imagerequest.FormatJXL:
|
||||
return exportJXL(img, quality)
|
||||
|
||||
case FormatOriginal:
|
||||
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
|
||||
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
|
||||
|
||||
default:
|
||||
@@ -849,20 +813,22 @@ func jxlDistance(quality int) float64 {
|
||||
|
||||
// formatFromString converts a format string from detectFormat to Format. A
|
||||
// format detectFormat could not name is ErrUnsupportedInputFormat.
|
||||
func (p *ImageProcessor) formatFromString(format string) (Format, error) {
|
||||
func (p *ImageProcessor) formatFromString(
|
||||
format string,
|
||||
) (imagerequest.Format, error) {
|
||||
switch format {
|
||||
case "jpeg":
|
||||
return FormatJPEG, nil
|
||||
return imagerequest.FormatJPEG, nil
|
||||
case "png":
|
||||
return FormatPNG, nil
|
||||
return imagerequest.FormatPNG, nil
|
||||
case "gif":
|
||||
return FormatGIF, nil
|
||||
return imagerequest.FormatGIF, nil
|
||||
case "webp":
|
||||
return FormatWebP, nil
|
||||
case string(FormatAVIF):
|
||||
return FormatAVIF, nil
|
||||
case string(FormatJXL):
|
||||
return FormatJXL, nil
|
||||
return imagerequest.FormatWebP, nil
|
||||
case string(imagerequest.FormatAVIF):
|
||||
return imagerequest.FormatAVIF, nil
|
||||
case string(imagerequest.FormatJXL):
|
||||
return imagerequest.FormatJXL, nil
|
||||
default:
|
||||
return "", fmt.Errorf("%w: %s", ErrUnsupportedInputFormat, format)
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"github.com/davidbyttow/govips/v2/vips"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
@@ -87,7 +88,7 @@ func isAVIF(data []byte) bool {
|
||||
|
||||
brand := string(data[8:12])
|
||||
|
||||
return brand == string(FormatAVIF) || brand == "avis"
|
||||
return brand == string(imagerequest.FormatAVIF) || brand == "avis"
|
||||
}
|
||||
|
||||
// detectMIME is a minimal magic-byte detector for test assertions.
|
||||
@@ -123,14 +124,14 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, 800, 600)
|
||||
|
||||
req := &Request{
|
||||
Size: Size{Width: 400, Height: 300},
|
||||
Format: FormatJPEG,
|
||||
transform := &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 400, Height: 300},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), req)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -169,13 +170,13 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
|
||||
req := &Request{
|
||||
Size: Size{Width: 200, Height: 150},
|
||||
Format: FormatPNG,
|
||||
FitMode: FitCover,
|
||||
transform := &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 200, Height: 150},
|
||||
Format: imagerequest.FormatPNG,
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), req)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -196,7 +197,7 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
|
||||
// processAndCheckSize processes a test JPEG of the given input dimensions
|
||||
// with the requested size and asserts the resulting dimensions.
|
||||
func processAndCheckSize(
|
||||
t *testing.T, inputW, inputH int, size Size, wantW, wantH int,
|
||||
t *testing.T, inputW, inputH int, size imagerequest.Size, wantW, wantH int,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
@@ -205,14 +206,14 @@ func processAndCheckSize(
|
||||
|
||||
input := createTestJPEG(t, inputW, inputH)
|
||||
|
||||
req := &Request{
|
||||
transform := &imagerequest.Transform{
|
||||
Size: size,
|
||||
Format: FormatJPEG,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), req)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -232,7 +233,7 @@ func TestImageProcessor_OriginalSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Width and height 0: keep original size
|
||||
processAndCheckSize(t, 640, 480, Size{Width: 0, Height: 0}, 640, 480)
|
||||
processAndCheckSize(t, 640, 480, imagerequest.Size{Width: 0, Height: 0}, 640, 480)
|
||||
}
|
||||
|
||||
func TestImageProcessor_FitContain(t *testing.T) {
|
||||
@@ -245,14 +246,14 @@ func TestImageProcessor_FitContain(t *testing.T) {
|
||||
// Should result in 400x200 (maintaining aspect ratio)
|
||||
input := createTestJPEG(t, 800, 400)
|
||||
|
||||
req := &Request{
|
||||
Size: Size{Width: 400, Height: 400},
|
||||
Format: FormatJPEG,
|
||||
transform := &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 400, Height: 400},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitContain,
|
||||
FitMode: imagerequest.FitContain,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), req)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -270,7 +271,7 @@ func TestImageProcessor_ProportionalScale_WidthOnly(t *testing.T) {
|
||||
|
||||
// 800x600 image, request width=400 height=0
|
||||
// Should scale proportionally to 400x300
|
||||
processAndCheckSize(t, 800, 600, Size{Width: 400, Height: 0}, 400, 300)
|
||||
processAndCheckSize(t, 800, 600, imagerequest.Size{Width: 400, Height: 0}, 400, 300)
|
||||
}
|
||||
|
||||
func TestImageProcessor_ProportionalScale_HeightOnly(t *testing.T) {
|
||||
@@ -278,7 +279,7 @@ func TestImageProcessor_ProportionalScale_HeightOnly(t *testing.T) {
|
||||
|
||||
// 800x600 image, request width=0 height=300
|
||||
// Should scale proportionally to 400x300
|
||||
processAndCheckSize(t, 800, 600, Size{Width: 0, Height: 300}, 400, 300)
|
||||
processAndCheckSize(t, 800, 600, imagerequest.Size{Width: 0, Height: 300}, 400, 300)
|
||||
}
|
||||
|
||||
func TestImageProcessor_ProcessPNG(t *testing.T) {
|
||||
@@ -289,13 +290,13 @@ func TestImageProcessor_ProcessPNG(t *testing.T) {
|
||||
|
||||
input := createTestPNG(t, 400, 300)
|
||||
|
||||
req := &Request{
|
||||
Size: Size{Width: 200, Height: 150},
|
||||
Format: FormatPNG,
|
||||
FitMode: FitCover,
|
||||
transform := &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 200, Height: 150},
|
||||
Format: imagerequest.FormatPNG,
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), req)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
@@ -349,14 +350,14 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
input := createTestJPEG(t, tt.width, tt.height)
|
||||
|
||||
req := &Request{
|
||||
Size: Size{Width: 100, Height: 100},
|
||||
Format: FormatJPEG,
|
||||
transform := &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 100, Height: 100},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input), req)
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err == nil {
|
||||
t.Error("Process() should reject oversized input images")
|
||||
}
|
||||
@@ -377,14 +378,14 @@ func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
|
||||
// Create an image at exactly MaxInputDimension - should be accepted
|
||||
input := createTestJPEG(t, MaxInputDimension, 100)
|
||||
|
||||
req := &Request{
|
||||
Size: Size{Width: 100, Height: 100},
|
||||
Format: FormatJPEG,
|
||||
transform := &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 100, Height: 100},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), req)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf(
|
||||
"Process() should accept images at MaxInputDimension, got error: %v",
|
||||
@@ -397,7 +398,9 @@ func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
|
||||
|
||||
// encodeAndCheck processes a 200x150 test JPEG into a 100x75 output of the
|
||||
// given format and asserts the output MIME type and dimensions.
|
||||
func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) {
|
||||
func encodeAndCheck(
|
||||
t *testing.T, format imagerequest.Format, quality int, wantMIME string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
proc := New(Params{})
|
||||
@@ -405,14 +408,14 @@ func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) {
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
|
||||
req := &Request{
|
||||
Size: Size{Width: 100, Height: 75},
|
||||
transform := &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 100, Height: 75},
|
||||
Format: format,
|
||||
Quality: quality,
|
||||
FitMode: FitCover,
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), req)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v, want nil", err)
|
||||
}
|
||||
@@ -443,7 +446,7 @@ func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) {
|
||||
func TestImageProcessor_EncodeWebP(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
encodeAndCheck(t, FormatWebP, 80, mimeWebP)
|
||||
encodeAndCheck(t, imagerequest.FormatWebP, 80, mimeWebP)
|
||||
}
|
||||
|
||||
func TestImageProcessor_DecodeAVIF(t *testing.T) {
|
||||
@@ -459,14 +462,14 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
|
||||
}
|
||||
|
||||
// Request resize and convert to JPEG
|
||||
req := &Request{
|
||||
Size: Size{Width: 2, Height: 2},
|
||||
Format: FormatJPEG,
|
||||
transform := &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 2, Height: 2},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), req)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v, want nil (AVIF decoding should work)", err)
|
||||
}
|
||||
@@ -500,14 +503,14 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
|
||||
t.Fatalf("test JPEG must exceed %d bytes, got %d", limit, len(input))
|
||||
}
|
||||
|
||||
req := &Request{
|
||||
Size: Size{Width: 100, Height: 75},
|
||||
Format: FormatJPEG,
|
||||
transform := &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 100, Height: 75},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input), req)
|
||||
_, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err == nil {
|
||||
t.Fatal("Process() should reject input exceeding maxInputBytes")
|
||||
}
|
||||
@@ -527,14 +530,14 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
|
||||
proc := New(Params{MaxInputBytes: limit})
|
||||
ctx := context.Background()
|
||||
|
||||
req := &Request{
|
||||
Size: Size{Width: 10, Height: 10},
|
||||
Format: FormatJPEG,
|
||||
transform := &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 10, Height: 10},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), req)
|
||||
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v, want nil", err)
|
||||
}
|
||||
@@ -561,16 +564,18 @@ func TestImageProcessor_DefaultMaxInputBytes(t *testing.T) {
|
||||
func TestImageProcessor_EncodeAVIF(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
encodeAndCheck(t, FormatAVIF, 85, mimeAVIF)
|
||||
encodeAndCheck(t, imagerequest.FormatAVIF, 85, mimeAVIF)
|
||||
}
|
||||
|
||||
// processAndDecode runs input through Process and decodes the output with
|
||||
// vips, so a test can inspect the image a client would receive.
|
||||
func processAndDecode(t *testing.T, input []byte, req *Request) *vips.ImageRef {
|
||||
func processAndDecode(
|
||||
t *testing.T, input []byte, transform *imagerequest.Transform,
|
||||
) *vips.ImageRef {
|
||||
t.Helper()
|
||||
|
||||
result, err := New(Params{}).Process(
|
||||
context.Background(), bytes.NewReader(input), req,
|
||||
context.Background(), bytes.NewReader(input), transform,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
@@ -614,15 +619,17 @@ func TestImageProcessor_StripsEXIF(t *testing.T) {
|
||||
t.Fatal("testdata/gps-exif.jpg has no GPS latitude")
|
||||
}
|
||||
|
||||
formats := []Format{
|
||||
FormatJPEG, FormatPNG, FormatWebP, FormatAVIF, FormatGIF, FormatOriginal,
|
||||
formats := []imagerequest.Format{
|
||||
imagerequest.FormatJPEG, imagerequest.FormatPNG, imagerequest.FormatWebP,
|
||||
imagerequest.FormatAVIF, imagerequest.FormatGIF, imagerequest.FormatOriginal,
|
||||
}
|
||||
|
||||
for _, format := range formats {
|
||||
t.Run(string(format), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
output := processAndDecode(t, input, &Request{Format: format})
|
||||
output := processAndDecode(t, input,
|
||||
&imagerequest.Transform{Format: format})
|
||||
|
||||
if output.HasExif() {
|
||||
t.Errorf("output has EXIF: %v", output.GetExif())
|
||||
@@ -644,21 +651,21 @@ func TestImageProcessor_AppliesEXIFOrientation(t *testing.T) {
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
size Size
|
||||
size imagerequest.Size
|
||||
wantW int
|
||||
wantH int
|
||||
}{
|
||||
{name: "original size", size: Size{}, wantW: 8, wantH: 16},
|
||||
{name: "width only", size: Size{Width: 4}, wantW: 4, wantH: 8},
|
||||
{name: "original size", size: imagerequest.Size{}, wantW: 8, wantH: 16},
|
||||
{name: "width only", size: imagerequest.Size{Width: 4}, wantW: 4, wantH: 8},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
output := processAndDecode(t, input, &Request{
|
||||
output := processAndDecode(t, input, &imagerequest.Transform{
|
||||
Size: tt.size,
|
||||
Format: FormatPNG,
|
||||
Format: imagerequest.FormatPNG,
|
||||
})
|
||||
|
||||
if output.Width() != tt.wantW || output.Height() != tt.wantH {
|
||||
@@ -694,7 +701,8 @@ func TestImageProcessor_ConvertsWideGamutToSRGB(t *testing.T) {
|
||||
t.Fatalf("failed to read test JPEG: %v", err)
|
||||
}
|
||||
|
||||
output := processAndDecode(t, input, &Request{Format: FormatPNG})
|
||||
output := processAndDecode(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatPNG})
|
||||
|
||||
if output.HasICCProfile() {
|
||||
t.Error("output has an ICC profile")
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"github.com/davidbyttow/govips/v2/vips"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestImageProcessor_EncodeJPEGXL converts a JPEG to a smaller JPEG XL and
|
||||
@@ -15,10 +16,13 @@ import (
|
||||
func TestImageProcessor_EncodeJPEGXL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &Request{Size: Size{Width: 100, Height: 75}, Format: FormatJXL}
|
||||
transform := &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 100, Height: 75},
|
||||
Format: imagerequest.FormatJXL,
|
||||
}
|
||||
|
||||
result, err := New(Params{}).Process(
|
||||
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)), req,
|
||||
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)), transform,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
@@ -61,7 +65,8 @@ func TestImageProcessor_JPEGXLQuality(t *testing.T) {
|
||||
|
||||
for _, quality := range []int{30, 90} {
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input), &Request{Format: FormatJXL, Quality: quality})
|
||||
bytes.NewReader(input),
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: quality})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() at quality %d error = %v", quality, err)
|
||||
}
|
||||
@@ -89,7 +94,8 @@ func TestImageProcessor_JPEGXLDropsSourceEXIF(t *testing.T) {
|
||||
t.Fatalf("failed to read test JPEG: %v", err)
|
||||
}
|
||||
|
||||
exif := processAndDecode(t, input, &Request{Format: FormatJXL}).GetExif()
|
||||
exif := processAndDecode(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJXL}).GetExif()
|
||||
|
||||
for _, field := range []string{
|
||||
"exif-ifd0-Make", "exif-ifd0-Model", "exif-ifd2-BodySerialNumber",
|
||||
@@ -115,7 +121,8 @@ func TestImageProcessor_JPEGXLDropsSourceResolution(t *testing.T) {
|
||||
t.Fatalf("failed to read test JPEG: %v", err)
|
||||
}
|
||||
|
||||
output := processAndDecode(t, input, &Request{Format: FormatJXL})
|
||||
output := processAndDecode(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJXL})
|
||||
|
||||
// libvips gives the resolution in pixels per millimetre.
|
||||
xDPI := math.Round(output.ResX() * 25.4)
|
||||
@@ -140,7 +147,8 @@ func TestImageProcessor_JPEGXLAppliesEXIFOrientation(t *testing.T) {
|
||||
t.Fatalf("failed to read test JPEG: %v", err)
|
||||
}
|
||||
|
||||
output := processAndDecode(t, input, &Request{Format: FormatJXL})
|
||||
output := processAndDecode(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJXL})
|
||||
|
||||
if output.Width() != 8 || output.Height() != 16 {
|
||||
t.Fatalf("output is %dx%d, want 8x16", output.Width(), output.Height())
|
||||
@@ -176,7 +184,8 @@ func TestImageProcessor_JPEGXLConvertsWideGamutToSRGB(t *testing.T) {
|
||||
t.Fatalf("failed to read test JPEG: %v", err)
|
||||
}
|
||||
|
||||
output := processAndDecode(t, input, &Request{Format: FormatJXL})
|
||||
output := processAndDecode(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJXL})
|
||||
|
||||
pixel, err := output.GetPoint(4, 4)
|
||||
if err != nil {
|
||||
@@ -203,7 +212,8 @@ func TestImageProcessor_JPEGXLFromCMYK(t *testing.T) {
|
||||
t.Fatalf("failed to read test JPEG: %v", err)
|
||||
}
|
||||
|
||||
output := processAndDecode(t, input, &Request{Format: FormatJXL})
|
||||
output := processAndDecode(t, input,
|
||||
&imagerequest.Transform{Format: imagerequest.FormatJXL})
|
||||
|
||||
if output.Bands() != 3 {
|
||||
t.Fatalf("output has %d bands, want 3", output.Bands())
|
||||
|
||||
@@ -11,6 +11,8 @@ import (
|
||||
"testing"
|
||||
"testing/iotest"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// errTestReadFailed is the error the unreadable test input returns.
|
||||
@@ -76,13 +78,13 @@ func (r *gatedReader) Read(p []byte) (int, error) {
|
||||
return n, err
|
||||
}
|
||||
|
||||
// smallJPEGRequest asks for a 5x5 JPEG.
|
||||
func smallJPEGRequest() *Request {
|
||||
return &Request{
|
||||
Size: Size{Width: 5, Height: 5},
|
||||
Format: FormatJPEG,
|
||||
// smallJPEGTransform asks for a 5x5 JPEG.
|
||||
func smallJPEGTransform() *imagerequest.Transform {
|
||||
return &imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 5, Height: 5},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
FitMode: imagerequest.FitCover,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -92,7 +94,7 @@ func processInBackground(
|
||||
proc *ImageProcessor, reader *gatedReader, results chan<- error,
|
||||
) {
|
||||
go func() {
|
||||
result, err := proc.Process(context.Background(), reader, smallJPEGRequest())
|
||||
result, err := proc.Process(context.Background(), reader, smallJPEGTransform())
|
||||
if err == nil {
|
||||
_ = result.Content.Close()
|
||||
}
|
||||
@@ -237,7 +239,7 @@ func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) {
|
||||
start := time.Now()
|
||||
|
||||
_, err := proc.Process(context.Background(), bytes.NewReader(input),
|
||||
smallJPEGRequest())
|
||||
smallJPEGTransform())
|
||||
if !errors.Is(err, ErrTooManyImages) {
|
||||
t.Fatalf("Process() error = %v, want ErrTooManyImages", err)
|
||||
}
|
||||
@@ -263,38 +265,38 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
|
||||
|
||||
valid := createTestJPEG(t, 10, 10)
|
||||
|
||||
unsupported := smallJPEGRequest()
|
||||
unsupported := smallJPEGTransform()
|
||||
unsupported.Format = "bmp"
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
input io.Reader
|
||||
req *Request
|
||||
name string
|
||||
input io.Reader
|
||||
transform *imagerequest.Transform
|
||||
// want is the error Process must return; nil means any error.
|
||||
want error
|
||||
}{
|
||||
{
|
||||
name: "input cannot be read",
|
||||
input: iotest.ErrReader(errTestReadFailed),
|
||||
req: smallJPEGRequest(),
|
||||
want: errTestReadFailed,
|
||||
name: "input cannot be read",
|
||||
input: iotest.ErrReader(errTestReadFailed),
|
||||
transform: smallJPEGTransform(),
|
||||
want: errTestReadFailed,
|
||||
},
|
||||
{
|
||||
name: "input over the byte limit",
|
||||
input: bytes.NewReader(createTestJPEG(t, 800, 600)),
|
||||
req: smallJPEGRequest(),
|
||||
want: ErrInputDataTooLarge,
|
||||
name: "input over the byte limit",
|
||||
input: bytes.NewReader(createTestJPEG(t, 800, 600)),
|
||||
transform: smallJPEGTransform(),
|
||||
want: ErrInputDataTooLarge,
|
||||
},
|
||||
{
|
||||
name: "input not an image",
|
||||
input: strings.NewReader("not an image"),
|
||||
req: smallJPEGRequest(),
|
||||
name: "input not an image",
|
||||
input: strings.NewReader("not an image"),
|
||||
transform: smallJPEGTransform(),
|
||||
},
|
||||
{
|
||||
name: "output format not supported",
|
||||
input: bytes.NewReader(valid),
|
||||
req: unsupported,
|
||||
want: ErrUnsupportedOutputFormat,
|
||||
name: "output format not supported",
|
||||
input: bytes.NewReader(valid),
|
||||
transform: unsupported,
|
||||
want: ErrUnsupportedOutputFormat,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -305,7 +307,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
|
||||
proc := New(Params{MaxInputBytes: 4096, MaxConcurrentProcessing: 1})
|
||||
proc.processingWaitTimeout = 100 * time.Millisecond
|
||||
|
||||
_, err := proc.Process(context.Background(), tc.input, tc.req)
|
||||
_, err := proc.Process(context.Background(), tc.input, tc.transform)
|
||||
if err == nil || (tc.want != nil && !errors.Is(err, tc.want)) {
|
||||
t.Fatalf("Process() error = %v, want %v", err, tc.want)
|
||||
}
|
||||
@@ -315,7 +317,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
|
||||
}
|
||||
|
||||
result, err := proc.Process(context.Background(), bytes.NewReader(valid),
|
||||
smallJPEGRequest())
|
||||
smallJPEGTransform())
|
||||
if err != nil {
|
||||
t.Fatalf("Process() after the error = %v, want nil", err)
|
||||
}
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"github.com/davidbyttow/govips/v2/vips"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestImageProcessor_OrigOfUnnamedFormatRefused verifies that orig of a TIFF
|
||||
@@ -27,7 +28,8 @@ func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
|
||||
}
|
||||
|
||||
result, err := New(Params{}).Process(
|
||||
t.Context(), bytes.NewReader(tiff), &Request{Format: FormatOriginal},
|
||||
t.Context(), bytes.NewReader(tiff),
|
||||
&imagerequest.Transform{Format: imagerequest.FormatOriginal},
|
||||
)
|
||||
if result != nil {
|
||||
t.Logf("Process() returned %s", result.ContentType)
|
||||
|
||||
@@ -4,6 +4,8 @@ import (
|
||||
"bytes"
|
||||
"io"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestImageProcessor_LoadsEachSourceFormat verifies that an image in each of
|
||||
@@ -14,11 +16,13 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
|
||||
|
||||
proc := New(Params{})
|
||||
|
||||
for _, format := range []Format{
|
||||
FormatJPEG, FormatPNG, FormatGIF, FormatWebP, FormatAVIF, FormatJXL,
|
||||
for _, format := range []imagerequest.Format{
|
||||
imagerequest.FormatJPEG, imagerequest.FormatPNG, imagerequest.FormatGIF,
|
||||
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
|
||||
} {
|
||||
saved, err := proc.Process(t.Context(),
|
||||
bytes.NewReader(createTestPNG(t, 20, 20)), &Request{Format: format})
|
||||
bytes.NewReader(createTestPNG(t, 20, 20)),
|
||||
&imagerequest.Transform{Format: format})
|
||||
if err != nil {
|
||||
t.Fatalf("saving the test image as %s: %v", format, err)
|
||||
}
|
||||
@@ -29,7 +33,7 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
|
||||
}
|
||||
|
||||
result, err := proc.Process(t.Context(), bytes.NewReader(source),
|
||||
&Request{Format: FormatOriginal})
|
||||
&imagerequest.Transform{Format: imagerequest.FormatOriginal})
|
||||
if err != nil {
|
||||
t.Errorf("orig of the %s image: %v", format, err)
|
||||
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
// Package imagerequest holds the image request type, its parts, its checks,
|
||||
// the parsing of the /v1/image/ path, the variant key and the source key. It
|
||||
// imports no other package of this repo, so every package that handles an
|
||||
// image request can import it.
|
||||
package imagerequest
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/url"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrInvalidFitMode is returned when an invalid fit mode is provided.
|
||||
var ErrInvalidFitMode = errors.New("invalid fit mode")
|
||||
|
||||
// Format represents supported output image formats.
|
||||
type Format string
|
||||
|
||||
// Supported image output formats.
|
||||
const (
|
||||
FormatOriginal Format = "orig"
|
||||
FormatJPEG Format = "jpeg"
|
||||
FormatPNG Format = "png"
|
||||
FormatWebP Format = "webp"
|
||||
FormatAVIF Format = "avif"
|
||||
FormatJXL Format = "jxl"
|
||||
FormatGIF Format = "gif"
|
||||
|
||||
// FormatAuto stands for JPEG XL, AVIF, WebP or JPEG, chosen for each
|
||||
// request from its Accept header once the URL's signature or token has
|
||||
// been checked; it is never processed or cached as itself.
|
||||
FormatAuto Format = "auto"
|
||||
)
|
||||
|
||||
// Size represents requested image dimensions
|
||||
type Size struct {
|
||||
Width int
|
||||
Height int
|
||||
}
|
||||
|
||||
// OriginalSize returns true if this represents "keep original size"
|
||||
func (s Size) OriginalSize() bool {
|
||||
return s.Width == 0 && s.Height == 0
|
||||
}
|
||||
|
||||
// Fit represents how to fit image into requested dimensions.
|
||||
type Fit string
|
||||
|
||||
// Supported image fit modes.
|
||||
const (
|
||||
FitCover Fit = "cover"
|
||||
FitContain Fit = "contain"
|
||||
FitFill Fit = "fill"
|
||||
FitInside Fit = "inside"
|
||||
FitOutside Fit = "outside"
|
||||
)
|
||||
|
||||
// ValidateFitMode checks if the given fit mode is valid.
|
||||
// Returns ErrInvalidFitMode for unrecognized fit modes.
|
||||
func ValidateFitMode(fit Fit) error {
|
||||
switch fit {
|
||||
case FitCover, FitContain, FitFill, FitInside, FitOutside, "":
|
||||
return nil
|
||||
default:
|
||||
return ErrInvalidFitMode
|
||||
}
|
||||
}
|
||||
|
||||
// ValidateImageRequest checks a request's width and height with
|
||||
// ValidateDimension and its fit mode with ValidateFitMode. Both the plain
|
||||
// /v1/image/ route and the encrypted /v1/e/ route validate through this
|
||||
// function so a request from either source enforces identical bounds.
|
||||
func ValidateImageRequest(req *Request) error {
|
||||
err := ValidateDimension("width", req.Transform.Size.Width)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = ValidateDimension("height", req.Transform.Size.Height)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return ValidateFitMode(req.Transform.FitMode)
|
||||
}
|
||||
|
||||
// Request represents a request for a processed image: the source to fetch
|
||||
// and the transform to apply to it.
|
||||
type Request struct {
|
||||
// Source is what to fetch
|
||||
Source Source
|
||||
// Transform is what to do to it
|
||||
Transform Transform
|
||||
// Signature is the HMAC signature for non-allowlisted hosts
|
||||
Signature string
|
||||
// Expires is when the URL expires: the exp of a signed URL, or the expiry
|
||||
// of an encrypted URL; the zero time if it has none
|
||||
Expires time.Time
|
||||
// AllowHTTP indicates whether HTTP (non-TLS) is allowed for this request
|
||||
AllowHTTP bool
|
||||
}
|
||||
|
||||
// Source is the image a request fetches from the origin.
|
||||
type Source struct {
|
||||
// Host is the origin host (e.g., "cdn.example.com")
|
||||
Host string
|
||||
// Path is the path on the origin with its percent-escapes decoded, as
|
||||
// url.URL's Path holds it (e.g., "/photos/my cat.jpg")
|
||||
Path string
|
||||
// Query is the optional query string for the origin URL as the URL
|
||||
// writes it, percent-escapes included, as url.URL's RawQuery holds it
|
||||
Query string
|
||||
}
|
||||
|
||||
// Transform is what a request does to its source; the image processor
|
||||
// takes it alone.
|
||||
type Transform struct {
|
||||
// Size is the requested output dimensions
|
||||
Size Size
|
||||
// Format is the requested output format
|
||||
Format Format
|
||||
// Quality is the output quality (1-100) for lossy formats
|
||||
Quality int
|
||||
// FitMode is how to fit the image into requested dimensions
|
||||
FitMode Fit
|
||||
}
|
||||
|
||||
// SourceURL returns the full upstream URL to fetch: the source path
|
||||
// percent-encoded again, as net/url writes a path, so each source path is one
|
||||
// upstream path, and the source query as it is, with the characters
|
||||
// escapeQuery encodes percent-encoded. Uses http:// scheme when AllowHTTP is
|
||||
// true, otherwise https://.
|
||||
func (r *Request) SourceURL() string {
|
||||
scheme := "https"
|
||||
if r.AllowHTTP {
|
||||
scheme = "http"
|
||||
}
|
||||
|
||||
sourceURL := url.URL{
|
||||
Scheme: scheme,
|
||||
Host: r.Source.Host,
|
||||
Path: r.Source.Path,
|
||||
RawQuery: escapeQuery(r.Source.Query),
|
||||
}
|
||||
|
||||
return sourceURL.String()
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
package imagerequest
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// SourceKey is a SHA256 hash identifying a cached source image (hex-encoded).
|
||||
type SourceKey string
|
||||
|
||||
// Key returns the key the source is cached under, made as CacheKey makes a
|
||||
// variant key.
|
||||
// Format: sha256(host:path:query)
|
||||
// The path and query go through escapePercentAndColon, so that a ":" in
|
||||
// either cannot be taken for the one between them.
|
||||
func (s Source) Key() SourceKey {
|
||||
data := fmt.Sprintf("%s:%s:%s",
|
||||
s.Host,
|
||||
escapePercentAndColon(s.Path),
|
||||
escapePercentAndColon(s.Query),
|
||||
)
|
||||
h := sha256.Sum256([]byte(data))
|
||||
|
||||
return SourceKey(hex.EncodeToString(h[:]))
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package imagerequest
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestSourceKey_DiffersForEachSource checks that each source gets a key of
|
||||
// 64 hex characters, the same each time, and that no two of these sources
|
||||
// share one, also when they differ only by a ":" or "%" in the path or query.
|
||||
func TestSourceKey_DiffersForEachSource(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
sources := []Source{
|
||||
{Host: testHostCDN, Path: testPathCat},
|
||||
{Host: testHostCDN, Path: testPathCat, Query: "v=2"},
|
||||
{Host: "other.example.com", Path: testPathCat},
|
||||
{Host: testHostCDN, Path: "/c:d"},
|
||||
{Host: testHostCDN, Path: "/c", Query: "d:"},
|
||||
{Host: testHostCDN, Path: "/c%3Ad"},
|
||||
}
|
||||
|
||||
seen := make(map[SourceKey]Source)
|
||||
|
||||
for _, source := range sources {
|
||||
key := source.Key()
|
||||
t.Logf("Key() of %+v = %s", source, key)
|
||||
|
||||
if len(key) != 64 {
|
||||
t.Errorf("Key() of %+v length = %d, want 64", source, len(key))
|
||||
}
|
||||
|
||||
if again := source.Key(); again != key {
|
||||
t.Errorf("Key() of %+v not deterministic: %s vs %s", source, key, again)
|
||||
}
|
||||
|
||||
if earlier, found := seen[key]; found {
|
||||
t.Errorf("Key() gave %+v and %+v the same key", earlier, source)
|
||||
}
|
||||
|
||||
seen[key] = source
|
||||
}
|
||||
}
|
||||
+22
-16
@@ -1,14 +1,16 @@
|
||||
package imgcache
|
||||
package imagerequest
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestImageRequest_SourceURL_DefaultHTTPS(t *testing.T) {
|
||||
func TestRequest_SourceURL_DefaultHTTPS(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: testPathCat,
|
||||
SourceQuery: "v=2",
|
||||
req := &Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
Query: "v=2",
|
||||
},
|
||||
}
|
||||
|
||||
got := req.SourceURL()
|
||||
@@ -19,13 +21,15 @@ func TestImageRequest_SourceURL_DefaultHTTPS(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestImageRequest_SourceURL_AllowHTTP(t *testing.T) {
|
||||
func TestRequest_SourceURL_AllowHTTP(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: "localhost:8080",
|
||||
SourcePath: testPathCat,
|
||||
AllowHTTP: true,
|
||||
req := &Request{
|
||||
Source: Source{
|
||||
Host: "localhost:8080",
|
||||
Path: testPathCat,
|
||||
},
|
||||
AllowHTTP: true,
|
||||
}
|
||||
|
||||
got := req.SourceURL()
|
||||
@@ -36,13 +40,15 @@ func TestImageRequest_SourceURL_AllowHTTP(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestImageRequest_SourceURL_AllowHTTPFalse(t *testing.T) {
|
||||
func TestRequest_SourceURL_AllowHTTPFalse(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: "/img.jpg",
|
||||
AllowHTTP: false,
|
||||
req := &Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/img.jpg",
|
||||
},
|
||||
AllowHTTP: false,
|
||||
}
|
||||
|
||||
got := req.SourceURL()
|
||||
@@ -1,4 +1,4 @@
|
||||
package imgcache
|
||||
package imagerequest
|
||||
|
||||
import (
|
||||
"errors"
|
||||
@@ -8,8 +8,6 @@ import (
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
)
|
||||
|
||||
// URL parsing errors.
|
||||
@@ -25,8 +23,8 @@ var (
|
||||
)
|
||||
|
||||
// MaxDimension is the maximum allowed width or height, the same as the image
|
||||
// processor's maximum for an output image.
|
||||
const MaxDimension = imageprocessor.MaxOutputDimension
|
||||
// processor's maximum for an output image (imageprocessor.MaxOutputDimension).
|
||||
const MaxDimension = 8192
|
||||
|
||||
// ValidateDimension checks one requested width or height; name ("width" or
|
||||
// "height") appears in the error. 0 is valid: that side is then worked out from
|
||||
@@ -68,20 +66,6 @@ var queryEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}|[^!-~]|["#'<>%]`)
|
||||
var hostRegex = regexp.MustCompile(
|
||||
`^(?:[A-Za-z0-9_.-]+|\[([0-9A-Fa-f:.]+)\])(?::[0-9]*)?$`)
|
||||
|
||||
// ParsedURL contains the parsed components of an image proxy URL.
|
||||
type ParsedURL struct {
|
||||
// Host is the source origin host (e.g., "cdn.example.com")
|
||||
Host string
|
||||
// Path is the path on the origin (e.g., "/photos/cat.jpg")
|
||||
Path string
|
||||
// Query is the optional query string for the origin
|
||||
Query string
|
||||
// Size is the requested output dimensions
|
||||
Size Size
|
||||
// Format is the requested output format
|
||||
Format ImageFormat
|
||||
}
|
||||
|
||||
// ParseImagePath parses an image URL's path after /v1/image/, decoded once:
|
||||
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL.
|
||||
// Examples:
|
||||
@@ -89,7 +73,7 @@ type ParsedURL struct {
|
||||
// - cdn.example.com/photos/cat.jpg/0x0.jpeg
|
||||
// - cdn.example.com/photos/cat.jpg/orig.png
|
||||
// - cdn.example.com/photos/cat.jpg/800x600
|
||||
func ParseImagePath(path string) (*ParsedURL, error) {
|
||||
func ParseImagePath(path string) (*Request, error) {
|
||||
// Strip leading slash if present (chi may include it)
|
||||
path = strings.TrimPrefix(path, "/")
|
||||
if path == "" {
|
||||
@@ -103,7 +87,7 @@ func ParseImagePath(path string) (*ParsedURL, error) {
|
||||
// it: /v1/image/<host>/<path>/<size>.<format>, or /v1/image/<host>/<path>/<size>
|
||||
// for JPEG XL. A source query follows the source path after a "?", which the
|
||||
// URL writes as %3F.
|
||||
func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
||||
func ParseImageURL(urlPath string) (*Request, error) {
|
||||
// Remove the /v1/image/ prefix
|
||||
const prefix = "/v1/image/"
|
||||
if !strings.HasPrefix(urlPath, prefix) {
|
||||
@@ -120,7 +104,7 @@ func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
||||
|
||||
// parseImageComponents parses <host>/<path>/<size>.<format>, or
|
||||
// <host>/<path>/<size> for JPEG XL.
|
||||
func parseImageComponents(remainder string) (*ParsedURL, error) {
|
||||
func parseImageComponents(remainder string) (*Request, error) {
|
||||
// Find the last path segment, which holds "size" or "size.format"
|
||||
lastSlash := strings.LastIndex(remainder, "/")
|
||||
if lastSlash == -1 {
|
||||
@@ -209,12 +193,9 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
|
||||
path = "/" + path
|
||||
}
|
||||
|
||||
return &ParsedURL{
|
||||
Host: host,
|
||||
Path: path,
|
||||
Query: query,
|
||||
Size: size,
|
||||
Format: format,
|
||||
return &Request{
|
||||
Source: Source{Host: host, Path: path, Query: query},
|
||||
Transform: Transform{Size: size, Format: format},
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -297,7 +278,7 @@ func escapeQuery(query string) string {
|
||||
}
|
||||
|
||||
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
|
||||
func parseSizeFormat(s string) (Size, ImageFormat, error) {
|
||||
func parseSizeFormat(s string) (Size, Format, error) {
|
||||
matches := sizeFormatRegex.FindStringSubmatch(s)
|
||||
if matches == nil {
|
||||
return Size{}, "", ErrInvalidSize
|
||||
@@ -351,8 +332,8 @@ func parseSizeFormat(s string) (Size, ImageFormat, error) {
|
||||
return size, format, nil
|
||||
}
|
||||
|
||||
// parseFormat converts a format string to ImageFormat.
|
||||
func parseFormat(s string) (ImageFormat, error) {
|
||||
// parseFormat converts a format string to Format.
|
||||
func parseFormat(s string) (Format, error) {
|
||||
switch strings.ToLower(s) {
|
||||
case "orig", "original":
|
||||
return FormatOriginal, nil
|
||||
@@ -374,14 +355,3 @@ func parseFormat(s string) (ImageFormat, error) {
|
||||
return "", fmt.Errorf("%w: %s", ErrInvalidFormat, s)
|
||||
}
|
||||
}
|
||||
|
||||
// ToImageRequest converts a ParsedURL to an ImageRequest.
|
||||
func (p *ParsedURL) ToImageRequest() *ImageRequest {
|
||||
return &ImageRequest{
|
||||
SourceHost: p.Host,
|
||||
SourcePath: p.Path,
|
||||
SourceQuery: p.Query,
|
||||
Size: p.Size,
|
||||
Format: p.Format,
|
||||
}
|
||||
}
|
||||
+84
-98
@@ -1,33 +1,41 @@
|
||||
package imgcache
|
||||
package imagerequest
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// assertParsedURL compares all fields of a parsed URL against the
|
||||
// expected value.
|
||||
func assertParsedURL(t *testing.T, got, want *ParsedURL) {
|
||||
// Shared test fixture strings.
|
||||
const (
|
||||
testHostCDN = "cdn.example.com"
|
||||
testHostExample = "example.com"
|
||||
testPathCat = "/photos/cat.jpg"
|
||||
)
|
||||
|
||||
// assertRequest compares the source host, path and query and the size and
|
||||
// format of got against want.
|
||||
func assertRequest(t *testing.T, got, want *Request) {
|
||||
t.Helper()
|
||||
|
||||
if got.Host != want.Host {
|
||||
t.Errorf("Host = %q, want %q", got.Host, want.Host)
|
||||
if got.Source.Host != want.Source.Host {
|
||||
t.Errorf("Host = %q, want %q", got.Source.Host, want.Source.Host)
|
||||
}
|
||||
|
||||
if got.Path != want.Path {
|
||||
t.Errorf("Path = %q, want %q", got.Path, want.Path)
|
||||
if got.Source.Path != want.Source.Path {
|
||||
t.Errorf("Path = %q, want %q", got.Source.Path, want.Source.Path)
|
||||
}
|
||||
|
||||
if got.Query != want.Query {
|
||||
t.Errorf("Query = %q, want %q", got.Query, want.Query)
|
||||
if got.Source.Query != want.Source.Query {
|
||||
t.Errorf("Query = %q, want %q", got.Source.Query, want.Source.Query)
|
||||
}
|
||||
|
||||
if got.Size != want.Size {
|
||||
t.Errorf("Size = %v, want %v", got.Size, want.Size)
|
||||
if got.Transform.Size != want.Transform.Size {
|
||||
t.Errorf("Size = %v, want %v", got.Transform.Size, want.Transform.Size)
|
||||
}
|
||||
|
||||
if got.Format != want.Format {
|
||||
t.Errorf("Format = %q, want %q", got.Format, want.Format)
|
||||
if got.Transform.Format != want.Transform.Format {
|
||||
t.Errorf("Format = %q, want %q",
|
||||
got.Transform.Format, want.Transform.Format)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -37,62 +45,64 @@ func TestParseImageURL(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
want *ParsedURL
|
||||
want *Request
|
||||
}{
|
||||
{
|
||||
name: "basic path with size",
|
||||
input: "/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp",
|
||||
want: &ParsedURL{
|
||||
Host: testHostCDN, Path: testPathCat,
|
||||
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||
want: &Request{
|
||||
Source: Source{Host: testHostCDN, Path: testPathCat},
|
||||
Transform: Transform{Size: Size{Width: 800, Height: 600}, Format: FormatWebP},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "original size with 0x0",
|
||||
input: "/v1/image/cdn.example.com/photos/cat.jpg/0x0.jpeg",
|
||||
want: &ParsedURL{
|
||||
Host: testHostCDN, Path: testPathCat,
|
||||
Size: Size{Width: 0, Height: 0}, Format: FormatJPEG,
|
||||
want: &Request{
|
||||
Source: Source{Host: testHostCDN, Path: testPathCat},
|
||||
Transform: Transform{Size: Size{Width: 0, Height: 0}, Format: FormatJPEG},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "original size with orig keyword",
|
||||
input: "/v1/image/cdn.example.com/photos/cat.jpg/orig.png",
|
||||
want: &ParsedURL{
|
||||
Host: testHostCDN, Path: testPathCat,
|
||||
Size: Size{Width: 0, Height: 0}, Format: FormatPNG,
|
||||
want: &Request{
|
||||
Source: Source{Host: testHostCDN, Path: testPathCat},
|
||||
Transform: Transform{Size: Size{Width: 0, Height: 0}, Format: FormatPNG},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "path with query string",
|
||||
input: "/v1/image/cdn.example.com/photos/cat.jpg?arg1=val1&arg2=val2/800x600.webp",
|
||||
want: &ParsedURL{
|
||||
Host: testHostCDN, Path: testPathCat, Query: "arg1=val1&arg2=val2",
|
||||
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||
want: &Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN, Path: testPathCat, Query: "arg1=val1&arg2=val2",
|
||||
},
|
||||
Transform: Transform{Size: Size{Width: 800, Height: 600}, Format: FormatWebP},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "deep nested path",
|
||||
input: "/v1/image/cdn.example.com/a/b/c/d/image.jpg/1920x1080.avif",
|
||||
want: &ParsedURL{
|
||||
Host: testHostCDN, Path: "/a/b/c/d/image.jpg",
|
||||
Size: Size{Width: 1920, Height: 1080}, Format: FormatAVIF,
|
||||
want: &Request{
|
||||
Source: Source{Host: testHostCDN, Path: "/a/b/c/d/image.jpg"},
|
||||
Transform: Transform{Size: Size{Width: 1920, Height: 1080}, Format: FormatAVIF},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "jpg alias for jpeg",
|
||||
input: "/v1/image/example.com/img.png/100x100.jpg",
|
||||
want: &ParsedURL{
|
||||
Host: testHostExample, Path: "/img.png",
|
||||
Size: Size{Width: 100, Height: 100}, Format: FormatJPEG,
|
||||
want: &Request{
|
||||
Source: Source{Host: testHostExample, Path: "/img.png"},
|
||||
Transform: Transform{Size: Size{Width: 100, Height: 100}, Format: FormatJPEG},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "gif format",
|
||||
input: "/v1/image/example.com/animated.gif/200x200.gif",
|
||||
want: &ParsedURL{
|
||||
Host: testHostExample, Path: "/animated.gif",
|
||||
Size: Size{Width: 200, Height: 200}, Format: FormatGIF,
|
||||
want: &Request{
|
||||
Source: Source{Host: testHostExample, Path: "/animated.gif"},
|
||||
Transform: Transform{Size: Size{Width: 200, Height: 200}, Format: FormatGIF},
|
||||
},
|
||||
},
|
||||
}
|
||||
@@ -106,7 +116,7 @@ func TestParseImageURL(t *testing.T) {
|
||||
t.Fatalf("ParseImageURL() unexpected error = %v", err)
|
||||
}
|
||||
|
||||
assertParsedURL(t, got, tt.want)
|
||||
assertRequest(t, got, tt.want)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -121,8 +131,8 @@ func TestParseImageURL_AutoFormat(t *testing.T) {
|
||||
t.Fatalf("ParseImageURL() error = %v", err)
|
||||
}
|
||||
|
||||
if got.Format != FormatAuto {
|
||||
t.Errorf("Format = %q, want %q", got.Format, FormatAuto)
|
||||
if got.Transform.Format != FormatAuto {
|
||||
t.Errorf("Format = %q, want %q", got.Transform.Format, FormatAuto)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -189,27 +199,35 @@ func TestParseImagePath(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
want *ParsedURL
|
||||
want *Request
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "chi wildcard capture",
|
||||
input: "cdn.example.com/photos/cat.jpg/800x600.webp",
|
||||
want: &ParsedURL{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
want: &Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
},
|
||||
Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "with leading slash from chi",
|
||||
input: "/cdn.example.com/photos/cat.jpg/800x600.webp",
|
||||
want: &ParsedURL{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
want: &Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
},
|
||||
Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
@@ -229,7 +247,7 @@ func TestParseImagePath(t *testing.T) {
|
||||
return
|
||||
}
|
||||
|
||||
assertParsedURL(t, got, tt.want)
|
||||
assertRequest(t, got, tt.want)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -246,14 +264,14 @@ func TestParseImagePath_LowerCasesHost(t *testing.T) {
|
||||
t.Fatalf("ParseImagePath() error = %v", err)
|
||||
}
|
||||
|
||||
t.Logf("parsed host %q, path %q", got.Host, got.Path)
|
||||
t.Logf("parsed host %q, path %q", got.Source.Host, got.Source.Path)
|
||||
|
||||
if got.Host != testHostCDN {
|
||||
t.Errorf("Host = %q, want %q", got.Host, testHostCDN)
|
||||
if got.Source.Host != testHostCDN {
|
||||
t.Errorf("Host = %q, want %q", got.Source.Host, testHostCDN)
|
||||
}
|
||||
|
||||
if got.Path != "/Photos/Cat.jpg" {
|
||||
t.Errorf("Path = %q, want %q", got.Path, "/Photos/Cat.jpg")
|
||||
if got.Source.Path != "/Photos/Cat.jpg" {
|
||||
t.Errorf("Path = %q, want %q", got.Source.Path, "/Photos/Cat.jpg")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -277,40 +295,6 @@ func TestParseImagePath_RefusesUserinfo(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestParsedURL_ToImageRequest(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
parsed := &ParsedURL{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
Query: "version=2",
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
}
|
||||
|
||||
req := parsed.ToImageRequest()
|
||||
|
||||
if req.SourceHost != parsed.Host {
|
||||
t.Errorf("SourceHost = %q, want %q", req.SourceHost, parsed.Host)
|
||||
}
|
||||
|
||||
if req.SourcePath != parsed.Path {
|
||||
t.Errorf("SourcePath = %q, want %q", req.SourcePath, parsed.Path)
|
||||
}
|
||||
|
||||
if req.SourceQuery != parsed.Query {
|
||||
t.Errorf("SourceQuery = %q, want %q", req.SourceQuery, parsed.Query)
|
||||
}
|
||||
|
||||
if req.Size != parsed.Size {
|
||||
t.Errorf("Size = %v, want %v", req.Size, parsed.Size)
|
||||
}
|
||||
|
||||
if req.Format != parsed.Format {
|
||||
t.Errorf("Format = %q, want %q", req.Format, parsed.Format)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseImageURL_PathTraversal(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -417,22 +401,24 @@ func TestParseImageURL_SourceURLHoldingPercent25(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
want *ParsedURL
|
||||
want *Request
|
||||
}{
|
||||
{
|
||||
name: "path",
|
||||
input: "/v1/image/cdn.example.com/sale/50%25off.jpg/800x600.webp",
|
||||
want: &ParsedURL{
|
||||
Host: testHostCDN, Path: "/sale/50%25off.jpg",
|
||||
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||
want: &Request{
|
||||
Source: Source{Host: testHostCDN, Path: "/sale/50%25off.jpg"},
|
||||
Transform: Transform{Size: Size{Width: 800, Height: 600}, Format: FormatWebP},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "query",
|
||||
input: "/v1/image/cdn.example.com/photos/cat.jpg?discount=50%25/800x600.webp",
|
||||
want: &ParsedURL{
|
||||
Host: testHostCDN, Path: testPathCat, Query: "discount=50%25",
|
||||
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||
want: &Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN, Path: testPathCat, Query: "discount=50%25",
|
||||
},
|
||||
Transform: Transform{Size: Size{Width: 800, Height: 600}, Format: FormatWebP},
|
||||
},
|
||||
},
|
||||
}
|
||||
@@ -446,7 +432,7 @@ func TestParseImageURL_SourceURLHoldingPercent25(t *testing.T) {
|
||||
t.Fatalf("ParseImageURL() unexpected error = %v", err)
|
||||
}
|
||||
|
||||
assertParsedURL(t, got, tt.want)
|
||||
assertRequest(t, got, tt.want)
|
||||
})
|
||||
}
|
||||
}
|
||||
+19
-11
@@ -1,4 +1,4 @@
|
||||
package imgcache
|
||||
package imagerequest
|
||||
|
||||
import (
|
||||
"errors"
|
||||
@@ -10,35 +10,43 @@ func TestValidateImageRequest(t *testing.T) {
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
req ImageRequest
|
||||
req Request
|
||||
wantErr error
|
||||
}{
|
||||
{
|
||||
name: "within bounds",
|
||||
req: ImageRequest{Size: Size{Width: 800, Height: 600}, FitMode: FitCover},
|
||||
req: Request{Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600}, FitMode: FitCover,
|
||||
}},
|
||||
},
|
||||
{
|
||||
name: "original size and empty fit",
|
||||
req: ImageRequest{Size: Size{Width: 0, Height: 0}},
|
||||
req: Request{Transform: Transform{Size: Size{Width: 0, Height: 0}}},
|
||||
},
|
||||
{
|
||||
name: "width over limit",
|
||||
req: ImageRequest{Size: Size{Width: MaxDimension + 1, Height: 600}},
|
||||
name: "width over limit",
|
||||
req: Request{Transform: Transform{
|
||||
Size: Size{Width: MaxDimension + 1, Height: 600},
|
||||
}},
|
||||
wantErr: ErrDimensionTooLarge,
|
||||
},
|
||||
{
|
||||
name: "height over limit",
|
||||
req: ImageRequest{Size: Size{Width: 800, Height: MaxDimension + 1}},
|
||||
name: "height over limit",
|
||||
req: Request{Transform: Transform{
|
||||
Size: Size{Width: 800, Height: MaxDimension + 1},
|
||||
}},
|
||||
wantErr: ErrDimensionTooLarge,
|
||||
},
|
||||
{
|
||||
name: "negative width",
|
||||
req: ImageRequest{Size: Size{Width: -1, Height: 600}},
|
||||
req: Request{Transform: Transform{Size: Size{Width: -1, Height: 600}}},
|
||||
wantErr: ErrInvalidSize,
|
||||
},
|
||||
{
|
||||
name: "invalid fit mode",
|
||||
req: ImageRequest{Size: Size{Width: 800, Height: 600}, FitMode: "bogus"},
|
||||
name: "invalid fit mode",
|
||||
req: Request{Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600}, FitMode: "bogus",
|
||||
}},
|
||||
wantErr: ErrInvalidFitMode,
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package imagerequest
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// VariantKey is a SHA256 hash identifying a specific image variant (hex-encoded).
|
||||
type VariantKey string
|
||||
|
||||
// CacheKey generates a unique key for a request variant.
|
||||
// Format: sha256(host:path:query:width:height:format:quality:fit_mode)
|
||||
// The path and query go through escapePercentAndColon, so that a ":" in
|
||||
// either cannot be taken for the one between them.
|
||||
func CacheKey(req *Request) VariantKey {
|
||||
data := fmt.Sprintf("%s:%s:%s:%d:%d:%s:%d:%s",
|
||||
req.Source.Host,
|
||||
escapePercentAndColon(req.Source.Path),
|
||||
escapePercentAndColon(req.Source.Query),
|
||||
req.Transform.Size.Width,
|
||||
req.Transform.Size.Height,
|
||||
req.Transform.Format,
|
||||
req.Transform.Quality,
|
||||
req.Transform.FitMode,
|
||||
)
|
||||
h := sha256.Sum256([]byte(data))
|
||||
|
||||
return VariantKey(hex.EncodeToString(h[:]))
|
||||
}
|
||||
|
||||
// escapePercentAndColon returns s with each "%" written as "%25" and each
|
||||
// ":" as "%3A", so that the result holds no ":".
|
||||
func escapePercentAndColon(s string) string {
|
||||
s = strings.ReplaceAll(s, "%", "%25")
|
||||
|
||||
return strings.ReplaceAll(s, ":", "%3A")
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
package imagerequest
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestCacheKey(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req1 := &Request{
|
||||
Source: Source{Host: testHostCDN, Path: testPathCat, Query: ""},
|
||||
Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||
Quality: 85, FitMode: FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
req2 := &Request{
|
||||
Source: Source{Host: testHostCDN, Path: testPathCat, Query: ""},
|
||||
Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||
Quality: 85, FitMode: FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Same request should produce same key
|
||||
key1 := CacheKey(req1)
|
||||
key2 := CacheKey(req2)
|
||||
|
||||
if key1 != key2 {
|
||||
t.Errorf("CacheKey() not deterministic: %s vs %s", key1, key2)
|
||||
}
|
||||
|
||||
// Key should be 64 hex chars
|
||||
if len(string(key1)) != 64 {
|
||||
t.Errorf("CacheKey() length = %d, want 64", len(string(key1)))
|
||||
}
|
||||
|
||||
// Different size should produce different key
|
||||
req3 := &Request{
|
||||
Source: Source{Host: testHostCDN, Path: testPathCat, Query: ""},
|
||||
Transform: Transform{
|
||||
Size: Size{Width: 400, Height: 300}, Format: FormatWebP, // Different size
|
||||
Quality: 85, FitMode: FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
key3 := CacheKey(req3)
|
||||
if key1 == key3 {
|
||||
t.Error("CacheKey() produced same key for different sizes")
|
||||
}
|
||||
|
||||
// Different format should produce different key
|
||||
req4 := &Request{
|
||||
Source: Source{Host: testHostCDN, Path: testPathCat, Query: ""},
|
||||
Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600}, Format: FormatPNG, // Different format
|
||||
Quality: 85, FitMode: FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
key4 := CacheKey(req4)
|
||||
if key1 == key4 {
|
||||
t.Error("CacheKey() produced same key for different formats")
|
||||
}
|
||||
|
||||
// Different quality should produce different key
|
||||
req5 := &Request{
|
||||
Source: Source{Host: testHostCDN, Path: testPathCat, Query: ""},
|
||||
Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||
Quality: 50, FitMode: FitCover, // Different quality
|
||||
},
|
||||
}
|
||||
|
||||
key5 := CacheKey(req5)
|
||||
if key1 == key5 {
|
||||
t.Error("CacheKey() produced same key for different quality")
|
||||
}
|
||||
}
|
||||
|
||||
// TestCacheKey_ColonOrPercentInPathOrQuery checks that two different source
|
||||
// URLs whose path or query holds ":" or "%" get different keys.
|
||||
func TestCacheKey_ColonOrPercentInPathOrQuery(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
path1, query1 string
|
||||
path2, query2 string
|
||||
}{
|
||||
{
|
||||
name: "colon moved from path to query",
|
||||
path1: "/a:b", query1: "",
|
||||
path2: "/a", query2: "b:",
|
||||
},
|
||||
{
|
||||
name: "colon in path and its percent-encoding",
|
||||
path1: "/a:b", query1: "",
|
||||
path2: "/a%3Ab", query2: "",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range cases {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
key1 := CacheKey(&Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN,
|
||||
Path: tt.path1,
|
||||
Query: tt.query1,
|
||||
},
|
||||
Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
},
|
||||
})
|
||||
key2 := CacheKey(&Request{
|
||||
Source: Source{
|
||||
Host: testHostCDN,
|
||||
Path: tt.path2,
|
||||
Query: tt.query2,
|
||||
},
|
||||
Transform: Transform{
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
},
|
||||
})
|
||||
|
||||
t.Logf("CacheKey() of path %q query %q = %s", tt.path1, tt.query1, key1)
|
||||
t.Logf("CacheKey() of path %q query %q = %s", tt.path2, tt.query2, key2)
|
||||
|
||||
if key1 == key2 {
|
||||
t.Errorf("CacheKey() gave path %q query %q and path %q query %q "+
|
||||
"the same key", tt.path1, tt.query1, tt.path2, tt.query2)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+77
-176
@@ -2,10 +2,7 @@ package imgcache
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
@@ -17,6 +14,7 @@ import (
|
||||
|
||||
lru "github.com/hashicorp/golang-lru/v2"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// Cache errors.
|
||||
@@ -25,9 +23,6 @@ var (
|
||||
ErrNegativeCache = errors.New("negative cache hit")
|
||||
)
|
||||
|
||||
// HTTP status code for successful fetch.
|
||||
const httpStatusOK = 200
|
||||
|
||||
// metaCacheSize is how many variants' content types metaCache holds. A
|
||||
// variant not among them is served as before, reading its .meta file.
|
||||
const metaCacheSize = 10000
|
||||
@@ -68,12 +63,11 @@ type CacheConfig struct {
|
||||
|
||||
// Cache implements the caching layer for the image proxy.
|
||||
type Cache struct {
|
||||
db *sql.DB
|
||||
srcContent *ContentStorage // source images by content hash
|
||||
variants *VariantStorage // processed variants by cache key
|
||||
srcMetadata *MetadataStorage // source metadata by host/path
|
||||
config CacheConfig
|
||||
log *slog.Logger
|
||||
db *sql.DB
|
||||
srcContent *ContentStorage // source images by source key
|
||||
variants *VariantStorage // processed variants by cache key
|
||||
config CacheConfig
|
||||
log *slog.Logger
|
||||
|
||||
// disabled means the disk cache is turned off entirely: lookups
|
||||
// always miss, stores are no-ops, and no eviction runs.
|
||||
@@ -111,20 +105,7 @@ type Cache struct {
|
||||
// metaCache holds the content types of the variants most recently
|
||||
// stored or served, so a hit does not read the variant's .meta file.
|
||||
// It never stands in for the variant file, which is always opened.
|
||||
metaCache *lru.Cache[VariantKey, string]
|
||||
|
||||
// contentLocks serializes StoreSource and evictSourceBlob per
|
||||
// content hash, closing the race window between an eviction's row
|
||||
// deletion and its file unlink against a concurrent store of
|
||||
// identical content.
|
||||
contentLocks *contentLock
|
||||
|
||||
// evictSourceBlobTestHook, when set, is invoked by evictSourceBlob
|
||||
// after its row-deletion transaction commits and before the
|
||||
// content file is unlinked. It exists solely so tests can
|
||||
// deterministically pause inside that window to exercise
|
||||
// concurrent stores against it; production code leaves it nil.
|
||||
evictSourceBlobTestHook func(ContentHash)
|
||||
metaCache *lru.Cache[imagerequest.VariantKey, string]
|
||||
|
||||
// reconciliationPageSize is the most rows one read of a content table
|
||||
// returns in the reconciliation pass and in Stats. newCache sets it to
|
||||
@@ -153,7 +134,7 @@ func newCache(
|
||||
log = slog.Default()
|
||||
}
|
||||
|
||||
metaCache, err := lru.New[VariantKey, string](metaCacheSize)
|
||||
metaCache, err := lru.New[imagerequest.VariantKey, string](metaCacheSize)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create variant content type cache: %w", err)
|
||||
}
|
||||
@@ -166,7 +147,6 @@ func newCache(
|
||||
evictionPressure: make(chan struct{}, 1),
|
||||
evictionDone: make(chan struct{}),
|
||||
metaCache: metaCache,
|
||||
contentLocks: newContentLock(),
|
||||
|
||||
pendingCountsAdded: make(chan struct{}, 1),
|
||||
pendingCountsDone: make(chan struct{}),
|
||||
@@ -192,16 +172,8 @@ func newCache(
|
||||
return nil, fmt.Errorf("failed to create variant storage: %w", err)
|
||||
}
|
||||
|
||||
srcMetadata, err := NewMetadataStorage(
|
||||
filepath.Join(config.StateDir, "cache", "metadata"),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create source metadata storage: %w", err)
|
||||
}
|
||||
|
||||
c.srcContent = srcContent
|
||||
c.variants = variants
|
||||
c.srcMetadata = srcMetadata
|
||||
|
||||
if config.UseDefaultMaxBytes {
|
||||
limit, err := c.computeDefaultMaxBytes(context.Background(), probe)
|
||||
@@ -223,7 +195,7 @@ func newCache(
|
||||
// LookupResult contains the result of a cache lookup.
|
||||
type LookupResult struct {
|
||||
Hit bool
|
||||
CacheKey VariantKey
|
||||
CacheKey imagerequest.VariantKey
|
||||
ContentType string
|
||||
SizeBytes int64
|
||||
CacheStatus CacheStatus
|
||||
@@ -231,8 +203,10 @@ type LookupResult struct {
|
||||
|
||||
// Lookup checks if a processed variant exists on disk. Hits touch the
|
||||
// variant's LRU timestamp; a disabled cache always misses.
|
||||
func (c *Cache) Lookup(ctx context.Context, req *ImageRequest) (*LookupResult, error) {
|
||||
cacheKey := CacheKey(req)
|
||||
func (c *Cache) Lookup(
|
||||
ctx context.Context, req *imagerequest.Request,
|
||||
) (*LookupResult, error) {
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
|
||||
// Check variant storage directly - no DB needed for cache hits
|
||||
if !c.disabled && c.variants.Exists(cacheKey) {
|
||||
@@ -257,7 +231,9 @@ func (c *Cache) Lookup(ctx context.Context, req *ImageRequest) (*LookupResult, e
|
||||
// variant's .meta file and is then kept in metaCache. A variant with
|
||||
// no .meta file is served as application/octet-stream, which is not
|
||||
// kept.
|
||||
func (c *Cache) GetVariant(cacheKey VariantKey) (io.ReadCloser, int64, string, error) {
|
||||
func (c *Cache) GetVariant(
|
||||
cacheKey imagerequest.VariantKey,
|
||||
) (io.ReadCloser, int64, string, error) {
|
||||
if c.disabled {
|
||||
return nil, 0, "", ErrNotFound
|
||||
}
|
||||
@@ -278,88 +254,59 @@ func (c *Cache) GetVariant(cacheKey VariantKey) (io.ReadCloser, int64, string, e
|
||||
return reader, size, contentType, nil
|
||||
}
|
||||
|
||||
// StoreSource stores fetched source content and metadata. On a
|
||||
// disabled cache it is a no-op returning an empty hash.
|
||||
// StoreSource stores fetched source content: it writes the file named by
|
||||
// the source's key, then the source's row. When the row cannot be written
|
||||
// the file is removed and the store fails, as a file with no row is never
|
||||
// served or evicted; if that removal fails too, the next reconciliation
|
||||
// pass removes the file. On a disabled cache it is a no-op returning an
|
||||
// empty key.
|
||||
func (c *Cache) StoreSource(
|
||||
ctx context.Context,
|
||||
req *ImageRequest,
|
||||
req *imagerequest.Request,
|
||||
content io.Reader,
|
||||
result *httpfetcher.FetchResult,
|
||||
) (ContentHash, error) {
|
||||
) (imagerequest.SourceKey, error) {
|
||||
if c.disabled {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// Hash the content ourselves (rather than via srcContent.Store,
|
||||
// which would hash internally) so the content hash is known before
|
||||
// any file or database work happens: that lets the entire store be
|
||||
// serialized, per hash, against a concurrent eviction of the same
|
||||
// content below.
|
||||
data, err := io.ReadAll(content)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to read source content: %w", err)
|
||||
}
|
||||
|
||||
sum := sha256.Sum256(data)
|
||||
contentHash := ContentHash(hex.EncodeToString(sum[:]))
|
||||
key := req.Source.Key()
|
||||
|
||||
// Hold the content hash's lock for the whole store operation. A
|
||||
// concurrent eviction of this exact hash (the real SHA-256 dedup
|
||||
// case: a different source path whose bytes hash identically)
|
||||
// deletes the accounting rows and unlinks the file inside the same
|
||||
// lock, so the two can never interleave: either this store
|
||||
// completes first (and a subsequent eviction removes it together
|
||||
// with its rows and file, correctly), or eviction completes first
|
||||
// (and this store finds the file already gone and recreates it
|
||||
// fresh) — never a fresh row left pointing at a file eviction is
|
||||
// mid-unlink on.
|
||||
unlock := c.contentLocks.Lock(string(contentHash))
|
||||
defer unlock()
|
||||
|
||||
size, err := c.srcContent.StoreHashed(contentHash, data)
|
||||
size, err := c.srcContent.Store(key, data)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to store source content: %w", err)
|
||||
}
|
||||
|
||||
// Store in database
|
||||
pathHash := HashPath(req.SourcePath + "?" + req.SourceQuery)
|
||||
|
||||
headersJSON, err := json.Marshal(result.Headers)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to marshal response headers: %w", err)
|
||||
}
|
||||
|
||||
_, err = c.db.ExecContext(ctx, `
|
||||
INSERT INTO source_content (content_hash, content_type, size_bytes)
|
||||
VALUES (?, ?, ?)
|
||||
ON CONFLICT(content_hash) DO NOTHING
|
||||
`, contentHash, result.ContentType, size)
|
||||
INSERT INTO source_content
|
||||
(source_key, source_host, source_path, source_query,
|
||||
content_type, size_bytes)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(source_key) DO UPDATE SET
|
||||
content_type = excluded.content_type,
|
||||
size_bytes = excluded.size_bytes,
|
||||
fetched_at = CURRENT_TIMESTAMP,
|
||||
last_accessed_at = CURRENT_TIMESTAMP
|
||||
`, string(key), req.Source.Host, req.Source.Path, req.Source.Query,
|
||||
result.ContentType, size)
|
||||
if err != nil {
|
||||
deleteErr := c.srcContent.Delete(key)
|
||||
if deleteErr != nil {
|
||||
c.log.Warn("failed to remove source file whose row was not written",
|
||||
"source_key", key, "error", deleteErr)
|
||||
}
|
||||
|
||||
return "", fmt.Errorf("failed to insert source content: %w", err)
|
||||
}
|
||||
|
||||
_, err = c.db.ExecContext(ctx, `
|
||||
INSERT INTO source_metadata
|
||||
(source_host, source_path, source_query, path_hash,
|
||||
content_hash, status_code, content_type, response_headers)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(source_host, source_path, source_query) DO UPDATE SET
|
||||
content_hash = excluded.content_hash,
|
||||
status_code = excluded.status_code,
|
||||
content_type = excluded.content_type,
|
||||
response_headers = excluded.response_headers,
|
||||
fetched_at = CURRENT_TIMESTAMP
|
||||
`, req.SourceHost, req.SourcePath, req.SourceQuery, pathHash,
|
||||
contentHash, httpStatusOK, result.ContentType, string(headersJSON))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to insert source metadata: %w", err)
|
||||
}
|
||||
|
||||
c.writeMetadataSidecar(req, pathHash, contentHash, result)
|
||||
|
||||
c.notifyWritePressure()
|
||||
|
||||
return contentHash, nil
|
||||
return key, nil
|
||||
}
|
||||
|
||||
// StoreVariant stores a processed variant by its cache key and records
|
||||
@@ -367,7 +314,8 @@ func (c *Cache) StoreSource(
|
||||
// accounting insert is best-effort (the startup reconciliation pass
|
||||
// adopts any variant file that misses its accounting row).
|
||||
func (c *Cache) StoreVariant(
|
||||
ctx context.Context, cacheKey VariantKey, content io.Reader, contentType string,
|
||||
ctx context.Context, cacheKey imagerequest.VariantKey,
|
||||
content io.Reader, contentType string,
|
||||
) error {
|
||||
if c.disabled {
|
||||
return nil
|
||||
@@ -399,22 +347,24 @@ func (c *Cache) StoreVariant(
|
||||
}
|
||||
|
||||
// LookupSource checks if we have cached source content for a request.
|
||||
// Returns the content hash and content type if found, or empty values
|
||||
// if not. Hits touch the blob's LRU timestamp; a disabled cache always
|
||||
// reports no cached source.
|
||||
// Returns the source key and content type if found, or empty values
|
||||
// if not; a row whose file is missing is not found. Hits touch the
|
||||
// source's LRU timestamp; a disabled cache always reports no cached
|
||||
// source.
|
||||
func (c *Cache) LookupSource(
|
||||
ctx context.Context, req *ImageRequest,
|
||||
) (ContentHash, string, error) {
|
||||
ctx context.Context, req *imagerequest.Request,
|
||||
) (imagerequest.SourceKey, string, error) {
|
||||
if c.disabled {
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
var hashStr, contentType string
|
||||
key := req.Source.Key()
|
||||
|
||||
var contentType string
|
||||
|
||||
err := c.db.QueryRowContext(ctx, `
|
||||
SELECT content_hash, content_type FROM source_metadata
|
||||
WHERE source_host = ? AND source_path = ? AND source_query = ?
|
||||
`, req.SourceHost, req.SourcePath, req.SourceQuery).Scan(&hashStr, &contentType)
|
||||
SELECT content_type FROM source_content WHERE source_key = ?
|
||||
`, string(key)).Scan(&contentType)
|
||||
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return "", "", nil
|
||||
@@ -424,21 +374,19 @@ func (c *Cache) LookupSource(
|
||||
return "", "", fmt.Errorf("failed to lookup source: %w", err)
|
||||
}
|
||||
|
||||
contentHash := ContentHash(hashStr)
|
||||
|
||||
// Verify the content file exists
|
||||
if !c.srcContent.Exists(contentHash) {
|
||||
if !c.srcContent.Exists(key) {
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
c.touchSourceContent(ctx, contentHash)
|
||||
c.touchSourceContent(ctx, key)
|
||||
|
||||
return contentHash, contentType, nil
|
||||
return key, contentType, nil
|
||||
}
|
||||
|
||||
// StoreNegative stores a negative cache entry for a failed fetch.
|
||||
func (c *Cache) StoreNegative(
|
||||
ctx context.Context, req *ImageRequest, statusCode int, errMsg string,
|
||||
ctx context.Context, req *imagerequest.Request, statusCode int, errMsg string,
|
||||
) error {
|
||||
expiresAt := time.Now().UTC().Add(c.config.NegativeTTL)
|
||||
|
||||
@@ -452,7 +400,7 @@ func (c *Cache) StoreNegative(
|
||||
error_message = excluded.error_message,
|
||||
fetched_at = CURRENT_TIMESTAMP,
|
||||
expires_at = excluded.expires_at
|
||||
`, req.SourceHost, req.SourcePath, req.SourceQuery, statusCode, errMsg, expiresAt)
|
||||
`, req.Source.Host, req.Source.Path, req.Source.Query, statusCode, errMsg, expiresAt)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to insert negative cache: %w", err)
|
||||
}
|
||||
@@ -460,33 +408,16 @@ func (c *Cache) StoreNegative(
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetSourceMetadataID returns the source metadata ID for a request.
|
||||
func (c *Cache) GetSourceMetadataID(
|
||||
ctx context.Context, req *ImageRequest,
|
||||
) (int64, error) {
|
||||
var id int64
|
||||
|
||||
err := c.db.QueryRowContext(ctx, `
|
||||
SELECT id FROM source_metadata
|
||||
WHERE source_host = ? AND source_path = ? AND source_query = ?
|
||||
`, req.SourceHost, req.SourcePath, req.SourceQuery).Scan(&id)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("failed to get source metadata ID: %w", err)
|
||||
}
|
||||
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// GetSourceContent returns a reader for cached source content by its hash,
|
||||
// GetSourceContent returns a reader for cached source content by its key,
|
||||
// and the content's size in bytes.
|
||||
func (c *Cache) GetSourceContent(
|
||||
contentHash ContentHash,
|
||||
key imagerequest.SourceKey,
|
||||
) (io.ReadCloser, int64, error) {
|
||||
if c.disabled {
|
||||
return nil, 0, ErrNotFound
|
||||
}
|
||||
|
||||
return c.srcContent.LoadWithSize(contentHash)
|
||||
return c.srcContent.LoadWithSize(key)
|
||||
}
|
||||
|
||||
// CleanExpired removes the negative cache rows past their expiry, one
|
||||
@@ -665,7 +596,7 @@ func (c *Cache) readHitAndMissCounts(ctx context.Context, stats *CacheStats) err
|
||||
// application/octet-stream and keeps nothing, so metaCache only ever
|
||||
// holds a type read from a .meta file or passed to StoreVariant.
|
||||
func (c *Cache) loadVariantWithMeta(
|
||||
cacheKey VariantKey,
|
||||
cacheKey imagerequest.VariantKey,
|
||||
) (io.ReadCloser, int64, string, error) {
|
||||
reader, size, contentType, err := c.variants.LoadWithMeta(cacheKey)
|
||||
if err != nil {
|
||||
@@ -681,39 +612,9 @@ func (c *Cache) loadVariantWithMeta(
|
||||
return reader, size, contentType, nil
|
||||
}
|
||||
|
||||
// writeMetadataSidecar writes the JSON metadata sidecar of a stored source.
|
||||
// A failure is logged and is otherwise non-fatal; the metadata is in the
|
||||
// database.
|
||||
func (c *Cache) writeMetadataSidecar(
|
||||
req *ImageRequest,
|
||||
pathHash PathHash,
|
||||
contentHash ContentHash,
|
||||
result *httpfetcher.FetchResult,
|
||||
) {
|
||||
meta := &SourceMetadata{
|
||||
Host: req.SourceHost,
|
||||
Path: req.SourcePath,
|
||||
Query: req.SourceQuery,
|
||||
ContentHash: string(contentHash),
|
||||
StatusCode: result.StatusCode,
|
||||
ContentType: result.ContentType,
|
||||
ContentLength: result.ContentLength,
|
||||
ResponseHeaders: result.Headers,
|
||||
FetchedAt: time.Now().UTC().Unix(),
|
||||
FetchDurationMs: result.FetchDurationMs,
|
||||
RemoteAddr: result.RemoteAddr,
|
||||
}
|
||||
|
||||
err := c.srcMetadata.Store(req.SourceHost, pathHash, meta)
|
||||
if err != nil {
|
||||
c.log.Warn("failed to write metadata sidecar",
|
||||
"host", req.SourceHost, "path_hash", pathHash, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// touchVariant updates the LRU timestamp of a variant, best-effort:
|
||||
// a failed touch only makes the entry look colder to eviction.
|
||||
func (c *Cache) touchVariant(ctx context.Context, cacheKey VariantKey) {
|
||||
func (c *Cache) touchVariant(ctx context.Context, cacheKey imagerequest.VariantKey) {
|
||||
_, err := c.db.ExecContext(ctx, `
|
||||
UPDATE variant_content SET last_accessed_at = CURRENT_TIMESTAMP
|
||||
WHERE cache_key = ?
|
||||
@@ -724,29 +625,29 @@ func (c *Cache) touchVariant(ctx context.Context, cacheKey VariantKey) {
|
||||
}
|
||||
}
|
||||
|
||||
// touchSourceContent updates the LRU timestamp of a source content
|
||||
// blob, best-effort: a failed touch only makes the blob look colder.
|
||||
func (c *Cache) touchSourceContent(ctx context.Context, contentHash ContentHash) {
|
||||
// touchSourceContent updates the LRU timestamp of a cached source,
|
||||
// best-effort: a failed touch only makes the source look colder.
|
||||
func (c *Cache) touchSourceContent(ctx context.Context, key imagerequest.SourceKey) {
|
||||
_, err := c.db.ExecContext(ctx, `
|
||||
UPDATE source_content SET last_accessed_at = CURRENT_TIMESTAMP
|
||||
WHERE content_hash = ?
|
||||
`, string(contentHash))
|
||||
WHERE source_key = ?
|
||||
`, string(key))
|
||||
if err != nil {
|
||||
c.log.Debug("failed to touch source content LRU timestamp",
|
||||
"content_hash", contentHash, "error", err)
|
||||
"source_key", key, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// checkNegativeCache checks if a request is in the negative cache.
|
||||
func (c *Cache) checkNegativeCache(
|
||||
ctx context.Context, req *ImageRequest,
|
||||
ctx context.Context, req *imagerequest.Request,
|
||||
) (bool, error) {
|
||||
var expiresAt time.Time
|
||||
|
||||
err := c.db.QueryRowContext(ctx, `
|
||||
SELECT expires_at FROM negative_cache
|
||||
WHERE source_host = ? AND source_path = ? AND source_query = ?
|
||||
`, req.SourceHost, req.SourcePath, req.SourceQuery).Scan(&expiresAt)
|
||||
`, req.Source.Host, req.Source.Path, req.Source.Query).Scan(&expiresAt)
|
||||
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return false, nil
|
||||
@@ -762,10 +663,10 @@ func (c *Cache) checkNegativeCache(
|
||||
_, err = c.db.ExecContext(ctx, `
|
||||
DELETE FROM negative_cache
|
||||
WHERE source_host = ? AND source_path = ? AND source_query = ?
|
||||
`, req.SourceHost, req.SourcePath, req.SourceQuery)
|
||||
`, req.Source.Host, req.Source.Path, req.Source.Query)
|
||||
if err != nil {
|
||||
c.log.Warn("failed to delete expired negative cache entry",
|
||||
"host", req.SourceHost, "path", req.SourcePath, "error", err)
|
||||
"host", req.Source.Host, "path", req.Source.Path, "error", err)
|
||||
}
|
||||
|
||||
return false, nil
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
func setupTestDB(t *testing.T) *sql.DB {
|
||||
@@ -23,26 +24,14 @@ func setupTestDB(t *testing.T) *sql.DB {
|
||||
// Create schema
|
||||
schema := `
|
||||
CREATE TABLE source_content (
|
||||
content_hash TEXT PRIMARY KEY,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes INTEGER NOT NULL,
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE TABLE source_metadata (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_key TEXT PRIMARY KEY,
|
||||
source_host TEXT NOT NULL,
|
||||
source_path TEXT NOT NULL,
|
||||
source_query TEXT NOT NULL DEFAULT '',
|
||||
path_hash TEXT NOT NULL,
|
||||
content_hash TEXT,
|
||||
status_code INTEGER NOT NULL,
|
||||
content_type TEXT,
|
||||
response_headers TEXT,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes INTEGER NOT NULL,
|
||||
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
expires_at DATETIME,
|
||||
etag TEXT,
|
||||
last_modified TEXT,
|
||||
UNIQUE(source_host, source_path, source_query)
|
||||
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE TABLE output_content (
|
||||
content_hash TEXT PRIMARY KEY,
|
||||
@@ -117,13 +106,17 @@ func TestCache_LookupMiss(t *testing.T) {
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: testPathCat,
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
result, err := cache.Lookup(ctx, req)
|
||||
@@ -146,13 +139,17 @@ func TestCache_StoreAndLookup(t *testing.T) {
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: testPathCat,
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Store source content
|
||||
@@ -162,18 +159,18 @@ func TestCache_StoreAndLookup(t *testing.T) {
|
||||
Headers: map[string][]string{testHeaderContentType: {testContentTypeJPEG}},
|
||||
}
|
||||
|
||||
contentHash, err := cache.StoreSource(
|
||||
sourceKey, err := cache.StoreSource(
|
||||
ctx, req, bytes.NewReader(sourceContent), fetchResult)
|
||||
if err != nil {
|
||||
t.Fatalf("StoreSource() error = %v", err)
|
||||
}
|
||||
|
||||
if contentHash == "" {
|
||||
t.Error("StoreSource() returned empty hash")
|
||||
if sourceKey == "" {
|
||||
t.Error("StoreSource() returned empty key")
|
||||
}
|
||||
|
||||
// Store variant
|
||||
cacheKey := CacheKey(req)
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
outputContent := []byte("fake webp data")
|
||||
|
||||
err = cache.StoreVariant(
|
||||
@@ -207,11 +204,15 @@ func TestCache_NegativeCache(t *testing.T) {
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: "/photos/notfound.jpg",
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/photos/notfound.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
},
|
||||
}
|
||||
|
||||
// Store negative cache entry
|
||||
@@ -248,11 +249,15 @@ func TestCache_NegativeCacheExpiry(t *testing.T) {
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: "/photos/expired.jpg",
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/photos/expired.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
},
|
||||
}
|
||||
|
||||
// Store negative cache entry
|
||||
@@ -281,17 +286,21 @@ func TestCache_VariantLookup(t *testing.T) {
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: "/photos/variant.jpg",
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/photos/variant.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Store variant
|
||||
cacheKey := CacheKey(req)
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
outputContent := []byte("output data")
|
||||
|
||||
err := cache.StoreVariant(
|
||||
@@ -331,17 +340,21 @@ func TestCache_GetVariant_ReturnsContentType(t *testing.T) {
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: "/photos/variantct.jpg",
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/photos/variantct.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Store variant
|
||||
cacheKey := CacheKey(req)
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
outputContent := []byte("output webp data")
|
||||
|
||||
err := cache.StoreVariant(
|
||||
@@ -383,17 +396,21 @@ func TestCache_GetVariant(t *testing.T) {
|
||||
cache := setupTestCache(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: "/photos/output.jpg",
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: "/photos/output.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Store variant
|
||||
cacheKey := CacheKey(req)
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
outputContent := []byte("the actual output content")
|
||||
|
||||
err := cache.StoreVariant(
|
||||
@@ -518,7 +535,6 @@ func TestCache_StorageDirectoriesCreated(t *testing.T) {
|
||||
dirs := []string{
|
||||
"cache/sources",
|
||||
"cache/variants",
|
||||
"cache/metadata",
|
||||
}
|
||||
|
||||
for _, dir := range dirs {
|
||||
|
||||
@@ -5,6 +5,8 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestEvictionReadsTheUsageTotal adds, stores again, resizes and evicts
|
||||
@@ -29,10 +31,10 @@ func TestEvictionReadsTheUsageTotal(t *testing.T) {
|
||||
storeEvictionTestVariant(t, cache, testVariantKeyTwo,
|
||||
bytes.Repeat([]byte{0x75}, 200))
|
||||
|
||||
// pixa never changes a source's size, but a statement that does must
|
||||
// change the total too.
|
||||
// Storing a source URL again changes its size, so the total must
|
||||
// follow a statement that changes it.
|
||||
_, err := cache.db.ExecContext(ctx,
|
||||
`UPDATE source_content SET size_bytes = 900 WHERE content_hash = ?`,
|
||||
`UPDATE source_content SET size_bytes = 900 WHERE source_key = ?`,
|
||||
string(kept))
|
||||
if err != nil {
|
||||
t.Fatalf("failed to change the source's size: %v", err)
|
||||
@@ -153,8 +155,9 @@ func TestUsageTotalNotCorrectedFromAnOlderSum(t *testing.T) {
|
||||
// pass. No read the pass makes of a content table, to check its rows or
|
||||
// to sum them, may cover more than two rows, so a request's query waits
|
||||
// for one page at most. Between them the reads must still cover every
|
||||
// row of both tables twice, once to check it and once to sum it, and the
|
||||
// sum must put a wrong total right.
|
||||
// variant row twice, once to check it and once to sum it, and every
|
||||
// source row once, as source rows are read only to sum them, and the sum
|
||||
// must put a wrong total right.
|
||||
func TestReconciliationReadsAPageAtATime(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -171,7 +174,8 @@ func TestReconciliationReadsAPageAtATime(t *testing.T) {
|
||||
|
||||
storeEvictionTestSource(t, cache, "pages.example.com",
|
||||
"/"+strconv.Itoa(i)+".jpg", bytes.Repeat(content, 100))
|
||||
storeEvictionTestVariant(t, cache, VariantKey("aabbccdd000"+strconv.Itoa(i)),
|
||||
storeEvictionTestVariant(t, cache,
|
||||
imagerequest.VariantKey("aabbccdd000"+strconv.Itoa(i)),
|
||||
bytes.Repeat(content, 10))
|
||||
}
|
||||
|
||||
@@ -204,9 +208,10 @@ func TestReconciliationReadsAPageAtATime(t *testing.T) {
|
||||
coveredRows += rows
|
||||
}
|
||||
|
||||
// Ten rows, each read once to check it and once to sum it.
|
||||
if coveredRows != 20 {
|
||||
t.Errorf("the reads covered %d rows in all, want 20", coveredRows)
|
||||
// Five variant rows, each read once to check it and once to sum it,
|
||||
// and five source rows, read only to sum them.
|
||||
if coveredRows != 15 {
|
||||
t.Errorf("the reads covered %d rows in all, want 15", coveredRows)
|
||||
}
|
||||
|
||||
usage, err := cache.UsageBytes(ctx)
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"github.com/getsentry/sentry-go"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
|
||||
@@ -80,14 +81,20 @@ func setupHeldFetchService(t *testing.T) (*Service, *TestFixtures, *heldFetcher)
|
||||
// photoVariant asks for the test photo, 100x100, at 50x25 as a JPEG of the
|
||||
// given quality and fit mode. Each call returns a new request, as Get writes
|
||||
// to the request it is given.
|
||||
func photoVariant(fixtures *TestFixtures, quality int, fit FitMode) *ImageRequest {
|
||||
return &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: Size{Width: 50, Height: 25},
|
||||
Format: FormatJPEG,
|
||||
Quality: quality,
|
||||
FitMode: fit,
|
||||
func photoVariant(
|
||||
fixtures *TestFixtures, quality int, fit imagerequest.Fit,
|
||||
) *imagerequest.Request {
|
||||
return &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 25},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: quality,
|
||||
FitMode: fit,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -100,7 +107,7 @@ type getResult struct {
|
||||
// startGet calls Get in a goroutine of its own and delivers what it returned
|
||||
// on the channel.
|
||||
func startGet(
|
||||
ctx context.Context, svc *Service, req *ImageRequest,
|
||||
ctx context.Context, svc *Service, req *imagerequest.Request,
|
||||
) <-chan getResult {
|
||||
results := make(chan getResult, 1)
|
||||
|
||||
@@ -146,7 +153,7 @@ func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
|
||||
pending := make([]<-chan getResult, 0, requests)
|
||||
for range requests {
|
||||
pending = append(pending,
|
||||
startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover)))
|
||||
startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover)))
|
||||
}
|
||||
|
||||
<-fetcher.started
|
||||
@@ -199,9 +206,11 @@ func TestService_Get_ConcurrentVariantsStayApart(t *testing.T) {
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
|
||||
cover := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover))
|
||||
lowQuality := startGet(t.Context(), svc, photoVariant(fixtures, 40, FitCover))
|
||||
contain := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitContain))
|
||||
cover := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
lowQuality := startGet(t.Context(), svc,
|
||||
photoVariant(fixtures, 40, imagerequest.FitCover))
|
||||
contain := startGet(t.Context(), svc,
|
||||
photoVariant(fixtures, 85, imagerequest.FitContain))
|
||||
|
||||
for range 3 {
|
||||
select {
|
||||
@@ -253,12 +262,12 @@ func TestService_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
|
||||
first := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover))
|
||||
first := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
<-fetcher.started
|
||||
|
||||
waiterCtx, cancelWaiter := context.WithCancel(t.Context())
|
||||
waiter := startGet(waiterCtx, svc, photoVariant(fixtures, 85, FitCover))
|
||||
waiter := startGet(waiterCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
time.Sleep(arrivalWait)
|
||||
cancelWaiter()
|
||||
@@ -300,11 +309,11 @@ func TestService_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
|
||||
firstCtx, cancelFirst := context.WithCancel(t.Context())
|
||||
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, FitCover))
|
||||
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
<-fetcher.started
|
||||
|
||||
second := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover))
|
||||
second := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
time.Sleep(arrivalWait)
|
||||
cancelFirst()
|
||||
@@ -355,9 +364,9 @@ func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
|
||||
request := func() *ImageRequest {
|
||||
req := photoVariant(fixtures, 85, FitCover)
|
||||
req.SourcePath = tc.path
|
||||
request := func() *imagerequest.Request {
|
||||
req := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
req.Source.Path = tc.path
|
||||
|
||||
return req
|
||||
}
|
||||
@@ -399,7 +408,7 @@ func TestService_Get_EndedRequestFetchesNothing(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
cancel()
|
||||
|
||||
_, err := svc.Get(ctx, photoVariant(fixtures, 85, FitCover))
|
||||
_, err := svc.Get(ctx, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
t.Errorf("Get() error = %v, want %v", err, context.Canceled)
|
||||
}
|
||||
@@ -423,7 +432,7 @@ func TestService_Get_ReturnsByItsDeadline(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(t.Context(), timeout)
|
||||
defer cancel()
|
||||
|
||||
results := startGet(ctx, svc, photoVariant(fixtures, 85, FitCover))
|
||||
results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
select {
|
||||
case got := <-results:
|
||||
@@ -456,11 +465,11 @@ func TestService_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
|
||||
firstCtx, cancelFirst := context.WithTimeout(t.Context(), firstTimeout)
|
||||
defer cancelFirst()
|
||||
|
||||
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, FitCover))
|
||||
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
<-fetcher.started
|
||||
|
||||
waiter := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover))
|
||||
waiter := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
time.Sleep(arrivalWait)
|
||||
fetcher.releaseFetches()
|
||||
@@ -503,7 +512,7 @@ func TestService_Get_PanicBecomesAnError(t *testing.T) {
|
||||
}
|
||||
}()
|
||||
|
||||
_, err = svc.Get(t.Context(), photoVariant(fixtures, 85, FitCover))
|
||||
_, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
}()
|
||||
|
||||
if err == nil || !strings.Contains(err.Error(), "upstream fetcher panicked") {
|
||||
@@ -540,7 +549,7 @@ func TestService_Get_PanicIsReportedToSentry(t *testing.T) {
|
||||
}
|
||||
}()
|
||||
|
||||
_, _ = svc.Get(ctx, photoVariant(fixtures, 85, FitCover))
|
||||
_, _ = svc.Get(ctx, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
}()
|
||||
|
||||
events := transport.Events()
|
||||
|
||||
@@ -1,64 +0,0 @@
|
||||
package imgcache
|
||||
|
||||
import "sync"
|
||||
|
||||
// contentLock provides per-key mutual exclusion for content-hash keyed
|
||||
// operations. StoreSource and evictSourceBlob each hold a content
|
||||
// hash's lock for the full duration of their file-plus-accounting-row
|
||||
// work, so a store and an eviction racing on identical content bytes
|
||||
// (the real SHA-256 content-addressed dedup case, not a contrived one)
|
||||
// can never interleave: the unlink of an evicted blob's file can never
|
||||
// race the creation of a fresh database row for a concurrently
|
||||
// re-stored copy of the same content. Entries are removed once no
|
||||
// goroutine holds or is waiting for them, so a long-running process
|
||||
// does not accumulate memory proportional to the number of distinct
|
||||
// content hashes it has ever seen.
|
||||
type contentLock struct {
|
||||
mu sync.Mutex
|
||||
entries map[string]*contentLockEntry
|
||||
}
|
||||
|
||||
// contentLockEntry is one key's exclusion lock plus a count of
|
||||
// goroutines currently holding or waiting to acquire it, used to know
|
||||
// when it is safe to remove the entry from the map.
|
||||
type contentLockEntry struct {
|
||||
mu sync.Mutex
|
||||
count int
|
||||
}
|
||||
|
||||
// newContentLock creates an empty contentLock.
|
||||
func newContentLock() *contentLock {
|
||||
return &contentLock{entries: make(map[string]*contentLockEntry)}
|
||||
}
|
||||
|
||||
// Lock acquires exclusive access for key, blocking until it is
|
||||
// available, and returns a function that releases it. The caller must
|
||||
// invoke the returned function exactly once to release the lock.
|
||||
func (c *contentLock) Lock(key string) func() {
|
||||
c.mu.Lock()
|
||||
|
||||
entry, ok := c.entries[key]
|
||||
if !ok {
|
||||
entry = &contentLockEntry{}
|
||||
c.entries[key] = entry
|
||||
}
|
||||
|
||||
entry.count++
|
||||
|
||||
c.mu.Unlock()
|
||||
|
||||
entry.mu.Lock()
|
||||
|
||||
return func() {
|
||||
entry.mu.Unlock()
|
||||
|
||||
c.mu.Lock()
|
||||
|
||||
entry.count--
|
||||
if entry.count == 0 {
|
||||
delete(c.entries, key)
|
||||
}
|
||||
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
@@ -1,142 +0,0 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestContentLockExcludesSameKey verifies that two goroutines locking
|
||||
// the same key never run their critical sections concurrently.
|
||||
func TestContentLockExcludesSameKey(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
lock := newContentLock()
|
||||
|
||||
var (
|
||||
active atomic.Int32
|
||||
maxSeen int32
|
||||
wg sync.WaitGroup
|
||||
)
|
||||
|
||||
const goroutines = 20
|
||||
|
||||
wg.Add(goroutines)
|
||||
|
||||
for range goroutines {
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
|
||||
unlock := lock.Lock("same-key")
|
||||
defer unlock()
|
||||
|
||||
n := active.Add(1)
|
||||
|
||||
for {
|
||||
seen := atomic.LoadInt32(&maxSeen)
|
||||
if n <= seen || atomic.CompareAndSwapInt32(&maxSeen, seen, n) {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
time.Sleep(time.Millisecond)
|
||||
|
||||
active.Add(-1)
|
||||
}()
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
|
||||
if maxSeen != 1 {
|
||||
t.Errorf("max concurrent holders of the same key = %d, want 1", maxSeen)
|
||||
}
|
||||
}
|
||||
|
||||
// TestContentLockAllowsDifferentKeys verifies that locking distinct
|
||||
// keys does not serialize unrelated work: all goroutines must be able
|
||||
// to enter their critical sections at once, proven by every one of
|
||||
// them reaching the rendezvous point before any is allowed to
|
||||
// proceed.
|
||||
func TestContentLockAllowsDifferentKeys(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
lock := newContentLock()
|
||||
|
||||
const goroutines = 20
|
||||
|
||||
var (
|
||||
wg sync.WaitGroup
|
||||
inside atomic.Int32
|
||||
reached = make(chan struct{}, goroutines)
|
||||
)
|
||||
|
||||
wg.Add(goroutines)
|
||||
|
||||
release := make(chan struct{})
|
||||
|
||||
for i := range goroutines {
|
||||
key := string(rune('a' + i))
|
||||
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
|
||||
unlock := lock.Lock(key)
|
||||
defer unlock()
|
||||
|
||||
inside.Add(1)
|
||||
|
||||
reached <- struct{}{}
|
||||
|
||||
<-release
|
||||
}()
|
||||
}
|
||||
|
||||
// Every goroutine must reach the rendezvous point (i.e. acquire its
|
||||
// own key's lock) without needing any other to release first. If
|
||||
// keys were incorrectly serialized onto one underlying lock, only
|
||||
// one would get here and this would time out.
|
||||
for i := range goroutines {
|
||||
select {
|
||||
case <-reached:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatalf("only %d/%d goroutines locking distinct keys made progress; "+
|
||||
"keys may be incorrectly serialized", i, goroutines)
|
||||
}
|
||||
}
|
||||
|
||||
if n := inside.Load(); n != goroutines {
|
||||
t.Errorf("goroutines inside their critical section = %d, want %d",
|
||||
n, goroutines)
|
||||
}
|
||||
|
||||
close(release)
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// TestContentLockRemovesEntryAfterUnlock verifies that the internal
|
||||
// entries map does not grow without bound: once no goroutine holds or
|
||||
// awaits a key, its entry is removed.
|
||||
func TestContentLockRemovesEntryAfterUnlock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
lock := newContentLock()
|
||||
|
||||
unlock := lock.Lock("k")
|
||||
|
||||
lock.mu.Lock()
|
||||
if _, ok := lock.entries["k"]; !ok {
|
||||
lock.mu.Unlock()
|
||||
t.Fatal("entry missing while lock is held")
|
||||
}
|
||||
lock.mu.Unlock()
|
||||
|
||||
unlock()
|
||||
|
||||
lock.mu.Lock()
|
||||
defer lock.mu.Unlock()
|
||||
|
||||
if _, ok := lock.entries["k"]; ok {
|
||||
t.Error("entry for key still present after the last holder unlocked")
|
||||
}
|
||||
}
|
||||
+55
-217
@@ -11,6 +11,8 @@ import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// DefaultEvictionInterval is how often the background evictor checks
|
||||
@@ -82,11 +84,11 @@ func (c *Cache) UsageBytes(ctx context.Context) (int64, error) {
|
||||
|
||||
// evictionCandidate is one LRU eviction victim candidate: either a
|
||||
// processed variant (isVariant true, identified by cacheKey) or a
|
||||
// source content blob (identified by contentHash).
|
||||
// source content blob (identified by sourceKey).
|
||||
type evictionCandidate struct {
|
||||
isVariant bool
|
||||
cacheKey VariantKey
|
||||
contentHash ContentHash
|
||||
cacheKey imagerequest.VariantKey
|
||||
sourceKey imagerequest.SourceKey
|
||||
sizeBytes int64
|
||||
lastAccessedAt string
|
||||
}
|
||||
@@ -158,7 +160,7 @@ func (c *Cache) evictBatch(ctx context.Context, excessBytes int64) (int64, error
|
||||
|
||||
c.log.Warn("failed to evict cache entry",
|
||||
"cache_key", candidate.cacheKey,
|
||||
"content_hash", candidate.contentHash,
|
||||
"source_key", candidate.sourceKey,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
@@ -177,7 +179,7 @@ func (c *Cache) evictCandidate(ctx context.Context, candidate evictionCandidate)
|
||||
return c.evictVariant(ctx, candidate.cacheKey)
|
||||
}
|
||||
|
||||
return c.evictSourceBlob(ctx, candidate.contentHash)
|
||||
return c.evictSourceBlob(ctx, candidate.sourceKey)
|
||||
}
|
||||
|
||||
// evictionCandidates returns up to evictionBatchSize variants and
|
||||
@@ -240,7 +242,7 @@ func (c *Cache) variantCandidates(ctx context.Context) ([]evictionCandidate, err
|
||||
return nil, fmt.Errorf("failed to scan variant candidate: %w", err)
|
||||
}
|
||||
|
||||
candidate.cacheKey = VariantKey(key)
|
||||
candidate.cacheKey = imagerequest.VariantKey(key)
|
||||
candidates = append(candidates, candidate)
|
||||
}
|
||||
|
||||
@@ -257,9 +259,9 @@ func (c *Cache) variantCandidates(ctx context.Context) ([]evictionCandidate, err
|
||||
// SQLite reads the rows in order from that column's index instead of
|
||||
// sorting the whole table.
|
||||
const sourceCandidatesQuery = `
|
||||
SELECT content_hash, size_bytes, last_accessed_at
|
||||
SELECT source_key, size_bytes, last_accessed_at
|
||||
FROM source_content
|
||||
ORDER BY last_accessed_at ASC, content_hash ASC
|
||||
ORDER BY last_accessed_at ASC, source_key ASC
|
||||
LIMIT ?`
|
||||
|
||||
// sourceCandidates returns the least recently used source blobs.
|
||||
@@ -276,14 +278,14 @@ func (c *Cache) sourceCandidates(ctx context.Context) ([]evictionCandidate, erro
|
||||
for rows.Next() {
|
||||
var candidate evictionCandidate
|
||||
|
||||
var hash string
|
||||
var key string
|
||||
|
||||
err := rows.Scan(&hash, &candidate.sizeBytes, &candidate.lastAccessedAt)
|
||||
err := rows.Scan(&key, &candidate.sizeBytes, &candidate.lastAccessedAt)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to scan source candidate: %w", err)
|
||||
}
|
||||
|
||||
candidate.contentHash = ContentHash(hash)
|
||||
candidate.sourceKey = imagerequest.SourceKey(key)
|
||||
candidates = append(candidates, candidate)
|
||||
}
|
||||
|
||||
@@ -300,7 +302,9 @@ func (c *Cache) sourceCandidates(ctx context.Context) ([]evictionCandidate, erro
|
||||
// file. The metaCache entry goes before the files; a GetVariant that
|
||||
// read them just before may put it back, and the next GetVariant then
|
||||
// fails to open the file and removes it again.
|
||||
func (c *Cache) evictVariant(ctx context.Context, cacheKey VariantKey) error {
|
||||
func (c *Cache) evictVariant(
|
||||
ctx context.Context, cacheKey imagerequest.VariantKey,
|
||||
) error {
|
||||
_, err := c.db.ExecContext(ctx,
|
||||
`DELETE FROM variant_content WHERE cache_key = ?`, string(cacheKey))
|
||||
if err != nil {
|
||||
@@ -317,117 +321,18 @@ func (c *Cache) evictVariant(ctx context.Context, cacheKey VariantKey) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// sourceReference identifies one source_metadata row's JSON sidecar.
|
||||
type sourceReference struct {
|
||||
host string
|
||||
pathHash PathHash
|
||||
}
|
||||
|
||||
// evictSourceBlob removes one source content blob. All source_metadata
|
||||
// rows referencing the blob are deleted together with its
|
||||
// source_content row in a single transaction BEFORE the file is
|
||||
// unlinked: a blob referenced by multiple source paths is only ever
|
||||
// removed together with all of its references, and database rows never
|
||||
// point at deleted files. The JSON metadata sidecars for the removed
|
||||
// rows are deleted afterwards.
|
||||
//
|
||||
// The whole operation holds the content hash's lock (the same one
|
||||
// StoreSource holds for its full store), so a concurrent store of
|
||||
// identical content bytes can never observe the file gone but a row
|
||||
// still present, or insert a fresh row between this transaction's
|
||||
// commit and the file unlink below: it either runs entirely before
|
||||
// this eviction starts, or is blocked until this eviction (row
|
||||
// deletion and unlink together) has fully completed.
|
||||
func (c *Cache) evictSourceBlob(ctx context.Context, contentHash ContentHash) error {
|
||||
unlock := c.contentLocks.Lock(string(contentHash))
|
||||
defer unlock()
|
||||
|
||||
references, err := c.sourceReferences(ctx, contentHash)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
tx, err := c.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to begin eviction transaction: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
_, err = tx.ExecContext(ctx,
|
||||
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash))
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to delete source metadata rows: %w", err)
|
||||
}
|
||||
|
||||
_, err = tx.ExecContext(ctx,
|
||||
`DELETE FROM source_content WHERE content_hash = ?`, string(contentHash))
|
||||
// evictSourceBlob removes one source: its row first, then its file, as
|
||||
// evictVariant does for a variant. It holds no lock: a store of the same
|
||||
// source in between can leave the source's row with no file, which
|
||||
// LookupSource treats as a miss.
|
||||
func (c *Cache) evictSourceBlob(ctx context.Context, key imagerequest.SourceKey) error {
|
||||
_, err := c.db.ExecContext(ctx,
|
||||
`DELETE FROM source_content WHERE source_key = ?`, string(key))
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to delete source content row: %w", err)
|
||||
}
|
||||
|
||||
err = tx.Commit()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to commit eviction transaction: %w", err)
|
||||
}
|
||||
|
||||
if c.evictSourceBlobTestHook != nil {
|
||||
c.evictSourceBlobTestHook(contentHash)
|
||||
}
|
||||
|
||||
// Only after the rows are gone may the files be removed.
|
||||
for _, reference := range references {
|
||||
err := c.srcMetadata.Delete(reference.host, reference.pathHash)
|
||||
if err != nil {
|
||||
c.log.Warn("failed to delete metadata sidecar",
|
||||
"host", reference.host, "path_hash", reference.pathHash, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
err = c.srcContent.Delete(contentHash)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// sourceReferences lists the metadata sidecar locations of every
|
||||
// source_metadata row referencing the given blob.
|
||||
func (c *Cache) sourceReferences(
|
||||
ctx context.Context, contentHash ContentHash,
|
||||
) ([]sourceReference, error) {
|
||||
rows, err := c.db.QueryContext(ctx, `
|
||||
SELECT source_host, path_hash FROM source_metadata WHERE content_hash = ?
|
||||
`, string(contentHash))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to query source references: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var references []sourceReference
|
||||
|
||||
for rows.Next() {
|
||||
var reference sourceReference
|
||||
|
||||
var pathHash string
|
||||
|
||||
err := rows.Scan(&reference.host, &pathHash)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to scan source reference: %w", err)
|
||||
}
|
||||
|
||||
reference.pathHash = PathHash(pathHash)
|
||||
references = append(references, reference)
|
||||
}
|
||||
|
||||
err = rows.Err()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("source reference iteration failed: %w", err)
|
||||
}
|
||||
|
||||
return references, nil
|
||||
return c.srcContent.Delete(key)
|
||||
}
|
||||
|
||||
// notifyWritePressure wakes the background evictor after a store, so
|
||||
@@ -582,12 +487,12 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
|
||||
// as long after the last pass ended as that pass took, off the request
|
||||
// hot path: it adopts variant files that predate the accounting table
|
||||
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
|
||||
// insert under transient DB contention), drops accounting rows whose
|
||||
// files are missing, removes source blob files the database does not
|
||||
// know (and rows whose files are gone), removes source and variant files
|
||||
// that are empty or not the size their rows record, sweeps stale temp
|
||||
// files left behind by crashed writes, and last checks the total cache
|
||||
// usage against the tables. It reads the tables a page at a time.
|
||||
// insert under transient DB contention), drops variant accounting rows
|
||||
// whose files are missing, removes source blob files the database does
|
||||
// not know, removes source and variant files that are empty or not the
|
||||
// size their rows record, sweeps stale temp files left behind by crashed
|
||||
// writes, and last checks the total cache usage against the tables. It
|
||||
// reads the tables a page at a time.
|
||||
// Running it periodically, not just once, bounds how long such drift can
|
||||
// accumulate unaccounted for on a long-running process to about one
|
||||
// eviction interval plus ten times the last pass's time.
|
||||
@@ -613,11 +518,6 @@ func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
|
||||
err = c.reconcileSourceRows(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return c.reconcileUsageTotal(ctx)
|
||||
}
|
||||
|
||||
@@ -648,7 +548,7 @@ func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
return c.reconcileVariantFile(ctx, path, entry, VariantKey(name))
|
||||
return c.reconcileVariantFile(ctx, path, entry, imagerequest.VariantKey(name))
|
||||
},
|
||||
)
|
||||
}
|
||||
@@ -659,7 +559,7 @@ func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
|
||||
// crash or a restored snapshot can leave it, is removed with its row and
|
||||
// a warning naming it, so the variant is converted again.
|
||||
func (c *Cache) reconcileVariantFile(
|
||||
ctx context.Context, path string, entry fs.DirEntry, cacheKey VariantKey,
|
||||
ctx context.Context, path string, entry fs.DirEntry, cacheKey imagerequest.VariantKey,
|
||||
) error {
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
@@ -691,7 +591,7 @@ func (c *Cache) reconcileVariantFile(
|
||||
// adoptVariantFile inserts an accounting row for a variant file that
|
||||
// has none, using the file's size and modification time.
|
||||
func (c *Cache) adoptVariantFile(
|
||||
ctx context.Context, path string, info fs.FileInfo, cacheKey VariantKey,
|
||||
ctx context.Context, path string, info fs.FileInfo, cacheKey imagerequest.VariantKey,
|
||||
) error {
|
||||
modTime := info.ModTime().UTC().Format(sqliteTimestampLayout)
|
||||
contentType := c.variantContentTypeFromSidecar(path)
|
||||
@@ -731,7 +631,7 @@ func (c *Cache) variantContentTypeFromSidecar(variantPath string) string {
|
||||
// reconcileVariantRows drops accounting rows whose variant files are
|
||||
// missing, so the database never references deleted content.
|
||||
func (c *Cache) reconcileVariantRows(ctx context.Context) error {
|
||||
var after VariantKey
|
||||
var after imagerequest.VariantKey
|
||||
|
||||
for {
|
||||
keys, err := c.variantKeysAfter(ctx, after)
|
||||
@@ -772,9 +672,9 @@ func (c *Cache) reconcileVariantRows(ctx context.Context) error {
|
||||
// variantKeysAfter returns, in order, up to c.reconciliationPageSize
|
||||
// tracked variant cache keys that sort after the given one.
|
||||
func (c *Cache) variantKeysAfter(
|
||||
ctx context.Context, after VariantKey,
|
||||
) ([]VariantKey, error) {
|
||||
return queryStringColumn[VariantKey](ctx, c.db, `
|
||||
ctx context.Context, after imagerequest.VariantKey,
|
||||
) ([]imagerequest.VariantKey, error) {
|
||||
return queryStringColumn[imagerequest.VariantKey](ctx, c.db, `
|
||||
SELECT cache_key FROM variant_content
|
||||
WHERE cache_key > ? ORDER BY cache_key LIMIT ?
|
||||
`, "variant keys", "variant key", string(after), c.reconciliationPageSize)
|
||||
@@ -838,19 +738,19 @@ func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
return c.reconcileSourceFile(ctx, path, entry, ContentHash(name))
|
||||
return c.reconcileSourceFile(ctx, path, entry, imagerequest.SourceKey(name))
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
// reconcileSourceFile checks one source blob file against its
|
||||
// source_content row. A file without a row is removed: it is
|
||||
// unreachable, as source lookups always go through source_metadata. A
|
||||
// file that is empty, or not the size its row records, as a crash or a
|
||||
// restored snapshot can leave it, is removed with its rows and a warning
|
||||
// naming it, so the source is fetched again.
|
||||
// source_content row. A file without a row is removed, as no lookup
|
||||
// serves it and no eviction removes it. A file that is empty, or not the
|
||||
// size its row records, as a crash or a restored snapshot can leave it,
|
||||
// is removed with its row and a warning naming it, so the source is
|
||||
// fetched again.
|
||||
func (c *Cache) reconcileSourceFile(
|
||||
ctx context.Context, path string, entry fs.DirEntry, contentHash ContentHash,
|
||||
ctx context.Context, path string, entry fs.DirEntry, key imagerequest.SourceKey,
|
||||
) error {
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
@@ -860,13 +760,13 @@ func (c *Cache) reconcileSourceFile(
|
||||
var recordedSize int64
|
||||
|
||||
err = c.db.QueryRowContext(ctx,
|
||||
`SELECT size_bytes FROM source_content WHERE content_hash = ?`,
|
||||
string(contentHash),
|
||||
`SELECT size_bytes FROM source_content WHERE source_key = ?`,
|
||||
string(key),
|
||||
).Scan(&recordedSize)
|
||||
|
||||
switch {
|
||||
case errors.Is(err, sql.ErrNoRows):
|
||||
return c.removeUntrackedSourceFile(ctx, path, contentHash)
|
||||
return c.removeUntrackedSourceFile(path, key)
|
||||
case err != nil:
|
||||
return fmt.Errorf("failed to check source content row: %w", err)
|
||||
case info.Size() > 0 && info.Size() == recordedSize:
|
||||
@@ -877,96 +777,34 @@ func (c *Cache) reconcileSourceFile(
|
||||
c.log.Warn("removing damaged cached source file", "path", path,
|
||||
"file_bytes", info.Size(), "recorded_bytes", recordedSize)
|
||||
|
||||
return c.evictSourceBlob(ctx, contentHash)
|
||||
return c.evictSourceBlob(ctx, key)
|
||||
}
|
||||
|
||||
// removeUntrackedSourceFile deletes a source blob file that has no
|
||||
// source_content row. Any source_metadata rows referencing the hash
|
||||
// are removed first so no row ever points at a deleted file.
|
||||
// source_content row.
|
||||
func (c *Cache) removeUntrackedSourceFile(
|
||||
ctx context.Context, path string, contentHash ContentHash,
|
||||
path string, key imagerequest.SourceKey,
|
||||
) error {
|
||||
_, err := c.db.ExecContext(ctx,
|
||||
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash))
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to delete metadata rows for untracked blob: %w", err)
|
||||
}
|
||||
|
||||
err = os.Remove(path)
|
||||
err := os.Remove(path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("failed to remove untracked source file: %w", err)
|
||||
}
|
||||
|
||||
c.log.Info("removed untracked source content file", "content_hash", contentHash)
|
||||
c.log.Info("removed untracked source content file", "source_key", key)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// reconcileSourceRows removes source_content rows (and their metadata
|
||||
// references and sidecars) whose blob files are missing on disk.
|
||||
func (c *Cache) reconcileSourceRows(ctx context.Context) error {
|
||||
var after ContentHash
|
||||
|
||||
for {
|
||||
hashes, err := c.sourceContentHashesAfter(ctx, after)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if c.reconciliationReadTestHook != nil {
|
||||
c.reconciliationReadTestHook(len(hashes))
|
||||
}
|
||||
|
||||
if len(hashes) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
for _, hash := range hashes {
|
||||
if ctx.Err() != nil {
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
if c.srcContent.Exists(hash) {
|
||||
continue
|
||||
}
|
||||
|
||||
// The blob file is already gone; evictSourceBlob removes the
|
||||
// rows and sidecars and tolerates the missing file.
|
||||
err := c.evictSourceBlob(ctx, hash)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
c.log.Info("dropped rows for missing source content file", "content_hash", hash)
|
||||
}
|
||||
|
||||
after = hashes[len(hashes)-1]
|
||||
}
|
||||
}
|
||||
|
||||
// sourceContentHashesAfter returns, in order, up to
|
||||
// c.reconciliationPageSize tracked source content hashes that sort after
|
||||
// the given one.
|
||||
func (c *Cache) sourceContentHashesAfter(
|
||||
ctx context.Context, after ContentHash,
|
||||
) ([]ContentHash, error) {
|
||||
return queryStringColumn[ContentHash](ctx, c.db, `
|
||||
SELECT content_hash FROM source_content
|
||||
WHERE content_hash > ? ORDER BY content_hash LIMIT ?
|
||||
`, "source content hashes", "content hash", string(after),
|
||||
c.reconciliationPageSize)
|
||||
}
|
||||
|
||||
// Each of these queries sums size_bytes over the next page of rows of
|
||||
// one content table, the rows that sort after a key, and returns the
|
||||
// page's last key, the sum and the number of rows in the page. Past the
|
||||
// last row the page has no rows and the key is NULL.
|
||||
const (
|
||||
sourceSizePageQuery = `
|
||||
SELECT MAX(content_hash), COALESCE(SUM(size_bytes), 0), COUNT(*)
|
||||
SELECT MAX(source_key), COALESCE(SUM(size_bytes), 0), COUNT(*)
|
||||
FROM (
|
||||
SELECT content_hash, size_bytes FROM source_content
|
||||
WHERE content_hash > ? ORDER BY content_hash LIMIT ?
|
||||
SELECT source_key, size_bytes FROM source_content
|
||||
WHERE source_key > ? ORDER BY source_key LIMIT ?
|
||||
)`
|
||||
variantSizePageQuery = `
|
||||
SELECT MAX(cache_key), COALESCE(SUM(size_bytes), 0), COUNT(*)
|
||||
|
||||
@@ -10,12 +10,14 @@ import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
"modernc.org/sqlite"
|
||||
"sneak.berlin/go/pixa/internal/database"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// sqliteTimestampFormat matches the format SQLite's CURRENT_TIMESTAMP
|
||||
@@ -27,10 +29,10 @@ const sqliteTimestampFormat = "2006-01-02 15:04:05"
|
||||
// keys reused across the eviction tests, in the order the tests store
|
||||
// them.
|
||||
const (
|
||||
testVariantKeyOne VariantKey = "aabbccdd0001"
|
||||
testVariantKeyTwo VariantKey = "aabbccdd0002"
|
||||
testVariantKeyThree VariantKey = "aabbccdd0003"
|
||||
testVariantKeyFour VariantKey = "aabbccdd0004"
|
||||
testVariantKeyOne imagerequest.VariantKey = "aabbccdd0001"
|
||||
testVariantKeyTwo imagerequest.VariantKey = "aabbccdd0002"
|
||||
testVariantKeyThree imagerequest.VariantKey = "aabbccdd0003"
|
||||
testVariantKeyFour imagerequest.VariantKey = "aabbccdd0004"
|
||||
)
|
||||
|
||||
// evictionTestDB creates an in-memory SQLite database with the real
|
||||
@@ -82,18 +84,22 @@ func newEvictionTestCache(t *testing.T, maxBytes int64) (*Cache, string) {
|
||||
}
|
||||
|
||||
// storeEvictionTestSource stores content as a fetched source for
|
||||
// host/path and returns the resulting content hash.
|
||||
// host/path and returns the resulting source key.
|
||||
func storeEvictionTestSource(
|
||||
t *testing.T, cache *Cache, host, path string, content []byte,
|
||||
) ContentHash {
|
||||
) imagerequest.SourceKey {
|
||||
t.Helper()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: host,
|
||||
SourcePath: path,
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: host,
|
||||
Path: path,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
result := &httpfetcher.FetchResult{
|
||||
@@ -105,20 +111,20 @@ func storeEvictionTestSource(
|
||||
},
|
||||
}
|
||||
|
||||
hash, err := cache.StoreSource(
|
||||
key, err := cache.StoreSource(
|
||||
context.Background(), req, bytes.NewReader(content), result,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("StoreSource(%s%s) failed: %v", host, path, err)
|
||||
}
|
||||
|
||||
return hash
|
||||
return key
|
||||
}
|
||||
|
||||
// storeEvictionTestVariant stores content as a processed variant under
|
||||
// the given cache key.
|
||||
func storeEvictionTestVariant(
|
||||
t *testing.T, cache *Cache, key VariantKey, content []byte,
|
||||
t *testing.T, cache *Cache, key imagerequest.VariantKey, content []byte,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
@@ -131,7 +137,7 @@ func storeEvictionTestVariant(
|
||||
// setVariantLastAccessed backdates the last access time of a tracked
|
||||
// variant, to make LRU ordering deterministic in tests.
|
||||
func setVariantLastAccessed(
|
||||
t *testing.T, cache *Cache, key VariantKey, when time.Time,
|
||||
t *testing.T, cache *Cache, key imagerequest.VariantKey, when time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
@@ -157,13 +163,13 @@ func setVariantLastAccessed(
|
||||
// setSourceLastAccessed backdates the last access time of a tracked
|
||||
// source content blob.
|
||||
func setSourceLastAccessed(
|
||||
t *testing.T, cache *Cache, hash ContentHash, when time.Time,
|
||||
t *testing.T, cache *Cache, key imagerequest.SourceKey, when time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
res, err := cache.db.ExecContext(t.Context(),
|
||||
`UPDATE source_content SET last_accessed_at = ? WHERE content_hash = ?`,
|
||||
when.UTC().Format(sqliteTimestampFormat), string(hash),
|
||||
`UPDATE source_content SET last_accessed_at = ? WHERE source_key = ?`,
|
||||
when.UTC().Format(sqliteTimestampFormat), string(key),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set source last_accessed_at: %v", err)
|
||||
@@ -175,7 +181,7 @@ func setSourceLastAccessed(
|
||||
}
|
||||
|
||||
if affected != 1 {
|
||||
t.Fatalf("source %s has no accounting row (affected=%d)", hash, affected)
|
||||
t.Fatalf("source %s has no accounting row (affected=%d)", key, affected)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -200,32 +206,31 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
|
||||
t.Helper()
|
||||
|
||||
rows, err := cache.db.QueryContext(t.Context(),
|
||||
`SELECT content_hash FROM source_metadata
|
||||
WHERE content_hash IS NOT NULL AND content_hash != ''`,
|
||||
`SELECT source_key FROM source_content`,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to query source_metadata: %v", err)
|
||||
t.Fatalf("failed to query source_content: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
for rows.Next() {
|
||||
var hash string
|
||||
var key string
|
||||
|
||||
err := rows.Scan(&hash)
|
||||
err := rows.Scan(&key)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to scan content_hash: %v", err)
|
||||
t.Fatalf("failed to scan source_key: %v", err)
|
||||
}
|
||||
|
||||
if !cache.srcContent.Exists(ContentHash(hash)) {
|
||||
t.Errorf("source_metadata references content %s but the file is missing",
|
||||
hash)
|
||||
if !cache.srcContent.Exists(imagerequest.SourceKey(key)) {
|
||||
t.Errorf("source_content references content %s but the file is missing",
|
||||
key)
|
||||
}
|
||||
}
|
||||
|
||||
err = rows.Err()
|
||||
if err != nil {
|
||||
t.Fatalf("source_metadata iteration failed: %v", err)
|
||||
t.Fatalf("source_content iteration failed: %v", err)
|
||||
}
|
||||
|
||||
variantRows, err := cache.db.QueryContext(t.Context(),
|
||||
@@ -244,7 +249,7 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
|
||||
t.Fatalf("failed to scan cache_key: %v", err)
|
||||
}
|
||||
|
||||
if !cache.variants.Exists(VariantKey(key)) {
|
||||
if !cache.variants.Exists(imagerequest.VariantKey(key)) {
|
||||
t.Errorf("variant_content references key %s but the file is missing", key)
|
||||
}
|
||||
}
|
||||
@@ -308,33 +313,6 @@ func TestUsageBytesAccountsSourceAndVariantBytes(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestUsageBytesCountsMultiReferencedBlobOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
content := bytes.Repeat([]byte{0xCC}, 1200)
|
||||
|
||||
hashOne := storeEvictionTestSource(t, cache,
|
||||
"src.example.com", "/one.jpg", content)
|
||||
hashTwo := storeEvictionTestSource(t, cache,
|
||||
"src.example.com", "/two.jpg", content)
|
||||
|
||||
if hashOne != hashTwo {
|
||||
t.Fatalf("identical content produced different hashes: %s vs %s",
|
||||
hashOne, hashTwo)
|
||||
}
|
||||
|
||||
usage, err := cache.UsageBytes(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes failed: %v", err)
|
||||
}
|
||||
|
||||
if usage != 1200 {
|
||||
t.Errorf("UsageBytes = %d, want 1200 (deduplicated blob counted once)", usage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -343,7 +321,7 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
|
||||
cache, _ := newEvictionTestCache(t, limit)
|
||||
|
||||
now := time.Now()
|
||||
keys := []VariantKey{
|
||||
keys := []imagerequest.VariantKey{
|
||||
testVariantKeyOne, testVariantKeyTwo,
|
||||
testVariantKeyThree, testVariantKeyFour,
|
||||
}
|
||||
@@ -388,102 +366,15 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
|
||||
assertNoDanglingReferences(t, cache)
|
||||
}
|
||||
|
||||
func TestEvictionRemovesMultiReferencedBlobTogetherWithAllReferences(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const limit = 1000
|
||||
|
||||
cache, _ := newEvictionTestCache(t, limit)
|
||||
|
||||
now := time.Now()
|
||||
|
||||
// One 800-byte blob referenced by two source paths.
|
||||
sharedContent := bytes.Repeat([]byte{0xDD}, 800)
|
||||
sharedHash := storeEvictionTestSource(t, cache,
|
||||
"src.example.com", "/a.jpg", sharedContent)
|
||||
|
||||
h := storeEvictionTestSource(t, cache,
|
||||
"src.example.com", "/b.jpg", sharedContent)
|
||||
if h != sharedHash {
|
||||
t.Fatalf("identical content produced different hashes: %s vs %s",
|
||||
h, sharedHash)
|
||||
}
|
||||
|
||||
// A newer 600-byte blob referenced by one source path.
|
||||
recentHash := storeEvictionTestSource(t, cache, "src.example.com", "/c.jpg",
|
||||
bytes.Repeat([]byte{0xEE}, 600))
|
||||
|
||||
setSourceLastAccessed(t, cache, sharedHash, now.Add(-2*time.Hour))
|
||||
setSourceLastAccessed(t, cache, recentHash, now.Add(-time.Minute))
|
||||
|
||||
err := cache.EvictToLimit(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("EvictToLimit failed: %v", err)
|
||||
}
|
||||
|
||||
usage, err := cache.UsageBytes(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes failed: %v", err)
|
||||
}
|
||||
|
||||
if usage > limit {
|
||||
t.Errorf("usage after eviction = %d, want <= %d", usage, limit)
|
||||
}
|
||||
|
||||
// The multi-referenced blob must be gone from disk, from
|
||||
// source_content, and from BOTH source_metadata rows: references
|
||||
// are removed together with the blob, never left dangling.
|
||||
if cache.srcContent.Exists(sharedHash) {
|
||||
t.Errorf("evicted blob %s still exists on disk", sharedHash)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM source_content WHERE content_hash = ?`, string(sharedHash),
|
||||
); n != 0 {
|
||||
t.Errorf("evicted blob %s still has %d source_content rows", sharedHash, n)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(sharedHash),
|
||||
); n != 0 {
|
||||
t.Errorf("evicted blob %s still has %d source_metadata references", sharedHash, n)
|
||||
}
|
||||
|
||||
// The JSON metadata sidecars for both referencing paths must be
|
||||
// removed along with the rows.
|
||||
for _, path := range []string{"/a.jpg", "/b.jpg"} {
|
||||
pathHash := HashPath(path + "?")
|
||||
if cache.srcMetadata.Exists("src.example.com", pathHash) {
|
||||
t.Errorf("metadata sidecar for %s must be removed with its row", path)
|
||||
}
|
||||
}
|
||||
|
||||
// The more recently used blob survives fully intact.
|
||||
if !cache.srcContent.Exists(recentHash) {
|
||||
t.Errorf("recently used blob %s must survive eviction", recentHash)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(recentHash),
|
||||
); n != 1 {
|
||||
t.Errorf("recently used blob %s has %d source_metadata rows, want 1",
|
||||
recentHash, n)
|
||||
}
|
||||
|
||||
assertNoDanglingReferences(t, cache)
|
||||
}
|
||||
|
||||
func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
content := bytes.Repeat([]byte{0xDF}, 800)
|
||||
hash := storeEvictionTestSource(t, cache, "src.example.com", "/a.jpg", content)
|
||||
|
||||
h := storeEvictionTestSource(t, cache, "src.example.com", "/b.jpg", content)
|
||||
if h != hash {
|
||||
t.Fatalf("identical content produced different hashes: %s vs %s", h, hash)
|
||||
keys := []imagerequest.SourceKey{
|
||||
storeEvictionTestSource(t, cache, "src.example.com", "/a.jpg", content),
|
||||
storeEvictionTestSource(t, cache, "src.example.com", "/b.jpg", content),
|
||||
}
|
||||
|
||||
storeEvictionTestVariant(t, cache, testVariantKeyOne,
|
||||
@@ -494,14 +385,16 @@ func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
|
||||
t.Fatalf("EvictToLimit failed: %v", err)
|
||||
}
|
||||
|
||||
if !cache.srcContent.Exists(hash) {
|
||||
t.Errorf("blob %s must not be evicted while usage is under the limit", hash)
|
||||
}
|
||||
for _, key := range keys {
|
||||
if !cache.srcContent.Exists(key) {
|
||||
t.Errorf("source %s must not be evicted while usage is under the limit", key)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(hash),
|
||||
); n != 2 {
|
||||
t.Errorf("blob %s has %d source_metadata rows, want 2", hash, n)
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM source_content WHERE source_key = ?`, string(key),
|
||||
); n != 1 {
|
||||
t.Errorf("source %s has %d source_content rows, want 1", key, n)
|
||||
}
|
||||
}
|
||||
|
||||
if !cache.variants.Exists(testVariantKeyOne) {
|
||||
@@ -516,12 +409,16 @@ func TestZeroMaxBytesDisablesDiskCache(t *testing.T) {
|
||||
|
||||
cache, tmpDir := newEvictionTestCache(t, 0)
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: "src.example.com",
|
||||
SourcePath: "/a.jpg",
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: "src.example.com",
|
||||
Path: "/a.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
assertDisabledCacheWritesAreNoOps(t, cache, req)
|
||||
@@ -533,14 +430,14 @@ func TestZeroMaxBytesDisablesDiskCache(t *testing.T) {
|
||||
// assertDisabledCacheWritesAreNoOps verifies that stores against a
|
||||
// disabled cache report success without recording anything.
|
||||
func assertDisabledCacheWritesAreNoOps(
|
||||
t *testing.T, cache *Cache, req *ImageRequest,
|
||||
t *testing.T, cache *Cache, req *imagerequest.Request,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
ctx := t.Context()
|
||||
|
||||
err := cache.StoreVariant(
|
||||
ctx, CacheKey(req), bytes.NewReader([]byte("data")), "image/webp",
|
||||
ctx, imagerequest.CacheKey(req), bytes.NewReader([]byte("data")), "image/webp",
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("StoreVariant on disabled cache must be a no-op, got error: %v", err)
|
||||
@@ -553,20 +450,20 @@ func assertDisabledCacheWritesAreNoOps(
|
||||
Headers: map[string][]string{},
|
||||
}
|
||||
|
||||
hash, err := cache.StoreSource(ctx, req, bytes.NewReader([]byte("data")), result)
|
||||
key, err := cache.StoreSource(ctx, req, bytes.NewReader([]byte("data")), result)
|
||||
if err != nil {
|
||||
t.Fatalf("StoreSource on disabled cache must be a no-op, got error: %v", err)
|
||||
}
|
||||
|
||||
if hash != "" {
|
||||
t.Errorf("StoreSource on disabled cache returned hash %q, want empty", hash)
|
||||
if key != "" {
|
||||
t.Errorf("StoreSource on disabled cache returned key %q, want empty", key)
|
||||
}
|
||||
}
|
||||
|
||||
// assertDisabledCacheReadsAlwaysMiss verifies that lookups against a
|
||||
// disabled cache never report a hit.
|
||||
func assertDisabledCacheReadsAlwaysMiss(
|
||||
t *testing.T, cache *Cache, req *ImageRequest,
|
||||
t *testing.T, cache *Cache, req *imagerequest.Request,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
@@ -581,14 +478,14 @@ func assertDisabledCacheReadsAlwaysMiss(
|
||||
t.Error("Lookup on disabled cache must always miss")
|
||||
}
|
||||
|
||||
srcHash, srcType, err := cache.LookupSource(ctx, req)
|
||||
key, srcType, err := cache.LookupSource(ctx, req)
|
||||
if err != nil {
|
||||
t.Fatalf("LookupSource on disabled cache failed: %v", err)
|
||||
}
|
||||
|
||||
if srcHash != "" || srcType != "" {
|
||||
if key != "" || srcType != "" {
|
||||
t.Errorf("LookupSource on disabled cache = (%q, %q), want empty",
|
||||
srcHash, srcType)
|
||||
key, srcType)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -609,10 +506,6 @@ func assertDisabledCacheTracksNothing(t *testing.T, cache *Cache) {
|
||||
if n := countRows(t, cache, `SELECT COUNT(*) FROM source_content`); n != 0 {
|
||||
t.Errorf("disabled cache wrote %d source_content rows, want 0", n)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache, `SELECT COUNT(*) FROM source_metadata`); n != 0 {
|
||||
t.Errorf("disabled cache wrote %d source_metadata rows, want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
// assertDisabledCacheWroteNothingToDisk verifies that a disabled cache
|
||||
@@ -661,7 +554,7 @@ func TestEvictionRunsUnderWritePressure(t *testing.T) {
|
||||
cache.StartEviction(time.Hour)
|
||||
defer func() { _ = cache.StopEviction(t.Context()) }()
|
||||
|
||||
keys := []VariantKey{
|
||||
keys := []imagerequest.VariantKey{
|
||||
testVariantKeyOne, testVariantKeyTwo, testVariantKeyThree,
|
||||
}
|
||||
fills := []byte{0x11, 0x12, 0x13}
|
||||
@@ -715,7 +608,7 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
|
||||
keys := []VariantKey{
|
||||
keys := []imagerequest.VariantKey{
|
||||
testVariantKeyOne, testVariantKeyTwo, testVariantKeyThree,
|
||||
}
|
||||
fills := []byte{0x21, 0x22, 0x23}
|
||||
@@ -1055,9 +948,15 @@ func TestStopEvictionInterruptsPassInProgress(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestEvictToLimitStopsAtNextCandidateOnceCancelled cancels the context
|
||||
// while the oldest of three source blobs is being evicted, and checks that
|
||||
// EvictToLimit then returns context.Canceled without evicting the other
|
||||
// two or logging a warning for either of them.
|
||||
// once the oldest of three sources is evicted, while the next one waits
|
||||
// for the database, and checks that EvictToLimit then returns
|
||||
// context.Canceled, with that source gone and the other two kept, and
|
||||
// logs no warning. The test database has one connection. When the
|
||||
// deletion of the oldest source's row commits, a commit hook holds the
|
||||
// deletion until the test waits for the connection, so the test takes it
|
||||
// as soon as the deletion is done. The eviction of the next source then
|
||||
// waits for the connection, and the test cancels the context while it
|
||||
// waits.
|
||||
func TestEvictToLimitStopsAtNextCandidateOnceCancelled(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -1067,7 +966,7 @@ func TestEvictToLimitStopsAtNextCandidateOnceCancelled(t *testing.T) {
|
||||
|
||||
cache.log = slog.New(slog.NewJSONHandler(&logBuf, nil))
|
||||
|
||||
hashes := []ContentHash{
|
||||
keys := []imagerequest.SourceKey{
|
||||
storeEvictionTestSource(t, cache, "cancel.example.com", "/a.jpg",
|
||||
bytes.Repeat([]byte{0x61}, 1000)),
|
||||
storeEvictionTestSource(t, cache, "cancel.example.com", "/b.jpg",
|
||||
@@ -1078,69 +977,155 @@ func TestEvictToLimitStopsAtNextCandidateOnceCancelled(t *testing.T) {
|
||||
|
||||
base := time.Now().Add(-time.Hour)
|
||||
|
||||
for i, hash := range hashes {
|
||||
setSourceLastAccessed(t, cache, hash, base.Add(time.Duration(i)*time.Minute))
|
||||
for i, key := range keys {
|
||||
setSourceLastAccessed(t, cache, key, base.Add(time.Duration(i)*time.Minute))
|
||||
}
|
||||
|
||||
waitsBefore := cache.db.Stats().WaitCount
|
||||
oldestRowDeleted := make(chan struct{})
|
||||
|
||||
var oldestRowDeletedOnce sync.Once
|
||||
|
||||
setCommitHook(t, cache, func() int32 {
|
||||
oldestRowDeletedOnce.Do(func() {
|
||||
close(oldestRowDeleted)
|
||||
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
|
||||
for cache.db.Stats().WaitCount == waitsBefore && time.Now().Before(deadline) {
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
})
|
||||
|
||||
return 0
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
|
||||
cache.evictSourceBlobTestHook = func(ContentHash) { cancel() }
|
||||
evicted := make(chan error, 1)
|
||||
|
||||
err := cache.EvictToLimit(ctx)
|
||||
go func() { evicted <- cache.EvictToLimit(ctx) }()
|
||||
|
||||
select {
|
||||
case <-oldestRowDeleted:
|
||||
case err := <-evicted:
|
||||
t.Fatalf("EvictToLimit() returned %v without deleting a row", err)
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("EvictToLimit never deleted a row")
|
||||
}
|
||||
|
||||
releaseDatabase := holdDatabase(t, cache)
|
||||
defer releaseDatabase()
|
||||
|
||||
waitForConnectionWaits(t, cache.db, waitsBefore+2)
|
||||
cancel()
|
||||
|
||||
err := <-evicted
|
||||
t.Logf("EvictToLimit() error = %v", err)
|
||||
t.Logf("log output: %s", logBuf.String())
|
||||
|
||||
// The checks below read the database, so the connection goes back.
|
||||
releaseDatabase()
|
||||
|
||||
setCommitHook(t, cache, nil)
|
||||
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
t.Errorf("EvictToLimit() error = %v, want context.Canceled", err)
|
||||
}
|
||||
|
||||
if cache.srcContent.Exists(hashes[0]) {
|
||||
t.Errorf("source blob %s, evicted when the context was cancelled, "+
|
||||
"is still on disk", hashes[0])
|
||||
}
|
||||
|
||||
for _, hash := range hashes[1:] {
|
||||
if !cache.srcContent.Exists(hash) {
|
||||
t.Errorf("source blob %s was evicted after the context was cancelled", hash)
|
||||
}
|
||||
}
|
||||
assertOnlyOldestSourceEvicted(t, cache, keys)
|
||||
|
||||
if strings.Contains(logBuf.String(), `"level":"WARN"`) {
|
||||
t.Errorf("EvictToLimit logged a warning after the context was cancelled")
|
||||
}
|
||||
}
|
||||
|
||||
// setCommitHook makes the SQLite driver call hook each time a write on
|
||||
// the test database's one connection commits, or stops it when hook is
|
||||
// nil.
|
||||
func setCommitHook(t *testing.T, cache *Cache, hook sqlite.CommitHookFn) {
|
||||
t.Helper()
|
||||
|
||||
conn, err := cache.db.Conn(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("failed to take the database connection: %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = conn.Close() }()
|
||||
|
||||
err = conn.Raw(func(driverConn any) error {
|
||||
hooks, ok := driverConn.(sqlite.HookRegisterer)
|
||||
if !ok {
|
||||
t.Fatalf("the database connection is a %T, which takes no hooks",
|
||||
driverConn)
|
||||
}
|
||||
|
||||
hooks.RegisterCommitHook(hook)
|
||||
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to set the commit hook: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// assertOnlyOldestSourceEvicted checks that the first of keys has neither
|
||||
// its file nor its row, and that every other one has both.
|
||||
func assertOnlyOldestSourceEvicted(
|
||||
t *testing.T, cache *Cache, keys []imagerequest.SourceKey,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
if cache.srcContent.Exists(keys[0]) {
|
||||
t.Errorf("source %s, evicted when the context was cancelled, "+
|
||||
"is still on disk", keys[0])
|
||||
}
|
||||
|
||||
for _, key := range keys[1:] {
|
||||
if !cache.srcContent.Exists(key) {
|
||||
t.Errorf("source %s was evicted after the context was cancelled", key)
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM source_content WHERE source_key = ?`, string(key),
|
||||
); n != 1 {
|
||||
t.Errorf("source %s, kept, has %d source_content rows, want 1", key, n)
|
||||
}
|
||||
}
|
||||
|
||||
assertNoDanglingReferences(t, cache)
|
||||
}
|
||||
|
||||
// TestStopEvictionReturnsWhenItsContextEnds pauses an eviction pass where
|
||||
// cancellation cannot reach it, after a source blob's rows are deleted and
|
||||
// before its file is removed, and checks that StopEviction returns its
|
||||
// context's error when that context ends instead of waiting for the pass.
|
||||
// Once the pass goes on, the goroutine exits and no row points at a
|
||||
// missing file.
|
||||
// TestStopEvictionReturnsWhenItsContextEnds pauses the startup
|
||||
// reconciliation pass in the hook called after each of its reads, where
|
||||
// cancellation cannot reach it, and checks that StopEviction returns an
|
||||
// error wrapping its context's error when that context ends, instead of
|
||||
// waiting for the pass. Once the pass goes on, the goroutine exits, and a
|
||||
// second StopEviction returns nil.
|
||||
func TestStopEvictionReturnsWhenItsContextEnds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1)
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
paused := make(chan struct{})
|
||||
resume := make(chan struct{})
|
||||
|
||||
cache.evictSourceBlobTestHook = func(ContentHash) {
|
||||
close(paused)
|
||||
<-resume
|
||||
}
|
||||
var pauseOnce sync.Once
|
||||
|
||||
hash := storeEvictionTestSource(t, cache, "stop.example.com", "/a.jpg",
|
||||
bytes.Repeat([]byte{0x61}, 1000))
|
||||
cache.reconciliationReadTestHook = func(int) {
|
||||
pauseOnce.Do(func() {
|
||||
close(paused)
|
||||
<-resume
|
||||
})
|
||||
}
|
||||
|
||||
cache.StartEviction(time.Hour)
|
||||
|
||||
select {
|
||||
case <-paused:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("the eviction pass never reached the source blob")
|
||||
t.Fatal("the reconciliation pass never made a read")
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 50*time.Millisecond)
|
||||
@@ -1159,12 +1144,6 @@ func TestStopEvictionReturnsWhenItsContextEnds(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("second StopEviction() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
assertNoDanglingReferences(t, cache)
|
||||
|
||||
if cache.srcContent.Exists(hash) {
|
||||
t.Errorf("source blob %s is still on disk after its rows were deleted", hash)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconciliationWalksStopOnceCancelled checks that both directory
|
||||
@@ -1236,125 +1215,3 @@ func TestReconciliationPassLogsNoWarningOnceCancelled(t *testing.T) {
|
||||
t.Errorf("runReconciliationPass logged a warning with a cancelled context")
|
||||
}
|
||||
}
|
||||
|
||||
// TestEvictSourceBlobExcludesConcurrentStoreOfIdenticalContent exercises
|
||||
// the exact TOCTOU window between evictSourceBlob's row-deletion
|
||||
// transaction commit and its content file unlink: a concurrent
|
||||
// StoreSource for a different source path whose content hashes to the
|
||||
// same value (real SHA-256 dedup, not a contrived case) must not be
|
||||
// able to insert a fresh row referencing the file while eviction is
|
||||
// mid-unlink, and must not lose its own store once eviction has fully
|
||||
// released the content hash.
|
||||
func TestEvictSourceBlobExcludesConcurrentStoreOfIdenticalContent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
ctx := context.Background()
|
||||
|
||||
content := bytes.Repeat([]byte{0x55}, 400)
|
||||
|
||||
hash := storeEvictionTestSource(t, cache,
|
||||
"race.example.com", "/first.jpg", content)
|
||||
|
||||
proceed := make(chan struct{})
|
||||
storeAttempted := make(chan struct{})
|
||||
|
||||
cache.evictSourceBlobTestHook = func(gotHash ContentHash) {
|
||||
if gotHash != hash {
|
||||
t.Errorf("test hook invoked for hash %s, want %s", gotHash, hash)
|
||||
}
|
||||
|
||||
close(storeAttempted)
|
||||
<-proceed
|
||||
}
|
||||
|
||||
evictDone := make(chan error, 1)
|
||||
|
||||
go func() {
|
||||
evictDone <- cache.evictSourceBlob(ctx, hash)
|
||||
}()
|
||||
|
||||
// Wait until eviction has committed its delete transaction and is
|
||||
// paused (inside the test hook) immediately before unlinking the
|
||||
// content file: exactly the window the review flagged.
|
||||
<-storeAttempted
|
||||
|
||||
storeDone := storeIdenticalContentConcurrently(ctx, cache, content)
|
||||
|
||||
// The concurrent store must not be able to complete while eviction
|
||||
// still holds the content hash (i.e. before the file is unlinked):
|
||||
// if it could, it would insert a row referencing a file about to be
|
||||
// removed out from under it.
|
||||
select {
|
||||
case err := <-storeDone:
|
||||
t.Fatalf("StoreSource for identical content completed (err=%v) while eviction "+
|
||||
"still held the content hash open between commit and unlink; the store and "+
|
||||
"the evict of identical content are not mutually exclusive", err)
|
||||
case <-time.After(200 * time.Millisecond):
|
||||
// Expected: the store is blocked behind eviction's exclusion.
|
||||
}
|
||||
|
||||
close(proceed)
|
||||
|
||||
err := <-evictDone
|
||||
if err != nil {
|
||||
t.Fatalf("evictSourceBlob failed: %v", err)
|
||||
}
|
||||
|
||||
err = <-storeDone
|
||||
if err != nil {
|
||||
t.Fatalf("StoreSource failed: %v", err)
|
||||
}
|
||||
|
||||
assertNoDanglingReferences(t, cache)
|
||||
|
||||
dupHash, _, err := cache.LookupSource(ctx, &ImageRequest{
|
||||
SourceHost: "race.example.com",
|
||||
SourcePath: "/dup.jpg",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("LookupSource failed: %v", err)
|
||||
}
|
||||
|
||||
if dupHash == "" {
|
||||
t.Fatal("re-stored blob was lost: the store legitimately ran after eviction " +
|
||||
"released the content hash and must have recreated the file and row")
|
||||
}
|
||||
|
||||
if !cache.srcContent.Exists(dupHash) {
|
||||
t.Errorf("source_content/source_metadata references %s but its file is missing",
|
||||
dupHash)
|
||||
}
|
||||
}
|
||||
|
||||
// storeIdenticalContentConcurrently starts a StoreSource for a second
|
||||
// source path whose body hashes to the same content hash, returning the
|
||||
// channel its error is delivered on.
|
||||
func storeIdenticalContentConcurrently(
|
||||
ctx context.Context, cache *Cache, content []byte,
|
||||
) chan error {
|
||||
storeDone := make(chan error, 1)
|
||||
|
||||
go func() {
|
||||
req := &ImageRequest{
|
||||
SourceHost: "race.example.com",
|
||||
SourcePath: "/dup.jpg",
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
}
|
||||
result := &httpfetcher.FetchResult{
|
||||
StatusCode: 200,
|
||||
ContentType: testContentTypeJPEG,
|
||||
ContentLength: int64(len(content)),
|
||||
Headers: map[string][]string{
|
||||
testHeaderContentType: {testContentTypeJPEG},
|
||||
},
|
||||
}
|
||||
|
||||
_, err := cache.StoreSource(ctx, req, bytes.NewReader(content), result)
|
||||
storeDone <- err
|
||||
}()
|
||||
|
||||
return storeDone
|
||||
}
|
||||
|
||||
@@ -3,134 +3,12 @@ package imgcache
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// ErrInvalidFitMode is returned when an invalid fit mode is provided.
|
||||
var ErrInvalidFitMode = errors.New("invalid fit mode")
|
||||
|
||||
// ImageFormat represents supported output image formats.
|
||||
type ImageFormat string
|
||||
|
||||
// Supported image output formats.
|
||||
const (
|
||||
FormatOriginal ImageFormat = "orig"
|
||||
FormatJPEG ImageFormat = "jpeg"
|
||||
FormatPNG ImageFormat = "png"
|
||||
FormatWebP ImageFormat = "webp"
|
||||
FormatAVIF ImageFormat = "avif"
|
||||
FormatJXL ImageFormat = "jxl"
|
||||
FormatGIF ImageFormat = "gif"
|
||||
|
||||
// FormatAuto stands for JPEG XL, AVIF, WebP or JPEG, chosen for each
|
||||
// request from its Accept header once the URL's signature or token has
|
||||
// been checked; it is never processed or cached as itself.
|
||||
FormatAuto ImageFormat = "auto"
|
||||
)
|
||||
|
||||
// Size represents requested image dimensions
|
||||
type Size struct {
|
||||
Width int
|
||||
Height int
|
||||
}
|
||||
|
||||
// OriginalSize returns true if this represents "keep original size"
|
||||
func (s Size) OriginalSize() bool {
|
||||
return s.Width == 0 && s.Height == 0
|
||||
}
|
||||
|
||||
// FitMode represents how to fit image into requested dimensions.
|
||||
type FitMode string
|
||||
|
||||
// Supported image fit modes.
|
||||
const (
|
||||
FitCover FitMode = "cover"
|
||||
FitContain FitMode = "contain"
|
||||
FitFill FitMode = "fill"
|
||||
FitInside FitMode = "inside"
|
||||
FitOutside FitMode = "outside"
|
||||
)
|
||||
|
||||
// ValidateFitMode checks if the given fit mode is valid.
|
||||
// Returns ErrInvalidFitMode for unrecognized fit modes.
|
||||
func ValidateFitMode(fit FitMode) error {
|
||||
switch fit {
|
||||
case FitCover, FitContain, FitFill, FitInside, FitOutside, "":
|
||||
return nil
|
||||
default:
|
||||
return ErrInvalidFitMode
|
||||
}
|
||||
}
|
||||
|
||||
// ValidateImageRequest checks a request's width and height with
|
||||
// ValidateDimension and its fit mode with ValidateFitMode. Both the plain
|
||||
// /v1/image/ route and the encrypted /v1/e/ route validate through this
|
||||
// function so a request from either source enforces identical bounds.
|
||||
func ValidateImageRequest(req *ImageRequest) error {
|
||||
err := ValidateDimension("width", req.Size.Width)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = ValidateDimension("height", req.Size.Height)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return ValidateFitMode(req.FitMode)
|
||||
}
|
||||
|
||||
// ImageRequest represents a request for a processed image
|
||||
type ImageRequest struct {
|
||||
// SourceHost is the origin host (e.g., "cdn.example.com")
|
||||
SourceHost string
|
||||
// SourcePath is the path on the origin with its percent-escapes decoded,
|
||||
// as url.URL's Path holds it (e.g., "/photos/my cat.jpg")
|
||||
SourcePath string
|
||||
// SourceQuery is the optional query string for the origin URL as the URL
|
||||
// writes it, percent-escapes included, as url.URL's RawQuery holds it
|
||||
SourceQuery string
|
||||
// Size is the requested output dimensions
|
||||
Size Size
|
||||
// Format is the requested output format
|
||||
Format ImageFormat
|
||||
// Quality is the output quality (1-100) for lossy formats
|
||||
Quality int
|
||||
// FitMode is how to fit the image into requested dimensions
|
||||
FitMode FitMode
|
||||
// Signature is the HMAC signature for non-allowlisted hosts
|
||||
Signature string
|
||||
// Expires is when the URL expires: the exp of a signed URL, or the expiry
|
||||
// of an encrypted URL; the zero time if it has none
|
||||
Expires time.Time
|
||||
// AllowHTTP indicates whether HTTP (non-TLS) is allowed for this request
|
||||
AllowHTTP bool
|
||||
}
|
||||
|
||||
// SourceURL returns the full upstream URL to fetch: the source path
|
||||
// percent-encoded again, as net/url writes a path, so each source path is one
|
||||
// upstream path, and the source query as it is, with the characters
|
||||
// escapeQuery encodes percent-encoded. Uses http:// scheme when AllowHTTP is
|
||||
// true, otherwise https://.
|
||||
func (r *ImageRequest) SourceURL() string {
|
||||
scheme := "https"
|
||||
if r.AllowHTTP {
|
||||
scheme = "http"
|
||||
}
|
||||
|
||||
sourceURL := url.URL{
|
||||
Scheme: scheme,
|
||||
Host: r.SourceHost,
|
||||
Path: r.SourcePath,
|
||||
RawQuery: escapeQuery(r.SourceQuery),
|
||||
}
|
||||
|
||||
return sourceURL.String()
|
||||
}
|
||||
|
||||
// ImageResponse represents a processed image ready to serve
|
||||
type ImageResponse struct {
|
||||
// Content is the image data reader
|
||||
@@ -162,10 +40,10 @@ const (
|
||||
// ImageCache is the main interface for the image caching proxy
|
||||
type ImageCache interface {
|
||||
// Get retrieves a processed image, fetching and processing if necessary
|
||||
Get(ctx context.Context, req *ImageRequest) (*ImageResponse, error)
|
||||
Get(ctx context.Context, req *imagerequest.Request) (*ImageResponse, error)
|
||||
|
||||
// Warm pre-fetches and caches an image without returning it
|
||||
Warm(ctx context.Context, req *ImageRequest) error
|
||||
Warm(ctx context.Context, req *imagerequest.Request) error
|
||||
|
||||
// Stats returns cache statistics
|
||||
Stats(ctx context.Context) (*CacheStats, error)
|
||||
|
||||
@@ -9,18 +9,23 @@ import (
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// widthOnlyRequest asks for the test photo at width, its height scaled to
|
||||
// keep the photo's aspect ratio.
|
||||
func widthOnlyRequest(fixtures *TestFixtures, width int) *ImageRequest {
|
||||
return &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: Size{Width: width},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
func widthOnlyRequest(fixtures *TestFixtures, width int) *imagerequest.Request {
|
||||
return &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: width},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,7 +41,7 @@ func holdProcessingSlot(
|
||||
input, feed := io.Pipe()
|
||||
|
||||
go func() {
|
||||
_, _ = proc.Process(t.Context(), input, &imageprocessor.Request{})
|
||||
_, _ = proc.Process(t.Context(), input, &imagerequest.Transform{})
|
||||
}()
|
||||
|
||||
_, err := feed.Write([]byte{0})
|
||||
@@ -72,11 +77,11 @@ func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
contentHash, _, err := svc.cache.LookupSource(t.Context(),
|
||||
sourceKey, _, err := svc.cache.LookupSource(t.Context(),
|
||||
widthOnlyRequest(fixtures, 50))
|
||||
if err != nil || contentHash == "" {
|
||||
if err != nil || sourceKey == "" {
|
||||
t.Fatalf("LookupSource() = %q, %v; want the cached source",
|
||||
contentHash, err)
|
||||
sourceKey, err)
|
||||
}
|
||||
|
||||
release := holdProcessingSlot(t, svc.processor)
|
||||
@@ -98,7 +103,7 @@ func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
|
||||
// before waiting, it would have read it by now.
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
|
||||
err = os.WriteFile(svc.cache.srcContent.hashToPath(contentHash),
|
||||
err = os.WriteFile(svc.cache.srcContent.keyToPath(sourceKey),
|
||||
generateTestJPEG(t, 100, 50, color.RGBA{0, 0, 255, 255}), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to rewrite the cached source: %v", err)
|
||||
|
||||
@@ -10,23 +10,31 @@ import (
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// webpRequest returns a request for a 100x100 WebP variant of path.
|
||||
func webpRequest(path string) *ImageRequest {
|
||||
return &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: path,
|
||||
Size: Size{Width: 100, Height: 100},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
func webpRequest(path string) *imagerequest.Request {
|
||||
return &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: path,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 100, Height: 100},
|
||||
Format: imagerequest.FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// assertVariantServed checks that GetVariant serves key with the given
|
||||
// content and the image/webp content type storeEvictionTestVariant stores.
|
||||
func assertVariantServed(t *testing.T, cache *Cache, key VariantKey, content []byte) {
|
||||
func assertVariantServed(
|
||||
t *testing.T, cache *Cache, key imagerequest.VariantKey, content []byte,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
reader, size, contentType, err := cache.GetVariant(key)
|
||||
@@ -57,7 +65,9 @@ func assertVariantServed(t *testing.T, cache *Cache, key VariantKey, content []b
|
||||
|
||||
// assertVariantNotFound checks that GetVariant refuses key with
|
||||
// ErrNotFound.
|
||||
func assertVariantNotFound(t *testing.T, cache *Cache, key VariantKey) {
|
||||
func assertVariantNotFound(
|
||||
t *testing.T, cache *Cache, key imagerequest.VariantKey,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
reader, _, _, err := cache.GetVariant(key)
|
||||
@@ -71,16 +81,18 @@ func assertVariantNotFound(t *testing.T, cache *Cache, key VariantKey) {
|
||||
}
|
||||
|
||||
// assertLookupMisses checks that Lookup reports request as a miss.
|
||||
func assertLookupMisses(t *testing.T, cache *Cache, request *ImageRequest) {
|
||||
func assertLookupMisses(
|
||||
t *testing.T, cache *Cache, request *imagerequest.Request,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
lookup, err := cache.Lookup(t.Context(), request)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup(%s) error = %v", request.SourcePath, err)
|
||||
t.Fatalf("Lookup(%s) error = %v", request.Source.Path, err)
|
||||
}
|
||||
|
||||
if lookup.Hit {
|
||||
t.Errorf("Lookup(%s) is a hit, want a miss", request.SourcePath)
|
||||
t.Errorf("Lookup(%s) is a hit, want a miss", request.Source.Path)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -251,8 +263,8 @@ func TestEvictedVariantIsNotServed(t *testing.T) {
|
||||
|
||||
oldRequest := webpRequest("/old.jpg")
|
||||
newRequest := webpRequest("/new.jpg")
|
||||
oldKey := CacheKey(oldRequest)
|
||||
newKey := CacheKey(newRequest)
|
||||
oldKey := imagerequest.CacheKey(oldRequest)
|
||||
newKey := imagerequest.CacheKey(newRequest)
|
||||
oldContent := bytes.Repeat([]byte{0x01}, 1000)
|
||||
newContent := bytes.Repeat([]byte{0x02}, 1000)
|
||||
|
||||
@@ -280,7 +292,7 @@ func TestVariantDeletedFromDiskIsNotServed(t *testing.T) {
|
||||
cache, _ := newEvictionTestCache(t, 1<<20)
|
||||
|
||||
request := webpRequest("/deleted.jpg")
|
||||
key := CacheKey(request)
|
||||
key := imagerequest.CacheKey(request)
|
||||
content := []byte("webp variant bytes")
|
||||
|
||||
storeEvictionTestVariant(t, cache, key, content)
|
||||
@@ -307,7 +319,7 @@ func TestConcurrentVariantStoreReadAndEvict(t *testing.T) {
|
||||
|
||||
for goroutine := range 8 {
|
||||
wg.Go(func() {
|
||||
key := VariantKey(fmt.Sprintf("aabbccdd01%02d", goroutine))
|
||||
key := imagerequest.VariantKey(fmt.Sprintf("aabbccdd01%02d", goroutine))
|
||||
content := []byte(key)
|
||||
|
||||
for range 20 {
|
||||
|
||||
@@ -6,6 +6,8 @@ import (
|
||||
"strconv"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
func TestNegativeCache_StoreAndCheck(t *testing.T) {
|
||||
@@ -23,9 +25,11 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
req := &ImageRequest{
|
||||
SourceHost: testHostExample,
|
||||
SourcePath: "/missing.jpg",
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostExample,
|
||||
Path: "/missing.jpg",
|
||||
},
|
||||
}
|
||||
|
||||
// Initially should not be in negative cache
|
||||
@@ -70,9 +74,11 @@ func TestNegativeCache_Expired(t *testing.T) {
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
req := &ImageRequest{
|
||||
SourceHost: testHostExample,
|
||||
SourcePath: "/expired.jpg",
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostExample,
|
||||
Path: "/expired.jpg",
|
||||
},
|
||||
}
|
||||
|
||||
// Store a negative entry with very short TTL
|
||||
@@ -106,20 +112,21 @@ func TestService_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
|
||||
|
||||
svc, fixtures := SetupTestService(t)
|
||||
|
||||
resp, err := svc.Get(t.Context(), photoVariant(fixtures, 85, FitCover))
|
||||
resp, err := svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
if err != nil {
|
||||
t.Fatalf("Get() first request error = %v", err)
|
||||
}
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
err = svc.cache.StoreNegative(t.Context(), photoVariant(fixtures, 85, FitCover),
|
||||
err = svc.cache.StoreNegative(t.Context(),
|
||||
photoVariant(fixtures, 85, imagerequest.FitCover),
|
||||
httpStatusBadGateway, "upstream answered 503")
|
||||
if err != nil {
|
||||
t.Fatalf("StoreNegative() error = %v", err)
|
||||
}
|
||||
|
||||
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 85, FitCover))
|
||||
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
if err != nil {
|
||||
t.Fatalf("Get() of the cached variant error = %v", err)
|
||||
}
|
||||
@@ -131,7 +138,7 @@ func TestService_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
|
||||
resp.CacheStatus, CacheHit)
|
||||
}
|
||||
|
||||
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, FitCover))
|
||||
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
|
||||
t.Logf("Get() of a variant made from the cached source: error = %v", err)
|
||||
|
||||
if err != nil {
|
||||
@@ -140,18 +147,18 @@ func TestService_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
|
||||
|
||||
_ = resp.Content.Close()
|
||||
|
||||
contentHash, _, err := svc.cache.LookupSource(t.Context(),
|
||||
photoVariant(fixtures, 85, FitCover))
|
||||
if err != nil || contentHash == "" {
|
||||
t.Fatalf("LookupSource() = %q, %v, want the cached source", contentHash, err)
|
||||
sourceKey, _, err := svc.cache.LookupSource(t.Context(),
|
||||
photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
if err != nil || sourceKey == "" {
|
||||
t.Fatalf("LookupSource() = %q, %v, want the cached source", sourceKey, err)
|
||||
}
|
||||
|
||||
err = svc.cache.evictSourceBlob(t.Context(), contentHash)
|
||||
err = svc.cache.evictSourceBlob(t.Context(), sourceKey)
|
||||
if err != nil {
|
||||
t.Fatalf("evictSourceBlob() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = svc.Get(t.Context(), photoVariant(fixtures, 60, FitCover))
|
||||
_, err = svc.Get(t.Context(), photoVariant(fixtures, 60, imagerequest.FitCover))
|
||||
t.Logf("Get() of a variant whose source was evicted: error = %v", err)
|
||||
|
||||
if !errors.Is(err, ErrNegativeCached) {
|
||||
@@ -190,8 +197,8 @@ func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
|
||||
t.Fatalf("failed to create cache: %v", err)
|
||||
}
|
||||
|
||||
expired := &ImageRequest{
|
||||
SourceHost: testHostExample, SourcePath: "/gone.jpg",
|
||||
expired := &imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: testHostExample, Path: "/gone.jpg"},
|
||||
}
|
||||
|
||||
err = cache.StoreNegative(t.Context(), expired, 404, "not found")
|
||||
@@ -215,7 +222,7 @@ func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
|
||||
cache.StartEviction(100 * time.Millisecond)
|
||||
defer func() { _ = cache.StopEviction(t.Context()) }()
|
||||
|
||||
for negativeCacheRowCount(t, cache, expired.SourcePath) > 0 {
|
||||
for negativeCacheRowCount(t, cache, expired.Source.Path) > 0 {
|
||||
if time.Since(started) > 10*time.Second {
|
||||
t.Fatal("the expired row is still in negative_cache")
|
||||
}
|
||||
|
||||
@@ -2,31 +2,33 @@ package imgcache
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// getImage calls Get for req and returns the image it answered with and
|
||||
// its cache status.
|
||||
func getImage(t *testing.T, svc *Service, req *ImageRequest) ([]byte, CacheStatus) {
|
||||
func getImage(
|
||||
t *testing.T, svc *Service, req *imagerequest.Request,
|
||||
) ([]byte, CacheStatus) {
|
||||
t.Helper()
|
||||
|
||||
resp, err := svc.Get(t.Context(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Get() of quality %d error = %v", req.Quality, err)
|
||||
t.Fatalf("Get() of quality %d error = %v", req.Transform.Quality, err)
|
||||
}
|
||||
|
||||
defer func() { _ = resp.Content.Close() }()
|
||||
|
||||
body, err := io.ReadAll(resp.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("reading the image of quality %d: %v", req.Quality, err)
|
||||
t.Fatalf("reading the image of quality %d: %v", req.Transform.Quality, err)
|
||||
}
|
||||
|
||||
return body, resp.CacheStatus
|
||||
@@ -58,15 +60,17 @@ func truncateFile(t *testing.T, path string, size int64) {
|
||||
}
|
||||
|
||||
// cachedSourcePath returns the path of the cached source file req uses.
|
||||
func cachedSourcePath(t *testing.T, svc *Service, req *ImageRequest) string {
|
||||
func cachedSourcePath(
|
||||
t *testing.T, svc *Service, req *imagerequest.Request,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
contentHash, _, err := svc.cache.LookupSource(t.Context(), req)
|
||||
if err != nil || contentHash == "" {
|
||||
t.Fatalf("LookupSource() = %q, %v; want the cached source", contentHash, err)
|
||||
sourceKey, _, err := svc.cache.LookupSource(t.Context(), req)
|
||||
if err != nil || sourceKey == "" {
|
||||
t.Fatalf("LookupSource() = %q, %v; want the cached source", sourceKey, err)
|
||||
}
|
||||
|
||||
return svc.cache.srcContent.hashToPath(contentHash)
|
||||
return svc.cache.srcContent.keyToPath(sourceKey)
|
||||
}
|
||||
|
||||
// reconcileAsAtStartup runs the cache's reconciliation pass, as pixa does
|
||||
@@ -113,16 +117,16 @@ func TestService_Get_FetchesAgainForEmptyCachedSource(t *testing.T) {
|
||||
|
||||
svc, fixtures := SetupTestService(t)
|
||||
|
||||
getImage(t, svc, photoVariant(fixtures, 85, FitCover))
|
||||
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, FitCover))
|
||||
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
wholeSize := fileSize(t, path)
|
||||
|
||||
truncateFile(t, path, 0)
|
||||
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
|
||||
|
||||
getImage(t, svc, photoVariant(fixtures, 40, FitCover))
|
||||
getImage(t, svc, photoVariant(fixtures, 30, FitCover))
|
||||
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
|
||||
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
|
||||
|
||||
fetches := readCacheStatsCounters(t, svc.cache).upstreamFetchCount
|
||||
t.Logf("upstream fetches = %d", fetches)
|
||||
@@ -149,16 +153,16 @@ func TestService_Get_FetchesAgainForPartialCachedSource(t *testing.T) {
|
||||
|
||||
svc, fixtures := SetupTestService(t)
|
||||
|
||||
getImage(t, svc, photoVariant(fixtures, 85, FitCover))
|
||||
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, FitCover))
|
||||
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
wholeSize := fileSize(t, path)
|
||||
|
||||
truncateFile(t, path, wholeSize/2)
|
||||
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
|
||||
|
||||
getImage(t, svc, photoVariant(fixtures, 40, FitCover))
|
||||
getImage(t, svc, photoVariant(fixtures, 30, FitCover))
|
||||
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
|
||||
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
|
||||
|
||||
fetches := readCacheStatsCounters(t, svc.cache).upstreamFetchCount
|
||||
t.Logf("upstream fetches = %d", fetches)
|
||||
@@ -182,11 +186,11 @@ func assertConvertedAgain(t *testing.T, size int64) {
|
||||
t.Helper()
|
||||
|
||||
svc, fixtures := SetupTestService(t)
|
||||
req := photoVariant(fixtures, 85, FitCover)
|
||||
req := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
|
||||
want, _ := getImage(t, svc, req)
|
||||
|
||||
path := svc.cache.variants.keyToPath(CacheKey(req))
|
||||
path := svc.cache.variants.keyToPath(imagerequest.CacheKey(req))
|
||||
|
||||
truncateFile(t, path, size)
|
||||
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
|
||||
@@ -310,10 +314,10 @@ func TestVariantStorage_StoreReplacesMetaFile(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestContentStorage_StoreHashedReplacesFileOfAnotherSize checks that a
|
||||
// store over a file of another size at the content's path, such as a
|
||||
// TestContentStorage_StoreReplacesFileOfAnotherSize checks that a
|
||||
// store over a file of another size at the source's path, such as a
|
||||
// partial file a crash left, replaces it with the content.
|
||||
func TestContentStorage_StoreHashedReplacesFileOfAnotherSize(t *testing.T) {
|
||||
func TestContentStorage_StoreReplacesFileOfAnotherSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
storage, err := NewContentStorage(t.TempDir())
|
||||
@@ -322,13 +326,12 @@ func TestContentStorage_StoreHashedReplacesFileOfAnotherSize(t *testing.T) {
|
||||
}
|
||||
|
||||
content := []byte("the whole source image")
|
||||
sum := sha256.Sum256(content)
|
||||
hash := ContentHash(hex.EncodeToString(sum[:]))
|
||||
path := storage.hashToPath(hash)
|
||||
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
|
||||
path := storage.keyToPath(key)
|
||||
|
||||
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
||||
if err != nil {
|
||||
t.Fatalf("creating the content's directory: %v", err)
|
||||
t.Fatalf("creating the source's directory: %v", err)
|
||||
}
|
||||
|
||||
err = os.WriteFile(path, content[:5], StorageFilePerm)
|
||||
@@ -336,12 +339,12 @@ func TestContentStorage_StoreHashedReplacesFileOfAnotherSize(t *testing.T) {
|
||||
t.Fatalf("writing a partial file: %v", err)
|
||||
}
|
||||
|
||||
_, err = storage.StoreHashed(hash, content)
|
||||
_, err = storage.Store(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("StoreHashed() error = %v", err)
|
||||
t.Fatalf("Store() error = %v", err)
|
||||
}
|
||||
|
||||
stored, err := storage.Load(hash)
|
||||
stored, err := storage.Load(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -6,6 +6,8 @@ import (
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// pendingCountsWait is how long a test waits for the database's connection,
|
||||
@@ -92,7 +94,7 @@ func TestService_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy(t *testing.T) {
|
||||
|
||||
deadline, _ := ctx.Deadline()
|
||||
|
||||
results := startGet(ctx, svc, photoVariant(fixtures, 85, FitCover))
|
||||
results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
|
||||
|
||||
// The request has made its database reads by the time it fetches.
|
||||
select {
|
||||
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"sneak.berlin/go/pixa/internal/allowlist"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
@@ -139,7 +140,9 @@ var ErrProcessingTimedOut = errors.New(
|
||||
// While its source URL is in the negative cache, a cached variant is still
|
||||
// served and a variant not in the cache is still made from a cached source;
|
||||
// only a request that needs the source fetched is refused.
|
||||
func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, error) {
|
||||
func (s *Service) Get(
|
||||
ctx context.Context, req *imagerequest.Request,
|
||||
) (*ImageResponse, error) {
|
||||
// Propagate AllowHTTP setting to the request
|
||||
req.AllowHTTP = s.allowHTTP
|
||||
|
||||
@@ -186,7 +189,7 @@ func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, e
|
||||
}
|
||||
|
||||
// Warm pre-fetches and caches an image without returning it.
|
||||
func (s *Service) Warm(ctx context.Context, req *ImageRequest) error {
|
||||
func (s *Service) Warm(ctx context.Context, req *imagerequest.Request) error {
|
||||
_, err := s.Get(ctx, req)
|
||||
|
||||
return err
|
||||
@@ -204,7 +207,7 @@ func (s *Service) WaitForProcessing(ctx context.Context) int {
|
||||
}
|
||||
|
||||
// ValidateRequest validates the request signature if required.
|
||||
func (s *Service) ValidateRequest(req *ImageRequest) error {
|
||||
func (s *Service) ValidateRequest(req *imagerequest.Request) error {
|
||||
// Check if host is allowed (no signature required)
|
||||
sourceURL := req.SourceURL()
|
||||
|
||||
@@ -227,7 +230,7 @@ func (s *Service) ValidateRequest(req *ImageRequest) error {
|
||||
}
|
||||
|
||||
// Signature required for non-allowed hosts
|
||||
return s.signer.Verify(signatureRequest(req))
|
||||
return s.signer.Verify(req)
|
||||
}
|
||||
|
||||
// GenerateSignedURL generates a signed URL for the given request. The URL
|
||||
@@ -236,26 +239,22 @@ func (s *Service) ValidateRequest(req *ImageRequest) error {
|
||||
// set to 85 or cover, the values the route uses when a URL has no q or fit.
|
||||
func (s *Service) GenerateSignedURL(
|
||||
baseURL string,
|
||||
req *ImageRequest,
|
||||
req *imagerequest.Request,
|
||||
ttl time.Duration,
|
||||
) (string, error) {
|
||||
if req.Quality == 0 {
|
||||
req.Quality = 85
|
||||
if req.Transform.Quality == 0 {
|
||||
req.Transform.Quality = 85
|
||||
}
|
||||
|
||||
if req.FitMode == "" {
|
||||
req.FitMode = FitCover
|
||||
if req.Transform.FitMode == "" {
|
||||
req.Transform.FitMode = imagerequest.FitCover
|
||||
}
|
||||
|
||||
sigReq := signatureRequest(req)
|
||||
path, sig, exp := s.signer.GenerateSignedURL(sigReq, ttl)
|
||||
|
||||
// Propagate the generated signature and expiration back onto the request.
|
||||
req.Expires = sigReq.Expires
|
||||
req.Signature = sigReq.Signature
|
||||
// Sets the request's Expires and Signature
|
||||
path, sig, exp := s.signer.GenerateSignedURL(req, ttl)
|
||||
|
||||
return fmt.Sprintf("%s%s?sig=%s&exp=%d&q=%d&fit=%s",
|
||||
baseURL, path, sig, exp, req.Quality, req.FitMode), nil
|
||||
baseURL, path, sig, exp, req.Transform.Quality, req.Transform.FitMode), nil
|
||||
}
|
||||
|
||||
// errPanicked is returned when processing a variant panicked.
|
||||
@@ -268,14 +267,14 @@ var errPanicked = errors.New("panic while processing image")
|
||||
// the first request's cancellation, so the others are still served if that
|
||||
// client goes away, but keeps its deadline.
|
||||
func (s *Service) processOrWait(
|
||||
ctx context.Context, req *ImageRequest,
|
||||
ctx context.Context, req *imagerequest.Request,
|
||||
) (*ImageResponse, error) {
|
||||
// A request that has already ended starts no processing
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
|
||||
cacheKey := CacheKey(req)
|
||||
cacheKey := imagerequest.CacheKey(req)
|
||||
|
||||
// Closed when this request's own function runs, which singleflight does
|
||||
// only when no other request is processing the variant
|
||||
@@ -293,7 +292,7 @@ func (s *Service) processOrWait(
|
||||
recovered := recover()
|
||||
if recovered != nil {
|
||||
s.log.Error("panic while processing image",
|
||||
"host", req.SourceHost, "path", req.SourcePath,
|
||||
"host", req.Source.Host, "path", req.Source.Path,
|
||||
"panic", recovered, "stack", string(debug.Stack()))
|
||||
|
||||
if hub := sentry.GetHubFromContext(ctx); hub != nil {
|
||||
@@ -360,9 +359,9 @@ func withoutCancelKeepingDeadline(
|
||||
// returns it with its size; nil if the cached data is unavailable, empty or
|
||||
// exceeds maxResponseSize.
|
||||
func (s *Service) loadCachedSource(
|
||||
contentHash ContentHash,
|
||||
sourceKey imagerequest.SourceKey,
|
||||
) (io.ReadCloser, int64) {
|
||||
reader, size, err := s.cache.GetSourceContent(contentHash)
|
||||
reader, size, err := s.cache.GetSourceContent(sourceKey)
|
||||
if err != nil {
|
||||
s.log.Warn("failed to load cached source, fetching", "error", err)
|
||||
|
||||
@@ -373,7 +372,7 @@ func (s *Service) loadCachedSource(
|
||||
_ = reader.Close()
|
||||
|
||||
s.log.Warn("cached source exceeds max response size, discarding",
|
||||
"hash", contentHash,
|
||||
"source_key", sourceKey,
|
||||
"max_bytes", s.maxResponseSize,
|
||||
)
|
||||
|
||||
@@ -393,11 +392,11 @@ func (s *Service) loadCachedSource(
|
||||
// if available.
|
||||
func (s *Service) processFromSourceOrFetch(
|
||||
ctx context.Context,
|
||||
req *ImageRequest,
|
||||
cacheKey VariantKey,
|
||||
req *imagerequest.Request,
|
||||
cacheKey imagerequest.VariantKey,
|
||||
) (*processedVariant, error) {
|
||||
// Check if we have cached source content
|
||||
contentHash, _, err := s.cache.LookupSource(ctx, req)
|
||||
sourceKey, _, err := s.cache.LookupSource(ctx, req)
|
||||
if err != nil {
|
||||
s.log.Warn("source lookup failed", "error", err)
|
||||
}
|
||||
@@ -407,9 +406,9 @@ func (s *Service) processFromSourceOrFetch(
|
||||
sourceSize int64
|
||||
)
|
||||
|
||||
if contentHash != "" {
|
||||
s.log.Debug("using cached source", "hash", contentHash)
|
||||
source, sourceSize = s.loadCachedSource(contentHash)
|
||||
if sourceKey != "" {
|
||||
s.log.Debug("using cached source", "source_key", sourceKey)
|
||||
source, sourceSize = s.loadCachedSource(sourceKey)
|
||||
}
|
||||
|
||||
// Fetch from upstream if we don't have source data or it's empty
|
||||
@@ -431,8 +430,8 @@ func (s *Service) processFromSourceOrFetch(
|
||||
// source URL is in the negative cache.
|
||||
func (s *Service) fetchAndProcess(
|
||||
ctx context.Context,
|
||||
req *ImageRequest,
|
||||
cacheKey VariantKey,
|
||||
req *imagerequest.Request,
|
||||
cacheKey imagerequest.VariantKey,
|
||||
) (*processedVariant, error) {
|
||||
// Check negative cache - skip fetching for recently-failed URLs
|
||||
negHit, err := s.cache.checkNegativeCache(ctx, req)
|
||||
@@ -442,8 +441,8 @@ func (s *Service) fetchAndProcess(
|
||||
|
||||
if negHit {
|
||||
s.log.Debug("negative cache hit",
|
||||
"host", req.SourceHost,
|
||||
"path", req.SourcePath,
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
)
|
||||
|
||||
return nil, fmt.Errorf("%w: %w", httpfetcher.ErrUpstreamError, ErrNegativeCached)
|
||||
@@ -493,8 +492,8 @@ func (s *Service) fetchAndProcess(
|
||||
// Log upstream fetch details
|
||||
s.log.Info("upstream fetched",
|
||||
"request_id", middleware.GetReqID(ctx),
|
||||
"host", req.SourceHost,
|
||||
"path", req.SourcePath,
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"bytes", fetchBytes,
|
||||
"fetch_ms", fetchResult.FetchDurationMs,
|
||||
"rate", downloadRate,
|
||||
@@ -510,7 +509,7 @@ func (s *Service) fetchAndProcess(
|
||||
err = magic.ValidateMagicBytes(sourceData, fetchResult.ContentType)
|
||||
if err != nil {
|
||||
s.log.Warn("source refused", "request_id", middleware.GetReqID(ctx),
|
||||
"host", req.SourceHost, "path", req.SourcePath,
|
||||
"host", req.Source.Host, "path", req.Source.Path,
|
||||
"content_type", fetchResult.ContentType, "error", err)
|
||||
s.storeNegative(ctx, req, httpStatusUnsupportedMediaType, err)
|
||||
|
||||
@@ -533,12 +532,12 @@ func (s *Service) fetchAndProcess(
|
||||
// failed with err, so the source is not fetched again until the entry
|
||||
// expires. A failure to store the entry is only logged.
|
||||
func (s *Service) storeNegative(
|
||||
ctx context.Context, req *ImageRequest, statusCode int, err error,
|
||||
ctx context.Context, req *imagerequest.Request, statusCode int, err error,
|
||||
) {
|
||||
storeErr := s.cache.StoreNegative(ctx, req, statusCode, err.Error())
|
||||
if storeErr != nil {
|
||||
s.log.Warn("failed to store negative cache entry",
|
||||
"host", req.SourceHost, "path", req.SourcePath, "error", storeErr)
|
||||
"host", req.Source.Host, "path", req.Source.Path, "error", storeErr)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -554,24 +553,17 @@ type processedVariant struct {
|
||||
// result.
|
||||
func (s *Service) processAndStore(
|
||||
ctx context.Context,
|
||||
req *ImageRequest,
|
||||
cacheKey VariantKey,
|
||||
req *imagerequest.Request,
|
||||
cacheKey imagerequest.VariantKey,
|
||||
source io.Reader,
|
||||
fetchBytes int64,
|
||||
) (*processedVariant, error) {
|
||||
// Process the image
|
||||
processStart := time.Now()
|
||||
|
||||
processReq := &imageprocessor.Request{
|
||||
Size: imageprocessor.Size{Width: req.Size.Width, Height: req.Size.Height},
|
||||
Format: imageprocessor.Format(req.Format),
|
||||
Quality: req.Quality,
|
||||
FitMode: imageprocessor.FitMode(req.FitMode),
|
||||
}
|
||||
|
||||
// ctx has the deadline of the request doing the processing, which can
|
||||
// end while a request waiting for its image has time left
|
||||
processResult, err := s.processor.Process(ctx, source, processReq)
|
||||
processResult, err := s.processor.Process(ctx, source, &req.Transform)
|
||||
if errors.Is(err, context.DeadlineExceeded) {
|
||||
return nil, fmt.Errorf("%w: %w", ErrProcessingTimedOut, err)
|
||||
}
|
||||
@@ -604,10 +596,10 @@ func (s *Service) processAndStore(
|
||||
|
||||
s.log.Info("image converted",
|
||||
"request_id", middleware.GetReqID(ctx),
|
||||
"host", req.SourceHost,
|
||||
"path", req.SourcePath,
|
||||
"host", req.Source.Host,
|
||||
"path", req.Source.Path,
|
||||
"src_format", processResult.InputFormat,
|
||||
"dst_format", req.Format,
|
||||
"dst_format", req.Transform.Format,
|
||||
"src_bytes", fetchBytes,
|
||||
"dst_bytes", outputSize,
|
||||
"src_dimensions", fmt.Sprintf("%dx%d",
|
||||
@@ -616,8 +608,8 @@ func (s *Service) processAndStore(
|
||||
processResult.Width, processResult.Height),
|
||||
"size_ratio", fmt.Sprintf("%.1f%%", sizePercent),
|
||||
"convert_ms", processDuration.Milliseconds(),
|
||||
"quality", req.Quality,
|
||||
"fit", req.FitMode,
|
||||
"quality", req.Transform.Quality,
|
||||
"fit", req.Transform.FitMode,
|
||||
)
|
||||
|
||||
// Store variant to cache
|
||||
@@ -636,24 +628,6 @@ func (s *Service) processAndStore(
|
||||
}, nil
|
||||
}
|
||||
|
||||
// signatureRequest projects an ImageRequest onto the standalone
|
||||
// signature.Request type used by the signature package. This keeps the
|
||||
// import edge one-way: imgcache depends on signature, never the reverse.
|
||||
func signatureRequest(req *ImageRequest) *signature.Request {
|
||||
return &signature.Request{
|
||||
SourceHost: req.SourceHost,
|
||||
SourcePath: req.SourcePath,
|
||||
SourceQuery: req.SourceQuery,
|
||||
Width: req.Size.Width,
|
||||
Height: req.Size.Height,
|
||||
Format: string(req.Format),
|
||||
Quality: req.Quality,
|
||||
FitMode: string(req.FitMode),
|
||||
Signature: req.Signature,
|
||||
Expires: req.Expires,
|
||||
}
|
||||
}
|
||||
|
||||
// HTTP status codes for error responses.
|
||||
const (
|
||||
httpStatusBadGateway = 502
|
||||
@@ -680,7 +654,7 @@ func extractStatusCode(err error) int {
|
||||
const etagHashLength = 16
|
||||
|
||||
// formatETag formats a VariantKey as a quoted ETag value.
|
||||
func formatETag(key VariantKey) string {
|
||||
func formatETag(key imagerequest.VariantKey) string {
|
||||
hash := string(key)
|
||||
// Use first 16 characters of hash for a shorter but still unique ETag
|
||||
if len(hash) > etagHashLength {
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
@@ -25,13 +26,17 @@ func TestService_Get_AllowlistedHost(t *testing.T) {
|
||||
svc, fixtures := SetupTestService(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
resp, err := svc.Get(ctx, req)
|
||||
@@ -61,13 +66,17 @@ func TestService_Get_NonAllowlistedHost_NoSignature(t *testing.T) {
|
||||
|
||||
svc, fixtures := SetupTestService(t, WithSigningKey("test-key"))
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.OtherHost,
|
||||
SourcePath: testPathUpload,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.OtherHost,
|
||||
Path: testPathUpload,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Should fail validation - not allowlisted and no signature
|
||||
@@ -84,19 +93,23 @@ func TestService_Get_NonAllowlistedHost_ValidSignature(t *testing.T) {
|
||||
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.OtherHost,
|
||||
SourcePath: testPathUpload,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.OtherHost,
|
||||
Path: testPathUpload,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Generate a valid signature
|
||||
signer := signature.New(signingKey)
|
||||
req.Expires = time.Now().Add(time.Hour)
|
||||
req.Signature = signer.Sign(signatureRequest(req))
|
||||
req.Signature = signer.Sign(req)
|
||||
|
||||
// Should pass validation
|
||||
err := svc.ValidateRequest(req)
|
||||
@@ -128,19 +141,23 @@ func TestService_Get_NonAllowlistedHost_ExpiredSignature(t *testing.T) {
|
||||
signingKey := testSigningKey
|
||||
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.OtherHost,
|
||||
SourcePath: testPathUpload,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.OtherHost,
|
||||
Path: testPathUpload,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Generate an expired signature
|
||||
signer := signature.New(signingKey)
|
||||
req.Expires = time.Now().Add(-time.Hour) // Already expired
|
||||
req.Signature = signer.Sign(signatureRequest(req))
|
||||
req.Signature = signer.Sign(req)
|
||||
|
||||
// Should fail validation
|
||||
err := svc.ValidateRequest(req)
|
||||
@@ -155,13 +172,17 @@ func TestService_Get_NonAllowlistedHost_InvalidSignature(t *testing.T) {
|
||||
signingKey := testSigningKey
|
||||
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.OtherHost,
|
||||
SourcePath: testPathUpload,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.OtherHost,
|
||||
Path: testPathUpload,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// Set an invalid signature
|
||||
@@ -191,16 +212,20 @@ func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
|
||||
signer := signature.New(signingKey)
|
||||
|
||||
// Sign a request for "cdn.example.com"
|
||||
signedReq := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: testPathCat,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
signedReq := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
Expires: time.Now().Add(time.Hour),
|
||||
}
|
||||
signedReq.Signature = signer.Sign(signatureRequest(signedReq))
|
||||
signedReq.Signature = signer.Sign(signedReq)
|
||||
|
||||
// The original request should pass validation
|
||||
t.Run("exact host passes", func(t *testing.T) {
|
||||
@@ -228,23 +253,22 @@ func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
|
||||
t.Run(tt.name+" rejected", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: tt.host,
|
||||
SourcePath: signedReq.SourcePath,
|
||||
SourceQuery: signedReq.SourceQuery,
|
||||
Size: signedReq.Size,
|
||||
Format: signedReq.Format,
|
||||
Quality: signedReq.Quality,
|
||||
FitMode: signedReq.FitMode,
|
||||
Expires: signedReq.Expires,
|
||||
Signature: signedReq.Signature,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: tt.host,
|
||||
Path: signedReq.Source.Path,
|
||||
Query: signedReq.Source.Query,
|
||||
},
|
||||
Transform: signedReq.Transform,
|
||||
Expires: signedReq.Expires,
|
||||
Signature: signedReq.Signature,
|
||||
}
|
||||
|
||||
err := svc.ValidateRequest(req)
|
||||
if err == nil {
|
||||
t.Errorf(
|
||||
"ValidateRequest() should reject signature for host %q (signed for %q)",
|
||||
tt.host, signedReq.SourceHost)
|
||||
tt.host, signedReq.Source.Host)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -280,13 +304,17 @@ func TestService_ValidateRequest_HostWithPortNeedsSignature(t *testing.T) {
|
||||
t.Run(tt.host, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: tt.host,
|
||||
SourcePath: testPathCat,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: tt.host,
|
||||
Path: testPathCat,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
err := svc.ValidateRequest(req)
|
||||
@@ -320,13 +348,17 @@ func TestService_ValidateRequest_UnbracketedIPv6PortNeedsSignature(
|
||||
t.Run(host, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: host,
|
||||
SourcePath: testPathCat,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: host,
|
||||
Path: testPathCat,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
err := svc.ValidateRequest(req)
|
||||
@@ -347,14 +379,9 @@ func TestService_Get_InvalidFile(t *testing.T) {
|
||||
svc, fixtures := SetupTestService(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: "/images/fake.jpg",
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
}
|
||||
req := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
req.Source.Path = "/images/fake.jpg"
|
||||
req.Transform.Size = imagerequest.Size{Width: 50, Height: 50}
|
||||
|
||||
// Should fail because magic bytes don't match
|
||||
_, err := svc.Get(ctx, req)
|
||||
@@ -369,14 +396,9 @@ func TestService_Get_NotFound(t *testing.T) {
|
||||
svc, fixtures := SetupTestService(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: "/images/nonexistent.jpg",
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
}
|
||||
req := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
req.Source.Path = "/images/nonexistent.jpg"
|
||||
req.Transform.Size = imagerequest.Size{Width: 50, Height: 50}
|
||||
|
||||
_, err := svc.Get(ctx, req)
|
||||
if err == nil {
|
||||
@@ -393,25 +415,25 @@ func TestService_Get_FormatConversion(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
sourcePath string
|
||||
outFormat ImageFormat
|
||||
outFormat imagerequest.Format
|
||||
wantMIME string
|
||||
}{
|
||||
{
|
||||
name: "JPEG to PNG",
|
||||
sourcePath: testPathPhoto,
|
||||
outFormat: FormatPNG,
|
||||
outFormat: imagerequest.FormatPNG,
|
||||
wantMIME: "image/png",
|
||||
},
|
||||
{
|
||||
name: "PNG to JPEG",
|
||||
sourcePath: "/images/logo.png",
|
||||
outFormat: FormatJPEG,
|
||||
outFormat: imagerequest.FormatJPEG,
|
||||
wantMIME: testContentTypeJPEG,
|
||||
},
|
||||
{
|
||||
name: "GIF to PNG",
|
||||
sourcePath: "/images/animation.gif",
|
||||
outFormat: FormatPNG,
|
||||
outFormat: imagerequest.FormatPNG,
|
||||
wantMIME: "image/png",
|
||||
},
|
||||
}
|
||||
@@ -420,13 +442,12 @@ func TestService_Get_FormatConversion(t *testing.T) {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: tt.sourcePath,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: tt.outFormat,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: fixtures.GoodHost, Path: tt.sourcePath},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: tt.outFormat, Quality: 85, FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
resp, err := svc.Get(ctx, req)
|
||||
@@ -474,13 +495,17 @@ func TestService_Get_Caching(t *testing.T) {
|
||||
svc, fixtures := SetupTestService(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// First request - should be a cache miss
|
||||
@@ -525,26 +550,22 @@ func TestService_Get_Caching(t *testing.T) {
|
||||
|
||||
// TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo caches a source and
|
||||
// a variant of it and sets back two days every time pixa stored for them.
|
||||
// For the source: fetched_at in source_metadata, fetched_at and
|
||||
// last_accessed_at in source_content, the fetched_at in its metadata file,
|
||||
// and the access and modification times of its file and of its metadata
|
||||
// file. For the variant: created_at and last_accessed_at in variant_content,
|
||||
// the created_at in its .meta file, and the access and modification times of
|
||||
// its file and of its .meta file. It then checks that the variant is still
|
||||
// served from the cache and that a new variant is made from the cached source
|
||||
// without fetching it again.
|
||||
// For the source: fetched_at and last_accessed_at in source_content, and the
|
||||
// access and modification times of its file. For the variant: created_at and
|
||||
// last_accessed_at in variant_content, the created_at in its .meta file, and
|
||||
// the access and modification times of its file and of its .meta file. It
|
||||
// then checks that the variant is still served from the cache and that a new
|
||||
// variant is made from the cached source without fetching it again.
|
||||
//
|
||||
// Left at the present: the files' status change and birth times, which a
|
||||
// program cannot set, and the times of the directories holding the files,
|
||||
// which other entries share. The source's stored response headers are empty,
|
||||
// as the mock fetcher sends none. expires_at, etag and last_modified in
|
||||
// source_metadata and in the metadata file are never written, nor are the
|
||||
// request_cache and output_content tables.
|
||||
// which other entries share. The request_cache and output_content tables are
|
||||
// never written.
|
||||
func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures := SetupTestService(t)
|
||||
variant := photoVariant(fixtures, 85, FitCover)
|
||||
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
|
||||
|
||||
resp, err := svc.Get(t.Context(), variant)
|
||||
if err != nil {
|
||||
@@ -570,7 +591,7 @@ func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
|
||||
}
|
||||
|
||||
// Another quality is a new variant, made from the cached source
|
||||
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, FitCover))
|
||||
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
|
||||
if err != nil {
|
||||
t.Fatalf("Get() of a new variant error = %v", err)
|
||||
}
|
||||
@@ -586,43 +607,23 @@ func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
|
||||
// backdateSource sets every time pixa stored for req's cached source to when,
|
||||
// as listed on TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo.
|
||||
func backdateSource(
|
||||
t *testing.T, cache *Cache, req *ImageRequest, when time.Time,
|
||||
t *testing.T, cache *Cache, req *imagerequest.Request, when time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
pathHash := HashPath(req.SourcePath + "?" + req.SourceQuery)
|
||||
|
||||
meta, err := cache.srcMetadata.Load(req.SourceHost, pathHash)
|
||||
if err != nil {
|
||||
t.Fatalf("reading the source's metadata file: %v", err)
|
||||
}
|
||||
|
||||
meta.FetchedAt = when.Unix()
|
||||
|
||||
err = cache.srcMetadata.Store(req.SourceHost, pathHash, meta)
|
||||
if err != nil {
|
||||
t.Fatalf("writing the source's metadata file: %v", err)
|
||||
}
|
||||
|
||||
key := req.Source.Key()
|
||||
dbTime := when.UTC().Format(sqliteTimestampFormat)
|
||||
|
||||
updateOneRow(t, cache, `
|
||||
UPDATE source_metadata SET fetched_at = ?
|
||||
WHERE source_host = ? AND source_path = ? AND source_query = ?
|
||||
`, dbTime, req.SourceHost, req.SourcePath, req.SourceQuery)
|
||||
updateOneRow(t, cache,
|
||||
`UPDATE source_content SET fetched_at = ? WHERE content_hash = ?`,
|
||||
dbTime, meta.ContentHash)
|
||||
setSourceLastAccessed(t, cache, ContentHash(meta.ContentHash), when)
|
||||
`UPDATE source_content SET fetched_at = ? WHERE source_key = ?`,
|
||||
dbTime, string(key))
|
||||
setSourceLastAccessed(t, cache, key, when)
|
||||
|
||||
for _, path := range []string{
|
||||
cache.srcContent.hashToPath(ContentHash(meta.ContentHash)),
|
||||
cache.srcMetadata.metaPath(req.SourceHost, pathHash),
|
||||
} {
|
||||
err = os.Chtimes(path, when, when)
|
||||
if err != nil {
|
||||
t.Fatalf("setting back the times of %s: %v", path, err)
|
||||
}
|
||||
path := cache.srcContent.keyToPath(key)
|
||||
|
||||
err := os.Chtimes(path, when, when)
|
||||
if err != nil {
|
||||
t.Fatalf("setting back the times of %s: %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -630,11 +631,11 @@ func backdateSource(
|
||||
// for to when, as listed on
|
||||
// TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo.
|
||||
func backdateVariant(
|
||||
t *testing.T, cache *Cache, req *ImageRequest, when time.Time,
|
||||
t *testing.T, cache *Cache, req *imagerequest.Request, when time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
key := CacheKey(req)
|
||||
key := imagerequest.CacheKey(req)
|
||||
|
||||
content, size, contentType, err := cache.variants.LoadWithMeta(key)
|
||||
if err != nil || contentType == "" {
|
||||
@@ -696,7 +697,7 @@ func TestService_Get_DifferentSizes(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
// Request same image at different sizes
|
||||
sizes := []Size{
|
||||
sizes := []imagerequest.Size{
|
||||
{Width: 25, Height: 25},
|
||||
{Width: 50, Height: 50},
|
||||
{Width: 75, Height: 75},
|
||||
@@ -705,13 +706,17 @@ func TestService_Get_DifferentSizes(t *testing.T) {
|
||||
responses := make([][]byte, 0, len(sizes))
|
||||
|
||||
for _, size := range sizes {
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: size,
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: size,
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
resp, err := svc.Get(ctx, req)
|
||||
@@ -745,13 +750,17 @@ func TestService_ValidateRequest_NoSigningKey(t *testing.T) {
|
||||
// Service with no signing key - all non-allowlisted requests should fail
|
||||
svc, fixtures := SetupTestService(t, WithNoAllowlist())
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.OtherHost,
|
||||
SourcePath: testPathUpload,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.OtherHost,
|
||||
Path: testPathUpload,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
err := svc.ValidateRequest(req)
|
||||
@@ -770,13 +779,17 @@ func TestService_Get_ContextCancellation(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel() // Cancel immediately
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
_, err := svc.Get(ctx, req)
|
||||
@@ -791,13 +804,17 @@ func TestService_Get_ReturnsETag(t *testing.T) {
|
||||
svc, fixtures := SetupTestService(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
resp, err := svc.Get(ctx, req)
|
||||
@@ -824,13 +841,17 @@ func TestService_Get_ETagConsistency(t *testing.T) {
|
||||
svc, fixtures := SetupTestService(t)
|
||||
ctx := context.Background()
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// First request
|
||||
@@ -866,22 +887,30 @@ func TestService_Get_DifferentETagsForDifferentContent(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
// Request same image at different sizes - should get different ETags
|
||||
req1 := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: Size{Width: 25, Height: 25},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req1 := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 25, Height: 25},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
req2 := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req2 := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
resp1, err := svc.Get(ctx, req1)
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestStoreSource_StoresIdenticalBytesOfTwoURLsTwice stores the same bytes
|
||||
// for two source URLs and checks that each URL has a source_content row of
|
||||
// its own and a file of its own, named by its source key under
|
||||
// cache/sources/<ab>/<cd>/.
|
||||
func TestStoreSource_StoresIdenticalBytesOfTwoURLsTwice(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, stateDir := newEvictionTestCache(t, 1<<30)
|
||||
content := bytes.Repeat([]byte{0x5A}, 400)
|
||||
|
||||
keys := map[string]imagerequest.SourceKey{
|
||||
"/a.jpg": storeEvictionTestSource(t, cache, testHostCDN, "/a.jpg", content),
|
||||
"/b.jpg": storeEvictionTestSource(t, cache, testHostCDN, "/b.jpg", content),
|
||||
}
|
||||
t.Logf("source keys: %v", keys)
|
||||
|
||||
if keys["/a.jpg"] == keys["/b.jpg"] {
|
||||
t.Fatalf("both source URLs got the key %s", keys["/a.jpg"])
|
||||
}
|
||||
|
||||
for sourcePath, key := range keys {
|
||||
rows := countRows(t, cache, `
|
||||
SELECT COUNT(*) FROM source_content
|
||||
WHERE source_key = ? AND source_host = ? AND source_path = ?
|
||||
`, string(key), testHostCDN, sourcePath)
|
||||
if rows != 1 {
|
||||
t.Errorf("%s has %d source_content rows, want 1", sourcePath, rows)
|
||||
}
|
||||
|
||||
name := string(key)
|
||||
file := filepath.Join(stateDir, "cache", "sources", name[0:2], name[2:4], name)
|
||||
|
||||
stored, err := os.ReadFile(file) //nolint:gosec // path in the test's state dir
|
||||
if err != nil || !bytes.Equal(stored, content) {
|
||||
t.Errorf("file of %s at %s holds %d bytes (error %v), want the %d stored",
|
||||
sourcePath, file, len(stored), err, len(content))
|
||||
}
|
||||
}
|
||||
|
||||
usage, err := cache.UsageBytes(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("UsageBytes() error = %v", err)
|
||||
}
|
||||
|
||||
if usage != 800 {
|
||||
t.Errorf("UsageBytes() = %d, want 800, the 400 bytes once for each URL", usage)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed stores the same
|
||||
// bytes for two source URLs and evicts the first. Its row and file must be
|
||||
// gone, and the second must still be found and read from the cache.
|
||||
func TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
content := bytes.Repeat([]byte{0x5B}, 400)
|
||||
|
||||
evicted := storeEvictionTestSource(t, cache, testHostCDN, "/evicted.jpg", content)
|
||||
storeEvictionTestSource(t, cache, testHostCDN, "/kept.jpg", content)
|
||||
|
||||
err := cache.evictSourceBlob(t.Context(), evicted)
|
||||
if err != nil {
|
||||
t.Fatalf("evictSourceBlob() error = %v", err)
|
||||
}
|
||||
|
||||
if cache.srcContent.Exists(evicted) {
|
||||
t.Error("the evicted source's file is still on disk")
|
||||
}
|
||||
|
||||
if n := countRows(t, cache,
|
||||
`SELECT COUNT(*) FROM source_content WHERE source_key = ?`, string(evicted),
|
||||
); n != 0 {
|
||||
t.Errorf("the evicted source still has %d source_content rows", n)
|
||||
}
|
||||
|
||||
kept, contentType, err := cache.LookupSource(t.Context(), &imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: testHostCDN, Path: "/kept.jpg"},
|
||||
})
|
||||
t.Logf("LookupSource() of the kept source = %q, %q, error %v", kept, contentType, err)
|
||||
|
||||
if err != nil || kept == "" {
|
||||
t.Fatalf("LookupSource() of the kept source = %q, %v; want it cached", kept, err)
|
||||
}
|
||||
|
||||
reader, _, err := cache.GetSourceContent(kept)
|
||||
if err != nil {
|
||||
t.Fatalf("GetSourceContent() of the kept source error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = reader.Close() }()
|
||||
|
||||
served, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
t.Fatalf("reading the kept source: %v", err)
|
||||
}
|
||||
|
||||
if !bytes.Equal(served, content) {
|
||||
t.Errorf("the kept source reads as %d bytes, want the %d stored",
|
||||
len(served), len(content))
|
||||
}
|
||||
}
|
||||
|
||||
// TestStoreSource_FailedRowWriteLeavesNoFile drops source_content, so that
|
||||
// StoreSource cannot write the source's row, and checks that the store fails
|
||||
// and leaves no file under the source's key.
|
||||
func TestStoreSource_FailedRowWriteLeavesNoFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
_, err := cache.db.ExecContext(t.Context(), `DROP TABLE source_content`)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to drop source_content: %v", err)
|
||||
}
|
||||
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: testHostCDN, Path: testPathCat},
|
||||
}
|
||||
result := &httpfetcher.FetchResult{ContentType: testContentTypeJPEG}
|
||||
|
||||
key, err := cache.StoreSource(t.Context(), req,
|
||||
bytes.NewReader([]byte("source bytes")), result)
|
||||
t.Logf("StoreSource() = %q, error = %v", key, err)
|
||||
|
||||
if err == nil {
|
||||
t.Error("StoreSource() error = nil, want the failed row write")
|
||||
}
|
||||
|
||||
if cache.srcContent.Exists(req.Source.Key()) {
|
||||
t.Error("the source's file is on disk although its row was not written")
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
|
||||
"sneak.berlin/go/pixa/internal/database"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
func setupStatsTestDB(t *testing.T) *sql.DB {
|
||||
@@ -319,13 +320,17 @@ func TestService_Get_CountsStats(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, step := range steps {
|
||||
resp, err := svc.Get(t.Context(), &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: step.path,
|
||||
Size: Size{Width: step.size, Height: step.size},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
resp, err := svc.Get(t.Context(), &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: step.path,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: step.size, Height: step.size},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
})
|
||||
if (err != nil) != step.wantErr {
|
||||
t.Fatalf("%s: Get() error = %v, want error %t", step.name, err, step.wantErr)
|
||||
@@ -419,13 +424,17 @@ func TestService_Get_CountsInterruptedMisses(t *testing.T) {
|
||||
|
||||
svc.fetcher = upstream
|
||||
|
||||
resp, err := svc.Get(ctx, &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
resp, err := svc.Get(ctx, &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
})
|
||||
t.Logf("Get() error = %v", err)
|
||||
|
||||
@@ -453,13 +462,17 @@ func TestService_Get_CountsHitAfterRequestEnds(t *testing.T) {
|
||||
|
||||
svc, fixtures := SetupTestService(t)
|
||||
|
||||
req := &ImageRequest{
|
||||
SourceHost: fixtures.GoodHost,
|
||||
SourcePath: testPathPhoto,
|
||||
Size: Size{Width: 50, Height: 50},
|
||||
Format: FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: fixtures.GoodHost,
|
||||
Path: testPathPhoto,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 50, Height: 50},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}
|
||||
|
||||
// A first request caches the variant.
|
||||
|
||||
+47
-268
@@ -1,8 +1,6 @@
|
||||
package imgcache
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -10,8 +8,9 @@ import (
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// Storage constants.
|
||||
@@ -29,17 +28,8 @@ var (
|
||||
ErrNotFound = errors.New("content not found")
|
||||
)
|
||||
|
||||
// ContentHash is a SHA256 hash of file content (hex-encoded).
|
||||
type ContentHash string
|
||||
|
||||
// VariantKey is a SHA256 hash identifying a specific image variant (hex-encoded).
|
||||
type VariantKey string
|
||||
|
||||
// PathHash is a SHA256 hash of a URL path (hex-encoded).
|
||||
type PathHash string
|
||||
|
||||
// ContentStorage handles content-addressable file storage.
|
||||
// Files are stored at: <basedir>/<ab>/<cd>/<abcdef...sha256>
|
||||
// ContentStorage handles source image file storage, one file per source
|
||||
// URL. Files are stored at: <basedir>/<ab>/<cd>/<source key>
|
||||
type ContentStorage struct {
|
||||
baseDir string
|
||||
}
|
||||
@@ -54,38 +44,19 @@ func NewContentStorage(baseDir string) (*ContentStorage, error) {
|
||||
return &ContentStorage{baseDir: baseDir}, nil
|
||||
}
|
||||
|
||||
// Store writes content to storage and returns its SHA256 hash.
|
||||
// The content is read fully into memory to compute the hash before writing.
|
||||
func (s *ContentStorage) Store(r io.Reader) (ContentHash, int64, error) {
|
||||
// Read all content to compute hash
|
||||
data, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
return "", 0, fmt.Errorf("failed to read content: %w", err)
|
||||
}
|
||||
|
||||
// Compute hash
|
||||
h := sha256.Sum256(data)
|
||||
hash := ContentHash(hex.EncodeToString(h[:]))
|
||||
size := int64(len(data))
|
||||
|
||||
err = s.writeIfAbsent(hash, data)
|
||||
if err != nil {
|
||||
return "", 0, err
|
||||
}
|
||||
|
||||
return hash, size, nil
|
||||
}
|
||||
|
||||
// StoreHashed writes pre-hashed content to storage at the path derived
|
||||
// from hash, without recomputing it. Callers that already know the
|
||||
// hash before writing (e.g. because they must hold a hash-keyed lock
|
||||
// across the whole store operation) use this instead of Store. Like
|
||||
// Store, it is idempotent: a file of the content's size already at that
|
||||
// path is left untouched, and a file of another size is replaced.
|
||||
func (s *ContentStorage) StoreHashed(
|
||||
hash ContentHash, data []byte,
|
||||
// Store writes data with writeFileSynced to the path derived from key,
|
||||
// replacing any file already there, and returns its size.
|
||||
func (s *ContentStorage) Store(
|
||||
key imagerequest.SourceKey, data []byte,
|
||||
) (int64, error) {
|
||||
err := s.writeIfAbsent(hash, data)
|
||||
path := s.keyToPath(key)
|
||||
|
||||
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("failed to create directory: %w", err)
|
||||
}
|
||||
|
||||
err = writeFileSynced(path, data)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
@@ -93,11 +64,11 @@ func (s *ContentStorage) StoreHashed(
|
||||
return int64(len(data)), nil
|
||||
}
|
||||
|
||||
// Load returns a reader for the content with the given hash.
|
||||
func (s *ContentStorage) Load(hash ContentHash) (io.ReadCloser, error) {
|
||||
path := s.hashToPath(hash)
|
||||
// Load returns a reader for the content with the given key.
|
||||
func (s *ContentStorage) Load(key imagerequest.SourceKey) (io.ReadCloser, error) {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
f, err := os.Open(path) //nolint:gosec // path derived from content hash
|
||||
f, err := os.Open(path) //nolint:gosec // path derived from source key
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, ErrNotFound
|
||||
@@ -109,11 +80,13 @@ func (s *ContentStorage) Load(hash ContentHash) (io.ReadCloser, error) {
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// LoadWithSize returns a reader and file size for the content with the given hash.
|
||||
func (s *ContentStorage) LoadWithSize(hash ContentHash) (io.ReadCloser, int64, error) {
|
||||
path := s.hashToPath(hash)
|
||||
// LoadWithSize returns a reader and file size for the content with the given key.
|
||||
func (s *ContentStorage) LoadWithSize(
|
||||
key imagerequest.SourceKey,
|
||||
) (io.ReadCloser, int64, error) {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
f, err := os.Open(path) //nolint:gosec // path derived from content hash
|
||||
f, err := os.Open(path) //nolint:gosec // path derived from source key
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, 0, ErrNotFound
|
||||
@@ -132,9 +105,9 @@ func (s *ContentStorage) LoadWithSize(hash ContentHash) (io.ReadCloser, int64, e
|
||||
return f, stat.Size(), nil
|
||||
}
|
||||
|
||||
// Delete removes content with the given hash.
|
||||
func (s *ContentStorage) Delete(hash ContentHash) error {
|
||||
path := s.hashToPath(hash)
|
||||
// Delete removes content with the given key.
|
||||
func (s *ContentStorage) Delete(key imagerequest.SourceKey) error {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
err := os.Remove(path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
@@ -144,34 +117,14 @@ func (s *ContentStorage) Delete(hash ContentHash) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Exists checks if content with the given hash exists.
|
||||
func (s *ContentStorage) Exists(hash ContentHash) bool {
|
||||
path := s.hashToPath(hash)
|
||||
// Exists checks if content with the given key exists.
|
||||
func (s *ContentStorage) Exists(key imagerequest.SourceKey) bool {
|
||||
path := s.keyToPath(key)
|
||||
_, err := os.Stat(path)
|
||||
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// writeIfAbsent writes data with writeFileSynced to the path derived
|
||||
// from hash, unless a file of the data's size is already there.
|
||||
func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error {
|
||||
// Build path: <basedir>/<ab>/<cd>/<hash>
|
||||
path := s.hashToPath(hash)
|
||||
|
||||
// A file of another size there is damaged, and is replaced
|
||||
info, statErr := os.Stat(path)
|
||||
if statErr == nil && info.Size() == int64(len(data)) {
|
||||
return nil
|
||||
}
|
||||
|
||||
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create directory: %w", err)
|
||||
}
|
||||
|
||||
return writeFileSynced(path, data)
|
||||
}
|
||||
|
||||
// writeFileSynced writes data to path through a temp file in the same
|
||||
// directory: the temp file is synced to disk, renamed to path, and the
|
||||
// directory synced after, so that neither a concurrent reader nor a
|
||||
@@ -246,195 +199,19 @@ func syncDirectory(dir string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// hashToPath converts a hash to a file path: <basedir>/<ab>/<cd>/<hash>
|
||||
func (s *ContentStorage) hashToPath(hash ContentHash) string {
|
||||
h := string(hash)
|
||||
if len(h) < MinHashLength {
|
||||
return filepath.Clean(filepath.Join(s.baseDir, h))
|
||||
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
|
||||
func (s *ContentStorage) keyToPath(key imagerequest.SourceKey) string {
|
||||
k := string(key)
|
||||
if len(k) < MinHashLength {
|
||||
return filepath.Clean(filepath.Join(s.baseDir, k))
|
||||
}
|
||||
|
||||
return filepath.Clean(filepath.Join(s.baseDir, h[0:2], h[2:4], h))
|
||||
}
|
||||
|
||||
// MetadataStorage handles JSON metadata file storage.
|
||||
// Files are stored at: <basedir>/<hostname>/<path_hash>.json
|
||||
type MetadataStorage struct {
|
||||
baseDir string
|
||||
}
|
||||
|
||||
// NewMetadataStorage creates a new metadata storage at the given base directory.
|
||||
func NewMetadataStorage(baseDir string) (*MetadataStorage, error) {
|
||||
err := os.MkdirAll(baseDir, StorageDirPerm)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to create metadata directory: %w", err)
|
||||
}
|
||||
|
||||
return &MetadataStorage{baseDir: baseDir}, nil
|
||||
}
|
||||
|
||||
// SourceMetadata represents cached metadata about a source URL.
|
||||
//
|
||||
//nolint:tagliatelle // stored metadata format uses snake_case
|
||||
type SourceMetadata struct {
|
||||
Host string `json:"host"`
|
||||
Path string `json:"path"`
|
||||
Query string `json:"query,omitempty"`
|
||||
ContentHash string `json:"content_hash,omitempty"`
|
||||
StatusCode int `json:"status_code"`
|
||||
ContentType string `json:"content_type,omitempty"`
|
||||
ContentLength int64 `json:"content_length,omitempty"`
|
||||
ResponseHeaders map[string][]string `json:"response_headers,omitempty"`
|
||||
FetchedAt int64 `json:"fetched_at"`
|
||||
FetchDurationMs int64 `json:"fetch_duration_ms,omitempty"`
|
||||
ExpiresAt int64 `json:"expires_at,omitempty"`
|
||||
ETag string `json:"etag,omitempty"`
|
||||
LastModified string `json:"last_modified,omitempty"`
|
||||
RemoteAddr string `json:"remote_addr,omitempty"`
|
||||
}
|
||||
|
||||
// Store writes metadata to storage.
|
||||
func (s *MetadataStorage) Store(
|
||||
host string, pathHash PathHash, meta *SourceMetadata,
|
||||
) error {
|
||||
path := s.metaPath(host, pathHash)
|
||||
|
||||
// Create directory structure
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
err := os.MkdirAll(dir, StorageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create directory: %w", err)
|
||||
}
|
||||
|
||||
// Marshal to JSON
|
||||
data, err := json.MarshalIndent(meta, "", " ")
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to marshal metadata: %w", err)
|
||||
}
|
||||
|
||||
// Write to temp file first, then rename for atomicity
|
||||
tmpFile, err := os.CreateTemp(dir, ".tmp-*.json")
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create temp file: %w", err)
|
||||
}
|
||||
|
||||
tmpPath := tmpFile.Name()
|
||||
|
||||
_, err = tmpFile.Write(data)
|
||||
if err != nil {
|
||||
_ = tmpFile.Close()
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to write metadata: %w", err)
|
||||
}
|
||||
|
||||
err = tmpFile.Close()
|
||||
if err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to close temp file: %w", err)
|
||||
}
|
||||
|
||||
// Atomic rename
|
||||
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
|
||||
if err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
|
||||
return fmt.Errorf("failed to rename temp file: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Load reads metadata from storage.
|
||||
func (s *MetadataStorage) Load(
|
||||
host string, pathHash PathHash,
|
||||
) (*SourceMetadata, error) {
|
||||
path := s.metaPath(host, pathHash)
|
||||
|
||||
data, err := os.ReadFile(path) //nolint:gosec // path derived from host+hash
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("failed to read metadata: %w", err)
|
||||
}
|
||||
|
||||
var meta SourceMetadata
|
||||
|
||||
err = json.Unmarshal(data, &meta)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to unmarshal metadata: %w", err)
|
||||
}
|
||||
|
||||
return &meta, nil
|
||||
}
|
||||
|
||||
// Delete removes metadata for the given host and path hash.
|
||||
func (s *MetadataStorage) Delete(host string, pathHash PathHash) error {
|
||||
path := s.metaPath(host, pathHash)
|
||||
|
||||
err := os.Remove(path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("failed to delete metadata: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Exists checks if metadata exists for the given host and path hash.
|
||||
func (s *MetadataStorage) Exists(host string, pathHash PathHash) bool {
|
||||
path := s.metaPath(host, pathHash)
|
||||
_, err := os.Stat(path)
|
||||
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// metaPath returns the file path for metadata: <basedir>/<host>/<path_hash>.json
|
||||
func (s *MetadataStorage) metaPath(host string, pathHash PathHash) string {
|
||||
return filepath.Clean(filepath.Join(s.baseDir, host, string(pathHash)+".json"))
|
||||
}
|
||||
|
||||
// HashPath computes the SHA256 hash of a path string.
|
||||
func HashPath(path string) PathHash {
|
||||
h := sha256.Sum256([]byte(path))
|
||||
|
||||
return PathHash(hex.EncodeToString(h[:]))
|
||||
}
|
||||
|
||||
// CacheKey generates a unique key for a request variant.
|
||||
// Format: sha256(host:path:query:width:height:format:quality:fit_mode)
|
||||
// The path and query go through escapePercentAndColon, so that a ":" in
|
||||
// either cannot be taken for the one between them.
|
||||
func CacheKey(req *ImageRequest) VariantKey {
|
||||
data := fmt.Sprintf("%s:%s:%s:%d:%d:%s:%d:%s",
|
||||
req.SourceHost,
|
||||
escapePercentAndColon(req.SourcePath),
|
||||
escapePercentAndColon(req.SourceQuery),
|
||||
req.Size.Width,
|
||||
req.Size.Height,
|
||||
req.Format,
|
||||
req.Quality,
|
||||
req.FitMode,
|
||||
)
|
||||
h := sha256.Sum256([]byte(data))
|
||||
|
||||
return VariantKey(hex.EncodeToString(h[:]))
|
||||
}
|
||||
|
||||
// escapePercentAndColon returns s with each "%" written as "%25" and each
|
||||
// ":" as "%3A", so that the result holds no ":".
|
||||
func escapePercentAndColon(s string) string {
|
||||
s = strings.ReplaceAll(s, "%", "%25")
|
||||
|
||||
return strings.ReplaceAll(s, ":", "%3A")
|
||||
return filepath.Clean(filepath.Join(s.baseDir, k[0:2], k[2:4], k))
|
||||
}
|
||||
|
||||
// VariantStorage handles key-based file storage for processed image variants.
|
||||
// Files are stored at: <basedir>/<ab>/<cd>/<cache_key>
|
||||
// Metadata is stored at: <basedir>/<ab>/<cd>/<cache_key>.meta
|
||||
// Unlike ContentStorage, the key is provided by the caller (not computed from content).
|
||||
type VariantStorage struct {
|
||||
baseDir string
|
||||
log *slog.Logger
|
||||
@@ -464,7 +241,7 @@ func NewVariantStorage(baseDir string, log *slog.Logger) (*VariantStorage, error
|
||||
// with writeFileSynced. The .meta file is written first, so that a crash
|
||||
// cannot leave the content in place without it.
|
||||
func (s *VariantStorage) Store(
|
||||
key VariantKey, r io.Reader, contentType string,
|
||||
key imagerequest.VariantKey, r io.Reader, contentType string,
|
||||
) (int64, error) {
|
||||
data, err := io.ReadAll(r)
|
||||
if err != nil {
|
||||
@@ -508,7 +285,7 @@ func (s *VariantStorage) Store(
|
||||
}
|
||||
|
||||
// Load returns a reader for the content at the given key.
|
||||
func (s *VariantStorage) Load(key VariantKey) (io.ReadCloser, error) {
|
||||
func (s *VariantStorage) Load(key imagerequest.VariantKey) (io.ReadCloser, error) {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
f, err := os.Open(path) //nolint:gosec // path derived from cache key
|
||||
@@ -525,7 +302,9 @@ func (s *VariantStorage) Load(key VariantKey) (io.ReadCloser, error) {
|
||||
|
||||
// LoadWithSize returns a reader and file size for the content at the
|
||||
// given key.
|
||||
func (s *VariantStorage) LoadWithSize(key VariantKey) (io.ReadCloser, int64, error) {
|
||||
func (s *VariantStorage) LoadWithSize(
|
||||
key imagerequest.VariantKey,
|
||||
) (io.ReadCloser, int64, error) {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
f, err := os.Open(path) //nolint:gosec // path derived from cache key
|
||||
@@ -551,7 +330,7 @@ func (s *VariantStorage) LoadWithSize(key VariantKey) (io.ReadCloser, int64, err
|
||||
// the given key. The content type is read from the .meta file, and is
|
||||
// empty when that file is missing or unreadable.
|
||||
func (s *VariantStorage) LoadWithMeta(
|
||||
key VariantKey,
|
||||
key imagerequest.VariantKey,
|
||||
) (io.ReadCloser, int64, string, error) {
|
||||
f, size, err := s.LoadWithSize(key)
|
||||
if err != nil {
|
||||
@@ -574,7 +353,7 @@ func (s *VariantStorage) LoadWithMeta(
|
||||
}
|
||||
|
||||
// Exists checks if content exists at the given key.
|
||||
func (s *VariantStorage) Exists(key VariantKey) bool {
|
||||
func (s *VariantStorage) Exists(key imagerequest.VariantKey) bool {
|
||||
path := s.keyToPath(key)
|
||||
_, err := os.Stat(path)
|
||||
|
||||
@@ -583,7 +362,7 @@ func (s *VariantStorage) Exists(key VariantKey) bool {
|
||||
|
||||
// Delete removes the content at the given key together with its .meta
|
||||
// sidecar file. A missing file is not an error.
|
||||
func (s *VariantStorage) Delete(key VariantKey) error {
|
||||
func (s *VariantStorage) Delete(key imagerequest.VariantKey) error {
|
||||
path := s.keyToPath(key)
|
||||
|
||||
err := os.Remove(path)
|
||||
@@ -602,7 +381,7 @@ func (s *VariantStorage) Delete(key VariantKey) error {
|
||||
}
|
||||
|
||||
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
|
||||
func (s *VariantStorage) keyToPath(key VariantKey) string {
|
||||
func (s *VariantStorage) keyToPath(key imagerequest.VariantKey) string {
|
||||
k := string(key)
|
||||
if len(k) < MinHashLength {
|
||||
return filepath.Join(s.baseDir, k)
|
||||
|
||||
@@ -9,6 +9,8 @@ import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
func TestContentStorage_StoreAndLoad(t *testing.T) {
|
||||
@@ -22,8 +24,9 @@ func TestContentStorage_StoreAndLoad(t *testing.T) {
|
||||
}
|
||||
|
||||
content := []byte("hello world")
|
||||
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
|
||||
|
||||
hash, size, err := storage.Store(bytes.NewReader(content))
|
||||
size, err := storage.Store(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Store() error = %v", err)
|
||||
}
|
||||
@@ -32,14 +35,10 @@ func TestContentStorage_StoreAndLoad(t *testing.T) {
|
||||
t.Errorf("Store() size = %d, want %d", size, len(content))
|
||||
}
|
||||
|
||||
if hash == "" {
|
||||
t.Error("Store() returned empty hash")
|
||||
}
|
||||
|
||||
// Verify file exists at expected path
|
||||
hashStr := string(hash)
|
||||
keyStr := string(key)
|
||||
|
||||
expectedPath := filepath.Join(tmpDir, hashStr[0:2], hashStr[2:4], hashStr)
|
||||
expectedPath := filepath.Join(tmpDir, keyStr[0:2], keyStr[2:4], keyStr)
|
||||
|
||||
_, err = os.Stat(expectedPath)
|
||||
if err != nil {
|
||||
@@ -47,7 +46,7 @@ func TestContentStorage_StoreAndLoad(t *testing.T) {
|
||||
}
|
||||
|
||||
// Load and verify content
|
||||
r, err := storage.Load(hash)
|
||||
r, err := storage.Load(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
@@ -64,33 +63,6 @@ func TestContentStorage_StoreAndLoad(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestContentStorage_StoreIdempotent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewContentStorage(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewContentStorage() error = %v", err)
|
||||
}
|
||||
|
||||
content := []byte("same content")
|
||||
|
||||
hash1, _, err := storage.Store(bytes.NewReader(content))
|
||||
if err != nil {
|
||||
t.Fatalf("Store() first error = %v", err)
|
||||
}
|
||||
|
||||
hash2, _, err := storage.Store(bytes.NewReader(content))
|
||||
if err != nil {
|
||||
t.Fatalf("Store() second error = %v", err)
|
||||
}
|
||||
|
||||
if hash1 != hash2 {
|
||||
t.Errorf("Store() hashes differ: %s vs %s", hash1, hash2)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContentStorage_LoadNotFound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -101,7 +73,7 @@ func TestContentStorage_LoadNotFound(t *testing.T) {
|
||||
t.Fatalf("NewContentStorage() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = storage.Load(ContentHash("nonexistent"))
|
||||
_, err = storage.Load(imagerequest.SourceKey("nonexistent"))
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Load() error = %v, want ErrNotFound", err)
|
||||
}
|
||||
@@ -118,22 +90,23 @@ func TestContentStorage_Delete(t *testing.T) {
|
||||
}
|
||||
|
||||
content := []byte("to be deleted")
|
||||
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
|
||||
|
||||
hash, _, err := storage.Store(bytes.NewReader(content))
|
||||
_, err = storage.Store(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Store() error = %v", err)
|
||||
}
|
||||
|
||||
if !storage.Exists(hash) {
|
||||
if !storage.Exists(key) {
|
||||
t.Error("Exists() = false, want true")
|
||||
}
|
||||
|
||||
err = storage.Delete(hash)
|
||||
err = storage.Delete(key)
|
||||
if err != nil {
|
||||
t.Fatalf("Delete() error = %v", err)
|
||||
}
|
||||
|
||||
if storage.Exists(hash) {
|
||||
if storage.Exists(key) {
|
||||
t.Error("Exists() = true after delete, want false")
|
||||
}
|
||||
}
|
||||
@@ -149,7 +122,7 @@ func TestContentStorage_DeleteNonexistent(t *testing.T) {
|
||||
}
|
||||
|
||||
// Should not error
|
||||
err = storage.Delete(ContentHash("nonexistent"))
|
||||
err = storage.Delete(imagerequest.SourceKey("nonexistent"))
|
||||
if err != nil {
|
||||
t.Errorf("Delete() error = %v, want nil", err)
|
||||
}
|
||||
@@ -167,15 +140,16 @@ func TestContentStorage_HashToPath(t *testing.T) {
|
||||
|
||||
// Test by storing and verifying the resulting path structure
|
||||
content := []byte("test content for path verification")
|
||||
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
|
||||
|
||||
hash, _, err := storage.Store(bytes.NewReader(content))
|
||||
_, err = storage.Store(key, content)
|
||||
if err != nil {
|
||||
t.Fatalf("Store() error = %v", err)
|
||||
}
|
||||
|
||||
hashStr := string(hash)
|
||||
keyStr := string(key)
|
||||
|
||||
expectedPath := filepath.Join(tmpDir, hashStr[0:2], hashStr[2:4], hashStr)
|
||||
expectedPath := filepath.Join(tmpDir, keyStr[0:2], keyStr[2:4], keyStr)
|
||||
|
||||
_, err = os.Stat(expectedPath)
|
||||
if err != nil {
|
||||
@@ -183,286 +157,6 @@ func TestContentStorage_HashToPath(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetadataStorage_StoreAndLoad(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewMetadataStorage(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewMetadataStorage() error = %v", err)
|
||||
}
|
||||
|
||||
meta := &SourceMetadata{
|
||||
Host: testHostCDN,
|
||||
Path: testPathCat,
|
||||
ContentHash: "abc123",
|
||||
StatusCode: 200,
|
||||
ContentType: testContentTypeJPEG,
|
||||
FetchedAt: 1704067200,
|
||||
ETag: `"etag123"`,
|
||||
}
|
||||
|
||||
pathHash := HashPath(testPathCat)
|
||||
|
||||
err = storage.Store(testHostCDN, pathHash, meta)
|
||||
if err != nil {
|
||||
t.Fatalf("Store() error = %v", err)
|
||||
}
|
||||
|
||||
// Verify file exists at expected path
|
||||
expectedPath := filepath.Join(tmpDir, testHostCDN, string(pathHash)+".json")
|
||||
|
||||
_, err = os.Stat(expectedPath)
|
||||
if err != nil {
|
||||
t.Errorf("File not at expected path %s: %v", expectedPath, err)
|
||||
}
|
||||
|
||||
// Load and verify
|
||||
loaded, err := storage.Load(testHostCDN, pathHash)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
|
||||
if loaded.Host != meta.Host {
|
||||
t.Errorf("Host = %q, want %q", loaded.Host, meta.Host)
|
||||
}
|
||||
|
||||
if loaded.Path != meta.Path {
|
||||
t.Errorf("Path = %q, want %q", loaded.Path, meta.Path)
|
||||
}
|
||||
|
||||
if loaded.ContentHash != meta.ContentHash {
|
||||
t.Errorf("ContentHash = %q, want %q", loaded.ContentHash, meta.ContentHash)
|
||||
}
|
||||
|
||||
if loaded.StatusCode != meta.StatusCode {
|
||||
t.Errorf("StatusCode = %d, want %d", loaded.StatusCode, meta.StatusCode)
|
||||
}
|
||||
|
||||
if loaded.ETag != meta.ETag {
|
||||
t.Errorf("ETag = %q, want %q", loaded.ETag, meta.ETag)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetadataStorage_LoadNotFound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewMetadataStorage(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewMetadataStorage() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = storage.Load(testHostExample, PathHash("nonexistent"))
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Load() error = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetadataStorage_Delete(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
storage, err := NewMetadataStorage(tmpDir)
|
||||
if err != nil {
|
||||
t.Fatalf("NewMetadataStorage() error = %v", err)
|
||||
}
|
||||
|
||||
meta := &SourceMetadata{
|
||||
Host: testHostExample,
|
||||
Path: "/test.jpg",
|
||||
StatusCode: 200,
|
||||
}
|
||||
|
||||
pathHash := HashPath("/test.jpg")
|
||||
|
||||
err = storage.Store(testHostExample, pathHash, meta)
|
||||
if err != nil {
|
||||
t.Fatalf("Store() error = %v", err)
|
||||
}
|
||||
|
||||
if !storage.Exists(testHostExample, pathHash) {
|
||||
t.Error("Exists() = false, want true")
|
||||
}
|
||||
|
||||
err = storage.Delete(testHostExample, pathHash)
|
||||
if err != nil {
|
||||
t.Fatalf("Delete() error = %v", err)
|
||||
}
|
||||
|
||||
if storage.Exists(testHostExample, pathHash) {
|
||||
t.Error("Exists() = true after delete, want false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHashPath(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Same input should produce same hash
|
||||
hash1 := HashPath(testPathCat)
|
||||
hash2 := HashPath(testPathCat)
|
||||
|
||||
if hash1 != hash2 {
|
||||
t.Errorf("HashPath() not deterministic: %s vs %s", hash1, hash2)
|
||||
}
|
||||
|
||||
// Different input should produce different hash
|
||||
hash3 := HashPath("/photos/dog.jpg")
|
||||
|
||||
if hash1 == hash3 {
|
||||
t.Error("HashPath() produced same hash for different inputs")
|
||||
}
|
||||
|
||||
// Hash should be 64 hex chars (256 bits)
|
||||
if len(string(hash1)) != 64 {
|
||||
t.Errorf("HashPath() length = %d, want 64", len(string(hash1)))
|
||||
}
|
||||
}
|
||||
|
||||
func TestCacheKey(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req1 := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: testPathCat,
|
||||
SourceQuery: "",
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
}
|
||||
|
||||
req2 := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: testPathCat,
|
||||
SourceQuery: "",
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
}
|
||||
|
||||
// Same request should produce same key
|
||||
key1 := CacheKey(req1)
|
||||
key2 := CacheKey(req2)
|
||||
|
||||
if key1 != key2 {
|
||||
t.Errorf("CacheKey() not deterministic: %s vs %s", key1, key2)
|
||||
}
|
||||
|
||||
// Key should be 64 hex chars
|
||||
if len(string(key1)) != 64 {
|
||||
t.Errorf("CacheKey() length = %d, want 64", len(string(key1)))
|
||||
}
|
||||
|
||||
// Different size should produce different key
|
||||
req3 := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: testPathCat,
|
||||
SourceQuery: "",
|
||||
Size: Size{Width: 400, Height: 300}, // Different size
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
}
|
||||
|
||||
key3 := CacheKey(req3)
|
||||
if key1 == key3 {
|
||||
t.Error("CacheKey() produced same key for different sizes")
|
||||
}
|
||||
|
||||
// Different format should produce different key
|
||||
req4 := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: testPathCat,
|
||||
SourceQuery: "",
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatPNG, // Different format
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
}
|
||||
|
||||
key4 := CacheKey(req4)
|
||||
if key1 == key4 {
|
||||
t.Error("CacheKey() produced same key for different formats")
|
||||
}
|
||||
|
||||
// Different quality should produce different key
|
||||
req5 := &ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: testPathCat,
|
||||
SourceQuery: "",
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 50, // Different quality
|
||||
FitMode: FitCover,
|
||||
}
|
||||
|
||||
key5 := CacheKey(req5)
|
||||
if key1 == key5 {
|
||||
t.Error("CacheKey() produced same key for different quality")
|
||||
}
|
||||
}
|
||||
|
||||
// TestCacheKey_ColonOrPercentInPathOrQuery checks that two different source
|
||||
// URLs whose path or query holds ":" or "%" get different keys.
|
||||
func TestCacheKey_ColonOrPercentInPathOrQuery(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
path1, query1 string
|
||||
path2, query2 string
|
||||
}{
|
||||
{
|
||||
name: "colon moved from path to query",
|
||||
path1: "/a:b", query1: "",
|
||||
path2: "/a", query2: "b:",
|
||||
},
|
||||
{
|
||||
name: "colon in path and its percent-encoding",
|
||||
path1: "/a:b", query1: "",
|
||||
path2: "/a%3Ab", query2: "",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range cases {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
key1 := CacheKey(&ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: tt.path1,
|
||||
SourceQuery: tt.query1,
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
})
|
||||
key2 := CacheKey(&ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: tt.path2,
|
||||
SourceQuery: tt.query2,
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
})
|
||||
|
||||
t.Logf("CacheKey() of path %q query %q = %s", tt.path1, tt.query1, key1)
|
||||
t.Logf("CacheKey() of path %q query %q = %s", tt.path2, tt.query2, key2)
|
||||
|
||||
if key1 == key2 {
|
||||
t.Errorf("CacheKey() gave path %q query %q and path %q query %q "+
|
||||
"the same key", tt.path1, tt.query1, tt.path2, tt.query2)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestVariantStorage_StoreLogsFailedMetaWrite verifies that a .meta write
|
||||
// that fails is logged at warn and the store still succeeds.
|
||||
func TestVariantStorage_StoreLogsFailedMetaWrite(t *testing.T) {
|
||||
@@ -476,7 +170,9 @@ func TestVariantStorage_StoreLogsFailedMetaWrite(t *testing.T) {
|
||||
t.Fatalf("NewVariantStorage() error = %v", err)
|
||||
}
|
||||
|
||||
key := CacheKey(&ImageRequest{SourceHost: testHostCDN, SourcePath: testPathCat})
|
||||
key := imagerequest.CacheKey(&imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: testHostCDN, Path: testPathCat},
|
||||
})
|
||||
|
||||
// A directory where the .meta file goes makes the .meta write fail.
|
||||
err = os.MkdirAll(storage.keyToPath(key)+".meta", StorageDirPerm)
|
||||
@@ -497,7 +193,7 @@ func TestVariantStorage_StoreLogsFailedMetaWrite(t *testing.T) {
|
||||
|
||||
// storeTestVariant stores one variant, with its .meta file, in a new
|
||||
// VariantStorage and returns the storage and the variant's key.
|
||||
func storeTestVariant(t *testing.T) (*VariantStorage, VariantKey) {
|
||||
func storeTestVariant(t *testing.T) (*VariantStorage, imagerequest.VariantKey) {
|
||||
t.Helper()
|
||||
|
||||
storage, err := NewVariantStorage(t.TempDir(), slog.New(slog.DiscardHandler))
|
||||
@@ -505,7 +201,9 @@ func storeTestVariant(t *testing.T) (*VariantStorage, VariantKey) {
|
||||
t.Fatalf("NewVariantStorage() error = %v", err)
|
||||
}
|
||||
|
||||
key := CacheKey(&ImageRequest{SourceHost: testHostCDN, SourcePath: testPathCat})
|
||||
key := imagerequest.CacheKey(&imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: testHostCDN, Path: testPathCat},
|
||||
})
|
||||
|
||||
_, err = storage.Store(key, bytes.NewReader([]byte("variant data")), "image/webp")
|
||||
if err != nil {
|
||||
|
||||
@@ -9,7 +9,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestImageProxyRefusesComputedSideAboveMaximum requests sizes of a 3000x600
|
||||
@@ -70,9 +70,12 @@ func TestImageProxyRefusesComputedSideAboveMaximum(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
token, err := generator.Generate(&encurl.Payload{
|
||||
SourceHost: upstreamHost, SourcePath: tt.path,
|
||||
Width: tt.width, Height: tt.height, Format: imgcache.FormatJPEG,
|
||||
token, err := generator.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: upstreamHost, Path: tt.path},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: tt.width, Height: tt.height},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("encurl Generate: %v", err)
|
||||
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// TestImageProxyRefusesFitOutsideAboveMaximum requests sizes with fit=outside
|
||||
@@ -59,10 +59,13 @@ func TestImageProxyRefusesFitOutsideAboveMaximum(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
token, err := generator.Generate(&encurl.Payload{
|
||||
SourceHost: upstreamHost, SourcePath: tt.path,
|
||||
Width: tt.width, Height: tt.height, Format: imgcache.FormatJPEG,
|
||||
FitMode: imgcache.FitOutside,
|
||||
token, err := generator.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: upstreamHost, Path: tt.path},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: tt.width, Height: tt.height},
|
||||
Format: imagerequest.FormatJPEG,
|
||||
FitMode: imagerequest.FitOutside,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("encurl Generate: %v", err)
|
||||
|
||||
@@ -3,9 +3,7 @@ package server
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/jpeg"
|
||||
@@ -30,6 +28,7 @@ import (
|
||||
"sneak.berlin/go/pixa/internal/handlers"
|
||||
"sneak.berlin/go/pixa/internal/healthcheck"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/logger"
|
||||
"sneak.berlin/go/pixa/internal/middleware"
|
||||
)
|
||||
@@ -254,39 +253,26 @@ func getImage(t *testing.T, s *Server, target string) *httptest.ResponseRecorder
|
||||
return rec
|
||||
}
|
||||
|
||||
// checkSourceCached checks that source is stored under its SHA-256 in
|
||||
// cache/sources, recorded in source_content, and that the source URL's row in
|
||||
// source_metadata points at it.
|
||||
// checkSourceCached checks that source is stored in cache/sources under the
|
||||
// source key of upstreamHost and /photo.png, and recorded under that key in
|
||||
// source_content.
|
||||
func checkSourceCached(t *testing.T, db *sql.DB, stateDir string, source []byte) {
|
||||
t.Helper()
|
||||
|
||||
sum := sha256.Sum256(source)
|
||||
hash := hex.EncodeToString(sum[:])
|
||||
key := string(imagerequest.Source{Host: upstreamHost, Path: "/photo.png"}.Key())
|
||||
|
||||
checkFile(t, filepath.Join(stateDir, "cache", "sources", hash[0:2], hash[2:4], hash),
|
||||
checkFile(t, filepath.Join(stateDir, "cache", "sources", key[0:2], key[2:4], key),
|
||||
source)
|
||||
|
||||
var rows int
|
||||
|
||||
err := db.QueryRowContext(t.Context(),
|
||||
"SELECT COUNT(*) FROM source_content WHERE content_hash = ?", hash,
|
||||
"SELECT COUNT(*) FROM source_content WHERE source_key = ?", key,
|
||||
).Scan(&rows)
|
||||
if err != nil || rows != 1 {
|
||||
t.Errorf("source_content rows for the source = %d (error %v), want 1",
|
||||
rows, err)
|
||||
}
|
||||
|
||||
var metadataHash string
|
||||
|
||||
err = db.QueryRowContext(t.Context(),
|
||||
`SELECT content_hash FROM source_metadata
|
||||
WHERE source_host = ? AND source_path = ?`,
|
||||
upstreamHost, "/photo.png",
|
||||
).Scan(&metadataHash)
|
||||
if err != nil || metadataHash != hash {
|
||||
t.Errorf("source_metadata content_hash = %q (error %v), want %q",
|
||||
metadataHash, err, hash)
|
||||
}
|
||||
}
|
||||
|
||||
// checkVariantCached checks that the converted image served is recorded in
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/session"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
@@ -513,12 +514,16 @@ func signedImageURL(t *testing.T, sourceHost string) string {
|
||||
t.Helper()
|
||||
|
||||
path, sig, exp := signature.New(testSigningKey).GenerateSignedURL(
|
||||
&signature.Request{
|
||||
SourceHost: sourceHost,
|
||||
SourcePath: ipv6SourcePath,
|
||||
Format: string(imgcache.FormatPNG),
|
||||
Quality: 85,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
&imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: sourceHost,
|
||||
Path: ipv6SourcePath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Format: imagerequest.FormatPNG,
|
||||
Quality: 85,
|
||||
FitMode: imagerequest.FitCover,
|
||||
},
|
||||
}, time.Hour)
|
||||
|
||||
return path + "?sig=" + sig + "&exp=" + strconv.FormatInt(exp, 10)
|
||||
@@ -534,9 +539,9 @@ func encryptedImageURL(t *testing.T, sourceHost string) string {
|
||||
t.Fatalf("encurl.NewGenerator: %v", err)
|
||||
}
|
||||
|
||||
token, err := generator.Generate(&encurl.Payload{
|
||||
SourceHost: sourceHost, SourcePath: ipv6SourcePath,
|
||||
Format: imgcache.FormatPNG,
|
||||
token, err := generator.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: sourceHost, Path: ipv6SourcePath},
|
||||
Transform: imagerequest.Transform{Format: imagerequest.FormatPNG},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("encurl Generate: %v", err)
|
||||
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// svgServedAsAVIF is an SVG whose first bytes pass the first-bytes check for
|
||||
@@ -38,10 +38,13 @@ func TestImageProxyRefusesSVGServedAsAVIF(t *testing.T) {
|
||||
t.Fatalf("encurl.NewGenerator: %v", err)
|
||||
}
|
||||
|
||||
encryptedURL := func(format imgcache.ImageFormat) string {
|
||||
token, err := generator.Generate(&encurl.Payload{
|
||||
SourceHost: upstreamHost, SourcePath: "/drawing.avif",
|
||||
Width: 32, Height: 24, Format: format,
|
||||
encryptedURL := func(format imagerequest.Format) string {
|
||||
token, err := generator.Generate(&imagerequest.Request{
|
||||
Source: imagerequest.Source{Host: upstreamHost, Path: "/drawing.avif"},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 32, Height: 24},
|
||||
Format: format,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("encurl Generate: %v", err)
|
||||
@@ -53,8 +56,8 @@ func TestImageProxyRefusesSVGServedAsAVIF(t *testing.T) {
|
||||
for _, target := range []string{
|
||||
"/v1/image/" + upstreamHost + "/drawing.avif/32x24.png",
|
||||
"/v1/image/" + upstreamHost + "/drawing.avif/64x48.orig",
|
||||
encryptedURL(imgcache.FormatPNG),
|
||||
encryptedURL(imgcache.FormatOriginal),
|
||||
encryptedURL(imagerequest.FormatPNG),
|
||||
encryptedURL(imagerequest.FormatOriginal),
|
||||
} {
|
||||
rec := httptest.NewRecorder()
|
||||
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
@@ -16,32 +17,35 @@ func TestSigner_Verify_ColonCannotMovePathQueryBoundary(t *testing.T) {
|
||||
|
||||
signer := signature.New(goldenSigningKey)
|
||||
|
||||
signed := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: "/a:b",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
Expires: time.Now().Add(1 * time.Hour),
|
||||
signed := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: "/a:b",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
},
|
||||
Expires: time.Now().Add(1 * time.Hour),
|
||||
}
|
||||
signed.Signature = signer.Sign(signed)
|
||||
|
||||
replayed := *signed
|
||||
replayed.SourcePath = "/a"
|
||||
replayed.SourceQuery = "b:"
|
||||
replayed.Source.Path = "/a"
|
||||
replayed.Source.Query = "b:"
|
||||
|
||||
t.Logf("signature for path %q query %q: %s",
|
||||
signed.SourcePath, signed.SourceQuery, signed.Signature)
|
||||
signed.Source.Path, signed.Source.Query, signed.Signature)
|
||||
t.Logf("signature for path %q query %q: %s",
|
||||
replayed.SourcePath, replayed.SourceQuery, signer.Sign(&replayed))
|
||||
replayed.Source.Path, replayed.Source.Query, signer.Sign(&replayed))
|
||||
|
||||
err := signer.Verify(&replayed)
|
||||
if !errors.Is(err, signature.ErrInvalid) {
|
||||
t.Errorf("Verify() of path %q query %q with the signature for "+
|
||||
"path %q = %v, want %v",
|
||||
replayed.SourcePath, replayed.SourceQuery, signed.SourcePath,
|
||||
replayed.Source.Path, replayed.Source.Query, signed.Source.Path,
|
||||
err, signature.ErrInvalid)
|
||||
}
|
||||
}
|
||||
@@ -90,16 +94,19 @@ func TestSigner_EscapedGoldenVectors(t *testing.T) {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: tt.path,
|
||||
SourceQuery: tt.query,
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
Expires: time.Unix(goldenExpiresUnix, 0),
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: tt.path,
|
||||
Query: tt.query,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
},
|
||||
Expires: time.Unix(goldenExpiresUnix, 0),
|
||||
}
|
||||
|
||||
gotSignature := signer.Sign(req)
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
@@ -16,7 +17,7 @@ const goldenSigningKey = "golden-test-key"
|
||||
|
||||
type goldenVector struct {
|
||||
name string
|
||||
req signature.Request
|
||||
req imagerequest.Request
|
||||
// wantSignature is the exact base64url (RFC 4648 URL-safe,
|
||||
// padded) HMAC-SHA256 signature for the request with Expires
|
||||
// set to goldenExpiresUnix.
|
||||
@@ -34,15 +35,18 @@ func goldenVectors() []goldenVector {
|
||||
return []goldenVector{
|
||||
{
|
||||
name: "resized without query",
|
||||
req: signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
SourceQuery: "",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
req: imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
Query: "",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
},
|
||||
},
|
||||
// Signed data:
|
||||
// "cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200:85:cover"
|
||||
@@ -51,15 +55,18 @@ func goldenVectors() []goldenVector {
|
||||
},
|
||||
{
|
||||
name: "resized with query string",
|
||||
req: signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
SourceQuery: "token=abc&v=2",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
req: imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
Query: "token=abc&v=2",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
},
|
||||
},
|
||||
// Signed data:
|
||||
// "cdn.example.com:/photos/cat.jpg:token=abc&v=2:800:600:webp:1704067200:85:cover"
|
||||
@@ -69,15 +76,18 @@ func goldenVectors() []goldenVector {
|
||||
},
|
||||
{
|
||||
name: "original size without query",
|
||||
req: signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
SourceQuery: "",
|
||||
Width: 0,
|
||||
Height: 0,
|
||||
Format: testFormatPNG,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
req: imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
Query: "",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 0, Height: 0},
|
||||
Format: testFormatPNG,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
},
|
||||
},
|
||||
// Signed data:
|
||||
// "cdn.example.com:/photos/cat.jpg::0:0:png:1704067200:85:cover"
|
||||
@@ -86,15 +96,18 @@ func goldenVectors() []goldenVector {
|
||||
},
|
||||
{
|
||||
name: "non-default quality and fit",
|
||||
req: signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
SourceQuery: "",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Quality: 40,
|
||||
FitMode: testFitContain,
|
||||
req: imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
Query: "",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
Quality: 40,
|
||||
FitMode: testFitContain,
|
||||
},
|
||||
},
|
||||
// Signed data:
|
||||
// "cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200:40:contain"
|
||||
|
||||
@@ -5,22 +5,26 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
// signedQualityFitRequest returns a request signed for quality 85 and fit
|
||||
// mode "cover", the effective defaults the handler applies before
|
||||
// verification.
|
||||
func signedQualityFitRequest(signer *signature.Signer) *signature.Request {
|
||||
req := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
Expires: time.Now().Add(1 * time.Hour),
|
||||
func signedQualityFitRequest(signer *signature.Signer) *imagerequest.Request {
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
},
|
||||
Expires: time.Now().Add(1 * time.Hour),
|
||||
}
|
||||
req.Signature = signer.Sign(req)
|
||||
|
||||
@@ -40,15 +44,15 @@ func TestSigner_Verify_QualityAndFitAreSigned(t *testing.T) {
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
tamper func(r *signature.Request)
|
||||
tamper func(r *imagerequest.Request)
|
||||
}{
|
||||
{
|
||||
name: "replayed with different quality",
|
||||
tamper: func(r *signature.Request) { r.Quality = 40 },
|
||||
tamper: func(r *imagerequest.Request) { r.Transform.Quality = 40 },
|
||||
},
|
||||
{
|
||||
name: "replayed with different fit mode",
|
||||
tamper: func(r *signature.Request) { r.FitMode = testFitContain },
|
||||
tamper: func(r *imagerequest.Request) { r.Transform.FitMode = testFitContain },
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -12,6 +12,8 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
)
|
||||
|
||||
// Signature errors.
|
||||
@@ -22,37 +24,6 @@ var (
|
||||
ErrMissingExpiration = errors.New("signature expiration is required")
|
||||
)
|
||||
|
||||
// Request carries the components an image request signature covers. It is a
|
||||
// standalone type so that this package does not depend on imgcache, keeping
|
||||
// the import edge one-way (imgcache depends on signature, never the reverse).
|
||||
type Request struct {
|
||||
// SourceHost is the origin host (e.g. "cdn.example.com").
|
||||
SourceHost string
|
||||
// SourcePath is the path on the origin (e.g. "/photos/cat.jpg").
|
||||
SourcePath string
|
||||
// SourceQuery is the optional query string for the origin URL.
|
||||
SourceQuery string
|
||||
// Width is the requested output width in pixels.
|
||||
Width int
|
||||
// Height is the requested output height in pixels.
|
||||
Height int
|
||||
// Format is the requested output format (e.g. "webp").
|
||||
Format string
|
||||
// Quality is the requested output quality (1-100) for lossy formats.
|
||||
// It is the effective value the request resolves to: callers pass the
|
||||
// default quality when the request omits the parameter, so an omitted
|
||||
// quality signs identically to that same value stated explicitly.
|
||||
Quality int
|
||||
// FitMode is how the image is fit into the requested dimensions
|
||||
// (e.g. "cover"). Like Quality it is the effective value: callers pass
|
||||
// the default fit mode when the request omits the parameter.
|
||||
FitMode string
|
||||
// Signature is the HMAC signature to verify.
|
||||
Signature string
|
||||
// Expires is the signature expiration timestamp.
|
||||
Expires time.Time
|
||||
}
|
||||
|
||||
// Signer handles HMAC-SHA256 signature generation and verification.
|
||||
type Signer struct {
|
||||
secretKey []byte
|
||||
@@ -67,8 +38,11 @@ func New(secretKey string) *Signer {
|
||||
|
||||
// Sign generates an HMAC-SHA256 signature for the given request.
|
||||
// The signature covers: host + path + query + width + height + format +
|
||||
// expiration + quality + fit.
|
||||
func (s *Signer) Sign(req *Request) string {
|
||||
// expiration + quality + fit. The quality and fit mode are the effective
|
||||
// values the request resolves to: callers pass the defaults when the request
|
||||
// omits those parameters, so an omitted quality or fit mode signs
|
||||
// identically to that same value stated explicitly.
|
||||
func (s *Signer) Sign(req *imagerequest.Request) string {
|
||||
data := s.buildSignatureData(req)
|
||||
mac := hmac.New(sha256.New, s.secretKey)
|
||||
mac.Write([]byte(data))
|
||||
@@ -84,7 +58,7 @@ func (s *Signer) Sign(req *Request) string {
|
||||
// No suffix matching, wildcard matching, or partial matching is supported.
|
||||
// A signature for "cdn.example.com" will NOT verify for "example.com" or
|
||||
// "other.cdn.example.com", and vice versa.
|
||||
func (s *Signer) Verify(req *Request) error {
|
||||
func (s *Signer) Verify(req *imagerequest.Request) error {
|
||||
// Check expiration first
|
||||
if req.Expires.IsZero() {
|
||||
return ErrMissingExpiration
|
||||
@@ -108,7 +82,7 @@ func (s *Signer) Verify(req *Request) error {
|
||||
// GenerateSignedURL creates a complete URL with signature and expiration.
|
||||
// Returns the path portion that should be appended to the base URL.
|
||||
func (s *Signer) GenerateSignedURL(
|
||||
req *Request, ttl time.Duration,
|
||||
req *imagerequest.Request, ttl time.Duration,
|
||||
) (string, string, int64) {
|
||||
// Set expiration
|
||||
req.Expires = time.Now().Add(ttl)
|
||||
@@ -120,10 +94,11 @@ func (s *Signer) GenerateSignedURL(
|
||||
|
||||
// Build the size component
|
||||
var sizeStr string
|
||||
if req.Width == 0 && req.Height == 0 {
|
||||
if req.Transform.Size.Width == 0 && req.Transform.Size.Height == 0 {
|
||||
sizeStr = "orig"
|
||||
} else {
|
||||
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
|
||||
sizeStr = fmt.Sprintf("%dx%d",
|
||||
req.Transform.Size.Width, req.Transform.Size.Height)
|
||||
}
|
||||
|
||||
// Build the path. A source query follows the source path after a "?".
|
||||
@@ -131,13 +106,13 @@ func (s *Signer) GenerateSignedURL(
|
||||
// written in Go's own percent-encoding of a URL path, which writes "?"
|
||||
// as %3F and "%" as %25, and leaves "/", ";", ",", "=" and "&" as they
|
||||
// are.
|
||||
sourcePathAndQuery := req.SourcePath
|
||||
if req.SourceQuery != "" {
|
||||
sourcePathAndQuery += "?" + req.SourceQuery
|
||||
sourcePathAndQuery := req.Source.Path
|
||||
if req.Source.Query != "" {
|
||||
sourcePathAndQuery += "?" + req.Source.Query
|
||||
}
|
||||
|
||||
unescapedPath := fmt.Sprintf("/v1/image/%s%s/%s.%s",
|
||||
req.SourceHost, sourcePathAndQuery, sizeStr, req.Format)
|
||||
req.Source.Host, sourcePathAndQuery, sizeStr, req.Transform.Format)
|
||||
|
||||
return (&url.URL{Path: unescapedPath}).EscapedPath(), sig, exp
|
||||
}
|
||||
@@ -150,17 +125,17 @@ func (s *Signer) GenerateSignedURL(
|
||||
// expansion is performed. Quality and fit are the effective transform
|
||||
// values, so replaying a signed URL with a different quality or fit mode
|
||||
// fails verification.
|
||||
func (s *Signer) buildSignatureData(req *Request) string {
|
||||
func (s *Signer) buildSignatureData(req *imagerequest.Request) string {
|
||||
return fmt.Sprintf("%s:%s:%s:%d:%d:%s:%d:%d:%s",
|
||||
req.SourceHost,
|
||||
escapePercentAndColon(req.SourcePath),
|
||||
escapePercentAndColon(req.SourceQuery),
|
||||
req.Width,
|
||||
req.Height,
|
||||
req.Format,
|
||||
req.Source.Host,
|
||||
escapePercentAndColon(req.Source.Path),
|
||||
escapePercentAndColon(req.Source.Query),
|
||||
req.Transform.Size.Width,
|
||||
req.Transform.Size.Height,
|
||||
req.Transform.Format,
|
||||
req.Expires.Unix(),
|
||||
req.Quality,
|
||||
req.FitMode,
|
||||
req.Transform.Quality,
|
||||
req.Transform.FitMode,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imagerequest"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
@@ -26,14 +27,17 @@ func TestSigner_Sign(t *testing.T) {
|
||||
|
||||
signer := signature.New("test-secret-key")
|
||||
|
||||
req := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
SourceQuery: "",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Expires: time.Unix(1704067200, 0), // Fixed timestamp for reproducibility
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
Query: "",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
},
|
||||
Expires: time.Unix(1704067200, 0), // Fixed timestamp for reproducibility
|
||||
}
|
||||
|
||||
sig1 := signer.Sign(req)
|
||||
@@ -51,14 +55,17 @@ func TestSigner_Sign(t *testing.T) {
|
||||
}
|
||||
|
||||
// Different input should produce different signature
|
||||
req2 := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: "/photos/dog.jpg", // Different path
|
||||
SourceQuery: "",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Expires: time.Unix(1704067200, 0),
|
||||
req2 := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: "/photos/dog.jpg", // Different path
|
||||
Query: "",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
},
|
||||
Expires: time.Unix(1704067200, 0),
|
||||
}
|
||||
|
||||
sig3 := signer.Sign(req2)
|
||||
@@ -69,20 +76,23 @@ func TestSigner_Sign(t *testing.T) {
|
||||
|
||||
// validVerifyRequest returns a fully-populated request that verifies
|
||||
// successfully once signed.
|
||||
func validVerifyRequest() *signature.Request {
|
||||
return &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Expires: time.Now().Add(1 * time.Hour),
|
||||
func validVerifyRequest() *imagerequest.Request {
|
||||
return &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
},
|
||||
Expires: time.Now().Add(1 * time.Hour),
|
||||
}
|
||||
}
|
||||
|
||||
type verifyCase struct {
|
||||
name string
|
||||
setup func() *signature.Request
|
||||
setup func() *imagerequest.Request
|
||||
wantErr error
|
||||
}
|
||||
|
||||
@@ -90,7 +100,7 @@ func verifyCases(signer *signature.Signer) []verifyCase {
|
||||
return []verifyCase{
|
||||
{
|
||||
name: "valid signature",
|
||||
setup: func() *signature.Request {
|
||||
setup: func() *imagerequest.Request {
|
||||
req := validVerifyRequest()
|
||||
req.Signature = signer.Sign(req)
|
||||
|
||||
@@ -100,7 +110,7 @@ func verifyCases(signer *signature.Signer) []verifyCase {
|
||||
},
|
||||
{
|
||||
name: "expired signature",
|
||||
setup: func() *signature.Request {
|
||||
setup: func() *imagerequest.Request {
|
||||
req := validVerifyRequest()
|
||||
req.Expires = time.Now().Add(-1 * time.Hour)
|
||||
req.Signature = signer.Sign(req)
|
||||
@@ -111,7 +121,7 @@ func verifyCases(signer *signature.Signer) []verifyCase {
|
||||
},
|
||||
{
|
||||
name: "invalid signature",
|
||||
setup: func() *signature.Request {
|
||||
setup: func() *imagerequest.Request {
|
||||
req := validVerifyRequest()
|
||||
req.Signature = "invalid-signature"
|
||||
|
||||
@@ -121,7 +131,7 @@ func verifyCases(signer *signature.Signer) []verifyCase {
|
||||
},
|
||||
{
|
||||
name: "missing expiration",
|
||||
setup: func() *signature.Request {
|
||||
setup: func() *imagerequest.Request {
|
||||
req := validVerifyRequest()
|
||||
req.Expires = time.Time{}
|
||||
req.Signature = "some-signature"
|
||||
@@ -132,10 +142,10 @@ func verifyCases(signer *signature.Signer) []verifyCase {
|
||||
},
|
||||
{
|
||||
name: "tampered request",
|
||||
setup: func() *signature.Request {
|
||||
setup: func() *imagerequest.Request {
|
||||
req := validVerifyRequest()
|
||||
req.Signature = signer.Sign(req)
|
||||
req.SourcePath = "/photos/secret.jpg"
|
||||
req.Source.Path = "/photos/secret.jpg"
|
||||
|
||||
return req
|
||||
},
|
||||
@@ -173,7 +183,7 @@ func TestSigner_Verify(t *testing.T) {
|
||||
|
||||
type tamperCase struct {
|
||||
name string
|
||||
tamper func(r *signature.Request)
|
||||
tamper func(r *imagerequest.Request)
|
||||
}
|
||||
|
||||
// exactMatchTamperCases mutates one signed component per case; every
|
||||
@@ -182,59 +192,59 @@ func exactMatchTamperCases() []tamperCase {
|
||||
return []tamperCase{
|
||||
{
|
||||
name: "parent domain does not match subdomain",
|
||||
tamper: func(r *signature.Request) { r.SourceHost = "example.com" },
|
||||
tamper: func(r *imagerequest.Request) { r.Source.Host = "example.com" },
|
||||
},
|
||||
{
|
||||
name: "subdomain does not match parent domain",
|
||||
tamper: func(r *signature.Request) { r.SourceHost = "images.cdn.example.com" },
|
||||
tamper: func(r *imagerequest.Request) { r.Source.Host = "images.cdn.example.com" },
|
||||
},
|
||||
{
|
||||
name: "sibling subdomain does not match",
|
||||
tamper: func(r *signature.Request) { r.SourceHost = "images.example.com" },
|
||||
tamper: func(r *imagerequest.Request) { r.Source.Host = "images.example.com" },
|
||||
},
|
||||
{
|
||||
name: "host with suffix appended does not match",
|
||||
tamper: func(r *signature.Request) { r.SourceHost = testHost + ".evil.com" },
|
||||
tamper: func(r *imagerequest.Request) { r.Source.Host = testHost + ".evil.com" },
|
||||
},
|
||||
{
|
||||
name: "host with prefix does not match",
|
||||
tamper: func(r *signature.Request) { r.SourceHost = "evilcdn.example.com" },
|
||||
tamper: func(r *imagerequest.Request) { r.Source.Host = "evilcdn.example.com" },
|
||||
},
|
||||
{
|
||||
name: "different path does not match",
|
||||
tamper: func(r *signature.Request) { r.SourcePath = "/photos/dog.jpg" },
|
||||
tamper: func(r *imagerequest.Request) { r.Source.Path = "/photos/dog.jpg" },
|
||||
},
|
||||
{
|
||||
name: "path suffix does not match",
|
||||
tamper: func(r *signature.Request) { r.SourcePath = testPath + "/extra" },
|
||||
tamper: func(r *imagerequest.Request) { r.Source.Path = testPath + "/extra" },
|
||||
},
|
||||
{
|
||||
name: "path prefix does not match",
|
||||
tamper: func(r *signature.Request) { r.SourcePath = "/other" + testPath },
|
||||
tamper: func(r *imagerequest.Request) { r.Source.Path = "/other" + testPath },
|
||||
},
|
||||
{
|
||||
name: "different query does not match",
|
||||
tamper: func(r *signature.Request) { r.SourceQuery = "token=xyz" },
|
||||
tamper: func(r *imagerequest.Request) { r.Source.Query = "token=xyz" },
|
||||
},
|
||||
{
|
||||
name: "added query does not match empty query",
|
||||
tamper: func(r *signature.Request) { r.SourceQuery = "extra=1" },
|
||||
tamper: func(r *imagerequest.Request) { r.Source.Query = "extra=1" },
|
||||
},
|
||||
{
|
||||
name: "removed query does not match",
|
||||
tamper: func(r *signature.Request) { r.SourceQuery = "" },
|
||||
tamper: func(r *imagerequest.Request) { r.Source.Query = "" },
|
||||
},
|
||||
{
|
||||
name: "different width does not match",
|
||||
tamper: func(r *signature.Request) { r.Width = 801 },
|
||||
tamper: func(r *imagerequest.Request) { r.Transform.Size.Width = 801 },
|
||||
},
|
||||
{
|
||||
name: "different height does not match",
|
||||
tamper: func(r *signature.Request) { r.Height = 601 },
|
||||
tamper: func(r *imagerequest.Request) { r.Transform.Size.Height = 601 },
|
||||
},
|
||||
{
|
||||
name: "different format does not match",
|
||||
tamper: func(r *signature.Request) { r.Format = testFormatPNG },
|
||||
tamper: func(r *imagerequest.Request) { r.Transform.Format = testFormatPNG },
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -248,15 +258,18 @@ func TestSigner_Verify_ExactMatchOnly(t *testing.T) {
|
||||
signer := signature.New("test-secret-key")
|
||||
|
||||
// Base request that we'll sign, then tamper with individual fields.
|
||||
baseReq := func() *signature.Request {
|
||||
req := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
SourceQuery: "token=abc",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Expires: time.Now().Add(1 * time.Hour),
|
||||
baseReq := func() *imagerequest.Request {
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
Query: "token=abc",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
},
|
||||
Expires: time.Now().Add(1 * time.Hour),
|
||||
}
|
||||
req.Signature = signer.Sign(req)
|
||||
|
||||
@@ -309,14 +322,17 @@ func TestSigner_Sign_ExactHostInData(t *testing.T) {
|
||||
sigs := make(map[string]string)
|
||||
|
||||
for _, host := range hosts {
|
||||
req := &signature.Request{
|
||||
SourceHost: host,
|
||||
SourcePath: testPath,
|
||||
SourceQuery: "",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Expires: time.Unix(1704067200, 0),
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: host,
|
||||
Path: testPath,
|
||||
Query: "",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
},
|
||||
Expires: time.Unix(1704067200, 0),
|
||||
}
|
||||
|
||||
sig := signer.Sign(req)
|
||||
@@ -334,13 +350,16 @@ func TestSigner_DifferentKeys(t *testing.T) {
|
||||
signer1 := signature.New("secret-key-1")
|
||||
signer2 := signature.New("secret-key-2")
|
||||
|
||||
req := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Expires: time.Now().Add(1 * time.Hour),
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
},
|
||||
Expires: time.Now().Add(1 * time.Hour),
|
||||
}
|
||||
|
||||
// Sign with key 1
|
||||
@@ -364,13 +383,16 @@ func TestGenerateSignedURL(t *testing.T) {
|
||||
|
||||
signer := signature.New("test-secret-key")
|
||||
|
||||
req := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
SourceQuery: "",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
Query: "",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
},
|
||||
}
|
||||
|
||||
ttl := 1 * time.Hour
|
||||
@@ -405,12 +427,15 @@ func TestGenerateSignedURL_OrigSize(t *testing.T) {
|
||||
|
||||
signer := signature.New("test-secret-key")
|
||||
|
||||
req := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
Width: 0, // Original size
|
||||
Height: 0,
|
||||
Format: testFormatPNG,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 0, Height: 0}, // Original size
|
||||
Format: testFormatPNG,
|
||||
},
|
||||
}
|
||||
|
||||
path, _, _ := signer.GenerateSignedURL(req, time.Hour)
|
||||
@@ -426,13 +451,16 @@ func TestGenerateSignedURL_WithQueryString(t *testing.T) {
|
||||
|
||||
signer := signature.New("test-secret-key-for-testing!")
|
||||
|
||||
req := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
SourceQuery: "token=abc&v=2",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
Query: "token=abc&v=2",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
},
|
||||
}
|
||||
|
||||
path, _, _ := signer.GenerateSignedURL(req, time.Hour)
|
||||
@@ -460,12 +488,15 @@ func TestGenerateSignedURL_WithoutQueryString(t *testing.T) {
|
||||
|
||||
signer := signature.New("test-secret-key-for-testing!")
|
||||
|
||||
req := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: testPath,
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
req := &imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: testPath,
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
},
|
||||
}
|
||||
|
||||
path, _, _ := signer.GenerateSignedURL(req, time.Hour)
|
||||
@@ -485,16 +516,19 @@ func TestGenerateSignedURL_ReadmeQueryExample(t *testing.T) {
|
||||
|
||||
signer := signature.New("example-signing-key-for-documentation")
|
||||
|
||||
req := signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: "/img",
|
||||
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
Expires: time.Unix(1704067200, 0),
|
||||
req := imagerequest.Request{
|
||||
Source: imagerequest.Source{
|
||||
Host: testHost,
|
||||
Path: "/img",
|
||||
Query: "url=https://origin.example.com/my%20cat.jpg",
|
||||
},
|
||||
Transform: imagerequest.Transform{
|
||||
Size: imagerequest.Size{Width: 800, Height: 600},
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
},
|
||||
Expires: time.Unix(1704067200, 0),
|
||||
}
|
||||
|
||||
const wantSig = "37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA="
|
||||
|
||||
+3
-3
@@ -1,8 +1,8 @@
|
||||
#!/bin/sh
|
||||
# script/docker-smoke: build the Docker image, start it, and wait up to
|
||||
# 30 seconds for its HEALTHCHECK to report healthy. Needs a Docker
|
||||
# daemon, so it is not part of script/check; the Gitea workflow runs it
|
||||
# after script/cibuild.
|
||||
# 30 seconds for its HEALTHCHECK to report healthy. It is not part of
|
||||
# script/check, which never starts the image, and the Gitea workflow
|
||||
# does not run it. Run it by hand.
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||
|
||||
Reference in New Issue
Block a user