sneak/pixa
1
0
Fork 0
Code Issues 51 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

base: sneak:next
Branches Tags
sneak:main sneak:next sneak:feat/upaas-deployment-setup sneak:remove-whitelist-suffix-matching sneak:fix/remove-whitelist-suffix-matching sneak:fix/20-split-dockerfile sneak:ci/make-check sneak:chore/remove-stale-files sneak:fix/issue-1 sneak:fix/issue-3 sneak:fix/issue-4
..
compare: sneak:71782f2e9b7c170d3738619770c23ab458ec7a53
Branches Tags
sneak:next sneak:main sneak:feat/upaas-deployment-setup sneak:remove-whitelist-suffix-matching sneak:fix/remove-whitelist-suffix-matching sneak:fix/20-split-dockerfile sneak:ci/make-check sneak:chore/remove-stale-files sneak:fix/issue-1 sneak:fix/issue-3 sneak:fix/issue-4

3 Commits
next ... 71782f2e9b

Author SHA1 Message Date
sneak
71782f2e9b docs: record startup config validation in TODO.md (closes #52)
Some checks failed
check / check (push) Has been cancelled
Details
2026-08-07 16:37:15 +00:00
sneak
f4c7dc4cd7 feat: validate configuration on startup, fail fast on bad config (#52) 
A config value that is set but unparseable or invalid now aborts
startup with an error naming the offending key and value; defaults
apply only to omitted keys. Unknown top-level config keys and unknown
metrics subkeys abort startup naming each unknown key, so typos like
whitelist_hosts fail immediately instead of being silently ignored. A
config file that exists at a standard location but fails to parse is
now a fatal error instead of being skipped with a warning. state_dir
is verified creatable and writable with a probe file before the
listener binds. Port must be in 1-65535 (fractional values are
rejected, not truncated), upstream_connections_per_host must be at
least 1, allowlist_hosts entries must be bare hostnames, sentry_dsn
must be a valid URL when set, and metrics credentials must be set
together. The stale signing_key comment in config.example.yml (keyless
mode was never implemented) now states the actual requirement.
 2026-08-07 16:36:34 +00:00
sneak
f19da2c02c test: add failing startup config validation tests (#52) 
Encode the required fail-fast behavior as tests ahead of the
implementation: a config value that is SET but unparseable or invalid
must abort startup (defaults apply only to OMITTED keys), unknown
top-level keys and unknown metrics subkeys must abort naming the key,
a malformed config file at a standard location must abort instead of
being skipped with a warning, and state_dir must be creatable and
writable at startup. Mechanically extracts newFromSmartConfig from
config.New so the construction path is testable without fx; current
lenient behavior is unchanged, so the new enforcement tests fail.
 2026-08-07 16:31:03 +00:00
66 changed files with 2148 additions and 6022 deletions
Expand all files Collapse all files

133
.golangci.yml
View File

@@ -1,34 +1,117 @@
version: "2"
# Config schema uses the golangci-lint v2 layout (settings live under
# linters.settings, not top-level linters-settings) so that the
# thresholds below are actually applied by golangci-lint >= v2.
run:
timeout: 5m
modules-download-mode: readonly
go: "1.24"
tests: false
linters:
default: all
disable:
# Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields
- depguard # Dependency allow/block lists
- godot # Requires comments to end with periods
- wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
settings:
lll:
line-length: 88
funlen:
lines: 80
statements: 50
cyclop:
max-complexity: 15
dupl:
threshold: 100
enable:
# Additional linters requested
- testifylint # Checks usage of github.com/stretchr/testify
- usetesting # usetesting is an analyzer that detects using os.Setenv instead of t.Setenv since Go 1.17
# - tagliatelle # Disabled: we need snake_case for external API compatibility
- nlreturn # nlreturn checks for a new line before return and branch statements
- nilnil # Checks that there is no simultaneous return of nil error and an invalid value
- nestif # Reports deeply nested if statements
- mnd # An analyzer to detect magic numbers
- lll # Reports long lines
- intrange # intrange is a linter to find places where for loops could make use of an integer range
- gochecknoglobals # Check that no global variables exist
# Default/existing linters that are commonly useful
- govet
- errcheck
- staticcheck
- unused
- ineffassign
- misspell
- revive
- gosec
- unconvert
- unparam
linters-settings:
lll:
line-length: 120
nestif:
min-complexity: 4
nlreturn:
block-size: 2
revive:
rules:
- name: var-naming
arguments:
- []
- []
- "upperCaseConst=true"
tagliatelle:
case:
rules:
json: snake
yaml: snake
xml: snake
bson: snake
testifylint:
enable-all: true
usetesting: {}
issues:
max-issues-per-linter: 0
max-same-issues: 0
exclude-rules:
# Exclude unused parameter warnings for cobra command signatures
- text: "parameter '(args|cmd)' seems to be unused"
linters:
- revive
# Allow ALL_CAPS constant names
- text: "don't use ALL_CAPS in Go names"
linters:
- revive
# Allow snake_case JSON tags for external API compatibility
- path: "internal/types/ris.go"
linters:
- tagliatelle
# Allow snake_case JSON tags for database models
- path: "internal/database/models.go"
linters:
- tagliatelle
# Allow generic package name for types that define data structures
- path: "internal/types/"
text: "avoid meaningless package names"
linters:
- revive
# Allow globals in the globals package (by design)
- path: "internal/globals/"
linters:
- gochecknoglobals
# Allow globals in main (Version/Buildarch set by ldflags)
- path: "cmd/"
linters:
- gochecknoglobals
# Allow blank imports for driver registration
- text: "blank-imports"
linters:
- revive
# Allow unused fx.Lifecycle parameters (required by fx signature)
- text: "parameter 'lc' seems to be unused"
linters:
- revive
# Allow unused context parameters in fx hooks
- text: "parameter 'ctx' seems to be unused"
linters:
- revive

4
Dockerfile
View File

@@ -1,6 +1,6 @@
# Lint stage
# golangci/golangci-lint:v2.12.2-alpine, 2026-08-07
FROM golangci/golangci-lint:v2.12.2-alpine@sha256:91b27804074a0bacea298707f016911e60cf0cdbc6c7bf5ccacb5f0606d18d60 AS lint
# golangci/golangci-lint:v2.10.1-alpine, 2026-02-17
FROM golangci/golangci-lint:v2.10.1-alpine@sha256:33bc6b6156d4c7da87175f187090019769903d04dd408833b83083ed214b0ddf AS lint
RUN apk add --no-cache make build-base vips-dev libheif-dev pkgconfig

3
README.md
View File

@@ -115,9 +115,6 @@ Configured via YAML file (`--config`). Key settings:
- `upstream_max_response_size` — max origin response size
- `downstream_timeout` — client response timeout
- `signing_key` — HMAC secret for URL signatures
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the
disk cache entirely; omitted defaults to 75% of the free space on
the filesystem containing `<state_dir>/cache/` (minimum 500 MiB)
See `config.example.yml` for all options with defaults.

79
TODO.md
View File

@@ -12,84 +12,27 @@
pre-1.0. No git tags exist. Recent work extracted the internal/magic,
internal/allowlist, internal/httpfetcher, and internal/signature
packages. The gosec findings from the 2026-07-06 survey are resolved
and `make check` is green on main. The disk cache is now size-bounded
with LRU eviction (`cache_max_bytes`), closing the unbounded disk
growth DoS vector.
packages. The gosec findings from the 2026-07-06 survey are resolved:
the last two open findings (G124, session cookie attributes in
internal/session) are fixed as of this change, so `make check` is green
on main.
# Next Step
P1: implement blocked networks configuration to extend SSRF protection
P0: manual test pass of the auth and encrypted URL flows, then commit
the checked-off results to TODO.md: visit / and see the login form;
wrong key shows an error; correct signing key shows the generator form;
a generated encrypted URL serves the image; an expired URL (short TTL)
returns 410; logout redirects back to login
# Completed Steps
- 2026-08-07 update golangci-lint to v2.12.2 with the canonical
`.golangci.yml` (v2 schema, `default: all` minus six disabled
linters, `lll` 88, tests included): bumped the pinned
`golangci/golangci-lint:v2.12.2-alpine` image in `Dockerfile` and the
release-archive sha256 pins in `script/bootstrap`; fixed the findings
the stricter config surfaced (notably `paralleltest`, `wsl_v5`,
`goconst`, `lll`, `noinlineerr`, `err113`, `errcheck`, `testpackage` —
white-box test files renamed to `*_internal_test.go`), including #55's
code absorbed after it merged, iterating the pinned linter to
`0 issues.`; no single finding total is substantiable, since
golangci-lint's `uniq-by-line` reveals new findings on a line as
others there are fixed — the documented re-measurements were 81 after
the #53 merge and 149 after the #55 merge; three behavior changes, so
not a pure no-op: `Cache.StoreVariant` now takes a `context.Context`
(`noctx`), so a cancelled request skips its best-effort accounting
row; `MetadataStorage.Store`'s cleanup defer was dead on `main` and
leaked `.tmp-*.json` on failure, now fixed with explicit removals; and
the `signing_key` validation error text gained `value too short: `;
the eviction loop's uncancellable context is deferred to #102 under a
`//nolint:contextcheck`; three `//nolint:tagliatelle` directives keep
the snake_case JSON wire/disk formats unchanged; `make check` green
- 2026-08-07 implement cache size management and eviction (closes
#51): new `cache_max_bytes` config key validated by the startup
framework (explicit values used exactly with no floor, `0` disables
the disk cache entirely, omitted defaults to max(75% of free space
on the filesystem containing `<state_dir>/cache/`, 500 MiB), logged
at startup); processed variants are now tracked in the database (a
new `variant_content` table and an LRU timestamp on `source_content`)
so total usage is two SUMs, never a directory scan on the hot path; a
background goroutine evicts globally least-recently-used entries
(variants and source blobs merged) to the limit, woken by a periodic
ticker and by write-pressure notifications from stores; a source
blob and ALL of its `source_metadata` references are deleted in one
transaction before the file is unlinked, so multi-referenced blobs
are never removed while referenced and rows never point at deleted
files; a startup and periodic reconciliation pass adopts untracked
variant files, drops rows for missing files, removes unreachable
source blobs, and sweeps stale temp files
- 2026-08-07 validate configuration on startup, fail fast on bad
config (closes #52): a config value that is set but unparseable or
invalid aborts startup naming the key and value (defaults apply only
to omitted keys), unknown config keys abort startup, a malformed
config file aborts instead of being skipped, and `state_dir` is
verified creatable and writable before the listener binds
- 2026-08-07 manual test pass of the auth and encrypted URL flows
against a locally built and running `pixad` (built from `main` at
`6573b9d`, port 18099, local throwaway config); all six checks
passed, plus all nine tests in `scripts/manual-test.sh` (closes #49):
- [x] visit `/` and see the login form: HTTP 200, `Pixa - Login`
page with `name="key"` password form
- [x] wrong key shows an error: POST `/` with `key=wrong-key`
returned HTTP 200 login page containing "Invalid signing key"
- [x] correct signing key shows the generator form: POST `/`
returned HTTP 303 to `/` with
`Set-Cookie: pixa_session=...; HttpOnly; Secure; SameSite=Strict`;
GET `/` with that cookie rendered `Pixa - URL Generator` with the
`/generate` form and logout link
- [x] a generated encrypted URL serves the image: POST `/generate`
(ttl=3600) produced a `/v1/e/<token>/img.jpeg` URL that returned
HTTP 200, `Content-Type: image/jpeg`, an 800x600 baseline JPEG of
61706 bytes
- [x] an expired URL (short TTL) returns 410: a ttl=1 URL fetched
after 3 s returned HTTP 410 Gone with
`{"error":"URL has expired","status":410,...}`
- [x] logout redirects back to login: GET `/logout` returned HTTP
303 to `/` with `Set-Cookie: pixa_session=; Max-Age=0`;
subsequent GET `/` rendered the login form again
- 2026-08-07 fix the two remaining gosec findings (G124 in
internal/session): session cookies now always carry
Secure/HttpOnly/SameSite=Strict on both the set and clear paths;
@@ -116,6 +59,10 @@ P1: implement blocked networks configuration to extend SSRF protection
# Future Steps
- P0: implement cache size management and eviction so the disk cannot
fill up
- P1: implement blocked networks configuration to extend SSRF
protection
- P1: rate limit global concurrent upstream fetches to prevent
resource exhaustion
- P1: strip EXIF and other metadata from processed images (privacy)

3
cmd/pixad/main.go
View File

@@ -30,8 +30,7 @@ func main() {
rootCmd.Flags().StringVarP(&configPath, "config", "c", "", "path to config file")
err := rootCmd.Execute()
if err != nil {
if err := rootCmd.Execute(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}

7
config.example.yml
View File

@@ -28,13 +28,6 @@ allow_http: false
# Maximum concurrent connections per upstream host (default: 20)
upstream_connections_per_host: 20
# Maximum disk cache size in bytes. Explicit values are used exactly as
# given; 0 disables the disk cache entirely (every request fetches and
# processes uncached). When omitted, the default is 75% of the free
# space on the filesystem containing <state_dir>/cache/ at startup,
# with a minimum of 500 MiB.
# cache_max_bytes: 10737418240
# Sentry error reporting (optional)
sentry_dsn: ""

3
internal/allowlist/allowlist.go
View File

@@ -10,8 +10,7 @@ import (
type HostAllowList struct {
// exactHosts contains hosts that must match exactly (e.g., "cdn.example.com")
exactHosts map[string]struct{}
// suffixHosts contains domain suffixes to match
// (e.g., ".example.com" matches "cdn.example.com")
// suffixHosts contains domain suffixes to match (e.g., ".example.com" matches "cdn.example.com")
suffixHosts []string
}

209
internal/allowlist/allowlist_test.go
View File

@@ -7,37 +7,104 @@ import (
"sneak.berlin/go/pixa/internal/allowlist"
)
const (
testExactHost = "cdn.example.com"
testImageURL = "https://cdn.example.com/image.jpg"
testSuffix = ".example.com"
)
type isAllowedCase struct {
name string
patterns []string
testURL string
want bool
}
func runIsAllowedCases(t *testing.T, tests []isAllowedCase) {
t.Helper()
func TestHostAllowList_IsAllowed(t *testing.T) {
tests := []struct {
name string
patterns []string
testURL string
want bool
}{
{
name: "exact match",
patterns: []string{"cdn.example.com"},
testURL: "https://cdn.example.com/image.jpg",
want: true,
},
{
name: "exact match case insensitive",
patterns: []string{"CDN.Example.COM"},
testURL: "https://cdn.example.com/image.jpg",
want: true,
},
{
name: "exact match not found",
patterns: []string{"cdn.example.com"},
testURL: "https://other.example.com/image.jpg",
want: false,
},
{
name: "suffix match",
patterns: []string{".example.com"},
testURL: "https://cdn.example.com/image.jpg",
want: true,
},
{
name: "suffix match deep subdomain",
patterns: []string{".example.com"},
testURL: "https://cdn.images.example.com/image.jpg",
want: true,
},
{
name: "suffix match apex domain",
patterns: []string{".example.com"},
testURL: "https://example.com/image.jpg",
want: true,
},
{
name: "suffix match not found",
patterns: []string{".example.com"},
testURL: "https://notexample.com/image.jpg",
want: false,
},
{
name: "suffix match partial not allowed",
patterns: []string{".example.com"},
testURL: "https://fakeexample.com/image.jpg",
want: false,
},
{
name: "multiple patterns",
patterns: []string{"cdn.example.com", ".images.org", "static.test.net"},
testURL: "https://photos.images.org/image.jpg",
want: true,
},
{
name: "empty allow list",
patterns: []string{},
testURL: "https://cdn.example.com/image.jpg",
want: false,
},
{
name: "nil url",
patterns: []string{"cdn.example.com"},
testURL: "",
want: false,
},
{
name: "url with port",
patterns: []string{"cdn.example.com"},
testURL: "https://cdn.example.com:443/image.jpg",
want: true,
},
{
name: "whitespace in patterns",
patterns: []string{" cdn.example.com ", " .other.com "},
testURL: "https://cdn.example.com/image.jpg",
want: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
w := allowlist.New(tt.patterns)
var u *url.URL
if tt.testURL != "" {
parsed, err := url.Parse(tt.testURL)
var err error
u, err = url.Parse(tt.testURL)
if err != nil {
t.Fatalf("failed to parse test URL: %v", err)
}
u = parsed
}
got := w.IsAllowed(u)
@@ -48,101 +115,7 @@ func runIsAllowedCases(t *testing.T, tests []isAllowedCase) {
}
}
func TestHostAllowList_IsAllowed_ExactMatch(t *testing.T) {
t.Parallel()
runIsAllowedCases(t, []isAllowedCase{
{
name: "exact match",
patterns: []string{testExactHost},
testURL: testImageURL,
want: true,
},
{
name: "exact match case insensitive",
patterns: []string{"CDN.Example.COM"},
testURL: testImageURL,
want: true,
},
{
name: "exact match not found",
patterns: []string{testExactHost},
testURL: "https://other.example.com/image.jpg",
want: false,
},
{
name: "multiple patterns",
patterns: []string{testExactHost, ".images.org", "static.test.net"},
testURL: "https://photos.images.org/image.jpg",
want: true,
},
{
name: "empty allow list",
patterns: []string{},
testURL: testImageURL,
want: false,
},
{
name: "nil url",
patterns: []string{testExactHost},
testURL: "",
want: false,
},
{
name: "url with port",
patterns: []string{testExactHost},
testURL: "https://cdn.example.com:443/image.jpg",
want: true,
},
{
name: "whitespace in patterns",
patterns: []string{" cdn.example.com ", " .other.com "},
testURL: testImageURL,
want: true,
},
})
}
func TestHostAllowList_IsAllowed_SuffixMatch(t *testing.T) {
t.Parallel()
runIsAllowedCases(t, []isAllowedCase{
{
name: "suffix match",
patterns: []string{testSuffix},
testURL: testImageURL,
want: true,
},
{
name: "suffix match deep subdomain",
patterns: []string{testSuffix},
testURL: "https://cdn.images.example.com/image.jpg",
want: true,
},
{
name: "suffix match apex domain",
patterns: []string{testSuffix},
testURL: "https://example.com/image.jpg",
want: true,
},
{
name: "suffix match not found",
patterns: []string{testSuffix},
testURL: "https://notexample.com/image.jpg",
want: false,
},
{
name: "suffix match partial not allowed",
patterns: []string{testSuffix},
testURL: "https://fakeexample.com/image.jpg",
want: false,
},
})
}
func TestHostAllowList_IsEmpty(t *testing.T) {
t.Parallel()
tests := []struct {
name string
patterns []string
@@ -172,8 +145,6 @@ func TestHostAllowList_IsEmpty(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
w := allowlist.New(tt.patterns)
if got := w.IsEmpty(); got != tt.want {
t.Errorf("IsEmpty() = %v, want %v", got, tt.want)
@@ -183,8 +154,6 @@ func TestHostAllowList_IsEmpty(t *testing.T) {
}
func TestHostAllowList_Count(t *testing.T) {
t.Parallel()
tests := []struct {
name string
patterns []string
@@ -214,8 +183,6 @@ func TestHostAllowList_Count(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
w := allowlist.New(tt.patterns)
if got := w.Count(); got != tt.want {
t.Errorf("Count() = %v, want %v", got, tt.want)

305
internal/config/cache_max_bytes_internal_test.go
View File

@@ -1,305 +0,0 @@
package config
import (
"errors"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
)
// Static errors returned by the stub free-space probes below.
var (
errTestStatfsFailed = errors.New("statfs failed")
errTestProbeNotExpected = errors.New("probe must not be called")
)
// discardLogger returns a logger that swallows all output, for tests
// that exercise code paths which log.
func discardLogger() *slog.Logger {
return slog.New(slog.DiscardHandler)
}
// TestCacheMaxBytesExplicitValueUsedWithoutFloor verifies that an
// explicitly configured cache_max_bytes value is used exactly as
// given: the 500 MiB floor applies only to the computed default, never
// to explicit values.
func TestCacheMaxBytesExplicitValueUsedWithoutFloor(t *testing.T) {
t.Parallel()
yamlContent := "signing_key: " + validTestSigningKey + "\ncache_max_bytes: 1024\n"
c, err := configFromYAML(t, yamlContent)
if err != nil {
t.Fatalf("explicit cache_max_bytes must be accepted, got error: %v", err)
}
if c.CacheMaxBytes != 1024 {
t.Errorf("CacheMaxBytes = %d, want 1024 (no floor for explicit values)",
c.CacheMaxBytes)
}
}
// TestCacheMaxBytesZeroIsValidAndDisablesCache verifies that an
// explicit zero is a valid value (it disables the disk cache), not an
// error.
func TestCacheMaxBytesZeroIsValidAndDisablesCache(t *testing.T) {
t.Parallel()
yamlContent := "signing_key: " + validTestSigningKey + "\ncache_max_bytes: 0\n"
c, err := configFromYAML(t, yamlContent)
if err != nil {
t.Fatalf("cache_max_bytes: 0 must be accepted, got error: %v", err)
}
if c.CacheMaxBytes != 0 {
t.Errorf("CacheMaxBytes = %d, want 0", c.CacheMaxBytes)
}
}
// TestCacheMaxBytesLargeExplicitValueParses verifies that values above
// 32-bit range parse correctly (the field is an int64 byte count).
func TestCacheMaxBytesLargeExplicitValueParses(t *testing.T) {
t.Parallel()
yamlContent := "signing_key: " + validTestSigningKey +
"\ncache_max_bytes: 10737418240\n"
c, err := configFromYAML(t, yamlContent)
if err != nil {
t.Fatalf("large cache_max_bytes must be accepted, got error: %v", err)
}
if c.CacheMaxBytes != 10737418240 {
t.Errorf("CacheMaxBytes = %d, want 10737418240", c.CacheMaxBytes)
}
}
// TestCacheMaxBytesInvalidValuesAbortStartup verifies that a SET but
// invalid cache_max_bytes value aborts startup naming the key and the
// offending value, per the no-silent-fallback rule: defaults apply
// only to omitted keys.
func TestCacheMaxBytesInvalidValuesAbortStartup(t *testing.T) {
t.Parallel()
signingKeyLine := "signing_key: " + validTestSigningKey + "\n"
cases := []struct {
name string
yaml string
// wantErrSubstrings must all appear in the error message.
wantErrSubstrings []string
}{
{
name: "negative",
yaml: signingKeyLine + "cache_max_bytes: -1024\n",
wantErrSubstrings: []string{keyCacheMaxBytes, "-1024"},
},
{
name: "float",
yaml: signingKeyLine + "cache_max_bytes: 3.5\n",
wantErrSubstrings: []string{keyCacheMaxBytes, "3.5"},
},
{
name: "non-numeric string",
yaml: signingKeyLine + "cache_max_bytes: banana\n",
wantErrSubstrings: []string{keyCacheMaxBytes, "banana"},
},
{
name: "explicit null",
yaml: signingKeyLine + "cache_max_bytes: null\n",
wantErrSubstrings: []string{keyCacheMaxBytes, "null"},
},
{
name: "bare key no value",
yaml: signingKeyLine + "cache_max_bytes:\n",
wantErrSubstrings: []string{keyCacheMaxBytes, "null"},
},
{
name: "boolean",
yaml: signingKeyLine + "cache_max_bytes: true\n",
wantErrSubstrings: []string{keyCacheMaxBytes, "true"},
},
{
name: "list",
yaml: signingKeyLine + "cache_max_bytes:\n - 1\n",
wantErrSubstrings: []string{keyCacheMaxBytes},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, tc.yaml)
if err == nil {
t.Fatalf(
"config with %s cache_max_bytes must abort startup, got config: %+v",
tc.name, c)
}
t.Logf("got expected error: %v", err)
for _, want := range tc.wantErrSubstrings {
if !strings.Contains(err.Error(), want) {
t.Errorf("error %q does not mention %q", err.Error(), want)
}
}
})
}
}
// TestComputeDefaultCacheMaxBytesUses75PercentOfFreeSpace verifies the
// computed default is 75% of the probed free space when that exceeds
// the floor.
func TestComputeDefaultCacheMaxBytesUses75PercentOfFreeSpace(t *testing.T) {
t.Parallel()
// 4 GiB free -> 3 GiB default.
probe := func(string) (uint64, error) { return 4294967296, nil }
got, err := ComputeDefaultCacheMaxBytes(t.TempDir(), probe)
if err != nil {
t.Fatalf("ComputeDefaultCacheMaxBytes returned error: %v", err)
}
if got != 3221225472 {
t.Errorf("ComputeDefaultCacheMaxBytes = %d, want 3221225472 (75%% of 4 GiB)",
got)
}
}
// TestComputeDefaultCacheMaxBytesAppliesFloorToComputedDefault
// verifies that when 75% of free space is below 500 MiB, the computed
// default is floored at DefaultCacheMaxBytesFloor.
func TestComputeDefaultCacheMaxBytesAppliesFloorToComputedDefault(t *testing.T) {
t.Parallel()
cases := []struct {
name string
freeBytes uint64
}{
{name: "100 MiB free", freeBytes: 104857600},
{name: "zero free", freeBytes: 0},
{name: "just below floor threshold", freeBytes: 699050665},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
probe := func(string) (uint64, error) { return tc.freeBytes, nil }
got, err := ComputeDefaultCacheMaxBytes(t.TempDir(), probe)
if err != nil {
t.Fatalf("ComputeDefaultCacheMaxBytes returned error: %v", err)
}
if got != DefaultCacheMaxBytesFloor {
t.Errorf("ComputeDefaultCacheMaxBytes = %d, want floor %d",
got, DefaultCacheMaxBytesFloor)
}
})
}
}
// TestComputeDefaultCacheMaxBytesPropagatesProbeError verifies that a
// failing free-space probe produces an error naming the config key,
// instead of a silently wrong default.
func TestComputeDefaultCacheMaxBytesPropagatesProbeError(t *testing.T) {
t.Parallel()
probe := func(string) (uint64, error) { return 0, errTestStatfsFailed }
_, err := ComputeDefaultCacheMaxBytes(t.TempDir(), probe)
if err == nil {
t.Fatal("probe failure must produce an error, got nil")
}
t.Logf("got expected error: %v", err)
if !strings.Contains(err.Error(), keyCacheMaxBytes) {
t.Errorf("error %q does not name the config key cache_max_bytes", err.Error())
}
}
// TestResolveCacheMaxBytesComputesDefaultWhenOmitted verifies that an
// omitted cache_max_bytes key resolves to the computed default, that
// the probe is pointed at <state_dir>/cache/ (which must be created
// first so statfs measures the right filesystem), and that the result
// lands on the Config.
func TestResolveCacheMaxBytesComputesDefaultWhenOmitted(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, "signing_key: "+validTestSigningKey+"\n")
if err != nil {
t.Fatalf("minimal config should be valid, got error: %v", err)
}
c.StateDir = t.TempDir()
wantCacheDir := filepath.Join(c.StateDir, "cache")
var probedPath string
// 4 GiB free -> 3 GiB default.
probe := func(path string) (uint64, error) {
probedPath = path
return 4294967296, nil
}
err = c.resolveCacheMaxBytes(discardLogger(), probe)
if err != nil {
t.Fatalf("resolveCacheMaxBytes returned error: %v", err)
}
if c.CacheMaxBytes != 3221225472 {
t.Errorf("CacheMaxBytes = %d, want computed default 3221225472",
c.CacheMaxBytes)
}
if probedPath != wantCacheDir {
t.Errorf("free space probed at %q, want cache directory %q",
probedPath, wantCacheDir)
}
info, err := os.Stat(wantCacheDir)
if err != nil || !info.IsDir() {
t.Errorf("cache directory %q was not created before probing: info=%v err=%v",
wantCacheDir, info, err)
}
}
// TestResolveCacheMaxBytesDoesNotOverrideExplicitValue verifies that
// an explicitly configured value survives resolution untouched and
// that the free-space probe is never consulted for it.
func TestResolveCacheMaxBytesDoesNotOverrideExplicitValue(t *testing.T) {
t.Parallel()
yamlContent := "signing_key: " + validTestSigningKey + "\ncache_max_bytes: 1024\n"
c, err := configFromYAML(t, yamlContent)
if err != nil {
t.Fatalf("explicit cache_max_bytes must be accepted, got error: %v", err)
}
c.StateDir = t.TempDir()
probe := func(string) (uint64, error) {
t.Error("free-space probe must not be consulted for explicit values")
return 0, errTestProbeNotExpected
}
err = c.resolveCacheMaxBytes(discardLogger(), probe)
if err != nil {
t.Fatalf("resolveCacheMaxBytes returned error: %v", err)
}
if c.CacheMaxBytes != 1024 {
t.Errorf("CacheMaxBytes = %d, want explicit 1024 (no floor, no recompute)",
c.CacheMaxBytes)
}
}

116
internal/config/cachesize.go
View File

@@ -1,116 +0,0 @@
package config
import (
"fmt"
"log/slog"
"math"
"os"
"path/filepath"
"syscall"
)
// DefaultCacheMaxBytesFloor is the minimum computed default for the
// cache_max_bytes setting: 500 MiB. The floor applies only to the
// computed default (when the key is omitted from the configuration),
// never to explicitly configured values.
const DefaultCacheMaxBytesFloor int64 = 524288000
// cacheDirPerms is the permission mode for the cache directory created
// before probing free space, matching the state directory permissions.
const cacheDirPerms = 0o750
// freeSpaceFractionNumerator and freeSpaceFractionDenominator express
// the 75% share of free space used for the computed default limit as
// integer arithmetic (dividing before multiplying avoids overflow).
const (
freeSpaceFractionNumerator uint64 = 3
freeSpaceFractionDenominator uint64 = 4
)
// FreeSpaceProbeFunc reports the number of free bytes available on the
// filesystem containing path. It is a function type so tests can
// inject a fake probe instead of depending on the host disk.
type FreeSpaceProbeFunc func(path string) (uint64, error)
// defaultFreeSpaceProbe reports free filesystem bytes via statfs on
// the given path, as available to unprivileged processes.
func defaultFreeSpaceProbe(path string) (uint64, error) {
var stat syscall.Statfs_t
err := syscall.Statfs(path, &stat)
if err != nil {
return 0, err
}
if stat.Bsize < 0 {
return 0, fmt.Errorf("%w %d for %q", errNegativeBlockSize, stat.Bsize, path)
}
blockSize := uint64(stat.Bsize)
return stat.Bavail * blockSize, nil
}
// ComputeDefaultCacheMaxBytes returns the default cache size limit for
// the filesystem containing cacheDir: 75% of the free bytes reported
// by probe, with a floor of DefaultCacheMaxBytesFloor.
func ComputeDefaultCacheMaxBytes(
cacheDir string, probe FreeSpaceProbeFunc,
) (int64, error) {
freeBytes, err := probe(cacheDir)
if err != nil {
return 0, fmt.Errorf("config key %q: cannot determine free space for %q: %w",
"cache_max_bytes", cacheDir, err)
}
computed := freeBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
computed = min(computed, math.MaxInt64)
// gosec cannot see that min() above bounds computed, so it reads
// this conversion as potentially overflowing. It cannot: computed is
// at most math.MaxInt64 on every path here.
//nolint:gosec // G115: clamped to MaxInt64 by min above
limit := int64(computed)
limit = max(limit, DefaultCacheMaxBytesFloor)
return limit, nil
}
// resolveCacheMaxBytes finalizes CacheMaxBytes after state_dir
// validation: an explicitly configured value is kept as-is (no floor
// applies), while an omitted key receives the computed default based
// on free space in <state_dir>/cache/. The cache directory is created
// first so statfs measures the filesystem that will actually hold the
// cache. The effective limit is logged either way.
func (c *Config) resolveCacheMaxBytes(
log *slog.Logger, probe FreeSpaceProbeFunc,
) error {
if !c.cacheMaxBytesExplicit {
cacheDir := filepath.Join(c.StateDir, "cache")
err := os.MkdirAll(cacheDir, cacheDirPerms)
if err != nil {
return fmt.Errorf("config key %q: cannot create cache directory %q: %w",
keyCacheMaxBytes, cacheDir, err)
}
limit, err := ComputeDefaultCacheMaxBytes(cacheDir, probe)
if err != nil {
return err
}
c.CacheMaxBytes = limit
log.Info("computed default cache size limit from free space",
"cache_max_bytes", limit,
"cache_dir", cacheDir,
)
}
log.Info("effective cache size limit",
"cache_max_bytes", c.CacheMaxBytes,
"cache_disabled", c.CacheMaxBytes == 0,
)
return nil
}

425
internal/config/config.go
View File

@@ -2,7 +2,6 @@
package config
import (
"errors"
"fmt"
"log/slog"
"math"
@@ -26,59 +25,9 @@ const (
DefaultUpstreamConnectionsPerHost = 20
)
// Configuration key names.
const (
keyDebug = "debug"
keyMaintenanceMode = "maintenance_mode"
keyPort = "port"
keyStateDir = "state_dir"
keySentryDSN = "sentry_dsn"
keyDBURL = "db_url"
keyMetrics = "metrics"
keyMetricsUsername = "metrics.username"
keyMetricsPassword = "metrics.password"
keySigningKey = "signing_key"
keyAllowlistHosts = "allowlist_hosts"
keyAllowHTTP = "allow_http"
keyUpstreamConnectionsPerHost = "upstream_connections_per_host"
keyCacheMaxBytes = "cache_max_bytes"
)
// Static validation errors. Each use site attaches the offending key
// and value by wrapping these with fmt.Errorf and %w.
var (
errValueRequired = errors.New("a value is required")
errValueEmpty = errors.New("value must not be empty")
errUnknownConfigKeys = errors.New("unknown config keys")
errNotAString = errors.New("not a string")
errNotAnInteger = errors.New("not an integer")
errNotABoolean = errors.New("not a boolean")
errNotAStringList = errors.New("not a list of strings")
errNotAMetricsMap = errors.New("not a map of metrics settings")
errEmptyListEntry = errors.New("list contains an empty entry")
errEmptyEntry = errors.New("contains an empty entry")
errNotAValidURL = errors.New("not a valid URL")
errPortOutOfRange = errors.New("outside the valid port range")
errTooFewConnections = errors.New("must be at least 1")
errValueTooShort = errors.New("value too short")
errMustBeSetTogether = errors.New("must be set together")
errMustNotBeNegative = errors.New("must not be negative")
errOverflowsInt64 = errors.New("overflows a 64-bit integer")
errNegativeBlockSize = errors.New(
"statfs reported negative block size")
errValueNull = errors.New(
"value is null; omit the key entirely to use the default")
errValuesNull = errors.New(
"value is null; omit a key entirely to use its default")
errNotBareHostname = errors.New(
"must be a bare hostname without scheme, path, or whitespace")
errNoHostnameLabels = errors.New("contains no hostname labels")
)
// Params defines dependencies for Config.
type Params struct {
fx.In
Globals *globals.Globals
Logger *logger.Logger
}
@@ -99,19 +48,6 @@ type Config struct {
AllowlistHosts []string // Hosts that don't require signatures
AllowHTTP bool // Allow non-TLS upstream (testing only)
UpstreamConnectionsPerHost int // Max concurrent connections per upstream host
// CacheMaxBytes is the disk cache size limit in bytes. Zero
// disables the disk cache entirely. When cache_max_bytes is
// omitted from the configuration, this holds the computed default
// (75% of free space on the filesystem containing
// <state_dir>/cache/, floored at DefaultCacheMaxBytesFloor).
CacheMaxBytes int64
// cacheMaxBytesExplicit records whether cache_max_bytes was
// explicitly set in the configuration file. Explicit values are
// used exactly as given; only an omitted key gets the computed
// default (and its floor) in resolveCacheMaxBytes.
cacheMaxBytesExplicit bool
}
// New creates a new Config instance by loading configuration from file.
@@ -133,13 +69,7 @@ func New(_ fx.Lifecycle, params Params) (*Config, error) {
return nil, err
}
err = c.ensureStateDirWritable()
if err != nil {
return nil, err
}
err = c.resolveCacheMaxBytes(log, defaultFreeSpaceProbe)
if err != nil {
if err := c.ensureStateDirWritable(); err != nil {
return nil, err
}
@@ -157,13 +87,11 @@ func New(_ fx.Lifecycle, params Params) (*Config, error) {
// to omitted keys, never to invalid explicit values.
func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
if sc != nil {
err := validateKnownKeys(sc)
if err != nil {
if err := validateKnownKeys(sc); err != nil {
return nil, err
}
err = validateAllowlistHostsValue(sc)
if err != nil {
if err := validateAllowlistHostsValue(sc); err != nil {
return nil, err
}
}
@@ -171,43 +99,23 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
loader := &strictLoader{sc: sc}
c := &Config{
Debug: loader.boolVal(keyDebug, false),
MaintenanceMode: loader.boolVal(keyMaintenanceMode, false),
Port: loader.intVal(keyPort, DefaultPort),
StateDir: loader.stringVal(keyStateDir, DefaultStateDir),
SentryDSN: loader.stringVal(keySentryDSN, ""),
MetricsUsername: loader.stringVal(keyMetricsUsername, ""),
MetricsPassword: loader.stringVal(keyMetricsPassword, ""),
SigningKey: loader.stringVal(keySigningKey, ""),
AllowlistHosts: getStringSlice(sc),
AllowHTTP: loader.boolVal(keyAllowHTTP, false),
Debug: loader.boolVal("debug", false),
MaintenanceMode: loader.boolVal("maintenance_mode", false),
Port: loader.intVal("port", DefaultPort),
StateDir: loader.stringVal("state_dir", DefaultStateDir),
SentryDSN: loader.stringVal("sentry_dsn", ""),
MetricsUsername: loader.stringVal("metrics.username", ""),
MetricsPassword: loader.stringVal("metrics.password", ""),
SigningKey: loader.stringVal("signing_key", ""),
AllowlistHosts: getStringSlice(sc, "allowlist_hosts"),
AllowHTTP: loader.boolVal("allow_http", false),
UpstreamConnectionsPerHost: loader.intVal(
keyUpstreamConnectionsPerHost, DefaultUpstreamConnectionsPerHost),
CacheMaxBytes: loader.int64Val(keyCacheMaxBytes, 0),
"upstream_connections_per_host", DefaultUpstreamConnectionsPerHost),
}
// The computed default for cache_max_bytes needs a validated
// state_dir, so it is resolved later (resolveCacheMaxBytes); here
// we only record whether the operator set the key explicitly.
if sc != nil {
if _, present := sc.Get(keyCacheMaxBytes); present {
c.cacheMaxBytesExplicit = true
}
}
// Build DBURL from StateDir if not explicitly set. The derived URL
// is a default: it applies only when db_url is omitted, never to an
// explicitly empty value.
c.DBURL = loader.stringVal(keyDBURL, "")
if c.DBURL == "" && loader.err == nil {
if sc != nil {
if _, present := sc.Get(keyDBURL); present {
return nil, fmt.Errorf(
"config key %q: %w; omit the key to derive it from state_dir",
keyDBURL, errValueEmpty)
}
}
// Build DBURL from StateDir if not explicitly set
c.DBURL = loader.stringVal("db_url", "")
if c.DBURL == "" {
c.DBURL = fmt.Sprintf("file:%s/state.sqlite3?_journal_mode=WAL", c.StateDir)
}
@@ -215,8 +123,7 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
return nil, loader.err
}
err := c.validate()
if err != nil {
if err := c.validate(); err != nil {
return nil, err
}
@@ -225,12 +132,10 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
// validateKnownKeys rejects configuration files containing keys the
// application does not understand, so typos fail at startup instead of
// being silently ignored, and rejects keys that are explicitly set to
// null: a null is a SET value, never an omission, so it must not
// silently take the default. The env section is permitted because
// being silently ignored. The env section is permitted because
// smartconfig consumes it for environment variable injection.
func validateKnownKeys(sc *smartconfig.Config) error {
var unknown, nullKeys []string
var unknown []string
for key, value := range sc.Data() {
if !isKnownConfigKey(key) {
@@ -239,28 +144,17 @@ func validateKnownKeys(sc *smartconfig.Config) error {
continue
}
if value == nil {
nullKeys = append(nullKeys, key)
continue
}
if key == keyMetrics {
metricsMap, ok := value.(map[string]any)
if key == "metrics" {
metricsMap, ok := value.(map[string]interface{})
if !ok {
return fmt.Errorf("config key %q: value %v is %w",
keyMetrics, value, errNotAMetricsMap)
return fmt.Errorf(
"config key %q: value %v is not a map of metrics settings",
"metrics", value)
}
for subkey, subvalue := range metricsMap {
for subkey := range metricsMap {
if subkey != "username" && subkey != "password" {
unknown = append(unknown, keyMetrics+"."+subkey)
continue
}
if subvalue == nil {
nullKeys = append(nullKeys, keyMetrics+"."+subkey)
unknown = append(unknown, "metrics."+subkey)
}
}
}
@@ -269,37 +163,19 @@ func validateKnownKeys(sc *smartconfig.Config) error {
if len(unknown) > 0 {
sort.Strings(unknown)
return fmt.Errorf("%w: %s", errUnknownConfigKeys, strings.Join(unknown, ", "))
}
if len(nullKeys) > 0 {
sort.Strings(nullKeys)
if len(nullKeys) == 1 {
return errNullConfigValue(nullKeys[0])
}
return fmt.Errorf("config keys %s: %w",
strings.Join(nullKeys, ", "), errValuesNull)
return fmt.Errorf("unknown config keys: %s", strings.Join(unknown, ", "))
}
return nil
}
// errNullConfigValue reports a config key that is explicitly set to
// null (including the bare "key:" form and the "~" alias). Silently
// applying the default would mask a truncated or typo'd config entry.
func errNullConfigValue(key string) error {
return fmt.Errorf("config key %q: %w", key, errValueNull)
}
// isKnownConfigKey reports whether key is a permitted top-level
// configuration key.
func isKnownConfigKey(key string) bool {
switch key {
case keyDebug, keyMaintenanceMode, keyPort, keyStateDir, keySentryDSN,
keyDBURL, keyMetrics, keySigningKey, keyAllowlistHosts, keyAllowHTTP,
keyUpstreamConnectionsPerHost, keyCacheMaxBytes, "env":
case "debug", "maintenance_mode", "port", "state_dir", "sentry_dsn",
"db_url", "metrics", "signing_key", "allowlist_hosts", "allow_http",
"upstream_connections_per_host", "env":
return true
}
@@ -312,30 +188,27 @@ func isKnownConfigKey(key string) bool {
func (c *Config) ensureStateDirWritable() error {
const stateDirPerms = 0o750
err := os.MkdirAll(c.StateDir, stateDirPerms)
if err != nil {
if err := os.MkdirAll(c.StateDir, stateDirPerms); err != nil {
return fmt.Errorf("config key %q: cannot create directory %q: %w",
keyStateDir, c.StateDir, err)
"state_dir", c.StateDir, err)
}
probe, err := os.CreateTemp(c.StateDir, ".startup-write-probe-*")
if err != nil {
return fmt.Errorf("config key %q: directory %q is not writable: %w",
keyStateDir, c.StateDir, err)
"state_dir", c.StateDir, err)
}
probePath := probe.Name()
err = probe.Close()
if err != nil {
if err := probe.Close(); err != nil {
return fmt.Errorf("config key %q: cannot close probe file %q: %w",
keyStateDir, probePath, err)
"state_dir", probePath, err)
}
err = os.Remove(probePath)
if err != nil {
if err := os.Remove(probePath); err != nil {
return fmt.Errorf("config key %q: cannot remove probe file %q: %w",
keyStateDir, probePath, err)
"state_dir", probePath, err)
}
return nil
@@ -344,44 +217,34 @@ func (c *Config) ensureStateDirWritable() error {
// validate checks that all required configuration values are set and
// that every value is within its valid range.
func (c *Config) validate() error {
// The signing key value is never echoed in error messages.
if c.SigningKey == "" {
return fmt.Errorf("config key %q: %w", keySigningKey, errValueRequired)
return fmt.Errorf("signing_key is required")
}
// Minimum key length for security (32 bytes = 256 bits)
const minKeyLength = 32
if len(c.SigningKey) < minKeyLength {
return fmt.Errorf("config key %q: %w: must be at least %d characters, got %d",
keySigningKey, errValueTooShort, minKeyLength, len(c.SigningKey))
return fmt.Errorf("signing_key must be at least %d characters, got %d",
minKeyLength, len(c.SigningKey))
}
const maxPort = 65535
if c.Port < 1 || c.Port > maxPort {
return fmt.Errorf("config key %q: value %d is %w 1-%d",
keyPort, c.Port, errPortOutOfRange, maxPort)
return fmt.Errorf("config key %q: value %d is outside the valid port range 1-%d",
"port", c.Port, maxPort)
}
if c.UpstreamConnectionsPerHost < 1 {
return fmt.Errorf("config key %q: value %d %w",
keyUpstreamConnectionsPerHost, c.UpstreamConnectionsPerHost,
errTooFewConnections)
return fmt.Errorf("config key %q: value %d must be at least 1",
"upstream_connections_per_host", c.UpstreamConnectionsPerHost)
}
if c.StateDir == "" {
return fmt.Errorf("config key %q: %w", keyStateDir, errValueEmpty)
}
// Zero is valid (it disables the disk cache); only negative
// values are rejected. No floor applies to explicit values.
if c.CacheMaxBytes < 0 {
return fmt.Errorf("config key %q: value %d %w",
keyCacheMaxBytes, c.CacheMaxBytes, errMustNotBeNegative)
return fmt.Errorf("config key %q: value must not be empty", "state_dir")
}
for _, host := range c.AllowlistHosts {
err := validateAllowlistHost(host)
if err != nil {
if err := validateAllowlistHost(host); err != nil {
return err
}
}
@@ -389,14 +252,14 @@ func (c *Config) validate() error {
if c.SentryDSN != "" {
parsed, err := url.Parse(c.SentryDSN)
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
return fmt.Errorf("config key %q: value %q is %w",
keySentryDSN, c.SentryDSN, errNotAValidURL)
return fmt.Errorf("config key %q: value %q is not a valid URL",
"sentry_dsn", c.SentryDSN)
}
}
if (c.MetricsUsername == "") != (c.MetricsPassword == "") {
return fmt.Errorf("config keys %q and %q %w",
keyMetricsUsername, keyMetricsPassword, errMustBeSetTogether)
return fmt.Errorf("config keys %q and %q must be set together",
"metrics.username", "metrics.password")
}
return nil
@@ -404,27 +267,18 @@ func (c *Config) validate() error {
// validateAllowlistHost checks that an allowlist_hosts entry is a bare
// hostname, optionally with a leading dot for suffix matching. URLs,
// paths, and whitespace indicate a misconfigured entry. An entry with
// no hostname labels (such as ".") is rejected: the allowlist matcher
// treats a leading dot as a suffix pattern, so a bare "." would match
// any upstream host written in FQDN trailing-dot form and effectively
// disable URL signing.
// paths, and whitespace indicate a misconfigured entry.
func validateAllowlistHost(host string) error {
if strings.Contains(host, "://") || strings.ContainsAny(host, "/ \t") {
return fmt.Errorf("config key %q: entry %q %w",
keyAllowlistHosts, host, errNotBareHostname)
}
if strings.Trim(host, ".") == "" {
return fmt.Errorf("config key %q: entry %q %w",
keyAllowlistHosts, host, errNoHostnameLabels)
return fmt.Errorf(
"config key %q: entry %q must be a bare hostname without scheme, path, or whitespace",
"allowlist_hosts", host)
}
return nil
}
// loadConfigFile loads configuration from the PIXA_CONFIG_PATH env var
// or standard locations.
// loadConfigFile loads configuration from PIXA_CONFIG_PATH env var or standard locations.
func loadConfigFile(log *slog.Logger, appName string) (*smartconfig.Config, error) {
// Check for explicit config path from environment
if envPath := os.Getenv("PIXA_CONFIG_PATH"); envPath != "" {
@@ -450,9 +304,8 @@ func loadConfigFile(log *slog.Logger, appName string) (*smartconfig.Config, erro
for _, path := range configPaths {
cleanPath := filepath.Clean(path)
_, statErr := os.Stat(cleanPath)
if statErr == nil {
//nolint:gosec // G703: paths are hardcoded config locations
if _, statErr := os.Stat(cleanPath); statErr == nil {
// A config file that exists but does not parse is a fatal
// startup error, never something to skip over.
sc, err := smartconfig.NewFromConfigPath(path)
@@ -503,19 +356,6 @@ func (l *strictLoader) intVal(key string, defaultVal int) int {
return val
}
func (l *strictLoader) int64Val(key string, defaultVal int64) int64 {
if l.err != nil {
return 0
}
val, err := getInt64(l.sc, key, defaultVal)
if err != nil {
l.err = err
}
return val
}
func (l *strictLoader) boolVal(key string, defaultVal bool) bool {
if l.err != nil {
return false
@@ -530,48 +370,39 @@ func (l *strictLoader) boolVal(key string, defaultVal bool) bool {
}
// getString returns the string value for key, or defaultVal if the key
// is omitted. A present value that is not a string, or is explicitly
// null, is an error.
// is omitted. A present value that is not a string is an error.
func getString(sc *smartconfig.Config, key, defaultVal string) (string, error) {
if sc == nil {
return defaultVal, nil
}
raw, ok := sc.Get(key)
if !ok {
if !ok || raw == nil {
return defaultVal, nil
}
if raw == nil {
return "", errNullConfigValue(key)
}
str, ok := raw.(string)
if !ok {
return "", fmt.Errorf("config key %q: value %v (%T) is %w",
key, raw, raw, errNotAString)
return "", fmt.Errorf("config key %q: value %v (%T) is not a string",
key, raw, raw)
}
return str, nil
}
// getInt returns the integer value for key, or defaultVal if the key is
// omitted. A present value that is not a whole number, or is explicitly
// null, is an error; fractional values are never truncated.
// omitted. A present value that is not a whole number is an error;
// fractional values are never truncated.
func getInt(sc *smartconfig.Config, key string, defaultVal int) (int, error) {
if sc == nil {
return defaultVal, nil
}
raw, ok := sc.Get(key)
if !ok {
if !ok || raw == nil {
return defaultVal, nil
}
if raw == nil {
return 0, errNullConfigValue(key)
}
switch val := raw.(type) {
case int:
return val, nil
@@ -579,138 +410,76 @@ func getInt(sc *smartconfig.Config, key string, defaultVal int) (int, error) {
return int(val), nil
case float64:
if val != math.Trunc(val) {
return 0, fmt.Errorf("config key %q: value %v is %w",
key, val, errNotAnInteger)
return 0, fmt.Errorf("config key %q: value %v is not an integer", key, val)
}
return int(val), nil
case string:
parsed, err := strconv.Atoi(strings.TrimSpace(val))
if err != nil {
return 0, fmt.Errorf("config key %q: value %q is %w",
key, val, errNotAnInteger)
return 0, fmt.Errorf("config key %q: value %q is not an integer", key, val)
}
return parsed, nil
default:
return 0, fmt.Errorf("config key %q: value %v (%T) is %w",
key, raw, raw, errNotAnInteger)
}
}
// getInt64 returns the 64-bit integer value for key, or defaultVal if
// the key is omitted. A present value that is not a whole number, or
// is explicitly null, is an error; fractional values are never
// truncated and out-of-range values are never clamped.
func getInt64(sc *smartconfig.Config, key string, defaultVal int64) (int64, error) {
if sc == nil {
return defaultVal, nil
}
raw, ok := sc.Get(key)
if !ok {
return defaultVal, nil
}
if raw == nil {
return 0, errNullConfigValue(key)
}
switch val := raw.(type) {
case int:
return int64(val), nil
case int64:
return val, nil
case uint64:
if val > math.MaxInt64 {
return 0, fmt.Errorf("config key %q: value %d %w",
key, val, errOverflowsInt64)
}
return int64(val), nil
case float64:
if val != math.Trunc(val) {
return 0, fmt.Errorf("config key %q: value %v is %w",
key, val, errNotAnInteger)
}
return int64(val), nil
case string:
parsed, err := strconv.ParseInt(strings.TrimSpace(val), 10, 64)
if err != nil {
return 0, fmt.Errorf("config key %q: value %q is %w",
key, val, errNotAnInteger)
}
return parsed, nil
default:
return 0, fmt.Errorf("config key %q: value %v (%T) is %w",
key, raw, raw, errNotAnInteger)
return 0, fmt.Errorf("config key %q: value %v (%T) is not an integer",
key, raw, raw)
}
}
// getBool returns the boolean value for key, or defaultVal if the key
// is omitted. A present value that is not a boolean (or a ParseBool-able
// string), or is explicitly null, is an error; numbers are not accepted
// as booleans.
// string) is an error; numbers are not accepted as booleans.
func getBool(sc *smartconfig.Config, key string, defaultVal bool) (bool, error) {
if sc == nil {
return defaultVal, nil
}
raw, ok := sc.Get(key)
if !ok {
if !ok || raw == nil {
return defaultVal, nil
}
if raw == nil {
return false, errNullConfigValue(key)
}
switch val := raw.(type) {
case bool:
return val, nil
case string:
parsed, err := strconv.ParseBool(strings.TrimSpace(val))
if err != nil {
return false, fmt.Errorf("config key %q: value %q is %w",
key, val, errNotABoolean)
return false, fmt.Errorf("config key %q: value %q is not a boolean", key, val)
}
return parsed, nil
default:
return false, fmt.Errorf("config key %q: value %v (%T) is %w",
key, raw, raw, errNotABoolean)
return false, fmt.Errorf("config key %q: value %v (%T) is not a boolean",
key, raw, raw)
}
}
// validateAllowlistHostsValue checks the raw shape of the
// allowlist_hosts value before the lenient extraction in getStringSlice
// runs: an explicitly null value, a value that is not a list of strings
// (or a comma-separated string), a non-string entry, or an empty entry
// is an error, never silently skipped.
// runs: a value that is not a list of strings (or a comma-separated
// string), a non-string entry, or an empty entry is an error, never
// silently skipped.
func validateAllowlistHostsValue(sc *smartconfig.Config) error {
raw, ok := sc.Get(keyAllowlistHosts)
if !ok {
const key = "allowlist_hosts"
raw, ok := sc.Get(key)
if !ok || raw == nil {
return nil
}
if raw == nil {
return errNullConfigValue(keyAllowlistHosts)
}
switch val := raw.(type) {
case []any:
case []interface{}:
for _, item := range val {
str, ok := item.(string)
if !ok {
return fmt.Errorf("config key %q: list entry %v (%T) is %w",
keyAllowlistHosts, item, item, errNotAString)
return fmt.Errorf(
"config key %q: list entry %v (%T) is not a string", key, item, item)
}
if strings.TrimSpace(str) == "" {
return fmt.Errorf("config key %q: %w",
keyAllowlistHosts, errEmptyListEntry)
return fmt.Errorf("config key %q: list contains an empty entry", key)
}
}
case string:
@@ -718,36 +487,36 @@ func validateAllowlistHostsValue(sc *smartconfig.Config) error {
return nil
}
for part := range strings.SplitSeq(val, ",") {
for _, part := range strings.Split(val, ",") {
if strings.TrimSpace(part) == "" {
return fmt.Errorf("config key %q: value %q %w",
keyAllowlistHosts, val, errEmptyEntry)
return fmt.Errorf(
"config key %q: value %q contains an empty entry", key, val)
}
}
default:
return fmt.Errorf("config key %q: value %v (%T) is %w",
keyAllowlistHosts, raw, raw, errNotAStringList)
return fmt.Errorf("config key %q: value %v (%T) is not a list of strings",
key, raw, raw)
}
return nil
}
// getStringSlice returns the allowlist_hosts list of strings, or nil if
// the key is omitted. It accepts a YAML list of strings or a
// comma-separated string (backwards compatibility). Malformed entries
// are rejected beforehand by validateAllowlistHostsValue.
func getStringSlice(sc *smartconfig.Config) []string {
// getStringSlice returns the list of strings for key, or nil if the key
// is omitted. It accepts a YAML list of strings or a comma-separated
// string (backwards compatibility). Malformed entries are rejected
// beforehand by validateAllowlistHostsValue.
func getStringSlice(sc *smartconfig.Config, key string) []string {
if sc == nil {
return nil
}
val, ok := sc.Get(keyAllowlistHosts)
val, ok := sc.Get(key)
if !ok || val == nil {
return nil
}
// Handle YAML list format
if slice, ok := val.([]any); ok {
if slice, ok := val.([]interface{}); ok {
result := make([]string, 0, len(slice))
for _, item := range slice {
if str, ok := item.(string); ok {

98
internal/config/config_internal_test.go
View File

@@ -1,98 +0,0 @@
package config
import (
"os"
"path/filepath"
"testing"
"git.eeqj.de/sneak/smartconfig"
)
// writeTestConfig writes yamlContent to a temp config file and returns
// the file path.
func writeTestConfig(t *testing.T, yamlContent string) string {
t.Helper()
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.yml")
err := os.WriteFile(configPath, []byte(yamlContent), 0o600)
if err != nil {
t.Fatalf("failed to write test config: %v", err)
}
return configPath
}
// checkAllowlistHosts loads the config at configPath and asserts that
// getStringSlice returns the three expected hosts.
func checkAllowlistHosts(t *testing.T, configPath string) {
t.Helper()
sc, err := loadTestConfig(configPath)
if err != nil {
t.Fatalf("failed to load config: %v", err)
}
hosts := getStringSlice(sc)
if len(hosts) != 3 {
t.Errorf("expected 3 hosts, got %d: %v", len(hosts), hosts)
}
expected := []string{"static.sneak.cloud", "sneak.berlin", testHostS3}
for i, want := range expected {
if i >= len(hosts) {
t.Errorf("missing host at index %d: want %q", i, want)
continue
}
if hosts[i] != want {
t.Errorf("host[%d] = %q, want %q", i, hosts[i], want)
}
}
}
func TestGetStringSlice_YAMLList(t *testing.T) {
t.Parallel()
yamlContent := `
allowlist_hosts:
- static.sneak.cloud
- sneak.berlin
- s3.sneak.cloud
`
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
}
func TestGetStringSlice_CommaSeparated(t *testing.T) {
t.Parallel()
// Backwards compatibility with comma-separated string values.
yamlContent := `allowlist_hosts: "static.sneak.cloud, sneak.berlin, s3.sneak.cloud"`
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
}
func TestGetStringSlice_Empty(t *testing.T) {
t.Parallel()
configPath := writeTestConfig(t, `port: 8080`)
sc, err := loadTestConfig(configPath)
if err != nil {
t.Fatalf("failed to load config: %v", err)
}
hosts := getStringSlice(sc)
if len(hosts) != 0 {
t.Errorf("expected nil or empty slice, got %v", hosts)
}
}
// loadTestConfig is a helper to load a config file for testing.
func loadTestConfig(path string) (*smartconfig.Config, error) {
return smartconfig.NewFromConfigPath(path)
}

113
internal/config/config_test.go Normal file
View File

@@ -0,0 +1,113 @@
package config
import (
"os"
"path/filepath"
"testing"
"git.eeqj.de/sneak/smartconfig"
)
func TestGetStringSlice_YAMLList(t *testing.T) {
// Create a temp config file with YAML list format
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.yml")
yamlContent := `
allowlist_hosts:
- static.sneak.cloud
- sneak.berlin
- s3.sneak.cloud
`
err := os.WriteFile(configPath, []byte(yamlContent), 0644)
if err != nil {
t.Fatalf("failed to write test config: %v", err)
}
// Load config using smartconfig
sc, err := loadTestConfig(configPath)
if err != nil {
t.Fatalf("failed to load config: %v", err)
}
// Test that getStringSlice correctly parses YAML list
hosts := getStringSlice(sc, "allowlist_hosts")
if len(hosts) != 3 {
t.Errorf("expected 3 hosts, got %d: %v", len(hosts), hosts)
}
expected := []string{"static.sneak.cloud", "sneak.berlin", "s3.sneak.cloud"}
for i, want := range expected {
if i >= len(hosts) {
t.Errorf("missing host at index %d: want %q", i, want)
continue
}
if hosts[i] != want {
t.Errorf("host[%d] = %q, want %q", i, hosts[i], want)
}
}
}
func TestGetStringSlice_CommaSeparated(t *testing.T) {
// Test backwards compatibility with comma-separated string
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.yml")
yamlContent := `allowlist_hosts: "static.sneak.cloud, sneak.berlin, s3.sneak.cloud"`
err := os.WriteFile(configPath, []byte(yamlContent), 0644)
if err != nil {
t.Fatalf("failed to write test config: %v", err)
}
sc, err := loadTestConfig(configPath)
if err != nil {
t.Fatalf("failed to load config: %v", err)
}
hosts := getStringSlice(sc, "allowlist_hosts")
if len(hosts) != 3 {
t.Errorf("expected 3 hosts, got %d: %v", len(hosts), hosts)
}
expected := []string{"static.sneak.cloud", "sneak.berlin", "s3.sneak.cloud"}
for i, want := range expected {
if i >= len(hosts) {
t.Errorf("missing host at index %d: want %q", i, want)
continue
}
if hosts[i] != want {
t.Errorf("host[%d] = %q, want %q", i, hosts[i], want)
}
}
}
func TestGetStringSlice_Empty(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.yml")
yamlContent := `port: 8080`
err := os.WriteFile(configPath, []byte(yamlContent), 0644)
if err != nil {
t.Fatalf("failed to write test config: %v", err)
}
sc, err := loadTestConfig(configPath)
if err != nil {
t.Fatalf("failed to load config: %v", err)
}
hosts := getStringSlice(sc, "allowlist_hosts")
if hosts != nil && len(hosts) != 0 {
t.Errorf("expected nil or empty slice, got %v", hosts)
}
}
// loadTestConfig is a helper to load a config file for testing
func loadTestConfig(path string) (*smartconfig.Config, error) {
return smartconfig.NewFromConfigPath(path)
}

485
internal/config/config_validation_internal_test.go → internal/config/config_validation_test.go
View File

@@ -1,6 +1,7 @@
package config
import (
"io"
"log/slog"
"os"
"path/filepath"
@@ -14,26 +15,6 @@ import (
// minimum length requirement in validate().
const validTestSigningKey = "0123456789abcdef0123456789abcdef"
// signingKeyLine is a valid signing_key config line used as the base of
// test config files.
const signingKeyLine = "signing_key: " + validTestSigningKey + "\n"
// testHostS3 is an allowlist host entry used across the config tests.
const testHostS3 = "s3.sneak.cloud"
// nullValueText is the substring that error messages about explicitly
// null config values must contain.
const nullValueText = "null"
// abortCase describes a config file that must abort startup with an
// error mentioning every string in wantErrSubstrings.
type abortCase struct {
name string
yaml string
// wantErrSubstrings must all appear in the error message.
wantErrSubstrings []string
}
// configFromYAML writes yamlContent to a temporary config file, loads it
// via smartconfig, and constructs a Config from it using the same code
// path the server uses at startup.
@@ -43,8 +24,7 @@ func configFromYAML(t *testing.T, yamlContent string) (*Config, error) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.yml")
err := os.WriteFile(configPath, []byte(yamlContent), 0o600)
if err != nil {
if err := os.WriteFile(configPath, []byte(yamlContent), 0o600); err != nil {
t.Fatalf("failed to write test config: %v", err)
}
@@ -57,9 +37,7 @@ func configFromYAML(t *testing.T, yamlContent string) (*Config, error) {
}
func TestOmittedValuesUseDefaults(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, signingKeyLine)
c, err := configFromYAML(t, "signing_key: "+validTestSigningKey+"\n")
if err != nil {
t.Fatalf("minimal config should be valid, got error: %v", err)
}
@@ -100,8 +78,6 @@ func TestOmittedValuesUseDefaults(t *testing.T) {
}
func TestExplicitValidValuesAreUsed(t *testing.T) {
t.Parallel()
yamlContent := `
port: 9090
debug: true
@@ -142,10 +118,9 @@ metrics:
t.Errorf("DBURL = %q, want explicit value", c.DBURL)
}
if len(c.AllowlistHosts) != 2 || c.AllowlistHosts[0] != testHostS3 ||
if len(c.AllowlistHosts) != 2 || c.AllowlistHosts[0] != "s3.sneak.cloud" ||
c.AllowlistHosts[1] != ".example.com" {
t.Errorf("AllowlistHosts = %v, want [s3.sneak.cloud .example.com]",
c.AllowlistHosts)
t.Errorf("AllowlistHosts = %v, want [s3.sneak.cloud .example.com]", c.AllowlistHosts)
}
if c.UpstreamConnectionsPerHost != 5 {
@@ -163,10 +138,8 @@ metrics:
}
func TestCommaSeparatedAllowlistStillSupported(t *testing.T) {
t.Parallel()
yamlContent := signingKeyLine +
`allowlist_hosts: "s3.sneak.cloud, sneak.berlin"
yamlContent := `signing_key: ` + validTestSigningKey + `
allowlist_hosts: "s3.sneak.cloud, sneak.berlin"
`
c, err := configFromYAML(t, yamlContent)
@@ -174,22 +147,138 @@ func TestCommaSeparatedAllowlistStillSupported(t *testing.T) {
t.Fatalf("comma-separated allowlist should load, got error: %v", err)
}
if len(c.AllowlistHosts) != 2 || c.AllowlistHosts[0] != testHostS3 ||
if len(c.AllowlistHosts) != 2 || c.AllowlistHosts[0] != "s3.sneak.cloud" ||
c.AllowlistHosts[1] != "sneak.berlin" {
t.Errorf("AllowlistHosts = %v, want [s3.sneak.cloud sneak.berlin]",
c.AllowlistHosts)
t.Errorf("AllowlistHosts = %v, want [s3.sneak.cloud sneak.berlin]", c.AllowlistHosts)
}
}
// runAbortCases asserts that each case's config aborts startup with an
// error message mentioning every expected substring.
func runAbortCases(t *testing.T, cases []abortCase) {
t.Helper()
// TestSetButInvalidValueAbortsStartup verifies the no-silent-fallback
// rule: a key that is explicitly set to an unparseable or out-of-range
// value must produce a startup error naming the offending key, never
// silently fall back to the default.
func TestSetButInvalidValueAbortsStartup(t *testing.T) {
signingKeyLine := "signing_key: " + validTestSigningKey + "\n"
cases := []struct {
name string
yaml string
// wantErrSubstrings must all appear in the error message.
wantErrSubstrings []string
}{
{
name: "port not a number",
yaml: signingKeyLine + "port: banana\n",
wantErrSubstrings: []string{"port", "banana"},
},
{
name: "port zero",
yaml: signingKeyLine + "port: 0\n",
wantErrSubstrings: []string{"port", "0"},
},
{
name: "port above 65535",
yaml: signingKeyLine + "port: 99999\n",
wantErrSubstrings: []string{"port", "99999"},
},
{
name: "port fractional",
yaml: signingKeyLine + "port: 8080.5\n",
wantErrSubstrings: []string{"port", "8080.5"},
},
{
name: "debug not a bool",
yaml: signingKeyLine + "debug: notabool\n",
wantErrSubstrings: []string{"debug", "notabool"},
},
{
name: "maintenance_mode not a bool",
yaml: signingKeyLine + "maintenance_mode: sometimes\n",
wantErrSubstrings: []string{"maintenance_mode", "sometimes"},
},
{
name: "allow_http numeric",
yaml: signingKeyLine + "allow_http: 2\n",
wantErrSubstrings: []string{"allow_http", "2"},
},
{
name: "upstream_connections_per_host zero",
yaml: signingKeyLine + "upstream_connections_per_host: 0\n",
wantErrSubstrings: []string{"upstream_connections_per_host", "0"},
},
{
name: "upstream_connections_per_host negative",
yaml: signingKeyLine + "upstream_connections_per_host: -3\n",
wantErrSubstrings: []string{"upstream_connections_per_host", "-3"},
},
{
name: "upstream_connections_per_host not a number",
yaml: signingKeyLine + "upstream_connections_per_host: many\n",
wantErrSubstrings: []string{"upstream_connections_per_host", "many"},
},
{
name: "allowlist host with scheme",
yaml: signingKeyLine + "allowlist_hosts:\n - https://example.com\n",
wantErrSubstrings: []string{
"allowlist_hosts", "https://example.com",
},
},
{
name: "allowlist host with path",
yaml: signingKeyLine + "allowlist_hosts:\n - example.com/images\n",
wantErrSubstrings: []string{
"allowlist_hosts", "example.com/images",
},
},
{
name: "allowlist host with whitespace",
yaml: signingKeyLine + "allowlist_hosts:\n - \"exa mple.com\"\n",
wantErrSubstrings: []string{"allowlist_hosts", "exa mple.com"},
},
{
name: "allowlist entry not a string",
yaml: signingKeyLine + "allowlist_hosts:\n - 123\n",
wantErrSubstrings: []string{"allowlist_hosts", "123"},
},
{
name: "allowlist not a list",
yaml: signingKeyLine + "allowlist_hosts:\n key: value\n",
wantErrSubstrings: []string{"allowlist_hosts"},
},
{
name: "signing_key too short",
yaml: "signing_key: short\n",
wantErrSubstrings: []string{"signing_key"},
},
{
name: "signing_key missing",
yaml: "port: 8080\n",
wantErrSubstrings: []string{"signing_key"},
},
{
name: "state_dir explicitly empty",
yaml: signingKeyLine + "state_dir: \"\"\n",
wantErrSubstrings: []string{"state_dir"},
},
{
name: "sentry_dsn not a URL",
yaml: signingKeyLine + "sentry_dsn: \"not a url\"\n",
wantErrSubstrings: []string{"sentry_dsn", "not a url"},
},
{
name: "metrics username without password",
yaml: signingKeyLine + "metrics:\n username: bob\n",
wantErrSubstrings: []string{"metrics"},
},
{
name: "metrics password without username",
yaml: signingKeyLine + "metrics:\n password: hunter2\n",
wantErrSubstrings: []string{"metrics"},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, tc.yaml)
if err == nil {
t.Fatalf("config with %s must abort startup, got config: %+v", tc.name, c)
@@ -206,283 +295,9 @@ func runAbortCases(t *testing.T, cases []abortCase) {
}
}
// invalidScalarValueCases are configs where a scalar key is explicitly
// set to an unparseable or out-of-range value; each must abort startup
// naming the offending key, never silently fall back to the default.
func invalidScalarValueCases() []abortCase {
return []abortCase{
{
name: "port not a number",
yaml: signingKeyLine + "port: banana\n",
wantErrSubstrings: []string{keyPort, "banana"},
},
{
name: "port zero",
yaml: signingKeyLine + "port: 0\n",
wantErrSubstrings: []string{keyPort, "0"},
},
{
name: "port above 65535",
yaml: signingKeyLine + "port: 99999\n",
wantErrSubstrings: []string{keyPort, "99999"},
},
{
name: "port fractional",
yaml: signingKeyLine + "port: 8080.5\n",
wantErrSubstrings: []string{keyPort, "8080.5"},
},
{
name: "debug not a bool",
yaml: signingKeyLine + "debug: notabool\n",
wantErrSubstrings: []string{keyDebug, "notabool"},
},
{
name: "maintenance_mode not a bool",
yaml: signingKeyLine + "maintenance_mode: sometimes\n",
wantErrSubstrings: []string{keyMaintenanceMode, "sometimes"},
},
{
name: "allow_http numeric",
yaml: signingKeyLine + "allow_http: 2\n",
wantErrSubstrings: []string{keyAllowHTTP, "2"},
},
{
name: "upstream_connections_per_host zero",
yaml: signingKeyLine + "upstream_connections_per_host: 0\n",
wantErrSubstrings: []string{keyUpstreamConnectionsPerHost, "0"},
},
{
name: "upstream_connections_per_host negative",
yaml: signingKeyLine + "upstream_connections_per_host: -3\n",
wantErrSubstrings: []string{keyUpstreamConnectionsPerHost, "-3"},
},
{
name: "upstream_connections_per_host not a number",
yaml: signingKeyLine + "upstream_connections_per_host: many\n",
wantErrSubstrings: []string{keyUpstreamConnectionsPerHost, "many"},
},
}
}
// invalidHostAndCredentialCases are configs where allowlist_hosts,
// signing_key, state_dir, sentry_dsn, or metrics is explicitly set to
// an invalid value; each must abort startup naming the offending key.
func invalidHostAndCredentialCases() []abortCase {
return []abortCase{
{
name: "allowlist host with scheme",
yaml: signingKeyLine + "allowlist_hosts:\n - https://example.com\n",
wantErrSubstrings: []string{
keyAllowlistHosts, "https://example.com",
},
},
{
name: "allowlist host with path",
yaml: signingKeyLine + "allowlist_hosts:\n - example.com/images\n",
wantErrSubstrings: []string{
keyAllowlistHosts, "example.com/images",
},
},
{
name: "allowlist host with whitespace",
yaml: signingKeyLine + "allowlist_hosts:\n - \"exa mple.com\"\n",
wantErrSubstrings: []string{keyAllowlistHosts, "exa mple.com"},
},
{
name: "allowlist entry not a string",
yaml: signingKeyLine + "allowlist_hosts:\n - 123\n",
wantErrSubstrings: []string{keyAllowlistHosts, "123"},
},
{
name: "allowlist not a list",
yaml: signingKeyLine + "allowlist_hosts:\n key: value\n",
wantErrSubstrings: []string{keyAllowlistHosts},
},
{
name: "signing_key too short",
yaml: "signing_key: short\n",
wantErrSubstrings: []string{keySigningKey},
},
{
name: "signing_key missing",
yaml: "port: 8080\n",
wantErrSubstrings: []string{keySigningKey},
},
{
name: "state_dir explicitly empty",
yaml: signingKeyLine + "state_dir: \"\"\n",
wantErrSubstrings: []string{keyStateDir},
},
{
name: "sentry_dsn not a URL",
yaml: signingKeyLine + "sentry_dsn: \"not a url\"\n",
wantErrSubstrings: []string{keySentryDSN, "not a url"},
},
{
name: "metrics username without password",
yaml: signingKeyLine + "metrics:\n username: bob\n",
wantErrSubstrings: []string{keyMetrics},
},
{
name: "metrics password without username",
yaml: signingKeyLine + "metrics:\n password: hunter2\n",
wantErrSubstrings: []string{keyMetrics},
},
}
}
// TestSetButInvalidValueAbortsStartup verifies the no-silent-fallback
// rule: a key that is explicitly set to an unparseable or out-of-range
// value must produce a startup error naming the offending key, never
// silently fall back to the default.
func TestSetButInvalidValueAbortsStartup(t *testing.T) {
t.Parallel()
runAbortCases(t, append(
invalidScalarValueCases(), invalidHostAndCredentialCases()...))
}
// explicitNullValueCases are configs where a key is explicitly set to
// null (including the bare "key:" form and the "~" alias); each must
// abort startup naming the key.
func explicitNullValueCases() []abortCase {
return []abortCase{
{
name: "port explicit null",
yaml: signingKeyLine + "port: null\n",
wantErrSubstrings: []string{keyPort, nullValueText},
},
{
name: "port bare key no value",
yaml: signingKeyLine + "port:\n",
wantErrSubstrings: []string{keyPort, nullValueText},
},
{
name: "debug tilde null",
yaml: signingKeyLine + "debug: ~\n",
wantErrSubstrings: []string{keyDebug, nullValueText},
},
{
name: "maintenance_mode null",
yaml: signingKeyLine + "maintenance_mode: null\n",
wantErrSubstrings: []string{keyMaintenanceMode, nullValueText},
},
{
name: "allow_http null",
yaml: signingKeyLine + "allow_http: null\n",
wantErrSubstrings: []string{keyAllowHTTP, nullValueText},
},
{
name: "state_dir null",
yaml: signingKeyLine + "state_dir: null\n",
wantErrSubstrings: []string{keyStateDir, nullValueText},
},
{
name: "db_url null",
yaml: signingKeyLine + "db_url: null\n",
wantErrSubstrings: []string{keyDBURL, nullValueText},
},
{
name: "sentry_dsn null",
yaml: signingKeyLine + "sentry_dsn: null\n",
wantErrSubstrings: []string{keySentryDSN, nullValueText},
},
{
name: "upstream_connections_per_host null",
yaml: signingKeyLine + "upstream_connections_per_host: null\n",
wantErrSubstrings: []string{keyUpstreamConnectionsPerHost, nullValueText},
},
{
name: "allowlist_hosts null",
yaml: signingKeyLine + "allowlist_hosts: null\n",
wantErrSubstrings: []string{keyAllowlistHosts, nullValueText},
},
{
name: "signing_key null",
yaml: "signing_key: null\n",
wantErrSubstrings: []string{keySigningKey, nullValueText},
},
{
name: "metrics null",
yaml: signingKeyLine + "metrics: null\n",
wantErrSubstrings: []string{keyMetrics, nullValueText},
},
{
name: "metrics subkeys null",
yaml: signingKeyLine + "metrics:\n username: null\n password: null\n",
wantErrSubstrings: []string{
keyMetricsUsername, keyMetricsPassword, nullValueText,
},
},
}
}
// TestExplicitNullValueAbortsStartup verifies that a key explicitly
// set to null (including the bare "key:" form and the "~" alias) aborts
// startup naming the key. An explicit null is a SET value: it must
// never silently fall back to the default the way an omitted key does.
func TestExplicitNullValueAbortsStartup(t *testing.T) {
t.Parallel()
runAbortCases(t, explicitNullValueCases())
}
// TestExplicitlyEmptyDBURLAbortsStartup verifies that db_url set to an
// empty string aborts startup: the derived file:...state.sqlite3 URL is
// a default, and defaults apply only to omitted keys. This matches
// state_dir, where an explicitly empty value already aborts.
func TestExplicitlyEmptyDBURLAbortsStartup(t *testing.T) {
t.Parallel()
yamlContent := signingKeyLine + "db_url: \"\"\n"
c, err := configFromYAML(t, yamlContent)
if err == nil {
t.Fatalf("explicitly empty db_url must abort startup, got config: %+v", c)
}
t.Logf("got expected error: %v", err)
if !strings.Contains(err.Error(), keyDBURL) {
t.Errorf("error %q does not name the offending key db_url", err.Error())
}
}
// TestAllowlistHostsRejectsDotOnlyEntries verifies that entries with no
// hostname labels are rejected. The allowlist matcher treats a leading
// dot as a suffix pattern, so a bare "." entry would match any upstream
// host written in FQDN trailing-dot form (e.g. evil.com.) and
// effectively disable URL signing with a single character.
func TestAllowlistHostsRejectsDotOnlyEntries(t *testing.T) {
t.Parallel()
for _, entry := range []string{".", ".."} {
t.Run(entry, func(t *testing.T) {
t.Parallel()
yamlContent := signingKeyLine +
"allowlist_hosts:\n - \"" + entry + "\"\n"
c, err := configFromYAML(t, yamlContent)
if err == nil {
t.Fatalf("allowlist entry %q must abort startup, got config: %+v",
entry, c)
}
t.Logf("got expected error: %v", err)
if !strings.Contains(err.Error(), keyAllowlistHosts) {
t.Errorf("error %q does not name the offending key allowlist_hosts",
err.Error())
}
})
}
}
func TestUnknownTopLevelKeyAbortsStartup(t *testing.T) {
t.Parallel()
yamlContent := signingKeyLine + `whitelist_hosts:
yamlContent := `signing_key: ` + validTestSigningKey + `
whitelist_hosts:
- example.com
`
@@ -499,9 +314,8 @@ func TestUnknownTopLevelKeyAbortsStartup(t *testing.T) {
}
func TestUnknownMetricsSubkeyAbortsStartup(t *testing.T) {
t.Parallel()
yamlContent := signingKeyLine + `metrics:
yamlContent := `signing_key: ` + validTestSigningKey + `
metrics:
username: bob
password: hunter2
port: 9100
@@ -520,17 +334,13 @@ func TestUnknownMetricsSubkeyAbortsStartup(t *testing.T) {
}
func TestEnvSectionIsPermitted(t *testing.T) {
t.Parallel()
yamlContent := signingKeyLine + `env:
yamlContent := `signing_key: ` + validTestSigningKey + `
env:
PIXA_TEST_ENV_INJECTION: injected
`
_, err := configFromYAML(t, yamlContent)
if err != nil {
t.Fatalf(
"env section must be permitted (smartconfig consumes it), got error: %v",
err)
if _, err := configFromYAML(t, yamlContent); err != nil {
t.Fatalf("env section must be permitted (smartconfig consumes it), got error: %v", err)
}
}
@@ -538,8 +348,7 @@ func TestMalformedConfigFileAbortsStartup(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.yml")
err := os.WriteFile(configPath, []byte("port: [unclosed\n"), 0o600)
if err != nil {
if err := os.WriteFile(configPath, []byte("port: [unclosed\n"), 0o600); err != nil {
t.Fatalf("failed to write malformed config: %v", err)
}
@@ -548,7 +357,7 @@ func TestMalformedConfigFileAbortsStartup(t *testing.T) {
t.Setenv("PIXA_CONFIG_PATH", "")
t.Chdir(tmpDir)
log := slog.New(slog.DiscardHandler)
log := slog.New(slog.NewTextHandler(io.Discard, nil))
sc, err := loadConfigFile(log, "pixa-test-nonexistent-app")
if err == nil {
@@ -559,14 +368,10 @@ func TestMalformedConfigFileAbortsStartup(t *testing.T) {
}
func TestEnsureStateDirCreatesDirectory(t *testing.T) {
t.Parallel()
stateDir := filepath.Join(t.TempDir(), "nested", "state")
c := &Config{StateDir: stateDir}
err := c.ensureStateDirWritable()
if err != nil {
if err := c.ensureStateDirWritable(); err != nil {
t.Fatalf("creatable state_dir must validate, got error: %v", err)
}
@@ -577,8 +382,6 @@ func TestEnsureStateDirCreatesDirectory(t *testing.T) {
}
func TestEnsureStateDirFailsOnUncreatablePath(t *testing.T) {
t.Parallel()
// A path below /dev/null can never be created, even when running
// as root (as in the Docker build).
c := &Config{StateDir: "/dev/null/pixa-state"}
@@ -590,7 +393,7 @@ func TestEnsureStateDirFailsOnUncreatablePath(t *testing.T) {
t.Logf("got expected error: %v", err)
if !strings.Contains(err.Error(), keyStateDir) {
if !strings.Contains(err.Error(), "state_dir") {
t.Errorf("error %q does not name the offending key state_dir", err.Error())
}
}

77
internal/database/database.go
View File

@@ -5,7 +5,6 @@ import (
"context"
"database/sql"
"embed"
"errors"
"fmt"
"log/slog"
"path/filepath"
@@ -30,15 +29,10 @@ const bootstrapVersion = 0
// Params defines dependencies for Database.
type Params struct {
fx.In
Logger *logger.Logger
Config *config.Config
}
// errInvalidMigrationFilename is returned when a migration filename does
// not match the "<version>[_<description>].sql" pattern.
var errInvalidMigrationFilename = errors.New("invalid migration filename")
// Database wraps the SQL database connection.
type Database struct {
db *sql.DB
@@ -54,31 +48,33 @@ type Database struct {
func ParseMigrationVersion(filename string) (int, error) {
name := strings.TrimSuffix(filename, filepath.Ext(filename))
if name == "" {
return 0, fmt.Errorf("%w %q: empty name", errInvalidMigrationFilename, filename)
return 0, fmt.Errorf("invalid migration filename %q: empty name", filename)
}
// Split on underscore to separate version from description.
// If there's no underscore, the entire stem is the version.
versionStr, _, _ := strings.Cut(name, "_")
versionStr := name
if idx := strings.IndexByte(name, '_'); idx >= 0 {
versionStr = name[:idx]
}
if versionStr == "" {
return 0, fmt.Errorf(
"%w %q: empty version prefix", errInvalidMigrationFilename, filename,
)
return 0, fmt.Errorf("invalid migration filename %q: empty version prefix", filename)
}
// Validate the version is purely numeric.
for _, ch := range versionStr {
if ch < '0' || ch > '9' {
return 0, fmt.Errorf(
"%w %q: version %q contains non-numeric character %q",
errInvalidMigrationFilename, filename, versionStr, string(ch),
"invalid migration filename %q: version %q contains non-numeric character %q",
filename, versionStr, string(ch),
)
}
}
version, err := strconv.Atoi(versionStr)
if err != nil {
return 0, fmt.Errorf("%w %q: %w", errInvalidMigrationFilename, filename, err)
return 0, fmt.Errorf("invalid migration filename %q: %w", filename, err)
}
return version, nil
@@ -101,7 +97,6 @@ func New(lc fx.Lifecycle, params Params) (*Database, error) {
},
OnStop: func(_ context.Context) error {
s.log.Info("Database OnStop Hook")
if s.db != nil {
return s.db.Close()
}
@@ -113,6 +108,30 @@ func New(lc fx.Lifecycle, params Params) (*Database, error) {
return s, nil
}
func (s *Database) connect(ctx context.Context) error {
dbURL := s.config.DBURL
s.log.Info("connecting to database", "url", dbURL)
db, err := sql.Open("sqlite", dbURL)
if err != nil {
s.log.Error("failed to open database", "error", err)
return err
}
if err := db.PingContext(ctx); err != nil {
s.log.Error("failed to ping database", "error", err)
return err
}
s.db = db
s.log.Info("database connected")
return ApplyMigrations(ctx, s.db, s.log)
}
// collectMigrations reads the embedded schema directory and returns
// migration filenames sorted lexicographically.
func collectMigrations() ([]string, error) {
@@ -172,8 +191,7 @@ func bootstrapMigrationsTable(ctx context.Context, db *sql.DB, log *slog.Logger)
// This is exported so tests can apply the real schema without the full fx
// lifecycle.
func ApplyMigrations(ctx context.Context, db *sql.DB, log *slog.Logger) error {
err := bootstrapMigrationsTable(ctx, db, log)
if err != nil {
if err := bootstrapMigrationsTable(ctx, db, log); err != nil {
return err
}
@@ -243,28 +261,3 @@ func ApplyMigrations(ctx context.Context, db *sql.DB, log *slog.Logger) error {
func (s *Database) DB() *sql.DB {
return s.db
}
func (s *Database) connect(ctx context.Context) error {
dbURL := s.config.DBURL
s.log.Info("connecting to database", "url", dbURL)
db, err := sql.Open("sqlite", dbURL)
if err != nil {
s.log.Error("failed to open database", "error", err)
return err
}
err = db.PingContext(ctx)
if err != nil {
s.log.Error("failed to ping database", "error", err)
return err
}
s.db = db
s.log.Info("database connected")
return ApplyMigrations(ctx, s.db, s.log)
}

73
internal/database/database_internal_test.go → internal/database/database_test.go
View File

@@ -1,6 +1,7 @@
package database
import (
"context"
"database/sql"
"testing"
@@ -16,14 +17,12 @@ func openTestDB(t *testing.T) *sql.DB {
t.Fatalf("failed to open test db: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
t.Cleanup(func() { db.Close() })
return db
}
func TestParseMigrationVersion(t *testing.T) {
t.Parallel()
tests := []struct {
name string
filename string
@@ -79,8 +78,6 @@ func TestParseMigrationVersion(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := ParseMigrationVersion(tt.filename)
if tt.wantErr {
if err == nil {
@@ -104,50 +101,37 @@ func TestParseMigrationVersion(t *testing.T) {
}
func TestApplyMigrations_CreatesSchemaAndTables(t *testing.T) {
t.Parallel()
db := openTestDB(t)
ctx := t.Context()
ctx := context.Background()
err := ApplyMigrations(ctx, db, nil)
if err != nil {
if err := ApplyMigrations(ctx, db, nil); err != nil {
t.Fatalf("ApplyMigrations failed: %v", err)
}
// The schema_migrations table must exist and contain at least
// version 0 (the bootstrap) and 1 (the initial schema).
rows, err := db.QueryContext(
ctx, "SELECT version FROM schema_migrations ORDER BY version",
)
rows, err := db.Query("SELECT version FROM schema_migrations ORDER BY version")
if err != nil {
t.Fatalf("failed to query schema_migrations: %v", err)
}
defer func() { _ = rows.Close() }()
defer rows.Close()
var versions []int
for rows.Next() {
var v int
scanErr := rows.Scan(&v)
if scanErr != nil {
t.Fatalf("failed to scan version: %v", scanErr)
if err := rows.Scan(&v); err != nil {
t.Fatalf("failed to scan version: %v", err)
}
versions = append(versions, v)
}
err = rows.Err()
if err != nil {
if err := rows.Err(); err != nil {
t.Fatalf("row iteration error: %v", err)
}
if len(versions) < 2 {
t.Fatalf(
"expected at least 2 migrations recorded, got %d: %v",
len(versions), versions,
)
t.Fatalf("expected at least 2 migrations recorded, got %d: %v", len(versions), versions)
}
if versions[0] != 0 {
@@ -159,15 +143,10 @@ func TestApplyMigrations_CreatesSchemaAndTables(t *testing.T) {
}
// Verify that the application tables created by 001.sql exist.
tables := []string{
"source_content", "source_metadata", "output_content",
"request_cache", "negative_cache", "cache_stats",
}
for _, table := range tables {
for _, table := range []string{"source_content", "source_metadata", "output_content", "request_cache", "negative_cache", "cache_stats"} {
var count int
err := db.QueryRowContext(
ctx,
err := db.QueryRow(
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name=?",
table,
).Scan(&count)
@@ -182,28 +161,22 @@ func TestApplyMigrations_CreatesSchemaAndTables(t *testing.T) {
}
func TestApplyMigrations_Idempotent(t *testing.T) {
t.Parallel()
db := openTestDB(t)
ctx := t.Context()
ctx := context.Background()
err := ApplyMigrations(ctx, db, nil)
if err != nil {
if err := ApplyMigrations(ctx, db, nil); err != nil {
t.Fatalf("first ApplyMigrations failed: %v", err)
}
// Running a second time must succeed without errors.
err = ApplyMigrations(ctx, db, nil)
if err != nil {
if err := ApplyMigrations(ctx, db, nil); err != nil {
t.Fatalf("second ApplyMigrations failed: %v", err)
}
// Verify no duplicate rows in schema_migrations.
var count int
err = db.QueryRowContext(
ctx, "SELECT COUNT(*) FROM schema_migrations WHERE version = 0",
).Scan(&count)
err := db.QueryRow("SELECT COUNT(*) FROM schema_migrations WHERE version = 0").Scan(&count)
if err != nil {
t.Fatalf("failed to count version 0 rows: %v", err)
}
@@ -214,21 +187,17 @@ func TestApplyMigrations_Idempotent(t *testing.T) {
}
func TestBootstrapMigrationsTable_FreshDatabase(t *testing.T) {
t.Parallel()
db := openTestDB(t)
ctx := t.Context()
ctx := context.Background()
err := bootstrapMigrationsTable(ctx, db, nil)
if err != nil {
if err := bootstrapMigrationsTable(ctx, db, nil); err != nil {
t.Fatalf("bootstrapMigrationsTable failed: %v", err)
}
// schema_migrations table must exist.
var tableCount int
err = db.QueryRowContext(
ctx,
err := db.QueryRow(
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='schema_migrations'",
).Scan(&tableCount)
if err != nil {
@@ -242,8 +211,8 @@ func TestBootstrapMigrationsTable_FreshDatabase(t *testing.T) {
// Version 0 must be recorded.
var recorded int
err = db.QueryRowContext(
ctx, "SELECT COUNT(*) FROM schema_migrations WHERE version = 0",
err = db.QueryRow(
"SELECT COUNT(*) FROM schema_migrations WHERE version = 0",
).Scan(&recorded)
if err != nil {
t.Fatalf("failed to check version: %v", err)

23
internal/database/schema/001_initial_schema.sql
View File

@@ -3,17 +3,12 @@
-- Source content blobs
-- Files stored at: cache/src-content/<ab>/<cd>/<sha256>
-- last_accessed_at is NULL until the first LRU touch; eviction falls
-- back to fetched_at for rows that have never been touched.
CREATE TABLE IF NOT EXISTS source_content (
content_hash TEXT PRIMARY KEY,
content_type TEXT NOT NULL,
size_bytes INTEGER NOT NULL,
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP,
last_accessed_at DATETIME
fetched_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_source_content_last_accessed
ON source_content(last_accessed_at);
-- Source URL metadata - maps URLs to content hashes
-- JSON stored at: cache/src-metadata/<hostname>/<path_hash>.json
@@ -39,22 +34,6 @@ CREATE INDEX IF NOT EXISTS idx_source_meta_path_hash ON source_metadata(path_has
CREATE INDEX IF NOT EXISTS idx_source_meta_expires ON source_metadata(expires_at);
CREATE INDEX IF NOT EXISTS idx_source_meta_content_hash ON source_metadata(content_hash);
-- Processed variant blobs
-- Files stored at: cache/variants/<ab>/<cd>/<cache_key> (plus a .meta
-- sidecar with the content type). Tracked here (like source content
-- blobs above) so total cache usage can be computed with a SUM query,
-- never a directory scan, and so LRU eviction has a timestamp to order
-- on.
CREATE TABLE IF NOT EXISTS variant_content (
cache_key TEXT PRIMARY KEY,
size_bytes INTEGER NOT NULL,
content_type TEXT NOT NULL,
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_variant_content_last_accessed
ON variant_content(last_accessed_at);
-- Output/transformed content blobs
-- Files stored at: cache/dst-content/<ab>/<cd>/<sha256>
CREATE TABLE IF NOT EXISTS output_content (

10
internal/encurl/encurl.go
View File

@@ -48,8 +48,7 @@ type Generator struct {
key [seal.KeySize]byte
}
// NewGenerator creates an encrypted URL generator with a key derived
// from the signing key.
// NewGenerator creates an encrypted URL generator with a key derived from the signing key.
func NewGenerator(signingKey string) (*Generator, error) {
key, err := seal.DeriveKey([]byte(signingKey), urlKeySalt)
if err != nil {
@@ -78,8 +77,7 @@ func (g *Generator) Parse(token string) (*Payload, error) {
// Decrypt
data, err := seal.Decrypt(g.key, token)
if err != nil {
if errors.Is(err, seal.ErrDecryptionFailed) ||
errors.Is(err, seal.ErrInvalidPayload) {
if errors.Is(err, seal.ErrDecryptionFailed) || errors.Is(err, seal.ErrInvalidPayload) {
return nil, ErrDecryptFailed
}
@@ -88,9 +86,7 @@ func (g *Generator) Parse(token string) (*Payload, error) {
// CBOR decode
var p Payload
err = cbor.Unmarshal(data, &p)
if err != nil {
if err := cbor.Unmarshal(data, &p); err != nil {
return nil, ErrInvalidFormat
}

152
internal/encurl/encurl_test.go
View File

@@ -1,33 +1,22 @@
package encurl_test
package encurl
import (
"errors"
"testing"
"time"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
)
// Shared test fixture strings.
const (
testSourceHost = "cdn.example.com"
testSourcePath = "/images/photo.jpg"
testSourceQuery = "v=2"
)
func TestGenerator_GenerateAndParse(t *testing.T) {
t.Parallel()
gen, err := encurl.NewGenerator("test-signing-key-12345")
gen, err := NewGenerator("test-signing-key-12345")
if err != nil {
t.Fatalf("NewGenerator() error = %v", err)
}
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
SourceQuery: testSourceQuery,
payload := &Payload{
SourceHost: "cdn.example.com",
SourcePath: "/images/photo.jpg",
SourceQuery: "v=2",
Width: 800,
Height: 600,
Format: imgcache.FormatWebP,
@@ -54,48 +43,38 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
if parsed.SourceHost != payload.SourceHost {
t.Errorf("SourceHost = %q, want %q", parsed.SourceHost, payload.SourceHost)
}
if parsed.SourcePath != payload.SourcePath {
t.Errorf("SourcePath = %q, want %q", parsed.SourcePath, payload.SourcePath)
}
if parsed.SourceQuery != payload.SourceQuery {
t.Errorf("SourceQuery = %q, want %q", parsed.SourceQuery, payload.SourceQuery)
}
if parsed.Width != payload.Width {
t.Errorf("Width = %d, want %d", parsed.Width, payload.Width)
}
if parsed.Height != payload.Height {
t.Errorf("Height = %d, want %d", parsed.Height, payload.Height)
}
if parsed.Format != payload.Format {
t.Errorf("Format = %q, want %q", parsed.Format, payload.Format)
}
if parsed.Quality != payload.Quality {
t.Errorf("Quality = %d, want %d", parsed.Quality, payload.Quality)
}
if parsed.FitMode != payload.FitMode {
t.Errorf("FitMode = %q, want %q", parsed.FitMode, payload.FitMode)
}
if parsed.ExpiresAt != payload.ExpiresAt {
t.Errorf("ExpiresAt = %d, want %d", parsed.ExpiresAt, payload.ExpiresAt)
}
}
func TestGenerator_Parse_Expired(t *testing.T) {
t.Parallel()
gen, _ := NewGenerator("test-signing-key-12345")
gen, _ := encurl.NewGenerator("test-signing-key-12345")
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
payload := &Payload{
SourceHost: "cdn.example.com",
SourcePath: "/images/photo.jpg",
ExpiresAt: time.Now().Add(-time.Hour).Unix(), // Already expired
}
@@ -109,15 +88,13 @@ func TestGenerator_Parse_Expired(t *testing.T) {
t.Error("Parse() should fail for expired token")
}
if !errors.Is(err, encurl.ErrExpired) {
t.Errorf("Parse() error = %v, want %v", err, encurl.ErrExpired)
if err != ErrExpired {
t.Errorf("Parse() error = %v, want %v", err, ErrExpired)
}
}
func TestGenerator_Parse_InvalidToken(t *testing.T) {
t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345")
gen, _ := NewGenerator("test-signing-key-12345")
_, err := gen.Parse("not-a-valid-token")
if err == nil {
@@ -126,13 +103,11 @@ func TestGenerator_Parse_InvalidToken(t *testing.T) {
}
func TestGenerator_Parse_TamperedToken(t *testing.T) {
t.Parallel()
gen, _ := NewGenerator("test-signing-key-12345")
gen, _ := encurl.NewGenerator("test-signing-key-12345")
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
payload := &Payload{
SourceHost: "cdn.example.com",
SourcePath: "/images/photo.jpg",
ExpiresAt: time.Now().Add(time.Hour).Unix(),
}
@@ -151,14 +126,12 @@ func TestGenerator_Parse_TamperedToken(t *testing.T) {
}
func TestGenerator_Parse_WrongKey(t *testing.T) {
t.Parallel()
gen1, _ := NewGenerator("signing-key-1")
gen2, _ := NewGenerator("signing-key-2")
gen1, _ := encurl.NewGenerator("signing-key-1")
gen2, _ := encurl.NewGenerator("signing-key-2")
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
payload := &Payload{
SourceHost: "cdn.example.com",
SourcePath: "/images/photo.jpg",
ExpiresAt: time.Now().Add(time.Hour).Unix(),
}
@@ -171,12 +144,10 @@ func TestGenerator_Parse_WrongKey(t *testing.T) {
}
func TestPayload_ToImageRequest(t *testing.T) {
t.Parallel()
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
SourceQuery: testSourceQuery,
payload := &Payload{
SourceHost: "cdn.example.com",
SourcePath: "/images/photo.jpg",
SourceQuery: "v=2",
Width: 800,
Height: 600,
Format: imgcache.FormatWebP,
@@ -190,68 +161,55 @@ func TestPayload_ToImageRequest(t *testing.T) {
if req.SourceHost != payload.SourceHost {
t.Errorf("SourceHost = %q, want %q", req.SourceHost, payload.SourceHost)
}
if req.SourcePath != payload.SourcePath {
t.Errorf("SourcePath = %q, want %q", req.SourcePath, payload.SourcePath)
}
if req.SourceQuery != payload.SourceQuery {
t.Errorf("SourceQuery = %q, want %q", req.SourceQuery, payload.SourceQuery)
}
if req.Size.Width != payload.Width {
t.Errorf("Width = %d, want %d", req.Size.Width, payload.Width)
}
if req.Size.Height != payload.Height {
t.Errorf("Height = %d, want %d", req.Size.Height, payload.Height)
}
if req.Format != payload.Format {
t.Errorf("Format = %q, want %q", req.Format, payload.Format)
}
if req.Quality != payload.Quality {
t.Errorf("Quality = %d, want %d", req.Quality, payload.Quality)
}
if req.FitMode != payload.FitMode {
t.Errorf("FitMode = %q, want %q", req.FitMode, payload.FitMode)
}
}
func TestPayload_ToImageRequest_Defaults(t *testing.T) {
t.Parallel()
// Payload with only required fields - should get defaults
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
payload := &Payload{
SourceHost: "cdn.example.com",
SourcePath: "/images/photo.jpg",
ExpiresAt: time.Now().Add(time.Hour).Unix(),
}
req := payload.ToImageRequest()
if req.Format != encurl.DefaultFormat {
t.Errorf("Format = %q, want default %q", req.Format, encurl.DefaultFormat)
if req.Format != DefaultFormat {
t.Errorf("Format = %q, want default %q", req.Format, DefaultFormat)
}
if req.Quality != encurl.DefaultQuality {
t.Errorf("Quality = %d, want default %d", req.Quality, encurl.DefaultQuality)
if req.Quality != DefaultQuality {
t.Errorf("Quality = %d, want default %d", req.Quality, DefaultQuality)
}
if req.FitMode != encurl.DefaultFitMode {
t.Errorf("FitMode = %q, want default %q", req.FitMode, encurl.DefaultFitMode)
if req.FitMode != DefaultFitMode {
t.Errorf("FitMode = %q, want default %q", req.FitMode, DefaultFitMode)
}
}
func TestFromImageRequest(t *testing.T) {
t.Parallel()
req := &imgcache.ImageRequest{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
SourceQuery: testSourceQuery,
SourceHost: "cdn.example.com",
SourcePath: "/images/photo.jpg",
SourceQuery: "v=2",
Size: imgcache.Size{Width: 800, Height: 600},
Format: imgcache.FormatWebP,
Quality: 90,
@@ -259,62 +217,52 @@ func TestFromImageRequest(t *testing.T) {
}
expiresAt := time.Now().Add(time.Hour)
payload := encurl.FromImageRequest(req, expiresAt)
payload := FromImageRequest(req, expiresAt)
if payload.SourceHost != req.SourceHost {
t.Errorf("SourceHost = %q, want %q", payload.SourceHost, req.SourceHost)
}
if payload.SourcePath != req.SourcePath {
t.Errorf("SourcePath = %q, want %q", payload.SourcePath, req.SourcePath)
}
if payload.Width != req.Size.Width {
t.Errorf("Width = %d, want %d", payload.Width, req.Size.Width)
}
if payload.ExpiresAt != expiresAt.Unix() {
t.Errorf("ExpiresAt = %d, want %d", payload.ExpiresAt, expiresAt.Unix())
}
}
func TestFromImageRequest_OmitsDefaults(t *testing.T) {
t.Parallel()
// Request with default values - payload should omit them for
// smaller encoding
// Request with default values - payload should omit them for smaller encoding
req := &imgcache.ImageRequest{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
Format: encurl.DefaultFormat,
Quality: encurl.DefaultQuality,
FitMode: encurl.DefaultFitMode,
SourceHost: "cdn.example.com",
SourcePath: "/images/photo.jpg",
Format: DefaultFormat,
Quality: DefaultQuality,
FitMode: DefaultFitMode,
}
payload := encurl.FromImageRequest(req, time.Now().Add(time.Hour))
payload := FromImageRequest(req, time.Now().Add(time.Hour))
// These should be zero/empty because they match defaults
if payload.Format != "" {
t.Errorf("Format should be empty for default, got %q", payload.Format)
}
if payload.Quality != 0 {
t.Errorf("Quality should be 0 for default, got %d", payload.Quality)
}
if payload.FitMode != "" {
t.Errorf("FitMode should be empty for default, got %q", payload.FitMode)
}
}
func TestGenerator_TokenIsURLSafe(t *testing.T) {
t.Parallel()
gen, _ := NewGenerator("test-signing-key-12345")
gen, _ := encurl.NewGenerator("test-signing-key-12345")
payload := &encurl.Payload{
SourceHost: testSourceHost,
SourcePath: testSourcePath,
payload := &Payload{
SourceHost: "cdn.example.com",
SourcePath: "/images/photo.jpg",
ExpiresAt: time.Now().Add(time.Hour).Unix(),
}

141
internal/handlers/auth.go
View File

@@ -35,8 +35,7 @@ func (s *Handlers) HandleRoot() http.HandlerFunc {
// handleLoginPost handles login form submission.
func (s *Handlers) handleLoginPost(w http.ResponseWriter, r *http.Request) {
err := r.ParseForm()
if err != nil {
if err := r.ParseForm(); err != nil {
s.renderLogin(w, "Invalid form data")
return
@@ -53,8 +52,7 @@ func (s *Handlers) handleLoginPost(w http.ResponseWriter, r *http.Request) {
}
// Create session
err = s.sessMgr.CreateSession(w)
if err != nil {
if err := s.sessMgr.CreateSession(w); err != nil {
s.log.Error("failed to create session", "error", err)
s.renderLogin(w, "Failed to create session")
@@ -85,14 +83,20 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
return
}
err := r.ParseForm()
if err != nil {
if err := r.ParseForm(); err != nil {
s.renderGenerator(w, &generatorData{Error: "Invalid form data"})
return
}
// Parse form values
sourceURL := r.FormValue("url")
widthStr := r.FormValue("width")
heightStr := r.FormValue("height")
format := r.FormValue("format")
qualityStr := r.FormValue("quality")
fit := r.FormValue("fit")
ttlStr := r.FormValue("ttl")
// Validate source URL
parsed, err := url.Parse(sourceURL)
@@ -102,7 +106,38 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
return
}
payload, expiresAt, ttl := buildGeneratePayload(parsed, r.Form)
// Parse dimensions
width, _ := strconv.Atoi(widthStr)
height, _ := strconv.Atoi(heightStr)
quality, _ := strconv.Atoi(qualityStr)
ttl, _ := strconv.Atoi(ttlStr)
if quality <= 0 {
quality = 85
}
// Create payload
// ttl=0 means never expires
var expiresAt time.Time
var expiresAtUnix int64
if ttl > 0 {
expiresAt = time.Now().Add(time.Duration(ttl) * time.Second)
expiresAtUnix = expiresAt.Unix()
}
// else expiresAtUnix stays 0 (never expires)
payload := &encurl.Payload{
SourceHost: parsed.Host,
SourcePath: parsed.Path,
SourceQuery: parsed.RawQuery,
Width: width,
Height: height,
Format: imgcache.ImageFormat(format),
Quality: quality,
FitMode: imgcache.FitMode(fit),
ExpiresAt: expiresAtUnix,
}
// Generate encrypted token
token, err := s.encGen.Generate(payload)
@@ -113,7 +148,20 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
return
}
generatedURL := s.buildGeneratedURL(r, token, r.FormValue("format"))
// Build full URL (URL-encode the token for safety)
scheme := "https"
if s.config.Debug {
scheme = "http"
}
// Determine file extension for the trailing filename
ext := format
if ext == "" || ext == "orig" {
ext = "jpg" // Default extension
}
host := r.Host
generatedURL := scheme + "://" + host + "/v1/e/" + url.PathEscape(token) + "/img." + ext
// Format expiry for display
expiresAtStr := "Never"
@@ -125,55 +173,16 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
GeneratedURL: generatedURL,
ExpiresAt: expiresAtStr,
FormURL: sourceURL,
FormWidth: r.FormValue("width"),
FormHeight: r.FormValue("height"),
FormFormat: r.FormValue("format"),
FormQuality: r.FormValue("quality"),
FormFit: r.FormValue("fit"),
FormTTL: r.FormValue("ttl"),
FormWidth: widthStr,
FormHeight: heightStr,
FormFormat: format,
FormQuality: qualityStr,
FormFit: fit,
FormTTL: ttlStr,
})
}
}
// buildGeneratePayload parses the numeric form fields and assembles the
// encrypted URL payload. ttl=0 means never expires (ExpiresAt stays 0).
func buildGeneratePayload(
parsed *url.URL, form url.Values,
) (*encurl.Payload, time.Time, int) {
width, _ := strconv.Atoi(form.Get("width"))
height, _ := strconv.Atoi(form.Get("height"))
quality, _ := strconv.Atoi(form.Get("quality"))
ttl, _ := strconv.Atoi(form.Get("ttl"))
if quality <= 0 {
quality = 85
}
var (
expiresAt time.Time
expiresAtUnix int64
)
if ttl > 0 {
expiresAt = time.Now().Add(time.Duration(ttl) * time.Second)
expiresAtUnix = expiresAt.Unix()
}
payload := &encurl.Payload{
SourceHost: parsed.Host,
SourcePath: parsed.Path,
SourceQuery: parsed.RawQuery,
Width: width,
Height: height,
Format: imgcache.ImageFormat(form.Get("format")),
Quality: quality,
FitMode: imgcache.FitMode(form.Get("fit")),
ExpiresAt: expiresAtUnix,
}
return payload, expiresAt, ttl
}
// generatorData holds template data for the generator page.
type generatorData struct {
GeneratedURL string
@@ -197,8 +206,7 @@ func (s *Handlers) renderLogin(w http.ResponseWriter, errorMsg string) {
Error: errorMsg,
}
err := templates.Render(w, "login.html", data)
if err != nil {
if err := templates.Render(w, "login.html", data); err != nil {
s.log.Error("failed to render login template", "error", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
}
@@ -211,16 +219,13 @@ func (s *Handlers) renderGenerator(w http.ResponseWriter, data *generatorData) {
data = &generatorData{}
}
err := templates.Render(w, "generator.html", data)
if err != nil {
if err := templates.Render(w, "generator.html", data); err != nil {
s.log.Error("failed to render generator template", "error", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
}
}
func (s *Handlers) renderGeneratorWithForm(
w http.ResponseWriter, errorMsg string, form url.Values,
) {
func (s *Handlers) renderGeneratorWithForm(w http.ResponseWriter, errorMsg string, form url.Values) {
s.renderGenerator(w, &generatorData{
Error: errorMsg,
FormURL: form.Get("url"),
@@ -232,19 +237,3 @@ func (s *Handlers) renderGeneratorWithForm(
FormTTL: form.Get("ttl"),
})
}
func (s *Handlers) buildGeneratedURL(r *http.Request, token, format string) string {
// Build full URL (URL-encode the token for safety)
scheme := "https"
if s.config.Debug {
scheme = "http"
}
// Determine file extension for the trailing filename
ext := format
if ext == "" || ext == "orig" {
ext = "jpg" // Default extension
}
return scheme + "://" + r.Host + "/v1/e/" + url.PathEscape(token) + "/img." + ext
}

40
internal/handlers/handlers.go
View File

@@ -22,7 +22,6 @@ import (
// Params defines dependencies for Handlers.
type Params struct {
fx.In
Logger *logger.Logger
Healthcheck *healthcheck.Healthcheck
Database *database.Database
@@ -51,24 +50,9 @@ func New(lc fx.Lifecycle, params Params) (*Handlers, error) {
}
lc.Append(fx.Hook{
// The eviction goroutine must outlive OnStart, so it cannot
// inherit this hook's context. It makes its own instead, which
// leaves it uncancellable: an in-flight pass runs to completion
// during OnStop regardless of the shutdown deadline. Making the
// loop cancellable changes shutdown semantics and is tracked
// separately in issue #102, rather than being folded into the
// lint-conformance change that surfaced it.
//nolint:contextcheck // see issue #102
OnStart: func(_ context.Context) error {
return s.initImageService()
},
OnStop: func(_ context.Context) error {
if s.imgCache != nil {
s.imgCache.StopEviction()
}
return nil
},
})
return s, nil
@@ -76,15 +60,11 @@ func New(lc fx.Lifecycle, params Params) (*Handlers, error) {
// initImageService initializes the image cache and service.
func (s *Handlers) initImageService() error {
// Create the cache. cache_max_bytes: 0 disables the disk cache
// entirely; any other value is the eviction limit in bytes.
// Create the cache
cache, err := imgcache.NewCache(s.db.DB(), imgcache.CacheConfig{
StateDir: s.config.StateDir,
CacheTTL: imgcache.DefaultCacheTTL,
NegativeTTL: imgcache.DefaultNegativeTTL,
MaxBytes: s.config.CacheMaxBytes,
DisableDiskCache: s.config.CacheMaxBytes == 0,
Logger: s.log,
StateDir: s.config.StateDir,
CacheTTL: imgcache.DefaultCacheTTL,
NegativeTTL: imgcache.DefaultNegativeTTL,
})
if err != nil {
return err
@@ -92,14 +72,9 @@ func (s *Handlers) initImageService() error {
s.imgCache = cache
// Background eviction: startup reconciliation, then periodic and
// write-pressure passes. No-op when the disk cache is disabled.
cache.StartEviction(imgcache.DefaultEvictionInterval)
// Create the fetcher config
fetcherCfg := httpfetcher.DefaultConfig()
fetcherCfg.AllowHTTP = s.config.AllowHTTP
if s.config.UpstreamConnectionsPerHost > 0 {
fetcherCfg.MaxConnectionsPerHost = s.config.UpstreamConnectionsPerHost
}
@@ -125,7 +100,6 @@ func (s *Handlers) initImageService() error {
if err != nil {
return err
}
s.sessMgr = sessMgr
// Initialize encrypted URL generator
@@ -133,7 +107,6 @@ func (s *Handlers) initImageService() error {
if err != nil {
return err
}
s.encGen = encGen
s.log.Info("session manager and URL generator initialized")
@@ -141,10 +114,9 @@ func (s *Handlers) initImageService() error {
return nil
}
func (s *Handlers) respondJSON(w http.ResponseWriter, data any, status int) {
func (s *Handlers) respondJSON(w http.ResponseWriter, data interface{}, status int) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
if data != nil {
err := json.NewEncoder(w).Encode(data)
if err != nil {
@@ -154,7 +126,7 @@ func (s *Handlers) respondJSON(w http.ResponseWriter, data any, status int) {
}
func (s *Handlers) respondError(w http.ResponseWriter, message string, status int) {
s.respondJSON(w, map[string]any{
s.respondJSON(w, map[string]interface{}{
"error": message,
"status": status,
"timestamp": time.Now().UTC().Format(time.RFC3339),

46
internal/handlers/handlers_internal_test.go → internal/handlers/handlers_test.go
View File

@@ -83,8 +83,7 @@ func setupTestDB(t *testing.T) *sql.DB {
t.Fatalf("failed to open test db: %v", err)
}
err = database.ApplyMigrations(context.Background(), db, nil)
if err != nil {
if err := database.ApplyMigrations(context.Background(), db, nil); err != nil {
t.Fatalf("failed to apply migrations: %v", err)
}
@@ -95,16 +94,14 @@ func generateTestJPEG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 85})
if err != nil {
if err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 85}); err != nil {
t.Fatalf("failed to encode test JPEG: %v", err)
}
@@ -120,9 +117,7 @@ func newMockFetcher(fs fs.FS) *mockFetcher {
return &mockFetcher{fs: fs}
}
func (f *mockFetcher) Fetch(
_ context.Context, url string,
) (*httpfetcher.FetchResult, error) {
func (f *mockFetcher) Fetch(ctx context.Context, url string) (*httpfetcher.FetchResult, error) {
// Remove https:// prefix
path := url[8:] // Remove "https://"
@@ -139,16 +134,13 @@ func (f *mockFetcher) Fetch(
}
func TestHandleImage_HEAD_ReturnsHeadersOnly(t *testing.T) {
t.Parallel()
fix := setupTestHandler(t)
// Create a chi router to properly handle wildcards
r := chi.NewRouter()
r.Head("/v1/image/*", fix.handler.HandleImage())
req := httptest.NewRequestWithContext(t.Context(), http.MethodHead,
"/v1/image/"+fix.goodHost+"/images/photo.jpg/50x50.jpeg", nil)
req := httptest.NewRequest(http.MethodHead, "/v1/image/"+fix.goodHost+"/images/photo.jpg/50x50.jpeg", nil)
rec := httptest.NewRecorder()
r.ServeHTTP(rec, req)
@@ -175,16 +167,13 @@ func TestHandleImage_HEAD_ReturnsHeadersOnly(t *testing.T) {
}
func TestHandleImage_ConditionalRequest_IfNoneMatch_Returns304(t *testing.T) {
t.Parallel()
fix := setupTestHandler(t)
r := chi.NewRouter()
r.Get("/v1/image/*", fix.handler.HandleImage())
// First request to get the ETag
req1 := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"/v1/image/"+fix.goodHost+"/images/photo.jpg/50x50.jpeg", nil)
req1 := httptest.NewRequest(http.MethodGet, "/v1/image/"+fix.goodHost+"/images/photo.jpg/50x50.jpeg", nil)
rec1 := httptest.NewRecorder()
r.ServeHTTP(rec1, req1)
@@ -199,18 +188,15 @@ func TestHandleImage_ConditionalRequest_IfNoneMatch_Returns304(t *testing.T) {
}
// Second request with If-None-Match header
req2 := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"/v1/image/"+fix.goodHost+"/images/photo.jpg/50x50.jpeg", nil)
req2 := httptest.NewRequest(http.MethodGet, "/v1/image/"+fix.goodHost+"/images/photo.jpg/50x50.jpeg", nil)
req2.Header.Set("If-None-Match", etag)
rec2 := httptest.NewRecorder()
r.ServeHTTP(rec2, req2)
// Should return 304 Not Modified
if rec2.Code != http.StatusNotModified {
t.Errorf("Conditional request status = %d, want %d",
rec2.Code, http.StatusNotModified)
t.Errorf("Conditional request status = %d, want %d", rec2.Code, http.StatusNotModified)
}
// Body should be empty for 304 response
@@ -220,26 +206,21 @@ func TestHandleImage_ConditionalRequest_IfNoneMatch_Returns304(t *testing.T) {
}
func TestHandleImage_ConditionalRequest_IfNoneMatch_DifferentETag(t *testing.T) {
t.Parallel()
fix := setupTestHandler(t)
r := chi.NewRouter()
r.Get("/v1/image/*", fix.handler.HandleImage())
// Request with non-matching ETag
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"/v1/image/"+fix.goodHost+"/images/photo.jpg/50x50.jpeg", nil)
req := httptest.NewRequest(http.MethodGet, "/v1/image/"+fix.goodHost+"/images/photo.jpg/50x50.jpeg", nil)
req.Header.Set("If-None-Match", `"different-etag"`)
rec := httptest.NewRecorder()
r.ServeHTTP(rec, req)
// Should return 200 OK with full response
if rec.Code != http.StatusOK {
t.Errorf("Request with non-matching ETag status = %d, want %d",
rec.Code, http.StatusOK)
t.Errorf("Request with non-matching ETag status = %d, want %d", rec.Code, http.StatusOK)
}
// Body should not be empty
@@ -249,15 +230,12 @@ func TestHandleImage_ConditionalRequest_IfNoneMatch_DifferentETag(t *testing.T)
}
func TestHandleImage_ETagHeader(t *testing.T) {
t.Parallel()
fix := setupTestHandler(t)
r := chi.NewRouter()
r.Get("/v1/image/*", fix.handler.HandleImage())
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
"/v1/image/"+fix.goodHost+"/images/photo.jpg/50x50.jpeg", nil)
req := httptest.NewRequest(http.MethodGet, "/v1/image/"+fix.goodHost+"/images/photo.jpg/50x50.jpeg", nil)
rec := httptest.NewRecorder()
r.ServeHTTP(rec, req)

287
internal/handlers/image.go
View File

@@ -16,14 +16,64 @@ import (
// /v1/image/<host>/<path>/<width>x<height>.<format>
func (s *Handlers) HandleImage() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
req, ok := s.parseImageRequest(w, r)
if !ok {
ctx := r.Context()
// Get the wildcard path from chi
pathParam := chi.URLParam(r, "*")
// Parse the URL path
parsed, err := imgcache.ParseImagePath(pathParam)
if err != nil {
s.log.Warn("failed to parse image URL",
"path", pathParam,
"error", err,
)
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
return
}
// Convert to ImageRequest
req := parsed.ToImageRequest()
// Parse signature params from query string
query := r.URL.Query()
req.Signature = query.Get("sig")
if expStr := query.Get("exp"); expStr != "" {
if exp, err := strconv.ParseInt(expStr, 10, 64); err == nil {
req.Expires = time.Unix(exp, 0)
}
}
// Parse optional quality and fit params
if qStr := query.Get("q"); qStr != "" {
if q, err := strconv.Atoi(qStr); err == nil && q > 0 && q <= 100 {
req.Quality = q
}
}
if fit := query.Get("fit"); fit != "" {
req.FitMode = imgcache.FitMode(fit)
if err := imgcache.ValidateFitMode(req.FitMode); err != nil {
s.respondError(w, "invalid fit mode: "+fit, http.StatusBadRequest)
return
}
}
// Default quality if not set
if req.Quality == 0 {
req.Quality = 85
}
// Default fit mode if not set
if req.FitMode == "" {
req.FitMode = imgcache.FitCover
}
// Validate signature if required
err := s.imgSvc.ValidateRequest(req)
if err != nil {
if err := s.imgSvc.ValidateRequest(req); err != nil {
s.log.Warn("signature validation failed",
"host", req.SourceHost,
"path", req.SourcePath,
@@ -39,17 +89,83 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
// Get the image (from cache or fetch/process)
startTime := time.Now()
resp, err := s.imgSvc.Get(r.Context(), req)
resp, err := s.imgSvc.Get(ctx, req)
if err != nil {
s.respondImageError(w, req, err)
s.log.Error("failed to get image",
"host", req.SourceHost,
"path", req.SourcePath,
"error", err,
)
// Check for specific error types
if errors.Is(err, httpfetcher.ErrSSRFBlocked) {
s.respondError(w, "forbidden", http.StatusForbidden)
return
}
if errors.Is(err, httpfetcher.ErrUpstreamError) {
s.respondError(w, "upstream error", http.StatusBadGateway)
return
}
s.respondError(w, "internal error", http.StatusInternalServerError)
return
}
defer func() { _ = resp.Content.Close() }()
// Set response headers
w.Header().Set("Content-Type", resp.ContentType)
if resp.ContentLength > 0 {
w.Header().Set("Content-Length", strconv.FormatInt(resp.ContentLength, 10))
}
// Cache control headers
w.Header().Set("Cache-Control", "public, max-age=31536000, immutable")
w.Header().Set("X-Pixa-Cache", string(resp.CacheStatus))
if resp.ETag != "" {
w.Header().Set("ETag", resp.ETag)
// Check for conditional request (If-None-Match)
if ifNoneMatch := r.Header.Get("If-None-Match"); ifNoneMatch != "" {
if ifNoneMatch == resp.ETag {
w.WriteHeader(http.StatusNotModified)
return
}
}
}
// Handle HEAD request - return headers only
if r.Method == http.MethodHead {
w.WriteHeader(http.StatusOK)
return
}
defer func() { _ = resp.Content.Close() }()
// Stream the response
w.WriteHeader(http.StatusOK)
s.writeImageResponse(w, r, req, resp, cacheKey, startTime)
servedBytes, err := io.Copy(w, resp.Content)
if err != nil {
s.log.Error("failed to write response",
"error", err,
)
}
// Log cache status and timing after serving
duration := time.Since(startTime)
s.log.Info("image served",
"cache_key", cacheKey,
"cache_status", resp.CacheStatus,
"duration_ms", duration.Milliseconds(),
"format", req.Format,
"served_bytes", servedBytes,
"fetched_bytes", resp.FetchedBytes,
)
}
}
@@ -64,156 +180,3 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
_, _ = w.Write(robotsTxt)
}
}
// parseImageRequest parses the wildcard path and query parameters into
// an ImageRequest. On invalid input it writes an error response and
// returns false.
func (s *Handlers) parseImageRequest(
w http.ResponseWriter, r *http.Request,
) (*imgcache.ImageRequest, bool) {
// Get the wildcard path from chi
pathParam := chi.URLParam(r, "*")
// Parse the URL path
parsed, err := imgcache.ParseImagePath(pathParam)
if err != nil {
s.log.Warn("failed to parse image URL",
"path", pathParam,
"error", err,
)
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
return nil, false
}
// Convert to ImageRequest
req := parsed.ToImageRequest()
// Parse signature params from query string
query := r.URL.Query()
req.Signature = query.Get("sig")
if expStr := query.Get("exp"); expStr != "" {
exp, parseErr := strconv.ParseInt(expStr, 10, 64)
if parseErr == nil {
req.Expires = time.Unix(exp, 0)
}
}
// Parse optional quality and fit params
if qStr := query.Get("q"); qStr != "" {
q, parseErr := strconv.Atoi(qStr)
if parseErr == nil && q > 0 && q <= 100 {
req.Quality = q
}
}
if fit := query.Get("fit"); fit != "" {
req.FitMode = imgcache.FitMode(fit)
fitErr := imgcache.ValidateFitMode(req.FitMode)
if fitErr != nil {
s.respondError(w, "invalid fit mode: "+fit, http.StatusBadRequest)
return nil, false
}
}
// Default quality if not set
if req.Quality == 0 {
req.Quality = 85
}
// Default fit mode if not set
if req.FitMode == "" {
req.FitMode = imgcache.FitCover
}
return req, true
}
// respondImageError maps image retrieval errors to HTTP responses.
func (s *Handlers) respondImageError(
w http.ResponseWriter, req *imgcache.ImageRequest, err error,
) {
s.log.Error("failed to get image",
"host", req.SourceHost,
"path", req.SourcePath,
"error", err,
)
// Check for specific error types
if errors.Is(err, httpfetcher.ErrSSRFBlocked) {
s.respondError(w, "forbidden", http.StatusForbidden)
return
}
if errors.Is(err, httpfetcher.ErrUpstreamError) {
s.respondError(w, "upstream error", http.StatusBadGateway)
return
}
s.respondError(w, "internal error", http.StatusInternalServerError)
}
// writeImageResponse writes headers and streams the image content,
// handling conditional and HEAD requests.
func (s *Handlers) writeImageResponse(
w http.ResponseWriter, r *http.Request,
req *imgcache.ImageRequest, resp *imgcache.ImageResponse,
cacheKey imgcache.VariantKey, startTime time.Time,
) {
// Set response headers
w.Header().Set("Content-Type", resp.ContentType)
if resp.ContentLength > 0 {
w.Header().Set("Content-Length", strconv.FormatInt(resp.ContentLength, 10))
}
// Cache control headers
w.Header().Set("Cache-Control", "public, max-age=31536000, immutable")
w.Header().Set("X-Pixa-Cache", string(resp.CacheStatus))
if resp.ETag != "" {
w.Header().Set("ETag", resp.ETag)
// Check for conditional request (If-None-Match)
if ifNoneMatch := r.Header.Get("If-None-Match"); ifNoneMatch != "" {
if ifNoneMatch == resp.ETag {
w.WriteHeader(http.StatusNotModified)
return
}
}
}
// Handle HEAD request - return headers only
if r.Method == http.MethodHead {
w.WriteHeader(http.StatusOK)
return
}
// Stream the response
w.WriteHeader(http.StatusOK)
servedBytes, err := io.Copy(w, resp.Content)
if err != nil {
s.log.Error("failed to write response",
"error", err,
)
}
// Log cache status and timing after serving
duration := time.Since(startTime)
s.log.Info("image served",
"cache_key", cacheKey,
"cache_status", resp.CacheStatus,
"duration_ms", duration.Milliseconds(),
"format", req.Format,
"served_bytes", servedBytes,
"fetched_bytes", resp.FetchedBytes,
)
}

10
internal/handlers/imageenc.go
View File

@@ -15,9 +15,8 @@ import (
"sneak.berlin/go/pixa/internal/imgcache"
)
// HandleImageEnc handles requests to /v1/e/{token}/* for encrypted
// image URLs. The trailing path (e.g., /img.jpg) is ignored but helps
// browsers identify the content type.
// HandleImageEnc handles requests to /v1/e/{token}/* for encrypted image URLs.
// The trailing path (e.g., /img.jpg) is ignored but helps browsers identify the content type.
func (s *Handlers) HandleImageEnc() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
@@ -58,20 +57,17 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
"format", req.Format,
)
// Fetch and process the image (no signature validation
// needed - encrypted URL is trusted)
// Fetch and process the image (no signature validation needed - encrypted URL is trusted)
resp, err := s.imgSvc.Get(ctx, req)
if err != nil {
s.handleImageError(w, err)
return
}
defer func() { _ = resp.Content.Close() }()
// Set response headers
w.Header().Set("Content-Type", resp.ContentType)
if resp.ContentLength > 0 {
w.Header().Set("Content-Length", strconv.FormatInt(resp.ContentLength, 10))
}

11
internal/healthcheck/healthcheck.go
View File

@@ -16,7 +16,6 @@ import (
// Params defines dependencies for Healthcheck.
type Params struct {
fx.In
Globals *globals.Globals
Config *config.Config
Logger *logger.Logger
@@ -54,8 +53,6 @@ func New(lc fx.Lifecycle, params Params) (*Healthcheck, error) {
}
// Response is the JSON response for health checks.
//
//nolint:tagliatelle // health endpoint response format uses snake_case
type Response struct {
Status string `json:"status"`
Now string `json:"now"`
@@ -66,6 +63,10 @@ type Response struct {
Maintenance bool `json:"maintenance_mode"`
}
func (s *Healthcheck) uptime() time.Duration {
return time.Since(s.StartupTime)
}
// Healthcheck returns the current health status.
func (s *Healthcheck) Healthcheck() *Response {
resp := &Response{
@@ -80,7 +81,3 @@ func (s *Healthcheck) Healthcheck() *Response {
return resp
}
func (s *Healthcheck) uptime() time.Duration {
return time.Since(s.StartupTime)
}

133
internal/httpfetcher/httpfetcher.go
View File

@@ -12,7 +12,6 @@ import (
"net/http"
"net/http/httptrace"
neturl "net/url"
"slices"
"strings"
"sync"
"time"
@@ -29,23 +28,6 @@ const (
DefaultMaxConnectionsPerHost = 20
)
// MIME content types.
const (
contentTypeJPEG = "image/jpeg"
contentTypePNG = "image/png"
contentTypeGIF = "image/gif"
contentTypeWebP = "image/webp"
contentTypeAVIF = "image/avif"
contentTypeSVG = "image/svg+xml"
contentTypeOctetStream = "application/octet-stream"
)
// Loopback addresses blocked by SSRF protection.
const (
localhostIPv4 = "127.0.0.1"
localhostIPv6 = "::1"
)
// Fetcher errors.
var (
ErrSSRFBlocked = errors.New("request blocked: private or internal IP")
@@ -57,12 +39,6 @@ var (
ErrUpstreamTimeout = errors.New("upstream request timeout")
)
// Internal fetcher errors.
var (
errTooManyRedirects = errors.New("too many redirects")
errConnectFailed = errors.New("failed to connect")
)
// Fetcher retrieves content from upstream origins.
type Fetcher interface {
// Fetch retrieves content from the given URL.
@@ -116,12 +92,12 @@ func DefaultConfig() *Config {
MaxResponseSize: DefaultMaxResponseSize,
UserAgent: "pixa/1.0",
AllowedContentTypes: []string{
contentTypeJPEG,
contentTypePNG,
contentTypeGIF,
contentTypeWebP,
contentTypeAVIF,
contentTypeSVG,
"image/jpeg",
"image/png",
"image/gif",
"image/webp",
"image/avif",
"image/svg+xml",
},
AllowHTTP: false,
MaxConnectionsPerHost: DefaultMaxConnectionsPerHost,
@@ -156,12 +132,10 @@ func New(config *Config) *HTTPFetcher {
// Don't follow redirects automatically - we need to validate each hop
CheckRedirect: func(req *http.Request, via []*http.Request) error {
if len(via) >= DefaultMaxRedirects {
return errTooManyRedirects
return errors.New("too many redirects")
}
// Validate the redirect target
err := validateURL(req.Context(), req.URL.String(), config.AllowHTTP)
if err != nil {
if err := validateURL(req.URL.String(), config.AllowHTTP); err != nil {
return fmt.Errorf("redirect blocked: %w", err)
}
@@ -176,11 +150,24 @@ func New(config *Config) *HTTPFetcher {
}
}
// getHostSemaphore returns the semaphore for a host, creating it if necessary.
func (f *HTTPFetcher) getHostSemaphore(host string) chan struct{} {
f.hostSemMu.Lock()
defer f.hostSemMu.Unlock()
sem, ok := f.hostSems[host]
if !ok {
sem = make(chan struct{}, f.config.MaxConnectionsPerHost)
f.hostSems[host] = sem
}
return sem
}
// Fetch retrieves content from the given URL with SSRF protection.
func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, error) {
// Validate URL before making request
err := validateURL(ctx, url, f.config.AllowHTTP)
if err != nil {
if err := validateURL(url, f.config.AllowHTTP); err != nil {
return nil, err
}
@@ -198,7 +185,6 @@ func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
// If we fail before returning a result, release the slot
success := false
defer func() {
if !success {
<-sem
@@ -215,6 +201,7 @@ func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
URL: parsedURL,
Header: make(http.Header),
}
req = req.WithContext(ctx)
req.Header.Set("User-Agent", f.config.UserAgent)
req.Header.Set("Accept", strings.Join(f.config.AllowedContentTypes, ", "))
@@ -229,10 +216,11 @@ func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
}
},
}
req = req.WithContext(httptrace.WithClientTrace(ctx, trace))
req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
startTime := time.Now()
//nolint:gosec // G704: URL validated by validateURL() above
resp, err := f.client.Do(req)
fetchDuration := time.Since(startTime)
@@ -245,39 +233,6 @@ func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
return nil, fmt.Errorf("upstream request failed: %w", err)
}
result, err := f.buildResult(resp, remoteAddr, fetchDuration, sem)
if err != nil {
return nil, err
}
// Mark success so defer doesn't release the semaphore
success = true
return result, nil
}
// getHostSemaphore returns the semaphore for a host, creating it if necessary.
func (f *HTTPFetcher) getHostSemaphore(host string) chan struct{} {
f.hostSemMu.Lock()
defer f.hostSemMu.Unlock()
sem, ok := f.hostSems[host]
if !ok {
sem = make(chan struct{}, f.config.MaxConnectionsPerHost)
f.hostSems[host] = sem
}
return sem
}
// buildResult validates the upstream response and assembles a FetchResult
// whose Content releases the host semaphore slot when closed.
func (f *HTTPFetcher) buildResult(
resp *http.Response,
remoteAddr string,
fetchDuration time.Duration,
sem chan struct{},
) (*FetchResult, error) {
// Extract HTTP version (strip "HTTP/" prefix)
httpVersion := strings.TrimPrefix(resp.Proto, "HTTP/")
@@ -310,6 +265,9 @@ func (f *HTTPFetcher) buildResult(
remaining: f.config.MaxResponseSize,
}
// Mark success so defer doesn't release the semaphore
success = true
return &FetchResult{
Content: &semaphoreReleasingReadCloser{limitedBody, resp.Body, sem},
ContentLength: resp.ContentLength,
@@ -339,7 +297,7 @@ func (f *HTTPFetcher) isAllowedContentType(contentType string) bool {
}
// validateURL checks if a URL is safe to fetch (not internal/private).
func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
func validateURL(rawURL string, allowHTTP bool) error {
if !allowHTTP && !strings.HasPrefix(rawURL, "https://") {
return ErrUnsupportedScheme
}
@@ -351,8 +309,7 @@ func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
}
// Remove port if present
h, _, err := net.SplitHostPort(host)
if err == nil {
if h, _, err := net.SplitHostPort(host); err == nil {
host = h
}
@@ -362,16 +319,15 @@ func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
}
// Resolve the host to check IP addresses
addrs, err := net.DefaultResolver.LookupIPAddr(ctx, host)
ips, err := net.LookupIP(host)
if err != nil {
return fmt.Errorf("%w: %s", ErrInvalidHost, host)
}
private := slices.ContainsFunc(addrs, func(addr net.IPAddr) bool {
return isPrivateIP(addr.IP)
})
if private {
return ErrSSRFBlocked
for _, ip := range ips {
if isPrivateIP(ip) {
return ErrSSRFBlocked
}
}
return nil
@@ -384,11 +340,9 @@ func extractHost(rawURL string) string {
if idx := strings.Index(url, "://"); idx != -1 {
url = url[idx+3:]
}
if idx := strings.Index(url, "/"); idx != -1 {
url = url[:idx]
}
if idx := strings.Index(url, "?"); idx != -1 {
url = url[:idx]
}
@@ -401,8 +355,8 @@ func isLocalhost(host string) bool {
host = strings.ToLower(host)
return host == "localhost" ||
host == localhostIPv4 ||
host == localhostIPv6 ||
host == "127.0.0.1" ||
host == "::1" ||
host == "[::1]" ||
strings.HasSuffix(host, ".localhost") ||
strings.HasSuffix(host, ".local")
@@ -468,23 +422,23 @@ func ssrfSafeDialer(ctx context.Context, network, addr string) (net.Conn, error)
}
// Check all resolved IPs
if slices.ContainsFunc(ips, isPrivateIP) {
return nil, ErrSSRFBlocked
for _, ip := range ips {
if isPrivateIP(ip) {
return nil, ErrSSRFBlocked
}
}
// Connect using the first valid IP
var dialer net.Dialer
for _, ip := range ips {
addr := net.JoinHostPort(ip.String(), port)
conn, err := dialer.DialContext(ctx, network, addr)
if err == nil {
return conn, nil
}
}
return nil, fmt.Errorf("%w to %s", errConnectFailed, host)
return nil, fmt.Errorf("failed to connect to %s", host)
}
// limitedReader wraps a reader and limits the number of bytes read.
@@ -511,7 +465,6 @@ func (r *limitedReader) Read(p []byte) (int, error) {
// semaphoreReleasingReadCloser releases a semaphore slot when closed.
type semaphoreReleasingReadCloser struct {
*limitedReader
closer io.Closer
sem chan struct{}
}

103
internal/httpfetcher/httpfetcher_internal_test.go → internal/httpfetcher/httpfetcher_test.go
View File

@@ -9,12 +9,7 @@ import (
"testing/fstest"
)
// testHost is the hostname used by mock fetch tests.
const testHost = "example.com"
func TestDefaultConfig(t *testing.T) {
t.Parallel()
cfg := DefaultConfig()
if cfg.Timeout != DefaultFetchTimeout {
@@ -40,8 +35,6 @@ func TestDefaultConfig(t *testing.T) {
}
func TestNewWithNilConfigUsesDefaults(t *testing.T) {
t.Parallel()
f := New(nil)
if f == nil {
@@ -58,28 +51,24 @@ func TestNewWithNilConfigUsesDefaults(t *testing.T) {
}
func TestIsAllowedContentType(t *testing.T) {
t.Parallel()
f := New(DefaultConfig())
tests := []struct {
contentType string
want bool
}{
{contentTypeJPEG, true},
{contentTypePNG, true},
{contentTypeWebP, true},
{"image/jpeg", true},
{"image/png", true},
{"image/webp", true},
{"image/jpeg; charset=utf-8", true},
{"IMAGE/JPEG", true},
{"text/html", false},
{contentTypeOctetStream, false},
{"application/octet-stream", false},
{"", false},
}
for _, tc := range tests {
t.Run(tc.contentType, func(t *testing.T) {
t.Parallel()
got := f.isAllowedContentType(tc.contentType)
if got != tc.want {
t.Errorf("isAllowedContentType(%q) = %v, want %v", tc.contentType, got, tc.want)
@@ -89,24 +78,20 @@ func TestIsAllowedContentType(t *testing.T) {
}
func TestExtractHost(t *testing.T) {
t.Parallel()
tests := []struct {
url string
want string
}{
{"https://example.com/path", testHost},
{"https://example.com/path", "example.com"},
{"http://example.com:8080/path", "example.com:8080"},
{"https://example.com", testHost},
{"https://example.com?q=1", testHost},
{"example.com/path", testHost},
{"https://example.com", "example.com"},
{"https://example.com?q=1", "example.com"},
{"example.com/path", "example.com"},
{"", ""},
}
for _, tc := range tests {
t.Run(tc.url, func(t *testing.T) {
t.Parallel()
got := extractHost(tc.url)
if got != tc.want {
t.Errorf("extractHost(%q) = %q, want %q", tc.url, got, tc.want)
@@ -116,27 +101,23 @@ func TestExtractHost(t *testing.T) {
}
func TestIsLocalhost(t *testing.T) {
t.Parallel()
tests := []struct {
host string
want bool
}{
{"localhost", true},
{"LOCALHOST", true},
{localhostIPv4, true},
{localhostIPv6, true},
{"127.0.0.1", true},
{"::1", true},
{"[::1]", true},
{"foo.localhost", true},
{"foo.local", true},
{testHost, false},
{"example.com", false},
{"127.0.0.2", false}, // Handled by isPrivateIP, not isLocalhost string match
}
for _, tc := range tests {
t.Run(tc.host, func(t *testing.T) {
t.Parallel()
got := isLocalhost(tc.host)
if got != tc.want {
t.Errorf("isLocalhost(%q) = %v, want %v", tc.host, got, tc.want)
@@ -146,20 +127,18 @@ func TestIsLocalhost(t *testing.T) {
}
func TestIsPrivateIP(t *testing.T) {
t.Parallel()
tests := []struct {
ip string
want bool
}{
{localhostIPv4, true}, // loopback
{"127.0.0.1", true}, // loopback
{"10.0.0.1", true}, // private
{"192.168.1.1", true}, // private
{"172.16.0.1", true}, // private
{"169.254.1.1", true}, // link-local
{"0.0.0.0", true}, // unspecified
{"224.0.0.1", true}, // multicast
{localhostIPv6, true}, // IPv6 loopback
{"::1", true}, // IPv6 loopback
{"fe80::1", true}, // IPv6 link-local
{"8.8.8.8", false}, // public
{"2001:4860:4860::8888", false}, // public IPv6
@@ -167,8 +146,6 @@ func TestIsPrivateIP(t *testing.T) {
for _, tc := range tests {
t.Run(tc.ip, func(t *testing.T) {
t.Parallel()
ip := net.ParseIP(tc.ip)
if ip == nil {
t.Fatalf("failed to parse IP %q", tc.ip)
@@ -187,19 +164,15 @@ func TestIsPrivateIP(t *testing.T) {
}
func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
t.Parallel()
err := validateURL(t.Context(), "http://example.com/path", false)
err := validateURL("http://example.com/path", false)
if !errors.Is(err, ErrUnsupportedScheme) {
t.Errorf("validateURL http = %v, want ErrUnsupportedScheme", err)
}
}
func TestValidateURL_AllowsHTTPWhenConfigured(t *testing.T) {
t.Parallel()
// Use a host that won't resolve (explicit .invalid TLD) so we don't hit DNS.
err := validateURL(t.Context(), "http://nonexistent.invalid/path", true)
err := validateURL("http://nonexistent.invalid/path", true)
// We expect a host resolution error, not ErrUnsupportedScheme.
if errors.Is(err, ErrUnsupportedScheme) {
t.Error("validateURL with AllowHTTP should not return ErrUnsupportedScheme")
@@ -207,26 +180,20 @@ func TestValidateURL_AllowsHTTPWhenConfigured(t *testing.T) {
}
func TestValidateURL_RejectsLocalhost(t *testing.T) {
t.Parallel()
err := validateURL(t.Context(), "https://localhost/path", false)
err := validateURL("https://localhost/path", false)
if !errors.Is(err, ErrSSRFBlocked) {
t.Errorf("validateURL localhost = %v, want ErrSSRFBlocked", err)
}
}
func TestValidateURL_EmptyHost(t *testing.T) {
t.Parallel()
err := validateURL(t.Context(), "https:///path", false)
err := validateURL("https:///path", false)
if !errors.Is(err, ErrInvalidHost) {
t.Errorf("validateURL empty host = %v, want ErrInvalidHost", err)
}
}
func TestMockFetcher_FetchesFile(t *testing.T) {
t.Parallel()
mockFS := fstest.MapFS{
"example.com/images/photo.jpg": &fstest.MapFile{Data: []byte("fake-jpeg-data")},
}
@@ -237,10 +204,9 @@ func TestMockFetcher_FetchesFile(t *testing.T) {
if err != nil {
t.Fatalf("Fetch() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
if result.ContentType != contentTypeJPEG {
if result.ContentType != "image/jpeg" {
t.Errorf("ContentType = %q, want image/jpeg", result.ContentType)
}
@@ -259,8 +225,6 @@ func TestMockFetcher_FetchesFile(t *testing.T) {
}
func TestMockFetcher_MissingFileReturnsUpstreamError(t *testing.T) {
t.Parallel()
mockFS := fstest.MapFS{}
m := NewMock(mockFS)
@@ -271,8 +235,6 @@ func TestMockFetcher_MissingFileReturnsUpstreamError(t *testing.T) {
}
func TestMockFetcher_RespectsContextCancellation(t *testing.T) {
t.Parallel()
mockFS := fstest.MapFS{
"example.com/photo.jpg": &fstest.MapFile{Data: []byte("data")},
}
@@ -288,28 +250,24 @@ func TestMockFetcher_RespectsContextCancellation(t *testing.T) {
}
func TestDetectContentTypeFromPath(t *testing.T) {
t.Parallel()
tests := []struct {
path string
want string
}{
{"foo/bar.jpg", contentTypeJPEG},
{"foo/bar.JPG", contentTypeJPEG},
{"foo/bar.jpeg", contentTypeJPEG},
{"foo/bar.png", contentTypePNG},
{"foo/bar.gif", contentTypeGIF},
{"foo/bar.webp", contentTypeWebP},
{"foo/bar.avif", contentTypeAVIF},
{"foo/bar.svg", contentTypeSVG},
{"foo/bar.bin", contentTypeOctetStream},
{"foo/bar", contentTypeOctetStream},
{"foo/bar.jpg", "image/jpeg"},
{"foo/bar.JPG", "image/jpeg"},
{"foo/bar.jpeg", "image/jpeg"},
{"foo/bar.png", "image/png"},
{"foo/bar.gif", "image/gif"},
{"foo/bar.webp", "image/webp"},
{"foo/bar.avif", "image/avif"},
{"foo/bar.svg", "image/svg+xml"},
{"foo/bar.bin", "application/octet-stream"},
{"foo/bar", "application/octet-stream"},
}
for _, tc := range tests {
t.Run(tc.path, func(t *testing.T) {
t.Parallel()
got := detectContentTypeFromPath(tc.path)
if got != tc.want {
t.Errorf("detectContentTypeFromPath(%q) = %q, want %q", tc.path, got, tc.want)
@@ -319,8 +277,6 @@ func TestDetectContentTypeFromPath(t *testing.T) {
}
func TestLimitedReader_EnforcesLimit(t *testing.T) {
t.Parallel()
src := make([]byte, 100)
r := &limitedReader{
reader: &byteReader{data: src},
@@ -342,11 +298,10 @@ func TestLimitedReader_EnforcesLimit(t *testing.T) {
total := n
for total < 50 {
nn, err := r.Read(buf)
total += nn
if err != nil {
t.Fatalf("during drain: %v", err)
}
total += nn
}
// Now the limit is exhausted — next read should error.

22
internal/httpfetcher/mock.go
View File

@@ -4,14 +4,12 @@ import (
"context"
"errors"
"fmt"
"io"
"io/fs"
"net/http"
"strings"
)
// errEmptyURLPath is returned when a mock URL has no usable path.
var errEmptyURLPath = errors.New("empty URL path")
// MockFetcher implements Fetcher using an embedded filesystem.
// Files are organized as: hostname/path/to/file.ext
// URLs like https://example.com/images/photo.jpg map to example.com/images/photo.jpg.
@@ -61,7 +59,7 @@ func (m *MockFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
contentType := detectContentTypeFromPath(path)
return &FetchResult{
Content: f,
Content: f.(io.ReadCloser),
ContentLength: stat.Size(),
ContentType: contentType,
Headers: make(http.Header),
@@ -88,7 +86,7 @@ func urlToFSPath(rawURL string) (string, error) {
}
if url == "" {
return "", errEmptyURLPath
return "", errors.New("empty URL path")
}
return url, nil
@@ -100,18 +98,18 @@ func detectContentTypeFromPath(path string) string {
switch {
case strings.HasSuffix(path, ".jpg"), strings.HasSuffix(path, ".jpeg"):
return contentTypeJPEG
return "image/jpeg"
case strings.HasSuffix(path, ".png"):
return contentTypePNG
return "image/png"
case strings.HasSuffix(path, ".gif"):
return contentTypeGIF
return "image/gif"
case strings.HasSuffix(path, ".webp"):
return contentTypeWebP
return "image/webp"
case strings.HasSuffix(path, ".avif"):
return contentTypeAVIF
return "image/avif"
case strings.HasSuffix(path, ".svg"):
return contentTypeSVG
return "image/svg+xml"
default:
return contentTypeOctetStream
return "application/octet-stream"
}
}

101
internal/imageprocessor/imageprocessor.go
View File

@@ -13,9 +13,7 @@ import (
)
// vipsOnce ensures vips is initialized exactly once.
//
//nolint:gochecknoglobals // package-level sync.Once for one-time vips init
var vipsOnce sync.Once
var vipsOnce sync.Once //nolint:gochecknoglobals // package-level sync.Once for one-time vips init
// initVips initializes libvips with quiet logging.
func initVips() {
@@ -98,12 +96,10 @@ const DefaultMaxInputBytes = 50 << 20
// ErrInputTooLarge is returned when input image dimensions exceed MaxInputDimension.
var ErrInputTooLarge = errors.New("input image dimensions exceed maximum")
// ErrInputDataTooLarge is returned when the raw input data exceeds the
// configured byte limit.
// ErrInputDataTooLarge is returned when the raw input data exceeds the configured byte limit.
var ErrInputDataTooLarge = errors.New("input data exceeds maximum allowed size")
// ErrUnsupportedOutputFormat is returned when the requested output format is
// not supported.
// ErrUnsupportedOutputFormat is returned when the requested output format is not supported.
var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
// ImageProcessor implements image transformation using libvips via govips.
@@ -174,12 +170,25 @@ func (p *ImageProcessor) Process(
}
// Determine target dimensions
targetWidth, targetHeight := targetDimensions(req.Size, origWidth, origHeight)
targetWidth := req.Size.Width
targetHeight := req.Size.Height
// Handle dimension calculation
if targetWidth == 0 && targetHeight == 0 {
// Both are 0: keep original size
targetWidth = origWidth
targetHeight = origHeight
} else if targetWidth == 0 {
// Only height specified: calculate width proportionally
targetWidth = origWidth * targetHeight / origHeight
} else if targetHeight == 0 {
// Only width specified: calculate height proportionally
targetHeight = origHeight * targetWidth / origWidth
}
// Resize if needed
if targetWidth != origWidth || targetHeight != origHeight {
err := p.resize(img, targetWidth, targetHeight, req.FitMode)
if err != nil {
if err := p.resize(img, targetWidth, targetHeight, req.FitMode); err != nil {
return nil, fmt.Errorf("failed to resize: %w", err)
}
}
@@ -208,42 +217,14 @@ func (p *ImageProcessor) Process(
}, nil
}
// targetDimensions calculates the output dimensions for a requested size,
// scaling proportionally when only one dimension is given and keeping the
// original dimensions when both are zero.
func targetDimensions(size Size, origWidth, origHeight int) (int, int) {
switch {
case size.Width == 0 && size.Height == 0:
// Both are 0: keep original size
return origWidth, origHeight
case size.Width == 0:
// Only height specified: calculate width proportionally
return origWidth * size.Height / origHeight, size.Height
case size.Height == 0:
// Only width specified: calculate height proportionally
return size.Width, origHeight * size.Width / origWidth
default:
return size.Width, size.Height
}
}
// MIME types for the supported image formats.
const (
mimeJPEG = "image/jpeg"
mimePNG = "image/png"
mimeGIF = "image/gif"
mimeWebP = "image/webp"
mimeAVIF = "image/avif"
)
// SupportedInputFormats returns MIME types this processor can read.
func (p *ImageProcessor) SupportedInputFormats() []string {
return []string{
mimeJPEG,
mimePNG,
mimeGIF,
mimeWebP,
mimeAVIF,
"image/jpeg",
"image/png",
"image/gif",
"image/webp",
"image/avif",
}
}
@@ -262,17 +243,15 @@ func (p *ImageProcessor) SupportedOutputFormats() []Format {
func FormatToMIME(format Format) string {
switch format {
case FormatJPEG:
return mimeJPEG
return "image/jpeg"
case FormatPNG:
return mimePNG
return "image/png"
case FormatWebP:
return mimeWebP
return "image/webp"
case FormatGIF:
return mimeGIF
return "image/gif"
case FormatAVIF:
return mimeAVIF
case FormatOriginal:
return "application/octet-stream"
return "image/avif"
default:
return "application/octet-stream"
}
@@ -291,20 +270,14 @@ func (p *ImageProcessor) detectFormat(img *vips.ImageRef) string {
case vips.ImageTypeWEBP:
return "webp"
case vips.ImageTypeAVIF, vips.ImageTypeHEIF:
return string(FormatAVIF)
case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF,
vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP,
vips.ImageTypeJP2K, vips.ImageTypeJXL:
return "unknown"
return "avif"
default:
return "unknown"
}
}
// resize resizes the image according to the fit mode.
func (p *ImageProcessor) resize(
img *vips.ImageRef, width, height int, fit FitMode,
) error {
func (p *ImageProcessor) resize(img *vips.ImageRef, width, height int, fit FitMode) error {
switch fit {
case FitCover, "":
// Resize and crop to fill exact dimensions (default)
@@ -330,7 +303,6 @@ func (p *ImageProcessor) resize(
if img.Width() <= width && img.Height() <= height {
return nil // Already fits
}
imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
@@ -359,9 +331,7 @@ func (p *ImageProcessor) resize(
const defaultQuality = 85
// encode encodes an image to the specified format.
func (p *ImageProcessor) encode(
img *vips.ImageRef, format Format, quality int,
) ([]byte, error) {
func (p *ImageProcessor) encode(img *vips.ImageRef, format Format, quality int) ([]byte, error) {
if quality <= 0 {
quality = defaultQuality
}
@@ -397,11 +367,8 @@ func (p *ImageProcessor) encode(
Quality: quality,
}
case FormatOriginal:
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
default:
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
return nil, fmt.Errorf("unsupported output format: %s", format)
}
output, _, err := img.Export(&params)
@@ -423,7 +390,7 @@ func (p *ImageProcessor) formatFromString(format string) Format {
return FormatGIF
case "webp":
return FormatWebP
case string(FormatAVIF):
case "avif":
return FormatAVIF
default:
return FormatJPEG

326
internal/imageprocessor/imageprocessor_internal_test.go → internal/imageprocessor/imageprocessor_test.go
View File

@@ -3,7 +3,6 @@ package imageprocessor
import (
"bytes"
"context"
"errors"
"image"
"image/color"
"image/jpeg"
@@ -17,9 +16,7 @@ import (
func TestMain(m *testing.M) {
initVips()
code := m.Run()
vips.Shutdown()
os.Exit(code)
}
@@ -30,11 +27,11 @@ func createTestJPEG(t *testing.T, width, height int) []byte {
img := image.NewRGBA(image.Rect(0, 0, width, height))
// Fill with a gradient
for y := range height {
for x := range width {
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
img.Set(x, y, color.RGBA{
R: uint8((x * 255 / width) & 0xff),
G: uint8((y * 255 / height) & 0xff),
R: uint8(x * 255 / width),
G: uint8(y * 255 / height),
B: 128,
A: 255,
})
@@ -42,9 +39,7 @@ func createTestJPEG(t *testing.T, width, height int) []byte {
}
var buf bytes.Buffer
err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 90})
if err != nil {
if err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 90}); err != nil {
t.Fatalf("failed to encode test JPEG: %v", err)
}
@@ -56,11 +51,11 @@ func createTestPNG(t *testing.T, width, height int) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
img.Set(x, y, color.RGBA{
R: uint8((x * 255 / width) & 0xff),
G: uint8((y * 255 / height) & 0xff),
R: uint8(x * 255 / width),
G: uint8(y * 255 / height),
B: 128,
A: 255,
})
@@ -68,54 +63,37 @@ func createTestPNG(t *testing.T, width, height int) []byte {
}
var buf bytes.Buffer
err := png.Encode(&buf, img)
if err != nil {
if err := png.Encode(&buf, img); err != nil {
t.Fatalf("failed to encode test PNG: %v", err)
}
return buf.Bytes()
}
// isAVIF reports whether data starts with an AVIF ftyp box.
func isAVIF(data []byte) bool {
if len(data) < 12 || string(data[4:8]) != "ftyp" {
return false
}
brand := string(data[8:12])
return brand == string(FormatAVIF) || brand == "avis"
}
// detectMIME is a minimal magic-byte detector for test assertions.
func detectMIME(data []byte) string {
if len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF {
return mimeJPEG
return "image/jpeg"
}
if len(data) >= 8 && string(data[:8]) == "\x89PNG\r\n\x1a\n" {
return mimePNG
return "image/png"
}
if len(data) >= 4 && string(data[:4]) == "GIF8" {
return mimeGIF
return "image/gif"
}
if len(data) >= 12 && string(data[:4]) == "RIFF" && string(data[8:12]) == "WEBP" {
return mimeWebP
return "image/webp"
}
if isAVIF(data) {
return mimeAVIF
if len(data) >= 12 && string(data[4:8]) == "ftyp" {
brand := string(data[8:12])
if brand == "avif" || brand == "avis" {
return "image/avif"
}
}
return ""
}
func TestImageProcessor_ResizeJPEG(t *testing.T) {
t.Parallel()
proc := New(Params{})
ctx := context.Background()
@@ -132,8 +110,7 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
defer result.Content.Close()
if result.Width != 400 {
t.Errorf("Process() width = %d, want 400", result.Width)
@@ -154,14 +131,12 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
}
mime := detectMIME(data)
if mime != mimeJPEG {
if mime != "image/jpeg" {
t.Errorf("Output format = %v, want image/jpeg", mime)
}
}
func TestImageProcessor_ConvertToPNG(t *testing.T) {
t.Parallel()
proc := New(Params{})
ctx := context.Background()
@@ -177,8 +152,7 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
defer result.Content.Close()
data, err := io.ReadAll(result.Content)
if err != nil {
@@ -186,25 +160,19 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
}
mime := detectMIME(data)
if mime != mimePNG {
if mime != "image/png" {
t.Errorf("Output format = %v, want image/png", mime)
}
}
// processAndCheckSize processes a test JPEG of the given input dimensions
// with the requested size and asserts the resulting dimensions.
func processAndCheckSize(
t *testing.T, inputW, inputH int, size Size, wantW, wantH int,
) {
t.Helper()
func TestImageProcessor_OriginalSize(t *testing.T) {
proc := New(Params{})
ctx := context.Background()
input := createTestJPEG(t, inputW, inputH)
input := createTestJPEG(t, 640, 480)
req := &Request{
Size: size,
Size: Size{Width: 0, Height: 0}, // Original size
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
@@ -214,28 +182,18 @@ func processAndCheckSize(
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer result.Content.Close()
defer func() { _ = result.Content.Close() }()
if result.Width != wantW {
t.Errorf("Process() width = %d, want %d", result.Width, wantW)
if result.Width != 640 {
t.Errorf("Process() width = %d, want 640", result.Width)
}
if result.Height != wantH {
t.Errorf("Process() height = %d, want %d", result.Height, wantH)
if result.Height != 480 {
t.Errorf("Process() height = %d, want 480", result.Height)
}
}
func TestImageProcessor_OriginalSize(t *testing.T) {
t.Parallel()
// Width and height 0: keep original size
processAndCheckSize(t, 640, 480, Size{Width: 0, Height: 0}, 640, 480)
}
func TestImageProcessor_FitContain(t *testing.T) {
t.Parallel()
proc := New(Params{})
ctx := context.Background()
@@ -254,8 +212,7 @@ func TestImageProcessor_FitContain(t *testing.T) {
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
defer result.Content.Close()
// With contain, the image should fit within the box
if result.Width > 400 || result.Height > 400 {
@@ -264,24 +221,66 @@ func TestImageProcessor_FitContain(t *testing.T) {
}
func TestImageProcessor_ProportionalScale_WidthOnly(t *testing.T) {
t.Parallel()
proc := New(Params{})
ctx := context.Background()
// 800x600 image, request width=400 height=0
// Should scale proportionally to 400x300
processAndCheckSize(t, 800, 600, Size{Width: 400, Height: 0}, 400, 300)
input := createTestJPEG(t, 800, 600)
req := &Request{
Size: Size{Width: 400, Height: 0},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer result.Content.Close()
if result.Width != 400 {
t.Errorf("Process() width = %d, want 400", result.Width)
}
if result.Height != 300 {
t.Errorf("Process() height = %d, want 300", result.Height)
}
}
func TestImageProcessor_ProportionalScale_HeightOnly(t *testing.T) {
t.Parallel()
proc := New(Params{})
ctx := context.Background()
// 800x600 image, request width=0 height=300
// Should scale proportionally to 400x300
processAndCheckSize(t, 800, 600, Size{Width: 0, Height: 300}, 400, 300)
input := createTestJPEG(t, 800, 600)
req := &Request{
Size: Size{Width: 0, Height: 300},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer result.Content.Close()
if result.Width != 400 {
t.Errorf("Process() width = %d, want 400", result.Width)
}
if result.Height != 300 {
t.Errorf("Process() height = %d, want 300", result.Height)
}
}
func TestImageProcessor_ProcessPNG(t *testing.T) {
t.Parallel()
proc := New(Params{})
ctx := context.Background()
@@ -297,8 +296,7 @@ func TestImageProcessor_ProcessPNG(t *testing.T) {
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
defer result.Content.Close()
if result.Width != 200 {
t.Errorf("Process() width = %d, want 200", result.Width)
@@ -310,8 +308,6 @@ func TestImageProcessor_ProcessPNG(t *testing.T) {
}
func TestImageProcessor_SupportedFormats(t *testing.T) {
t.Parallel()
proc := New(Params{})
inputFormats := proc.SupportedInputFormats()
@@ -326,49 +322,55 @@ func TestImageProcessor_SupportedFormats(t *testing.T) {
}
func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
t.Parallel()
proc := New(Params{})
ctx := context.Background()
// Images exceeding MaxInputDimension in either dimension must be
// rejected before processing to prevent DoS.
tests := []struct {
name string
width int
height int
}{
{name: "oversized width", width: 10000, height: 100},
{name: "oversized height", width: 100, height: 10000},
// Create an image that exceeds MaxInputDimension (e.g., 10000x100)
// This should be rejected before processing to prevent DoS
input := createTestJPEG(t, 10000, 100)
req := &Request{
Size: Size{Width: 100, Height: 100},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
_, err := proc.Process(ctx, bytes.NewReader(input), req)
if err == nil {
t.Error("Process() should reject oversized input images")
}
proc := New(Params{})
ctx := context.Background()
input := createTestJPEG(t, tt.width, tt.height)
if err != ErrInputTooLarge {
t.Errorf("Process() error = %v, want ErrInputTooLarge", err)
}
}
req := &Request{
Size: Size{Width: 100, Height: 100},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
func TestImageProcessor_RejectsOversizedInputHeight(t *testing.T) {
proc := New(Params{})
ctx := context.Background()
_, err := proc.Process(ctx, bytes.NewReader(input), req)
if err == nil {
t.Error("Process() should reject oversized input images")
}
// Create an image with oversized height
input := createTestJPEG(t, 100, 10000)
if !errors.Is(err, ErrInputTooLarge) {
t.Errorf("Process() error = %v, want ErrInputTooLarge", err)
}
})
req := &Request{
Size: Size{Width: 100, Height: 100},
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
_, err := proc.Process(ctx, bytes.NewReader(input), req)
if err == nil {
t.Error("Process() should reject oversized input images")
}
if err != ErrInputTooLarge {
t.Errorf("Process() error = %v, want ErrInputTooLarge", err)
}
}
func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
t.Parallel()
proc := New(Params{})
ctx := context.Background()
@@ -384,20 +386,12 @@ func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
result, err := proc.Process(ctx, bytes.NewReader(input), req)
if err != nil {
t.Fatalf(
"Process() should accept images at MaxInputDimension, got error: %v",
err,
)
t.Fatalf("Process() should accept images at MaxInputDimension, got error: %v", err)
}
defer func() { _ = result.Content.Close() }()
defer result.Content.Close()
}
// encodeAndCheck processes a 200x150 test JPEG into a 100x75 output of the
// given format and asserts the output MIME type and dimensions.
func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) {
t.Helper()
func TestImageProcessor_EncodeWebP(t *testing.T) {
proc := New(Params{})
ctx := context.Background()
@@ -405,8 +399,8 @@ func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) {
req := &Request{
Size: Size{Width: 100, Height: 75},
Format: format,
Quality: quality,
Format: FormatWebP,
Quality: 80,
FitMode: FitCover,
}
@@ -414,39 +408,29 @@ func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) {
if err != nil {
t.Fatalf("Process() error = %v, want nil", err)
}
defer result.Content.Close()
defer func() { _ = result.Content.Close() }()
// Verify output format
// Verify output is valid WebP
data, err := io.ReadAll(result.Content)
if err != nil {
t.Fatalf("failed to read result: %v", err)
}
mime := detectMIME(data)
if mime != wantMIME {
t.Errorf("Output format = %v, want %v", mime, wantMIME)
if mime != "image/webp" {
t.Errorf("Output format = %v, want image/webp", mime)
}
// Verify dimensions
if result.Width != 100 {
t.Errorf("Width = %d, want 100", result.Width)
}
if result.Height != 75 {
t.Errorf("Height = %d, want 75", result.Height)
}
}
func TestImageProcessor_EncodeWebP(t *testing.T) {
t.Parallel()
encodeAndCheck(t, FormatWebP, 80, mimeWebP)
}
func TestImageProcessor_DecodeAVIF(t *testing.T) {
t.Parallel()
proc := New(Params{})
ctx := context.Background()
@@ -468,8 +452,7 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
if err != nil {
t.Fatalf("Process() error = %v, want nil (AVIF decoding should work)", err)
}
defer func() { _ = result.Content.Close() }()
defer result.Content.Close()
// Verify output is valid JPEG
data, err := io.ReadAll(result.Content)
@@ -478,17 +461,14 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
}
mime := detectMIME(data)
if mime != mimeJPEG {
if mime != "image/jpeg" {
t.Errorf("Output format = %v, want image/jpeg", mime)
}
}
func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
t.Parallel()
// Create a processor with a very small byte limit
const limit = 1024
proc := New(Params{MaxInputBytes: limit})
ctx := context.Background()
@@ -510,14 +490,12 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
t.Fatal("Process() should reject input exceeding maxInputBytes")
}
if !errors.Is(err, ErrInputDataTooLarge) {
if err != ErrInputDataTooLarge {
t.Errorf("Process() error = %v, want ErrInputDataTooLarge", err)
}
}
func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
t.Parallel()
// Create a small image and set limit well above its size
input := createTestJPEG(t, 10, 10)
limit := int64(len(input)) * 10 // 10× headroom
@@ -536,13 +514,10 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
if err != nil {
t.Fatalf("Process() error = %v, want nil", err)
}
defer func() { _ = result.Content.Close() }()
defer result.Content.Close()
}
func TestImageProcessor_DefaultMaxInputBytes(t *testing.T) {
t.Parallel()
// Passing 0 should use the default
proc := New(Params{})
if proc.maxInputBytes != DefaultMaxInputBytes {
@@ -557,7 +532,40 @@ func TestImageProcessor_DefaultMaxInputBytes(t *testing.T) {
}
func TestImageProcessor_EncodeAVIF(t *testing.T) {
t.Parallel()
proc := New(Params{})
ctx := context.Background()
encodeAndCheck(t, FormatAVIF, 85, mimeAVIF)
input := createTestJPEG(t, 200, 150)
req := &Request{
Size: Size{Width: 100, Height: 75},
Format: FormatAVIF,
Quality: 85,
FitMode: FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input), req)
if err != nil {
t.Fatalf("Process() error = %v, want nil (AVIF encoding should work)", err)
}
defer result.Content.Close()
// Verify output is valid AVIF
data, err := io.ReadAll(result.Content)
if err != nil {
t.Fatalf("failed to read result: %v", err)
}
mime := detectMIME(data)
if mime != "image/avif" {
t.Errorf("Output format = %v, want image/avif", mime)
}
// Verify dimensions
if result.Width != 100 {
t.Errorf("Width = %d, want 100", result.Width)
}
if result.Height != 75 {
t.Errorf("Height = %d, want 75", result.Height)
}
}

332
internal/imgcache/cache.go
View File

@@ -2,16 +2,12 @@ package imgcache
import (
"context"
"crypto/sha256"
"database/sql"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"path/filepath"
"sync"
"time"
"sneak.berlin/go/pixa/internal/httpfetcher"
@@ -31,22 +27,6 @@ type CacheConfig struct {
StateDir string
CacheTTL time.Duration
NegativeTTL time.Duration
// MaxBytes is the disk cache size limit in bytes that eviction
// enforces. Zero means no limit is enforced (no eviction). The
// config layer supplies the computed default when the operator
// omits cache_max_bytes.
MaxBytes int64
// DisableDiskCache turns the disk cache off entirely: no cache
// directories are created, lookups always miss, stores are
// no-ops, and no eviction machinery runs. The config layer sets
// this when the operator configures cache_max_bytes: 0.
DisableDiskCache bool
// Logger receives accounting and eviction log output. A nil
// Logger means slog.Default().
Logger *slog.Logger
}
// variantMeta stores content type for fast cache hits without reading .meta file.
@@ -62,87 +42,36 @@ type Cache struct {
variants *VariantStorage // processed variants by cache key
srcMetadata *MetadataStorage // source metadata by host/path
config CacheConfig
log *slog.Logger
// disabled means the disk cache is turned off entirely: lookups
// always miss, stores are no-ops, and no eviction runs.
disabled bool
// Eviction machinery. The channels are created in NewCache so
// stores can signal write pressure without racing StartEviction.
evictionPressure chan struct{}
evictionStop chan struct{}
evictionDone chan struct{}
evictionStarted bool
evictionStopOnce sync.Once
// In-memory cache of variant metadata (content type, size) to avoid
// reading .meta files
// In-memory cache of variant metadata (content type, size) to avoid reading .meta files
metaCache map[VariantKey]variantMeta
// contentLocks serializes StoreSource and evictSourceBlob per
// content hash, closing the race window between an eviction's row
// deletion and its file unlink against a concurrent store of
// identical content.
contentLocks *contentLock
// evictSourceBlobTestHook, when set, is invoked by evictSourceBlob
// after its row-deletion transaction commits and before the
// content file is unlinked. It exists solely so tests can
// deterministically pause inside that window to exercise
// concurrent stores against it; production code leaves it nil.
evictSourceBlobTestHook func(ContentHash)
}
// NewCache creates a new cache instance.
func NewCache(db *sql.DB, config CacheConfig) (*Cache, error) {
log := config.Logger
if log == nil {
log = slog.Default()
}
c := &Cache{
db: db,
config: config,
log: log,
disabled: config.DisableDiskCache,
evictionPressure: make(chan struct{}, 1),
evictionStop: make(chan struct{}),
evictionDone: make(chan struct{}),
metaCache: make(map[VariantKey]variantMeta),
contentLocks: newContentLock(),
}
if c.disabled {
return c, nil
}
srcContent, err := NewContentStorage(
filepath.Join(config.StateDir, "cache", "sources"),
)
srcContent, err := NewContentStorage(filepath.Join(config.StateDir, "cache", "sources"))
if err != nil {
return nil, fmt.Errorf("failed to create source content storage: %w", err)
}
variants, err := NewVariantStorage(
filepath.Join(config.StateDir, "cache", "variants"),
)
variants, err := NewVariantStorage(filepath.Join(config.StateDir, "cache", "variants"))
if err != nil {
return nil, fmt.Errorf("failed to create variant storage: %w", err)
}
srcMetadata, err := NewMetadataStorage(
filepath.Join(config.StateDir, "cache", "metadata"),
)
srcMetadata, err := NewMetadataStorage(filepath.Join(config.StateDir, "cache", "metadata"))
if err != nil {
return nil, fmt.Errorf("failed to create source metadata storage: %w", err)
}
c.srcContent = srcContent
c.variants = variants
c.srcMetadata = srcMetadata
return c, nil
return &Cache{
db: db,
srcContent: srcContent,
variants: variants,
srcMetadata: srcMetadata,
config: config,
metaCache: make(map[VariantKey]variantMeta),
}, nil
}
// LookupResult contains the result of a cache lookup.
@@ -154,15 +83,12 @@ type LookupResult struct {
CacheStatus CacheStatus
}
// Lookup checks if a processed variant exists on disk. Hits touch the
// variant's LRU timestamp; a disabled cache always misses.
func (c *Cache) Lookup(ctx context.Context, req *ImageRequest) (*LookupResult, error) {
// Lookup checks if a processed variant exists on disk (no DB access for hits).
func (c *Cache) Lookup(_ context.Context, req *ImageRequest) (*LookupResult, error) {
cacheKey := CacheKey(req)
// Check variant storage directly - no DB needed for cache hits
if !c.disabled && c.variants.Exists(cacheKey) {
c.touchVariant(ctx, cacheKey)
if c.variants.Exists(cacheKey) {
return &LookupResult{
Hit: true,
CacheKey: cacheKey,
@@ -179,63 +105,25 @@ func (c *Cache) Lookup(ctx context.Context, req *ImageRequest) (*LookupResult, e
// GetVariant returns a reader, size, and content type for a cached variant.
func (c *Cache) GetVariant(cacheKey VariantKey) (io.ReadCloser, int64, string, error) {
if c.disabled {
return nil, 0, "", ErrNotFound
}
return c.variants.LoadWithMeta(cacheKey)
}
// StoreSource stores fetched source content and metadata. On a
// disabled cache it is a no-op returning an empty hash.
// StoreSource stores fetched source content and metadata.
func (c *Cache) StoreSource(
ctx context.Context,
req *ImageRequest,
content io.Reader,
result *httpfetcher.FetchResult,
) (ContentHash, error) {
if c.disabled {
return "", nil
}
// Hash the content ourselves (rather than via srcContent.Store,
// which would hash internally) so the content hash is known before
// any file or database work happens: that lets the entire store be
// serialized, per hash, against a concurrent eviction of the same
// content below.
data, err := io.ReadAll(content)
if err != nil {
return "", fmt.Errorf("failed to read source content: %w", err)
}
sum := sha256.Sum256(data)
contentHash := ContentHash(hex.EncodeToString(sum[:]))
// Hold the content hash's lock for the whole store operation. A
// concurrent eviction of this exact hash (the real SHA-256 dedup
// case: a different source path whose bytes hash identically)
// deletes the accounting rows and unlinks the file inside the same
// lock, so the two can never interleave: either this store
// completes first (and a subsequent eviction removes it together
// with its rows and file, correctly), or eviction completes first
// (and this store finds the file already gone and recreates it
// fresh) — never a fresh row left pointing at a file eviction is
// mid-unlink on.
unlock := c.contentLocks.Lock(string(contentHash))
defer unlock()
size, err := c.srcContent.StoreHashed(contentHash, data)
// Store content
contentHash, size, err := c.srcContent.Store(content)
if err != nil {
return "", fmt.Errorf("failed to store source content: %w", err)
}
// Store in database
pathHash := HashPath(req.SourcePath + "?" + req.SourceQuery)
headersJSON, err := json.Marshal(result.Headers)
if err != nil {
return "", fmt.Errorf("failed to marshal response headers: %w", err)
}
headersJSON, _ := json.Marshal(result.Headers)
_, err = c.db.ExecContext(ctx, `
INSERT INTO source_content (content_hash, content_type, size_bytes)
@@ -278,59 +166,24 @@ func (c *Cache) StoreSource(
RemoteAddr: result.RemoteAddr,
}
// A failure here is non-fatal; the metadata is in the database.
_ = c.srcMetadata.Store(req.SourceHost, pathHash, meta)
c.notifyWritePressure()
if err := c.srcMetadata.Store(req.SourceHost, pathHash, meta); err != nil {
// Non-fatal, we have it in the database
_ = err
}
return contentHash, nil
}
// StoreVariant stores a processed variant by its cache key and records
// it in the size accounting. On a disabled cache it is a no-op. The
// accounting insert is best-effort (the startup reconciliation pass
// adopts any variant file that misses its accounting row).
func (c *Cache) StoreVariant(
ctx context.Context, cacheKey VariantKey, content io.Reader, contentType string,
) error {
if c.disabled {
return nil
}
// StoreVariant stores a processed variant by its cache key.
func (c *Cache) StoreVariant(cacheKey VariantKey, content io.Reader, contentType string) error {
_, err := c.variants.Store(cacheKey, content, contentType)
size, err := c.variants.Store(cacheKey, content, contentType)
if err != nil {
return err
}
_, err = c.db.ExecContext(ctx, `
INSERT INTO variant_content (cache_key, size_bytes, content_type)
VALUES (?, ?, ?)
ON CONFLICT(cache_key) DO UPDATE SET
size_bytes = excluded.size_bytes,
content_type = excluded.content_type,
last_accessed_at = CURRENT_TIMESTAMP
`, string(cacheKey), size, contentType)
if err != nil {
c.log.Warn("failed to record variant in size accounting",
"cache_key", cacheKey, "error", err)
}
c.notifyWritePressure()
return nil
return err
}
// LookupSource checks if we have cached source content for a request.
// Returns the content hash and content type if found, or empty values
// if not. Hits touch the blob's LRU timestamp; a disabled cache always
// reports no cached source.
func (c *Cache) LookupSource(
ctx context.Context, req *ImageRequest,
) (ContentHash, string, error) {
if c.disabled {
return "", "", nil
}
// Returns the content hash and content type if found, or empty values if not.
func (c *Cache) LookupSource(ctx context.Context, req *ImageRequest) (ContentHash, string, error) {
var hashStr, contentType string
err := c.db.QueryRowContext(ctx, `
@@ -353,21 +206,15 @@ func (c *Cache) LookupSource(
return "", "", nil
}
c.touchSourceContent(ctx, contentHash)
return contentHash, contentType, nil
}
// StoreNegative stores a negative cache entry for a failed fetch.
func (c *Cache) StoreNegative(
ctx context.Context, req *ImageRequest, statusCode int, errMsg string,
) error {
func (c *Cache) StoreNegative(ctx context.Context, req *ImageRequest, statusCode int, errMsg string) error {
expiresAt := time.Now().UTC().Add(c.config.NegativeTTL)
_, err := c.db.ExecContext(ctx, `
INSERT INTO negative_cache
(source_host, source_path, source_query, status_code,
error_message, expires_at)
INSERT INTO negative_cache (source_host, source_path, source_query, status_code, error_message, expires_at)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(source_host, source_path, source_query) DO UPDATE SET
status_code = excluded.status_code,
@@ -382,16 +229,46 @@ func (c *Cache) StoreNegative(
return nil
}
// checkNegativeCache checks if a request is in the negative cache.
func (c *Cache) checkNegativeCache(ctx context.Context, req *ImageRequest) (bool, error) {
var expiresAt time.Time
err := c.db.QueryRowContext(ctx, `
SELECT expires_at FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.SourceHost, req.SourcePath, req.SourceQuery).Scan(&expiresAt)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("failed to check negative cache: %w", err)
}
// Check if expired
if time.Now().After(expiresAt) {
// Clean up expired entry
_, _ = c.db.ExecContext(ctx, `
DELETE FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.SourceHost, req.SourcePath, req.SourceQuery)
return false, nil
}
return true, nil
}
// GetSourceMetadataID returns the source metadata ID for a request.
func (c *Cache) GetSourceMetadataID(
ctx context.Context, req *ImageRequest,
) (int64, error) {
func (c *Cache) GetSourceMetadataID(ctx context.Context, req *ImageRequest) (int64, error) {
var id int64
err := c.db.QueryRowContext(ctx, `
SELECT id FROM source_metadata
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.SourceHost, req.SourcePath, req.SourceQuery).Scan(&id)
if err != nil {
return 0, fmt.Errorf("failed to get source metadata ID: %w", err)
}
@@ -401,10 +278,6 @@ func (c *Cache) GetSourceMetadataID(
// GetSourceContent returns a reader for cached source content by its hash.
func (c *Cache) GetSourceContent(contentHash ContentHash) (io.ReadCloser, error) {
if c.disabled {
return nil, ErrNotFound
}
return c.srcContent.Load(contentHash)
}
@@ -436,12 +309,8 @@ func (c *Cache) Stats(ctx context.Context) (*CacheStats, error) {
}
// Get actual item count and total size from content tables
_ = c.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM request_cache`,
).Scan(&stats.TotalItems)
_ = c.db.QueryRowContext(ctx,
`SELECT COALESCE(SUM(size_bytes), 0) FROM output_content`,
).Scan(&stats.TotalSizeBytes)
_ = c.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM request_cache`).Scan(&stats.TotalItems)
_ = c.db.QueryRowContext(ctx, `SELECT COALESCE(SUM(size_bytes), 0) FROM output_content`).Scan(&stats.TotalSizeBytes)
// Compute hit rate as a ratio
if stats.HitCount+stats.MissCount > 0 {
@@ -455,17 +324,11 @@ func (c *Cache) Stats(ctx context.Context) (*CacheStats, error) {
func (c *Cache) IncrementStats(ctx context.Context, hit bool, fetchBytes int64) {
if hit {
_, _ = c.db.ExecContext(ctx, `
UPDATE cache_stats
SET hit_count = hit_count + 1,
last_updated_at = CURRENT_TIMESTAMP
WHERE id = 1
UPDATE cache_stats SET hit_count = hit_count + 1, last_updated_at = CURRENT_TIMESTAMP WHERE id = 1
`)
} else {
_, _ = c.db.ExecContext(ctx, `
UPDATE cache_stats
SET miss_count = miss_count + 1,
last_updated_at = CURRENT_TIMESTAMP
WHERE id = 1
UPDATE cache_stats SET miss_count = miss_count + 1, last_updated_at = CURRENT_TIMESTAMP WHERE id = 1
`)
}
@@ -479,62 +342,3 @@ func (c *Cache) IncrementStats(ctx context.Context, hit bool, fetchBytes int64)
`, fetchBytes)
}
}
// touchVariant updates the LRU timestamp of a variant, best-effort:
// a failed touch only makes the entry look colder to eviction.
func (c *Cache) touchVariant(ctx context.Context, cacheKey VariantKey) {
_, err := c.db.ExecContext(ctx, `
UPDATE variant_content SET last_accessed_at = CURRENT_TIMESTAMP
WHERE cache_key = ?
`, string(cacheKey))
if err != nil {
c.log.Debug("failed to touch variant LRU timestamp",
"cache_key", cacheKey, "error", err)
}
}
// touchSourceContent updates the LRU timestamp of a source content
// blob, best-effort: a failed touch only makes the blob look colder.
func (c *Cache) touchSourceContent(ctx context.Context, contentHash ContentHash) {
_, err := c.db.ExecContext(ctx, `
UPDATE source_content SET last_accessed_at = CURRENT_TIMESTAMP
WHERE content_hash = ?
`, string(contentHash))
if err != nil {
c.log.Debug("failed to touch source content LRU timestamp",
"content_hash", contentHash, "error", err)
}
}
// checkNegativeCache checks if a request is in the negative cache.
func (c *Cache) checkNegativeCache(
ctx context.Context, req *ImageRequest,
) (bool, error) {
var expiresAt time.Time
err := c.db.QueryRowContext(ctx, `
SELECT expires_at FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.SourceHost, req.SourcePath, req.SourceQuery).Scan(&expiresAt)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("failed to check negative cache: %w", err)
}
// Check if expired
if time.Now().After(expiresAt) {
// Clean up expired entry
_, _ = c.db.ExecContext(ctx, `
DELETE FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.SourceHost, req.SourcePath, req.SourceQuery)
return false, nil
}
return true, nil
}

108
internal/imgcache/cache_internal_test.go → internal/imgcache/cache_test.go
View File

@@ -86,15 +86,14 @@ func setupTestDB(t *testing.T) *sql.DB {
INSERT INTO cache_stats (id) VALUES (1);
`
_, err = db.ExecContext(t.Context(), schema)
if err != nil {
if _, err := db.Exec(schema); err != nil {
t.Fatalf("failed to create schema: %v", err)
}
return db
}
func setupTestCache(t *testing.T) *Cache {
func setupTestCache(t *testing.T) (*Cache, string) {
t.Helper()
tmpDir := t.TempDir()
@@ -109,18 +108,16 @@ func setupTestCache(t *testing.T) *Cache {
t.Fatalf("failed to create cache: %v", err)
}
return cache
return cache, tmpDir
}
func TestCache_LookupMiss(t *testing.T) {
t.Parallel()
cache := setupTestCache(t)
cache, _ := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
@@ -142,14 +139,12 @@ func TestCache_LookupMiss(t *testing.T) {
}
func TestCache_StoreAndLookup(t *testing.T) {
t.Parallel()
cache := setupTestCache(t)
cache, _ := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
Quality: 85,
@@ -159,12 +154,11 @@ func TestCache_StoreAndLookup(t *testing.T) {
// Store source content
sourceContent := []byte("fake jpeg data")
fetchResult := &httpfetcher.FetchResult{
ContentType: testContentTypeJPEG,
Headers: map[string][]string{testHeaderContentType: {testContentTypeJPEG}},
ContentType: "image/jpeg",
Headers: map[string][]string{"Content-Type": {"image/jpeg"}},
}
contentHash, err := cache.StoreSource(
ctx, req, bytes.NewReader(sourceContent), fetchResult)
contentHash, err := cache.StoreSource(ctx, req, bytes.NewReader(sourceContent), fetchResult)
if err != nil {
t.Fatalf("StoreSource() error = %v", err)
}
@@ -176,9 +170,7 @@ func TestCache_StoreAndLookup(t *testing.T) {
// Store variant
cacheKey := CacheKey(req)
outputContent := []byte("fake webp data")
err = cache.StoreVariant(
t.Context(), cacheKey, bytes.NewReader(outputContent), "image/webp")
err = cache.StoreVariant(cacheKey, bytes.NewReader(outputContent), "image/webp")
if err != nil {
t.Fatalf("StoreVariant() error = %v", err)
}
@@ -203,13 +195,11 @@ func TestCache_StoreAndLookup(t *testing.T) {
}
func TestCache_NegativeCache(t *testing.T) {
t.Parallel()
cache := setupTestCache(t)
cache, _ := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourceHost: "cdn.example.com",
SourcePath: "/photos/notfound.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
@@ -233,8 +223,6 @@ func TestCache_NegativeCache(t *testing.T) {
}
func TestCache_NegativeCacheExpiry(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
db := setupTestDB(t)
@@ -251,7 +239,7 @@ func TestCache_NegativeCacheExpiry(t *testing.T) {
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourceHost: "cdn.example.com",
SourcePath: "/photos/expired.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
@@ -278,13 +266,11 @@ func TestCache_NegativeCacheExpiry(t *testing.T) {
}
func TestCache_VariantLookup(t *testing.T) {
t.Parallel()
cache := setupTestCache(t)
cache, _ := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourceHost: "cdn.example.com",
SourcePath: "/photos/variant.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
@@ -295,9 +281,7 @@ func TestCache_VariantLookup(t *testing.T) {
// Store variant
cacheKey := CacheKey(req)
outputContent := []byte("output data")
err := cache.StoreVariant(
t.Context(), cacheKey, bytes.NewReader(outputContent), "image/webp")
err := cache.StoreVariant(cacheKey, bytes.NewReader(outputContent), "image/webp")
if err != nil {
t.Fatalf("StoreVariant() error = %v", err)
}
@@ -328,13 +312,11 @@ func TestCache_VariantLookup(t *testing.T) {
}
func TestCache_GetVariant_ReturnsContentType(t *testing.T) {
t.Parallel()
cache := setupTestCache(t)
cache, _ := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourceHost: "cdn.example.com",
SourcePath: "/photos/variantct.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
@@ -345,9 +327,7 @@ func TestCache_GetVariant_ReturnsContentType(t *testing.T) {
// Store variant
cacheKey := CacheKey(req)
outputContent := []byte("output webp data")
err := cache.StoreVariant(
t.Context(), cacheKey, bytes.NewReader(outputContent), "image/webp")
err := cache.StoreVariant(cacheKey, bytes.NewReader(outputContent), "image/webp")
if err != nil {
t.Fatalf("StoreVariant() error = %v", err)
}
@@ -367,8 +347,7 @@ func TestCache_GetVariant_ReturnsContentType(t *testing.T) {
if err != nil {
t.Fatalf("GetVariant() error = %v", err)
}
defer func() { _ = reader.Close() }()
defer reader.Close()
if contentType != "image/webp" {
t.Errorf("GetVariant() ContentType = %q, want %q", contentType, "image/webp")
@@ -380,13 +359,11 @@ func TestCache_GetVariant_ReturnsContentType(t *testing.T) {
}
func TestCache_GetVariant(t *testing.T) {
t.Parallel()
cache := setupTestCache(t)
cache, _ := setupTestCache(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostCDN,
SourceHost: "cdn.example.com",
SourcePath: "/photos/output.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
@@ -397,9 +374,7 @@ func TestCache_GetVariant(t *testing.T) {
// Store variant
cacheKey := CacheKey(req)
outputContent := []byte("the actual output content")
err := cache.StoreVariant(
t.Context(), cacheKey, bytes.NewReader(outputContent), "image/webp")
err := cache.StoreVariant(cacheKey, bytes.NewReader(outputContent), "image/webp")
if err != nil {
t.Fatalf("StoreVariant() error = %v", err)
}
@@ -415,8 +390,7 @@ func TestCache_GetVariant(t *testing.T) {
if err != nil {
t.Fatalf("GetVariant() error = %v", err)
}
defer func() { _ = reader.Close() }()
defer reader.Close()
buf := make([]byte, 100)
n, _ := reader.Read(buf)
@@ -427,9 +401,7 @@ func TestCache_GetVariant(t *testing.T) {
}
func TestCache_Stats(t *testing.T) {
t.Parallel()
cache := setupTestCache(t)
cache, _ := setupTestCache(t)
ctx := context.Background()
// Increment some stats
@@ -452,8 +424,6 @@ func TestCache_Stats(t *testing.T) {
}
func TestCache_CleanExpired(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
db := setupTestDB(t)
@@ -466,8 +436,7 @@ func TestCache_CleanExpired(t *testing.T) {
// Insert expired negative cache entry directly
_, err := db.ExecContext(ctx, `
INSERT INTO negative_cache
(source_host, source_path, source_query, status_code, expires_at)
INSERT INTO negative_cache (source_host, source_path, source_query, status_code, expires_at)
VALUES ('example.com', '/old.jpg', '', 404, datetime('now', '-1 hour'))
`)
if err != nil {
@@ -476,12 +445,7 @@ func TestCache_CleanExpired(t *testing.T) {
// Verify it exists
var count int
err = db.QueryRowContext(ctx, `SELECT COUNT(*) FROM negative_cache`).Scan(&count)
if err != nil {
t.Fatalf("failed to count negative cache entries: %v", err)
}
db.QueryRowContext(ctx, `SELECT COUNT(*) FROM negative_cache`).Scan(&count)
if count != 1 {
t.Fatalf("expected 1 negative cache entry, got %d", count)
}
@@ -493,19 +457,13 @@ func TestCache_CleanExpired(t *testing.T) {
}
// Verify it's gone
err = db.QueryRowContext(ctx, `SELECT COUNT(*) FROM negative_cache`).Scan(&count)
if err != nil {
t.Fatalf("failed to count negative cache entries: %v", err)
}
db.QueryRowContext(ctx, `SELECT COUNT(*) FROM negative_cache`).Scan(&count)
if count != 0 {
t.Errorf("expected 0 negative cache entries after clean, got %d", count)
}
}
func TestCache_StorageDirectoriesCreated(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
db := setupTestDB(t)
@@ -525,9 +483,7 @@ func TestCache_StorageDirectoriesCreated(t *testing.T) {
for _, dir := range dirs {
path := tmpDir + "/" + dir
_, err := os.Stat(path)
if os.IsNotExist(err) {
if _, err := os.Stat(path); os.IsNotExist(err) {
t.Errorf("directory %s was not created", dir)
}
}

64
internal/imgcache/contentlock.go
View File

@@ -1,64 +0,0 @@
package imgcache
import "sync"
// contentLock provides per-key mutual exclusion for content-hash keyed
// operations. StoreSource and evictSourceBlob each hold a content
// hash's lock for the full duration of their file-plus-accounting-row
// work, so a store and an eviction racing on identical content bytes
// (the real SHA-256 content-addressed dedup case, not a contrived one)
// can never interleave: the unlink of an evicted blob's file can never
// race the creation of a fresh database row for a concurrently
// re-stored copy of the same content. Entries are removed once no
// goroutine holds or is waiting for them, so a long-running process
// does not accumulate memory proportional to the number of distinct
// content hashes it has ever seen.
type contentLock struct {
mu sync.Mutex
entries map[string]*contentLockEntry
}
// contentLockEntry is one key's exclusion lock plus a count of
// goroutines currently holding or waiting to acquire it, used to know
// when it is safe to remove the entry from the map.
type contentLockEntry struct {
mu sync.Mutex
count int
}
// newContentLock creates an empty contentLock.
func newContentLock() *contentLock {
return &contentLock{entries: make(map[string]*contentLockEntry)}
}
// Lock acquires exclusive access for key, blocking until it is
// available, and returns a function that releases it. The caller must
// invoke the returned function exactly once to release the lock.
func (c *contentLock) Lock(key string) func() {
c.mu.Lock()
entry, ok := c.entries[key]
if !ok {
entry = &contentLockEntry{}
c.entries[key] = entry
}
entry.count++
c.mu.Unlock()
entry.mu.Lock()
return func() {
entry.mu.Unlock()
c.mu.Lock()
entry.count--
if entry.count == 0 {
delete(c.entries, key)
}
c.mu.Unlock()
}
}

142
internal/imgcache/contentlock_internal_test.go
View File

@@ -1,142 +0,0 @@
package imgcache
import (
"sync"
"sync/atomic"
"testing"
"time"
)
// TestContentLockExcludesSameKey verifies that two goroutines locking
// the same key never run their critical sections concurrently.
func TestContentLockExcludesSameKey(t *testing.T) {
t.Parallel()
lock := newContentLock()
var (
active atomic.Int32
maxSeen int32
wg sync.WaitGroup
)
const goroutines = 20
wg.Add(goroutines)
for range goroutines {
go func() {
defer wg.Done()
unlock := lock.Lock("same-key")
defer unlock()
n := active.Add(1)
for {
seen := atomic.LoadInt32(&maxSeen)
if n <= seen || atomic.CompareAndSwapInt32(&maxSeen, seen, n) {
break
}
}
time.Sleep(time.Millisecond)
active.Add(-1)
}()
}
wg.Wait()
if maxSeen != 1 {
t.Errorf("max concurrent holders of the same key = %d, want 1", maxSeen)
}
}
// TestContentLockAllowsDifferentKeys verifies that locking distinct
// keys does not serialize unrelated work: all goroutines must be able
// to enter their critical sections at once, proven by every one of
// them reaching the rendezvous point before any is allowed to
// proceed.
func TestContentLockAllowsDifferentKeys(t *testing.T) {
t.Parallel()
lock := newContentLock()
const goroutines = 20
var (
wg sync.WaitGroup
inside atomic.Int32
reached = make(chan struct{}, goroutines)
)
wg.Add(goroutines)
release := make(chan struct{})
for i := range goroutines {
key := string(rune('a' + i))
go func() {
defer wg.Done()
unlock := lock.Lock(key)
defer unlock()
inside.Add(1)
reached <- struct{}{}
<-release
}()
}
// Every goroutine must reach the rendezvous point (i.e. acquire its
// own key's lock) without needing any other to release first. If
// keys were incorrectly serialized onto one underlying lock, only
// one would get here and this would time out.
for i := range goroutines {
select {
case <-reached:
case <-time.After(2 * time.Second):
t.Fatalf("only %d/%d goroutines locking distinct keys made progress; "+
"keys may be incorrectly serialized", i, goroutines)
}
}
if n := inside.Load(); n != goroutines {
t.Errorf("goroutines inside their critical section = %d, want %d",
n, goroutines)
}
close(release)
wg.Wait()
}
// TestContentLockRemovesEntryAfterUnlock verifies that the internal
// entries map does not grow without bound: once no goroutine holds or
// awaits a key, its entry is removed.
func TestContentLockRemovesEntryAfterUnlock(t *testing.T) {
t.Parallel()
lock := newContentLock()
unlock := lock.Lock("k")
lock.mu.Lock()
if _, ok := lock.entries["k"]; !ok {
lock.mu.Unlock()
t.Fatal("entry missing while lock is held")
}
lock.mu.Unlock()
unlock()
lock.mu.Lock()
defer lock.mu.Unlock()
if _, ok := lock.entries["k"]; ok {
t.Error("entry for key still present after the last holder unlocked")
}
}

4
internal/imgcache/divzero_internal_test.go → internal/imgcache/divzero_test.go
View File

@@ -7,8 +7,6 @@ import (
)
func TestSizePercentSafeWithZeroFetchBytes(t *testing.T) {
t.Parallel()
// Simulate the calculation from processAndStore
fetchBytes := int64(0)
outputSize := int64(100)
@@ -31,8 +29,6 @@ func TestSizePercentSafeWithZeroFetchBytes(t *testing.T) {
}
func TestSizePercentNormalCase(t *testing.T) {
t.Parallel()
fetchBytes := int64(1000)
outputSize := int64(500)

803
internal/imgcache/eviction.go
View File

@@ -1,803 +0,0 @@
package imgcache
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"io/fs"
"os"
"path/filepath"
"strings"
"time"
)
// DefaultEvictionInterval is how often the background evictor checks
// cache usage against the configured limit, in addition to the
// write-pressure wakeups triggered by stores.
const DefaultEvictionInterval = 5 * time.Minute
// evictionBatchSize is how many LRU candidates of each class (variants
// and source blobs) one eviction pass fetches from the database.
const evictionBatchSize = 100
// staleTempFileAge is how old an orphaned temp file (left behind by a
// crashed write) must be before reconciliation removes it. Fresh temp
// files may still belong to an in-flight store.
const staleTempFileAge = time.Hour
// sqliteTimestampLayout matches SQLite's CURRENT_TIMESTAMP format, so
// timestamps written by reconciliation order correctly against ones
// written by the hot path.
const sqliteTimestampLayout = "2006-01-02 15:04:05"
// tempFilePrefix is the prefix os.CreateTemp uses for in-flight cache
// writes (".tmp-*" patterns in the storage layer).
const tempFilePrefix = ".tmp-"
// variantMetaSuffix is the sidecar suffix VariantStorage writes next
// to each variant file.
const variantMetaSuffix = ".meta"
// fallbackContentType is recorded when a reconciled variant file has
// no readable .meta sidecar.
const fallbackContentType = "application/octet-stream"
// UsageBytes returns the total number of bytes of cache content
// tracked in the database (source content blobs plus processed
// variants). It never scans the cache directories.
func (c *Cache) UsageBytes(ctx context.Context) (int64, error) {
if c.disabled {
return 0, nil
}
var total int64
err := c.db.QueryRowContext(ctx, `
SELECT (SELECT COALESCE(SUM(size_bytes), 0) FROM source_content)
+ (SELECT COALESCE(SUM(size_bytes), 0) FROM variant_content)
`).Scan(&total)
if err != nil {
return 0, fmt.Errorf("failed to compute cache usage: %w", err)
}
return total, nil
}
// evictionCandidate is one LRU eviction victim candidate: either a
// processed variant (isVariant true, identified by cacheKey) or a
// source content blob (identified by contentHash).
type evictionCandidate struct {
isVariant bool
cacheKey VariantKey
contentHash ContentHash
sizeBytes int64
lastAccessedAt string
}
// EvictToLimit evicts least-recently-used cache entries until total
// tracked usage is at or below the configured MaxBytes limit. It is a
// no-op when the cache is disabled or no limit is configured.
func (c *Cache) EvictToLimit(ctx context.Context) error {
if c.disabled || c.config.MaxBytes <= 0 {
return nil
}
for {
usage, err := c.UsageBytes(ctx)
if err != nil {
return err
}
if usage <= c.config.MaxBytes {
return nil
}
freed, err := c.evictBatch(ctx, usage-c.config.MaxBytes)
if err != nil {
return err
}
if freed == 0 {
c.log.Warn("cache eviction made no progress",
"usage_bytes", usage,
"cache_max_bytes", c.config.MaxBytes,
)
return nil
}
c.log.Info("evicted cache content",
"freed_bytes", freed,
"usage_bytes", usage-freed,
"cache_max_bytes", c.config.MaxBytes,
)
}
}
// evictBatch fetches one batch of LRU candidates across variants and
// source blobs and evicts them oldest-first until excessBytes are
// freed or the batch is exhausted. It returns the bytes freed.
func (c *Cache) evictBatch(ctx context.Context, excessBytes int64) (int64, error) {
candidates, err := c.evictionCandidates(ctx)
if err != nil {
return 0, err
}
var freed int64
for _, candidate := range candidates {
if freed >= excessBytes {
break
}
err := c.evictCandidate(ctx, candidate)
if err != nil {
c.log.Warn("failed to evict cache entry",
"cache_key", candidate.cacheKey,
"content_hash", candidate.contentHash,
"error", err,
)
continue
}
freed += candidate.sizeBytes
}
return freed, nil
}
// evictCandidate removes a single eviction victim.
func (c *Cache) evictCandidate(ctx context.Context, candidate evictionCandidate) error {
if candidate.isVariant {
return c.evictVariant(ctx, candidate.cacheKey)
}
return c.evictSourceBlob(ctx, candidate.contentHash)
}
// evictionCandidates returns up to evictionBatchSize variants and
// evictionBatchSize source blobs, merged into a single list ordered by
// last access time (oldest first).
func (c *Cache) evictionCandidates(ctx context.Context) ([]evictionCandidate, error) {
variants, err := c.variantCandidates(ctx)
if err != nil {
return nil, err
}
sources, err := c.sourceCandidates(ctx)
if err != nil {
return nil, err
}
// Merge the two lists, each already sorted oldest-first. SQLite
// CURRENT_TIMESTAMP strings compare correctly lexicographically.
merged := make([]evictionCandidate, 0, len(variants)+len(sources))
for len(variants) > 0 && len(sources) > 0 {
if variants[0].lastAccessedAt <= sources[0].lastAccessedAt {
merged = append(merged, variants[0])
variants = variants[1:]
} else {
merged = append(merged, sources[0])
sources = sources[1:]
}
}
merged = append(merged, variants...)
merged = append(merged, sources...)
return merged, nil
}
// variantCandidates returns the least recently used variants.
func (c *Cache) variantCandidates(ctx context.Context) ([]evictionCandidate, error) {
rows, err := c.db.QueryContext(ctx, `
SELECT cache_key, size_bytes, last_accessed_at
FROM variant_content
ORDER BY last_accessed_at ASC, cache_key ASC
LIMIT ?
`, evictionBatchSize)
if err != nil {
return nil, fmt.Errorf("failed to query variant eviction candidates: %w", err)
}
defer func() { _ = rows.Close() }()
var candidates []evictionCandidate
for rows.Next() {
candidate := evictionCandidate{isVariant: true}
var key string
err := rows.Scan(&key, &candidate.sizeBytes, &candidate.lastAccessedAt)
if err != nil {
return nil, fmt.Errorf("failed to scan variant candidate: %w", err)
}
candidate.cacheKey = VariantKey(key)
candidates = append(candidates, candidate)
}
err = rows.Err()
if err != nil {
return nil, fmt.Errorf("variant candidate iteration failed: %w", err)
}
return candidates, nil
}
// sourceCandidates returns the least recently used source blobs. Rows
// written before the LRU column existed fall back to fetched_at.
func (c *Cache) sourceCandidates(ctx context.Context) ([]evictionCandidate, error) {
rows, err := c.db.QueryContext(ctx, `
SELECT content_hash, size_bytes,
COALESCE(last_accessed_at, fetched_at, '1970-01-01 00:00:00') AS lru
FROM source_content
ORDER BY lru ASC, content_hash ASC
LIMIT ?
`, evictionBatchSize)
if err != nil {
return nil, fmt.Errorf("failed to query source eviction candidates: %w", err)
}
defer func() { _ = rows.Close() }()
var candidates []evictionCandidate
for rows.Next() {
var candidate evictionCandidate
var hash string
err := rows.Scan(&hash, &candidate.sizeBytes, &candidate.lastAccessedAt)
if err != nil {
return nil, fmt.Errorf("failed to scan source candidate: %w", err)
}
candidate.contentHash = ContentHash(hash)
candidates = append(candidates, candidate)
}
err = rows.Err()
if err != nil {
return nil, fmt.Errorf("source candidate iteration failed: %w", err)
}
return candidates, nil
}
// evictVariant removes one variant: accounting row first, then the
// content and .meta files, so the database never references a deleted
// file.
func (c *Cache) evictVariant(ctx context.Context, cacheKey VariantKey) error {
_, err := c.db.ExecContext(ctx,
`DELETE FROM variant_content WHERE cache_key = ?`, string(cacheKey))
if err != nil {
return fmt.Errorf("failed to delete variant accounting row: %w", err)
}
err = c.variants.DeleteWithMeta(cacheKey)
if err != nil {
return err
}
return nil
}
// sourceReference identifies one source_metadata row's JSON sidecar.
type sourceReference struct {
host string
pathHash PathHash
}
// evictSourceBlob removes one source content blob. All source_metadata
// rows referencing the blob are deleted together with its
// source_content row in a single transaction BEFORE the file is
// unlinked: a blob referenced by multiple source paths is only ever
// removed together with all of its references, and database rows never
// point at deleted files. The JSON metadata sidecars for the removed
// rows are deleted afterwards.
//
// The whole operation holds the content hash's lock (the same one
// StoreSource holds for its full store), so a concurrent store of
// identical content bytes can never observe the file gone but a row
// still present, or insert a fresh row between this transaction's
// commit and the file unlink below: it either runs entirely before
// this eviction starts, or is blocked until this eviction (row
// deletion and unlink together) has fully completed.
func (c *Cache) evictSourceBlob(ctx context.Context, contentHash ContentHash) error {
unlock := c.contentLocks.Lock(string(contentHash))
defer unlock()
references, err := c.sourceReferences(ctx, contentHash)
if err != nil {
return err
}
tx, err := c.db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("failed to begin eviction transaction: %w", err)
}
defer func() { _ = tx.Rollback() }()
_, err = tx.ExecContext(ctx,
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash))
if err != nil {
return fmt.Errorf("failed to delete source metadata rows: %w", err)
}
_, err = tx.ExecContext(ctx,
`DELETE FROM source_content WHERE content_hash = ?`, string(contentHash))
if err != nil {
return fmt.Errorf("failed to delete source content row: %w", err)
}
err = tx.Commit()
if err != nil {
return fmt.Errorf("failed to commit eviction transaction: %w", err)
}
if c.evictSourceBlobTestHook != nil {
c.evictSourceBlobTestHook(contentHash)
}
// Only after the rows are gone may the files be removed.
for _, reference := range references {
err := c.srcMetadata.Delete(reference.host, reference.pathHash)
if err != nil {
c.log.Warn("failed to delete metadata sidecar",
"host", reference.host, "path_hash", reference.pathHash, "error", err)
}
}
err = c.srcContent.Delete(contentHash)
if err != nil {
return err
}
return nil
}
// sourceReferences lists the metadata sidecar locations of every
// source_metadata row referencing the given blob.
func (c *Cache) sourceReferences(
ctx context.Context, contentHash ContentHash,
) ([]sourceReference, error) {
rows, err := c.db.QueryContext(ctx, `
SELECT source_host, path_hash FROM source_metadata WHERE content_hash = ?
`, string(contentHash))
if err != nil {
return nil, fmt.Errorf("failed to query source references: %w", err)
}
defer func() { _ = rows.Close() }()
var references []sourceReference
for rows.Next() {
var reference sourceReference
var pathHash string
err := rows.Scan(&reference.host, &pathHash)
if err != nil {
return nil, fmt.Errorf("failed to scan source reference: %w", err)
}
reference.pathHash = PathHash(pathHash)
references = append(references, reference)
}
err = rows.Err()
if err != nil {
return nil, fmt.Errorf("source reference iteration failed: %w", err)
}
return references, nil
}
// notifyWritePressure wakes the background evictor after a store, so
// eviction under write pressure happens promptly without blocking the
// storing request. The notification channel has capacity one and drops
// when a wakeup is already pending.
func (c *Cache) notifyWritePressure() {
if c.disabled || c.config.MaxBytes <= 0 {
return
}
select {
case c.evictionPressure <- struct{}{}:
default:
}
}
// StartEviction launches the background eviction goroutine, which
// reconciles the database accounting with the cache directories at
// startup and again on every periodic tick thereafter, and evicts to
// the configured limit on the given periodic interval and on
// write-pressure notifications. It is a no-op on a disabled cache or
// when already started.
func (c *Cache) StartEviction(interval time.Duration) {
if c.disabled || c.evictionStarted {
return
}
c.evictionStarted = true
go c.evictionLoop(interval)
}
// StopEviction stops the background eviction goroutine and waits for
// it to exit. It is safe to call when eviction was never started, and
// safe to call more than once.
func (c *Cache) StopEviction() {
if !c.evictionStarted {
return
}
c.evictionStopOnce.Do(func() {
close(c.evictionStop)
<-c.evictionDone
})
}
// evictionLoop is the body of the background eviction goroutine.
func (c *Cache) evictionLoop(interval time.Duration) {
defer close(c.evictionDone)
ctx := context.Background()
c.runReconciliationPass(ctx)
c.runEvictionPass(ctx)
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-c.evictionStop:
return
case <-ticker.C:
// Reconciliation walks the cache directories, so it only
// runs on the periodic ticker rather than on every
// write-pressure wakeup, keeping it off the per-store hot
// path. Reusing the eviction interval itself (rather than a
// separate, longer one) is a deliberate choice: it is the
// simplest option that still bounds how long a store's
// best-effort accounting insert can stay silently
// unaccounted for to one interval, on a process that is
// already running this loop regardless.
c.runReconciliationPass(ctx)
case <-c.evictionPressure:
}
c.runEvictionPass(ctx)
}
}
// runEvictionPass runs one eviction pass, logging failures instead of
// propagating them (the loop must keep running).
func (c *Cache) runEvictionPass(ctx context.Context) {
err := c.EvictToLimit(ctx)
if err != nil {
c.log.Warn("cache eviction pass failed", "error", err)
}
}
// runReconciliationPass runs one reconciliation pass, logging failures
// instead of propagating them (the loop must keep running).
func (c *Cache) runReconciliationPass(ctx context.Context) {
err := c.reconcileAccounting(ctx)
if err != nil {
c.log.Warn("cache accounting reconciliation failed", "error", err)
}
}
// reconcileAccounting synchronizes the database size accounting with
// the actual contents of the cache directories. It runs at startup and
// again on every periodic eviction tick thereafter, off the request
// hot path: it adopts variant files that predate the accounting table
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
// insert under transient DB contention), drops accounting rows whose
// files are missing, removes source blob files the database does not
// know (and rows whose files are gone), and sweeps stale temp files
// left behind by crashed writes. Running it periodically, not just
// once, bounds how long such drift can accumulate unaccounted for on a
// long-running process to one eviction interval.
func (c *Cache) reconcileAccounting(ctx context.Context) error {
if c.disabled {
return nil
}
err := c.reconcileVariantFiles(ctx)
if err != nil {
return err
}
err = c.reconcileVariantRows(ctx)
if err != nil {
return err
}
err = c.reconcileSourceFiles(ctx)
if err != nil {
return err
}
err = c.reconcileSourceRows(ctx)
if err != nil {
return err
}
return nil
}
// reconcileVariantFiles walks the variant storage directory, adopting
// files without accounting rows and sweeping stale temp files.
func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
return filepath.WalkDir(
c.variants.baseDir,
func(path string, entry fs.DirEntry, err error) error {
if err != nil || entry.IsDir() {
return err
}
name := entry.Name()
if strings.HasPrefix(name, tempFilePrefix) {
c.sweepStaleTempFile(path, entry)
return nil
}
if strings.HasSuffix(name, variantMetaSuffix) {
return nil
}
return c.adoptVariantFile(ctx, path, entry, VariantKey(name))
},
)
}
// adoptVariantFile inserts an accounting row for a variant file that
// has none, using the file's size and modification time.
func (c *Cache) adoptVariantFile(
ctx context.Context, path string, entry fs.DirEntry, cacheKey VariantKey,
) error {
var rowExists int
err := c.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, string(cacheKey),
).Scan(&rowExists)
if err != nil {
return fmt.Errorf("failed to check variant accounting row: %w", err)
}
if rowExists > 0 {
return nil
}
info, err := entry.Info()
if err != nil {
return fmt.Errorf("failed to stat variant file: %w", err)
}
modTime := info.ModTime().UTC().Format(sqliteTimestampLayout)
contentType := c.variantContentTypeFromSidecar(path)
_, err = c.db.ExecContext(ctx, `
INSERT INTO variant_content
(cache_key, size_bytes, content_type, created_at, last_accessed_at)
VALUES (?, ?, ?, ?, ?)
`, string(cacheKey), info.Size(), contentType, modTime, modTime)
if err != nil {
return fmt.Errorf("failed to adopt variant file into accounting: %w", err)
}
c.log.Info("adopted untracked variant file into size accounting",
"cache_key", cacheKey, "size_bytes", info.Size())
return nil
}
// variantContentTypeFromSidecar reads the content type from a variant
// .meta sidecar, falling back to application/octet-stream.
func (c *Cache) variantContentTypeFromSidecar(variantPath string) string {
//nolint:gosec // path from cache walk
metaData, err := os.ReadFile(variantPath + variantMetaSuffix)
if err != nil {
return fallbackContentType
}
var meta VariantMeta
if json.Unmarshal(metaData, &meta) != nil || meta.ContentType == "" {
return fallbackContentType
}
return meta.ContentType
}
// reconcileVariantRows drops accounting rows whose variant files are
// missing, so the database never references deleted content.
func (c *Cache) reconcileVariantRows(ctx context.Context) error {
keys, err := c.allVariantKeys(ctx)
if err != nil {
return err
}
for _, key := range keys {
if c.variants.Exists(key) {
continue
}
_, err := c.db.ExecContext(ctx,
`DELETE FROM variant_content WHERE cache_key = ?`, string(key))
if err != nil {
return fmt.Errorf("failed to drop stale variant accounting row: %w", err)
}
c.log.Info("dropped accounting row for missing variant file", "cache_key", key)
}
return nil
}
// allVariantKeys returns every tracked variant cache key.
func (c *Cache) allVariantKeys(ctx context.Context) ([]VariantKey, error) {
return queryStringColumn[VariantKey](ctx, c.db,
`SELECT cache_key FROM variant_content`, "variant keys", "variant key")
}
// queryStringColumn runs a single-column query and returns the column
// values as T. plural names the set for the query and scan failure
// messages; singular names one row for the scan and iteration failure
// messages.
func queryStringColumn[T ~string](
ctx context.Context, db *sql.DB, query, plural, singular string,
) ([]T, error) {
rows, err := db.QueryContext(ctx, query)
if err != nil {
return nil, fmt.Errorf("failed to query %s: %w", plural, err)
}
defer func() { _ = rows.Close() }()
var values []T
for rows.Next() {
var value string
err := rows.Scan(&value)
if err != nil {
return nil, fmt.Errorf("failed to scan %s: %w", singular, err)
}
values = append(values, T(value))
}
err = rows.Err()
if err != nil {
return nil, fmt.Errorf("%s iteration failed: %w", singular, err)
}
return values, nil
}
// reconcileSourceFiles walks the source content directory, removing
// blob files the database does not track (they are unreachable: source
// lookups always go through source_metadata) and sweeping stale temp
// files.
func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
return filepath.WalkDir(
c.srcContent.baseDir,
func(path string, entry fs.DirEntry, err error) error {
if err != nil || entry.IsDir() {
return err
}
name := entry.Name()
if strings.HasPrefix(name, tempFilePrefix) {
c.sweepStaleTempFile(path, entry)
return nil
}
return c.removeUntrackedSourceFile(ctx, path, ContentHash(name))
},
)
}
// removeUntrackedSourceFile deletes a source blob file that has no
// source_content row. Any source_metadata rows referencing the hash
// are removed first so no row ever points at a deleted file.
func (c *Cache) removeUntrackedSourceFile(
ctx context.Context, path string, contentHash ContentHash,
) error {
var rowExists int
err := c.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM source_content WHERE content_hash = ?`, string(contentHash),
).Scan(&rowExists)
if err != nil {
return fmt.Errorf("failed to check source content row: %w", err)
}
if rowExists > 0 {
return nil
}
_, err = c.db.ExecContext(ctx,
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash))
if err != nil {
return fmt.Errorf("failed to delete metadata rows for untracked blob: %w", err)
}
err = os.Remove(path)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("failed to remove untracked source file: %w", err)
}
c.log.Info("removed untracked source content file", "content_hash", contentHash)
return nil
}
// reconcileSourceRows removes source_content rows (and their metadata
// references and sidecars) whose blob files are missing on disk.
func (c *Cache) reconcileSourceRows(ctx context.Context) error {
hashes, err := c.allSourceContentHashes(ctx)
if err != nil {
return err
}
for _, hash := range hashes {
if c.srcContent.Exists(hash) {
continue
}
// The blob file is already gone; evictSourceBlob removes the
// rows and sidecars and tolerates the missing file.
err := c.evictSourceBlob(ctx, hash)
if err != nil {
return err
}
c.log.Info("dropped rows for missing source content file", "content_hash", hash)
}
return nil
}
// allSourceContentHashes returns every tracked source content hash.
func (c *Cache) allSourceContentHashes(ctx context.Context) ([]ContentHash, error) {
return queryStringColumn[ContentHash](ctx, c.db,
`SELECT content_hash FROM source_content`,
"source content hashes", "content hash")
}
// sweepStaleTempFile removes a temp file left behind by a crashed
// write once it is old enough that no in-flight store can own it.
func (c *Cache) sweepStaleTempFile(path string, entry fs.DirEntry) {
info, err := entry.Info()
if err != nil {
return
}
if time.Since(info.ModTime()) < staleTempFileAge {
return
}
err = os.Remove(path)
if err != nil && !os.IsNotExist(err) {
c.log.Warn("failed to remove stale temp file", "path", path, "error", err)
return
}
c.log.Info("removed stale temp file", "path", path)
}

985
internal/imgcache/eviction_internal_test.go
View File

@@ -1,985 +0,0 @@
package imgcache
import (
"bytes"
"context"
"database/sql"
"io/fs"
"os"
"path/filepath"
"testing"
"time"
_ "modernc.org/sqlite"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher"
)
// sqliteTimestampFormat matches the format SQLite's CURRENT_TIMESTAMP
// produces, so injected timestamps compare correctly against ones the
// implementation writes.
const sqliteTimestampFormat = "2006-01-02 15:04:05"
// testVariantKeyOne through testVariantKeyFour are the variant cache
// keys reused across the eviction tests, in the order the tests store
// them.
const (
testVariantKeyOne VariantKey = "aabbccdd0001"
testVariantKeyTwo VariantKey = "aabbccdd0002"
testVariantKeyThree VariantKey = "aabbccdd0003"
testVariantKeyFour VariantKey = "aabbccdd0004"
)
// evictionTestDB creates an in-memory SQLite database with the real
// production schema, limited to a single connection so the background
// eviction goroutine shares the same in-memory database as the test.
func evictionTestDB(t *testing.T) *sql.DB {
t.Helper()
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatalf("failed to open test db: %v", err)
}
db.SetMaxOpenConns(1)
err = database.ApplyMigrations(context.Background(), db, nil)
if err != nil {
t.Fatalf("failed to apply migrations: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
return db
}
// newEvictionTestCache creates a Cache backed by a temp directory and
// an in-memory database, with the given size limit.
func newEvictionTestCache(t *testing.T, maxBytes int64) (*Cache, string) {
t.Helper()
tmpDir := t.TempDir()
db := evictionTestDB(t)
// maxBytes zero mirrors the production mapping of
// cache_max_bytes: 0 (handlers sets DisableDiskCache); at the
// CacheConfig layer itself a zero MaxBytes means "no limit" for
// backwards compatibility with existing fixtures.
cache, err := NewCache(db, CacheConfig{
StateDir: tmpDir,
CacheTTL: time.Hour,
NegativeTTL: 5 * time.Minute,
MaxBytes: maxBytes,
DisableDiskCache: maxBytes == 0,
})
if err != nil {
t.Fatalf("failed to create cache: %v", err)
}
return cache, tmpDir
}
// storeEvictionTestSource stores content as a fetched source for
// host/path and returns the resulting content hash.
func storeEvictionTestSource(
t *testing.T, cache *Cache, host, path string, content []byte,
) ContentHash {
t.Helper()
req := &ImageRequest{
SourceHost: host,
SourcePath: path,
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
result := &httpfetcher.FetchResult{
StatusCode: 200,
ContentType: testContentTypeJPEG,
ContentLength: int64(len(content)),
Headers: map[string][]string{
testHeaderContentType: {testContentTypeJPEG},
},
}
hash, err := cache.StoreSource(
context.Background(), req, bytes.NewReader(content), result,
)
if err != nil {
t.Fatalf("StoreSource(%s%s) failed: %v", host, path, err)
}
return hash
}
// storeEvictionTestVariant stores content as a processed variant under
// the given cache key.
func storeEvictionTestVariant(
t *testing.T, cache *Cache, key VariantKey, content []byte,
) {
t.Helper()
err := cache.StoreVariant(t.Context(), key, bytes.NewReader(content), "image/webp")
if err != nil {
t.Fatalf("StoreVariant(%s) failed: %v", key, err)
}
}
// setVariantLastAccessed backdates the last access time of a tracked
// variant, to make LRU ordering deterministic in tests.
func setVariantLastAccessed(
t *testing.T, cache *Cache, key VariantKey, when time.Time,
) {
t.Helper()
res, err := cache.db.ExecContext(t.Context(),
`UPDATE variant_content SET last_accessed_at = ? WHERE cache_key = ?`,
when.UTC().Format(sqliteTimestampFormat), string(key),
)
if err != nil {
t.Fatalf("failed to set variant last_accessed_at: %v", err)
}
affected, err := res.RowsAffected()
if err != nil {
t.Fatalf("failed to read affected rows: %v", err)
}
if affected != 1 {
t.Fatalf("variant %s has no accounting row (affected=%d); "+
"stores must track variants in the database", key, affected)
}
}
// setSourceLastAccessed backdates the last access time of a tracked
// source content blob.
func setSourceLastAccessed(
t *testing.T, cache *Cache, hash ContentHash, when time.Time,
) {
t.Helper()
res, err := cache.db.ExecContext(t.Context(),
`UPDATE source_content SET last_accessed_at = ? WHERE content_hash = ?`,
when.UTC().Format(sqliteTimestampFormat), string(hash),
)
if err != nil {
t.Fatalf("failed to set source last_accessed_at: %v", err)
}
affected, err := res.RowsAffected()
if err != nil {
t.Fatalf("failed to read affected rows: %v", err)
}
if affected != 1 {
t.Fatalf("source %s has no accounting row (affected=%d)", hash, affected)
}
}
// countRows returns the number of rows the given query yields.
func countRows(t *testing.T, cache *Cache, query string, args ...any) int {
t.Helper()
var n int
err := cache.db.QueryRowContext(t.Context(), query, args...).Scan(&n)
if err != nil {
t.Fatalf("count query %q failed: %v", query, err)
}
return n
}
// assertNoDanglingReferences verifies the core eviction invariant:
// every database row that references cache content on disk points at a
// file that actually exists.
func assertNoDanglingReferences(t *testing.T, cache *Cache) {
t.Helper()
rows, err := cache.db.QueryContext(t.Context(),
`SELECT content_hash FROM source_metadata
WHERE content_hash IS NOT NULL AND content_hash != ''`,
)
if err != nil {
t.Fatalf("failed to query source_metadata: %v", err)
}
defer func() { _ = rows.Close() }()
for rows.Next() {
var hash string
err := rows.Scan(&hash)
if err != nil {
t.Fatalf("failed to scan content_hash: %v", err)
}
if !cache.srcContent.Exists(ContentHash(hash)) {
t.Errorf("source_metadata references content %s but the file is missing",
hash)
}
}
err = rows.Err()
if err != nil {
t.Fatalf("source_metadata iteration failed: %v", err)
}
variantRows, err := cache.db.QueryContext(t.Context(),
`SELECT cache_key FROM variant_content`)
if err != nil {
t.Fatalf("failed to query variant_content: %v", err)
}
defer func() { _ = variantRows.Close() }()
for variantRows.Next() {
var key string
err := variantRows.Scan(&key)
if err != nil {
t.Fatalf("failed to scan cache_key: %v", err)
}
if !cache.variants.Exists(VariantKey(key)) {
t.Errorf("variant_content references key %s but the file is missing", key)
}
}
err = variantRows.Err()
if err != nil {
t.Fatalf("variant_content iteration failed: %v", err)
}
}
// waitForUsageAtOrBelow polls UsageBytes until it reaches limit or the
// timeout expires, returning the last observed usage.
func waitForUsageAtOrBelow(
t *testing.T, cache *Cache, limit int64, timeout time.Duration,
) int64 {
t.Helper()
deadline := time.Now().Add(timeout)
var usage int64
for time.Now().Before(deadline) {
var err error
usage, err = cache.UsageBytes(context.Background())
if err != nil {
t.Fatalf("UsageBytes failed: %v", err)
}
if usage <= limit {
return usage
}
time.Sleep(25 * time.Millisecond)
}
return usage
}
func TestUsageBytesAccountsSourceAndVariantBytes(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
storeEvictionTestSource(t, cache, "src.example.com", "/a.jpg",
bytes.Repeat([]byte{0xAA}, 1000))
storeEvictionTestSource(t, cache, "src.example.com", "/b.jpg",
bytes.Repeat([]byte{0xAB}, 2000))
storeEvictionTestVariant(t, cache, testVariantKeyOne,
bytes.Repeat([]byte{0xAC}, 500))
storeEvictionTestVariant(t, cache, testVariantKeyTwo,
bytes.Repeat([]byte{0xAD}, 250))
usage, err := cache.UsageBytes(context.Background())
if err != nil {
t.Fatalf("UsageBytes failed: %v", err)
}
if usage != 3750 {
t.Errorf("UsageBytes = %d, want 3750 (1000+2000+500+250)", usage)
}
}
func TestUsageBytesCountsMultiReferencedBlobOnce(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
content := bytes.Repeat([]byte{0xCC}, 1200)
hashOne := storeEvictionTestSource(t, cache,
"src.example.com", "/one.jpg", content)
hashTwo := storeEvictionTestSource(t, cache,
"src.example.com", "/two.jpg", content)
if hashOne != hashTwo {
t.Fatalf("identical content produced different hashes: %s vs %s",
hashOne, hashTwo)
}
usage, err := cache.UsageBytes(context.Background())
if err != nil {
t.Fatalf("UsageBytes failed: %v", err)
}
if usage != 1200 {
t.Errorf("UsageBytes = %d, want 1200 (deduplicated blob counted once)", usage)
}
}
func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
t.Parallel()
const limit = 3000
cache, _ := newEvictionTestCache(t, limit)
now := time.Now()
keys := []VariantKey{
testVariantKeyOne, testVariantKeyTwo,
testVariantKeyThree, testVariantKeyFour,
}
fills := []byte{0x01, 0x02, 0x03, 0x04}
ages := []time.Duration{4 * time.Hour, 3 * time.Hour, 2 * time.Hour, 1 * time.Hour}
for i, key := range keys {
storeEvictionTestVariant(t, cache, key, bytes.Repeat([]byte{fills[i]}, 1000))
setVariantLastAccessed(t, cache, key, now.Add(-ages[i]))
}
err := cache.EvictToLimit(context.Background())
if err != nil {
t.Fatalf("EvictToLimit failed: %v", err)
}
usage, err := cache.UsageBytes(context.Background())
if err != nil {
t.Fatalf("UsageBytes failed: %v", err)
}
if usage > limit {
t.Errorf("usage after eviction = %d, want <= %d", usage, limit)
}
if cache.variants.Exists(keys[0]) {
t.Errorf("least recently used variant %s must be evicted", keys[0])
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, string(keys[0]),
); n != 0 {
t.Errorf("evicted variant %s still has %d accounting rows", keys[0], n)
}
for _, key := range keys[1:] {
if !cache.variants.Exists(key) {
t.Errorf("more recently used variant %s must survive eviction", key)
}
}
assertNoDanglingReferences(t, cache)
}
func TestEvictionRemovesMultiReferencedBlobTogetherWithAllReferences(t *testing.T) {
t.Parallel()
const limit = 1000
cache, _ := newEvictionTestCache(t, limit)
now := time.Now()
// One 800-byte blob referenced by two source paths.
sharedContent := bytes.Repeat([]byte{0xDD}, 800)
sharedHash := storeEvictionTestSource(t, cache,
"src.example.com", "/a.jpg", sharedContent)
h := storeEvictionTestSource(t, cache,
"src.example.com", "/b.jpg", sharedContent)
if h != sharedHash {
t.Fatalf("identical content produced different hashes: %s vs %s",
h, sharedHash)
}
// A newer 600-byte blob referenced by one source path.
recentHash := storeEvictionTestSource(t, cache, "src.example.com", "/c.jpg",
bytes.Repeat([]byte{0xEE}, 600))
setSourceLastAccessed(t, cache, sharedHash, now.Add(-2*time.Hour))
setSourceLastAccessed(t, cache, recentHash, now.Add(-time.Minute))
err := cache.EvictToLimit(context.Background())
if err != nil {
t.Fatalf("EvictToLimit failed: %v", err)
}
usage, err := cache.UsageBytes(context.Background())
if err != nil {
t.Fatalf("UsageBytes failed: %v", err)
}
if usage > limit {
t.Errorf("usage after eviction = %d, want <= %d", usage, limit)
}
// The multi-referenced blob must be gone from disk, from
// source_content, and from BOTH source_metadata rows: references
// are removed together with the blob, never left dangling.
if cache.srcContent.Exists(sharedHash) {
t.Errorf("evicted blob %s still exists on disk", sharedHash)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM source_content WHERE content_hash = ?`, string(sharedHash),
); n != 0 {
t.Errorf("evicted blob %s still has %d source_content rows", sharedHash, n)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(sharedHash),
); n != 0 {
t.Errorf("evicted blob %s still has %d source_metadata references", sharedHash, n)
}
// The JSON metadata sidecars for both referencing paths must be
// removed along with the rows.
for _, path := range []string{"/a.jpg", "/b.jpg"} {
pathHash := HashPath(path + "?")
if cache.srcMetadata.Exists("src.example.com", pathHash) {
t.Errorf("metadata sidecar for %s must be removed with its row", path)
}
}
// The more recently used blob survives fully intact.
if !cache.srcContent.Exists(recentHash) {
t.Errorf("recently used blob %s must survive eviction", recentHash)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(recentHash),
); n != 1 {
t.Errorf("recently used blob %s has %d source_metadata rows, want 1",
recentHash, n)
}
assertNoDanglingReferences(t, cache)
}
func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
content := bytes.Repeat([]byte{0xDF}, 800)
hash := storeEvictionTestSource(t, cache, "src.example.com", "/a.jpg", content)
h := storeEvictionTestSource(t, cache, "src.example.com", "/b.jpg", content)
if h != hash {
t.Fatalf("identical content produced different hashes: %s vs %s", h, hash)
}
storeEvictionTestVariant(t, cache, testVariantKeyOne,
bytes.Repeat([]byte{0xE0}, 500))
err := cache.EvictToLimit(context.Background())
if err != nil {
t.Fatalf("EvictToLimit failed: %v", err)
}
if !cache.srcContent.Exists(hash) {
t.Errorf("blob %s must not be evicted while usage is under the limit", hash)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM source_metadata WHERE content_hash = ?`, string(hash),
); n != 2 {
t.Errorf("blob %s has %d source_metadata rows, want 2", hash, n)
}
if !cache.variants.Exists(testVariantKeyOne) {
t.Error("variant must not be evicted while usage is under the limit")
}
assertNoDanglingReferences(t, cache)
}
func TestZeroMaxBytesDisablesDiskCache(t *testing.T) {
t.Parallel()
cache, tmpDir := newEvictionTestCache(t, 0)
req := &ImageRequest{
SourceHost: "src.example.com",
SourcePath: "/a.jpg",
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
assertDisabledCacheWritesAreNoOps(t, cache, req)
assertDisabledCacheReadsAlwaysMiss(t, cache, req)
assertDisabledCacheTracksNothing(t, cache)
assertDisabledCacheWroteNothingToDisk(t, tmpDir)
}
// assertDisabledCacheWritesAreNoOps verifies that stores against a
// disabled cache report success without recording anything.
func assertDisabledCacheWritesAreNoOps(
t *testing.T, cache *Cache, req *ImageRequest,
) {
t.Helper()
ctx := t.Context()
err := cache.StoreVariant(
ctx, CacheKey(req), bytes.NewReader([]byte("data")), "image/webp",
)
if err != nil {
t.Fatalf("StoreVariant on disabled cache must be a no-op, got error: %v", err)
}
result := &httpfetcher.FetchResult{
StatusCode: 200,
ContentType: testContentTypeJPEG,
ContentLength: 4,
Headers: map[string][]string{},
}
hash, err := cache.StoreSource(ctx, req, bytes.NewReader([]byte("data")), result)
if err != nil {
t.Fatalf("StoreSource on disabled cache must be a no-op, got error: %v", err)
}
if hash != "" {
t.Errorf("StoreSource on disabled cache returned hash %q, want empty", hash)
}
}
// assertDisabledCacheReadsAlwaysMiss verifies that lookups against a
// disabled cache never report a hit.
func assertDisabledCacheReadsAlwaysMiss(
t *testing.T, cache *Cache, req *ImageRequest,
) {
t.Helper()
ctx := t.Context()
lookup, err := cache.Lookup(ctx, req)
if err != nil {
t.Fatalf("Lookup on disabled cache failed: %v", err)
}
if lookup.Hit {
t.Error("Lookup on disabled cache must always miss")
}
srcHash, srcType, err := cache.LookupSource(ctx, req)
if err != nil {
t.Fatalf("LookupSource on disabled cache failed: %v", err)
}
if srcHash != "" || srcType != "" {
t.Errorf("LookupSource on disabled cache = (%q, %q), want empty",
srcHash, srcType)
}
}
// assertDisabledCacheTracksNothing verifies that a disabled cache
// records no usage and writes no accounting rows.
func assertDisabledCacheTracksNothing(t *testing.T, cache *Cache) {
t.Helper()
usage, err := cache.UsageBytes(t.Context())
if err != nil {
t.Fatalf("UsageBytes failed: %v", err)
}
if usage != 0 {
t.Errorf("UsageBytes on disabled cache = %d, want 0", usage)
}
if n := countRows(t, cache, `SELECT COUNT(*) FROM source_content`); n != 0 {
t.Errorf("disabled cache wrote %d source_content rows, want 0", n)
}
if n := countRows(t, cache, `SELECT COUNT(*) FROM source_metadata`); n != 0 {
t.Errorf("disabled cache wrote %d source_metadata rows, want 0", n)
}
}
// assertDisabledCacheWroteNothingToDisk verifies that a disabled cache
// creates neither the cache directory tree nor any file under stateDir.
func assertDisabledCacheWroteNothingToDisk(t *testing.T, stateDir string) {
t.Helper()
_, err := os.Stat(filepath.Join(stateDir, "cache"))
if !os.IsNotExist(err) {
t.Errorf("disabled cache must not create the cache directory tree "+
"(stat err=%v)", err)
}
var foundFiles []string
walkErr := filepath.WalkDir(stateDir,
func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if !d.IsDir() {
foundFiles = append(foundFiles, path)
}
return nil
})
if walkErr != nil {
t.Fatalf("failed to walk state dir: %v", walkErr)
}
if len(foundFiles) != 0 {
t.Errorf("disabled cache wrote files to disk: %v", foundFiles)
}
}
func TestEvictionRunsUnderWritePressure(t *testing.T) {
t.Parallel()
const limit = 1500
cache, _ := newEvictionTestCache(t, limit)
// An interval far longer than the test ensures only write
// pressure can trigger eviction here.
cache.StartEviction(time.Hour)
defer cache.StopEviction()
keys := []VariantKey{
testVariantKeyOne, testVariantKeyTwo, testVariantKeyThree,
}
fills := []byte{0x11, 0x12, 0x13}
for i, key := range keys {
storeEvictionTestVariant(t, cache, key,
bytes.Repeat([]byte{fills[i]}, 1000))
}
usage := waitForUsageAtOrBelow(t, cache, limit, 5*time.Second)
if usage > limit {
t.Errorf("write pressure did not trigger eviction: usage = %d, want <= %d",
usage, limit)
}
assertNoDanglingReferences(t, cache)
}
func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
t.Parallel()
const limit = 1500
cache, _ := newEvictionTestCache(t, limit)
// Start the evictor while the cache is empty, then create tracked
// over-limit state WITHOUT going through the store methods, so no
// write-pressure notification fires and only the periodic ticker
// can trigger eviction.
cache.StartEviction(100 * time.Millisecond)
defer cache.StopEviction()
keys := []VariantKey{
testVariantKeyOne, testVariantKeyTwo, testVariantKeyThree,
}
fills := []byte{0x21, 0x22, 0x23}
for i, key := range keys {
content := bytes.Repeat([]byte{fills[i]}, 1000)
_, err := cache.variants.Store(key, bytes.NewReader(content), "image/webp")
if err != nil {
t.Fatalf("failed to store variant file: %v", err)
}
_, err = cache.db.ExecContext(t.Context(),
`INSERT INTO variant_content (cache_key, size_bytes, content_type)
VALUES (?, ?, ?)`,
string(key), len(content), "image/webp",
)
if err != nil {
t.Fatalf("failed to insert variant accounting row: %v", err)
}
}
usage := waitForUsageAtOrBelow(t, cache, limit, 5*time.Second)
if usage > limit {
t.Errorf("periodic schedule did not trigger eviction: usage = %d, want <= %d",
usage, limit)
}
assertNoDanglingReferences(t, cache)
}
func TestStartEvictionReconcilesAccountingWithDisk(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
// An untracked variant file on disk (e.g. written before this
// feature existed) must be adopted into the accounting.
untracked := bytes.Repeat([]byte{0x31}, 1000)
_, err := cache.variants.Store(
testVariantKeyOne, bytes.NewReader(untracked), "image/webp",
)
if err != nil {
t.Fatalf("failed to store untracked variant file: %v", err)
}
// An accounting row whose file is missing must be dropped.
_, err = cache.db.ExecContext(t.Context(),
`INSERT INTO variant_content (cache_key, size_bytes, content_type)
VALUES (?, ?, ?)`,
"deadbeef0001", 700, "image/webp",
)
if err != nil {
t.Fatalf("failed to insert stale variant accounting row: %v", err)
}
cache.StartEviction(time.Hour)
defer cache.StopEviction()
deadline := time.Now().Add(5 * time.Second)
var usage int64
for time.Now().Before(deadline) {
var err error
usage, err = cache.UsageBytes(context.Background())
if err != nil {
t.Fatalf("UsageBytes failed: %v", err)
}
if usage == 1000 {
break
}
time.Sleep(25 * time.Millisecond)
}
if usage != 1000 {
t.Errorf("usage after reconciliation = %d, want 1000 "+
"(untracked file adopted, stale row dropped)", usage)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`,
string(testVariantKeyOne),
); n != 1 {
t.Errorf("untracked variant file was not adopted into accounting (rows=%d)", n)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, "deadbeef0001",
); n != 0 {
t.Errorf("stale accounting row without a file was not dropped (rows=%d)", n)
}
}
// TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup proves
// reconciliation is not a one-shot startup-only pass: it must also run
// on the periodic ticker, so a variant file that lands on disk with no
// accounting row well after startup (e.g. because StoreVariant's
// best-effort accounting insert failed under transient contention, or
// any other cause of an untracked file appearing during steady-state
// operation) is still adopted into accounting eventually, rather than
// staying invisible to UsageBytes/EvictToLimit until the next process
// restart.
func TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
const interval = 100 * time.Millisecond
cache.StartEviction(interval)
defer cache.StopEviction()
// Let startup reconciliation run and settle on an empty cache
// before introducing the untracked file, so the adoption we assert
// below can only be the work of a later, periodic pass.
time.Sleep(3 * interval)
// Simulate a variant whose accounting insert failed after the
// process was already running and serving requests: the content
// file is written directly, bypassing StoreVariant's (and thus its
// accounting insert) entirely, exactly as would happen if that
// insert had failed and only the file write had succeeded.
untracked := bytes.Repeat([]byte{0x41}, 900)
_, err := cache.variants.Store(
"aabbccdd0099", bytes.NewReader(untracked), "image/webp",
)
if err != nil {
t.Fatalf("failed to store untracked variant file: %v", err)
}
deadline := time.Now().Add(5 * time.Second)
var usage int64
for time.Now().Before(deadline) {
var err error
usage, err = cache.UsageBytes(context.Background())
if err != nil {
t.Fatalf("UsageBytes failed: %v", err)
}
if usage == 900 {
break
}
time.Sleep(25 * time.Millisecond)
}
if usage != 900 {
t.Errorf("usage after periodic reconciliation = %d, want 900 "+
"(a file that appeared after startup reconciliation already ran must still "+
"be adopted by a later periodic pass)", usage)
}
if n := countRows(t, cache,
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, "aabbccdd0099",
); n != 1 {
t.Errorf("file that appeared after startup was not adopted by periodic "+
"reconciliation (rows=%d)", n)
}
}
// TestEvictSourceBlobExcludesConcurrentStoreOfIdenticalContent exercises
// the exact TOCTOU window between evictSourceBlob's row-deletion
// transaction commit and its content file unlink: a concurrent
// StoreSource for a different source path whose content hashes to the
// same value (real SHA-256 dedup, not a contrived case) must not be
// able to insert a fresh row referencing the file while eviction is
// mid-unlink, and must not lose its own store once eviction has fully
// released the content hash.
func TestEvictSourceBlobExcludesConcurrentStoreOfIdenticalContent(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
ctx := context.Background()
content := bytes.Repeat([]byte{0x55}, 400)
hash := storeEvictionTestSource(t, cache,
"race.example.com", "/first.jpg", content)
proceed := make(chan struct{})
storeAttempted := make(chan struct{})
cache.evictSourceBlobTestHook = func(gotHash ContentHash) {
if gotHash != hash {
t.Errorf("test hook invoked for hash %s, want %s", gotHash, hash)
}
close(storeAttempted)
<-proceed
}
evictDone := make(chan error, 1)
go func() {
evictDone <- cache.evictSourceBlob(ctx, hash)
}()
// Wait until eviction has committed its delete transaction and is
// paused (inside the test hook) immediately before unlinking the
// content file: exactly the window the review flagged.
<-storeAttempted
storeDone := storeIdenticalContentConcurrently(ctx, cache, content)
// The concurrent store must not be able to complete while eviction
// still holds the content hash (i.e. before the file is unlinked):
// if it could, it would insert a row referencing a file about to be
// removed out from under it.
select {
case err := <-storeDone:
t.Fatalf("StoreSource for identical content completed (err=%v) while eviction "+
"still held the content hash open between commit and unlink; the store and "+
"the evict of identical content are not mutually exclusive", err)
case <-time.After(200 * time.Millisecond):
// Expected: the store is blocked behind eviction's exclusion.
}
close(proceed)
err := <-evictDone
if err != nil {
t.Fatalf("evictSourceBlob failed: %v", err)
}
err = <-storeDone
if err != nil {
t.Fatalf("StoreSource failed: %v", err)
}
assertNoDanglingReferences(t, cache)
dupHash, _, err := cache.LookupSource(ctx, &ImageRequest{
SourceHost: "race.example.com",
SourcePath: "/dup.jpg",
})
if err != nil {
t.Fatalf("LookupSource failed: %v", err)
}
if dupHash == "" {
t.Fatal("re-stored blob was lost: the store legitimately ran after eviction " +
"released the content hash and must have recreated the file and row")
}
if !cache.srcContent.Exists(dupHash) {
t.Errorf("source_content/source_metadata references %s but its file is missing",
dupHash)
}
}
// storeIdenticalContentConcurrently starts a StoreSource for a second
// source path whose body hashes to the same content hash, returning the
// channel its error is delivered on.
func storeIdenticalContentConcurrently(
ctx context.Context, cache *Cache, content []byte,
) chan error {
storeDone := make(chan error, 1)
go func() {
req := &ImageRequest{
SourceHost: "race.example.com",
SourcePath: "/dup.jpg",
Format: FormatJPEG,
Quality: 85,
FitMode: FitCover,
}
result := &httpfetcher.FetchResult{
StatusCode: 200,
ContentType: testContentTypeJPEG,
ContentLength: int64(len(content)),
Headers: map[string][]string{
testHeaderContentType: {testContentTypeJPEG},
},
}
_, err := cache.StoreSource(ctx, req, bytes.NewReader(content), result)
storeDone <- err
}()
return storeDone
}

1
internal/imgcache/imgcache.go
View File

@@ -90,7 +90,6 @@ func (r *ImageRequest) SourceURL() string {
if r.AllowHTTP {
scheme = "http"
}
url := scheme + "://" + r.SourceHost + r.SourcePath
if r.SourceQuery != "" {
url += "?" + r.SourceQuery

13
internal/imgcache/negative_cache_internal_test.go → internal/imgcache/negative_cache_test.go
View File

@@ -8,8 +8,6 @@ import (
)
func TestNegativeCache_StoreAndCheck(t *testing.T) {
t.Parallel()
db := setupTestDB(t)
dir := t.TempDir()
@@ -24,7 +22,7 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostExample,
SourceHost: "example.com",
SourcePath: "/missing.jpg",
}
@@ -33,7 +31,6 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if hit {
t.Error("expected no negative cache hit initially")
}
@@ -49,15 +46,12 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if !hit {
t.Error("expected negative cache hit after storing")
}
}
func TestNegativeCache_Expired(t *testing.T) {
t.Parallel()
db := setupTestDB(t)
dir := t.TempDir()
@@ -72,7 +66,7 @@ func TestNegativeCache_Expired(t *testing.T) {
ctx := context.Background()
req := &ImageRequest{
SourceHost: testHostExample,
SourceHost: "example.com",
SourcePath: "/expired.jpg",
}
@@ -90,15 +84,12 @@ func TestNegativeCache_Expired(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if hit {
t.Error("expected expired negative cache entry to be a miss")
}
}
func TestService_Get_ReturnsErrorForNegativeCachedURL(t *testing.T) {
t.Parallel()
// This test verifies that Service.Get() checks the negative cache
// We can't easily test the full pipeline without vips, but we can
// verify the error type

152
internal/imgcache/service.go
View File

@@ -18,8 +18,7 @@ import (
"sneak.berlin/go/pixa/internal/signature"
)
// Service implements the ImageCache interface, orchestrating cache,
// fetcher, and processor.
// Service implements the ImageCache interface, orchestrating cache, fetcher, and processor.
type Service struct {
cache *Cache
fetcher httpfetcher.Fetcher
@@ -47,21 +46,14 @@ type ServiceConfig struct {
Logger *slog.Logger
}
// Static errors for service construction and unimplemented operations.
var (
errCacheRequired = errors.New("cache is required")
errSigningKeyRequired = errors.New("signing key is required")
errPurgeNotImplemented = errors.New("purge not implemented")
)
// NewService creates a new image service.
func NewService(cfg *ServiceConfig) (*Service, error) {
if cfg.Cache == nil {
return nil, errCacheRequired
return nil, errors.New("cache is required")
}
if cfg.SigningKey == "" {
return nil, errSigningKeyRequired
return nil, errors.New("signing key is required")
}
// Resolve fetcher config for defaults
@@ -91,14 +83,11 @@ func NewService(cfg *ServiceConfig) (*Service, error) {
}
maxResponseSize := fetcherCfg.MaxResponseSize
processor := imageprocessor.New(
imageprocessor.Params{MaxInputBytes: maxResponseSize},
)
return &Service{
cache: cfg.Cache,
fetcher: fetcher,
processor: processor,
processor: imageprocessor.New(imageprocessor.Params{MaxInputBytes: maxResponseSize}),
signer: signer,
allowlist: allowlist.New(cfg.Allowlist),
log: log,
@@ -120,7 +109,6 @@ func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, e
if err != nil {
s.log.Warn("negative cache check failed", "error", err)
}
if negHit {
s.log.Debug("negative cache hit",
"host", req.SourceHost,
@@ -157,7 +145,6 @@ func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, e
// Cache miss - check if we have source content cached
cacheKey := CacheKey(req)
s.cache.IncrementStats(ctx, false, 0)
response, err := s.processFromSourceOrFetch(ctx, req, cacheKey)
@@ -170,57 +157,6 @@ func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, e
return response, nil
}
// Warm pre-fetches and caches an image without returning it.
func (s *Service) Warm(ctx context.Context, req *ImageRequest) error {
_, err := s.Get(ctx, req)
return err
}
// Purge removes a cached image. Purging is not implemented yet.
func (s *Service) Purge(_ context.Context, _ *ImageRequest) error {
return errPurgeNotImplemented
}
// Stats returns cache statistics.
func (s *Service) Stats(ctx context.Context) (*CacheStats, error) {
return s.cache.Stats(ctx)
}
// ValidateRequest validates the request signature if required.
func (s *Service) ValidateRequest(req *ImageRequest) error {
// Check if host is allowed (no signature required)
sourceURL := req.SourceURL()
parsedURL, err := url.Parse(sourceURL)
if err != nil {
return fmt.Errorf("invalid source URL: %w", err)
}
if s.allowlist.IsAllowed(parsedURL) {
return nil
}
// Signature required for non-allowed hosts
return s.signer.Verify(signatureRequest(req))
}
// GenerateSignedURL generates a signed URL for the given request.
func (s *Service) GenerateSignedURL(
baseURL string,
req *ImageRequest,
ttl time.Duration,
) (string, error) {
sigReq := signatureRequest(req)
path, sig, exp := s.signer.GenerateSignedURL(sigReq, ttl)
// Propagate the generated signature and expiration back onto the request.
req.Expires = sigReq.Expires
req.Signature = sigReq.Signature
return fmt.Sprintf("%s%s?sig=%s&exp=%d", baseURL, path, sig, exp), nil
}
// loadCachedSource attempts to load source content from cache, returning nil
// if the cached data is unavailable or exceeds maxResponseSize.
func (s *Service) loadCachedSource(contentHash ContentHash) []byte {
@@ -255,8 +191,7 @@ func (s *Service) loadCachedSource(contentHash ContentHash) []byte {
return data
}
// processFromSourceOrFetch processes an image, using cached source content
// if available.
// processFromSourceOrFetch processes an image, using cached source content if available.
func (s *Service) processFromSourceOrFetch(
ctx context.Context,
req *ImageRequest,
@@ -268,10 +203,8 @@ func (s *Service) processFromSourceOrFetch(
s.log.Warn("source lookup failed", "error", err)
}
var (
sourceData []byte
fetchBytes int64
)
var sourceData []byte
var fetchBytes int64
if contentHash != "" {
s.log.Debug("using cached source", "hash", contentHash)
@@ -315,7 +248,6 @@ func (s *Service) fetchAndProcess(
return nil, fmt.Errorf("upstream fetch failed: %w", err)
}
defer func() { _ = fetchResult.Content.Close() }()
// Read and validate the source content
@@ -326,7 +258,6 @@ func (s *Service) fetchAndProcess(
// Calculate download bitrate
fetchBytes := int64(len(sourceData))
var downloadRate string
if fetchResult.FetchDurationMs > 0 {
@@ -349,8 +280,7 @@ func (s *Service) fetchAndProcess(
)
// Validate magic bytes match content type
err = magic.ValidateMagicBytes(sourceData, fetchResult.ContentType)
if err != nil {
if err := magic.ValidateMagicBytes(sourceData, fetchResult.ContentType); err != nil {
return nil, fmt.Errorf("content validation failed: %w", err)
}
@@ -402,8 +332,7 @@ func (s *Service) processAndStore(
var sizePercent float64
if fetchBytes > 0 {
//nolint:mnd // percentage calculation
sizePercent = float64(outputSize) / float64(fetchBytes) * 100.0
sizePercent = float64(outputSize) / float64(fetchBytes) * 100.0 //nolint:mnd // percentage calculation
}
s.log.Info("image converted",
@@ -413,10 +342,8 @@ func (s *Service) processAndStore(
"dst_format", req.Format,
"src_bytes", fetchBytes,
"dst_bytes", outputSize,
"src_dimensions", fmt.Sprintf("%dx%d",
processResult.InputWidth, processResult.InputHeight),
"dst_dimensions", fmt.Sprintf("%dx%d",
processResult.Width, processResult.Height),
"src_dimensions", fmt.Sprintf("%dx%d", processResult.InputWidth, processResult.InputHeight),
"dst_dimensions", fmt.Sprintf("%dx%d", processResult.Width, processResult.Height),
"size_ratio", fmt.Sprintf("%.1f%%", sizePercent),
"convert_ms", processDuration.Milliseconds(),
"quality", req.Quality,
@@ -424,10 +351,7 @@ func (s *Service) processAndStore(
)
// Store variant to cache
err = s.cache.StoreVariant(
ctx, cacheKey, bytes.NewReader(processedData), processResult.ContentType,
)
if err != nil {
if err := s.cache.StoreVariant(cacheKey, bytes.NewReader(processedData), processResult.ContentType); err != nil {
s.log.Warn("failed to store variant", "error", err)
// Continue even if caching fails
}
@@ -441,6 +365,58 @@ func (s *Service) processAndStore(
}, nil
}
// Warm pre-fetches and caches an image without returning it.
func (s *Service) Warm(ctx context.Context, req *ImageRequest) error {
_, err := s.Get(ctx, req)
return err
}
// Purge removes a cached image.
func (s *Service) Purge(_ context.Context, _ *ImageRequest) error {
// TODO: Implement purge
return errors.New("purge not implemented")
}
// Stats returns cache statistics.
func (s *Service) Stats(ctx context.Context) (*CacheStats, error) {
return s.cache.Stats(ctx)
}
// ValidateRequest validates the request signature if required.
func (s *Service) ValidateRequest(req *ImageRequest) error {
// Check if host is allowed (no signature required)
sourceURL := req.SourceURL()
parsedURL, err := url.Parse(sourceURL)
if err != nil {
return fmt.Errorf("invalid source URL: %w", err)
}
if s.allowlist.IsAllowed(parsedURL) {
return nil
}
// Signature required for non-allowed hosts
return s.signer.Verify(signatureRequest(req))
}
// GenerateSignedURL generates a signed URL for the given request.
func (s *Service) GenerateSignedURL(
baseURL string,
req *ImageRequest,
ttl time.Duration,
) (string, error) {
sigReq := signatureRequest(req)
path, sig, exp := s.signer.GenerateSignedURL(sigReq, ttl)
// Propagate the generated signature and expiration back onto the request.
req.Expires = sigReq.Expires
req.Signature = sigReq.Signature
return fmt.Sprintf("%s%s?sig=%s&exp=%d", baseURL, path, sig, exp), nil
}
// signatureRequest projects an ImageRequest onto the standalone
// signature.Request type used by the signature package. This keeps the
// import edge one-way: imgcache depends on signature, never the reverse.

142
internal/imgcache/service_internal_test.go → internal/imgcache/service_test.go
View File

@@ -10,22 +10,13 @@ import (
"sneak.berlin/go/pixa/internal/signature"
)
// Test data literals used repeatedly in this file (goconst).
const (
testPathPhoto = "/images/photo.jpg"
testPathUpload = "/uploads/image.jpg"
testSigningKey = "test-signing-key-12345"
)
func TestService_Get_AllowlistedHost(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
SourcePath: "/images/photo.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -36,8 +27,7 @@ func TestService_Get_AllowlistedHost(t *testing.T) {
if err != nil {
t.Fatalf("Get() error = %v", err)
}
defer func() { _ = resp.Content.Close() }()
defer resp.Content.Close()
// Verify we got content
data, err := io.ReadAll(resp.Content)
@@ -49,19 +39,17 @@ func TestService_Get_AllowlistedHost(t *testing.T) {
t.Error("expected non-empty response")
}
if resp.ContentType != testContentTypeJPEG {
t.Errorf("ContentType = %q, want %q", resp.ContentType, testContentTypeJPEG)
if resp.ContentType != "image/jpeg" {
t.Errorf("ContentType = %q, want %q", resp.ContentType, "image/jpeg")
}
}
func TestService_Get_NonAllowlistedHost_NoSignature(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t, WithSigningKey("test-key"))
req := &ImageRequest{
SourceHost: fixtures.OtherHost,
SourcePath: testPathUpload,
SourcePath: "/uploads/image.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -76,15 +64,13 @@ func TestService_Get_NonAllowlistedHost_NoSignature(t *testing.T) {
}
func TestService_Get_NonAllowlistedHost_ValidSignature(t *testing.T) {
t.Parallel()
signingKey := testSigningKey
signingKey := "test-signing-key-12345"
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.OtherHost,
SourcePath: testPathUpload,
SourcePath: "/uploads/image.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -107,8 +93,7 @@ func TestService_Get_NonAllowlistedHost_ValidSignature(t *testing.T) {
if err != nil {
t.Fatalf("Get() error = %v", err)
}
defer func() { _ = resp.Content.Close() }()
defer resp.Content.Close()
data, err := io.ReadAll(resp.Content)
if err != nil {
@@ -121,14 +106,12 @@ func TestService_Get_NonAllowlistedHost_ValidSignature(t *testing.T) {
}
func TestService_Get_NonAllowlistedHost_ExpiredSignature(t *testing.T) {
t.Parallel()
signingKey := testSigningKey
signingKey := "test-signing-key-12345"
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
req := &ImageRequest{
SourceHost: fixtures.OtherHost,
SourcePath: testPathUpload,
SourcePath: "/uploads/image.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -148,14 +131,12 @@ func TestService_Get_NonAllowlistedHost_ExpiredSignature(t *testing.T) {
}
func TestService_Get_NonAllowlistedHost_InvalidSignature(t *testing.T) {
t.Parallel()
signingKey := testSigningKey
signingKey := "test-signing-key-12345"
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
req := &ImageRequest{
SourceHost: fixtures.OtherHost,
SourcePath: testPathUpload,
SourcePath: "/uploads/image.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -178,8 +159,6 @@ func TestService_Get_NonAllowlistedHost_InvalidSignature(t *testing.T) {
// signature for one host must not verify for a different host, even
// if they share a domain suffix.
func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
t.Parallel()
signingKey := "test-signing-key-must-be-32-chars"
svc, _ := SetupTestService(t,
WithSigningKey(signingKey),
@@ -190,8 +169,8 @@ func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
// Sign a request for "cdn.example.com"
signedReq := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -202,8 +181,6 @@ func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
// The original request should pass validation
t.Run("exact host passes", func(t *testing.T) {
t.Parallel()
err := svc.ValidateRequest(signedReq)
if err != nil {
t.Errorf("ValidateRequest() exact host failed: %v", err)
@@ -215,7 +192,7 @@ func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
name string
host string
}{
{"parent domain", testHostExample},
{"parent domain", "example.com"},
{"sibling subdomain", "images.example.com"},
{"deeper subdomain", "a.cdn.example.com"},
{"evil suffix domain", "cdn.example.com.evil.com"},
@@ -224,8 +201,6 @@ func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name+" rejected", func(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: tt.host,
SourcePath: signedReq.SourcePath,
@@ -240,8 +215,7 @@ func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
err := svc.ValidateRequest(req)
if err == nil {
t.Errorf(
"ValidateRequest() should reject signature for host %q (signed for %q)",
t.Errorf("ValidateRequest() should reject signature for host %q (signed for %q)",
tt.host, signedReq.SourceHost)
}
})
@@ -249,8 +223,6 @@ func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
}
func TestService_Get_InvalidFile(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
ctx := context.Background()
@@ -271,8 +243,6 @@ func TestService_Get_InvalidFile(t *testing.T) {
}
func TestService_Get_NotFound(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
ctx := context.Background()
@@ -292,8 +262,6 @@ func TestService_Get_NotFound(t *testing.T) {
}
func TestService_Get_FormatConversion(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
ctx := context.Background()
@@ -305,7 +273,7 @@ func TestService_Get_FormatConversion(t *testing.T) {
}{
{
name: "JPEG to PNG",
sourcePath: testPathPhoto,
sourcePath: "/images/photo.jpg",
outFormat: FormatPNG,
wantMIME: "image/png",
},
@@ -313,7 +281,7 @@ func TestService_Get_FormatConversion(t *testing.T) {
name: "PNG to JPEG",
sourcePath: "/images/logo.png",
outFormat: FormatJPEG,
wantMIME: testContentTypeJPEG,
wantMIME: "image/jpeg",
},
{
name: "GIF to PNG",
@@ -325,8 +293,6 @@ func TestService_Get_FormatConversion(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: tt.sourcePath,
@@ -340,8 +306,7 @@ func TestService_Get_FormatConversion(t *testing.T) {
if err != nil {
t.Fatalf("Get() error = %v", err)
}
defer func() { _ = resp.Content.Close() }()
defer resp.Content.Close()
if resp.ContentType != tt.wantMIME {
t.Errorf("ContentType = %q, want %q", resp.ContentType, tt.wantMIME)
@@ -376,14 +341,12 @@ func TestService_Get_FormatConversion(t *testing.T) {
}
func TestService_Get_Caching(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
SourcePath: "/images/photo.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -404,8 +367,7 @@ func TestService_Get_Caching(t *testing.T) {
if err != nil {
t.Fatalf("failed to read first response: %v", err)
}
_ = resp1.Content.Close()
resp1.Content.Close()
// Second request - should be a cache hit
resp2, err := svc.Get(ctx, req)
@@ -421,8 +383,7 @@ func TestService_Get_Caching(t *testing.T) {
if err != nil {
t.Fatalf("failed to read second response: %v", err)
}
_ = resp2.Content.Close()
resp2.Content.Close()
// Content should be identical
if len(data1) != len(data2) {
@@ -431,8 +392,6 @@ func TestService_Get_Caching(t *testing.T) {
}
func TestService_Get_DifferentSizes(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
ctx := context.Background()
@@ -443,12 +402,12 @@ func TestService_Get_DifferentSizes(t *testing.T) {
{Width: 75, Height: 75},
}
responses := make([][]byte, 0, len(sizes))
var responses [][]byte
for _, size := range sizes {
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
SourcePath: "/images/photo.jpg",
Size: size,
Format: FormatJPEG,
Quality: 85,
@@ -464,31 +423,27 @@ func TestService_Get_DifferentSizes(t *testing.T) {
if err != nil {
t.Fatalf("failed to read response: %v", err)
}
_ = resp.Content.Close()
resp.Content.Close()
responses = append(responses, data)
}
// All responses should be different sizes (different cache entries)
for i := range len(responses) - 1 {
for i := 0; i < len(responses)-1; i++ {
if len(responses[i]) == len(responses[i+1]) {
// Not necessarily an error, but worth noting
t.Logf("responses %d and %d have same size: %d bytes",
i, i+1, len(responses[i]))
t.Logf("responses %d and %d have same size: %d bytes", i, i+1, len(responses[i]))
}
}
}
func TestService_ValidateRequest_NoSigningKey(t *testing.T) {
t.Parallel()
// Service with no signing key - all non-allowlisted requests should fail
svc, fixtures := SetupTestService(t, WithNoAllowlist())
req := &ImageRequest{
SourceHost: fixtures.OtherHost,
SourcePath: testPathUpload,
SourcePath: "/uploads/image.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -497,15 +452,11 @@ func TestService_ValidateRequest_NoSigningKey(t *testing.T) {
err := svc.ValidateRequest(req)
if err == nil {
t.Error(
"ValidateRequest() expected error when no signing key and host not allowlisted",
)
t.Error("ValidateRequest() expected error when no signing key and host not allowlisted")
}
}
func TestService_Get_ContextCancellation(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
ctx, cancel := context.WithCancel(context.Background())
@@ -513,7 +464,7 @@ func TestService_Get_ContextCancellation(t *testing.T) {
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
SourcePath: "/images/photo.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -527,14 +478,12 @@ func TestService_Get_ContextCancellation(t *testing.T) {
}
func TestService_Get_ReturnsETag(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
SourcePath: "/images/photo.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -545,8 +494,7 @@ func TestService_Get_ReturnsETag(t *testing.T) {
if err != nil {
t.Fatalf("Get() error = %v", err)
}
defer func() { _ = resp.Content.Close() }()
defer resp.Content.Close()
// ETag should be set
if resp.ETag == "" {
@@ -560,14 +508,12 @@ func TestService_Get_ReturnsETag(t *testing.T) {
}
func TestService_Get_ETagConsistency(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
ctx := context.Background()
req := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
SourcePath: "/images/photo.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -579,20 +525,16 @@ func TestService_Get_ETagConsistency(t *testing.T) {
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
etag1 := resp1.ETag
_ = resp1.Content.Close()
resp1.Content.Close()
// Second request (from cache)
resp2, err := svc.Get(ctx, req)
if err != nil {
t.Fatalf("Get() second request error = %v", err)
}
etag2 := resp2.ETag
_ = resp2.Content.Close()
resp2.Content.Close()
// ETags should be identical for the same content
if etag1 != etag2 {
@@ -601,15 +543,13 @@ func TestService_Get_ETagConsistency(t *testing.T) {
}
func TestService_Get_DifferentETagsForDifferentContent(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
ctx := context.Background()
// Request same image at different sizes - should get different ETags
req1 := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
SourcePath: "/images/photo.jpg",
Size: Size{Width: 25, Height: 25},
Format: FormatJPEG,
Quality: 85,
@@ -618,7 +558,7 @@ func TestService_Get_DifferentETagsForDifferentContent(t *testing.T) {
req2 := &ImageRequest{
SourceHost: fixtures.GoodHost,
SourcePath: testPathPhoto,
SourcePath: "/images/photo.jpg",
Size: Size{Width: 50, Height: 50},
Format: FormatJPEG,
Quality: 85,
@@ -629,19 +569,15 @@ func TestService_Get_DifferentETagsForDifferentContent(t *testing.T) {
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
etag1 := resp1.ETag
_ = resp1.Content.Close()
resp1.Content.Close()
resp2, err := svc.Get(ctx, req2)
if err != nil {
t.Fatalf("Get() second request error = %v", err)
}
etag2 := resp2.ETag
_ = resp2.Content.Close()
resp2.Content.Close()
// ETags should be different for different content
if etag1 == etag2 {

16
internal/imgcache/sourceurl_internal_test.go → internal/imgcache/sourceurl_test.go
View File

@@ -3,16 +3,13 @@ package imgcache
import "testing"
func TestImageRequest_SourceURL_DefaultHTTPS(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "v=2",
}
got := req.SourceURL()
want := "https://cdn.example.com/photos/cat.jpg?v=2"
if got != want {
t.Errorf("SourceURL() = %q, want %q", got, want)
@@ -20,16 +17,13 @@ func TestImageRequest_SourceURL_DefaultHTTPS(t *testing.T) {
}
func TestImageRequest_SourceURL_AllowHTTP(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: "localhost:8080",
SourcePath: testPathCat,
SourcePath: "/photos/cat.jpg",
AllowHTTP: true,
}
got := req.SourceURL()
want := "http://localhost:8080/photos/cat.jpg"
if got != want {
t.Errorf("SourceURL() = %q, want %q", got, want)
@@ -37,10 +31,8 @@ func TestImageRequest_SourceURL_AllowHTTP(t *testing.T) {
}
func TestImageRequest_SourceURL_AllowHTTPFalse(t *testing.T) {
t.Parallel()
req := &ImageRequest{
SourceHost: testHostCDN,
SourceHost: "cdn.example.com",
SourcePath: "/img.jpg",
AllowHTTP: false,
}

21
internal/imgcache/stats_internal_test.go → internal/imgcache/stats_test.go
View File

@@ -12,25 +12,18 @@ import (
func setupStatsTestDB(t *testing.T) *sql.DB {
t.Helper()
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatal(err)
}
err = database.ApplyMigrations(context.Background(), db, nil)
if err != nil {
if err := database.ApplyMigrations(context.Background(), db, nil); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
t.Cleanup(func() { db.Close() })
return db
}
func TestStats_HitRateIsRatio(t *testing.T) {
t.Parallel()
db := setupStatsTestDB(t)
dir := t.TempDir()
@@ -47,9 +40,7 @@ func TestStats_HitRateIsRatio(t *testing.T) {
// Set some hit/miss counts and a transform_count
_, err = db.ExecContext(ctx, `
UPDATE cache_stats
SET hit_count = 75, miss_count = 25, transform_count = 9999
WHERE id = 1
UPDATE cache_stats SET hit_count = 75, miss_count = 25, transform_count = 9999 WHERE id = 1
`)
if err != nil {
t.Fatal(err)
@@ -63,7 +54,6 @@ func TestStats_HitRateIsRatio(t *testing.T) {
if stats.HitCount != 75 {
t.Errorf("HitCount = %d, want 75", stats.HitCount)
}
if stats.MissCount != 25 {
t.Errorf("MissCount = %d, want 25", stats.MissCount)
}
@@ -71,14 +61,11 @@ func TestStats_HitRateIsRatio(t *testing.T) {
// HitRate should be 0.75, NOT 9999 (transform_count)
expectedRate := 0.75
if math.Abs(stats.HitRate-expectedRate) > 0.001 {
t.Errorf("HitRate = %f, want %f (was it scanning transform_count?)",
stats.HitRate, expectedRate)
t.Errorf("HitRate = %f, want %f (was it scanning transform_count?)", stats.HitRate, expectedRate)
}
}
func TestStats_ZeroCounts(t *testing.T) {
t.Parallel()
db := setupStatsTestDB(t)
dir := t.TempDir()

240
internal/imgcache/storage.go
View File

@@ -44,8 +44,7 @@ type ContentStorage struct {
// NewContentStorage creates a new content storage at the given base directory.
func NewContentStorage(baseDir string) (*ContentStorage, error) {
err := os.MkdirAll(baseDir, StorageDirPerm)
if err != nil {
if err := os.MkdirAll(baseDir, StorageDirPerm); err != nil {
return nil, fmt.Errorf("failed to create storage directory: %w", err)
}
@@ -54,7 +53,7 @@ func NewContentStorage(baseDir string) (*ContentStorage, error) {
// Store writes content to storage and returns its SHA256 hash.
// The content is read fully into memory to compute the hash before writing.
func (s *ContentStorage) Store(r io.Reader) (ContentHash, int64, error) {
func (s *ContentStorage) Store(r io.Reader) (hash ContentHash, size int64, err error) {
// Read all content to compute hash
data, err := io.ReadAll(r)
if err != nil {
@@ -63,34 +62,55 @@ func (s *ContentStorage) Store(r io.Reader) (ContentHash, int64, error) {
// Compute hash
h := sha256.Sum256(data)
hash := ContentHash(hex.EncodeToString(h[:]))
size := int64(len(data))
hash = ContentHash(hex.EncodeToString(h[:]))
size = int64(len(data))
err = s.writeIfAbsent(hash, data)
// Build path: <basedir>/<ab>/<cd>/<hash>
path := s.hashToPath(hash)
// Check if already exists
if _, err := os.Stat(path); err == nil {
return hash, size, nil
}
// Create directory structure
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, StorageDirPerm); err != nil {
return "", 0, fmt.Errorf("failed to create directory: %w", err)
}
// Write to temp file first, then rename for atomicity
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
if err != nil {
return "", 0, err
return "", 0, fmt.Errorf("failed to create temp file: %w", err)
}
tmpPath := tmpFile.Name()
defer func() {
if err != nil {
_ = os.Remove(tmpPath)
}
}()
if _, err := tmpFile.Write(data); err != nil {
_ = tmpFile.Close()
return "", 0, fmt.Errorf("failed to write content: %w", err)
}
if err := tmpFile.Close(); err != nil {
return "", 0, fmt.Errorf("failed to close temp file: %w", err)
}
// Atomic rename
//nolint:gosec // G703: paths from internal SHA256 hashes
if err := os.Rename(filepath.Clean(tmpPath), filepath.Clean(path)); err != nil {
return "", 0, fmt.Errorf("failed to rename temp file: %w", err)
}
return hash, size, nil
}
// StoreHashed writes pre-hashed content to storage at the path derived
// from hash, without recomputing it. Callers that already know the
// hash before writing (e.g. because they must hold a hash-keyed lock
// across the whole store operation) use this instead of Store. Like
// Store, it is idempotent: content already on disk at that path is
// left untouched.
func (s *ContentStorage) StoreHashed(
hash ContentHash, data []byte,
) (int64, error) {
err := s.writeIfAbsent(hash, data)
if err != nil {
return 0, err
}
return int64(len(data)), nil
}
// Load returns a reader for the content with the given hash.
func (s *ContentStorage) Load(hash ContentHash) (io.ReadCloser, error) {
path := s.hashToPath(hash)
@@ -150,67 +170,6 @@ func (s *ContentStorage) Exists(hash ContentHash) bool {
return err == nil
}
// writeIfAbsent writes data to the path derived from hash, unless
// content already exists there, via a temp-file-plus-rename so
// concurrent readers never observe a partial file.
func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error {
// Build path: <basedir>/<ab>/<cd>/<hash>
path := s.hashToPath(hash)
// Check if already exists
_, statErr := os.Stat(path)
if statErr == nil {
return nil
}
// Create directory structure
dir := filepath.Dir(path)
err := os.MkdirAll(dir, StorageDirPerm)
if err != nil {
return fmt.Errorf("failed to create directory: %w", err)
}
// Write to temp file first, then rename for atomicity
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
if err != nil {
return fmt.Errorf("failed to create temp file: %w", err)
}
tmpPath := tmpFile.Name()
// Each failure path below unlinks the temp file explicitly. This
// replaces a deferred cleanup that read a named result, which the
// canonical config does not permit; the set of paths that remove
// tmpPath, and the order relative to Close, is unchanged. This
// mirrors how MetadataStorage.Store and VariantStorage.Store below
// already express the same cleanup.
_, err = tmpFile.Write(data)
if err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to write content: %w", err)
}
err = tmpFile.Close()
if err != nil {
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to close temp file: %w", err)
}
// Atomic rename
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
if err != nil {
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to rename temp file: %w", err)
}
return nil
}
// hashToPath converts a hash to a file path: <basedir>/<ab>/<cd>/<hash>
func (s *ContentStorage) hashToPath(hash ContentHash) string {
h := string(hash)
@@ -229,8 +188,7 @@ type MetadataStorage struct {
// NewMetadataStorage creates a new metadata storage at the given base directory.
func NewMetadataStorage(baseDir string) (*MetadataStorage, error) {
err := os.MkdirAll(baseDir, StorageDirPerm)
if err != nil {
if err := os.MkdirAll(baseDir, StorageDirPerm); err != nil {
return nil, fmt.Errorf("failed to create metadata directory: %w", err)
}
@@ -238,8 +196,6 @@ func NewMetadataStorage(baseDir string) (*MetadataStorage, error) {
}
// SourceMetadata represents cached metadata about a source URL.
//
//nolint:tagliatelle // stored metadata format uses snake_case
type SourceMetadata struct {
Host string `json:"host"`
Path string `json:"path"`
@@ -258,16 +214,12 @@ type SourceMetadata struct {
}
// Store writes metadata to storage.
func (s *MetadataStorage) Store(
host string, pathHash PathHash, meta *SourceMetadata,
) error {
func (s *MetadataStorage) Store(host string, pathHash PathHash, meta *SourceMetadata) error {
path := s.metaPath(host, pathHash)
// Create directory structure
dir := filepath.Dir(path)
err := os.MkdirAll(dir, StorageDirPerm)
if err != nil {
if err := os.MkdirAll(dir, StorageDirPerm); err != nil {
return fmt.Errorf("failed to create directory: %w", err)
}
@@ -282,29 +234,27 @@ func (s *MetadataStorage) Store(
if err != nil {
return fmt.Errorf("failed to create temp file: %w", err)
}
tmpPath := tmpFile.Name()
_, err = tmpFile.Write(data)
if err != nil {
defer func() {
if err != nil {
_ = os.Remove(tmpPath)
}
}()
if _, err := tmpFile.Write(data); err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to write metadata: %w", err)
}
err = tmpFile.Close()
if err != nil {
_ = os.Remove(tmpPath)
if err := tmpFile.Close(); err != nil {
return fmt.Errorf("failed to close temp file: %w", err)
}
// Atomic rename
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
if err != nil {
_ = os.Remove(tmpPath)
//nolint:gosec // G703: paths from internal SHA256 hashes
if err := os.Rename(filepath.Clean(tmpPath), filepath.Clean(path)); err != nil {
return fmt.Errorf("failed to rename temp file: %w", err)
}
@@ -312,9 +262,7 @@ func (s *MetadataStorage) Store(
}
// Load reads metadata from storage.
func (s *MetadataStorage) Load(
host string, pathHash PathHash,
) (*SourceMetadata, error) {
func (s *MetadataStorage) Load(host string, pathHash PathHash) (*SourceMetadata, error) {
path := s.metaPath(host, pathHash)
data, err := os.ReadFile(path) //nolint:gosec // path derived from host+hash
@@ -327,9 +275,7 @@ func (s *MetadataStorage) Load(
}
var meta SourceMetadata
err = json.Unmarshal(data, &meta)
if err != nil {
if err := json.Unmarshal(data, &meta); err != nil {
return nil, fmt.Errorf("failed to unmarshal metadata: %w", err)
}
@@ -395,8 +341,6 @@ type VariantStorage struct {
}
// VariantMeta contains metadata about a cached variant.
//
//nolint:tagliatelle // stored metadata format uses snake_case
type VariantMeta struct {
ContentType string `json:"content_type"`
Size int64 `json:"size"`
@@ -405,8 +349,7 @@ type VariantMeta struct {
// NewVariantStorage creates a new variant storage at the given base directory.
func NewVariantStorage(baseDir string) (*VariantStorage, error) {
err := os.MkdirAll(baseDir, StorageDirPerm)
if err != nil {
if err := os.MkdirAll(baseDir, StorageDirPerm); err != nil {
return nil, fmt.Errorf("failed to create variant storage directory: %w", err)
}
@@ -414,23 +357,19 @@ func NewVariantStorage(baseDir string) (*VariantStorage, error) {
}
// Store writes content and metadata to storage at the given key.
func (s *VariantStorage) Store(
key VariantKey, r io.Reader, contentType string,
) (int64, error) {
func (s *VariantStorage) Store(key VariantKey, r io.Reader, contentType string) (size int64, err error) {
data, err := io.ReadAll(r)
if err != nil {
return 0, fmt.Errorf("failed to read content: %w", err)
}
size := int64(len(data))
size = int64(len(data))
path := s.keyToPath(key)
metaPath := path + ".meta"
// Create directory structure
dir := filepath.Dir(path)
err = os.MkdirAll(dir, StorageDirPerm)
if err != nil {
if err := os.MkdirAll(dir, StorageDirPerm); err != nil {
return 0, fmt.Errorf("failed to create directory: %w", err)
}
@@ -439,29 +378,27 @@ func (s *VariantStorage) Store(
if err != nil {
return 0, fmt.Errorf("failed to create temp file: %w", err)
}
tmpPath := tmpFile.Name()
_, err = tmpFile.Write(data)
if err != nil {
defer func() {
if err != nil {
_ = os.Remove(tmpPath)
}
}()
if _, err := tmpFile.Write(data); err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return 0, fmt.Errorf("failed to write content: %w", err)
}
err = tmpFile.Close()
if err != nil {
_ = os.Remove(tmpPath)
if err := tmpFile.Close(); err != nil {
return 0, fmt.Errorf("failed to close temp file: %w", err)
}
// Atomic rename content
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
if err != nil {
_ = os.Remove(tmpPath)
//nolint:gosec // G703: paths from internal SHA256 hashes
if err := os.Rename(filepath.Clean(tmpPath), filepath.Clean(path)); err != nil {
return 0, fmt.Errorf("failed to rename temp file: %w", err)
}
@@ -477,8 +414,10 @@ func (s *VariantStorage) Store(
return 0, fmt.Errorf("failed to marshal metadata: %w", err)
}
// Metadata write failure is non-fatal; content is already stored.
_ = os.WriteFile(metaPath, metaData, StorageFilePerm)
if err := os.WriteFile(metaPath, metaData, StorageFilePerm); err != nil {
// Non-fatal, content is stored
_ = err
}
return size, nil
}
@@ -499,11 +438,8 @@ func (s *VariantStorage) Load(key VariantKey) (io.ReadCloser, error) {
return f, nil
}
// LoadWithMeta returns a reader, size, and content type for the content at
// the given key.
func (s *VariantStorage) LoadWithMeta(
key VariantKey,
) (io.ReadCloser, int64, string, error) {
// LoadWithMeta returns a reader, size, and content type for the content at the given key.
func (s *VariantStorage) LoadWithMeta(key VariantKey) (io.ReadCloser, int64, string, error) {
path := s.keyToPath(key)
metaPath := path + ".meta"
@@ -557,24 +493,6 @@ func (s *VariantStorage) Delete(key VariantKey) error {
return nil
}
// DeleteWithMeta removes the content at the given key together with
// its .meta sidecar file. A missing file is not an error.
func (s *VariantStorage) DeleteWithMeta(key VariantKey) error {
err := s.Delete(key)
if err != nil {
return err
}
metaPath := s.keyToPath(key) + ".meta"
err = os.Remove(metaPath)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("failed to delete variant metadata: %w", err)
}
return nil
}
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
func (s *VariantStorage) keyToPath(key VariantKey) string {
k := string(key)

113
internal/imgcache/storage_internal_test.go → internal/imgcache/storage_test.go
View File

@@ -2,7 +2,6 @@ package imgcache
import (
"bytes"
"errors"
"io"
"os"
"path/filepath"
@@ -10,17 +9,13 @@ import (
)
func TestContentStorage_StoreAndLoad(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
content := []byte("hello world")
hash, size, err := storage.Store(bytes.NewReader(content))
if err != nil {
t.Fatalf("Store() error = %v", err)
@@ -36,11 +31,8 @@ func TestContentStorage_StoreAndLoad(t *testing.T) {
// Verify file exists at expected path
hashStr := string(hash)
expectedPath := filepath.Join(tmpDir, hashStr[0:2], hashStr[2:4], hashStr)
_, err = os.Stat(expectedPath)
if err != nil {
if _, err := os.Stat(expectedPath); err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
@@ -49,8 +41,7 @@ func TestContentStorage_StoreAndLoad(t *testing.T) {
if err != nil {
t.Fatalf("Load() error = %v", err)
}
defer func() { _ = r.Close() }()
defer r.Close()
loaded, err := io.ReadAll(r)
if err != nil {
@@ -63,10 +54,7 @@ func TestContentStorage_StoreAndLoad(t *testing.T) {
}
func TestContentStorage_StoreIdempotent(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
@@ -90,33 +78,26 @@ func TestContentStorage_StoreIdempotent(t *testing.T) {
}
func TestContentStorage_LoadNotFound(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
_, err = storage.Load(ContentHash("nonexistent"))
if !errors.Is(err, ErrNotFound) {
if err != ErrNotFound {
t.Errorf("Load() error = %v, want ErrNotFound", err)
}
}
func TestContentStorage_Delete(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
content := []byte("to be deleted")
hash, _, err := storage.Store(bytes.NewReader(content))
if err != nil {
t.Fatalf("Store() error = %v", err)
@@ -126,8 +107,7 @@ func TestContentStorage_Delete(t *testing.T) {
t.Error("Exists() = false, want true")
}
err = storage.Delete(hash)
if err != nil {
if err := storage.Delete(hash); err != nil {
t.Fatalf("Delete() error = %v", err)
}
@@ -137,27 +117,20 @@ func TestContentStorage_Delete(t *testing.T) {
}
func TestContentStorage_DeleteNonexistent(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
// Should not error
err = storage.Delete(ContentHash("nonexistent"))
if err != nil {
if err := storage.Delete(ContentHash("nonexistent")); err != nil {
t.Errorf("Delete() error = %v, want nil", err)
}
}
func TestContentStorage_HashToPath(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewContentStorage(tmpDir)
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
@@ -165,59 +138,50 @@ func TestContentStorage_HashToPath(t *testing.T) {
// Test by storing and verifying the resulting path structure
content := []byte("test content for path verification")
hash, _, err := storage.Store(bytes.NewReader(content))
if err != nil {
t.Fatalf("Store() error = %v", err)
}
hashStr := string(hash)
expectedPath := filepath.Join(tmpDir, hashStr[0:2], hashStr[2:4], hashStr)
_, err = os.Stat(expectedPath)
if err != nil {
if _, err := os.Stat(expectedPath); err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
}
func TestMetadataStorage_StoreAndLoad(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewMetadataStorage(tmpDir)
if err != nil {
t.Fatalf("NewMetadataStorage() error = %v", err)
}
meta := &SourceMetadata{
Host: testHostCDN,
Path: testPathCat,
Host: "cdn.example.com",
Path: "/photos/cat.jpg",
ContentHash: "abc123",
StatusCode: 200,
ContentType: testContentTypeJPEG,
ContentType: "image/jpeg",
FetchedAt: 1704067200,
ETag: `"etag123"`,
}
pathHash := HashPath(testPathCat)
pathHash := HashPath("/photos/cat.jpg")
err = storage.Store(testHostCDN, pathHash, meta)
err = storage.Store("cdn.example.com", pathHash, meta)
if err != nil {
t.Fatalf("Store() error = %v", err)
}
// Verify file exists at expected path
expectedPath := filepath.Join(tmpDir, testHostCDN, string(pathHash)+".json")
_, err = os.Stat(expectedPath)
if err != nil {
expectedPath := filepath.Join(tmpDir, "cdn.example.com", string(pathHash)+".json")
if _, err := os.Stat(expectedPath); err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
// Load and verify
loaded, err := storage.Load(testHostCDN, pathHash)
loaded, err := storage.Load("cdn.example.com", pathHash)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
@@ -244,64 +208,55 @@ func TestMetadataStorage_StoreAndLoad(t *testing.T) {
}
func TestMetadataStorage_LoadNotFound(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewMetadataStorage(tmpDir)
if err != nil {
t.Fatalf("NewMetadataStorage() error = %v", err)
}
_, err = storage.Load(testHostExample, PathHash("nonexistent"))
if !errors.Is(err, ErrNotFound) {
_, err = storage.Load("example.com", PathHash("nonexistent"))
if err != ErrNotFound {
t.Errorf("Load() error = %v, want ErrNotFound", err)
}
}
func TestMetadataStorage_Delete(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewMetadataStorage(tmpDir)
if err != nil {
t.Fatalf("NewMetadataStorage() error = %v", err)
}
meta := &SourceMetadata{
Host: testHostExample,
Host: "example.com",
Path: "/test.jpg",
StatusCode: 200,
}
pathHash := HashPath("/test.jpg")
err = storage.Store(testHostExample, pathHash, meta)
err = storage.Store("example.com", pathHash, meta)
if err != nil {
t.Fatalf("Store() error = %v", err)
}
if !storage.Exists(testHostExample, pathHash) {
if !storage.Exists("example.com", pathHash) {
t.Error("Exists() = false, want true")
}
err = storage.Delete(testHostExample, pathHash)
if err != nil {
if err := storage.Delete("example.com", pathHash); err != nil {
t.Fatalf("Delete() error = %v", err)
}
if storage.Exists(testHostExample, pathHash) {
if storage.Exists("example.com", pathHash) {
t.Error("Exists() = true after delete, want false")
}
}
func TestHashPath(t *testing.T) {
t.Parallel()
// Same input should produce same hash
hash1 := HashPath(testPathCat)
hash2 := HashPath(testPathCat)
hash1 := HashPath("/photos/cat.jpg")
hash2 := HashPath("/photos/cat.jpg")
if hash1 != hash2 {
t.Errorf("HashPath() not deterministic: %s vs %s", hash1, hash2)
@@ -321,11 +276,9 @@ func TestHashPath(t *testing.T) {
}
func TestCacheKey(t *testing.T) {
t.Parallel()
req1 := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
@@ -334,8 +287,8 @@ func TestCacheKey(t *testing.T) {
}
req2 := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
@@ -358,8 +311,8 @@ func TestCacheKey(t *testing.T) {
// Different size should produce different key
req3 := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "",
Size: Size{Width: 400, Height: 300}, // Different size
Format: FormatWebP,
@@ -374,8 +327,8 @@ func TestCacheKey(t *testing.T) {
// Different format should produce different key
req4 := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "",
Size: Size{Width: 800, Height: 600},
Format: FormatPNG, // Different format
@@ -390,8 +343,8 @@ func TestCacheKey(t *testing.T) {
// Different quality should produce different key
req5 := &ImageRequest{
SourceHost: testHostCDN,
SourcePath: testPathCat,
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,

45
internal/imgcache/testutil_internal_test.go → internal/imgcache/testutil_test.go
View File

@@ -18,15 +18,6 @@ import (
"sneak.berlin/go/pixa/internal/httpfetcher"
)
// Shared test data literals, extracted as constants for goconst.
const (
testHostCDN = "cdn.example.com"
testHostExample = "example.com"
testPathCat = "/photos/cat.jpg"
testContentTypeJPEG = "image/jpeg"
testHeaderContentType = "Content-Type"
)
// TestFixtures contains paths to test files in the mock filesystem.
type TestFixtures struct {
// Valid image files
@@ -98,16 +89,14 @@ func generateTestJPEG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 85})
if err != nil {
if err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 85}); err != nil {
t.Fatalf("failed to encode test JPEG: %v", err)
}
@@ -119,16 +108,14 @@ func generateTestPNG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
err := png.Encode(&buf, img)
if err != nil {
if err := png.Encode(&buf, img); err != nil {
t.Fatalf("failed to encode test PNG: %v", err)
}
@@ -139,20 +126,15 @@ func generateTestPNG(t *testing.T, width, height int, c color.Color) []byte {
func generateTestGIF(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewPaletted(
image.Rect(0, 0, width, height),
[]color.Color{c, color.White},
)
for y := range height {
for x := range width {
img := image.NewPaletted(image.Rect(0, 0, width, height), []color.Color{c, color.White})
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
img.SetColorIndex(x, y, 0)
}
}
var buf bytes.Buffer
err := gif.Encode(&buf, img, nil)
if err != nil {
if err := gif.Encode(&buf, img, nil); err != nil {
t.Fatalf("failed to encode test GIF: %v", err)
}
@@ -160,9 +142,7 @@ func generateTestGIF(t *testing.T, width, height int, c color.Color) []byte {
}
// SetupTestService creates a Service with mock fetcher for testing.
func SetupTestService(
t *testing.T, opts ...TestServiceOption,
) (*Service, *TestFixtures) {
func SetupTestService(t *testing.T, opts ...TestServiceOption) (*Service, *TestFixtures) {
t.Helper()
mockFS, fixtures := NewTestFS(t)
@@ -215,8 +195,7 @@ func setupServiceTestDB(t *testing.T) *sql.DB {
}
// Use the real production schema via migrations
err = database.ApplyMigrations(context.Background(), db, nil)
if err != nil {
if err := database.ApplyMigrations(context.Background(), db, nil); err != nil {
t.Fatalf("failed to apply migrations: %v", err)
}

16
internal/imgcache/urlparser.go
View File

@@ -40,8 +40,7 @@ type ParsedURL struct {
Format ImageFormat
}
// ParseImagePath parses the path captured by chi's wildcard:
// <host>/<path>/<size>.<format>
// ParseImagePath parses the path captured by chi's wildcard: <host>/<path>/<size>.<format>
// This is the primary entry point when using chi routing.
// Examples:
// - cdn.example.com/photos/cat.jpg/800x600.webp
@@ -77,8 +76,7 @@ func ParseImageURL(urlPath string) (*ParsedURL, error) {
// parseImageComponents parses <host>/<path>/<size>.<format> structure.
func parseImageComponents(remainder string) (*ParsedURL, error) {
// Check for path traversal before any other processing
err := checkPathTraversal(remainder)
if err != nil {
if err := checkPathTraversal(remainder); err != nil {
return nil, err
}
@@ -104,7 +102,6 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
// Split host from path
// The first segment is the host, everything after is the path
firstSlash := strings.Index(hostAndPath, "/")
var host, path, query string
if firstSlash == -1 {
@@ -184,7 +181,8 @@ func checkPathTraversal(path string) error {
// Also check for ".." as a path segment in the original path
// This catches cases where the path hasn't been normalized
for seg := range strings.SplitSeq(path, "/") {
segments := strings.Split(path, "/")
for _, seg := range segments {
// URL decode the segment
decodedSeg, _ := url.PathUnescape(seg)
decodedSeg = strings.ReplaceAll(decodedSeg, "\\", "/")
@@ -204,10 +202,8 @@ func parseSizeFormat(s string) (Size, ImageFormat, error) {
return Size{}, "", ErrInvalidSize
}
var (
size Size
formatStr string
)
var size Size
var formatStr string
if matches[4] == "orig" {
// "orig.format" pattern

198
internal/imgcache/urlparser_internal_test.go → internal/imgcache/urlparser_test.go
View File

@@ -1,124 +1,93 @@
package imgcache
import (
"errors"
"testing"
)
// assertParsedURL compares all fields of a parsed URL against the
// expected value.
func assertParsedURL(t *testing.T, got, want *ParsedURL) {
t.Helper()
if got.Host != want.Host {
t.Errorf("Host = %q, want %q", got.Host, want.Host)
}
if got.Path != want.Path {
t.Errorf("Path = %q, want %q", got.Path, want.Path)
}
if got.Query != want.Query {
t.Errorf("Query = %q, want %q", got.Query, want.Query)
}
if got.Size != want.Size {
t.Errorf("Size = %v, want %v", got.Size, want.Size)
}
if got.Format != want.Format {
t.Errorf("Format = %q, want %q", got.Format, want.Format)
}
}
func TestParseImageURL(t *testing.T) {
t.Parallel()
tests := []struct {
name string
input string
want *ParsedURL
name string
input string
want *ParsedURL
wantErr error
}{
{
name: "basic path with size",
input: "/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat,
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
Host: "cdn.example.com",
Path: "/photos/cat.jpg",
Query: "",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
},
},
{
name: "original size with 0x0",
input: "/v1/image/cdn.example.com/photos/cat.jpg/0x0.jpeg",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat,
Size: Size{Width: 0, Height: 0}, Format: FormatJPEG,
Host: "cdn.example.com",
Path: "/photos/cat.jpg",
Query: "",
Size: Size{Width: 0, Height: 0},
Format: FormatJPEG,
},
},
{
name: "original size with orig keyword",
input: "/v1/image/cdn.example.com/photos/cat.jpg/orig.png",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat,
Size: Size{Width: 0, Height: 0}, Format: FormatPNG,
Host: "cdn.example.com",
Path: "/photos/cat.jpg",
Query: "",
Size: Size{Width: 0, Height: 0},
Format: FormatPNG,
},
},
{
name: "path with query string",
input: "/v1/image/cdn.example.com/photos/cat.jpg?arg1=val1&arg2=val2/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat, Query: "arg1=val1&arg2=val2",
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
Host: "cdn.example.com",
Path: "/photos/cat.jpg",
Query: "arg1=val1&arg2=val2",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
},
},
{
name: "deep nested path",
input: "/v1/image/cdn.example.com/a/b/c/d/image.jpg/1920x1080.avif",
want: &ParsedURL{
Host: testHostCDN, Path: "/a/b/c/d/image.jpg",
Size: Size{Width: 1920, Height: 1080}, Format: FormatAVIF,
Host: "cdn.example.com",
Path: "/a/b/c/d/image.jpg",
Query: "",
Size: Size{Width: 1920, Height: 1080},
Format: FormatAVIF,
},
},
{
name: "jpg alias for jpeg",
input: "/v1/image/example.com/img.png/100x100.jpg",
want: &ParsedURL{
Host: testHostExample, Path: "/img.png",
Size: Size{Width: 100, Height: 100}, Format: FormatJPEG,
Host: "example.com",
Path: "/img.png",
Query: "",
Size: Size{Width: 100, Height: 100},
Format: FormatJPEG,
},
},
{
name: "gif format",
input: "/v1/image/example.com/animated.gif/200x200.gif",
want: &ParsedURL{
Host: testHostExample, Path: "/animated.gif",
Size: Size{Width: 200, Height: 200}, Format: FormatGIF,
Host: "example.com",
Path: "/animated.gif",
Query: "",
Size: Size{Width: 200, Height: 200},
Format: FormatGIF,
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := ParseImageURL(tt.input)
if err != nil {
t.Fatalf("ParseImageURL() unexpected error = %v", err)
}
assertParsedURL(t, got, tt.want)
})
}
}
func TestParseImageURL_Errors(t *testing.T) {
t.Parallel()
tests := []struct {
name string
input string
wantErr error
}{
{
name: "missing prefix",
input: "/image/cdn.example.com/photo.jpg/800x600.webp",
@@ -153,23 +122,47 @@ func TestParseImageURL_Errors(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := ParseImageURL(tt.input)
_, err := ParseImageURL(tt.input)
if err == nil {
t.Fatalf("ParseImageURL() error = nil, wantErr %v", tt.wantErr)
if tt.wantErr != nil {
if err == nil {
t.Errorf("ParseImageURL() error = nil, wantErr %v", tt.wantErr)
return
}
if !errorIs(err, tt.wantErr) {
t.Errorf("ParseImageURL() error = %v, wantErr %v", err, tt.wantErr)
}
return
}
if !errorIs(err, tt.wantErr) {
t.Errorf("ParseImageURL() error = %v, wantErr %v", err, tt.wantErr)
if err != nil {
t.Errorf("ParseImageURL() unexpected error = %v", err)
return
}
if got.Host != tt.want.Host {
t.Errorf("Host = %q, want %q", got.Host, tt.want.Host)
}
if got.Path != tt.want.Path {
t.Errorf("Path = %q, want %q", got.Path, tt.want.Path)
}
if got.Query != tt.want.Query {
t.Errorf("Query = %q, want %q", got.Query, tt.want.Query)
}
if got.Size != tt.want.Size {
t.Errorf("Size = %v, want %v", got.Size, tt.want.Size)
}
if got.Format != tt.want.Format {
t.Errorf("Format = %q, want %q", got.Format, tt.want.Format)
}
})
}
}
func TestParseImagePath(t *testing.T) {
t.Parallel()
// ParseImagePath is for chi wildcard capture (no /v1/image/ prefix)
tests := []struct {
name string
@@ -181,8 +174,8 @@ func TestParseImagePath(t *testing.T) {
name: "chi wildcard capture",
input: "cdn.example.com/photos/cat.jpg/800x600.webp",
want: &ParsedURL{
Host: testHostCDN,
Path: testPathCat,
Host: "cdn.example.com",
Path: "/photos/cat.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
},
@@ -191,8 +184,8 @@ func TestParseImagePath(t *testing.T) {
name: "with leading slash from chi",
input: "/cdn.example.com/photos/cat.jpg/800x600.webp",
want: &ParsedURL{
Host: testHostCDN,
Path: testPathCat,
Host: "cdn.example.com",
Path: "/photos/cat.jpg",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
},
@@ -201,30 +194,35 @@ func TestParseImagePath(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := ParseImagePath(tt.input)
if (err != nil) != tt.wantErr {
t.Errorf("ParseImagePath() error = %v, wantErr %v", err, tt.wantErr)
return
}
if err != nil {
return
}
assertParsedURL(t, got, tt.want)
if got.Host != tt.want.Host {
t.Errorf("Host = %q, want %q", got.Host, tt.want.Host)
}
if got.Path != tt.want.Path {
t.Errorf("Path = %q, want %q", got.Path, tt.want.Path)
}
if got.Size != tt.want.Size {
t.Errorf("Size = %v, want %v", got.Size, tt.want.Size)
}
if got.Format != tt.want.Format {
t.Errorf("Format = %q, want %q", got.Format, tt.want.Format)
}
})
}
}
func TestParsedURL_ToImageRequest(t *testing.T) {
t.Parallel()
parsed := &ParsedURL{
Host: testHostCDN,
Path: testPathCat,
Host: "cdn.example.com",
Path: "/photos/cat.jpg",
Query: "version=2",
Size: Size{Width: 800, Height: 600},
Format: FormatWebP,
@@ -235,27 +233,21 @@ func TestParsedURL_ToImageRequest(t *testing.T) {
if req.SourceHost != parsed.Host {
t.Errorf("SourceHost = %q, want %q", req.SourceHost, parsed.Host)
}
if req.SourcePath != parsed.Path {
t.Errorf("SourcePath = %q, want %q", req.SourcePath, parsed.Path)
}
if req.SourceQuery != parsed.Query {
t.Errorf("SourceQuery = %q, want %q", req.SourceQuery, parsed.Query)
}
if req.Size != parsed.Size {
t.Errorf("Size = %v, want %v", req.Size, parsed.Size)
}
if req.Format != parsed.Format {
t.Errorf("Format = %q, want %q", req.Format, parsed.Format)
}
}
func TestParseImageURL_PathTraversal(t *testing.T) {
t.Parallel()
// All path traversal attempts should be rejected
tests := []struct {
name string
@@ -301,14 +293,12 @@ func TestParseImageURL_PathTraversal(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
_, err := ParseImageURL(tt.input)
if err == nil {
t.Error("ParseImageURL() should reject path traversal attempts")
}
if !errors.Is(err, ErrPathTraversal) {
if err != ErrPathTraversal {
t.Errorf("ParseImageURL() error = %v, want ErrPathTraversal", err)
}
})
@@ -316,8 +306,6 @@ func TestParseImageURL_PathTraversal(t *testing.T) {
}
func TestParseImagePath_PathTraversal(t *testing.T) {
t.Parallel()
// Test path traversal via ParseImagePath (chi wildcard)
tests := []struct {
name string
@@ -335,14 +323,12 @@ func TestParseImagePath_PathTraversal(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
_, err := ParseImagePath(tt.input)
if err == nil {
t.Error("ParseImagePath() should reject path traversal attempts")
}
if !errors.Is(err, ErrPathTraversal) {
if err != ErrPathTraversal {
t.Errorf("ParseImagePath() error = %v, want ErrPathTraversal", err)
}
})
@@ -351,7 +337,7 @@ func TestParseImagePath_PathTraversal(t *testing.T) {
// errorIs checks if err matches target (handles wrapped errors).
func errorIs(err, target error) bool {
if errors.Is(err, target) {
if err == target {
return true
}
// Check if error message contains target message for wrapped errors

1
internal/logger/logger.go
View File

@@ -15,7 +15,6 @@ import (
// Params defines dependencies for Logger.
type Params struct {
fx.In
Globals *globals.Globals
}

19
internal/magic/magic.go
View File

@@ -46,10 +46,6 @@ const (
// MinMagicBytes is the minimum number of bytes needed to detect format.
const MinMagicBytes = 12
// mimeOctetStream is the fallback MIME type for formats without a
// specific MIME type.
const mimeOctetStream = "application/octet-stream"
// Magic byte signatures for supported formats.
// These are effectively constants but Go doesn't support const slices.
//
@@ -194,17 +190,14 @@ func IsSupportedMIMEType(mimeType string) bool {
func PeekAndValidate(r io.Reader, declaredType string) (io.Reader, error) {
// Read minimum bytes for detection
buf := make([]byte, MinMagicBytes)
n, err := io.ReadFull(r, buf)
if err != nil && !errors.Is(err, io.ErrUnexpectedEOF) {
if err != nil && err != io.ErrUnexpectedEOF {
return nil, err
}
buf = buf[:n]
// Validate magic bytes
err = ValidateMagicBytes(buf, declaredType)
if err != nil {
if err := ValidateMagicBytes(buf, declaredType); err != nil {
return nil, err
}
@@ -226,9 +219,6 @@ func MIMEToImageFormat(mimeType string) (ImageFormat, bool) {
return FormatGIF, true
case MIMETypeAVIF:
return FormatAVIF, true
case MIMETypeSVG:
// SVG has no corresponding output format.
return "", false
default:
return "", false
}
@@ -247,10 +237,7 @@ func ImageFormatToMIME(format ImageFormat) string {
return string(MIMETypeGIF)
case FormatAVIF:
return string(MIMETypeAVIF)
case FormatOriginal:
// Original format passes content through unchanged.
return mimeOctetStream
default:
return mimeOctetStream
return "application/octet-stream"
}
}

279
internal/magic/magic_internal_test.go → internal/magic/magic_test.go
View File

@@ -2,90 +2,121 @@ package magic
import (
"bytes"
"errors"
"io"
"slices"
"strings"
"testing"
)
// Shared test fixture strings.
const (
testNameEmpty = "empty"
testMIMEJPEG = "image/jpeg"
testMIMEJPEGParams = "image/jpeg; charset=utf-8"
testMIMEPNG = "image/png"
testMIMEWebP = "image/webp"
testMIMEGIF = "image/gif"
testMIMEAVIF = "image/avif"
)
// pad appends zero bytes so data is comfortably above MinMagicBytes.
func pad(b ...byte) []byte {
return append(b, make([]byte, 100)...)
}
func TestDetectFormat(t *testing.T) {
t.Parallel()
jpeg := pad(0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 0x4A, 0x46, 0x49, 0x46, 0x00, 0x01)
png := pad(0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x00, 0x00, 0x00, 0x0D)
gif87a := pad(0x47, 0x49, 0x46, 0x38, 0x37, 0x61, 0, 0, 0, 0, 0, 0)
gif89a := pad(0x47, 0x49, 0x46, 0x38, 0x39, 0x61, 0, 0, 0, 0, 0, 0)
// RIFF + size placeholder + WEBP
webp := pad(0x52, 0x49, 0x46, 0x46, 0, 0, 0, 0, 0x57, 0x45, 0x42, 0x50)
// box size + ftyp + brand
avif := pad(0x00, 0x00, 0x00, 0x1C, 0x66, 0x74, 0x79, 0x70, 0x61, 0x76, 0x69, 0x66)
avis := pad(0x00, 0x00, 0x00, 0x1C, 0x66, 0x74, 0x79, 0x70, 0x61, 0x76, 0x69, 0x73)
tests := []struct {
name string
data []byte
wantMIME MIMEType
wantErr error
}{
{name: "JPEG", data: jpeg, wantMIME: MIMETypeJPEG},
{name: "PNG", data: png, wantMIME: MIMETypePNG},
{name: "GIF87a", data: gif87a, wantMIME: MIMETypeGIF},
{name: "GIF89a", data: gif89a, wantMIME: MIMETypeGIF},
{name: "WebP", data: webp, wantMIME: MIMETypeWebP},
{name: "AVIF", data: avif, wantMIME: MIMETypeAVIF},
{name: "AVIF sequence", data: avis, wantMIME: MIMETypeAVIF},
{
name: "JPEG",
data: append([]byte{0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 0x4A, 0x46, 0x49, 0x46, 0x00, 0x01}, make([]byte, 100)...),
wantMIME: MIMETypeJPEG,
wantErr: nil,
},
{
name: "PNG",
data: append([]byte{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x00, 0x00, 0x00, 0x0D}, make([]byte, 100)...),
wantMIME: MIMETypePNG,
wantErr: nil,
},
{
name: "GIF87a",
data: append([]byte{0x47, 0x49, 0x46, 0x38, 0x37, 0x61, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}, make([]byte, 100)...),
wantMIME: MIMETypeGIF,
wantErr: nil,
},
{
name: "GIF89a",
data: append([]byte{0x47, 0x49, 0x46, 0x38, 0x39, 0x61, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}, make([]byte, 100)...),
wantMIME: MIMETypeGIF,
wantErr: nil,
},
{
name: "WebP",
data: append([]byte{
0x52, 0x49, 0x46, 0x46, // RIFF
0x00, 0x00, 0x00, 0x00, // file size (placeholder)
0x57, 0x45, 0x42, 0x50, // WEBP
}, make([]byte, 100)...),
wantMIME: MIMETypeWebP,
wantErr: nil,
},
{
name: "AVIF",
data: append([]byte{
0x00, 0x00, 0x00, 0x1C, // box size
0x66, 0x74, 0x79, 0x70, // ftyp
0x61, 0x76, 0x69, 0x66, // avif brand
}, make([]byte, 100)...),
wantMIME: MIMETypeAVIF,
wantErr: nil,
},
{
name: "AVIF sequence",
data: append([]byte{
0x00, 0x00, 0x00, 0x1C, // box size
0x66, 0x74, 0x79, 0x70, // ftyp
0x61, 0x76, 0x69, 0x73, // avis brand
}, make([]byte, 100)...),
wantMIME: MIMETypeAVIF,
wantErr: nil,
},
{
name: "SVG with XML declaration",
data: []byte(`<?xml version="1.0"?><svg></svg>`),
wantMIME: MIMETypeSVG,
wantErr: nil,
},
{
name: "SVG without declaration",
data: []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`),
wantMIME: MIMETypeSVG,
wantErr: nil,
},
{
name: "SVG with whitespace",
data: []byte(` <?xml version="1.0"?><svg></svg>`),
wantMIME: MIMETypeSVG,
wantErr: nil,
},
{
name: "SVG with BOM",
data: append([]byte{0xEF, 0xBB, 0xBF}, []byte(`<svg></svg>`)...),
wantMIME: MIMETypeSVG,
wantErr: nil,
},
{
name: "unknown format",
data: make([]byte, MinMagicBytes),
wantErr: ErrUnknownFormat,
name: "unknown format",
data: []byte{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00},
wantMIME: "",
wantErr: ErrUnknownFormat,
},
{
name: "too short",
data: []byte{0xFF, 0xD8},
wantMIME: "",
wantErr: ErrNotEnoughData,
},
{
name: "empty",
data: []byte{},
wantMIME: "",
wantErr: ErrNotEnoughData,
},
{name: "too short", data: []byte{0xFF, 0xD8}, wantErr: ErrNotEnoughData},
{name: testNameEmpty, data: []byte{}, wantErr: ErrNotEnoughData},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := DetectFormat(tt.data)
if !errors.Is(err, tt.wantErr) {
if err != tt.wantErr {
t.Errorf("DetectFormat() error = %v, wantErr %v", err, tt.wantErr)
return
@@ -99,10 +130,8 @@ func TestDetectFormat(t *testing.T) {
}
func TestValidateMagicBytes(t *testing.T) {
t.Parallel()
jpegData := pad(0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 0x4A, 0x46, 0x49, 0x46, 0x00, 0x01)
pngData := pad(0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x00, 0x00, 0x00, 0x0D)
jpegData := append([]byte{0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 0x4A, 0x46, 0x49, 0x46, 0x00, 0x01}, make([]byte, 100)...)
pngData := append([]byte{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x00, 0x00, 0x00, 0x0D}, make([]byte, 100)...)
tests := []struct {
name string
@@ -113,42 +142,40 @@ func TestValidateMagicBytes(t *testing.T) {
{
name: "matching JPEG",
data: jpegData,
declaredType: testMIMEJPEG,
declaredType: "image/jpeg",
wantErr: nil,
},
{
name: "matching JPEG with params",
data: jpegData,
declaredType: testMIMEJPEGParams,
declaredType: "image/jpeg; charset=utf-8",
wantErr: nil,
},
{
name: "matching PNG",
data: pngData,
declaredType: testMIMEPNG,
declaredType: "image/png",
wantErr: nil,
},
{
name: "mismatched type",
data: jpegData,
declaredType: testMIMEPNG,
declaredType: "image/png",
wantErr: ErrMagicByteMismatch,
},
{
name: "unknown data",
data: make([]byte, MinMagicBytes),
declaredType: testMIMEJPEG,
data: []byte{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00},
declaredType: "image/jpeg",
wantErr: ErrUnknownFormat,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
err := ValidateMagicBytes(tt.data, tt.declaredType)
if !errors.Is(err, tt.wantErr) {
if err != tt.wantErr {
t.Errorf("ValidateMagicBytes() error = %v, wantErr %v", err, tt.wantErr)
}
})
@@ -156,31 +183,27 @@ func TestValidateMagicBytes(t *testing.T) {
}
func TestIsSupportedMIMEType(t *testing.T) {
t.Parallel()
tests := []struct {
mimeType string
want bool
}{
{testMIMEJPEG, true},
{testMIMEPNG, true},
{testMIMEWebP, true},
{testMIMEGIF, true},
{testMIMEAVIF, true},
{"image/jpeg", true},
{"image/png", true},
{"image/webp", true},
{"image/gif", true},
{"image/avif", true},
{"image/svg+xml", true},
{"IMAGE/JPEG", true},
{testMIMEJPEGParams, true},
{"image/jpeg; charset=utf-8", true},
{"image/tiff", false},
{"image/bmp", false},
{mimeOctetStream, false},
{"application/octet-stream", false},
{"text/plain", false},
{"", false},
}
for _, tt := range tests {
t.Run(tt.mimeType, func(t *testing.T) {
t.Parallel()
if got := IsSupportedMIMEType(tt.mimeType); got != tt.want {
t.Errorf("IsSupportedMIMEType(%q) = %v, want %v", tt.mimeType, got, tt.want)
}
@@ -189,16 +212,8 @@ func TestIsSupportedMIMEType(t *testing.T) {
}
func TestPeekAndValidate(t *testing.T) {
t.Parallel()
jpegMagic := []byte{
0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 0x4A, 0x46, 0x49, 0x46, 0x00, 0x01,
}
pngMagic := []byte{
0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x00, 0x00, 0x00, 0x0D,
}
jpegData := slices.Concat(jpegMagic, []byte("rest of jpeg data"))
pngData := slices.Concat(pngMagic, []byte("rest of png data"))
jpegData := append([]byte{0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 0x4A, 0x46, 0x49, 0x46, 0x00, 0x01}, []byte("rest of jpeg data")...)
pngData := append([]byte{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x00, 0x00, 0x00, 0x0D}, []byte("rest of png data")...)
tests := []struct {
name string
@@ -210,32 +225,30 @@ func TestPeekAndValidate(t *testing.T) {
{
name: "valid JPEG",
data: jpegData,
declaredType: testMIMEJPEG,
declaredType: "image/jpeg",
wantErr: false,
wantData: jpegData,
},
{
name: "valid PNG",
data: pngData,
declaredType: testMIMEPNG,
declaredType: "image/png",
wantErr: false,
wantData: pngData,
},
{
name: "mismatched type",
data: jpegData,
declaredType: testMIMEPNG,
declaredType: "image/png",
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
r := bytes.NewReader(tt.data)
result, err := PeekAndValidate(r, tt.declaredType)
if tt.wantErr {
if err == nil {
t.Error("PeekAndValidate() expected error, got nil")
@@ -259,28 +272,23 @@ func TestPeekAndValidate(t *testing.T) {
}
if !bytes.Equal(got, tt.wantData) {
t.Errorf(
"PeekAndValidate() data mismatch: got %d bytes, want %d bytes",
len(got), len(tt.wantData),
)
t.Errorf("PeekAndValidate() data mismatch: got %d bytes, want %d bytes", len(got), len(tt.wantData))
}
})
}
}
func TestMIMEToImageFormat(t *testing.T) {
t.Parallel()
tests := []struct {
mimeType string
wantFormat ImageFormat
wantOk bool
}{
{testMIMEJPEG, FormatJPEG, true},
{testMIMEPNG, FormatPNG, true},
{testMIMEWebP, FormatWebP, true},
{testMIMEGIF, FormatGIF, true},
{testMIMEAVIF, FormatAVIF, true},
{"image/jpeg", FormatJPEG, true},
{"image/png", FormatPNG, true},
{"image/webp", FormatWebP, true},
{"image/gif", FormatGIF, true},
{"image/avif", FormatAVIF, true},
{"image/svg+xml", "", false}, // SVG doesn't convert to ImageFormat
{"image/tiff", "", false},
{"text/plain", "", false},
@@ -288,8 +296,6 @@ func TestMIMEToImageFormat(t *testing.T) {
for _, tt := range tests {
t.Run(tt.mimeType, func(t *testing.T) {
t.Parallel()
got, ok := MIMEToImageFormat(tt.mimeType)
if ok != tt.wantOk {
@@ -304,25 +310,21 @@ func TestMIMEToImageFormat(t *testing.T) {
}
func TestImageFormatToMIME(t *testing.T) {
t.Parallel()
tests := []struct {
format ImageFormat
wantMIME string
}{
{FormatJPEG, testMIMEJPEG},
{FormatPNG, testMIMEPNG},
{FormatWebP, testMIMEWebP},
{FormatGIF, testMIMEGIF},
{FormatAVIF, testMIMEAVIF},
{FormatOriginal, mimeOctetStream},
{"unknown", mimeOctetStream},
{FormatJPEG, "image/jpeg"},
{FormatPNG, "image/png"},
{FormatWebP, "image/webp"},
{FormatGIF, "image/gif"},
{FormatAVIF, "image/avif"},
{FormatOriginal, "application/octet-stream"},
{"unknown", "application/octet-stream"},
}
for _, tt := range tests {
t.Run(string(tt.format), func(t *testing.T) {
t.Parallel()
got := ImageFormatToMIME(tt.format)
if got != tt.wantMIME {
@@ -333,23 +335,19 @@ func TestImageFormatToMIME(t *testing.T) {
}
func TestNormalizeMIMEType(t *testing.T) {
t.Parallel()
tests := []struct {
input string
want string
}{
{testMIMEJPEG, testMIMEJPEG},
{"IMAGE/JPEG", testMIMEJPEG},
{testMIMEJPEGParams, testMIMEJPEG},
{" image/jpeg ", testMIMEJPEG},
{"image/jpeg; boundary=something", testMIMEJPEG},
{"image/jpeg", "image/jpeg"},
{"IMAGE/JPEG", "image/jpeg"},
{"image/jpeg; charset=utf-8", "image/jpeg"},
{" image/jpeg ", "image/jpeg"},
{"image/jpeg; boundary=something", "image/jpeg"},
}
for _, tt := range tests {
t.Run(tt.input, func(t *testing.T) {
t.Parallel()
got := normalizeMIMEType(tt.input)
if got != tt.want {
@@ -360,8 +358,6 @@ func TestNormalizeMIMEType(t *testing.T) {
}
func TestDetectSVG(t *testing.T) {
t.Parallel()
tests := []struct {
name string
data string
@@ -369,24 +365,17 @@ func TestDetectSVG(t *testing.T) {
}{
{"xml declaration", `<?xml version="1.0"?><svg></svg>`, true},
{"svg element", `<svg xmlns="http://www.w3.org/2000/svg"></svg>`, true},
{
"doctype",
`<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" ` +
`"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">`,
true,
},
{"doctype", `<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">`, true},
{"with whitespace", `
<?xml version="1.0"?><svg></svg>`, true},
{"uppercase", `<SVG></SVG>`, true},
{"not svg", `<html></html>`, false},
{"random text", `hello world`, false},
{testNameEmpty, ``, false},
{"empty", ``, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got := detectSVG([]byte(tt.data))
if got != tt.want {
@@ -397,8 +386,6 @@ func TestDetectSVG(t *testing.T) {
}
func TestSkipBOM(t *testing.T) {
t.Parallel()
tests := []struct {
name string
data []byte
@@ -406,15 +393,13 @@ func TestSkipBOM(t *testing.T) {
}{
{"with BOM", []byte{0xEF, 0xBB, 0xBF, 'h', 'e', 'l', 'l', 'o'}, []byte("hello")},
{"without BOM", []byte("hello"), []byte("hello")},
{testNameEmpty, []byte{}, []byte{}},
{"empty", []byte{}, []byte{}},
{"only BOM", []byte{0xEF, 0xBB, 0xBF}, []byte{}},
{"partial BOM", []byte{0xEF, 0xBB}, []byte{0xEF, 0xBB}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got := skipBOM(tt.data)
if !bytes.Equal(got, tt.want) {
@@ -425,8 +410,6 @@ func TestSkipBOM(t *testing.T) {
}
func TestRealWorldSVGPatterns(t *testing.T) {
t.Parallel()
// Test various real-world SVG patterns
svgPatterns := []string{
`<?xml version="1.0" encoding="UTF-8"?>
@@ -436,8 +419,7 @@ func TestRealWorldSVGPatterns(t *testing.T) {
`<svg viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M12 2L2 7l10 5 10-5-10-5z"/>
</svg>`,
`<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" ` +
`"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">` + `
`<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg">
</svg>`,
}
@@ -463,8 +445,6 @@ func TestRealWorldSVGPatterns(t *testing.T) {
}
func TestDetectFormatRIFFNotWebP(t *testing.T) {
t.Parallel()
// RIFF container but not WebP (e.g., WAV file)
wavData := []byte{
0x52, 0x49, 0x46, 0x46, // RIFF
@@ -473,14 +453,12 @@ func TestDetectFormatRIFFNotWebP(t *testing.T) {
}
_, err := DetectFormat(wavData)
if !errors.Is(err, ErrUnknownFormat) {
if err != ErrUnknownFormat {
t.Errorf("DetectFormat(WAV) error = %v, want %v", err, ErrUnknownFormat)
}
}
func TestDetectFormatFtypNotAVIF(t *testing.T) {
t.Parallel()
// ftyp container but not AVIF (e.g., MP4)
mp4Data := []byte{
0x00, 0x00, 0x00, 0x1C, // box size
@@ -489,24 +467,20 @@ func TestDetectFormatFtypNotAVIF(t *testing.T) {
}
_, err := DetectFormat(mp4Data)
if !errors.Is(err, ErrUnknownFormat) {
if err != ErrUnknownFormat {
t.Errorf("DetectFormat(MP4) error = %v, want %v", err, ErrUnknownFormat)
}
}
func TestPeekAndValidatePreservesReader(t *testing.T) {
t.Parallel()
// Ensure that after PeekAndValidate, we can read the complete
// original content
// Ensure that after PeekAndValidate, we can read the complete original content
originalContent := append(
[]byte{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x00, 0x00, 0x00, 0x0D},
[]byte(strings.Repeat("PNG IDAT chunk data here ", 100))...,
)
r := bytes.NewReader(originalContent)
validated, err := PeekAndValidate(r, testMIMEPNG)
validated, err := PeekAndValidate(r, "image/png")
if err != nil {
t.Fatalf("PeekAndValidate() error = %v", err)
}
@@ -518,9 +492,6 @@ func TestPeekAndValidatePreservesReader(t *testing.T) {
}
if !bytes.Equal(got, originalContent) {
t.Errorf(
"Content mismatch: got %d bytes, want %d bytes",
len(got), len(originalContent),
)
t.Errorf("Content mismatch: got %d bytes, want %d bytes", len(got), len(originalContent))
}
}

4
internal/middleware/middleware.go
View File

@@ -24,7 +24,6 @@ const CORSMaxAgeSeconds = 86400
// Params defines dependencies for Middleware.
type Params struct {
fx.In
Logger *logger.Logger
Config *config.Config
}
@@ -50,7 +49,6 @@ func ipFromHostPort(hp string) string {
if err != nil {
return ""
}
if len(h) > 0 && h[0] == '[' {
return h[1 : len(h)-1]
}
@@ -60,7 +58,6 @@ func ipFromHostPort(hp string) string {
type loggingResponseWriter struct {
http.ResponseWriter
statusCode int
bytesWritten int64
}
@@ -88,7 +85,6 @@ func (s *Middleware) Logging() func(http.Handler) http.Handler {
start := time.Now()
lrw := newLoggingResponseWriter(w)
ctx := r.Context()
defer func() {
latency := time.Since(start)
reqID, _ := ctx.Value(middleware.RequestIDKey).(string)

10
internal/middleware/middleware_internal_test.go → internal/middleware/middleware_test.go
View File

@@ -10,8 +10,6 @@ import (
)
func TestSecurityHeaders(t *testing.T) {
t.Parallel()
// Create middleware instance
cfg := &config.Config{}
mw := &Middleware{
@@ -28,7 +26,7 @@ func TestSecurityHeaders(t *testing.T) {
handler := mw.SecurityHeaders()(testHandler)
// Make a test request
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/test", nil)
req := httptest.NewRequest(http.MethodGet, "/test", nil)
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, req)
@@ -46,8 +44,6 @@ func TestSecurityHeaders(t *testing.T) {
for _, tt := range tests {
t.Run(tt.header, func(t *testing.T) {
t.Parallel()
got := rec.Header().Get(tt.header)
if got != tt.want {
t.Errorf("%s = %q, want %q", tt.header, got, tt.want)
@@ -57,8 +53,6 @@ func TestSecurityHeaders(t *testing.T) {
}
func TestSecurityHeaders_PreservesExistingHeaders(t *testing.T) {
t.Parallel()
cfg := &config.Config{}
mw := &Middleware{
log: slog.Default(),
@@ -74,7 +68,7 @@ func TestSecurityHeaders_PreservesExistingHeaders(t *testing.T) {
handler := mw.SecurityHeaders()(testHandler)
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/test", nil)
req := httptest.NewRequest(http.MethodGet, "/test", nil)
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, req)

7
internal/seal/crypto.go
View File

@@ -34,8 +34,7 @@ func DeriveKey(masterKey []byte, salt string) ([KeySize]byte, error) {
hkdfReader := hkdf.New(sha256.New, masterKey, []byte(salt), nil)
_, err := io.ReadFull(hkdfReader, key[:])
if err != nil {
if _, err := io.ReadFull(hkdfReader, key[:]); err != nil {
return key, ErrKeyDerivation
}
@@ -47,9 +46,7 @@ func DeriveKey(masterKey []byte, salt string) ([KeySize]byte, error) {
func Encrypt(key [KeySize]byte, plaintext []byte) (string, error) {
// Generate random nonce
var nonce [NonceSize]byte
_, err := rand.Read(nonce[:])
if err != nil {
if _, err := rand.Read(nonce[:]); err != nil {
return "", err
}

89
internal/seal/crypto_test.go
View File

@@ -1,25 +1,20 @@
package seal_test
package seal
import (
"bytes"
"errors"
"testing"
"sneak.berlin/go/pixa/internal/seal"
)
func TestDeriveKey_Consistent(t *testing.T) {
t.Parallel()
masterKey := []byte("test-master-key-12345")
salt := "test-salt-v1"
key1, err := seal.DeriveKey(masterKey, salt)
key1, err := DeriveKey(masterKey, salt)
if err != nil {
t.Fatalf("DeriveKey() error = %v", err)
}
key2, err := seal.DeriveKey(masterKey, salt)
key2, err := DeriveKey(masterKey, salt)
if err != nil {
t.Fatalf("DeriveKey() error = %v", err)
}
@@ -30,16 +25,14 @@ func TestDeriveKey_Consistent(t *testing.T) {
}
func TestDeriveKey_DifferentSalts(t *testing.T) {
t.Parallel()
masterKey := []byte("test-master-key-12345")
key1, err := seal.DeriveKey(masterKey, "salt-1")
key1, err := DeriveKey(masterKey, "salt-1")
if err != nil {
t.Fatalf("DeriveKey() error = %v", err)
}
key2, err := seal.DeriveKey(masterKey, "salt-2")
key2, err := DeriveKey(masterKey, "salt-2")
if err != nil {
t.Fatalf("DeriveKey() error = %v", err)
}
@@ -50,16 +43,14 @@ func TestDeriveKey_DifferentSalts(t *testing.T) {
}
func TestDeriveKey_DifferentMasterKeys(t *testing.T) {
t.Parallel()
salt := "test-salt"
key1, err := seal.DeriveKey([]byte("master-key-1"), salt)
key1, err := DeriveKey([]byte("master-key-1"), salt)
if err != nil {
t.Fatalf("DeriveKey() error = %v", err)
}
key2, err := seal.DeriveKey([]byte("master-key-2"), salt)
key2, err := DeriveKey([]byte("master-key-2"), salt)
if err != nil {
t.Fatalf("DeriveKey() error = %v", err)
}
@@ -70,21 +61,19 @@ func TestDeriveKey_DifferentMasterKeys(t *testing.T) {
}
func TestEncryptDecrypt_RoundTrip(t *testing.T) {
t.Parallel()
key, err := seal.DeriveKey([]byte("test-key"), "test-salt")
key, err := DeriveKey([]byte("test-key"), "test-salt")
if err != nil {
t.Fatalf("DeriveKey() error = %v", err)
}
plaintext := []byte("hello, world! this is a test message.")
ciphertext, err := seal.Encrypt(key, plaintext)
ciphertext, err := Encrypt(key, plaintext)
if err != nil {
t.Fatalf("Encrypt() error = %v", err)
}
decrypted, err := seal.Decrypt(key, ciphertext)
decrypted, err := Decrypt(key, ciphertext)
if err != nil {
t.Fatalf("Decrypt() error = %v", err)
}
@@ -95,17 +84,15 @@ func TestEncryptDecrypt_RoundTrip(t *testing.T) {
}
func TestEncryptDecrypt_EmptyPlaintext(t *testing.T) {
t.Parallel()
key, _ := seal.DeriveKey([]byte("test-key"), "test-salt")
key, _ := DeriveKey([]byte("test-key"), "test-salt")
plaintext := []byte{}
ciphertext, err := seal.Encrypt(key, plaintext)
ciphertext, err := Encrypt(key, plaintext)
if err != nil {
t.Fatalf("Encrypt() error = %v", err)
}
decrypted, err := seal.Decrypt(key, ciphertext)
decrypted, err := Decrypt(key, ciphertext)
if err != nil {
t.Fatalf("Decrypt() error = %v", err)
}
@@ -116,35 +103,31 @@ func TestEncryptDecrypt_EmptyPlaintext(t *testing.T) {
}
func TestDecrypt_WrongKey(t *testing.T) {
t.Parallel()
key1, _ := seal.DeriveKey([]byte("key-1"), "salt")
key2, _ := seal.DeriveKey([]byte("key-2"), "salt")
key1, _ := DeriveKey([]byte("key-1"), "salt")
key2, _ := DeriveKey([]byte("key-2"), "salt")
plaintext := []byte("secret message")
ciphertext, err := seal.Encrypt(key1, plaintext)
ciphertext, err := Encrypt(key1, plaintext)
if err != nil {
t.Fatalf("Encrypt() error = %v", err)
}
_, err = seal.Decrypt(key2, ciphertext)
_, err = Decrypt(key2, ciphertext)
if err == nil {
t.Error("Decrypt() should fail with wrong key")
}
if !errors.Is(err, seal.ErrDecryptionFailed) {
t.Errorf("Decrypt() error = %v, want %v", err, seal.ErrDecryptionFailed)
if err != ErrDecryptionFailed {
t.Errorf("Decrypt() error = %v, want %v", err, ErrDecryptionFailed)
}
}
func TestDecrypt_TamperedCiphertext(t *testing.T) {
t.Parallel()
key, _ := seal.DeriveKey([]byte("test-key"), "test-salt")
key, _ := DeriveKey([]byte("test-key"), "test-salt")
plaintext := []byte("secret message")
ciphertext, err := seal.Encrypt(key, plaintext)
ciphertext, err := Encrypt(key, plaintext)
if err != nil {
t.Fatalf("Encrypt() error = %v", err)
}
@@ -155,51 +138,45 @@ func TestDecrypt_TamperedCiphertext(t *testing.T) {
tampered[10] ^= 0x01
}
_, err = seal.Decrypt(key, string(tampered))
_, err = Decrypt(key, string(tampered))
if err == nil {
t.Error("Decrypt() should fail with tampered ciphertext")
}
}
func TestDecrypt_InvalidBase64(t *testing.T) {
t.Parallel()
key, _ := DeriveKey([]byte("test-key"), "test-salt")
key, _ := seal.DeriveKey([]byte("test-key"), "test-salt")
_, err := seal.Decrypt(key, "not-valid-base64!!!")
_, err := Decrypt(key, "not-valid-base64!!!")
if err == nil {
t.Error("Decrypt() should fail with invalid base64")
}
if !errors.Is(err, seal.ErrInvalidPayload) {
t.Errorf("Decrypt() error = %v, want %v", err, seal.ErrInvalidPayload)
if err != ErrInvalidPayload {
t.Errorf("Decrypt() error = %v, want %v", err, ErrInvalidPayload)
}
}
func TestDecrypt_TooShort(t *testing.T) {
t.Parallel()
key, _ := seal.DeriveKey([]byte("test-key"), "test-salt")
key, _ := DeriveKey([]byte("test-key"), "test-salt")
// Create a base64 string that's too short to contain nonce + auth tag
_, err := seal.Decrypt(key, "dG9vLXNob3J0")
_, err := Decrypt(key, "dG9vLXNob3J0")
if err == nil {
t.Error("Decrypt() should fail with too-short ciphertext")
}
if !errors.Is(err, seal.ErrInvalidPayload) {
t.Errorf("Decrypt() error = %v, want %v", err, seal.ErrInvalidPayload)
if err != ErrInvalidPayload {
t.Errorf("Decrypt() error = %v, want %v", err, ErrInvalidPayload)
}
}
func TestEncrypt_ProducesDifferentCiphertexts(t *testing.T) {
t.Parallel()
key, _ := seal.DeriveKey([]byte("test-key"), "test-salt")
key, _ := DeriveKey([]byte("test-key"), "test-salt")
plaintext := []byte("same message")
ciphertext1, _ := seal.Encrypt(key, plaintext)
ciphertext2, _ := seal.Encrypt(key, plaintext)
ciphertext1, _ := Encrypt(key, plaintext)
ciphertext2, _ := Encrypt(key, plaintext)
if ciphertext1 == ciphertext2 {
t.Error("Encrypt() should produce different ciphertexts due to random nonce")

6
internal/server/http.go
View File

@@ -1,7 +1,6 @@
package server
import (
"errors"
"fmt"
"net/http"
"time"
@@ -27,11 +26,8 @@ func (s *Server) serveUntilShutdown() {
s.SetupRoutes()
s.log.Info("http begin listen", "listenaddr", listenAddr)
err := s.httpServer.ListenAndServe()
if err != nil && !errors.Is(err, http.ErrServerClosed) {
if err := s.httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
s.log.Error("listen error", "error", err)
if s.cancelFunc != nil {
s.cancelFunc()
}

3
internal/server/routes.go
View File

@@ -56,8 +56,7 @@ func (s *Server) SetupRoutes() {
s.router.Head("/v1/image/*", s.h.HandleImage())
// Encrypted image URL route
// The trailing filename (e.g., /img.jpg) is ignored but helps
// browsers with content type
// The trailing filename (e.g., /img.jpg) is ignored but helps browsers with content type
s.router.Get("/v1/e/{token}/*", s.h.HandleImageEnc())
// Metrics endpoint with auth

42
internal/server/server.go
View File

@@ -30,7 +30,6 @@ const (
// Params defines dependencies for Server.
type Params struct {
fx.In
Logger *logger.Logger
Globals *globals.Globals
Config *config.Config
@@ -48,6 +47,7 @@ type Server struct {
startupTime time.Time
exitCode int
sentryEnabled bool
ctx context.Context
cancelFunc context.CancelFunc
httpServer *http.Server
router *chi.Mux
@@ -64,9 +64,9 @@ func New(lc fx.Lifecycle, params Params) (*Server, error) {
}
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
OnStart: func(_ context.Context) error {
s.startupTime = time.Now()
go s.Run(context.WithoutCancel(ctx))
go s.Run()
return nil
},
@@ -83,14 +83,9 @@ func New(lc fx.Lifecycle, params Params) (*Server, error) {
}
// Run starts the server.
func (s *Server) Run(ctx context.Context) {
func (s *Server) Run() {
s.enableSentry()
s.serve(ctx)
}
// MaintenanceMode returns whether maintenance mode is enabled.
func (s *Server) MaintenanceMode() bool {
return s.config.MaintenanceMode
s.serve()
}
func (s *Server) enableSentry() {
@@ -108,24 +103,19 @@ func (s *Server) enableSentry() {
s.log.Error("sentry init failure", "error", err)
os.Exit(1)
}
s.log.Info("sentry error reporting activated")
s.sentryEnabled = true
}
func (s *Server) serve(ctx context.Context) int {
ctx, cancelFunc := context.WithCancel(ctx)
s.cancelFunc = cancelFunc
func (s *Server) serve() int {
s.ctx, s.cancelFunc = context.WithCancel(context.Background())
go func() {
c := make(chan os.Signal, 1)
signal.Ignore(syscall.SIGPIPE)
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
sig := <-c
s.log.Info("signal received", "signal", sig)
if s.cancelFunc != nil {
s.cancelFunc()
}
@@ -133,22 +123,19 @@ func (s *Server) serve(ctx context.Context) int {
go s.serveUntilShutdown()
<-ctx.Done()
s.cleanShutdown(ctx)
<-s.ctx.Done()
s.cleanShutdown()
return s.exitCode
}
func (s *Server) cleanShutdown(ctx context.Context) {
func (s *Server) cleanShutdown() {
s.exitCode = 0
ctxShutdown, shutdownCancel := context.WithTimeout(
context.WithoutCancel(ctx), ShutdownTimeout)
ctxShutdown, shutdownCancel := context.WithTimeout(context.Background(), ShutdownTimeout)
defer shutdownCancel()
if s.httpServer != nil {
err := s.httpServer.Shutdown(ctxShutdown)
if err != nil {
if err := s.httpServer.Shutdown(ctxShutdown); err != nil {
s.log.Error("server clean shutdown failed", "error", err)
}
}
@@ -157,3 +144,8 @@ func (s *Server) cleanShutdown(ctx context.Context) {
sentry.Flush(SentryFlushTimeout)
}
}
// MaintenanceMode returns whether maintenance mode is enabled.
func (s *Server) MaintenanceMode() bool {
return s.config.MaintenanceMode
}

4
internal/session/session.go
View File

@@ -107,9 +107,7 @@ func (m *Manager) ValidateSession(r *http.Request) (*Data, error) {
}
var data Data
err = m.sc.Decode(CookieName, cookie.Value, &data)
if err != nil {
if err := m.sc.Decode(CookieName, cookie.Value, &data); err != nil {
return nil, ErrInvalidSession
}

19
internal/session/session_cookie_attributes_test.go
View File

@@ -1,11 +1,9 @@
package session_test
package session
import (
"net/http"
"net/http/httptest"
"testing"
"sneak.berlin/go/pixa/internal/session"
)
// TestSessionCookieAttributesAlwaysSecure verifies that every cookie
@@ -18,9 +16,7 @@ import (
// This covers both cookie-writing paths: CreateSession (the login
// set-cookie path) and ClearSession (the logout delete-cookie path).
func TestSessionCookieAttributesAlwaysSecure(t *testing.T) {
t.Parallel()
mgr, err := session.NewManager("test-signing-key-12345")
mgr, err := NewManager("test-signing-key-12345")
if err != nil {
t.Fatalf("NewManager() error = %v", err)
}
@@ -33,9 +29,7 @@ func TestSessionCookieAttributesAlwaysSecure(t *testing.T) {
name: "CreateSession",
setCookie: func(t *testing.T, w http.ResponseWriter) {
t.Helper()
err := mgr.CreateSession(w)
if err != nil {
if err := mgr.CreateSession(w); err != nil {
t.Fatalf("CreateSession() error = %v", err)
}
},
@@ -51,15 +45,12 @@ func TestSessionCookieAttributesAlwaysSecure(t *testing.T) {
for _, writePath := range writePaths {
t.Run(writePath.name, func(t *testing.T) {
t.Parallel()
w := httptest.NewRecorder()
writePath.setCookie(t, w)
var sessionCookie *http.Cookie
for _, c := range w.Result().Cookies() {
if c.Name == session.CookieName {
if c.Name == CookieName {
sessionCookie = c
break
@@ -67,7 +58,7 @@ func TestSessionCookieAttributesAlwaysSecure(t *testing.T) {
}
if sessionCookie == nil {
t.Fatalf("no cookie named %q was set", session.CookieName)
t.Fatalf("no cookie named %q was set", CookieName)
}
t.Logf("cookie attributes: HttpOnly=%v Secure=%v SameSite=%v",

95
internal/session/session_test.go
View File

@@ -1,55 +1,45 @@
package session_test
package session
import (
"errors"
"net/http"
"net/http/httptest"
"testing"
"time"
"sneak.berlin/go/pixa/internal/session"
)
func TestManager_CreateAndValidate(t *testing.T) {
t.Parallel()
mgr, err := session.NewManager("test-signing-key-12345")
mgr, err := NewManager("test-signing-key-12345")
if err != nil {
t.Fatalf("NewManager() error = %v", err)
}
// Create a session
w := httptest.NewRecorder()
err = mgr.CreateSession(w)
if err != nil {
if err := mgr.CreateSession(w); err != nil {
t.Fatalf("CreateSession() error = %v", err)
}
// Extract the cookie from response
resp := w.Result()
cookies := resp.Cookies()
if len(cookies) == 0 {
t.Fatal("CreateSession() did not set a cookie")
}
var sessionCookie *http.Cookie
for _, c := range cookies {
if c.Name == session.CookieName {
if c.Name == CookieName {
sessionCookie = c
break
}
}
if sessionCookie == nil {
t.Fatalf("CreateSession() did not set cookie named %q", session.CookieName)
t.Fatalf("CreateSession() did not set cookie named %q", CookieName)
}
// Validate the session
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req := httptest.NewRequest(http.MethodGet, "/", nil)
req.AddCookie(sessionCookie)
data, err := mgr.ValidateSession(req)
@@ -67,34 +57,27 @@ func TestManager_CreateAndValidate(t *testing.T) {
}
func TestManager_ValidateSession_NoCookie(t *testing.T) {
t.Parallel()
mgr, _ := NewManager("test-signing-key-12345")
mgr, _ := session.NewManager("test-signing-key-12345")
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req := httptest.NewRequest(http.MethodGet, "/", nil)
_, err := mgr.ValidateSession(req)
if err == nil {
t.Error("ValidateSession() should fail with no cookie")
}
if !errors.Is(err, session.ErrNoSession) {
t.Errorf("ValidateSession() error = %v, want %v", err, session.ErrNoSession)
if err != ErrNoSession {
t.Errorf("ValidateSession() error = %v, want %v", err, ErrNoSession)
}
}
func TestManager_ValidateSession_TamperedCookie(t *testing.T) {
t.Parallel()
mgr, _ := NewManager("test-signing-key-12345")
mgr, _ := session.NewManager("test-signing-key-12345")
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req := httptest.NewRequest(http.MethodGet, "/", nil)
req.AddCookie(&http.Cookie{
Name: session.CookieName,
Value: "tampered-invalid-cookie-value",
Secure: true,
HttpOnly: true,
SameSite: http.SameSiteStrictMode,
Name: CookieName,
Value: "tampered-invalid-cookie-value",
})
_, err := mgr.ValidateSession(req)
@@ -102,35 +85,30 @@ func TestManager_ValidateSession_TamperedCookie(t *testing.T) {
t.Error("ValidateSession() should fail with tampered cookie")
}
if !errors.Is(err, session.ErrInvalidSession) {
t.Errorf("ValidateSession() error = %v, want %v", err, session.ErrInvalidSession)
if err != ErrInvalidSession {
t.Errorf("ValidateSession() error = %v, want %v", err, ErrInvalidSession)
}
}
func TestManager_ValidateSession_WrongKey(t *testing.T) {
t.Parallel()
mgr1, _ := session.NewManager("signing-key-1")
mgr2, _ := session.NewManager("signing-key-2")
mgr1, _ := NewManager("signing-key-1")
mgr2, _ := NewManager("signing-key-2")
// Create session with mgr1
w := httptest.NewRecorder()
_ = mgr1.CreateSession(w)
resp := w.Result()
var sessionCookie *http.Cookie
for _, c := range resp.Cookies() {
if c.Name == session.CookieName {
if c.Name == CookieName {
sessionCookie = c
break
}
}
// Try to validate with mgr2 (different key)
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req := httptest.NewRequest(http.MethodGet, "/", nil)
req.AddCookie(sessionCookie)
_, err := mgr2.ValidateSession(req)
@@ -140,9 +118,7 @@ func TestManager_ValidateSession_WrongKey(t *testing.T) {
}
func TestManager_ClearSession(t *testing.T) {
t.Parallel()
mgr, _ := session.NewManager("test-signing-key-12345")
mgr, _ := NewManager("test-signing-key-12345")
w := httptest.NewRecorder()
mgr.ClearSession(w)
@@ -151,11 +127,9 @@ func TestManager_ClearSession(t *testing.T) {
cookies := resp.Cookies()
var sessionCookie *http.Cookie
for _, c := range cookies {
if c.Name == session.CookieName {
if c.Name == CookieName {
sessionCookie = c
break
}
}
@@ -170,12 +144,10 @@ func TestManager_ClearSession(t *testing.T) {
}
func TestManager_IsAuthenticated(t *testing.T) {
t.Parallel()
mgr, _ := session.NewManager("test-signing-key-12345")
mgr, _ := NewManager("test-signing-key-12345")
// No session - should return false
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req := httptest.NewRequest(http.MethodGet, "/", nil)
if mgr.IsAuthenticated(req) {
t.Error("IsAuthenticated() should return false with no session")
}
@@ -185,19 +157,16 @@ func TestManager_IsAuthenticated(t *testing.T) {
_ = mgr.CreateSession(w)
resp := w.Result()
var sessionCookie *http.Cookie
for _, c := range resp.Cookies() {
if c.Name == session.CookieName {
if c.Name == CookieName {
sessionCookie = c
break
}
}
// With valid session - should return true
req = httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req = httptest.NewRequest(http.MethodGet, "/", nil)
req.AddCookie(sessionCookie)
if !mgr.IsAuthenticated(req) {
@@ -206,21 +175,16 @@ func TestManager_IsAuthenticated(t *testing.T) {
}
func TestManager_CookieAttributes(t *testing.T) {
t.Parallel()
mgr, _ := session.NewManager("test-key")
mgr, _ := NewManager("test-key")
w := httptest.NewRecorder()
_ = mgr.CreateSession(w)
resp := w.Result()
var sessionCookie *http.Cookie
for _, c := range resp.Cookies() {
if c.Name == session.CookieName {
if c.Name == CookieName {
sessionCookie = c
break
}
}
@@ -234,7 +198,6 @@ func TestManager_CookieAttributes(t *testing.T) {
}
if sessionCookie.SameSite != http.SameSiteStrictMode {
t.Errorf("Cookie SameSite = %v, want %v",
sessionCookie.SameSite, http.SameSiteStrictMode)
t.Errorf("Cookie SameSite = %v, want %v", sessionCookie.SameSite, http.SameSiteStrictMode)
}
}

136
internal/signature/golden_test.go
View File

@@ -1,10 +1,8 @@
package signature_test
package signature
import (
"testing"
"time"
"sneak.berlin/go/pixa/internal/signature"
)
// goldenExpiresUnix is the fixed expiration timestamp used by all golden
@@ -14,74 +12,10 @@ const goldenExpiresUnix int64 = 1704067200
// goldenSigningKey is the fixed signing key used by all golden vectors.
const goldenSigningKey = "golden-test-key"
type goldenVector struct {
name string
req signature.Request
// wantSignature is the exact base64url (RFC 4648 URL-safe,
// padded) HMAC-SHA256 signature for the request with Expires
// set to goldenExpiresUnix.
wantSignature string
// wantSignedPath is the exact path returned by
// GenerateSignedURL for the request. The signature and
// expiration are returned separately by GenerateSignedURL and
// are not embedded in the path.
wantSignedPath string
}
// goldenVectors returns the known-answer vectors. The expected values
// were computed once and are hardcoded here.
func goldenVectors() []goldenVector {
return []goldenVector{
{
name: "resized without query",
req: signature.Request{
SourceHost: testHost,
SourcePath: testPath,
SourceQuery: "",
Width: 800,
Height: 600,
Format: testFormatWebP,
},
// Signed data: "cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200"
wantSignature: "x5PfPp8QSDo0cJT96od-AEgrQyOVLfqifH5sst61_-w=",
wantSignedPath: testSignedPath,
},
{
name: "resized with query string",
req: signature.Request{
SourceHost: testHost,
SourcePath: testPath,
SourceQuery: "token=abc&v=2",
Width: 800,
Height: 600,
Format: testFormatWebP,
},
// Signed data:
// "cdn.example.com:/photos/cat.jpg:token=abc&v=2:800:600:webp:1704067200"
wantSignature: "394_Vf9TdQFkpQ3XKFDQSyxgqKq8N7mApf2S4QaHqyo=",
wantSignedPath: "/v1/image/cdn.example.com/photos/cat.jpg" +
"%3Ftoken=abc&v=2/800x600.webp",
},
{
name: "original size without query",
req: signature.Request{
SourceHost: testHost,
SourcePath: testPath,
SourceQuery: "",
Width: 0,
Height: 0,
Format: testFormatPNG,
},
// Signed data: "cdn.example.com:/photos/cat.jpg::0:0:png:1704067200"
wantSignature: "7Be7oteeQwvnSPU4bchyQ4ZGYGsAGBKpeEtuQ02ox60=",
wantSignedPath: "/v1/image/cdn.example.com/photos/cat.jpg/orig.png",
},
}
}
// TestSigner_GoldenVectors pins the exact HMAC-SHA256 signature output and
// the exact generated signed URL path for fully-specified requests with a
// hardcoded signing key.
// hardcoded signing key. The expected values were computed once and are
// hardcoded here as known answers.
//
// If any of these assertions fail, the signed byte format
// ("host:path:query:width:height:format:expiration"), the base64url
@@ -90,14 +24,67 @@ func goldenVectors() []goldenVector {
// deliberately: update these constants only as part of an intentional,
// documented signature format migration.
func TestSigner_GoldenVectors(t *testing.T) {
t.Parallel()
signer := New(goldenSigningKey)
signer := signature.New(goldenSigningKey)
vectors := []struct {
name string
req Request
// wantSignature is the exact base64url (RFC 4648 URL-safe,
// padded) HMAC-SHA256 signature for the request with Expires
// set to goldenExpiresUnix.
wantSignature string
// wantSignedPath is the exact path returned by
// GenerateSignedURL for the request. The signature and
// expiration are returned separately by GenerateSignedURL and
// are not embedded in the path.
wantSignedPath string
}{
{
name: "resized without query",
req: Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "",
Width: 800,
Height: 600,
Format: "webp",
},
// Signed data: "cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200"
wantSignature: "x5PfPp8QSDo0cJT96od-AEgrQyOVLfqifH5sst61_-w=",
wantSignedPath: "/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp",
},
{
name: "resized with query string",
req: Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "token=abc&v=2",
Width: 800,
Height: 600,
Format: "webp",
},
// Signed data: "cdn.example.com:/photos/cat.jpg:token=abc&v=2:800:600:webp:1704067200"
wantSignature: "394_Vf9TdQFkpQ3XKFDQSyxgqKq8N7mApf2S4QaHqyo=",
wantSignedPath: "/v1/image/cdn.example.com/photos/cat.jpg%3Ftoken=abc&v=2/800x600.webp",
},
{
name: "original size without query",
req: Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "",
Width: 0,
Height: 0,
Format: "png",
},
// Signed data: "cdn.example.com:/photos/cat.jpg::0:0:png:1704067200"
wantSignature: "7Be7oteeQwvnSPU4bchyQ4ZGYGsAGBKpeEtuQ02ox60=",
wantSignedPath: "/v1/image/cdn.example.com/photos/cat.jpg/orig.png",
},
}
for _, tt := range goldenVectors() {
for _, tt := range vectors {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
signReq := tt.req
signReq.Expires = time.Unix(goldenExpiresUnix, 0)
@@ -108,7 +95,6 @@ func TestSigner_GoldenVectors(t *testing.T) {
}
urlReq := tt.req
gotPath, _, _ := signer.GenerateSignedURL(&urlReq, time.Hour)
if gotPath != tt.wantSignedPath {
t.Errorf("GenerateSignedURL() path = %q, want %q (signed URL layout changed?)",

51
internal/signature/signature.go
View File

@@ -93,17 +93,31 @@ func (s *Signer) Verify(req *Request) error {
return nil
}
// buildSignatureData creates the string to be signed.
// Format: "host:path:query:width:height:format:expiration"
// All components are used verbatim (exact match). No normalization,
// suffix matching, or wildcard expansion is performed.
func (s *Signer) buildSignatureData(req *Request) string {
return fmt.Sprintf("%s:%s:%s:%d:%d:%s:%d",
req.SourceHost,
req.SourcePath,
req.SourceQuery,
req.Width,
req.Height,
req.Format,
req.Expires.Unix(),
)
}
// GenerateSignedURL creates a complete URL with signature and expiration.
// Returns the path portion that should be appended to the base URL.
func (s *Signer) GenerateSignedURL(
req *Request, ttl time.Duration,
) (string, string, int64) {
func (s *Signer) GenerateSignedURL(req *Request, ttl time.Duration) (path string, sig string, exp int64) {
// Set expiration
req.Expires = time.Now().Add(ttl)
exp := req.Expires.Unix()
exp = req.Expires.Unix()
// Generate signature
sig := s.Sign(req)
sig = s.Sign(req)
req.Signature = sig
// Build the size component
@@ -120,7 +134,6 @@ func (s *Signer) GenerateSignedURL(
// it from the last-slash split. The "?" inside a path segment is
// percent-encoded by clients but chi delivers it decoded, which is
// exactly what the URL parser expects.
var path string
if req.SourceQuery != "" {
path = fmt.Sprintf("/v1/image/%s%s%%3F%s/%s.%s",
req.SourceHost,
@@ -141,26 +154,12 @@ func (s *Signer) GenerateSignedURL(
return path, sig, exp
}
// buildSignatureData creates the string to be signed.
// Format: "host:path:query:width:height:format:expiration"
// All components are used verbatim (exact match). No normalization,
// suffix matching, or wildcard expansion is performed.
func (s *Signer) buildSignatureData(req *Request) string {
return fmt.Sprintf("%s:%s:%s:%d:%d:%s:%d",
req.SourceHost,
req.SourcePath,
req.SourceQuery,
req.Width,
req.Height,
req.Format,
req.Expires.Unix(),
)
}
// ParseParams extracts signature and expiration from query parameters.
func ParseParams(sig, expStr string) (string, time.Time, error) {
func ParseParams(sig, expStr string) (parsed string, expires time.Time, err error) {
parsed = sig
if expStr == "" {
return sig, time.Time{}, nil
return parsed, time.Time{}, nil
}
expUnix, err := strconv.ParseInt(expStr, 10, 64)
@@ -168,5 +167,7 @@ func ParseParams(sig, expStr string) (string, time.Time, error) {
return "", time.Time{}, fmt.Errorf("invalid expiration: %w", err)
}
return sig, time.Unix(expUnix, 0), nil
expires = time.Unix(expUnix, 0)
return parsed, expires, nil
}

460
internal/signature/signature_test.go
View File

@@ -1,36 +1,21 @@
package signature_test
package signature
import (
"errors"
"strings"
"testing"
"time"
"sneak.berlin/go/pixa/internal/signature"
)
// Shared fixture values used across the signature tests.
const (
testHost = "cdn.example.com"
testPath = "/photos/cat.jpg"
testFormatWebP = "webp"
testFormatPNG = "png"
testSignedPath = "/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp"
testSig = "abc123"
)
func TestSigner_Sign(t *testing.T) {
t.Parallel()
signer := New("test-secret-key")
signer := signature.New("test-secret-key")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
req := &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "",
Width: 800,
Height: 600,
Format: testFormatWebP,
Format: "webp",
Expires: time.Unix(1704067200, 0), // Fixed timestamp for reproducibility
}
@@ -39,8 +24,7 @@ func TestSigner_Sign(t *testing.T) {
// Same input should produce same signature
if sig1 != sig2 {
t.Errorf("Sign() produced different signatures for same input: %q vs %q",
sig1, sig2)
t.Errorf("Sign() produced different signatures for same input: %q vs %q", sig1, sig2)
}
// Signature should be non-empty
@@ -49,13 +33,13 @@ func TestSigner_Sign(t *testing.T) {
}
// Different input should produce different signature
req2 := &signature.Request{
SourceHost: testHost,
req2 := &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/dog.jpg", // Different path
SourceQuery: "",
Width: 800,
Height: 600,
Format: testFormatWebP,
Format: "webp",
Expires: time.Unix(1704067200, 0),
}
@@ -65,31 +49,25 @@ func TestSigner_Sign(t *testing.T) {
}
}
// validVerifyRequest returns a fully-populated request that verifies
// successfully once signed.
func validVerifyRequest() *signature.Request {
return &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
Width: 800,
Height: 600,
Format: testFormatWebP,
Expires: time.Now().Add(1 * time.Hour),
}
}
func TestSigner_Verify(t *testing.T) {
signer := New("test-secret-key")
type verifyCase struct {
name string
setup func() *signature.Request
wantErr error
}
func verifyCases(signer *signature.Signer) []verifyCase {
return []verifyCase{
tests := []struct {
name string
setup func() *Request
wantErr error
}{
{
name: "valid signature",
setup: func() *signature.Request {
req := validVerifyRequest()
setup: func() *Request {
req := &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
Width: 800,
Height: 600,
Format: "webp",
Expires: time.Now().Add(1 * time.Hour),
}
req.Signature = signer.Sign(req)
return req
@@ -98,59 +76,74 @@ func verifyCases(signer *signature.Signer) []verifyCase {
},
{
name: "expired signature",
setup: func() *signature.Request {
req := validVerifyRequest()
req.Expires = time.Now().Add(-1 * time.Hour)
setup: func() *Request {
req := &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
Width: 800,
Height: 600,
Format: "webp",
Expires: time.Now().Add(-1 * time.Hour), // Expired
}
req.Signature = signer.Sign(req)
return req
},
wantErr: signature.ErrExpired,
wantErr: ErrExpired,
},
{
name: "invalid signature",
setup: func() *signature.Request {
req := validVerifyRequest()
req.Signature = "invalid-signature"
return req
setup: func() *Request {
return &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
Width: 800,
Height: 600,
Format: "webp",
Expires: time.Now().Add(1 * time.Hour),
Signature: "invalid-signature",
}
},
wantErr: signature.ErrInvalid,
wantErr: ErrInvalid,
},
{
name: "missing expiration",
setup: func() *signature.Request {
req := validVerifyRequest()
req.Expires = time.Time{}
req.Signature = "some-signature"
return req
setup: func() *Request {
return &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
Width: 800,
Height: 600,
Format: "webp",
Signature: "some-signature",
// Expires is zero
}
},
wantErr: signature.ErrMissingExpiration,
wantErr: ErrMissingExpiration,
},
{
name: "tampered request",
setup: func() *signature.Request {
req := validVerifyRequest()
setup: func() *Request {
req := &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
Width: 800,
Height: 600,
Format: "webp",
Expires: time.Now().Add(1 * time.Hour),
}
req.Signature = signer.Sign(req)
// Tamper with the request
req.SourcePath = "/photos/secret.jpg"
return req
},
wantErr: signature.ErrInvalid,
wantErr: ErrInvalid,
},
}
}
func TestSigner_Verify(t *testing.T) {
t.Parallel()
signer := signature.New("test-secret-key")
for _, tt := range verifyCases(signer) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
req := tt.setup()
err := signer.Verify(req)
@@ -158,102 +151,30 @@ func TestSigner_Verify(t *testing.T) {
if err != nil {
t.Errorf("Verify() unexpected error = %v", err)
}
return
}
if !errors.Is(err, tt.wantErr) {
t.Errorf("Verify() error = %v, wantErr %v", err, tt.wantErr)
} else {
if err != tt.wantErr {
t.Errorf("Verify() error = %v, wantErr %v", err, tt.wantErr)
}
}
})
}
}
type tamperCase struct {
name string
tamper func(r *signature.Request)
}
// exactMatchTamperCases mutates one signed component per case; every
// mutation must cause verification to fail with ErrInvalid.
func exactMatchTamperCases() []tamperCase {
return []tamperCase{
{
name: "parent domain does not match subdomain",
tamper: func(r *signature.Request) { r.SourceHost = "example.com" },
},
{
name: "subdomain does not match parent domain",
tamper: func(r *signature.Request) { r.SourceHost = "images.cdn.example.com" },
},
{
name: "sibling subdomain does not match",
tamper: func(r *signature.Request) { r.SourceHost = "images.example.com" },
},
{
name: "host with suffix appended does not match",
tamper: func(r *signature.Request) { r.SourceHost = testHost + ".evil.com" },
},
{
name: "host with prefix does not match",
tamper: func(r *signature.Request) { r.SourceHost = "evilcdn.example.com" },
},
{
name: "different path does not match",
tamper: func(r *signature.Request) { r.SourcePath = "/photos/dog.jpg" },
},
{
name: "path suffix does not match",
tamper: func(r *signature.Request) { r.SourcePath = testPath + "/extra" },
},
{
name: "path prefix does not match",
tamper: func(r *signature.Request) { r.SourcePath = "/other" + testPath },
},
{
name: "different query does not match",
tamper: func(r *signature.Request) { r.SourceQuery = "token=xyz" },
},
{
name: "added query does not match empty query",
tamper: func(r *signature.Request) { r.SourceQuery = "extra=1" },
},
{
name: "removed query does not match",
tamper: func(r *signature.Request) { r.SourceQuery = "" },
},
{
name: "different width does not match",
tamper: func(r *signature.Request) { r.Width = 801 },
},
{
name: "different height does not match",
tamper: func(r *signature.Request) { r.Height = 601 },
},
{
name: "different format does not match",
tamper: func(r *signature.Request) { r.Format = testFormatPNG },
},
}
}
// TestSigner_Verify_ExactMatchOnly verifies that signatures enforce exact
// matching on every URL component. No suffix matching, wildcard matching,
// or partial matching is supported.
func TestSigner_Verify_ExactMatchOnly(t *testing.T) {
t.Parallel()
signer := signature.New("test-secret-key")
signer := New("test-secret-key")
// Base request that we'll sign, then tamper with individual fields.
baseReq := func() *signature.Request {
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
baseReq := func() *Request {
req := &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "token=abc",
Width: 800,
Height: 600,
Format: testFormatWebP,
Format: "webp",
Expires: time.Now().Add(1 * time.Hour),
}
req.Signature = signer.Sign(req)
@@ -261,28 +182,117 @@ func TestSigner_Verify_ExactMatchOnly(t *testing.T) {
return req
}
for _, tt := range exactMatchTamperCases() {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
tests := []struct {
name string
tamper func(req *Request)
}{
{
name: "parent domain does not match subdomain",
tamper: func(req *Request) {
// Signed for cdn.example.com, try example.com
req.SourceHost = "example.com"
},
},
{
name: "subdomain does not match parent domain",
tamper: func(req *Request) {
// Signed for cdn.example.com, try images.cdn.example.com
req.SourceHost = "images.cdn.example.com"
},
},
{
name: "sibling subdomain does not match",
tamper: func(req *Request) {
// Signed for cdn.example.com, try images.example.com
req.SourceHost = "images.example.com"
},
},
{
name: "host with suffix appended does not match",
tamper: func(req *Request) {
// Signed for cdn.example.com, try cdn.example.com.evil.com
req.SourceHost = "cdn.example.com.evil.com"
},
},
{
name: "host with prefix does not match",
tamper: func(req *Request) {
// Signed for cdn.example.com, try evilcdn.example.com
req.SourceHost = "evilcdn.example.com"
},
},
{
name: "different path does not match",
tamper: func(req *Request) {
req.SourcePath = "/photos/dog.jpg"
},
},
{
name: "path suffix does not match",
tamper: func(req *Request) {
req.SourcePath = "/photos/cat.jpg/extra"
},
},
{
name: "path prefix does not match",
tamper: func(req *Request) {
req.SourcePath = "/other/photos/cat.jpg"
},
},
{
name: "different query does not match",
tamper: func(req *Request) {
req.SourceQuery = "token=xyz"
},
},
{
name: "added query does not match empty query",
tamper: func(req *Request) {
req.SourceQuery = "extra=1"
},
},
{
name: "removed query does not match",
tamper: func(req *Request) {
req.SourceQuery = ""
},
},
{
name: "different width does not match",
tamper: func(req *Request) {
req.Width = 801
},
},
{
name: "different height does not match",
tamper: func(req *Request) {
req.Height = 601
},
},
{
name: "different format does not match",
tamper: func(req *Request) {
req.Format = "png"
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
req := baseReq()
tt.tamper(req)
err := signer.Verify(req)
if !errors.Is(err, signature.ErrInvalid) {
t.Errorf("Verify() = %v, want %v", err, signature.ErrInvalid)
if err != ErrInvalid {
t.Errorf("Verify() = %v, want %v", err, ErrInvalid)
}
})
}
// Verify the unmodified base request still passes
t.Run("unmodified request passes", func(t *testing.T) {
t.Parallel()
req := baseReq()
err := signer.Verify(req)
if err != nil {
if err := signer.Verify(req); err != nil {
t.Errorf("Verify() unmodified request failed: %v", err)
}
})
@@ -292,12 +302,10 @@ func TestSigner_Verify_ExactMatchOnly(t *testing.T) {
// string in the signature data, producing different signatures for
// suffix-related hosts.
func TestSigner_Sign_ExactHostInData(t *testing.T) {
t.Parallel()
signer := signature.New("test-secret-key")
signer := New("test-secret-key")
hosts := []string{
testHost,
"cdn.example.com",
"example.com",
"images.example.com",
"images.cdn.example.com",
@@ -307,13 +315,13 @@ func TestSigner_Sign_ExactHostInData(t *testing.T) {
sigs := make(map[string]string)
for _, host := range hosts {
req := &signature.Request{
req := &Request{
SourceHost: host,
SourcePath: testPath,
SourcePath: "/photos/cat.jpg",
SourceQuery: "",
Width: 800,
Height: 600,
Format: testFormatWebP,
Format: "webp",
Expires: time.Unix(1704067200, 0),
}
@@ -327,17 +335,15 @@ func TestSigner_Sign_ExactHostInData(t *testing.T) {
}
func TestSigner_DifferentKeys(t *testing.T) {
t.Parallel()
signer1 := New("secret-key-1")
signer2 := New("secret-key-2")
signer1 := signature.New("secret-key-1")
signer2 := signature.New("secret-key-2")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
req := &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
Width: 800,
Height: 600,
Format: testFormatWebP,
Format: "webp",
Expires: time.Now().Add(1 * time.Hour),
}
@@ -345,38 +351,35 @@ func TestSigner_DifferentKeys(t *testing.T) {
req.Signature = signer1.Sign(req)
// Verify with key 1 should succeed
err := signer1.Verify(req)
if err != nil {
if err := signer1.Verify(req); err != nil {
t.Errorf("Verify() with same key failed: %v", err)
}
// Verify with key 2 should fail
err = signer2.Verify(req)
if !errors.Is(err, signature.ErrInvalid) {
if err := signer2.Verify(req); err != ErrInvalid {
t.Errorf("Verify() with different key should fail, got: %v", err)
}
}
func TestGenerateSignedURL(t *testing.T) {
t.Parallel()
signer := New("test-secret-key")
signer := signature.New("test-secret-key")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
req := &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "",
Width: 800,
Height: 600,
Format: testFormatWebP,
Format: "webp",
}
ttl := 1 * time.Hour
path, sig, exp := signer.GenerateSignedURL(req, ttl)
// Path should be correct format
if path != testSignedPath {
t.Errorf("GenerateSignedURL() path = %q, want %q", path, testSignedPath)
expectedPath := "/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp"
if path != expectedPath {
t.Errorf("GenerateSignedURL() path = %q, want %q", path, expectedPath)
}
// Signature should be non-empty
@@ -386,7 +389,6 @@ func TestGenerateSignedURL(t *testing.T) {
// Expiration should be approximately now + TTL
expTime := time.Unix(exp, 0)
expectedExp := time.Now().Add(ttl)
if expTime.Sub(expectedExp) > time.Second {
t.Errorf("GenerateSignedURL() exp time off by too much")
@@ -399,16 +401,14 @@ func TestGenerateSignedURL(t *testing.T) {
}
func TestGenerateSignedURL_OrigSize(t *testing.T) {
t.Parallel()
signer := New("test-secret-key")
signer := signature.New("test-secret-key")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
req := &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
Width: 0, // Original size
Height: 0,
Format: testFormatPNG,
Format: "png",
}
path, _, _ := signer.GenerateSignedURL(req, time.Hour)
@@ -420,24 +420,21 @@ func TestGenerateSignedURL_OrigSize(t *testing.T) {
}
func TestGenerateSignedURL_WithQueryString(t *testing.T) {
t.Parallel()
signer := New("test-secret-key-for-testing!")
signer := signature.New("test-secret-key-for-testing!")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
req := &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
SourceQuery: "token=abc&v=2",
Width: 800,
Height: 600,
Format: testFormatWebP,
Format: "webp",
}
path, _, _ := signer.GenerateSignedURL(req, time.Hour)
// The path must NOT contain a bare "?" that would be interpreted as
// a query string delimiter. The size segment must appear as the last
// path component.
// The path must NOT contain a bare "?" that would be interpreted as a query string delimiter.
// The size segment must appear as the last path component.
if strings.Contains(path, "?token=abc") {
t.Errorf("GenerateSignedURL() produced bare query string in path: %q", path)
}
@@ -454,28 +451,25 @@ func TestGenerateSignedURL_WithQueryString(t *testing.T) {
}
func TestGenerateSignedURL_WithoutQueryString(t *testing.T) {
t.Parallel()
signer := New("test-secret-key-for-testing!")
signer := signature.New("test-secret-key-for-testing!")
req := &signature.Request{
SourceHost: testHost,
SourcePath: testPath,
req := &Request{
SourceHost: "cdn.example.com",
SourcePath: "/photos/cat.jpg",
Width: 800,
Height: 600,
Format: testFormatWebP,
Format: "webp",
}
path, _, _ := signer.GenerateSignedURL(req, time.Hour)
if path != testSignedPath {
t.Errorf("GenerateSignedURL() path = %q, want %q", path, testSignedPath)
expected := "/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp"
if path != expected {
t.Errorf("GenerateSignedURL() path = %q, want %q", path, expected)
}
}
func TestParseParams(t *testing.T) {
t.Parallel()
tests := []struct {
name string
sig string
@@ -486,21 +480,21 @@ func TestParseParams(t *testing.T) {
}{
{
name: "valid params",
sig: testSig,
sig: "abc123",
expStr: "1704067200",
wantSig: testSig,
wantSig: "abc123",
wantErr: false,
},
{
name: "empty expiration",
sig: testSig,
sig: "abc123",
expStr: "",
wantSig: testSig,
wantSig: "abc123",
wantErr: false,
},
{
name: "invalid expiration",
sig: testSig,
sig: "abc123",
expStr: "not-a-number",
wantErr: true,
},
@@ -508,9 +502,7 @@ func TestParseParams(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
sig, exp, err := signature.ParseParams(tt.sig, tt.expStr)
sig, exp, err := ParseParams(tt.sig, tt.expStr)
if tt.wantErr {
if err == nil {

10
script/bootstrap
View File

@@ -11,11 +11,11 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-08-07. Never "latest"; exact versions only.
GOLANGCI_LINT_VERSION="2.12.2"
# sha256 of golangci-lint-2.12.2-linux-<arch>.tar.gz release archives
GOLANGCI_LINT_SHA256_AMD64="8df580d2670fed8fa984aac0507099af8df275e665215f5c7a2ae3943893a553"
GOLANGCI_LINT_SHA256_ARM64="44cd40a8c76c86755375adfeea52cfd3533cb43d7bd647771e0ae065e166df3a"
# Pinned versions, 2026-07-07. Never "latest"; exact versions only.
GOLANGCI_LINT_VERSION="2.10.1"
# sha256 of golangci-lint-2.10.1-linux-<arch>.tar.gz release archives
GOLANGCI_LINT_SHA256_AMD64="dfa775874cf0561b404a02a8f4481fc69b28091da95aa697259820d429b09c99"
GOLANGCI_LINT_SHA256_ARM64="6652b42ae02915eb2f9cb2a2e0cac99514c8eded8388d88ae3e06e1a52c00de8"
PKGMGR=""
SUDO=""

2
script/test
View File

@@ -17,7 +17,7 @@ run_with_cgo_deps() {
main() {
cd "$ROOT"
echo "Running tests..."
run_with_cgo_deps "CGO_ENABLED=1 go test -timeout 30s -race -v ./..."
run_with_cgo_deps "CGO_ENABLED=1 go test -timeout 30s -v ./..."
}
main "$@"