1 Commits
Author SHA1 Message Date
sneak 8a03d0dbe7 Take the signature, the expiry and the URL scheme off the image request (closes #314)
check / check (push) Canceled after 0s
The image request now holds only its source and its transform, exactly what
the variant key covers. The /v1/image/ handler checks the allowlist and
verifies the signature itself with Signer.Verify(req, sig, expires). Both
image handlers keep the URL's expiry as a local value for Cache-Control. The
image service holds no signer, allowlist or AllowHTTP, and ValidateRequest is
gone. The fetcher takes the source and builds its URL with Source.URL from its
own AllowHTTP, so no URL string is built and parsed again. chooseAutoFormat
returns the format it chooses instead of writing into the request. The signed
URL format and every route's answers are unchanged.

Model: opus-5-5
2026-10-26 12:18:23 +00:00
125 changed files with 6093 additions and 10216 deletions

No files matched your search

+1
View File
@@ -70,3 +70,4 @@ vendor/
# Local dev configs
config.yml
config.yaml
config.dev.yml
+51 -52
View File
@@ -1,76 +1,75 @@
# Repository Rules
Last Updated 2026-10-27
Last Updated 2026-01-08
These rules MUST be followed at all times. This is very important.
These rules MUST be followed at all times, it is very important.
- Never use `git add -A`. Stage specific changes for a deliberate commit. A
commit holds one change. After each change, make a commit with a good one-line
summary.
- Never use `git add -A` - add specific changes to a deliberate commit. A commit
should contain one change. After each change, make a commit with a good
one-line summary.
- NEVER modify the linter config without asking first.
- NEVER modify a test to exclude special cases or otherwise make it pass without
asking first. In almost all cases, change the code, NOT the test. You need
explicit user approval to modify an existing test. If you think a test needs
to change, make your case and ask for permission to proceed. Then stop. A bug
fix may change an existing test without asking only when that test pins
exactly the behavior the filed bug calls wrong. The change must then move the
test to the fixed behavior and nothing else. Any other change to an existing
test needs explicit user approval. Writing NEW tests needs no approval.
- NEVER modify tests to exclude special cases or otherwise get them to pass
without asking first. In almost all cases, the code should be changed, NOT the
tests. If you think the test needs to be changed, make your case for that and
ask for permission to proceed, then stop. You need explicit user approval to
modify existing tests. A bug fix may change an existing test without asking
when that test pins exactly the behavior the filed bug calls wrong, and the
change moves it to the fixed behavior and nothing else; any other change to an
existing test still needs explicit user approval. (You do not need user
approval for writing NEW tests.)
- When linting, assume the linter config is CORRECT. Treat each item the linter
reports as something that legitimately needs fixing in the code.
- When linting, assume the linter config is CORRECT, and that each item output
by the linter is something that legitimately needs fixing in the code.
- When running tests, use `make test`.
- Before committing, run `make check`. It runs `make test`, `make lint` and
`make fmt-check`. Any issue it finds MUST be resolved before committing,
unless you are explicitly told otherwise.
- Before commits, run `make check`. This runs `make lint` and `make test` and
`make check-fmt`. Any issues discovered MUST be resolved before committing
unless explicitly told otherwise.
- When fixing a bug, write a failing test for the bug FIRST. Add appropriate
logging to the test, to make sure the test is written correctly. Commit that.
Then work on the fix until the test passes, without changing the test further.
Then commit the fix.
logging to the test to ensure it is written correctly. Commit that. Then go
about fixing the bug until the test passes (without modifying the test
further). Then commit that.
- When adding a new feature, do the same: write a test first (test-driven
development, TDD). The test does not need to be very complex. Commit the test,
then commit the feature.
- When adding a new feature, do the same - implement a test first (TDD). It
doesn't have to be super complex. Commit the test, then commit the feature.
- When adding a new feature, use a feature branch. The feature branch reaches
`next` through a PR that passes review. It is merged only when the feature is
completely finished and the code is up to standards. The code is up to
standards when it passes `make check`. The branch is deleted after the merge.
Only the user merges `next` into `main`.
- When adding a new feature, use a feature branch. When the feature is
completely finished and the code is up to standards (passes `make check`) then
and only then can the feature branch be merged into `main` and the branch
deleted.
- Write `godoc` comments for all exported types and functions as you go along.
- Write godoc documentation comments for all exported types and functions as you
go along.
- ALWAYS be consistent in naming. If you name something one way in one place,
use the EXACT SAME name in every other place.
- ALWAYS be consistent in naming. If you name something one thing in one place,
name it the EXACT SAME THING in another place.
- Be descriptive and specific in naming. `al` is bad; `SourceHostAllowlist` is
- Be descriptive and specific in naming. `wl` is bad; `SourceHostWhitelist` is
good. `ConnsPerHost` is bad; `MaxConnectionsPerHost` is good.
- This is not prototype or teaching code. It is designed for production. Any
security issue (such as denial of service) or other web vulnerability is a P1
bug. It must be added at the top of `TODO.md`.
- This is not prototype or teaching code - this is designed for production. Any
security issues (such as denial of service) or other web vulnerabilities are
P1 bugs and must be added to TODO.md at the top.
- Do not stub an implementation unless you are specifically instructed to. This
is production code, and we need working implementations.
- As this is production code, no stubbing of implementations unless specifically
instructed. We need working implementations.
- NEVER silently fall back to a different setting when a user's parameter
explicitly specifies a value. For example, if a user requests `format=webp`
and WebP encoding is not supported, return an error. Do NOT silently output
PNG instead. If a user specifies `fit=invalid` and that fit mode does not
exist, return an error. Do NOT silently default to `cover`. A silent fallback
violates the principle of least surprise and masks bugs. Defaults are
acceptable only for OMITTED parameters, never for INVALID explicit values.
explicitly specifies a value. If a user requests format=webp and WebP encoding
is not supported, return an error - do NOT silently output PNG instead. If a
user specifies fit=invalid and that fit mode doesn't exist, return an error -
do NOT silently default to "cover". Silent fallbacks violate the principle of
least surprise and mask bugs. The only acceptable defaults are for OMITTED
parameters, never for INVALID explicit values.
- Avoid vendoring dependencies unless specifically instructed to. NEVER commit
the `vendor/` directory. NEVER commit compiled binaries. If such a directory
or file exists and `.gitignore` does not list it, add it to `.gitignore`.
Commit that `.gitignore` change. Keep the entire git repository, history
included, under 20 MiB. Go over 20 MiB only when you specifically must commit
a large file, such as a media file used as a test fixture. Only OUR source
code and its immediately supporting files, such as test examples, go into the
repository and its history.
- Avoid vendoring deps unless specifically instructed to. NEVER commit the
vendor directory, NEVER commit compiled binaries. If these directories or
files exist, add them to .gitignore (and commit the .gitignore) if they are
not already in there. Keep the entire git repository (with history) small -
under 20MiB, unless you specifically must commit larger files (e.g. test
fixture example media files). Only OUR source code and immediately supporting
files (such as test examples) goes into the repo/history.
+4 -7
View File
@@ -79,15 +79,12 @@ docker-versioned:
docker-test:
@script/test
# Run local dev server in Docker, with debug on and a new random signing
# key each run. Log in to the page with the key it prints.
# Run local dev server in Docker
devserver: docker-versioned devserver-stop
@key="$$(head -c 32 /dev/urandom | base64)" && \
docker run -d --name pixad-dev -p 8080:8080 \
-e PIXA_SIGNING_KEY="$$key" -e PIXA_DEBUG=true \
pixa:latest >/dev/null && \
echo "pixad running at http://localhost:8080" && \
echo "signing key: $$key"
-v $(CURDIR)/config.dev.yml:/etc/pixa/config.yml:ro \
pixa:latest
@echo "pixad running at http://localhost:8080"
# Stop dev server
devserver-stop:
+774 -1223
View File
File diff suppressed because it is too large. Load diff
+1207 -2
View File
File diff suppressed because it is too large. Load diff
+2 -13
View File
@@ -1,21 +1,10 @@
// Package main is the entry point for the pixad image proxy server. pixad
// also runs as a child process of itself, to read the size of an SVG source
// or to render it (see imageprocessor.RunAsSVGChildProcess).
// Package main is the entry point for the pixad image proxy server.
package main
import (
"os"
"sneak.berlin/go/pixa/internal/app"
"sneak.berlin/go/pixa/internal/imageprocessor"
)
import "sneak.berlin/go/pixa/internal/app"
var Version string //nolint:gochecknoglobals // set by ldflags
func main() {
if imageprocessor.IsSVGChildProcess(os.Args[1:]) {
os.Exit(imageprocessor.RunAsSVGChildProcess(os.Args[1:]))
}
app.Run(Version)
}
+2 -3
View File
@@ -158,9 +158,8 @@ access_control_allow_origin: "*"
# Sentry DSN for error reporting (default: empty, which turns it off)
sentry_dsn: ""
# Username and password for /metrics, set together (default: unset). pixa
# serves /metrics, and measures HTTP requests, only when both are set. It
# measures the pixa_ metrics either way.
# Username and password for /metrics, set together (default: unset). Metrics
# are measured and /metrics is served only when both are set.
# metrics:
# username: "admin"
# password: "secret"
+1 -1
View File
@@ -17,7 +17,6 @@ require (
github.com/prometheus/client_golang v1.23.2
github.com/slok/go-http-metrics v0.13.0
github.com/spf13/cobra v1.10.2
go.uber.org/dig v1.19.0
go.uber.org/fx v1.24.0
golang.org/x/crypto v0.41.0
golang.org/x/sync v0.19.0
@@ -127,6 +126,7 @@ require (
go.opentelemetry.io/otel v1.36.0 // indirect
go.opentelemetry.io/otel/metric v1.36.0 // indirect
go.opentelemetry.io/otel/trace v1.36.0 // indirect
go.uber.org/dig v1.19.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
+2 -43
View File
@@ -2,16 +2,13 @@
package app
import (
"context"
"fmt"
"io"
"os"
"os/signal"
"syscall"
"github.com/prometheus/client_golang/prometheus"
"github.com/spf13/cobra"
"go.uber.org/dig"
"go.uber.org/fx"
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/database"
@@ -56,10 +53,7 @@ func run(_ *cobra.Command, _ []string) {
// A write to a closed stdout or stderr must not end the process.
signal.Ignore(syscall.SIGPIPE)
app := fx.New(
// pixad logs with slog only; runApp prints a startup or shutdown
// error itself.
fx.NopLogger,
fx.New(
fx.Provide(
config.New,
database.New,
@@ -77,40 +71,5 @@ func run(_ *cobra.Command, _ []string) {
func(log *logger.Logger) { log.Identify() },
func(*server.Server) {},
),
)
exitCode := runApp(app, os.Stderr)
if exitCode != 0 {
os.Exit(exitCode)
}
}
// runApp starts app, waits for a signal or a shutdown request, stops app,
// and returns the status pixad exits with, as fx.App.Run does. It prints
// a start or stop error to stderr, once, as the error that caused it:
// dig.RootCause drops the frames fx and dig wrap around it.
func runApp(app *fx.App, stderr io.Writer) int {
startCtx, cancelStart := context.WithTimeout(context.Background(), app.StartTimeout())
defer cancelStart()
err := app.Start(startCtx)
if err != nil {
_, _ = fmt.Fprintln(stderr, dig.RootCause(err))
return 1
}
shutdown := <-app.Wait()
stopCtx, cancelStop := context.WithTimeout(context.Background(), app.StopTimeout())
defer cancelStop()
err = app.Stop(stopCtx)
if err != nil {
_, _ = fmt.Fprintln(stderr, dig.RootCause(err))
return 1
}
return shutdown.ExitCode
).Run()
}
-65
View File
@@ -1,65 +0,0 @@
package app
import (
"bytes"
"context"
"errors"
"io"
"testing"
"go.uber.org/fx"
)
var errTestStopFailed = errors.New("test stop hook failed")
// TestRunAppPrintsStopErrorOnce checks that when a stop hook fails, runApp
// prints the hook's error once and returns exit status 1.
func TestRunAppPrintsStopErrorOnce(t *testing.T) {
t.Parallel()
app := fx.New(
fx.NopLogger,
fx.Invoke(func(lifecycle fx.Lifecycle, shutdowner fx.Shutdowner) {
lifecycle.Append(fx.Hook{
// Ask to shut down as soon as the app has started, so
// that runApp goes on to stop it.
OnStart: func(context.Context) error { return shutdowner.Shutdown() },
OnStop: func(context.Context) error { return errTestStopFailed },
})
}),
)
var stderr bytes.Buffer
exitCode := runApp(app, &stderr)
if exitCode != 1 {
t.Errorf("runApp() = %d, want 1", exitCode)
}
want := errTestStopFailed.Error() + "\n"
if stderr.String() != want {
t.Errorf("stderr = %q, want %q", stderr.String(), want)
}
}
// TestRunAppReturnsShutdownExitCode checks that runApp returns the exit
// code a shutdown request names.
func TestRunAppReturnsShutdownExitCode(t *testing.T) {
t.Parallel()
app := fx.New(
fx.NopLogger,
fx.Invoke(func(lifecycle fx.Lifecycle, shutdowner fx.Shutdowner) {
lifecycle.Append(fx.Hook{
OnStart: func(context.Context) error {
return shutdowner.Shutdown(fx.ExitCode(3))
},
})
}),
)
exitCode := runApp(app, io.Discard)
if exitCode != 3 {
t.Errorf("runApp() = %d, want 3", exitCode)
}
}
-177
View File
@@ -1,177 +0,0 @@
package app_test
import (
"bytes"
"errors"
"fmt"
"net"
"net/http"
"os"
"os/exec"
"strconv"
"syscall"
"testing"
"time"
"sneak.berlin/go/pixa/internal/app"
)
// These tests run pixad in a child process: the test binary itself,
// started again with runPixadInChildEnvVar set, which makes TestMain run
// pixad instead of the tests.
const runPixadInChildEnvVar = "TEST_RUN_PIXAD_IN_CHILD"
const validTestSigningKey = "test-signing-key-for-app-tests-0123456789"
func TestMain(m *testing.M) {
if os.Getenv(runPixadInChildEnvVar) == "1" {
os.Args = []string{"pixad"}
app.Run("test")
os.Exit(0)
}
os.Exit(m.Run())
}
// pixadCommand returns a command that runs pixad with only the given
// environment, in an empty directory with an empty HOME, so that no
// config file is found.
func pixadCommand(t *testing.T, env ...string) (*exec.Cmd, *bytes.Buffer) {
t.Helper()
testBinary := os.Args[0]
cmd := exec.CommandContext(t.Context(), testBinary) //nolint:gosec // not tainted
cmd.Dir = t.TempDir()
// A test binary built with -cover warns on stderr at exit when
// GOCOVERDIR is not set.
cmd.Env = append([]string{
runPixadInChildEnvVar + "=1",
"HOME=" + t.TempDir(),
"GOCOVERDIR=" + t.TempDir(),
}, env...)
var stderr bytes.Buffer
cmd.Stderr = &stderr
return cmd, &stderr
}
// TestRunPrintsStartupErrorOnce checks that an invalid setting is
// printed once, as the line that names it, and nothing else goes to
// stderr.
func TestRunPrintsStartupErrorOnce(t *testing.T) {
t.Parallel()
cmd, stderr := pixadCommand(t, "PIXA_SIGNING_KEY=short")
err := cmd.Run()
var exitErr *exec.ExitError
if !errors.As(err, &exitErr) || exitErr.ExitCode() != 1 {
t.Fatalf("pixad: got %v, want exit status 1; stderr:\n%s", err, stderr)
}
want := `config key "signing_key" (environment variable PIXA_SIGNING_KEY): ` +
"value too short: must be at least 32 characters, got 5\n"
if stderr.String() != want {
t.Errorf("stderr:\n%s\nwant:\n%s", stderr, want)
}
}
// TestRunWritesNothingToStderrOnStartAndStop starts pixad, waits until
// it answers its health check, stops it with SIGTERM, and checks that it
// wrote nothing to stderr.
func TestRunWritesNothingToStderrOnStartAndStop(t *testing.T) {
t.Parallel()
port := freePort(t)
cmd, stderr := pixadCommand(t,
"PORT="+strconv.Itoa(port),
"PIXA_STATE_DIR="+t.TempDir(),
"PIXA_SIGNING_KEY="+validTestSigningKey,
)
err := cmd.Start()
if err != nil {
t.Fatalf("starting pixad: %v", err)
}
waitForHealthCheck(t, cmd, port, stderr)
err = cmd.Process.Signal(syscall.SIGTERM)
if err != nil {
t.Fatalf("signalling pixad: %v", err)
}
err = cmd.Wait()
if err != nil {
t.Fatalf("pixad: got %v, want exit status 0; stderr:\n%s", err, stderr)
}
if stderr.Len() != 0 {
t.Errorf("stderr is not empty:\n%s", stderr)
}
}
// freePort returns a TCP port on the loopback address that nothing
// listens on.
func freePort(t *testing.T) int {
t.Helper()
var listenConfig net.ListenConfig
listener, err := listenConfig.Listen(t.Context(), "tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("finding a free port: %v", err)
}
addr, ok := listener.Addr().(*net.TCPAddr)
if !ok {
t.Fatalf("listener address %v is not a TCP address", listener.Addr())
}
err = listener.Close()
if err != nil {
t.Fatalf("closing listener: %v", err)
}
return addr.Port
}
// waitForHealthCheck waits up to 30 seconds for pixad, started as cmd, to
// answer its health check on port. If it does not, it stops pixad and
// fails the test with what pixad wrote to stderr.
func waitForHealthCheck(t *testing.T, cmd *exec.Cmd, port int, stderr *bytes.Buffer) {
t.Helper()
url := fmt.Sprintf("http://127.0.0.1:%d/.well-known/healthcheck.json", port)
deadline := time.Now().Add(30 * time.Second)
for time.Now().Before(deadline) {
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, url, nil)
if err != nil {
t.Fatalf("health check request: %v", err)
}
resp, err := http.DefaultClient.Do(req)
if err == nil {
_ = resp.Body.Close()
if resp.StatusCode == http.StatusOK {
return
}
}
time.Sleep(100 * time.Millisecond)
}
// Stop pixad first: stderr is complete, and safe to read, only once
// cmd.Wait has returned.
_ = cmd.Process.Kill()
_ = cmd.Wait()
t.Fatalf("pixad did not answer its health check on port %d; stderr:\n%s",
port, stderr)
}
+586 -345
View File
File diff suppressed because it is too large. Load diff
+85
View File
@@ -1,12 +1,97 @@
package config
import (
"os"
"path/filepath"
"testing"
"git.eeqj.de/sneak/smartconfig"
)
// writeTestConfig writes yamlContent to a temp config file and returns
// the file path.
func writeTestConfig(t *testing.T, yamlContent string) string {
t.Helper()
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.yml")
err := os.WriteFile(configPath, []byte(yamlContent), 0o600)
if err != nil {
t.Fatalf("failed to write test config: %v", err)
}
return configPath
}
// checkAllowlistHosts loads the config at configPath and asserts that
// getStringSlice returns the three expected hosts.
func checkAllowlistHosts(t *testing.T, configPath string) {
t.Helper()
sc, err := loadTestConfig(configPath)
if err != nil {
t.Fatalf("failed to load config: %v", err)
}
hosts := getStringSlice(sc)
if len(hosts) != 3 {
t.Errorf("expected 3 hosts, got %d: %v", len(hosts), hosts)
}
expected := []string{"static.sneak.cloud", "sneak.berlin", testHostS3}
for i, want := range expected {
if i >= len(hosts) {
t.Errorf("missing host at index %d: want %q", i, want)
continue
}
if hosts[i] != want {
t.Errorf("host[%d] = %q, want %q", i, hosts[i], want)
}
}
}
func TestGetStringSlice_YAMLList(t *testing.T) {
t.Parallel()
yamlContent := `
allowlist_hosts:
- static.sneak.cloud
- sneak.berlin
- s3.sneak.cloud
`
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
}
func TestGetStringSlice_CommaSeparated(t *testing.T) {
t.Parallel()
// Backwards compatibility with comma-separated string values.
yamlContent := `allowlist_hosts: "static.sneak.cloud, sneak.berlin, s3.sneak.cloud"`
checkAllowlistHosts(t, writeTestConfig(t, yamlContent))
}
func TestGetStringSlice_Empty(t *testing.T) {
t.Parallel()
configPath := writeTestConfig(t, `port: 8080`)
sc, err := loadTestConfig(configPath)
if err != nil {
t.Fatalf("failed to load config: %v", err)
}
hosts := getStringSlice(sc)
if len(hosts) != 0 {
t.Errorf("expected nil or empty slice, got %v", hosts)
}
}
// TestExampleConfigAllowlistsNoHost loads configs/config.example.yml
// with a real signing key, as Getting Started in README.md has it copied
// and edited, and checks that it allowlists no host: every image on an
@@ -15,36 +15,8 @@ import (
_ "modernc.org/sqlite" // SQLite driver registration
)
// The config keys the tests name in the errors they expect.
const (
keyDebug = "debug"
keyMaintenanceMode = "maintenance_mode"
keyPort = "port"
keyStateDir = "state_dir"
keySentryDSN = "sentry_dsn"
keyDBURL = "db_url"
keyMetrics = "metrics"
keyMetricsUsername = "metrics.username"
keyMetricsPassword = "metrics.password"
keySigningKey = "signing_key"
keyAllowlistHosts = "allowlist_hosts"
keyRefererBlocklist = "referer_blocklist"
keyAllowHTTP = "allow_http"
keyUpstreamConnectionsPerHost = "upstream_connections_per_host"
keyUpstreamConnections = "upstream_connections"
keyMaxConcurrentProcessing = "max_concurrent_processing"
keyCacheMaxBytes = "cache_max_bytes"
keyBlockedNetworks = "blocked_networks"
keyTrustedProxies = "trusted_proxies"
keyAccessControlAllowOrigin = "access_control_allow_origin"
keyUpstreamFetchTimeout = "upstream_fetch_timeout"
keyUpstreamMaxResponseSize = "upstream_max_response_size"
keyDownstreamTimeout = "downstream_timeout"
keyImageRequestsPerMinute = "image_requests_per_minute"
)
// validTestSigningKey is a 32-character signing key that satisfies the
// minimum length requirement in validateSigningKey.
// minimum length requirement in validate().
const validTestSigningKey = "0123456789abcdef0123456789abcdef"
// signingKeyLine is a valid signing_key config line used as the base of
@@ -1,127 +0,0 @@
package config
import (
"os"
"path/filepath"
"regexp"
"strings"
"testing"
)
// TestReadmeNamesEverySetting checks that the list of settings in
// README.md names exactly the keys of the settings table, one item each.
// The list is the first one in the Configuration section, and it ends at
// the first blank line. Each item starts with its key in backticks.
func TestReadmeNamesEverySetting(t *testing.T) {
t.Parallel()
readme, err := os.ReadFile(filepath.Join("..", "..", "README.md"))
if err != nil {
t.Fatalf("failed to read README.md: %v", err)
}
_, section, found := strings.Cut(string(readme), "\n### Configuration\n")
if !found {
t.Fatal("README.md has no Configuration section")
}
// The section ends at the next heading of its level or above.
section, _, _ = strings.Cut(section, "\n## ")
section, _, _ = strings.Cut(section, "\n### ")
_, list, found := strings.Cut(section, "\n- ")
if !found {
t.Fatal("README.md's Configuration section has no list")
}
list, _, _ = strings.Cut("- "+list, "\n\n")
itemKey := regexp.MustCompile("^- `([^`]+)`")
named := map[string]bool{}
for line := range strings.SplitSeq(list, "\n") {
// A line that does not start with "- " goes on with the item above.
if !strings.HasPrefix(line, "- ") {
continue
}
match := itemKey.FindStringSubmatch(line)
if match == nil {
t.Errorf("README.md's list of settings has an item "+
"that does not start with a key in backticks: %s", line)
continue
}
if named[match[1]] {
t.Errorf("README.md's list of settings names %s twice", match[1])
}
named[match[1]] = true
}
for _, s := range new(Config).settings() {
if !named[s.key] {
t.Errorf("README.md's list of settings does not name %s", s.key)
}
delete(named, s.key)
}
for key := range named {
t.Errorf("README.md's list of settings names %s, "+
"which is not a setting", key)
}
}
// TestExampleConfigNamesEverySetting checks that
// configs/config.example.yml names exactly the keys of the settings
// table, each one set or commented out with "# ". A key line holds the
// key, a colon and at most a one-word value, which tells it from a
// comment line that starts with a word and a colon. An indented key is
// inside the map of the key above it, as username is inside metrics.
func TestExampleConfigNamesEverySetting(t *testing.T) {
t.Parallel()
const examplePath = "configs/config.example.yml"
example, err := os.ReadFile(filepath.Join("..", "..", examplePath))
if err != nil {
t.Fatalf("failed to read %s: %v", examplePath, err)
}
keyLine := regexp.MustCompile(`(?m)^(?:# )?( *)([a-z_]+):(?: \S+)?$`)
named := map[string]bool{}
var mapKey string
for _, match := range keyLine.FindAllStringSubmatch(string(example), -1) {
indent, key := match[1], match[2]
if indent == "" {
mapKey = key
named[key] = true
continue
}
// The key holding the map is not itself a setting.
delete(named, mapKey)
named[mapKey+"."+key] = true
}
// The env section sets environment variables; it is not a setting.
delete(named, "env")
for _, s := range new(Config).settings() {
if !named[s.key] {
t.Errorf("%s does not name %s", examplePath, s.key)
}
delete(named, s.key)
}
for key := range named {
t.Errorf("%s names %s, which is not a setting", examplePath, key)
}
}
+3 -4
View File
@@ -138,8 +138,7 @@ func TestPixaPortAbortsStartupPointingToPort(t *testing.T) {
// TestSettingVariablesAndConfigPathAreAccepted checks that every
// setting's variable and PIXA_CONFIG_PATH pass the check for unknown
// PIXA_ variables. TestEnvironmentSetsEveryKey pins the names in the
// settings table.
// PIXA_ variables. TestEnvironmentSetsEveryKey pins the names in the list.
func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) {
// A config file's env section loaded by another test can leave a
// PIXA_ variable set for the whole process, so every one is unset
@@ -160,8 +159,8 @@ func TestSettingVariablesAndConfigPathAreAccepted(t *testing.T) {
t.Setenv("PIXA_CONFIG_PATH", "/etc/pixa/config.yml")
for _, s := range new(Config).settings() {
t.Setenv(s.envVar, "")
for _, name := range envVarNames() {
t.Setenv(name, "")
}
err := validateKnownEnvVars()
+4 -24
View File
@@ -3,24 +3,14 @@
-- Source images, one row per source URL
-- Files stored at: cache/sources/<ab>/<cd>/<source_key>, where source_key
-- is the SHA-256 of the source's host, path and query.
-- fetched_at is when the cached bytes were fetched. checked_at is when
-- upstream last sent them or answered that they had not changed.
-- upstream_etag and upstream_last_modified are the ETag and Last-Modified
-- headers upstream sent with the bytes, or '' when it sent none.
-- is the SHA-256 of the source's host, path and query
CREATE TABLE IF NOT EXISTS source_content (
source_key TEXT PRIMARY KEY,
size_bytes INTEGER NOT NULL,
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
fetched_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
checked_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
upstream_etag TEXT NOT NULL DEFAULT '',
upstream_last_modified TEXT NOT NULL DEFAULT ''
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_source_content_last_accessed
ON source_content(last_accessed_at);
CREATE INDEX IF NOT EXISTS idx_source_content_checked
ON source_content(checked_at);
-- Processed variant blobs
-- Files stored at: cache/variants/<ab>/<cd>/<cache_key>. Tracked here
@@ -28,24 +18,14 @@ CREATE INDEX IF NOT EXISTS idx_source_content_checked
-- without a directory scan, and so LRU eviction has a timestamp to order
-- on. A cache hit does not read this table: it takes the content type
-- from the file's first bytes, and its last_accessed_at is written
-- later, in a batch. checked_at is the checked_at of the source the
-- variant was made from, when it was made or last renewed; the variant
-- file's modification time holds the same time for cache hits.
-- upstream_etag and upstream_last_modified are those of the source bytes
-- the variant was made from, so the variant can still be checked with
-- upstream after its source is evicted.
-- later, in a batch.
CREATE TABLE IF NOT EXISTS variant_content (
cache_key TEXT PRIMARY KEY,
size_bytes INTEGER NOT NULL,
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
checked_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
upstream_etag TEXT NOT NULL DEFAULT '',
upstream_last_modified TEXT NOT NULL DEFAULT ''
last_accessed_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_variant_content_last_accessed
ON variant_content(last_accessed_at);
CREATE INDEX IF NOT EXISTS idx_variant_content_checked
ON variant_content(checked_at);
-- Total cache usage: the sum of size_bytes over source_content and
-- variant_content, kept by the triggers below in the same statement
+12 -4
View File
@@ -16,6 +16,9 @@ const (
DefaultQuality = 85
DefaultFormat = imagerequest.FormatJXL
DefaultFitMode = imagerequest.FitCover
// HKDF salt for URL encryption key derivation
urlKeySalt = "pixa-urlenc-v1"
)
// Errors returned by encurl operations.
@@ -45,10 +48,15 @@ type Generator struct {
key [seal.KeySize]byte
}
// NewGenerator creates an encrypted URL generator that encrypts its tokens
// with key.
func NewGenerator(key [seal.KeySize]byte) *Generator {
return &Generator{key: key}
// NewGenerator creates an encrypted URL generator with a key derived
// from the signing key.
func NewGenerator(signingKey string) (*Generator, error) {
key, err := seal.DeriveKey([]byte(signingKey), urlKeySalt)
if err != nil {
return nil, err
}
return &Generator{key: key}, nil
}
// Generate creates an encrypted URL token from the request's source and
+9 -19
View File
@@ -7,7 +7,6 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
)
// Shared test fixture strings.
@@ -20,13 +19,11 @@ const (
func TestGenerator_GenerateAndParse(t *testing.T) {
t.Parallel()
keys, err := seal.DeriveKeys("test-signing-key-12345")
gen, err := encurl.NewGenerator("test-signing-key-12345")
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
t.Fatalf("NewGenerator() error = %v", err)
}
gen := encurl.NewGenerator(keys.EncryptedURLKey)
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
@@ -102,8 +99,7 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
func TestGenerator_Parse_Expired(t *testing.T) {
t.Parallel()
keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
gen, _ := encurl.NewGenerator("test-signing-key-12345")
req := &imagerequest.Request{
Source: imagerequest.Source{
@@ -130,8 +126,7 @@ func TestGenerator_Parse_Expired(t *testing.T) {
func TestGenerator_Parse_InvalidToken(t *testing.T) {
t.Parallel()
keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
gen, _ := encurl.NewGenerator("test-signing-key-12345")
_, _, err := gen.Parse("not-a-valid-token")
if err == nil {
@@ -142,8 +137,7 @@ func TestGenerator_Parse_InvalidToken(t *testing.T) {
func TestGenerator_Parse_TamperedToken(t *testing.T) {
t.Parallel()
keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
gen, _ := encurl.NewGenerator("test-signing-key-12345")
req := &imagerequest.Request{
Source: imagerequest.Source{
@@ -169,10 +163,8 @@ func TestGenerator_Parse_TamperedToken(t *testing.T) {
func TestGenerator_Parse_WrongKey(t *testing.T) {
t.Parallel()
keys1, _ := seal.DeriveKeys("signing-key-1")
keys2, _ := seal.DeriveKeys("signing-key-2")
gen1 := encurl.NewGenerator(keys1.EncryptedURLKey)
gen2 := encurl.NewGenerator(keys2.EncryptedURLKey)
gen1, _ := encurl.NewGenerator("signing-key-1")
gen2, _ := encurl.NewGenerator("signing-key-2")
req := &imagerequest.Request{
Source: imagerequest.Source{
@@ -192,8 +184,7 @@ func TestGenerator_Parse_WrongKey(t *testing.T) {
func TestGenerator_Parse_Defaults(t *testing.T) {
t.Parallel()
keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
gen, _ := encurl.NewGenerator("test-signing-key-12345")
// Request with only required fields - should get defaults
token, err := gen.Generate(&imagerequest.Request{
@@ -230,8 +221,7 @@ func TestGenerator_Parse_Defaults(t *testing.T) {
func TestGenerator_TokenIsURLSafe(t *testing.T) {
t.Parallel()
keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
gen, _ := encurl.NewGenerator("test-signing-key-12345")
req := &imagerequest.Request{
Source: imagerequest.Source{
+15 -6
View File
@@ -14,7 +14,6 @@ import (
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
@@ -45,17 +44,27 @@ func newCSRFTestRouter(t *testing.T) (*Handlers, http.Handler) {
cfg := &config.Config{SigningKey: testSigningKey, Debug: true}
keys, err := seal.DeriveKeys(testSigningKey)
sessMgr, err := session.NewManager(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
t.Fatalf("session.NewManager() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
protect, err := newCSRFProtect(testSigningKey, cfg.Debug)
if err != nil {
t.Fatalf("newCSRFProtect() error = %v", err)
}
h := &Handlers{
log: slog.New(slog.DiscardHandler),
config: cfg,
sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
encGen: encurl.NewGenerator(keys.EncryptedURLKey),
csrfProtect: newCSRFProtect(keys.CSRFKey, cfg.Debug, slog.New(slog.DiscardHandler)),
sessMgr: sessMgr,
encGen: encGen,
csrfProtect: protect,
}
r := chi.NewRouter()
@@ -10,7 +10,6 @@ import (
"testing"
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
@@ -23,15 +22,15 @@ func TestLoginLogLeavesOutSubmittedKey(t *testing.T) {
var buf bytes.Buffer
keys, err := seal.DeriveKeys(testSigningKey)
sessMgr, err := session.NewManager(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
t.Fatalf("session.NewManager() error = %v", err)
}
h := &Handlers{
log: slog.New(slog.NewJSONHandler(&buf, nil)),
config: &config.Config{SigningKey: testSigningKey},
sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
sessMgr: sessMgr,
}
submittedKeys := []string{wrongKey, testSigningKey}
@@ -11,7 +11,6 @@ import (
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
@@ -233,13 +232,8 @@ func TestHandleLogout_SaveFails_Answers500(t *testing.T) {
db := setupTestDB(t)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
sessMgr, err := session.NewManagerWithDatabase(
t.Context(), keys.SessionHashKey, keys.SessionBlockKey, db)
t.Context(), testSigningKey, db)
if err != nil {
t.Fatalf("session.NewManagerWithDatabase() error = %v", err)
}
@@ -12,11 +12,11 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError, which
// answers the errors of both image routes, answers 400 naming the AVIF pixel
// limit when the image processor refuses an AVIF output above it, wrapped as
// the service wraps it. The request's source only goes into a log line, so it
// is left empty.
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError and
// handleImageError, which answer the errors of the two image routes, answer
// 400 naming the AVIF pixel limit when the image processor refuses an AVIF
// output above it, wrapped as the service wraps it. The request's source only
// goes into a log line, so it is left empty.
func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
t.Parallel()
@@ -25,15 +25,25 @@ func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
imageprocessor.ErrAVIFOutputTooLarge)
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
rec := httptest.NewRecorder()
h.respondImageError(rec, &imagerequest.Request{}, err)
t.Logf("status = %d, body = %q", rec.Code, rec.Body.String())
plain := httptest.NewRecorder()
h.respondImageError(plain, &imagerequest.Request{}, err)
if rec.Code != http.StatusBadRequest {
t.Errorf("status = %d, want %d", rec.Code, http.StatusBadRequest)
}
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
if !strings.Contains(rec.Body.String(), wantBody) {
t.Errorf("body = %q, want it to say %q", rec.Body.String(), wantBody)
for name, rec := range map[string]*httptest.ResponseRecorder{
"respondImageError": plain, "handleImageError": encrypted,
} {
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status = %d, want %d",
name, rec.Code, http.StatusBadRequest)
}
if !strings.Contains(rec.Body.String(), wantBody) {
t.Errorf("%s: body = %q, want it to say %q",
name, rec.Body.String(), wantBody)
}
}
}
+15 -6
View File
@@ -11,7 +11,6 @@ import (
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
@@ -33,17 +32,27 @@ func newBodyLimitTestRouter(
cfg := &config.Config{SigningKey: testSigningKey, Debug: true}
keys, err := seal.DeriveKeys(testSigningKey)
sessMgr, err := session.NewManager(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
t.Fatalf("session.NewManager() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
protect, err := newCSRFProtect(testSigningKey, cfg.Debug)
if err != nil {
t.Fatalf("newCSRFProtect() error = %v", err)
}
h := &Handlers{
log: slog.New(slog.DiscardHandler),
config: cfg,
sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
encGen: encurl.NewGenerator(keys.EncryptedURLKey),
csrfProtect: newCSRFProtect(keys.CSRFKey, cfg.Debug, slog.New(slog.DiscardHandler)),
sessMgr: sessMgr,
encGen: encGen,
csrfProtect: protect,
}
r := chi.NewRouter()
+10 -14
View File
@@ -23,10 +23,8 @@ import (
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -123,7 +121,7 @@ func newCountingDatabaseServer(
tb.Fatalf("encoding the photo: %v", err)
}
cache, err := imgcache.NewCache(db, imgcache.Config{
cache, err := imgcache.NewCache(db, imgcache.CacheConfig{
StateDir: tb.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -132,7 +130,7 @@ func newCountingDatabaseServer(
tb.Fatalf("imgcache.NewCache() error = %v", err)
}
svc, err := imageproxy.New(&imageproxy.Config{
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
Cache: cache,
Fetcher: newMockFetcher(fstest.MapFS{
countingHost + photoPath: &fstest.MapFile{Data: photo.Bytes()},
@@ -143,18 +141,16 @@ func newCountingDatabaseServer(
}),
MaxResponseSize: 50 << 20,
Logger: slog.New(slog.DiscardHandler),
}, prometheus.NewRegistry())
})
if err != nil {
tb.Fatalf("imageproxy.New() error = %v", err)
tb.Fatalf("imgcache.NewService() error = %v", err)
}
keys, err := seal.DeriveKeys(testSigningKey)
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
tb.Fatalf("seal.DeriveKeys() error = %v", err)
tb.Fatalf("encurl.NewGenerator() error = %v", err)
}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
token, err := encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: countingHost, Path: photoPath},
Transform: imagerequest.Transform{
@@ -229,9 +225,9 @@ func TestImageRoutes_CacheHitsMakeNoDatabaseCall(t *testing.T) {
rec := serveImage(t, srv, target)
cacheStatus := rec.Header().Get("X-Pixa-Cache")
if cacheStatus != string(imageproxy.CacheHit) {
if cacheStatus != string(imgcache.CacheHit) {
t.Errorf("GET %s: X-Pixa-Cache = %q, want %s",
target, cacheStatus, imageproxy.CacheHit)
target, cacheStatus, imgcache.CacheHit)
}
}
@@ -261,9 +257,9 @@ func BenchmarkHandleImage_CacheHits(b *testing.B) {
rec := serveImage(b, srv, target)
cacheStatus := rec.Header().Get("X-Pixa-Cache")
if cacheStatus != string(imageproxy.CacheHit) {
if cacheStatus != string(imgcache.CacheHit) {
b.Errorf("GET %s: X-Pixa-Cache = %q, want %s",
target, cacheStatus, imageproxy.CacheHit)
target, cacheStatus, imgcache.CacheHit)
return
}
@@ -1,177 +0,0 @@
package handlers
import (
"bytes"
"context"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"slices"
"testing"
"time"
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// heldFetcher sends on started when a fetch begins, and holds the fetch until
// release is closed. It then fails the fetch as an upstream error.
type heldFetcher struct {
started chan struct{}
release chan struct{}
}
// Fetch sends on f.started, waits for f.release to be closed, and returns
// httpfetcher.ErrUpstreamError.
func (f heldFetcher) Fetch(
context.Context, imagerequest.Source, string, string,
) (*httpfetcher.FetchResult, error) {
f.started <- struct{}{}
<-f.release
return nil, httpfetcher.ErrUpstreamError
}
// readLogLines decodes the JSON log lines in logs.
func readLogLines(t *testing.T, logs *bytes.Buffer) []map[string]any {
t.Helper()
t.Logf("logged:\n%s", logs.String())
var lines []map[string]any
dec := json.NewDecoder(logs)
for dec.More() {
var line map[string]any
err := dec.Decode(&line)
if err != nil {
t.Fatalf("decoding log line: %v", err)
}
lines = append(lines, line)
}
return lines
}
// TestImageRoutes_ClientGoneWhileWaitingAnswers499 checks that both image
// routes answer 499 to a request whose client goes away while it waits for an
// image another request is making. A first request's fetch is held, so the
// image is being made, and a second request for the same image waits for it
// until its client goes away. The second request must be logged once at info
// level with its request ID and the source host and path. Nothing may be
// logged at error level.
func TestImageRoutes_ClientGoneWhileWaitingAnswers499(t *testing.T) {
t.Parallel()
const requestID = "test-request-id"
for route, imageURL := range photoURLsByRoute() {
t.Run(route, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
fetcher := heldFetcher{started: make(chan struct{}), release: make(chan struct{})}
h, routes := newRoutesFetchingThrough(t, fetcher,
slog.New(slog.NewJSONHandler(&logs, nil)))
target := imageURL(t, h)
firstDone := make(chan struct{})
go func() {
defer close(firstDone)
sendGet(t, routes, target)
}()
<-fetcher.started
ctx, cancel := context.WithCancel(context.WithValue(t.Context(),
middleware.RequestIDKey, requestID))
rec := httptest.NewRecorder()
secondDone := make(chan struct{})
go func() {
defer close(secondDone)
routes.ServeHTTP(rec, httptest.NewRequestWithContext(
ctx, http.MethodGet, target, nil))
}()
// Give the second request time to start waiting for the first
// request's image. The fetch is still held, so the second request
// can end only because its client went away
time.Sleep(100 * time.Millisecond)
cancel()
<-secondDone
close(fetcher.release)
<-firstDone
if rec.Code != 499 {
t.Errorf("status = %d, want 499", rec.Code)
}
var clientGoneLines int
for _, line := range readLogLines(t, &logs) {
if line["level"] == slog.LevelError.String() {
t.Errorf("logged at error level: %v", line["msg"])
}
if line["level"] == slog.LevelInfo.String() &&
line["msg"] == "client went away before the image was ready" &&
line["request_id"] == requestID &&
line["host"] == signedHost &&
line["path"] == photoPath {
clientGoneLines++
}
}
if clientGoneLines != 1 {
t.Errorf("info line naming the request logged %d times, want 1",
clientGoneLines)
}
})
}
}
// TestImageRoutes_CanceledWithLiveRequestAnswers500 checks that both image
// routes answer 500, logged as "failed to get image" at error level, for an
// image service error that wraps context.Canceled while the request's own
// context is live. Its client has not gone away, so the cancellation is
// pixa's own failure.
func TestImageRoutes_CanceledWithLiveRequestAnswers500(t *testing.T) {
t.Parallel()
for route, imageURL := range photoURLsByRoute() {
t.Run(route, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h, routes := newRoutesFetchingThrough(t,
failingFetcher{err: context.Canceled},
slog.New(slog.NewJSONHandler(&logs, nil)))
rec := sendGet(t, routes, imageURL(t, h))
if rec.Code != http.StatusInternalServerError {
t.Errorf("status = %d, want %d",
rec.Code, http.StatusInternalServerError)
}
if !slices.ContainsFunc(readLogLines(t, &logs),
func(line map[string]any) bool {
return line["level"] == slog.LevelError.String() &&
line["msg"] == "failed to get image"
}) {
t.Error(`no "failed to get image" line logged at error level`)
}
})
}
}
+14 -108
View File
@@ -1,43 +1,23 @@
package handlers
import (
"bytes"
"errors"
"html/template"
"log/slog"
"net/http"
"net/url"
"github.com/go-chi/chi/v5/middleware"
"github.com/gorilla/csrf"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/templates"
)
// The causes the form-refused page names, each with what to do about it.
const (
causePlainHTTP = "This page was opened over plain HTTP while debug is " +
"off. Open it over HTTPS, or turn debug on to use plain HTTP on " +
"your own machine."
causeHTTPSWithDebug = "This page was opened over HTTPS while debug is " +
"on. Turn debug off, or open the page over plain HTTP."
causeOtherHost = "The form was sent from another host than the one in " +
"the request's Host header. A reverse proxy in front of pixa must " +
"pass the Host header on unchanged."
causeNoReferer = "The browser sent neither an Origin nor a Referer " +
"header, and over HTTPS pixa needs one to check where the form " +
"came from. Let the browser send the Referer header to this site."
causeStaleToken = "The form token is missing or out of date. Reload " +
"the page and submit the form again."
)
// csrfKeySalt provides domain separation for the CSRF authentication key,
// derived from the signing key so tokens survive restarts without extra
// configuration and never reuse the session or encrypted-URL key material.
const csrfKeySalt = "pixa-csrf-v1"
// newCSRFProtect builds the CSRF-protection middleware for the
// state-mutating HTML form routes. The token lives in its own cookie,
// independent of the session cookie, so it also protects the login POST
// where no session exists yet (login CSRF). key signs that cookie. A
// refused form is answered by the handler that handleFormRefused returns.
// log gets a warning with the library's reason, such as "origin invalid".
// where no session exists yet (login CSRF).
//
// When plaintext is true (local HTTP development), requests are marked
// plaintext so the library neither demands an https Referer nor sets a
@@ -45,18 +25,22 @@ const (
// service runs behind a TLS-terminating proxy, so plaintext is false and
// the library enforces its https Referer origin check.
func newCSRFProtect(
key [seal.KeySize]byte, plaintext bool, log *slog.Logger,
) func(http.Handler) http.Handler {
signingKey string, plaintext bool,
) (func(http.Handler) http.Handler, error) {
key, err := seal.DeriveKey([]byte(signingKey), csrfKeySalt)
if err != nil {
return nil, err
}
protect := csrf.Protect(
key[:],
csrf.Path("/"),
csrf.Secure(!plaintext),
csrf.SameSite(csrf.SameSiteStrictMode),
csrf.ErrorHandler(handleFormRefused(plaintext, log)),
)
if !plaintext {
return protect
return protect, nil
}
return func(next http.Handler) http.Handler {
@@ -65,85 +49,7 @@ func newCSRFProtect(
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
protected.ServeHTTP(w, csrf.PlaintextHTTPRequest(r))
})
}
}
// handleFormRefused returns the handler the CSRF middleware calls when it
// refuses a request. It logs the library's reason with the request ID and
// answers 403 with a page naming the cause and what to do. A reason that is
// not a refusal, such as a failure to make a token, answers 500.
func handleFormRefused(plaintext bool, log *slog.Logger) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
reason := csrf.FailureReason(r)
requestID := middleware.GetReqID(r.Context())
cause := formRefusedCause(r, reason, plaintext)
if cause == "" {
log.Error("CSRF middleware failed",
"request_id", requestID, "error", reason)
http.Error(w, "Internal server error",
http.StatusInternalServerError)
return
}
log.Warn("form refused", "request_id", requestID, "reason", reason)
var page bytes.Buffer
err := templates.Render(&page, "form_refused.html",
struct{ Cause string }{Cause: cause})
if err != nil {
log.Error("failed to render form refused template", "error", err)
http.Error(w, "Internal server error",
http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(http.StatusForbidden)
_, err = page.WriteTo(w)
if err != nil {
log.Error("failed to write form refused page", "error", err)
}
})
}
// formRefusedCause returns the cause the form-refused page names for the
// CSRF middleware's reason, or "" when the reason is not a refusal.
func formRefusedCause(r *http.Request, reason error, plaintext bool) string {
switch {
case errors.Is(reason, csrf.ErrNoToken), errors.Is(reason, csrf.ErrBadToken):
return causeStaleToken
case errors.Is(reason, csrf.ErrNoReferer):
return causeNoReferer
case errors.Is(reason, csrf.ErrBadOrigin):
return pageAddressCause(r.Header.Get("Origin"), r.Host, plaintext)
case errors.Is(reason, csrf.ErrBadReferer):
return pageAddressCause(r.Referer(), r.Host, plaintext)
default:
return ""
}
}
// pageAddressCause returns the cause for a form whose page address, the
// Origin or Referer header, the CSRF middleware refused. The middleware
// expects the request's own host, with https when plaintext is false and
// http when it is true. So on the request's own host, only the scheme was
// wrong.
func pageAddressCause(pageAddress, host string, plaintext bool) string {
page, err := url.Parse(pageAddress)
if err != nil || page.Host != host {
return causeOtherHost
}
if plaintext {
return causeHTTPSWithDebug
}
return causePlainHTTP
}, nil
}
// CSRF returns the CSRF-protection middleware for the login and generator
@@ -1,205 +0,0 @@
package handlers
import (
"bytes"
"context"
"encoding/json"
"html/template"
"log/slog"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/go-chi/chi/v5/middleware"
"github.com/gorilla/csrf"
"sneak.berlin/go/pixa/internal/seal"
)
// refusedFormRequestID is the request ID of the requests postRefusedForm
// sends, which the warning for a refused form must carry.
const refusedFormRequestID = "test-request-id"
// TestFormRefusedNamesCause verifies that a form the CSRF middleware refuses
// answers 403 with a page naming the cause, and logs a warning with the
// request ID and the middleware's reason.
func TestFormRefusedNamesCause(t *testing.T) {
t.Parallel()
tests := []struct {
name string
debug bool
origin string
referer string
wantCause string
wantReason error
}{
{
name: "plain HTTP origin with debug off",
origin: "http://example.com",
wantCause: causePlainHTTP,
wantReason: csrf.ErrBadOrigin,
},
{
name: "plain HTTP referer with debug off",
referer: "http://example.com/",
wantCause: causePlainHTTP,
wantReason: csrf.ErrBadReferer,
},
{
name: "origin on another host",
origin: "https://proxy.example.net",
wantCause: causeOtherHost,
wantReason: csrf.ErrBadOrigin,
},
{
name: "referer on another host",
referer: "https://proxy.example.net/",
wantCause: causeOtherHost,
wantReason: csrf.ErrBadReferer,
},
{
name: "HTTPS origin with debug on",
debug: true,
origin: "https://example.com",
wantCause: causeHTTPSWithDebug,
wantReason: csrf.ErrBadOrigin,
},
{
name: "no origin or referer with debug off",
wantCause: causeNoReferer,
wantReason: csrf.ErrNoReferer,
},
{
name: "missing token",
debug: true,
origin: "http://example.com",
wantCause: causeStaleToken,
wantReason: csrf.ErrNoToken,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
rec, logLine := postRefusedForm(t, tt.debug, tt.origin, tt.referer,
nil, "")
checkFormRefused(t, rec, logLine, tt.wantCause, tt.wantReason)
})
}
}
// TestFormRefusedNamesCauseForForeignToken verifies that a form whose token
// does not match its CSRF cookie answers 403 with a page naming
// causeStaleToken, and logs a warning with the request ID and the
// middleware's reason.
func TestFormRefusedNamesCauseForForeignToken(t *testing.T) {
t.Parallel()
// The router and postRefusedForm derive their CSRF key from
// testSigningKey, so cookiesA is valid there and only tokenB is wrong.
_, router := newCSRFTestRouter(t)
cookiesA, _ := csrfCredentials(t, router, nil)
_, tokenB := csrfCredentials(t, router, nil)
rec, logLine := postRefusedForm(t, true, "http://example.com", "",
cookiesA, tokenB)
checkFormRefused(t, rec, logLine, causeStaleToken, csrf.ErrBadToken)
}
// postRefusedForm sends a POST with the given cookies to the CSRF
// middleware built for debug. It adds the form token and the Origin and
// Referer headers when they are not empty. It returns the response and the
// one JSON log line the middleware wrote.
func postRefusedForm(
t *testing.T, debug bool, origin, referer string,
cookies []*http.Cookie, token string,
) (*httptest.ResponseRecorder, map[string]any) {
t.Helper()
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
var logs bytes.Buffer
log := slog.New(slog.NewJSONHandler(&logs, nil))
reached := http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) {
t.Error("the refused form reached the handler behind the middleware")
})
srv := newCSRFProtect(keys.CSRFKey, debug, log)(reached)
form := url.Values{loginKeyField: {"x"}}
if token != "" {
form.Set(csrfTokenField, token)
}
ctx := context.WithValue(context.Background(),
middleware.RequestIDKey, refusedFormRequestID)
req := httptest.NewRequestWithContext(ctx, http.MethodPost,
"http://example.com/", strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
for _, c := range cookies {
req.AddCookie(c)
}
if origin != "" {
req.Header.Set("Origin", origin)
}
if referer != "" {
req.Header.Set("Referer", referer)
}
rec := httptest.NewRecorder()
srv.ServeHTTP(rec, req)
var logLine map[string]any
err = json.Unmarshal(logs.Bytes(), &logLine)
if err != nil {
t.Fatalf("decoding log line %q: %v", logs.String(), err)
}
return rec, logLine
}
// checkFormRefused checks that rec is a 403 page naming wantCause and that
// logLine is a warning carrying the request ID and wantReason.
func checkFormRefused(
t *testing.T, rec *httptest.ResponseRecorder, logLine map[string]any,
wantCause string, wantReason error,
) {
t.Helper()
if rec.Code != http.StatusForbidden {
t.Errorf("status = %d, want %d", rec.Code, http.StatusForbidden)
}
body := rec.Body.String()
t.Logf("body: %s", body)
if !strings.Contains(body, template.HTMLEscapeString(wantCause)) {
t.Errorf("page does not name the cause %q", wantCause)
}
t.Logf("log line: %v", logLine)
if logLine["level"] != "WARN" || logLine["msg"] != "form refused" {
t.Errorf("log line = %v, want a WARN \"form refused\"", logLine)
}
if logLine["request_id"] != refusedFormRequestID {
t.Errorf("logged request_id = %v, want %q",
logLine["request_id"], refusedFormRequestID)
}
if logLine["reason"] != wantReason.Error() {
t.Errorf("logged reason = %v, want %q", logLine["reason"], wantReason)
}
}
+6 -10
View File
@@ -17,10 +17,8 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -39,7 +37,7 @@ func newRoutesFetchingThrough(
) (*Handlers, http.Handler) {
t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -48,7 +46,7 @@ func newRoutesFetchingThrough(
t.Fatalf("imgcache.NewCache() error = %v", err)
}
svc, err := imageproxy.New(&imageproxy.Config{
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
Cache: cache,
Fetcher: fetcher,
Processor: imageprocessor.New(imageprocessor.Params{
@@ -57,18 +55,16 @@ func newRoutesFetchingThrough(
}),
MaxResponseSize: 50 << 20,
Logger: log,
}, prometheus.NewRegistry())
})
if err != nil {
t.Fatalf("imageproxy.New() error = %v", err)
t.Fatalf("imgcache.NewService() error = %v", err)
}
keys, err := seal.DeriveKeys(testSigningKey)
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{
log: log,
imgSvc: svc,
+4 -4
View File
@@ -65,20 +65,20 @@ func formatForAccept(accept string) (imagerequest.Format, error) {
return imagerequest.FormatJPEG, nil
}
if qualities[imagerequest.FormatJXL.MIMEType()] > 0 {
if qualities["image/jxl"] > 0 {
return imagerequest.FormatJXL, nil
}
if qualities[imagerequest.FormatAVIF.MIMEType()] > 0 {
if qualities["image/avif"] > 0 {
return imagerequest.FormatAVIF, nil
}
if qualities[imagerequest.FormatWebP.MIMEType()] > 0 {
if qualities["image/webp"] > 0 {
return imagerequest.FormatWebP, nil
}
// For JPEG, the most specific entry the header has decides
quality, named := qualities[imagerequest.FormatJPEG.MIMEType()]
quality, named := qualities["image/jpeg"]
if !named {
quality, named = qualities["image/*"]
}
@@ -75,9 +75,9 @@ func TestFormatForAccept(t *testing.T) {
// autoPhotoURLs returns a signed /v1/image/ URL and an encrypted /v1/e/ URL,
// both valid for a minute, for the JPEG at photoPath on signedHost at 50x50 in
// the format auto. The first is signed with testSigningKey, the key the
// handlers that newSignedHostServer builds check signatures with. The second
// is made with h's generator.
// the format auto. The first is signed with testSigningKey, the key of the
// image service newSignedHostServer builds, and the second is made with h's
// generator.
func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) {
t.Helper()
+11 -10
View File
@@ -13,9 +13,8 @@ import (
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/healthcheck"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/logger"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -27,9 +26,8 @@ type Params struct {
Logger *logger.Logger
Healthcheck *healthcheck.Healthcheck
Config *config.Config
Keys seal.Keys
ImageService *imageproxy.Proxy
ImageService *imgcache.Service
SessionManager *session.Manager
EncryptedURLGenerator *encurl.Generator
}
@@ -39,7 +37,7 @@ type Handlers struct {
log *slog.Logger
hc *healthcheck.Healthcheck
config *config.Config
imgSvc *imageproxy.Proxy
imgSvc *imgcache.Service
sessMgr *session.Manager
encGen *encurl.Generator
csrfProtect func(http.Handler) http.Handler
@@ -55,21 +53,24 @@ type Handlers struct {
}
// New creates a new Handlers instance.
func New(_ fx.Lifecycle, params Params) *Handlers {
log := params.Logger.Get()
func New(_ fx.Lifecycle, params Params) (*Handlers, error) {
csrfProtect, err := newCSRFProtect(params.Config.SigningKey, params.Config.Debug)
if err != nil {
return nil, err
}
return &Handlers{
log: log,
log: params.Logger.Get(),
hc: params.Healthcheck,
config: params.Config,
imgSvc: params.ImageService,
sessMgr: params.SessionManager,
encGen: params.EncryptedURLGenerator,
csrfProtect: newCSRFProtect(params.Keys.CSRFKey, params.Config.Debug, log),
csrfProtect: csrfProtect,
signer: signature.New(params.Config.SigningKey),
allowlist: allowlist.New(params.Config.AllowlistHosts),
refererBlocklist: allowlist.New(params.Config.RefererBlocklist),
}
}, nil
}
// WaitForProcessing waits until no image is being processed, or until ctx
+5 -90
View File
@@ -23,7 +23,6 @@ import (
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/signature"
@@ -50,7 +49,7 @@ func setupTestHandler(t *testing.T) *testFixtures {
tmpDir := t.TempDir()
db := setupTestDB(t)
cache, err := imgcache.NewCache(db, imgcache.Config{
cache, err := imgcache.NewCache(db, imgcache.CacheConfig{
StateDir: tmpDir,
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -59,7 +58,7 @@ func setupTestHandler(t *testing.T) *testFixtures {
t.Fatalf("failed to create cache: %v", err)
}
svc, err := imageproxy.New(&imageproxy.Config{
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
Cache: cache,
Fetcher: newMockFetcher(mockFS),
Processor: imageprocessor.New(imageprocessor.Params{
@@ -67,9 +66,9 @@ func setupTestHandler(t *testing.T) *testFixtures {
MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
}, prometheus.NewRegistry())
})
if err != nil {
t.Fatalf("failed to create proxy: %v", err)
t.Fatalf("failed to create service: %v", err)
}
h := &Handlers{
@@ -131,7 +130,7 @@ func newMockFetcher(fs fs.FS) *mockFetcher {
}
func (f *mockFetcher) Fetch(
_ context.Context, source imagerequest.Source, _, _ string,
_ context.Context, source imagerequest.Source,
) (*httpfetcher.FetchResult, error) {
path := source.Host + source.Path
@@ -257,90 +256,6 @@ func TestHandleImage_ConditionalRequest_IfNoneMatch_DifferentETag(t *testing.T)
}
}
// TestHandleImage_ConditionalRequest_EarlierETagGetsChangedImage serves an
// image, changes it upstream, and requests it again with the ETag first
// served, once the cache TTL has passed. The changed image must be served,
// not 304 Not Modified.
func TestHandleImage_ConditionalRequest_EarlierETagGetsChangedImage(t *testing.T) {
t.Parallel()
const host = "goodhost.example.com"
upstream := fstest.MapFS{
host + photoPath: &fstest.MapFile{
Data: generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255}),
},
}
// A cache TTL of one nanosecond has passed by the second request
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute,
CacheTTL: time.Nanosecond,
MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err)
}
svc, err := imageproxy.New(&imageproxy.Config{
Cache: cache,
Fetcher: newMockFetcher(upstream),
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
Logger: slog.New(slog.DiscardHandler),
}, prometheus.NewRegistry())
if err != nil {
t.Fatalf("imageproxy.New() error = %v", err)
}
h := &Handlers{
imgSvc: svc,
log: slog.New(slog.DiscardHandler),
signer: signature.New(testSigningKey),
allowlist: allowlist.New([]string{host}),
}
r := chi.NewRouter()
r.Get("/v1/image/*", h.HandleImage())
target := "/v1/image/" + host + photoPath + "/50x50.jpeg"
first := httptest.NewRecorder()
r.ServeHTTP(first,
httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil))
etag := first.Header().Get("ETag")
if first.Code != http.StatusOK || etag == "" {
t.Fatalf("first request: status = %d, ETag = %q, want %d and an ETag",
first.Code, etag, http.StatusOK)
}
upstream[host+photoPath] = &fstest.MapFile{
Data: generateTestJPEG(t, 100, 100, color.RGBA{0, 0, 255, 255}),
}
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil)
req.Header.Set("If-None-Match", etag)
second := httptest.NewRecorder()
r.ServeHTTP(second, req)
t.Logf("first ETag %s; with it after the change: status %d, ETag %s, %d bytes",
etag, second.Code, second.Header().Get("ETag"), second.Body.Len())
if second.Code != http.StatusOK {
t.Fatalf("status = %d, want %d with the changed image",
second.Code, http.StatusOK)
}
if bytes.Equal(second.Body.Bytes(), first.Body.Bytes()) {
t.Error("the image served is the one made before the change")
}
}
func TestHandleImage_ETagHeader(t *testing.T) {
t.Parallel()
+89 -166
View File
@@ -1,7 +1,6 @@
package handlers
import (
"context"
"errors"
"fmt"
"io"
@@ -15,17 +14,11 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic"
)
// statusClientClosedRequest is the status nginx calls "client closed
// request". The image routes answer it when the client goes away before its
// image is ready, unless its request is the one making the image. Nobody reads
// it; it is for logs and metrics.
const statusClientClosedRequest = 499
// HandleImage handles the main image proxy route:
// /v1/image/<host>/<path>/<width>x<height>.<format>, or with no format
// /v1/image/<host>/<path>/<width>x<height>
@@ -53,7 +46,30 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
return
}
s.serveImage(w, r, req, expires)
// The signature covers the format auto, not the format chosen
format, ok := s.chooseAutoFormat(w, r, req.Transform.Format)
if !ok {
return
}
req.Transform.Format = format
// Get cache key for logging
cacheKey := imagerequest.CacheKey(req)
// Get the image (from cache or fetch/process)
startTime := time.Now()
resp, err := s.imgSvc.Get(r.Context(), req)
if err != nil {
s.respondImageError(w, req, err)
return
}
defer func() { _ = resp.Content.Close() }()
s.writeImageResponse(w, r, req, expires, resp, cacheKey, startTime)
}
}
@@ -197,8 +213,8 @@ func (s *Handlers) validateSignature(
}
// parseExpires reads the exp query parameter, a Unix time in seconds. An exp
// missing from the URL gives the zero time, which the signature check refuses
// as a missing exp. An exp in the URL that is not a whole number, an empty
// missing from the URL gives the zero time, which the signature check takes
// as no expiration. An exp in the URL that is not a whole number, an empty
// one included, is an error naming exp and the value.
func parseExpires(query url.Values) (time.Time, error) {
if !query.Has("exp") {
@@ -226,101 +242,75 @@ func (s *Handlers) HandleTooManyRequests() http.HandlerFunc {
}
}
// respondImageError answers err, the image service's error for req, on both
// image routes. It logs a 500 at error level, as pixa's own failure. It logs
// the other answers at warn level, since the client, the upstream or a full
// limit caused them. The 415 for a source refused for its first bytes is not
// logged here, because the image service logs it.
// respondImageError maps image retrieval errors to HTTP responses.
func (s *Handlers) respondImageError(
w http.ResponseWriter, req *imagerequest.Request, err error,
) {
// The image service has logged this as a refused source
if isFirstBytesMismatch(err) {
// An upstream body whose first bytes are not those of the format its
// Content-Type names is the upstream's error, not pixa's: the image
// service has logged it as a refused source.
if errors.Is(err, magic.ErrMagicByteMismatch) ||
errors.Is(err, magic.ErrUnknownFormat) ||
errors.Is(err, magic.ErrNotEnoughData) {
s.respondError(w, "unsupported source format",
http.StatusUnsupportedMediaType)
return
}
var (
status int
message string
)
switch {
case errors.Is(err, httpfetcher.ErrSSRFBlocked):
status, message = http.StatusForbidden, "forbidden"
case errors.Is(err, httpfetcher.ErrUpstreamError),
errors.Is(err, httpfetcher.ErrInvalidHost),
errors.Is(err, httpfetcher.ErrUpstreamRequestFailed):
status, message = http.StatusBadGateway, "upstream error"
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
status, message = http.StatusGatewayTimeout, "upstream timeout"
case errors.Is(err, httpfetcher.ErrTooManyConnections),
errors.Is(err, imageprocessor.ErrTooManyImages),
errors.Is(err, imageproxy.ErrProcessingTimedOut):
status = http.StatusServiceUnavailable
message = "server busy, try again later"
case errors.Is(err, httpfetcher.ErrInvalidContentType),
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat):
status = http.StatusUnsupportedMediaType
message = "unsupported source format"
case errors.Is(err, imageprocessor.ErrSVGTooThin):
status = http.StatusUnsupportedMediaType
message = fmt.Sprintf("SVG source too thin to render: one side is "+
"more than %d times the other", imageprocessor.MaxInputDimension)
case errors.Is(err, imageprocessor.ErrSVGRenderFailed):
status = http.StatusUnsupportedMediaType
message = fmt.Sprintf("SVG source could not be rendered: it is not "+
"valid, or it takes more than %d seconds or %d GiB of memory to "+
"render", int(imageprocessor.SVGRenderTimeout.Seconds()),
imageprocessor.MaxSVGRenderMemory/bytesPerGiB)
case errors.Is(err, imageprocessor.ErrOutputTooLarge):
status = http.StatusBadRequest
message = fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d",
imageprocessor.MaxOutputDimension)
case errors.Is(err, imageprocessor.ErrSVGNotOriginalSize):
status = http.StatusBadRequest
message = "an SVG source is returned as SVG only at the orig size; " +
"ask for another format to resize it"
case errors.Is(err, imageprocessor.ErrSVGTooLargeForOrigSize):
status = http.StatusBadRequest
message = fmt.Sprintf("SVG source above %d on a side: give a size "+
"to render it at", imageprocessor.MaxOutputDimension)
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
status = http.StatusBadRequest
message = fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels)
default:
s.log.Error("failed to get image",
"host", req.Source.Host,
"path", req.Source.Path,
"error", err,
)
s.respondError(w, "internal error", http.StatusInternalServerError)
return
}
s.log.Warn("failed to get image",
s.log.Error("failed to get image",
"host", req.Source.Host,
"path", req.Source.Path,
"error", err,
)
s.respondError(w, message, status)
}
// bytesPerGiB is the number of bytes in a GiB.
const bytesPerGiB = 1 << 30
// Check for specific error types
if errors.Is(err, httpfetcher.ErrSSRFBlocked) {
s.respondError(w, "forbidden", http.StatusForbidden)
// isFirstBytesMismatch reports whether err refuses an upstream body whose
// first bytes are not those of the format its Content-Type names. That is the
// upstream's error, not pixa's.
func isFirstBytesMismatch(err error) bool {
return errors.Is(err, magic.ErrMagicByteMismatch) ||
errors.Is(err, magic.ErrUnknownFormat) ||
errors.Is(err, magic.ErrNotEnoughData)
return
}
if errors.Is(err, httpfetcher.ErrUpstreamError) {
s.respondError(w, "upstream error", http.StatusBadGateway)
return
}
if errors.Is(err, httpfetcher.ErrTooManyConnections) ||
errors.Is(err, imageprocessor.ErrTooManyImages) ||
errors.Is(err, imgcache.ErrProcessingTimedOut) {
s.respondError(w, "server busy, try again later",
http.StatusServiceUnavailable)
return
}
if errors.Is(err, httpfetcher.ErrInvalidContentType) ||
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat) {
s.respondError(w, "unsupported source format",
http.StatusUnsupportedMediaType)
return
}
if errors.Is(err, imageprocessor.ErrOutputTooLarge) {
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d",
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
return
}
if errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge) {
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
http.StatusBadRequest)
return
}
s.respondError(w, "internal error", http.StatusInternalServerError)
}
// cacheControl returns the Cache-Control header for an image served through a
@@ -341,26 +331,6 @@ func cacheControl(expires time.Time) string {
return fmt.Sprintf("public, max-age=%d, immutable", int64(maxAge/time.Second))
}
// svgContentSecurityPolicyValue is the Content-Security-Policy header value
// of an SVG response, in place of the middleware's. A browser that opens the
// SVG applies its styles and draws its embedded data: images, but runs none
// of its scripts and loads nothing else. The styles need unsafe-inline, as an
// SVG keeps them inline, in style elements and attributes.
const svgContentSecurityPolicyValue = "default-src 'none'; " +
"style-src 'unsafe-inline'; " +
"img-src data:; " +
"frame-ancestors 'none'; " +
"sandbox"
// setSVGContentSecurityPolicy sets the Content-Security-Policy in header to
// svgContentSecurityPolicyValue when contentType is that of SVG. Any other
// response keeps the middleware's policy.
func setSVGContentSecurityPolicy(header http.Header, contentType string) {
if contentType == imagerequest.FormatSVG.MIMEType() {
header.Set("Content-Security-Policy", svgContentSecurityPolicyValue)
}
}
// refuseBlockedReferer answers 403 with a JSON error when the request's Referer
// names a host on referer_blocklist, and reports whether it answered. A request
// with no Referer, or one that does not parse as a URL with a host, is not
@@ -397,53 +367,16 @@ func notModified(w http.ResponseWriter, r *http.Request, etag string) bool {
return true
}
// serveImage serves the image that req asks for. Both image routes call it
// once they have the request and its URL's expiry, which ends the
// Cache-Control max-age. It chooses the format for the format auto, answers
// If-None-Match with 304 and HEAD with the headers only. It answers 499 when
// the image service stopped because the client went away, and any other error
// of the image service with respondImageError.
func (s *Handlers) serveImage(
// writeImageResponse writes headers and streams the image content,
// handling conditional and HEAD requests. expires is the URL's expiry, which
// ends the Cache-Control max-age.
func (s *Handlers) writeImageResponse(
w http.ResponseWriter, r *http.Request,
req *imagerequest.Request, expires time.Time,
req *imagerequest.Request, expires time.Time, resp *imgcache.ImageResponse,
cacheKey imagerequest.VariantKey, startTime time.Time,
) {
// The signature or the token covers the format auto, not the format chosen
format, ok := s.chooseAutoFormat(w, r, req.Transform.Format)
if !ok {
return
}
req.Transform.Format = format
startTime := time.Now()
resp, err := s.imgSvc.Get(r.Context(), req)
// The client went away before its image was ready, which is not pixa's
// failure. A cancellation while the request's context is live is pixa's,
// and respondImageError answers it 500.
if errors.Is(err, context.Canceled) && r.Context().Err() != nil {
s.log.Info("client went away before the image was ready",
"request_id", middleware.GetReqID(r.Context()),
"host", req.Source.Host,
"path", req.Source.Path,
)
w.WriteHeader(statusClientClosedRequest)
return
}
if err != nil {
s.respondImageError(w, req, err)
return
}
defer func() { _ = resp.Content.Close() }()
// Set response headers
w.Header().Set("Content-Type", resp.ContentType)
setSVGContentSecurityPolicy(w.Header(), resp.ContentType)
if resp.ContentLength > 0 {
w.Header().Set("Content-Length", strconv.FormatInt(resp.ContentLength, 10))
@@ -467,28 +400,18 @@ func (s *Handlers) serveImage(
// Stream the response
w.WriteHeader(http.StatusOK)
// A failed write is most often a client that has gone away, not pixa's
// failure, so it is logged at warn level
servedBytes, err := io.Copy(w, resp.Content)
if err != nil {
s.log.Warn("failed to write response",
"request_id", middleware.GetReqID(r.Context()),
"host", req.Source.Host,
"path", req.Source.Path,
"served_bytes", servedBytes,
s.log.Error("failed to write response",
"error", err,
)
return
}
// Log cache status and timing after serving
duration := time.Since(startTime)
s.log.Info("image served",
"request_id", middleware.GetReqID(r.Context()),
"cache_key", imagerequest.CacheKey(req),
"host", req.Source.Host,
"path", req.Source.Path,
"cache_key", cacheKey,
"cache_status", resp.CacheStatus,
"duration_ms", duration.Milliseconds(),
"format", req.Transform.Format,
@@ -17,10 +17,8 @@ import (
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -36,7 +34,7 @@ func newSignedHostServer(
) (*Handlers, http.Handler) {
t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -47,7 +45,7 @@ func newSignedHostServer(
jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
svc, err := imageproxy.New(&imageproxy.Config{
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
Cache: cache,
Fetcher: newMockFetcher(fstest.MapFS{
signedHost + photoPath: &fstest.MapFile{Data: jpegData},
@@ -58,18 +56,16 @@ func newSignedHostServer(
}),
MaxResponseSize: 50 << 20,
Logger: log,
}, prometheus.NewRegistry())
})
if err != nil {
t.Fatalf("imageproxy.New() error = %v", err)
t.Fatalf("imgcache.NewService() error = %v", err)
}
keys, err := seal.DeriveKeys(testSigningKey)
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{
log: log,
imgSvc: svc,
@@ -185,13 +181,13 @@ func TestHandleImageEnc_MaxAge(t *testing.T) {
tests := []struct {
name string
ttl time.Duration
expires time.Time
wantAtLeast int
wantAtMost int
}{
{"60 second TTL", time.Minute, 50, 60},
{"two-year TTL", 2 * 365 * 24 * time.Hour, 31536000, 31536000},
{"no TTL", 0, 31536000, 31536000},
{"60 second TTL", time.Now().Add(time.Minute), 50, 60},
{"two-year TTL", time.Now().Add(2 * 365 * 24 * time.Hour), 31536000, 31536000},
{"no TTL", time.Time{}, 31536000, 31536000},
}
for _, tt := range tests {
@@ -200,14 +196,6 @@ func TestHandleImageEnc_MaxAge(t *testing.T) {
h, srv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
// The expiry is set here, not in the table: a parallel subtest
// can wait behind other tests for more than the 10 seconds the
// 60 second case allows.
var expires time.Time
if tt.ttl != 0 {
expires = time.Now().Add(tt.ttl)
}
token, err := h.encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{
Host: signedHost,
@@ -217,7 +205,7 @@ func TestHandleImageEnc_MaxAge(t *testing.T) {
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
}, expires)
}, tt.expires)
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
+5 -184
View File
@@ -1,29 +1,22 @@
package handlers
import (
"bytes"
"context"
"encoding/json"
"fmt"
"image/color"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"slices"
"strings"
"syscall"
"testing"
"testing/fstest"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/signature"
@@ -38,7 +31,7 @@ const allowlistedHost = "allowed.example.com"
func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -47,7 +40,7 @@ func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
t.Fatalf("failed to create cache: %v", err)
}
svc, err := imageproxy.New(&imageproxy.Config{
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
Cache: cache,
Fetcher: fetcher,
Processor: imageprocessor.New(imageprocessor.Params{
@@ -55,9 +48,9 @@ func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
}, prometheus.NewRegistry())
})
if err != nil {
t.Fatalf("failed to create proxy: %v", err)
t.Fatalf("failed to create service: %v", err)
}
h := &Handlers{
@@ -74,7 +67,7 @@ func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
}
// newPhotoFetcher returns a mock fetcher that serves a JPEG at photoPath on
// each of hosts, and answers any other source with an upstream error.
// each of hosts, and answers any other URL with an upstream error.
func newPhotoFetcher(t *testing.T, hosts ...string) *httpfetcher.MockFetcher {
t.Helper()
@@ -421,175 +414,3 @@ func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
})
}
}
// failingFetcher answers every fetch with its error.
type failingFetcher struct {
err error
}
// Fetch returns f's error.
func (f failingFetcher) Fetch(
context.Context, imagerequest.Source, string, string,
) (*httpfetcher.FetchResult, error) {
return nil, f.err
}
// photoURLsByRoute returns, for each image route, the function that makes the
// URL of the JPEG at photoPath on signedHost on that route.
func photoURLsByRoute() map[string]func(*testing.T, *Handlers) string {
return map[string]func(*testing.T, *Handlers) string{
"/v1/image/": signedPhotoURL,
"/v1/e/": encPhotoURL,
}
}
// TestImageRoutes_SameErrorAnswers checks that both image routes answer each
// upstream fetch error below with the same status. They log it as "failed to
// get image" at the same level: error for the 500 and warn for the others.
// Each request gets routes of its own, so no request finds another's failed
// fetch in the negative cache.
func TestImageRoutes_SameErrorAnswers(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
wantStatus int
wantLevel slog.Level
}{
{"blocked upstream address", httpfetcher.ErrSSRFBlocked,
http.StatusForbidden, slog.LevelWarn},
{"upstream error status",
fmt.Errorf("%w: status 404", httpfetcher.ErrUpstreamError),
http.StatusBadGateway, slog.LevelWarn},
{"upstream host with no address",
fmt.Errorf("%w: cdn.example.com", httpfetcher.ErrInvalidHost),
http.StatusBadGateway, slog.LevelWarn},
{"connection refused",
fmt.Errorf("%w: %w", httpfetcher.ErrUpstreamRequestFailed,
syscall.ECONNREFUSED),
http.StatusBadGateway, slog.LevelWarn},
{"no response headers within the fetch timeout",
httpfetcher.ErrUpstreamTimeout, http.StatusGatewayTimeout,
slog.LevelWarn},
{"no free upstream connection", httpfetcher.ErrTooManyConnections,
http.StatusServiceUnavailable, slog.LevelWarn},
{"Content-Type not a source format",
fmt.Errorf("%w: image/tiff", httpfetcher.ErrInvalidContentType),
http.StatusUnsupportedMediaType, slog.LevelWarn},
{"any other failure", io.ErrUnexpectedEOF,
http.StatusInternalServerError, slog.LevelError},
}
for _, tt := range tests {
for route, imageURL := range photoURLsByRoute() {
t.Run(route+" "+tt.name, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h, routes := newRoutesFetchingThrough(t,
failingFetcher{err: tt.err},
slog.New(slog.NewJSONHandler(&logs, nil)))
rec := sendGet(t, routes, imageURL(t, h))
if rec.Code != tt.wantStatus {
t.Errorf("status = %d, want %d: %s",
rec.Code, tt.wantStatus, rec.Body)
}
levels := loggedLevels(t, &logs)
if !slices.Contains(levels[tt.wantLevel.String()],
"failed to get image") {
t.Errorf(`no "failed to get image" line logged at level %s`,
tt.wantLevel)
}
if tt.wantLevel != slog.LevelError &&
len(levels[slog.LevelError.String()]) != 0 {
t.Errorf("logged at error level: %q",
levels[slog.LevelError.String()])
}
})
}
}
}
// failingResponseWriter is a response writer whose Write always fails, as when
// the client has gone away.
type failingResponseWriter struct {
*httptest.ResponseRecorder
}
// Write writes nothing and returns io.ErrClosedPipe.
func (failingResponseWriter) Write([]byte) (int, error) {
return 0, io.ErrClosedPipe
}
// TestImageRoutes_FailedWriteLoggedOnce checks that both image routes log a
// failed write of the image once, at warn level, with the request's ID, the
// source host and path, the bytes served and the error. Neither may also log
// the image as served.
func TestImageRoutes_FailedWriteLoggedOnce(t *testing.T) {
t.Parallel()
const requestID = "test-request-id"
for route, imageURL := range photoURLsByRoute() {
t.Run(route, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h, srv := newSignedHostServer(t,
slog.New(slog.NewJSONHandler(&logs, nil)))
ctx := context.WithValue(t.Context(),
middleware.RequestIDKey, requestID)
srv.ServeHTTP(failingResponseWriter{httptest.NewRecorder()},
httptest.NewRequestWithContext(
ctx, http.MethodGet, imageURL(t, h), nil))
t.Logf("logged:\n%s", logs.String())
var failedWrites []map[string]any
dec := json.NewDecoder(&logs)
for dec.More() {
var line map[string]any
err := dec.Decode(&line)
if err != nil {
t.Fatalf("decoding log line: %v", err)
}
switch line["msg"] {
case "image served":
t.Error(`"image served" logged after a failed write`)
case "failed to write response":
failedWrites = append(failedWrites, line)
}
}
if len(failedWrites) != 1 {
t.Fatalf(`"failed to write response" logged %d times, want 1`,
len(failedWrites))
}
want := map[string]any{
"level": slog.LevelWarn.String(),
"request_id": requestID,
"host": signedHost,
"path": photoPath,
"served_bytes": float64(0),
"error": io.ErrClosedPipe.Error(),
}
for key, wantValue := range want {
if got := failedWrites[0][key]; got != wantValue {
t.Errorf("%s = %v, want %v", key, got, wantValue)
}
}
})
}
}
+118 -5
View File
@@ -2,13 +2,21 @@ package handlers
import (
"errors"
"fmt"
"io"
"net/http"
"strconv"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic"
)
// HandleImageEnc handles requests to /v1/e/{token}/* for encrypted
@@ -20,12 +28,83 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
return
}
ctx := r.Context()
start := time.Now()
req, expires, ok := s.parseImageEncRequest(w, r)
if !ok {
return
}
s.serveImage(w, r, req, expires)
format, ok := s.chooseAutoFormat(w, r, req.Transform.Format)
if !ok {
return
}
req.Transform.Format = format
// Log the request
s.log.Debug("encrypted image request",
"host", req.Source.Host,
"path", req.Source.Path,
"dimensions", fmt.Sprintf("%dx%d",
req.Transform.Size.Width, req.Transform.Size.Height),
"format", req.Transform.Format,
)
// Fetch and process the image (no signature validation
// needed - encrypted URL is trusted)
resp, err := s.imgSvc.Get(ctx, req)
if err != nil {
s.handleImageError(w, err)
return
}
defer func() { _ = resp.Content.Close() }()
// Set response headers
w.Header().Set("Content-Type", resp.ContentType)
if resp.ContentLength > 0 {
w.Header().Set("Content-Length", strconv.FormatInt(resp.ContentLength, 10))
}
// Cache headers: max-age ends at the URL's expiry
w.Header().Set("Cache-Control", cacheControl(expires))
w.Header().Set("X-Pixa-Cache", string(resp.CacheStatus))
if notModified(w, r, resp.ETag) {
return
}
// A HEAD request gets the headers only
if r.Method == http.MethodHead {
w.WriteHeader(http.StatusOK)
return
}
// Stream the response
written, err := io.Copy(w, resp.Content)
if err != nil {
s.log.Error("failed to write response", "error", err)
return
}
// Log completion
duration := time.Since(start)
s.log.Info("image served",
"request_id", middleware.GetReqID(ctx),
"cache_key", imagerequest.CacheKey(req),
"host", req.Source.Host,
"path", req.Source.Path,
"format", req.Transform.Format,
"cache_status", resp.CacheStatus,
"served_bytes", written,
"duration_ms", duration.Milliseconds(),
)
}
}
@@ -54,7 +133,7 @@ func (s *Handlers) parseImageEncRequest(
return nil, time.Time{}, false
}
s.log.Debug("invalid encrypted URL", "error", err)
s.log.Debug("failed to decrypt URL", "error", err)
s.respondError(w, "invalid encrypted URL", http.StatusBadRequest)
return nil, time.Time{}, false
@@ -62,9 +141,8 @@ func (s *Handlers) parseImageEncRequest(
// Apply the same dimension and fit-mode bounds as the plain image
// route: a sealed payload is trusted for its origin, not for staying
// within limits. This refuses an over-limit size before the source is
// fetched. An unknown fit mode is a 400 here rather than a 500 from the
// processor.
// within limits, so an over-limit size or unknown fit mode is a 400
// here rather than an out-of-memory or a 500 from the processor.
err = imagerequest.ValidateImageRequest(req)
if err != nil {
s.log.Debug("encrypted URL failed validation", "error", err)
@@ -76,3 +154,38 @@ func (s *Handlers) parseImageEncRequest(
return req, expires, true
}
// handleImageError converts image service errors to HTTP responses.
func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
switch {
case errors.Is(err, httpfetcher.ErrSSRFBlocked):
s.respondError(w, "forbidden", http.StatusForbidden)
case errors.Is(err, httpfetcher.ErrUpstreamError):
s.respondError(w, "upstream error", http.StatusBadGateway)
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
s.respondError(w, "upstream timeout", http.StatusGatewayTimeout)
case errors.Is(err, httpfetcher.ErrTooManyConnections),
errors.Is(err, imageprocessor.ErrTooManyImages),
errors.Is(err, imgcache.ErrProcessingTimedOut):
s.respondError(w, "server busy, try again later",
http.StatusServiceUnavailable)
case errors.Is(err, httpfetcher.ErrInvalidContentType),
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat),
errors.Is(err, magic.ErrMagicByteMismatch),
errors.Is(err, magic.ErrUnknownFormat),
errors.Is(err, magic.ErrNotEnoughData):
s.respondError(w, "unsupported source format",
http.StatusUnsupportedMediaType)
case errors.Is(err, imageprocessor.ErrOutputTooLarge):
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d",
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
http.StatusBadRequest)
default:
s.log.Error("image request failed", "error", err)
s.respondError(w, "internal error", http.StatusInternalServerError)
}
}
+2 -5
View File
@@ -12,7 +12,6 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
)
// newEncTestServer builds a router serving the encrypted-URL route with a
@@ -22,13 +21,11 @@ import (
func newEncTestServer(t *testing.T) (*encurl.Generator, http.Handler) {
t.Helper()
keys, err := seal.DeriveKeys(testSigningKey)
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{
log: slog.New(slog.DiscardHandler),
encGen: encGen,
@@ -10,7 +10,6 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
)
// requireServedPhoto requires that rec answers 200 with the JPEG at photoPath
@@ -81,13 +80,11 @@ func TestHandleImageEnc_RejectedToken(t *testing.T) {
t.Fatalf("Generate() error = %v", err)
}
keys, err := seal.DeriveKeys("another-signing-key-fedcba9876543210")
otherGen, err := encurl.NewGenerator("another-signing-key-fedcba9876543210")
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
otherGen := encurl.NewGenerator(keys.EncryptedURLKey)
otherKey, err := otherGen.Generate(&photo, time.Time{})
if err != nil {
t.Fatalf("Generate() error = %v", err)
@@ -15,10 +15,8 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -36,11 +34,10 @@ type countingFetcher struct {
// Fetch counts the fetch and passes it on.
func (f *countingFetcher) Fetch(
ctx context.Context, source imagerequest.Source,
ifNoneMatch, ifModifiedSince string,
) (*httpfetcher.FetchResult, error) {
f.fetches.Add(1)
return f.Fetcher.Fetch(ctx, source, ifNoneMatch, ifModifiedSince)
return f.Fetcher.Fetch(ctx, source)
}
// newRefererRoutes returns both image routes of a Handlers whose
@@ -54,7 +51,7 @@ func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher)
Fetcher: newPhotoFetcher(t, allowlistedHost, signedHost),
}
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -63,7 +60,7 @@ func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher)
t.Fatalf("imgcache.NewCache() error = %v", err)
}
svc, err := imageproxy.New(&imageproxy.Config{
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
Cache: cache,
Fetcher: fetcher,
Processor: imageprocessor.New(imageprocessor.Params{
@@ -71,18 +68,16 @@ func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher)
MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
}, prometheus.NewRegistry())
})
if err != nil {
t.Fatalf("imageproxy.New() error = %v", err)
t.Fatalf("imgcache.NewService() error = %v", err)
}
keys, err := seal.DeriveKeys(testSigningKey)
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{
log: slog.New(slog.DiscardHandler),
imgSvc: svc,
@@ -18,8 +18,8 @@ import (
// signedPhotoURL returns a signed /v1/image/ URL, valid for a minute, for the
// JPEG at photoPath on signedHost at 50x50. It is signed with testSigningKey,
// the key the handlers that newSignedHostServer builds check signatures with.
// It takes a Handlers it does not use, so that it has the type of encPhotoURL.
// the key of the image service newSignedHostServer builds. It takes a Handlers
// it does not use, so that it has the type of encPhotoURL.
func signedPhotoURL(t *testing.T, _ *Handlers) string {
t.Helper()
+46 -30
View File
@@ -11,14 +11,13 @@ import (
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// TestServerBusyAnswers503 checks that respondImageError, which answers the
// errors of both image routes, answers 503 with a clear error when the image
// service gives up waiting for a free upstream connection or processing slot,
// wrapped as the service wraps them.
// TestServerBusyAnswers503 checks that both image routes answer 503 with
// a clear error when the image service gives up waiting for a free upstream
// connection or processing slot, wrapped as the service wraps them.
func TestServerBusyAnswers503(t *testing.T) {
t.Parallel()
@@ -31,44 +30,61 @@ func TestServerBusyAnswers503(t *testing.T) {
fmt.Errorf("upstream fetch failed: %w", httpfetcher.ErrTooManyConnections),
fmt.Errorf("image processing failed: %w", imageprocessor.ErrTooManyImages),
} {
rec := httptest.NewRecorder()
h.respondImageError(rec, req, err)
plain := httptest.NewRecorder()
h.respondImageError(plain, req, err)
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("for %v: status = %d, want %d",
err, rec.Code, http.StatusServiceUnavailable)
}
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("for %v: body = %q, want the server busy error",
err, rec.Body.String())
for route, rec := range map[string]*httptest.ResponseRecorder{
"/v1/image/": plain, "/v1/e/": encrypted,
} {
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("%s for %v: status = %d, want %d",
route, err, rec.Code, http.StatusServiceUnavailable)
}
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("%s for %v: body = %q, want the server busy error",
route, err, rec.Body.String())
}
}
}
}
// TestProcessingTimedOutAnswers503 checks that respondImageError, which
// answers the errors of both image routes, answers 503 with the server busy
// error when the image service reports that processing stopped at the
// deadline of the request doing it, wrapped as the service wraps it. A request
// that waited for that processing gets this error while its own deadline may
// have time left. The request's source only goes into a log line, so it is
// left empty.
// TestProcessingTimedOutAnswers503 checks that respondImageError and
// handleImageError, which answer the errors of the two image routes, answer
// 503 with the server busy error when the image service reports that
// processing stopped at the deadline of the request doing it, wrapped as the
// service wraps it. A request that waited for that processing gets this error
// while its own deadline may have time left. The request's source only goes
// into a log line, so it is left empty.
func TestProcessingTimedOutAnswers503(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
err := fmt.Errorf("%w: failed to encode: %w",
imageproxy.ErrProcessingTimedOut, context.DeadlineExceeded)
imgcache.ErrProcessingTimedOut, context.DeadlineExceeded)
rec := httptest.NewRecorder()
h.respondImageError(rec, &imagerequest.Request{}, err)
t.Logf("status = %d, body = %q", rec.Code, rec.Body.String())
plain := httptest.NewRecorder()
h.respondImageError(plain, &imagerequest.Request{}, err)
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("status = %d, want %d", rec.Code, http.StatusServiceUnavailable)
}
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("body = %q, want the server busy error", rec.Body.String())
for name, rec := range map[string]*httptest.ResponseRecorder{
"respondImageError": plain, "handleImageError": encrypted,
} {
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("%s: status = %d, want %d",
name, rec.Code, http.StatusServiceUnavailable)
}
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("%s: body = %q, want the server busy error",
name, rec.Body.String())
}
}
}
@@ -13,101 +13,44 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestUnsupportedSourceFormatAnswers415 checks that respondImageError, which
// answers the errors of both image routes, answers 415 with a clear error when
// the upstream answers with a content type pixa does not accept, such as TIFF,
// and when libvips does not read a source as the format its first bytes
// show, wrapped as the service wraps them.
// TestUnsupportedSourceFormatAnswers415 checks that both image routes answer
// 415 with a clear error when the upstream answers with a content type pixa
// does not accept, such as SVG, and when orig is asked of a source in a
// format the processor cannot name, wrapped as the service wraps them.
func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
req := &imagerequest.Request{
Source: imagerequest.Source{Host: "img.example.com", Path: "/a.tiff"},
Source: imagerequest.Source{Host: "img.example.com", Path: "/a.svg"},
}
for _, err := range []error{
fmt.Errorf("upstream fetch failed: %w: image/tiff",
fmt.Errorf("upstream fetch failed: %w: image/svg+xml",
httpfetcher.ErrInvalidContentType),
fmt.Errorf("image processing failed: %w: libvips does not read the input as avif",
fmt.Errorf("image processing failed: %w: unknown",
imageprocessor.ErrUnsupportedInputFormat),
} {
rec := httptest.NewRecorder()
h.respondImageError(rec, req, err)
t.Logf("for %v: %d %s", err, rec.Code, rec.Body.String())
plain := httptest.NewRecorder()
h.respondImageError(plain, req, err)
if rec.Code != http.StatusUnsupportedMediaType {
t.Errorf("for %v: status = %d, want %d",
err, rec.Code, http.StatusUnsupportedMediaType)
}
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
if !strings.Contains(rec.Body.String(), "unsupported source format") {
t.Errorf("for %v: body = %q, want the unsupported source format error",
err, rec.Body.String())
for route, rec := range map[string]*httptest.ResponseRecorder{
"HandleImage": plain, "HandleImageEnc": encrypted,
} {
t.Logf("%s for %v: %d %s", route, err, rec.Code, rec.Body.String())
if rec.Code != http.StatusUnsupportedMediaType {
t.Errorf("%s for %v: status = %d, want %d",
route, err, rec.Code, http.StatusUnsupportedMediaType)
}
if !strings.Contains(rec.Body.String(), "unsupported source format") {
t.Errorf("%s for %v: body = %q, want the unsupported source format error",
route, err, rec.Body.String())
}
}
}
}
// TestSVGTooThinAnswers415 checks that respondImageError answers 415 with a
// clear error for an SVG source too thin to render, wrapped as the service
// wraps it.
func TestSVGTooThinAnswers415(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
req := &imagerequest.Request{
Source: imagerequest.Source{Host: "svg.example.com", Path: "/a.svg"},
}
err := fmt.Errorf("image processing failed: %w: 81930x10",
imageprocessor.ErrSVGTooThin)
rec := httptest.NewRecorder()
h.respondImageError(rec, req, err)
t.Logf("for %v: %d %s", err, rec.Code, rec.Body.String())
if rec.Code != http.StatusUnsupportedMediaType {
t.Errorf("status = %d, want %d", rec.Code,
http.StatusUnsupportedMediaType)
}
if !strings.Contains(rec.Body.String(), "SVG source too thin to render") {
t.Errorf("body = %q, want the SVG too thin error", rec.Body.String())
}
}
// TestSVGRenderFailedAnswers415 checks that respondImageError answers 415,
// naming the limits, for an SVG source that a child process could not read
// or render, wrapped as the service wraps it.
func TestSVGRenderFailedAnswers415(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
err := fmt.Errorf("image processing failed: %w: it took more than 20s",
imageprocessor.ErrSVGRenderFailed)
rec := httptest.NewRecorder()
h.respondImageError(rec, &imagerequest.Request{}, err)
t.Logf("for %v: %d %s", err, rec.Code, rec.Body.String())
checkErrorBody(t, rec, http.StatusUnsupportedMediaType,
"SVG source could not be rendered: it is not valid, or it takes more "+
"than 20 seconds or 1 GiB of memory to render")
}
// TestSVGTooLargeForOrigSizeAnswers400 checks that respondImageError answers
// 400, asking for a size, for an SVG source above 8192 on a side asked for at
// the orig size in a raster format, wrapped as the service wraps it.
func TestSVGTooLargeForOrigSizeAnswers400(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
err := fmt.Errorf("image processing failed: %w: 10000x10 is above 8192",
imageprocessor.ErrSVGTooLargeForOrigSize)
rec := httptest.NewRecorder()
h.respondImageError(rec, &imagerequest.Request{}, err)
t.Logf("for %v: %d %s", err, rec.Code, rec.Body.String())
checkErrorBody(t, rec, http.StatusBadRequest,
"SVG source above 8192 on a side: give a size to render it at")
}
@@ -1,76 +0,0 @@
package httpfetcher
import (
"errors"
"io"
"net/http"
"net/http/httptest"
"sync"
"testing"
)
// TestFetch_SendsConditionalHeadersAndHandlesNotModified checks that Fetch
// sends a non-empty ifNoneMatch and ifModifiedSince as request headers, that
// upstream's 304 to such a request is a result with status 304 and no
// content, and that a 304 to a request with neither is ErrUpstreamError.
func TestFetch_SendsConditionalHeadersAndHandlesNotModified(t *testing.T) {
t.Parallel()
const (
etag = `"abc"`
lastModified = "Mon, 02 Nov 2026 10:00:00 GMT"
)
var (
mu sync.Mutex
received http.Header
)
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
received = r.Header.Clone()
mu.Unlock()
w.WriteHeader(http.StatusNotModified)
}))
t.Cleanup(srv.Close)
f, _ := newServerFetcher(t, srv, nil)
res, err := f.Fetch(testContext(t), upstreamSource("/image"), etag, lastModified)
if err != nil {
t.Fatalf("conditional Fetch() error = %v", err)
}
body, err := io.ReadAll(res.Content)
_ = res.Content.Close()
mu.Lock()
t.Logf("status = %d, body = %q, request headers = %v", res.StatusCode, body, received)
gotIfNoneMatch := received.Get("If-None-Match")
gotIfModifiedSince := received.Get("If-Modified-Since")
mu.Unlock()
if err != nil || len(body) != 0 {
t.Errorf("304 content = %q, error = %v; want empty", body, err)
}
if res.StatusCode != http.StatusNotModified {
t.Errorf("StatusCode = %d, want 304", res.StatusCode)
}
if gotIfNoneMatch != etag || gotIfModifiedSince != lastModified {
t.Errorf("If-None-Match = %q, If-Modified-Since = %q; want %q and %q",
gotIfNoneMatch, gotIfModifiedSince, etag, lastModified)
}
if held := len(f.allHostsSemaphore); held != 0 {
t.Errorf("connection still held after the 304: %d held", held)
}
_, err = f.Fetch(testContext(t), upstreamSource("/image"), "", "")
if !errors.Is(err, ErrUpstreamError) {
t.Errorf("unconditional Fetch() answered 304: error = %v, want ErrUpstreamError", err)
}
}
@@ -51,12 +51,12 @@ func TestDialContextReplacesOnlyTheDialer(t *testing.T) {
t.Errorf("body = %q, want %q", body, imagePayload)
}
_, err := f.Fetch(testContext(t), imageSourceOn("127.0.0.1"), "", "")
_, err := f.Fetch(testContext(t), imageSourceOn("127.0.0.1"))
if !errors.Is(err, ErrSSRFBlocked) {
t.Errorf("Fetch(loopback URL) error = %v, want ErrSSRFBlocked", err)
}
_, err = f.Fetch(testContext(t), upstreamSource("/redirect/private"), "", "")
_, err = f.Fetch(testContext(t), upstreamSource("/redirect/private"))
if !errors.Is(err, ErrSSRFBlocked) {
t.Errorf("Fetch(/redirect/private) error = %v, want ErrSSRFBlocked", err)
}
+7 -7
View File
@@ -83,7 +83,7 @@ func startUpstream(t *testing.T) *httptest.Server {
mux := http.NewServeMux()
mux.HandleFunc("/image", func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
w.Header().Set("Content-Type", contentTypeJPEG)
_, _ = io.WriteString(w, imagePayload)
})
mux.HandleFunc("/status/500", func(w http.ResponseWriter, _ *http.Request) {
@@ -160,7 +160,7 @@ func testContext(t *testing.T) context.Context {
func fetchImage(t *testing.T, f *HTTPFetcher, path string) *FetchResult {
t.Helper()
res, err := f.Fetch(testContext(t), upstreamSource(path), "", "")
res, err := f.Fetch(testContext(t), upstreamSource(path))
if err != nil {
t.Fatalf("Fetch(%s) error = %v", path, err)
}
@@ -172,7 +172,7 @@ func fetchImage(t *testing.T, f *HTTPFetcher, path string) *FetchResult {
func fetchExpectError(t *testing.T, f *HTTPFetcher, path string) error {
t.Helper()
res, err := f.Fetch(testContext(t), upstreamSource(path), "", "")
res, err := f.Fetch(testContext(t), upstreamSource(path))
if err == nil {
_ = res.Content.Close()
@@ -224,7 +224,7 @@ func TestFetchRedirectToPrivateIPBlocked(t *testing.T) {
srv := startUpstream(t)
f, dialer := newServerFetcher(t, srv, nil)
_, err := f.Fetch(testContext(t), upstreamSource("/redirect/private"), "", "")
_, err := f.Fetch(testContext(t), upstreamSource("/redirect/private"))
if !errors.Is(err, ErrSSRFBlocked) {
t.Fatalf("Fetch() error = %v, want ErrSSRFBlocked", err)
}
@@ -358,7 +358,7 @@ func TestFetchContentLengthBodyAtMaxResponseSize(t *testing.T) {
res := fetchWithMaxResponseSize(t, maxResponseSize,
func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
w.Header().Set("Content-Type", contentTypeJPEG)
w.Header().Set("Content-Length", strconv.Itoa(tc.size))
_, _ = io.WriteString(w, body)
})
@@ -406,7 +406,7 @@ func TestFetchChunkedBodyAtMaxResponseSize(t *testing.T) {
res := fetchWithMaxResponseSize(t, maxResponseSize,
func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
w.Header().Set("Content-Type", contentTypeJPEG)
_, _ = io.WriteString(w, body)
// With no Content-Length, flushing sends the body
// chunked; its end is sent when the handler returns.
@@ -624,7 +624,7 @@ func TestFetchSchemeFollowsAllowHTTP(t *testing.T) {
DialContext: dialer.dialContext,
})
res, err := f.Fetch(testContext(t), upstreamSource("/image"), "", "")
res, err := f.Fetch(testContext(t), upstreamSource("/image"))
if err == nil {
_ = res.Content.Close()
}
+42 -128
View File
@@ -38,6 +38,18 @@ const userAgent = "pixa/1.0"
// MaxConnections fetches are already in progress.
const ConnectionWaitTimeout = 10 * time.Second
// MIME content types.
const (
contentTypeJPEG = "image/jpeg"
contentTypePNG = "image/png"
contentTypeGIF = "image/gif"
contentTypeWebP = "image/webp"
contentTypeAVIF = "image/avif"
contentTypeJXL = "image/jxl"
contentTypeSVG = "image/svg+xml"
contentTypeOctetStream = "application/octet-stream"
)
// Loopback addresses blocked by SSRF protection.
const (
localhostIPv4 = "127.0.0.1"
@@ -69,11 +81,7 @@ var (
ErrInvalidContentType = errors.New("invalid or unsupported content type")
ErrUpstreamError = errors.New("upstream server error")
ErrUpstreamTimeout = errors.New("upstream request timeout")
// ErrUpstreamRequestFailed is a fetch that failed before the upstream
// answered, other than by timeout or cancellation: a failed connection
// or TLS handshake, or a redirect the fetcher refuses.
ErrUpstreamRequestFailed = errors.New("upstream request failed")
ErrTooManyConnections = errors.New("too many concurrent upstream connections")
ErrTooManyConnections = errors.New("too many concurrent upstream connections")
)
// Internal fetcher errors.
@@ -84,13 +92,8 @@ var (
// Fetcher retrieves content from upstream origins.
type Fetcher interface {
// Fetch retrieves the content of the source image. A non-empty
// ifNoneMatch or ifModifiedSince is sent as the request header of that
// name, and upstream may then answer 304 Not Modified with no content.
Fetch(
ctx context.Context, source imagerequest.Source,
ifNoneMatch, ifModifiedSince string,
) (*FetchResult, error)
// Fetch retrieves the content of the source image.
Fetch(ctx context.Context, source imagerequest.Source) (*FetchResult, error)
}
// FetchResult contains the result of fetching from upstream.
@@ -118,17 +121,15 @@ type FetchResult struct {
}
// allowedContentTypes returns the content types an upstream response may
// have, which Fetch also sends as its Accept header: the MIME types of the
// formats pixa reads.
// have, which Fetch also sends as its Accept header.
func allowedContentTypes() []string {
return []string{
imagerequest.FormatJPEG.MIMEType(),
imagerequest.FormatPNG.MIMEType(),
imagerequest.FormatGIF.MIMEType(),
imagerequest.FormatWebP.MIMEType(),
imagerequest.FormatAVIF.MIMEType(),
imagerequest.FormatJXL.MIMEType(),
imagerequest.FormatSVG.MIMEType(),
contentTypeJPEG,
contentTypePNG,
contentTypeGIF,
contentTypeWebP,
contentTypeAVIF,
contentTypeJXL,
}
}
@@ -233,13 +234,9 @@ func New(config *Config) *HTTPFetcher {
// Fetch retrieves the source image with SSRF protection, over https, or over
// http when AllowHTTP is set. When MaxConnections fetches are already in
// progress, it waits up to ConnectionWaitTimeout for one to finish, then
// fails with ErrTooManyConnections. A non-empty ifNoneMatch or
// ifModifiedSince is sent as the If-None-Match or If-Modified-Since header.
// Upstream's 304 Not Modified to such a request is a result with StatusCode
// 304 and empty Content; to any other request it is ErrUpstreamError.
// fails with ErrTooManyConnections.
func (f *HTTPFetcher) Fetch(
ctx context.Context, source imagerequest.Source,
ifNoneMatch, ifModifiedSince string,
) (*FetchResult, error) {
sourceURL := source.URL(f.config.AllowHTTP)
@@ -266,7 +263,17 @@ func (f *HTTPFetcher) Fetch(
req := &http.Request{
Method: http.MethodGet,
URL: sourceURL,
Header: requestHeaders(ctx, ifNoneMatch, ifModifiedSince),
Header: make(http.Header),
}
req.Header.Set("User-Agent", userAgent)
req.Header.Set("Accept", strings.Join(allowedContentTypes(), ", "))
// The ID of the request this fetch serves, so the fetch can be found in
// the upstream host's logs
requestID := middleware.GetReqID(ctx)
if requestID != "" {
req.Header.Set(middleware.RequestIDHeader, requestID)
}
// Use httptrace to capture connection details
@@ -288,21 +295,11 @@ func (f *HTTPFetcher) Fetch(
fetchDuration := time.Since(startTime)
if err != nil {
return nil, upstreamRequestError(err)
}
if errors.Is(err, context.DeadlineExceeded) {
return nil, ErrUpstreamTimeout
}
// The deferred release gives the connection back, as no body follows
conditional := ifNoneMatch != "" || ifModifiedSince != ""
if conditional && resp.StatusCode == http.StatusNotModified {
_ = resp.Body.Close()
return &FetchResult{
Content: http.NoBody,
Headers: resp.Header,
StatusCode: resp.StatusCode,
FetchDurationMs: fetchDuration.Milliseconds(),
RemoteAddr: remoteAddr,
}, nil
return nil, fmt.Errorf("upstream request failed: %w", err)
}
result, err := f.buildResult(resp, remoteAddr, fetchDuration, release)
@@ -317,60 +314,6 @@ func (f *HTTPFetcher) Fetch(
return result, nil
}
// upstreamRequestError returns the error Fetch reports when its upstream
// request fails with err:
// - err wrapped in ErrUpstreamTimeout for a timeout, so the log names what
// timed out.
// - err, still matching context.Canceled, when the caller cancelled the
// fetch.
// - err wrapped in ErrUpstreamRequestFailed otherwise.
func upstreamRequestError(err error) error {
var netErr net.Error
switch {
case errors.Is(err, context.DeadlineExceeded),
errors.Is(err, ErrUpstreamTimeout),
errors.As(err, &netErr) && netErr.Timeout():
// A timeout other than the fetch timeout, such as the TLS
// handshake timeout, does not match context.DeadlineExceeded. An
// address lookup or a connection that timed out is
// ErrUpstreamTimeout already, so its log line names that error twice.
return fmt.Errorf("%w: %w", ErrUpstreamTimeout, err)
case errors.Is(err, context.Canceled):
// The caller cancelled the fetch; the upstream did not fail
return fmt.Errorf("upstream request cancelled: %w", err)
default:
return fmt.Errorf("%w: %w", ErrUpstreamRequestFailed, err)
}
}
// requestHeaders returns the headers of a fetch: User-Agent, Accept, the
// ID of the request the fetch serves, so the fetch can be found in the
// upstream host's logs, and a non-empty ifNoneMatch or ifModifiedSince.
func requestHeaders(
ctx context.Context, ifNoneMatch, ifModifiedSince string,
) http.Header {
header := make(http.Header)
header.Set("User-Agent", userAgent)
header.Set("Accept", strings.Join(allowedContentTypes(), ", "))
requestID := middleware.GetReqID(ctx)
if requestID != "" {
header.Set(middleware.RequestIDHeader, requestID)
}
if ifNoneMatch != "" {
header.Set("If-None-Match", ifNoneMatch)
}
if ifModifiedSince != "" {
header.Set("If-Modified-Since", ifModifiedSince)
}
return header
}
// acquireConnection takes a slot for host, then one of the slots shared by
// all hosts, and returns the func that gives both back. The host's slot
// comes first, so fetches queued for one busy host hold no shared slot.
@@ -533,7 +476,7 @@ func validateURL(ctx context.Context, fetchURL *neturl.URL, allowHTTP bool) erro
// Resolve the host to check IP addresses
addrs, err := net.DefaultResolver.LookupIPAddr(ctx, host)
if err != nil {
return addressLookupError(host, err)
return fmt.Errorf("%w: %s", ErrInvalidHost, host)
}
private := slices.ContainsFunc(addrs, func(addr net.IPAddr) bool {
@@ -546,23 +489,6 @@ func validateURL(ctx context.Context, fetchURL *neturl.URL, allowHTTP bool) erro
return nil
}
// addressLookupError returns the error for a failed address lookup of host:
// - ErrUpstreamTimeout when the lookup timed out.
// - err, still matching context.Canceled, when the lookup was cancelled.
// - ErrInvalidHost otherwise.
func addressLookupError(host string, err error) error {
var netErr net.Error
if errors.As(err, &netErr) && netErr.Timeout() {
return fmt.Errorf("%w: address lookup of %s", ErrUpstreamTimeout, host)
}
if errors.Is(err, context.Canceled) {
return fmt.Errorf("address lookup cancelled: %w", err)
}
return fmt.Errorf("%w: %s", ErrInvalidHost, host)
}
// isLocalhost checks if the host is localhost.
func isLocalhost(host string) bool {
host = strings.ToLower(host)
@@ -691,7 +617,7 @@ func dialSSRFSafe(
// Resolve the address
ips, err := net.DefaultResolver.LookupIP(ctx, "ip", host)
if err != nil {
return nil, addressLookupError(host, err)
return nil, fmt.Errorf("%w: %s", ErrInvalidHost, host)
}
// Check all resolved IPs
@@ -702,10 +628,7 @@ func dialSSRFSafe(
}
// Connect using the first valid IP
var (
dialer net.Dialer
lastDialErr error
)
var dialer net.Dialer
for _, ip := range ips {
addr := net.JoinHostPort(ip.String(), port)
@@ -714,18 +637,9 @@ func dialSSRFSafe(
if err == nil {
return conn, nil
}
lastDialErr = err
}
// Only the last address tried decides whether the connection timed out
var netErr net.Error
if errors.As(lastDialErr, &netErr) && netErr.Timeout() {
return nil, fmt.Errorf("%w: connect to %s: %w",
ErrUpstreamTimeout, host, lastDialErr)
}
return nil, fmt.Errorf("%w to %s: %w", errConnectFailed, host, lastDialErr)
return nil, fmt.Errorf("%w to %s", errConnectFailed, host)
}
// limitedReader wraps a reader and refuses a body longer than remaining
@@ -30,9 +30,9 @@ func TestIsAllowedContentType(t *testing.T) {
contentType string
want bool
}{
{imagerequest.FormatJPEG.MIMEType(), true},
{imagerequest.FormatPNG.MIMEType(), true},
{imagerequest.FormatWebP.MIMEType(), true},
{contentTypeJPEG, true},
{contentTypePNG, true},
{contentTypeWebP, true},
{"image/jpeg; charset=utf-8", true},
{"IMAGE/JPEG", true},
{"text/html", false},
@@ -210,14 +210,14 @@ func TestMockFetcher_FetchesFile(t *testing.T) {
m := NewMock(mockFS)
result, err := m.Fetch(context.Background(),
imagerequest.Source{Host: testHost, Path: "/images/photo.jpg"}, "", "")
imagerequest.Source{Host: testHost, Path: "/images/photo.jpg"})
if err != nil {
t.Fatalf("Fetch() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
if result.ContentType != imagerequest.FormatJPEG.MIMEType() {
if result.ContentType != contentTypeJPEG {
t.Errorf("ContentType = %q, want image/jpeg", result.ContentType)
}
@@ -242,7 +242,7 @@ func TestMockFetcher_MissingFileReturnsUpstreamError(t *testing.T) {
m := NewMock(mockFS)
_, err := m.Fetch(context.Background(),
imagerequest.Source{Host: testHost, Path: "/missing.jpg"}, "", "")
imagerequest.Source{Host: testHost, Path: "/missing.jpg"})
if !errors.Is(err, ErrUpstreamError) {
t.Errorf("Fetch() error = %v, want ErrUpstreamError", err)
}
@@ -259,7 +259,7 @@ func TestMockFetcher_RespectsContextCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
_, err := m.Fetch(ctx, imagerequest.Source{Host: testHost, Path: "/photo.jpg"}, "", "")
_, err := m.Fetch(ctx, imagerequest.Source{Host: testHost, Path: "/photo.jpg"})
if !errors.Is(err, context.Canceled) {
t.Errorf("Fetch() error = %v, want context.Canceled", err)
}
@@ -272,14 +272,14 @@ func TestDetectContentTypeFromPath(t *testing.T) {
path string
want string
}{
{"foo/bar.jpg", imagerequest.FormatJPEG.MIMEType()},
{"foo/bar.JPG", imagerequest.FormatJPEG.MIMEType()},
{"foo/bar.jpeg", imagerequest.FormatJPEG.MIMEType()},
{"foo/bar.png", imagerequest.FormatPNG.MIMEType()},
{"foo/bar.gif", imagerequest.FormatGIF.MIMEType()},
{"foo/bar.webp", imagerequest.FormatWebP.MIMEType()},
{"foo/bar.avif", imagerequest.FormatAVIF.MIMEType()},
{"foo/bar.svg", imagerequest.FormatSVG.MIMEType()},
{"foo/bar.jpg", contentTypeJPEG},
{"foo/bar.JPG", contentTypeJPEG},
{"foo/bar.jpeg", contentTypeJPEG},
{"foo/bar.png", contentTypePNG},
{"foo/bar.gif", contentTypeGIF},
{"foo/bar.webp", contentTypeWebP},
{"foo/bar.avif", contentTypeAVIF},
{"foo/bar.svg", contentTypeSVG},
{"foo/bar.bin", contentTypeOctetStream},
{"foo/bar", contentTypeOctetStream},
}
@@ -45,12 +45,12 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
f, _ := newServerFetcher(t, srv, cfg)
f.connectionWaitTimeout = 100 * time.Millisecond
first, err := f.Fetch(testContext(t), imageSourceOnPort(81), "", "")
first, err := f.Fetch(testContext(t), imageSourceOnPort(81))
if err != nil {
t.Fatalf("first Fetch() error = %v", err)
}
second, err := f.Fetch(testContext(t), imageSourceOnPort(82), "", "")
second, err := f.Fetch(testContext(t), imageSourceOnPort(82))
if err != nil {
t.Fatalf("second Fetch() error = %v", err)
}
@@ -59,7 +59,7 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
start := time.Now()
_, err = f.Fetch(testContext(t), imageSourceOnPort(83), "", "")
_, err = f.Fetch(testContext(t), imageSourceOnPort(83))
if !errors.Is(err, ErrTooManyConnections) {
t.Fatalf("third Fetch() error = %v, want ErrTooManyConnections", err)
}
@@ -78,7 +78,7 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
t.Fatalf("close first body: %v", err)
}
third, err := f.Fetch(testContext(t), imageSourceOnPort(83), "", "")
third, err := f.Fetch(testContext(t), imageSourceOnPort(83))
if err != nil {
t.Fatalf("Fetch() after a response was closed: error = %v", err)
}
@@ -105,7 +105,7 @@ func TestFetchFreesHostSlotWhenContextEndsWaitingForConnection(t *testing.T) {
f, _ := newServerFetcher(t, srv, cfg)
first, err := f.Fetch(testContext(t), imageSourceOnPort(81), "", "")
first, err := f.Fetch(testContext(t), imageSourceOnPort(81))
if err != nil {
t.Fatalf("first Fetch() error = %v", err)
}
@@ -115,7 +115,7 @@ func TestFetchFreesHostSlotWhenContextEndsWaitingForConnection(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
_, err = f.Fetch(ctx, imageSourceOnPort(82), "", "")
_, err = f.Fetch(ctx, imageSourceOnPort(82))
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("second Fetch() error = %v, want context.DeadlineExceeded", err)
}
@@ -140,7 +140,7 @@ func TestFetchRemovesIdleHostSemaphores(t *testing.T) {
for i := range 100 {
wg.Go(func() {
res, err := f.Fetch(ctx, imageSourceOnPort(1+i%50), "", "")
res, err := f.Fetch(ctx, imageSourceOnPort(1+i%50))
if err != nil {
t.Errorf("Fetch() error = %v", err)
@@ -179,12 +179,12 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
f, _ := newServerFetcher(t, srv, cfg)
f.connectionWaitTimeout = 100 * time.Millisecond
open, err := f.Fetch(testContext(t), imageSourceOnPort(81), "", "")
open, err := f.Fetch(testContext(t), imageSourceOnPort(81))
if err != nil {
t.Fatalf("first Fetch() error = %v", err)
}
_, err = f.Fetch(testContext(t), imageSourceOnPort(82), "", "")
_, err = f.Fetch(testContext(t), imageSourceOnPort(82))
if !errors.Is(err, ErrTooManyConnections) {
t.Fatalf("Fetch() from another host: error = %v, "+
"want ErrTooManyConnections", err)
@@ -193,7 +193,7 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
_, err = f.Fetch(ctx, imageSourceOnPort(81), "", "")
_, err = f.Fetch(ctx, imageSourceOnPort(81))
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("Fetch() from the busy host: error = %v, "+
"want context.DeadlineExceeded", err)
@@ -205,7 +205,7 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
ctx, cancel = context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
_, err = f.Fetch(ctx, imageSourceOnPort(83), "", "")
_, err = f.Fetch(ctx, imageSourceOnPort(83))
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("Fetch() from a host with nothing open: error = %v, "+
"want context.DeadlineExceeded", err)
@@ -255,7 +255,7 @@ func TestFetchReleasesConnectionOnError(t *testing.T) {
f, _ := newServerFetcher(t, srv, cfg)
f.connectionWaitTimeout = 100 * time.Millisecond
_, err := f.Fetch(testContext(t), tc.source, "", "")
_, err := f.Fetch(testContext(t), tc.source)
if !errors.Is(err, tc.want) {
t.Fatalf("Fetch() error = %v, want %v", err, tc.want)
}
+9 -14
View File
@@ -24,10 +24,9 @@ func NewMock(fsys fs.FS) *MockFetcher {
return &MockFetcher{fs: fsys}
}
// Fetch retrieves the source's content from the mock filesystem. It ignores
// ifNoneMatch and ifModifiedSince, and always sends the content.
// Fetch retrieves the source's content from the mock filesystem.
func (m *MockFetcher) Fetch(
ctx context.Context, source imagerequest.Source, _, _ string,
ctx context.Context, source imagerequest.Source,
) (*FetchResult, error) {
// Check context cancellation
select {
@@ -67,29 +66,25 @@ func (m *MockFetcher) Fetch(
}, nil
}
// contentTypeOctetStream is the content type the mock fetcher gives files in
// no format pixa reads.
const contentTypeOctetStream = "application/octet-stream"
// detectContentTypeFromPath returns the MIME type based on file extension.
func detectContentTypeFromPath(path string) string {
path = strings.ToLower(path)
switch {
case strings.HasSuffix(path, ".jpg"), strings.HasSuffix(path, ".jpeg"):
return imagerequest.FormatJPEG.MIMEType()
return contentTypeJPEG
case strings.HasSuffix(path, ".png"):
return imagerequest.FormatPNG.MIMEType()
return contentTypePNG
case strings.HasSuffix(path, ".gif"):
return imagerequest.FormatGIF.MIMEType()
return contentTypeGIF
case strings.HasSuffix(path, ".webp"):
return imagerequest.FormatWebP.MIMEType()
return contentTypeWebP
case strings.HasSuffix(path, ".avif"):
return imagerequest.FormatAVIF.MIMEType()
return contentTypeAVIF
case strings.HasSuffix(path, ".jxl"):
return imagerequest.FormatJXL.MIMEType()
return contentTypeJXL
case strings.HasSuffix(path, ".svg"):
return imagerequest.FormatSVG.MIMEType()
return contentTypeSVG
default:
return contentTypeOctetStream
}
@@ -1,213 +0,0 @@
package httpfetcher
import (
"context"
"errors"
"net"
"os"
"syscall"
"testing"
"time"
)
// TestFetchFailedBeforeAnswer checks the error of a fetch whose connection
// fails:
// - refused: ErrUpstreamRequestFailed.
// - timed out: ErrUpstreamTimeout, still matching the dial's error, even for
// a timeout error that does not match context.DeadlineExceeded, such as
// the TLS handshake timeout.
// - cancelled by its caller: context.Canceled, not ErrUpstreamRequestFailed.
func TestFetchFailedBeforeAnswer(t *testing.T) {
t.Parallel()
tests := []struct {
name string
dial func(cancel context.CancelFunc) error
wantErrs []error
wantNotErr error
}{
{
name: "connection refused",
dial: func(context.CancelFunc) error {
return syscall.ECONNREFUSED
},
wantErrs: []error{ErrUpstreamRequestFailed},
wantNotErr: context.Canceled,
},
{
name: "timed out while connecting",
dial: func(context.CancelFunc) error {
return os.ErrDeadlineExceeded
},
wantErrs: []error{ErrUpstreamTimeout, os.ErrDeadlineExceeded},
wantNotErr: ErrUpstreamRequestFailed,
},
{
name: "cancelled while connecting",
dial: func(cancel context.CancelFunc) error {
cancel()
return context.Canceled
},
wantErrs: []error{context.Canceled},
wantNotErr: ErrUpstreamRequestFailed,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(testContext(t))
defer cancel()
f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
AllowHTTP: true,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
DialContext: func(context.Context, string, string) (net.Conn, error) {
return nil, tt.dial(cancel)
},
})
_, err := f.Fetch(ctx, upstreamSource("/image"), "", "")
t.Logf("Fetch() error: %v", err)
for _, wantErr := range tt.wantErrs {
if !errors.Is(err, wantErr) {
t.Errorf("Fetch() error = %v, want %v", err, wantErr)
}
}
if errors.Is(err, tt.wantNotErr) {
t.Errorf("Fetch() error = %v, must not be %v", err, tt.wantNotErr)
}
})
}
}
// TestFetchAddressLookupTimedOut checks that a fetch whose address lookup
// times out fails with ErrUpstreamTimeout, both for the lookup of the source
// host and for the lookup while connecting. The error must match neither
// ErrInvalidHost nor ErrUpstreamRequestFailed, which the image routes answer
// 502. A context whose deadline has passed makes the lookup time out.
func TestFetchAddressLookupTimedOut(t *testing.T) {
t.Parallel()
checkTimedOut := func(t *testing.T, err error) {
t.Helper()
t.Logf("Fetch() error: %v", err)
if !errors.Is(err, ErrUpstreamTimeout) {
t.Errorf("Fetch() error = %v, want ErrUpstreamTimeout", err)
}
if errors.Is(err, ErrInvalidHost) || errors.Is(err, ErrUpstreamRequestFailed) {
t.Errorf("Fetch() error = %v, must match neither ErrInvalidHost "+
"nor ErrUpstreamRequestFailed", err)
}
}
t.Run("lookup of the source host", func(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithTimeout(testContext(t), 0)
defer cancel()
f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
})
_, err := f.Fetch(ctx, imageSourceOn("cdn.example.com"), "", "")
checkTimedOut(t, err)
})
t.Run("lookup while connecting", func(t *testing.T) {
t.Parallel()
// The source host is an address, which validateURL does not look up;
// the dialer looks up a name
f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) {
ctx, cancel := context.WithTimeout(ctx, 0)
defer cancel()
return dialSSRFSafe(ctx, network, "img.example.com:443", nil)
},
})
_, err := f.Fetch(testContext(t), upstreamSource("/image"), "", "")
checkTimedOut(t, err)
})
}
// TestFetchConnectionTimedOut checks that a fetch whose connection in
// dialSSRFSafe times out fails with ErrUpstreamTimeout, which the image
// routes answer 504. The error must match neither ErrUpstreamRequestFailed
// nor ErrInvalidHost, which they answer 502. A context whose deadline has
// passed makes the connection time out without touching the network.
func TestFetchConnectionTimedOut(t *testing.T) {
t.Parallel()
f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
ctx, cancel := context.WithTimeout(ctx, 0)
defer cancel()
return dialSSRFSafe(ctx, network, addr, nil)
},
})
_, err := f.Fetch(testContext(t), upstreamSource("/image"), "", "")
t.Logf("Fetch() error: %v", err)
if !errors.Is(err, ErrUpstreamTimeout) {
t.Errorf("Fetch() error = %v, want ErrUpstreamTimeout", err)
}
if errors.Is(err, ErrUpstreamRequestFailed) || errors.Is(err, ErrInvalidHost) {
t.Errorf("Fetch() error = %v, must match neither "+
"ErrUpstreamRequestFailed nor ErrInvalidHost", err)
}
}
// TestFetchAddressLookupCancelled checks that a fetch whose address lookup
// its caller cancelled fails with an error matching context.Canceled, not
// ErrInvalidHost, which the image routes answer 502.
func TestFetchAddressLookupCancelled(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(testContext(t))
cancel()
f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
})
_, err := f.Fetch(ctx, imageSourceOn("cdn.example.com"), "", "")
t.Logf("Fetch() error: %v", err)
if !errors.Is(err, context.Canceled) {
t.Errorf("Fetch() error = %v, want context.Canceled", err)
}
if errors.Is(err, ErrInvalidHost) {
t.Errorf("Fetch() error = %v, must not be ErrInvalidHost", err)
}
}
@@ -8,7 +8,6 @@ import (
"testing"
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestFetchSendsRequestID verifies that a fetch sends the ID of the request
@@ -26,7 +25,7 @@ func TestFetchSendsRequestID(t *testing.T) {
func(w http.ResponseWriter, r *http.Request) {
received <- r.Header.Get("X-Request-Id")
w.Header().Set("Content-Type", imagerequest.FormatJPEG.MIMEType())
w.Header().Set("Content-Type", contentTypeJPEG)
_, _ = io.WriteString(w, imagePayload)
}))
t.Cleanup(srv.Close)
@@ -35,7 +34,7 @@ func TestFetchSendsRequestID(t *testing.T) {
ctx := context.WithValue(testContext(t), middleware.RequestIDKey, requestID)
res, err := f.Fetch(ctx, upstreamSource("/image"), "", "")
res, err := f.Fetch(ctx, upstreamSource("/image"))
if err != nil {
t.Fatalf("Fetch() error = %v", err)
}
@@ -39,31 +39,30 @@ func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
tests := []struct {
name string
input []byte
sourceFormat imagerequest.Format
format imagerequest.Format
width, height int
fit imagerequest.Fit
refused bool
}{
{"AVIF at the limit", cutOff, imagerequest.FormatPNG,
imagerequest.FormatAVIF, 4096, 4096, imagerequest.FitCover, false},
{"AVIF above the limit", cutOff, imagerequest.FormatPNG,
imagerequest.FormatAVIF, 4096, 4097, imagerequest.FitCover, true},
{"WebP above the AVIF limit", cutOff, imagerequest.FormatPNG,
imagerequest.FormatWebP, 4096, 4097, imagerequest.FitCover, false},
{"AVIF at the limit", cutOff, imagerequest.FormatAVIF,
4096, 4096, imagerequest.FitCover, false},
{"AVIF above the limit", cutOff, imagerequest.FormatAVIF,
4096, 4097, imagerequest.FitCover, true},
{"WebP above the AVIF limit", cutOff, imagerequest.FormatWebP,
4096, 4097, imagerequest.FitCover, false},
// FitInside does not enlarge, so the output stays 64x32
{"AVIF inside a larger size", cutOff, imagerequest.FormatPNG,
imagerequest.FormatAVIF, 8192, 8192, imagerequest.FitInside, false},
{"AVIF inside a larger size", cutOff, imagerequest.FormatAVIF,
8192, 8192, imagerequest.FitInside, false},
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
{"AVIF outside a smaller size", cutOff, imagerequest.FormatPNG,
imagerequest.FormatAVIF, 8192, 1, imagerequest.FitOutside, true},
{"AVIF outside a smaller size", cutOff, imagerequest.FormatAVIF,
8192, 1, imagerequest.FitOutside, true},
// With no size, the output keeps the source's 4097x4096
{"AVIF of the source's size", largeAVIF, imagerequest.FormatAVIF,
imagerequest.FormatAVIF, 0, 0, imagerequest.FitCover, true},
{"orig of an AVIF source", largeAVIF, imagerequest.FormatAVIF,
imagerequest.FormatOriginal, 0, 0, imagerequest.FitCover, true},
{"JPEG of the source's size", largeAVIF, imagerequest.FormatAVIF,
imagerequest.FormatJPEG, 0, 0, imagerequest.FitCover, false},
0, 0, imagerequest.FitCover, true},
{"orig of an AVIF source", largeAVIF, imagerequest.FormatOriginal,
0, 0, imagerequest.FitCover, true},
{"JPEG of the source's size", largeAVIF, imagerequest.FormatJPEG,
0, 0, imagerequest.FitCover, false},
}
for _, tt := range tests {
@@ -73,7 +72,7 @@ func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
_, err := New(Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(), bytes.NewReader(tt.input), tt.sourceFormat,
}).Process(t.Context(), bytes.NewReader(tt.input),
&imagerequest.Transform{
Size: imagerequest.Size{Width: tt.width, Height: tt.height},
Format: tt.format,
@@ -42,7 +42,7 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
bytes.NewReader(input),
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
@@ -111,7 +111,7 @@ func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
bytes.NewReader(input),
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
@@ -7,8 +7,6 @@ import (
"io"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// readerAfterDeadline is a Process input whose first Read waits until done
@@ -88,8 +86,7 @@ func checkProcessStopsAtDeadline(
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
result, err := proc.Process(ctx, input, imagerequest.FormatJPEG,
smallJPEGTransform())
result, err := proc.Process(ctx, input, smallJPEGTransform())
if result != nil {
t.Logf("Process() made a %dx%d image after its deadline passed",
result.Width, result.Height)
@@ -19,7 +19,7 @@ func TestImageProcessor_EmptyFormatRefused(t *testing.T) {
MaxConcurrentProcessing: 1,
}).Process(
t.Context(), bytes.NewReader(createTestJPEG(t, 20, 20)),
imagerequest.FormatJPEG, &imagerequest.Transform{},
&imagerequest.Transform{},
)
if !errors.Is(err, ErrUnsupportedOutputFormat) {
t.Errorf("Process() error = %v, want %v", err, ErrUnsupportedOutputFormat)
@@ -11,18 +11,17 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest"
)
// processedSize runs input, an image in sourceFormat, through Process and
// returns the output's size in bytes.
// processedSize runs input through Process and returns the output's size in
// bytes.
func processedSize(
t *testing.T, input []byte, sourceFormat imagerequest.Format,
transform *imagerequest.Transform,
t *testing.T, input []byte, transform *imagerequest.Transform,
) int64 {
t.Helper()
result, err := New(Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(), bytes.NewReader(input), sourceFormat, transform)
}).Process(t.Context(), bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -75,7 +74,7 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
}
}
size := processedSize(t, encodePNG(t, flat), imagerequest.FormatPNG,
size := processedSize(t, encodePNG(t, flat),
&imagerequest.Transform{Format: imagerequest.FormatPNG})
const rawBytes = width * height * 3
@@ -93,7 +92,7 @@ func TestImageProcessor_WebPAtEffort2(t *testing.T) {
input := createTestJPEG(t, 200, 150)
size := processedSize(t, input, imagerequest.FormatJPEG,
size := processedSize(t, input,
&imagerequest.Transform{Format: imagerequest.FormatWebP, Quality: 85})
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
@@ -135,7 +134,7 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
input := encodePNG(t, source)
size := processedSize(t, input, imagerequest.FormatPNG,
size := processedSize(t, input,
&imagerequest.Transform{Format: imagerequest.FormatAVIF, Quality: 85})
want, _, err := decode(t, input).ExportAvif(&vips.AvifExportParams{
@@ -162,7 +161,7 @@ func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
input := createTestJPEG(t, 200, 150)
size := processedSize(t, input, imagerequest.FormatJPEG,
size := processedSize(t, input,
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: 85})
params := vips.NewJxlExportParams()
@@ -40,7 +40,7 @@ func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
bytes.NewReader(input),
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
@@ -113,7 +113,7 @@ func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatPNG,
bytes.NewReader(input),
&imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatPNG,
@@ -240,7 +240,7 @@ func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatPNG,
bytes.NewReader(input),
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatPNG,
@@ -45,7 +45,7 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
bytes.NewReader(input),
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
@@ -102,7 +102,7 @@ func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
bytes.NewReader(input),
&imagerequest.Transform{
Size: tt.size,
Format: imagerequest.FormatJPEG,
+140 -323
View File
@@ -1,7 +1,4 @@
// Package imageprocessor provides image format conversion and resizing using libvips.
// It reads the size of an SVG source and renders it in child processes, each
// the running program started again (see RunAsSVGChildProcess). pixa kills
// them after SVGRenderTimeout, and each may use at most MaxSVGRenderMemory.
package imageprocessor
import (
@@ -10,8 +7,6 @@ import (
"errors"
"fmt"
"io"
"math"
"strconv"
"sync"
"time"
@@ -75,6 +70,8 @@ type Result struct {
InputWidth int
// InputHeight is the original image height before processing.
InputHeight int
// InputFormat is the detected input format (e.g., "jpeg", "png").
InputFormat string
}
// MaxInputDimension is the maximum allowed width or height for input images.
@@ -116,45 +113,11 @@ var ErrAVIFOutputTooLarge = errors.New("AVIF output has too many pixels")
var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
// ErrUnsupportedInputFormat is returned, before the input is decoded, when
// libvips reads the input's first bytes as another format than its source
// format, or as none. An SVG whose first bytes pass the first-bytes check for
// AVIF is one example. A source format other than the seven pixa reads is
// also refused with it.
// libvips reads the input's first bytes as another format than the source
// formats, or as none: for example an SVG whose first bytes pass the
// first-bytes check for AVIF.
var ErrUnsupportedInputFormat = errors.New("unsupported input format")
// ErrSVGNotOriginalSize is returned when orig is asked of an SVG source at a
// size other than orig. pixa returns an SVG only as it is; a raster format
// gives it a size.
var ErrSVGNotOriginalSize = errors.New(
"an SVG is returned as SVG only at the orig size")
// ErrSVGTooThin is returned, before rendering, when an SVG source has one
// side more than MaxInputDimension times the other. Rendered with its longer
// side at most MaxInputDimension, its shorter side would be under 1 pixel.
var ErrSVGTooThin = errors.New("SVG source too thin to render")
// ErrSVGTooLargeForOrigSize is returned, before rendering, when a format other
// than orig asks for an SVG source at the orig size and the SVG has a side
// above MaxOutputDimension.
var ErrSVGTooLargeForOrigSize = errors.New(
"SVG source too large to render at its own size")
// SVGRenderTimeout is how long the child processes that read the size of one
// SVG source and render it may run, counted from the start of the first,
// before pixa kills them.
const SVGRenderTimeout = 20 * time.Second
// MaxSVGRenderMemory is the most memory, in bytes, that each child process
// reading the size of an SVG source or rendering it may use. It is the
// child's RLIMIT_DATA, which limits its heap and its other private writable
// memory.
const MaxSVGRenderMemory = 1 << 30 // 1 GiB
// ErrSVGRenderFailed is returned when a child process fails to read the size
// of an SVG source or to render it: librsvg cannot read the SVG, or the
// child takes more than SVGRenderTimeout or MaxSVGRenderMemory.
var ErrSVGRenderFailed = errors.New("SVG source could not be rendered")
// ErrTooManyImages is returned when MaxConcurrentProcessing images are being
// processed and none finishes within ProcessingWaitTimeout.
var ErrTooManyImages = errors.New("too many images being processed at once")
@@ -172,10 +135,6 @@ type ImageProcessor struct {
processingSemaphore chan struct{}
// processingWaitTimeout is ProcessingWaitTimeout; tests shorten it.
processingWaitTimeout time.Duration
// svgRenderTimeout is SVGRenderTimeout; tests shorten it.
svgRenderTimeout time.Duration
// maxSVGRenderMemory is MaxSVGRenderMemory; tests lower it.
maxSVGRenderMemory int
}
// Params holds configuration for creating an ImageProcessor. Both fields
@@ -195,27 +154,20 @@ func New(params Params) *ImageProcessor {
maxInputBytes: params.MaxInputBytes,
processingSemaphore: make(chan struct{}, params.MaxConcurrentProcessing),
processingWaitTimeout: ProcessingWaitTimeout,
svgRenderTimeout: SVGRenderTimeout,
maxSVGRenderMemory: MaxSVGRenderMemory,
}
}
// Process transforms the image read from input according to transform.
// sourceFormat is the input's format, which magic detected from its first
// bytes. When MaxConcurrentProcessing images are already being processed,
// it waits up to ProcessingWaitTimeout for one to finish, then fails with
// Process transforms an image according to transform. When
// MaxConcurrentProcessing images are already being processed, it waits up
// to ProcessingWaitTimeout for one to finish, then fails with
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
// encodes, it returns ctx's error and gives the slot back. govips gives pixa
// no way to stop libvips once an encode has begun, so an encode keeps its
// slot until it finishes. An AVIF output of more than MaxAVIFOutputPixels is
// refused with ErrAVIFOutputTooLarge before anything is decoded. orig of an
// SVG source is the SVG as it is, since libvips cannot save SVG; any other
// format renders it at the output size. Child processes read an SVG's size
// and render it, as processSVG describes.
// refused with ErrAVIFOutputTooLarge before anything is decoded.
func (p *ImageProcessor) Process(
ctx context.Context,
input io.Reader,
sourceFormat imagerequest.Format,
transform *imagerequest.Transform,
) (*Result, error) {
release, err := p.acquireSlot(ctx)
@@ -230,18 +182,39 @@ func (p *ImageProcessor) Process(
return nil, err
}
if sourceFormat == imagerequest.FormatSVG {
return p.processSVG(ctx, data, transform)
}
// govips picks the libvips loader from the input's first bytes with
// DetermineImageType. Input whose first bytes it reads as no source
// format, such as SVG, which librsvg would render, is refused before
// libvips reads it.
inputFormat := p.detectFormat(vips.DetermineImageType(data))
img, err := loadImage(data, sourceFormat)
sourceFormat, err := p.formatFromString(inputFormat)
if err != nil {
return nil, err
}
// Decode image
img, err := vips.NewImageFromBuffer(data)
if err != nil {
return nil, fmt.Errorf("failed to decode image: %w", err)
}
defer img.Close()
origWidth, origHeight := img.Width(), img.Height()
outputFormat := resolveOutputFormat(transform.Format, sourceFormat)
// Turn the image upright now: encode strips the EXIF orientation tag,
// and sizes below must be worked out on the upright image.
err = img.AutoRotate()
if err != nil {
return nil, fmt.Errorf("failed to auto-rotate: %w", err)
}
// Get original dimensions
origWidth := img.Width()
origHeight := img.Height()
// Validate input dimensions to prevent DoS via decompression bombs
if origWidth > MaxInputDimension || origHeight > MaxInputDimension {
return nil, ErrInputTooLarge
}
// Determine target dimensions; too large ones are refused before resizing
targetWidth, targetHeight, err := targetDimensions(transform.Size,
@@ -250,6 +223,12 @@ func (p *ImageProcessor) Process(
return nil, err
}
// orig is the source's own format; encode refuses an empty format
outputFormat := transform.Format
if outputFormat == imagerequest.FormatOriginal {
outputFormat = sourceFormat
}
// Determine output dimensions; too large ones are refused before
// resizing, which with FitCover already decodes the input
outputWidth, outputHeight, err := outputDimensions(origWidth, origHeight,
@@ -266,104 +245,8 @@ func (p *ImageProcessor) Process(
}
}
return p.encodeResult(ctx, img, outputFormat, transform.Quality,
origWidth, origHeight)
}
// WaitForProcessing waits until no image is being processed, or until ctx
// ends, and returns how many images were still being processed then. It
// waits by taking each slot in processingSemaphore as it frees up until it
// holds them all, or until ctx ends, then gives back the slots it took.
func (p *ImageProcessor) WaitForProcessing(ctx context.Context) int {
taken := 0
defer func() {
for range taken {
<-p.processingSemaphore
}
}()
for taken < cap(p.processingSemaphore) {
select {
case p.processingSemaphore <- struct{}{}:
taken++
case <-ctx.Done():
return len(p.processingSemaphore) - taken
}
}
return 0
}
// processSVG is Process for an SVG source, data. Apart from the first-bytes
// check, pixad hands the SVG to libvips only in child processes (see
// runSVGChildProcess). One reads the SVG's size. For a raster format a second
// renders it as PNG, which is then resized and encoded as any other image.
// pixa kills them once svgRenderTimeout has passed since the first one
// started.
func (p *ImageProcessor) processSVG(
ctx context.Context, data []byte, transform *imagerequest.Transform,
) (*Result, error) {
err := checkInputFormat(data, imagerequest.FormatSVG)
if err != nil {
return nil, err
}
deadline := time.Now().Add(p.svgRenderTimeout)
origWidth, origHeight, err := p.readSVGSize(ctx, deadline, data)
if err != nil {
return nil, err
}
outputFormat := resolveOutputFormat(transform.Format, imagerequest.FormatSVG)
if outputFormat == imagerequest.FormatSVG {
return svgAsItIs(data, transform.Size, origWidth, origHeight)
}
err = checkSVGOrigSize(transform.Size, origWidth, origHeight)
if err != nil {
return nil, err
}
targetWidth, targetHeight, err := targetDimensions(transform.Size,
origWidth, origHeight)
if err != nil {
return nil, err
}
outputWidth, outputHeight, err := outputDimensions(origWidth, origHeight,
targetWidth, targetHeight, transform.FitMode, outputFormat)
if err != nil {
return nil, err
}
// The SVG is rendered at about the output size, to stay sharp
img, err := p.renderSVG(ctx, deadline, data, origWidth, origHeight,
outputWidth, outputHeight)
if err != nil {
return nil, err
}
defer img.Close()
if targetWidth != origWidth || targetHeight != origHeight {
err := p.resize(img, outputWidth, outputHeight, transform.FitMode)
if err != nil {
return nil, fmt.Errorf("failed to resize: %w", err)
}
}
return p.encodeResult(ctx, img, outputFormat, transform.Quality,
origWidth, origHeight)
}
// encodeResult encodes img as format at quality and returns it as the result
// for a source of origWidth x origHeight.
func (p *ImageProcessor) encodeResult(
ctx context.Context, img *vips.ImageRef, format imagerequest.Format,
quality, origWidth, origHeight int,
) (*Result, error) {
output, err := p.encode(ctx, img, format, quality)
// Encode to target format
output, err := p.encode(ctx, img, outputFormat, transform.Quality)
if err != nil {
return nil, fmt.Errorf("failed to encode: %w", err)
}
@@ -371,11 +254,12 @@ func (p *ImageProcessor) encodeResult(
return &Result{
Content: io.NopCloser(bytes.NewReader(output)),
ContentLength: int64(len(output)),
ContentType: format.MIMEType(),
ContentType: FormatToMIME(outputFormat),
Width: img.Width(),
Height: img.Height(),
InputWidth: origWidth,
InputHeight: origHeight,
InputFormat: inputFormat,
}, nil
}
@@ -410,125 +294,61 @@ func targetDimensions(
return width, height, nil
}
// resolveOutputFormat returns the format of the output for the requested
// format: sourceFormat for orig, and format itself for any other. encode
// refuses an empty format.
func resolveOutputFormat(
format, sourceFormat imagerequest.Format,
) imagerequest.Format {
if format == imagerequest.FormatOriginal {
return sourceFormat
}
// MIME types for the supported image formats.
const (
mimeJPEG = "image/jpeg"
mimePNG = "image/png"
mimeGIF = "image/gif"
mimeWebP = "image/webp"
mimeAVIF = "image/avif"
mimeJXL = "image/jxl"
)
return format
// FormatToMIME converts a Format to its MIME type string.
func FormatToMIME(format imagerequest.Format) string {
switch format {
case imagerequest.FormatJPEG:
return mimeJPEG
case imagerequest.FormatPNG:
return mimePNG
case imagerequest.FormatWebP:
return mimeWebP
case imagerequest.FormatGIF:
return mimeGIF
case imagerequest.FormatAVIF:
return mimeAVIF
case imagerequest.FormatJXL:
return mimeJXL
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
return "application/octet-stream"
default:
return "application/octet-stream"
}
}
// loadImage refuses data that libvips would not load as sourceFormat, then
// decodes it and turns the image upright: encode strips the EXIF orientation
// tag, and sizes must be worked out on the upright image. An image with a
// side above MaxInputDimension is ErrInputTooLarge, which guards against
// decompression bombs.
func loadImage(
data []byte, sourceFormat imagerequest.Format,
) (*vips.ImageRef, error) {
err := checkInputFormat(data, sourceFormat)
if err != nil {
return nil, err
// WaitForProcessing waits until no image is being processed, or until ctx
// ends, and returns how many images were still being processed then. It
// waits by taking each slot in processingSemaphore as it frees up until it
// holds them all, or until ctx ends, then gives back the slots it took.
func (p *ImageProcessor) WaitForProcessing(ctx context.Context) int {
taken := 0
defer func() {
for range taken {
<-p.processingSemaphore
}
}()
for taken < cap(p.processingSemaphore) {
select {
case p.processingSemaphore <- struct{}{}:
taken++
case <-ctx.Done():
return len(p.processingSemaphore) - taken
}
}
img, err := vips.NewImageFromBuffer(data)
if err != nil {
return nil, fmt.Errorf("failed to decode image: %w", err)
}
err = img.AutoRotate()
if err != nil {
img.Close()
return nil, fmt.Errorf("failed to auto-rotate: %w", err)
}
if img.Width() > MaxInputDimension || img.Height() > MaxInputDimension {
img.Close()
return nil, ErrInputTooLarge
}
return img, nil
}
// svgAsItIs returns the SVG data unchanged, which is orig of an SVG source
// at the orig size. width x height is the SVG's own size. Any other size is
// ErrSVGNotOriginalSize.
func svgAsItIs(
data []byte, size imagerequest.Size, width, height int,
) (*Result, error) {
if !size.OriginalSize() {
return nil, fmt.Errorf("%w: asked for %dx%d", ErrSVGNotOriginalSize,
size.Width, size.Height)
}
return &Result{
Content: io.NopCloser(bytes.NewReader(data)),
ContentLength: int64(len(data)),
ContentType: imagerequest.FormatSVG.MIMEType(),
Width: width,
Height: height,
InputWidth: width,
InputHeight: height,
}, nil
}
// checkSVGOrigSize returns ErrSVGTooLargeForOrigSize when size is the orig
// size and the SVG source, of width x height, has a side above
// MaxOutputDimension. processSVG calls it after returning orig of the SVG, so
// the SVG would be rendered at its own size.
func checkSVGOrigSize(size imagerequest.Size, width, height int) error {
if size.OriginalSize() && max(width, height) > MaxOutputDimension {
return fmt.Errorf("%w: %dx%d is above %d", ErrSVGTooLargeForOrigSize,
width, height, MaxOutputDimension)
}
return nil
}
// svgRenderSize returns the least size in the proportions of width x height,
// an SVG's own size, that covers outputWidth x outputHeight, so that resize
// hardly changes it. Neither side is above MaxInputDimension.
func svgRenderSize(width, height, outputWidth, outputHeight int) (int, int) {
scale := min(
max(float64(outputWidth)/float64(width),
float64(outputHeight)/float64(height)),
float64(MaxInputDimension)/float64(max(width, height)))
return int(math.Round(float64(width) * scale)),
int(math.Round(float64(height) * scale))
}
// renderSVG renders the SVG data, whose own size is width x height, at
// svgRenderSize for outputWidth x outputHeight, in a child process that pixa
// kills at deadline (see runSVGChildProcess). It returns the rendered image,
// read from the child's PNG. An SVG with one side more than
// MaxInputDimension times the other is ErrSVGTooThin.
func (p *ImageProcessor) renderSVG(
ctx context.Context, deadline time.Time, data []byte,
width, height, outputWidth, outputHeight int,
) (*vips.ImageRef, error) {
if max(width, height) > MaxInputDimension*min(width, height) {
return nil, fmt.Errorf("%w: %dx%d", ErrSVGTooThin, width, height)
}
renderWidth, renderHeight := svgRenderSize(width, height,
outputWidth, outputHeight)
png, err := p.runSVGChildProcess(ctx, deadline, data,
renderSVGAsPNGArgument, strconv.Itoa(renderWidth),
strconv.Itoa(renderHeight))
if err != nil {
return nil, err
}
return loadImage(png, imagerequest.FormatPNG)
return 0
}
// acquireSlot takes a slot in processingSemaphore, waiting at most
@@ -574,51 +394,27 @@ func (p *ImageProcessor) readInput(input io.Reader) ([]byte, error) {
return data, nil
}
// checkInputFormat returns ErrUnsupportedInputFormat unless libvips would
// load data with the loader of sourceFormat. govips picks the libvips loader
// from the input's first bytes with DetermineImageType. It refuses an SVG
// whose first bytes pass the first-bytes check for AVIF, which librsvg would
// otherwise render.
func checkInputFormat(data []byte, sourceFormat imagerequest.Format) error {
// A format with no loader, such as orig or auto, has the type govips also
// gives input it cannot read, which the check below would let through.
sourceType := vipsImageType(sourceFormat)
if sourceType == vips.ImageTypeUnknown {
return fmt.Errorf("%w: %q is not a source format",
ErrUnsupportedInputFormat, sourceFormat)
}
if vips.DetermineImageType(data) != sourceType {
return fmt.Errorf("%w: libvips does not read the input as %s",
ErrUnsupportedInputFormat, sourceFormat)
}
return nil
}
// vipsImageType returns the govips image type whose libvips loader reads
// format. orig and auto stand for another format, so they have
// vips.ImageTypeUnknown, which govips has no loader for.
func vipsImageType(format imagerequest.Format) vips.ImageType {
switch format {
case imagerequest.FormatJPEG:
return vips.ImageTypeJPEG
case imagerequest.FormatPNG:
return vips.ImageTypePNG
case imagerequest.FormatGIF:
return vips.ImageTypeGIF
case imagerequest.FormatWebP:
return vips.ImageTypeWEBP
case imagerequest.FormatAVIF:
return vips.ImageTypeAVIF
case imagerequest.FormatJXL:
return vips.ImageTypeJXL
case imagerequest.FormatSVG:
return vips.ImageTypeSVG
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
return vips.ImageTypeUnknown
// detectFormat returns the format string for a govips image type.
func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
switch imageType {
case vips.ImageTypeJPEG:
return "jpeg"
case vips.ImageTypePNG:
return "png"
case vips.ImageTypeGIF:
return "gif"
case vips.ImageTypeWEBP:
return "webp"
case vips.ImageTypeAVIF, vips.ImageTypeHEIF:
return string(imagerequest.FormatAVIF)
case vips.ImageTypeJXL:
return string(imagerequest.FormatJXL)
case vips.ImageTypeUnknown, vips.ImageTypeMagick, vips.ImageTypePDF,
vips.ImageTypeSVG, vips.ImageTypeTIFF, vips.ImageTypeBMP,
vips.ImageTypeJP2K:
return "unknown"
default:
return vips.ImageTypeUnknown
return "unknown"
}
}
@@ -793,9 +589,7 @@ func (p *ImageProcessor) encode(
case imagerequest.FormatJXL:
return exportJXL(img, quality)
// libvips cannot save SVG; Process returns an SVG source as it is
case imagerequest.FormatSVG, imagerequest.FormatOriginal,
imagerequest.FormatAuto:
case imagerequest.FormatOriginal, imagerequest.FormatAuto:
return nil, fmt.Errorf("%w: %s", ErrUnsupportedOutputFormat, format)
default:
@@ -974,3 +768,26 @@ func jxlDistance(quality int) float64 {
return 53.0/3000.0*q*q - 23.0/20.0*q + 25.0
}
// formatFromString converts a format string from detectFormat to Format. A
// format detectFormat could not name is ErrUnsupportedInputFormat.
func (p *ImageProcessor) formatFromString(
format string,
) (imagerequest.Format, error) {
switch format {
case "jpeg":
return imagerequest.FormatJPEG, nil
case "png":
return imagerequest.FormatPNG, nil
case "gif":
return imagerequest.FormatGIF, nil
case "webp":
return imagerequest.FormatWebP, nil
case string(imagerequest.FormatAVIF):
return imagerequest.FormatAVIF, nil
case string(imagerequest.FormatJXL):
return imagerequest.FormatJXL, nil
default:
return "", fmt.Errorf("%w: %s", ErrUnsupportedInputFormat, format)
}
}
@@ -19,11 +19,6 @@ import (
)
func TestMain(m *testing.M) {
// The SVG tests start this test binary again as SVG child processes
if IsSVGChildProcess(os.Args[1:]) {
os.Exit(RunAsSVGChildProcess(os.Args[1:]))
}
initVips()
code := m.Run()
@@ -99,23 +94,23 @@ func isAVIF(data []byte) bool {
// detectMIME is a minimal magic-byte detector for test assertions.
func detectMIME(data []byte) string {
if len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF {
return imagerequest.FormatJPEG.MIMEType()
return mimeJPEG
}
if len(data) >= 8 && string(data[:8]) == "\x89PNG\r\n\x1a\n" {
return imagerequest.FormatPNG.MIMEType()
return mimePNG
}
if len(data) >= 4 && string(data[:4]) == "GIF8" {
return imagerequest.FormatGIF.MIMEType()
return mimeGIF
}
if len(data) >= 12 && string(data[:4]) == "RIFF" && string(data[8:12]) == "WEBP" {
return imagerequest.FormatWebP.MIMEType()
return mimeWebP
}
if isAVIF(data) {
return imagerequest.FormatAVIF.MIMEType()
return mimeAVIF
}
return ""
@@ -136,8 +131,7 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -163,7 +157,7 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
}
mime := detectMIME(data)
if mime != imagerequest.FormatJPEG.MIMEType() {
if mime != mimeJPEG {
t.Errorf("Output format = %v, want image/jpeg", mime)
}
}
@@ -182,8 +176,7 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -196,7 +189,7 @@ func TestImageProcessor_ConvertToPNG(t *testing.T) {
}
mime := detectMIME(data)
if mime != imagerequest.FormatPNG.MIMEType() {
if mime != mimePNG {
t.Errorf("Output format = %v, want image/png", mime)
}
}
@@ -220,8 +213,7 @@ func processAndCheckSize(
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -261,8 +253,7 @@ func TestImageProcessor_FitContain(t *testing.T) {
FitMode: imagerequest.FitContain,
}
result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -305,8 +296,7 @@ func TestImageProcessor_ProcessPNG(t *testing.T) {
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatPNG, transform)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -351,8 +341,7 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
FitMode: imagerequest.FitCover,
}
_, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
_, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err == nil {
t.Error("Process() should reject oversized input images")
}
@@ -380,8 +369,7 @@ func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf(
"Process() should accept images at MaxInputDimension, got error: %v",
@@ -411,8 +399,7 @@ func encodeAndCheck(
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v, want nil", err)
}
@@ -443,7 +430,7 @@ func encodeAndCheck(
func TestImageProcessor_EncodeWebP(t *testing.T) {
t.Parallel()
encodeAndCheck(t, imagerequest.FormatWebP, 80, imagerequest.FormatWebP.MIMEType())
encodeAndCheck(t, imagerequest.FormatWebP, 80, mimeWebP)
}
func TestImageProcessor_DecodeAVIF(t *testing.T) {
@@ -466,8 +453,7 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatAVIF, transform)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v, want nil (AVIF decoding should work)", err)
}
@@ -481,7 +467,7 @@ func TestImageProcessor_DecodeAVIF(t *testing.T) {
}
mime := detectMIME(data)
if mime != imagerequest.FormatJPEG.MIMEType() {
if mime != mimeJPEG {
t.Errorf("Output format = %v, want image/jpeg", mime)
}
}
@@ -508,8 +494,7 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
FitMode: imagerequest.FitCover,
}
_, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
_, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err == nil {
t.Fatal("Process() should reject input exceeding maxInputBytes")
}
@@ -536,8 +521,7 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
FitMode: imagerequest.FitCover,
}
result, err := proc.Process(ctx, bytes.NewReader(input),
imagerequest.FormatJPEG, transform)
result, err := proc.Process(ctx, bytes.NewReader(input), transform)
if err != nil {
t.Fatalf("Process() error = %v, want nil", err)
}
@@ -548,11 +532,11 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
func TestImageProcessor_EncodeAVIF(t *testing.T) {
t.Parallel()
encodeAndCheck(t, imagerequest.FormatAVIF, 85, imagerequest.FormatAVIF.MIMEType())
encodeAndCheck(t, imagerequest.FormatAVIF, 85, mimeAVIF)
}
// processAndDecode runs input, a JPEG, through Process and decodes the output
// with vips, so a test can inspect the image a client would receive.
// processAndDecode runs input through Process and decodes the output with
// vips, so a test can inspect the image a client would receive.
func processAndDecode(
t *testing.T, input []byte, transform *imagerequest.Transform,
) *vips.ImageRef {
@@ -562,8 +546,7 @@ func processAndDecode(
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(
context.Background(), bytes.NewReader(input), imagerequest.FormatJPEG,
transform,
context.Background(), bytes.NewReader(input), transform,
)
if err != nil {
t.Fatalf("Process() error = %v", err)
@@ -25,8 +25,7 @@ func TestImageProcessor_EncodeJPEGXL(t *testing.T) {
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)),
imagerequest.FormatJPEG, transform,
t.Context(), bytes.NewReader(createTestJPEG(t, 200, 150)), transform,
)
if err != nil {
t.Fatalf("Process() error = %v", err)
@@ -72,7 +71,7 @@ func TestImageProcessor_JPEGXLQuality(t *testing.T) {
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(t.Context(),
bytes.NewReader(input), imagerequest.FormatJPEG,
bytes.NewReader(input),
&imagerequest.Transform{Format: imagerequest.FormatJXL, Quality: quality})
if err != nil {
t.Fatalf("Process() at quality %d error = %v", quality, err)
@@ -93,8 +93,7 @@ func processInBackground(
proc *ImageProcessor, reader *gatedReader, results chan<- error,
) {
go func() {
result, err := proc.Process(context.Background(), reader,
imagerequest.FormatJPEG, smallJPEGTransform())
result, err := proc.Process(context.Background(), reader, smallJPEGTransform())
if err == nil {
_ = result.Content.Close()
}
@@ -207,7 +206,7 @@ func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) {
start := time.Now()
_, err := proc.Process(context.Background(), bytes.NewReader(input),
imagerequest.FormatJPEG, smallJPEGTransform())
smallJPEGTransform())
if !errors.Is(err, ErrTooManyImages) {
t.Fatalf("Process() error = %v, want ErrTooManyImages", err)
}
@@ -275,8 +274,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
proc := New(Params{MaxInputBytes: 4096, MaxConcurrentProcessing: 1})
proc.processingWaitTimeout = 100 * time.Millisecond
_, err := proc.Process(context.Background(), tc.input,
imagerequest.FormatJPEG, tc.transform)
_, err := proc.Process(context.Background(), tc.input, tc.transform)
if err == nil || (tc.want != nil && !errors.Is(err, tc.want)) {
t.Fatalf("Process() error = %v, want %v", err, tc.want)
}
@@ -286,7 +284,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
}
result, err := proc.Process(context.Background(), bytes.NewReader(valid),
imagerequest.FormatJPEG, smallJPEGTransform())
smallJPEGTransform())
if err != nil {
t.Fatalf("Process() after the error = %v, want nil", err)
}
@@ -10,9 +10,8 @@ import (
)
// TestImageProcessor_OrigOfUnnamedFormatRefused verifies that orig of a TIFF
// input is refused with ErrUnsupportedInputFormat before it is decoded, and
// not saved as JPEG. TIFF is not a source format, and the test passes the
// input to Process as a JPEG.
// input, which is not a source format, is refused with
// ErrUnsupportedInputFormat before it is decoded, and not saved as JPEG.
func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
t.Parallel()
@@ -32,7 +31,7 @@ func TestImageProcessor_OrigOfUnnamedFormatRefused(t *testing.T) {
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 1,
}).Process(
t.Context(), bytes.NewReader(tiff), imagerequest.FormatJPEG,
t.Context(), bytes.NewReader(tiff),
&imagerequest.Transform{Format: imagerequest.FormatOriginal},
)
if result != nil {
@@ -2,7 +2,6 @@ package imageprocessor
import (
"bytes"
"errors"
"io"
"testing"
@@ -22,7 +21,7 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
} {
saved, err := proc.Process(t.Context(),
bytes.NewReader(createTestPNG(t, 20, 20)), imagerequest.FormatPNG,
bytes.NewReader(createTestPNG(t, 20, 20)),
&imagerequest.Transform{Format: format})
if err != nil {
t.Fatalf("saving the test image as %s: %v", format, err)
@@ -33,7 +32,7 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
t.Fatalf("reading the %s image: %v", format, err)
}
result, err := proc.Process(t.Context(), bytes.NewReader(source), format,
result, err := proc.Process(t.Context(), bytes.NewReader(source),
&imagerequest.Transform{Format: imagerequest.FormatOriginal})
if err != nil {
t.Errorf("orig of the %s image: %v", format, err)
@@ -43,33 +42,9 @@ func TestImageProcessor_LoadsEachSourceFormat(t *testing.T) {
t.Logf("orig of the %s image: %s", format, result.ContentType)
if result.ContentType != format.MIMEType() {
if result.ContentType != FormatToMIME(format) {
t.Errorf("orig of the %s image is %s, want %s",
format, result.ContentType, format.MIMEType())
}
}
}
// TestImageProcessor_NotASourceFormatRefused verifies that a source format
// other than the seven, such as orig, auto or none, is refused with
// ErrUnsupportedInputFormat. The input is text, which libvips reads as no
// format.
func TestImageProcessor_NotASourceFormatRefused(t *testing.T) {
t.Parallel()
proc := New(Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1})
for _, sourceFormat := range []imagerequest.Format{
imagerequest.FormatOriginal, imagerequest.FormatAuto, "",
} {
result, err := proc.Process(t.Context(),
bytes.NewReader([]byte("this is text, not an image")), sourceFormat,
&imagerequest.Transform{Format: imagerequest.FormatJPEG})
t.Logf("source format %q: error = %v", sourceFormat, err)
if result != nil || !errors.Is(err, ErrUnsupportedInputFormat) {
t.Errorf("Process() with source format %q = %v, error %v; want %v",
sourceFormat, result, err, ErrUnsupportedInputFormat)
format, result.ContentType, FormatToMIME(format))
}
}
}
@@ -1,255 +0,0 @@
package imageprocessor
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"os"
"os/exec"
"strconv"
"syscall"
"time"
"github.com/davidbyttow/govips/v2/vips"
)
// An SVG child process is the running program started again with one of
// these arguments first, naming what it does. Its command line, after the
// program's name, is one of:
//
// read-svg-size MEMORY_LIMIT
// render-svg-as-png MEMORY_LIMIT WIDTH HEIGHT
//
// MEMORY_LIMIT is in bytes. The child reads the SVG on standard input. It
// writes the SVG's size as WIDTH HEIGHT, or the SVG rendered at WIDTH x
// HEIGHT as PNG, to standard output.
const (
readSVGSizeArgument = "read-svg-size"
renderSVGAsPNGArgument = "render-svg-as-png"
)
// errSVGChildProcessArguments is returned for a command line that is not
// that of an SVG child process.
var errSVGChildProcessArguments = errors.New(
"not the command line of an SVG child process")
// IsSVGChildProcess reports whether args, a process's command line after the
// program's name, are those of an SVG child process.
func IsSVGChildProcess(args []string) bool {
return len(args) > 0 &&
(args[0] == readSVGSizeArgument || args[0] == renderSVGAsPNGArgument)
}
// RunAsSVGChildProcess runs this process as the SVG child process that args,
// its command line after the program's name, ask for. It returns the status
// to exit with: 0 once it has written its result, and 1 after writing an
// error to standard error, which the parent discards.
func RunAsSVGChildProcess(args []string) int {
err := runAsSVGChildProcess(args, os.Stdin, os.Stdout)
if err != nil {
_, _ = fmt.Fprintln(os.Stderr, err)
return 1
}
return 0
}
// runAsSVGChildProcess does the work of RunAsSVGChildProcess, reading the SVG
// from stdin and writing the result to stdout. It sets the process's limits
// before it reads the SVG.
func runAsSVGChildProcess(
args []string, stdin io.Reader, stdout io.Writer,
) error {
numbers, err := parseSVGChildProcessArguments(args)
if err != nil {
return err
}
err = setSVGChildProcessLimits(numbers[0])
if err != nil {
return err
}
initVips()
svg, err := io.ReadAll(stdin)
if err != nil {
return fmt.Errorf("failed to read the SVG: %w", err)
}
if args[0] == readSVGSizeArgument {
return writeSVGSize(svg, stdout)
}
return writeSVGAsPNG(svg, numbers[1], numbers[2], stdout)
}
// parseSVGChildProcessArguments returns the numbers on the command line of an
// SVG child process, args: the memory limit, then the width and height for
// render-svg-as-png.
func parseSVGChildProcessArguments(args []string) ([]int, error) {
wantArgs := 2
if len(args) > 0 && args[0] == renderSVGAsPNGArgument {
wantArgs = 4
}
if !IsSVGChildProcess(args) || len(args) != wantArgs {
return nil, fmt.Errorf("%w: %q", errSVGChildProcessArguments, args)
}
numbers := make([]int, 0, len(args)-1)
for _, arg := range args[1:] {
number, err := strconv.Atoi(arg)
if err != nil {
return nil, fmt.Errorf("%w: %q: %w", errSVGChildProcessArguments,
args, err)
}
numbers = append(numbers, number)
}
return numbers, nil
}
// setSVGChildProcessLimits sets this process's RLIMIT_DATA to memoryLimit
// bytes, so that allocating heap or other private writable memory past it
// fails. RLIMIT_AS would also count the address space the Go runtime reserves
// but does not use. It also sets RLIMIT_CORE to 0: librsvg crashes when an
// allocation fails, and the crash would write a core file of up to
// memoryLimit bytes into pixad's working directory.
func setSVGChildProcessLimits(memoryLimit int) error {
if memoryLimit < 1 {
return fmt.Errorf("%w: memory limit %d", errSVGChildProcessArguments,
memoryLimit)
}
err := syscall.Setrlimit(syscall.RLIMIT_DATA, &syscall.Rlimit{
Cur: uint64(memoryLimit),
Max: uint64(memoryLimit),
})
if err != nil {
return fmt.Errorf("failed to limit memory to %d bytes: %w",
memoryLimit, err)
}
err = syscall.Setrlimit(syscall.RLIMIT_CORE, &syscall.Rlimit{})
if err != nil {
return fmt.Errorf("failed to turn off core files: %w", err)
}
return nil
}
// writeSVGSize writes the size of svg to stdout as WIDTH HEIGHT. librsvg
// reads the whole SVG for it.
func writeSVGSize(svg []byte, stdout io.Writer) error {
img, err := vips.NewImageFromBuffer(svg)
if err != nil {
return fmt.Errorf("failed to read the SVG: %w", err)
}
defer img.Close()
_, err = fmt.Fprintf(stdout, "%d %d\n", img.Width(), img.Height())
return err
}
// writeSVGAsPNG renders svg at width x height and writes it to stdout as PNG.
// libvips' thumbnail has librsvg render the SVG at that size directly, at
// any scale: govips takes a density only in whole dots per inch, which cannot
// render an SVG below 1/72 of its own size. Given the SVG as bytes with no
// file name, librsvg loads no file or URL the SVG refers to.
func writeSVGAsPNG(svg []byte, width, height int, stdout io.Writer) error {
img, err := vips.NewThumbnailWithSizeFromBuffer(svg, width, height,
vips.InterestingNone, vips.SizeBoth)
if err != nil {
return fmt.Errorf("failed to render the SVG: %w", err)
}
defer img.Close()
png, err := exportPNG(img)
if err != nil {
return fmt.Errorf("failed to save the SVG as PNG: %w", err)
}
_, err = stdout.Write(png)
return err
}
// readSVGSize reads the size of the SVG data in a child process that pixa
// kills at deadline (see runSVGChildProcess).
func (p *ImageProcessor) readSVGSize(
ctx context.Context, deadline time.Time, data []byte,
) (int, int, error) {
output, err := p.runSVGChildProcess(ctx, deadline, data,
readSVGSizeArgument)
if err != nil {
return 0, 0, err
}
var width, height int
_, err = fmt.Sscan(string(output), &width, &height)
if err != nil {
return 0, 0, fmt.Errorf("failed to read the size %q an SVG child "+
"process wrote: %w", output, err)
}
return width, height, nil
}
// runSVGChildProcess starts the running program again as an SVG child
// process, with argument, p.maxSVGRenderMemory and extraArgs as its command
// line, and waits for it. It gives the child data on standard input,
// discards its standard error, and returns its standard output. pixa kills
// the child at deadline. A child that fails or is killed is
// ErrSVGRenderFailed, unless ctx has ended: runSVGChildProcess then returns
// ctx's error. Failing to start the child is another error.
func (p *ImageProcessor) runSVGChildProcess(
ctx context.Context, deadline time.Time, data []byte,
argument string, extraArgs ...string,
) ([]byte, error) {
executable, err := os.Executable()
if err != nil {
return nil, fmt.Errorf("failed to find the running program: %w", err)
}
childCtx, cancel := context.WithDeadline(ctx, deadline)
defer cancel()
args := append([]string{argument, strconv.Itoa(p.maxSVGRenderMemory)},
extraArgs...)
//nolint:gosec // the running program, with arguments pixa makes
cmd := exec.CommandContext(childCtx, executable, args...)
cmd.Stdin = bytes.NewReader(data)
var stdout bytes.Buffer
cmd.Stdout = &stdout
err = cmd.Run()
var exitError *exec.ExitError
switch {
case err == nil:
return stdout.Bytes(), nil
case ctx.Err() != nil:
return nil, ctx.Err()
case childCtx.Err() != nil:
return nil, fmt.Errorf("%w: it took more than %v", ErrSVGRenderFailed,
p.svgRenderTimeout)
case errors.As(err, &exitError):
return nil, fmt.Errorf("%w: %w", ErrSVGRenderFailed, err)
default:
return nil, fmt.Errorf("failed to run an SVG child process: %w", err)
}
}
@@ -1,503 +0,0 @@
package imageprocessor
import (
"bytes"
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// testSVG is a 16x12 SVG, black on its left half and white on its right.
const testSVG = `<svg xmlns="http://www.w3.org/2000/svg" width="16" height="12">` +
`<rect width="16" height="12" fill="white"/>` +
`<rect width="8" height="12" fill="black"/></svg>`
// newSVGTestProcessor returns a processor for the SVG tests.
func newSVGTestProcessor() *ImageProcessor {
return New(Params{MaxInputBytes: 1 << 20, MaxConcurrentProcessing: 1})
}
// processSVG processes svg, as an SVG source, with transform.
func processSVG(
t *testing.T, svg string, transform imagerequest.Transform,
) (*Result, []byte, error) {
t.Helper()
result, err := newSVGTestProcessor().Process(t.Context(),
bytes.NewReader([]byte(svg)), imagerequest.FormatSVG, &transform)
if err != nil {
return nil, nil, err
}
output, err := io.ReadAll(result.Content)
if err != nil {
t.Fatalf("reading the output: %v", err)
}
return result, output, nil
}
// TestImageProcessor_SVGOrigIsTheSVG checks that orig of an SVG at the orig
// size is the SVG itself, never another format, and that orig at another
// size is ErrSVGNotOriginalSize.
func TestImageProcessor_SVGOrigIsTheSVG(t *testing.T) {
t.Parallel()
result, output, err := processSVG(t, testSVG,
imagerequest.Transform{Format: imagerequest.FormatOriginal})
if err != nil {
t.Fatalf("orig of the SVG: %v", err)
}
t.Logf("orig of the SVG: %s, %dx%d", result.ContentType, result.Width,
result.Height)
if result.ContentType != "image/svg+xml" || string(output) != testSVG {
t.Errorf("orig of the SVG = %s %q, want image/svg+xml %q",
result.ContentType, output, testSVG)
}
_, _, err = processSVG(t, testSVG, imagerequest.Transform{
Size: imagerequest.Size{Width: 32, Height: 24},
Format: imagerequest.FormatOriginal,
})
if !errors.Is(err, ErrSVGNotOriginalSize) {
t.Errorf("orig of the SVG at 32x24: error = %v, want %v", err,
ErrSVGNotOriginalSize)
}
}
// TestImageProcessor_RendersSVGToEachRasterFormat checks that an SVG is
// rendered to each raster format at the requested size.
func TestImageProcessor_RendersSVGToEachRasterFormat(t *testing.T) {
t.Parallel()
for _, format := range []imagerequest.Format{
imagerequest.FormatJPEG, imagerequest.FormatPNG, imagerequest.FormatGIF,
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
} {
result, output, err := processSVG(t, testSVG, imagerequest.Transform{
Size: imagerequest.Size{Width: 40, Height: 30},
Format: format,
})
if err != nil {
t.Errorf("SVG as %s: %v", format, err)
continue
}
t.Logf("SVG as %s: %s, %dx%d, %d bytes", format, result.ContentType,
result.Width, result.Height, len(output))
if result.ContentType != format.MIMEType() ||
vips.DetermineImageType(output) != vipsImageType(format) ||
result.Width != 40 || result.Height != 30 {
t.Errorf("SVG as %s = %s, %dx%d; want %s, 40x30", format,
result.ContentType, result.Width, result.Height, format.MIMEType())
}
}
}
// TestImageProcessor_RendersSVGSharpWhenEnlarged checks that a 16x12 SVG
// asked for at 1600x1200 is rendered at that size, not rendered at 16x12 and
// enlarged: 10 pixels either side of the edge between its black and white
// halves are black and white, where an enlarged image is grey there.
func TestImageProcessor_RendersSVGSharpWhenEnlarged(t *testing.T) {
t.Parallel()
_, output, err := processSVG(t, testSVG, imagerequest.Transform{
Size: imagerequest.Size{Width: 1600, Height: 1200},
Format: imagerequest.FormatPNG,
})
if err != nil {
t.Fatalf("SVG as PNG at 1600x1200: %v", err)
}
img, err := vips.NewImageFromBuffer(output)
if err != nil {
t.Fatalf("decoding the PNG: %v", err)
}
t.Cleanup(img.Close)
for x, want := range map[int]float64{790: 0, 810: 255} {
point, err := img.GetPoint(x, 600)
if err != nil {
t.Fatalf("GetPoint(%d, 600): %v", x, err)
}
t.Logf("pixel at %d,600: %v", x, point)
if point[0] != want {
t.Errorf("pixel at %d,600 = %v, want %v", x, point[0], want)
}
}
}
// TestImageProcessor_SVGFetchesNothing checks that rendering an SVG draws no
// image it refers to: one at a URL on a local server, by href or xlink:href,
// or a local file that exists. Each SVG renders without error, white where
// the image would be. An SVG that uses an external entity at that server is
// a separate case. No SVG makes a request to the server.
func TestImageProcessor_SVGFetchesNothing(t *testing.T) {
t.Parallel()
var requests atomic.Int64
// Blue is 128 in every pixel of this PNG.
png := createTestPNG(t, 4, 4)
server := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
requests.Add(1)
t.Logf("request for %s", r.URL)
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(png)
}))
t.Cleanup(server.Close)
pngPath := filepath.Join(t.TempDir(), "image.png")
err := os.WriteFile(pngPath, png, 0o600)
if err != nil {
t.Fatalf("writing the PNG file: %v", err)
}
for _, reference := range []string{
`href="` + server.URL + `/image.png"`,
`xlink:href="` + server.URL + `/xlink.png"`,
`href="file://` + pngPath + `"`,
} {
svg := `<svg xmlns="http://www.w3.org/2000/svg" ` +
`xmlns:xlink="http://www.w3.org/1999/xlink" width="16" height="12">` +
`<rect width="16" height="12" fill="white"/>` +
`<image ` + reference + ` width="16" height="12"/></svg>`
_, output, err := processSVG(t, svg, imagerequest.Transform{
Format: imagerequest.FormatPNG,
})
if err != nil {
t.Errorf("SVG with image %s as PNG: %v", reference, err)
continue
}
img, err := vips.NewImageFromBuffer(output)
if err != nil {
t.Fatalf("decoding the PNG: %v", err)
}
point, err := img.GetPoint(8, 6)
img.Close()
if err != nil {
t.Fatalf("GetPoint(8, 6): %v", err)
}
t.Logf("SVG with image %s: pixel at 8,6 = %v", reference, point)
if point[2] != 255 {
t.Errorf("SVG with image %s: blue at 8,6 = %v, want 255: the "+
"image is drawn", reference, point[2])
}
}
svg := `<?xml version="1.0"?>` +
`<!DOCTYPE svg [<!ENTITY outside SYSTEM "` + server.URL + `/entity">]>` +
`<svg xmlns="http://www.w3.org/2000/svg" width="16" height="12">` +
`<text y="10">&outside;</text></svg>`
// librsvg loads no external entity, so it refuses this SVG: outside is
// not defined.
_, _, err = processSVG(t, svg, imagerequest.Transform{
Format: imagerequest.FormatPNG,
})
t.Logf("SVG with an external entity as PNG: error = %v", err)
if requests.Load() != 0 {
t.Errorf("rendering the SVGs made %d requests, want none",
requests.Load())
}
}
// TestImageProcessor_RendersSVGLargerThanMaxInputDimension checks that an
// SVG whose own size is above MaxInputDimension is limited only by the
// output checks: it is rendered at a raster size, orig of it is the SVG
// itself, and a raster format at its own size is ErrSVGTooLargeForOrigSize.
// The 1000000x1000000 one is rendered at a ten-thousandth of its own size.
func TestImageProcessor_RendersSVGLargerThanMaxInputDimension(t *testing.T) {
t.Parallel()
for _, svg := range []string{
`<svg xmlns="http://www.w3.org/2000/svg" width="10000" height="10">` +
`<rect width="10000" height="10"/></svg>`,
`<svg xmlns="http://www.w3.org/2000/svg" width="1000000" ` +
`height="1000000"><rect width="1000000" height="1000000"/></svg>`,
} {
result, _, err := processSVG(t, svg, imagerequest.Transform{
Size: imagerequest.Size{Width: 100, Height: 100},
Format: imagerequest.FormatPNG,
})
if err != nil {
t.Errorf("%s as PNG at 100x100: %v", svg, err)
} else {
t.Logf("%s as PNG at 100x100: %dx%d", svg, result.Width, result.Height)
if result.Width != 100 || result.Height != 100 {
t.Errorf("%s as PNG at 100x100 = %dx%d, want 100x100", svg,
result.Width, result.Height)
}
}
result, output, err := processSVG(t, svg,
imagerequest.Transform{Format: imagerequest.FormatOriginal})
if err != nil || result.ContentType != imagerequest.FormatSVG.MIMEType() ||
string(output) != svg {
t.Errorf("orig of %s: error = %v, want the SVG itself", svg, err)
}
_, _, err = processSVG(t, svg,
imagerequest.Transform{Format: imagerequest.FormatPNG})
if !errors.Is(err, ErrSVGTooLargeForOrigSize) {
t.Errorf("%s as PNG at its own size: error = %v, want %v", svg,
err, ErrSVGTooLargeForOrigSize)
}
}
}
// TestImageProcessor_RefusesSVGTooThin checks that an SVG with one side more
// than MaxInputDimension times the other is ErrSVGTooThin at a raster size,
// that one at exactly that proportion is rendered, and that orig of either
// is the SVG itself.
func TestImageProcessor_RefusesSVGTooThin(t *testing.T) {
t.Parallel()
tooThin := `<svg xmlns="http://www.w3.org/2000/svg" width="81930" ` +
`height="10"><rect width="81930" height="10"/></svg>`
atTheLimit := `<svg xmlns="http://www.w3.org/2000/svg" width="81920" ` +
`height="10"><rect width="81920" height="10"/></svg>`
size := imagerequest.Size{Width: 100, Height: 100}
_, _, err := processSVG(t, tooThin,
imagerequest.Transform{Size: size, Format: imagerequest.FormatPNG})
if !errors.Is(err, ErrSVGTooThin) {
t.Errorf("81930x10 SVG as PNG at 100x100: error = %v, want %v", err,
ErrSVGTooThin)
}
result, _, err := processSVG(t, atTheLimit,
imagerequest.Transform{Size: size, Format: imagerequest.FormatPNG})
if err != nil {
t.Errorf("81920x10 SVG as PNG at 100x100: %v", err)
} else if result.Width != 100 || result.Height != 100 {
t.Errorf("81920x10 SVG as PNG at 100x100 = %dx%d, want 100x100",
result.Width, result.Height)
}
for _, svg := range []string{tooThin, atTheLimit} {
result, output, err := processSVG(t, svg,
imagerequest.Transform{Format: imagerequest.FormatOriginal})
if err != nil || result.ContentType != imagerequest.FormatSVG.MIMEType() ||
string(output) != svg {
t.Errorf("orig of %s: error = %v, want the SVG itself", svg, err)
}
}
}
// testSVGRenderTimeout is the time limit of the SVG time limit tests, short
// so that they finish in a few seconds. Under the race detector a child
// process takes about a second to start and read an SVG's size.
const testSVGRenderTimeout = 3 * time.Second
// processSVGTimed has processor process svg, as an SVG source, with transform,
// and returns the error and how long it took.
func processSVGTimed(
t *testing.T, processor *ImageProcessor, svg string,
transform imagerequest.Transform,
) (time.Duration, error) {
t.Helper()
start := time.Now()
_, err := processor.Process(t.Context(), strings.NewReader(svg),
imagerequest.FormatSVG, &transform)
return time.Since(start), err
}
// checkProcessesAnotherImage checks that processor, after refusing an SVG,
// processes a PNG.
func checkProcessesAnotherImage(t *testing.T, processor *ImageProcessor) {
t.Helper()
_, err := processor.Process(t.Context(),
bytes.NewReader(createTestPNG(t, 64, 48)), imagerequest.FormatPNG,
&imagerequest.Transform{
Size: imagerequest.Size{Width: 32, Height: 24},
Format: imagerequest.FormatJPEG,
})
if err != nil {
t.Errorf("a PNG after the SVG: %v", err)
}
}
// TestImageProcessor_StopsSlowSVGRender checks that an SVG that takes about
// 12 seconds to render, 600 shapes each drawn with an feTurbulence filter
// effect, is ErrSVGRenderFailed soon after a time limit of
// testSVGRenderTimeout, not when the render would end. The processor then
// processes another image.
func TestImageProcessor_StopsSlowSVGRender(t *testing.T) {
t.Parallel()
svg := `<svg xmlns="http://www.w3.org/2000/svg" width="256" height="256">` +
`<filter id="noise">` +
`<feTurbulence baseFrequency="0.05" numOctaves="4"/></filter>` +
strings.Repeat(`<rect width="256" height="256" filter="url(#noise)"/>`,
600) +
`</svg>`
processor := newSVGTestProcessor()
processor.svgRenderTimeout = testSVGRenderTimeout
elapsed, err := processSVGTimed(t, processor, svg, imagerequest.Transform{
Size: imagerequest.Size{Width: 256, Height: 256},
Format: imagerequest.FormatPNG,
})
t.Logf("slow SVG as PNG: error %v after %v", err, elapsed)
if !errors.Is(err, ErrSVGRenderFailed) {
t.Errorf("slow SVG as PNG: error = %v, want %v", err,
ErrSVGRenderFailed)
}
if elapsed > testSVGRenderTimeout+2*time.Second {
t.Errorf("slow SVG as PNG: refused after %v, want soon after %v",
elapsed, testSVGRenderTimeout)
}
checkProcessesAnotherImage(t, processor)
}
// TestImageProcessor_StopsSlowSVGSizeRead checks that orig of an SVG whose
// size takes about 10 seconds to read is ErrSVGRenderFailed soon after a time
// limit of testSVGRenderTimeout. To read the size, librsvg matches each of the
// SVG's 2500 CSS rules against each of its 2500 shapes, walking up the 75
// groups around it. The processor then processes another image.
func TestImageProcessor_StopsSlowSVGSizeRead(t *testing.T) {
t.Parallel()
var rules strings.Builder
for i := range 2500 {
fmt.Fprintf(&rules, ".group%d rect { fill: red }", i)
}
svg := `<svg xmlns="http://www.w3.org/2000/svg" width="16" height="12">` +
`<style>` + rules.String() + `</style>` + strings.Repeat(`<g>`, 75) +
strings.Repeat(`<rect width="1" height="1"/>`, 2500) +
strings.Repeat(`</g>`, 75) + `</svg>`
processor := newSVGTestProcessor()
processor.svgRenderTimeout = testSVGRenderTimeout
elapsed, err := processSVGTimed(t, processor, svg,
imagerequest.Transform{Format: imagerequest.FormatOriginal})
t.Logf("orig of the SVG: error %v after %v", err, elapsed)
if !errors.Is(err, ErrSVGRenderFailed) {
t.Errorf("orig of the SVG: error = %v, want %v", err,
ErrSVGRenderFailed)
}
if elapsed > testSVGRenderTimeout+2*time.Second {
t.Errorf("orig of the SVG: refused after %v, want soon after %v",
elapsed, testSVGRenderTimeout)
}
checkProcessesAnotherImage(t, processor)
}
// TestImageProcessor_StopsSVGRenderOverMemoryLimit checks that an SVG that
// needs about 500 MiB more than a child process's own memory to render, a
// chain of 2000 blurs that each keep their result, is ErrSVGRenderFailed with
// a memory limit of 512 MiB, and that a plain SVG renders under the same
// limit. A child process needs about 130 MiB of its own, and about 350 MiB
// under the race detector, so the chain also renders under MaxSVGRenderMemory.
func TestImageProcessor_StopsSVGRenderOverMemoryLimit(t *testing.T) {
t.Parallel()
var blurs strings.Builder
blurs.WriteString(`<feGaussianBlur stdDeviation="3" result="blur0"/>`)
for i := 1; i < 2000; i++ {
fmt.Fprintf(&blurs,
`<feGaussianBlur in="blur%d" stdDeviation="3" result="blur%d"/>`,
i-1, i)
}
svg := `<svg xmlns="http://www.w3.org/2000/svg" width="256" height="256">` +
`<filter id="blurs" color-interpolation-filters="sRGB">` +
blurs.String() + `</filter>` +
`<rect width="256" height="256" fill="red" filter="url(#blurs)"/></svg>`
size := imagerequest.Size{Width: 256, Height: 256}
processor := newSVGTestProcessor()
processor.maxSVGRenderMemory = 512 << 20
elapsed, err := processSVGTimed(t, processor, svg,
imagerequest.Transform{Size: size, Format: imagerequest.FormatPNG})
t.Logf("SVG of 2000 blurs as PNG: error %v after %v", err, elapsed)
if !errors.Is(err, ErrSVGRenderFailed) {
t.Errorf("SVG of 2000 blurs as PNG: error = %v, want %v", err,
ErrSVGRenderFailed)
}
_, err = processSVGTimed(t, processor, testSVG,
imagerequest.Transform{Size: size, Format: imagerequest.FormatPNG})
if err != nil {
t.Errorf("plain SVG as PNG after it: %v", err)
}
}
// TestSVGRenderSize checks the size at which an SVG is rendered to cover an
// output size, and that it is held to MaxInputDimension.
func TestSVGRenderSize(t *testing.T) {
t.Parallel()
tests := []struct {
name string
width, height int
outputWidth, outputHeight int
wantWidth, wantHeight int
}{
{"own size", 16, 12, 16, 12, 16, 12},
{"100 times larger", 16, 12, 1600, 1200, 1600, 1200},
{"wider than the SVG's proportions", 16, 12, 160, 12, 160, 120},
{"10 times smaller", 100, 100, 10, 10, 10, 10},
{"held to MaxInputDimension", 8192, 1, 8192, 8192, 8192, 1},
{"held to MaxInputDimension when enlarged", 1024, 1, 8192, 8192, 8192, 8},
{"far above MaxInputDimension", 1000000, 1000000, 100, 100, 100, 100},
}
for _, tt := range tests {
gotWidth, gotHeight := svgRenderSize(tt.width, tt.height,
tt.outputWidth, tt.outputHeight)
if gotWidth != tt.wantWidth || gotHeight != tt.wantHeight {
t.Errorf("%s: svgRenderSize(%d, %d, %d, %d) = %dx%d, want %dx%d",
tt.name, tt.width, tt.height, tt.outputWidth, tt.outputHeight,
gotWidth, gotHeight, tt.wantWidth, tt.wantHeight)
}
}
}
@@ -1,100 +0,0 @@
package imageproxy
import (
"bytes"
"os"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// TestProxy_Get_MakesAgainVariantOfNoOutputFormat replaces a cached
// variant's file with text, and checks that the next request is a miss
// answered with the variant made again, and the one after a hit.
func TestProxy_Get_MakesAgainVariantOfNoOutputFormat(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := variantFilePath(svc, imagerequest.CacheKey(req))
err := os.WriteFile(path, []byte("this is text, not an image"),
imgcache.StorageFilePerm)
if err != nil {
t.Fatalf("replacing the variant file: %v", err)
}
got, status := getImage(t, svc, req)
t.Logf("after the replacement: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the replacement: Get() = %d bytes, %s; want the %d bytes "+
"made again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestProxy_Get_HitWaitsForNoDatabaseWrite holds the database's one
// connection while a cached variant is requested. The hit must be answered
// while the connection is still held, and its access time must reach the
// database once it is free.
func TestProxy_Get_HitWaitsForNoDatabaseWrite(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
key := imagerequest.CacheKey(req)
getImage(t, svc, req)
// Each connection to an in-memory SQLite database opens a new, empty one,
// so the goroutine that writes the access times must share the
// connection that holds the schema.
svc.db.SetMaxOpenConns(1)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
anHourAgo := time.Now().Add(-time.Hour)
setVariantLastAccessed(t, svc.db, key, anHourAgo)
releaseDatabase := holdDatabase(t, svc.db)
defer releaseDatabase()
results := startGet(t.Context(), svc, req)
select {
case got := <-results:
if got.err != nil {
t.Fatalf("Get() of the cached variant error = %v", got.err)
}
case <-time.After(pendingAccessTimesWait):
t.Fatal("the cache hit waited for the database")
}
releaseDatabase()
deadline := time.Now().Add(pendingAccessTimesWait)
for countRows(t, svc.db, `
SELECT COUNT(*) FROM variant_content
WHERE cache_key = ? AND last_accessed_at > ?
`, string(key), anHourAgo.UTC().Format(sqliteTimestampFormat)) == 0 {
if time.Now().After(deadline) {
t.Fatal("the hit's access time was not written")
}
time.Sleep(10 * time.Millisecond)
}
}
@@ -1,493 +0,0 @@
package imageproxy
import (
"bytes"
"context"
"errors"
"fmt"
"image/color"
"io"
"net/http"
"os"
"sync"
"testing"
"time"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// Validators the test upstream sends with its image.
const (
testETag = `"v1"`
testLastModified = "Mon, 02 Nov 2026 10:00:00 GMT"
)
// errConnectionRefused stands for an upstream that cannot be reached.
var errConnectionRefused = errors.New("connection refused")
// upstreamFailure is one way a fetch from upstream fails.
type upstreamFailure struct {
name string
err error
}
// upstreamFailures returns the ways a check of a cached image with upstream
// can fail: an error status, a timeout and no connection.
func upstreamFailures() []upstreamFailure {
return []upstreamFailure{
{"error status", fmt.Errorf("%w: status 503", httpfetcher.ErrUpstreamError)},
{"timeout", httpfetcher.ErrUpstreamTimeout},
{"no connection", errConnectionRefused},
}
}
// conditionalUpstream serves one JPEG image with an ETag or a Last-Modified
// header, or both. It answers 304 Not Modified to a fetch whose If-None-Match
// is its ETag, or, when it has no ETag, whose If-Modified-Since is its
// Last-Modified. While err is set, every fetch fails with it. It records the
// conditional headers of every fetch.
type conditionalUpstream struct {
mu sync.Mutex
image []byte
etag string
lastModified string
err error
ifNoneMatch []string
ifModifiedSince []string
}
func (u *conditionalUpstream) Fetch(
_ context.Context, _ imagerequest.Source,
ifNoneMatch, ifModifiedSince string,
) (*httpfetcher.FetchResult, error) {
u.mu.Lock()
defer u.mu.Unlock()
u.ifNoneMatch = append(u.ifNoneMatch, ifNoneMatch)
u.ifModifiedSince = append(u.ifModifiedSince, ifModifiedSince)
if u.err != nil {
return nil, u.err
}
unchanged := ifNoneMatch == u.etag
if u.etag == "" {
unchanged = ifModifiedSince == u.lastModified
}
if unchanged {
return &httpfetcher.FetchResult{
Content: http.NoBody,
StatusCode: http.StatusNotModified,
}, nil
}
headers := http.Header{}
if u.etag != "" {
headers.Set("ETag", u.etag)
}
if u.lastModified != "" {
headers.Set("Last-Modified", u.lastModified)
}
return &httpfetcher.FetchResult{
Content: io.NopCloser(bytes.NewReader(u.image)),
ContentLength: int64(len(u.image)),
ContentType: testContentTypeJPEG,
Headers: headers,
StatusCode: http.StatusOK,
}, nil
}
// change replaces the upstream image and its ETag.
func (u *conditionalUpstream) change(image []byte, etag string) {
u.mu.Lock()
defer u.mu.Unlock()
u.image = image
u.etag = etag
}
// fail makes every later fetch fail with err.
func (u *conditionalUpstream) fail(err error) {
u.mu.Lock()
defer u.mu.Unlock()
u.err = err
}
// fetches returns how many fetches upstream was asked for.
func (u *conditionalUpstream) fetches() int {
u.mu.Lock()
defer u.mu.Unlock()
return len(u.ifNoneMatch)
}
// lastConditionalHeaders returns the If-None-Match and If-Modified-Since of
// the last fetch.
func (u *conditionalUpstream) lastConditionalHeaders() (string, string) {
u.mu.Lock()
defer u.mu.Unlock()
last := len(u.ifNoneMatch) - 1
return u.ifNoneMatch[last], u.ifModifiedSince[last]
}
// setupConditionalUpstreamProxy returns a proxy whose upstream is a
// conditionalUpstream serving a red JPEG with an ETag.
func setupConditionalUpstreamProxy(
t *testing.T,
) (*testProxy, *TestFixtures, *conditionalUpstream) {
t.Helper()
svc, fixtures := setupTestProxy(t)
upstream := &conditionalUpstream{
image: generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255}),
etag: testETag,
}
svc.fetcher = upstream
return svc, fixtures, upstream
}
// TestProxy_Get_FreshVariantAsksUpstreamNothing checks that a variant whose
// source was checked with upstream less than the cache TTL ago is served from
// the cache without asking upstream.
func TestProxy_Get_FreshVariantAsksUpstreamNothing(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
second, status := getImage(t, svc, variant)
t.Logf("cache status = %s, upstream fetches = %d", status, upstream.fetches())
if status != CacheHit {
t.Errorf("CacheStatus = %q, want %q", status, CacheHit)
}
if !bytes.Equal(first, second) {
t.Error("the cached variant differs from the one first served")
}
if n := upstream.fetches(); n != 1 {
t.Errorf("upstream fetches = %d, want 1", n)
}
}
// TestProxy_Get_UnchangedSourceAfterTTLIsNotDownloadedAgain caches a variant,
// sets its source's check time and the variant's file time two days back,
// and requests it again, with the source still cached and with the source's
// row and file removed, as eviction does. Upstream must be asked with the
// validator it sent, answer 304, and send no bytes; the variant must be served
// again without being made again, and then be served from the cache until the
// TTL passes.
func TestProxy_Get_UnchangedSourceAfterTTLIsNotDownloadedAgain(t *testing.T) {
t.Parallel()
tests := []struct {
name string
sourceEvicted bool
etag string
lastModified string
wantIfNoneMatch string
wantIfModifiedSince string
}{
{"ETag", false, testETag, "", testETag, ""},
{"Last-Modified only", false, "", testLastModified, "", testLastModified},
{"both", false, testETag, testLastModified, testETag, testLastModified},
{"ETag, source evicted", true, testETag, "", testETag, ""},
{"Last-Modified only, source evicted", true, "", testLastModified, "",
testLastModified},
{"both, source evicted", true, testETag, testLastModified, testETag,
testLastModified},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
upstream.etag = tt.etag
upstream.lastModified = tt.lastModified
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
if tt.sourceEvicted {
evictSource(t, svc, variant.Source)
}
second, status := getImage(t, svc, variant)
ifNoneMatch, ifModifiedSince := upstream.lastConditionalHeaders()
counters := readCounters(t, svc)
t.Logf("cache status = %s, If-None-Match = %q, If-Modified-Since = %q, "+
"counters = %+v", status, ifNoneMatch, ifModifiedSince, counters)
if ifNoneMatch != tt.wantIfNoneMatch {
t.Errorf("If-None-Match = %q, want %q", ifNoneMatch, tt.wantIfNoneMatch)
}
if ifModifiedSince != tt.wantIfModifiedSince {
t.Errorf("If-Modified-Since = %q, want %q",
ifModifiedSince, tt.wantIfModifiedSince)
}
if !bytes.Equal(first, second) {
t.Error("the variant served after the check differs from the cached one")
}
if counters.upstreamFetchCount != 1 || counters.transformCount != 1 {
t.Errorf("upstream downloads = %d, transforms = %d; want 1 and 1",
counters.upstreamFetchCount, counters.transformCount)
}
_, status = getImage(t, svc, variant)
if status != CacheHit {
t.Errorf("CacheStatus after the check = %q, want %q", status, CacheHit)
}
if n := upstream.fetches(); n != 2 {
t.Errorf("upstream fetches = %d, want 2", n)
}
})
}
}
// TestProxy_Get_ChangedSourceAfterTTLReplacesSourceAndVariants caches two
// variants of one source, changes the image upstream, and sets the source's
// check time and both variants' file times two days back. Each variant must
// then be made again from the new image, which is downloaded once.
func TestProxy_Get_ChangedSourceAfterTTLReplacesSourceAndVariants(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variants := []*imagerequest.Request{
photoVariant(fixtures, 85, imagerequest.FitCover),
photoVariant(fixtures, 40, imagerequest.FitCover),
}
oldImages := make([][]byte, len(variants))
for i, variant := range variants {
oldImages[i], _ = getImage(t, svc, variant)
}
newSource := generateTestJPEG(t, 120, 90, color.RGBA{0, 0, 255, 255})
upstream.change(newSource, `"v2"`)
twoDaysAgo := time.Now().AddDate(0, 0, -2)
for _, variant := range variants {
setBackSourceAndVariant(t, svc, variant, twoDaysAgo)
}
for i, variant := range variants {
image, _ := getImage(t, svc, variant)
if bytes.Equal(image, oldImages[i]) {
t.Errorf("variant %d is still made from the old image", i)
}
}
counters := readCounters(t, svc)
t.Logf("counters = %+v, upstream fetches = %d", counters, upstream.fetches())
if counters.upstreamFetchCount != 2 {
t.Errorf("upstream downloads = %d, want 2", counters.upstreamFetchCount)
}
cached, err := os.ReadFile(sourceFilePath(svc, variants[0].Source.Key()))
if err != nil {
t.Fatalf("reading the cached source: %v", err)
}
if !bytes.Equal(cached, newSource) {
t.Error("the cached source is not the new image")
}
}
// TestProxy_Get_UpstreamFailureAfterTTLServesCachedImage caches a variant,
// sets its source's check time and the variant's file time two days back, and
// makes upstream fail. The cached variant must still be served, and a variant
// not yet cached must be made from the cached source.
func TestProxy_Get_UpstreamFailureAfterTTLServesCachedImage(t *testing.T) {
t.Parallel()
for _, failure := range upstreamFailures() {
t.Run(failure.name, func(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
upstream.fail(failure.err)
second, _ := getImage(t, svc, variant)
t.Logf("upstream fetches = %d", upstream.fetches())
if upstream.fetches() != 2 {
t.Errorf("upstream fetches = %d, want 2", upstream.fetches())
}
if !bytes.Equal(first, second) {
t.Error("the variant served while upstream fails differs from the cached one")
}
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
})
}
}
// TestProxy_Get_FailingUpstreamIsAskedOnceWhileImageIsServedOften caches a
// variant, sets its source's check time and the variant's file time two days
// back, and makes every fetch time out. Serving the variant ten times, and a
// variant not yet cached, must ask upstream once: the failed check puts the
// source URL in the negative cache.
func TestProxy_Get_FailingUpstreamIsAskedOnceWhileImageIsServedOften(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
upstream.fail(httpfetcher.ErrUpstreamTimeout)
for range 10 {
getImage(t, svc, variant)
}
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
t.Logf("upstream fetches = %d", upstream.fetches())
if n := upstream.fetches(); n != 2 {
t.Errorf("upstream fetches = %d, want 2: the first fetch and one check", n)
}
}
// TestProxy_Get_UpstreamFailureAfterTTLServesVariantOfEvictedSource caches a
// variant, sets its file time two days back, removes its source's row and
// file, as eviction does, and makes upstream fail. The cached variant must
// still be served.
func TestProxy_Get_UpstreamFailureAfterTTLServesVariantOfEvictedSource(t *testing.T) {
t.Parallel()
for _, failure := range upstreamFailures() {
t.Run(failure.name, func(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
evictSource(t, svc, variant.Source)
upstream.fail(failure.err)
second, _ := getImage(t, svc, variant)
t.Logf("upstream fetches = %d", upstream.fetches())
if !bytes.Equal(first, second) {
t.Error("the variant served while upstream fails differs from the cached one")
}
if n := upstream.fetches(); n != 2 {
t.Errorf("upstream fetches = %d, want 2", n)
}
})
}
}
// TestProxy_Get_FailingUpstreamIsAskedOnceForVariantOfEvictedSource caches a
// variant, sets its file time two days back, removes its source's row and
// file, as eviction does, and makes every fetch time out. Serving the variant
// ten times must ask upstream once: the failed check puts the source URL in
// the negative cache.
func TestProxy_Get_FailingUpstreamIsAskedOnceForVariantOfEvictedSource(t *testing.T) {
t.Parallel()
svc, fixtures, upstream := setupConditionalUpstreamProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
first, _ := getImage(t, svc, variant)
setBackSourceAndVariant(t, svc, variant, time.Now().AddDate(0, 0, -2))
evictSource(t, svc, variant.Source)
upstream.fail(httpfetcher.ErrUpstreamTimeout)
for range 10 {
image, _ := getImage(t, svc, variant)
if !bytes.Equal(image, first) {
t.Fatal("the variant served while upstream fails differs from the cached one")
}
}
t.Logf("upstream fetches = %d", upstream.fetches())
if n := upstream.fetches(); n != 2 {
t.Errorf("upstream fetches = %d, want 2: the first fetch and one check", n)
}
}
// evictSource removes the row and then the file of src's cached source, as
// the cache's eviction does, and leaves its variants cached.
func evictSource(t *testing.T, svc *testProxy, src imagerequest.Source) {
t.Helper()
_, err := svc.db.ExecContext(t.Context(),
`DELETE FROM source_content WHERE source_key = ?`, string(src.Key()))
if err != nil {
t.Fatalf("deleting the source's row: %v", err)
}
err = os.Remove(sourceFilePath(svc, src.Key()))
if err != nil {
t.Fatalf("removing the source's file: %v", err)
}
}
// setBackSourceAndVariant sets back to when the times the cache keeps for
// req's source and for the variant req asks for: the source's fetch and check
// times, both last access times, the variant's check time, and the variant
// file's modification time.
func setBackSourceAndVariant(
t *testing.T, svc *testProxy, req *imagerequest.Request, when time.Time,
) {
t.Helper()
key := req.Source.Key()
setSourceLastAccessed(t, svc.db, key, when)
_, err := svc.db.ExecContext(t.Context(),
`UPDATE source_content SET fetched_at = ?, checked_at = ?
WHERE source_key = ?`, when.UTC(), when.UTC(), string(key))
if err != nil {
t.Fatalf("setting back the source's fetch and check times: %v", err)
}
variantKey := imagerequest.CacheKey(req)
setVariantLastAccessed(t, svc.db, variantKey, when)
_, err = svc.db.ExecContext(t.Context(),
`UPDATE variant_content SET checked_at = ? WHERE cache_key = ?`,
when.UTC(), string(variantKey))
if err != nil {
t.Fatalf("setting back the variant's check time: %v", err)
}
path := variantFilePath(svc, variantKey)
err = os.Chtimes(path, when, when)
if err != nil {
t.Fatalf("setting back the times of %s: %v", path, err)
}
}
@@ -1,77 +0,0 @@
package imageproxy
import (
"errors"
"testing"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/testutil"
"sneak.berlin/go/pixa/internal/imgcache"
)
// TestNew_RegistersItsCountersOnItsRegistry checks that New registers the
// proxy's counters on the registry it is given, under the names README.md
// gives them. The cache registers its own counters on another registry.
func TestNew_RegistersItsCountersOnItsRegistry(t *testing.T) {
t.Parallel()
cache, err := imgcache.NewCache(setupTestDB(t),
imgcache.Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err)
}
registry := prometheus.NewRegistry()
_, err = New(&Config{Cache: cache}, registry)
if err != nil {
t.Fatalf("New() error = %v", err)
}
names := []string{
"pixa_upstream_fetches_total",
"pixa_upstream_fetch_bytes_total",
"pixa_transforms_total",
}
count, err := testutil.GatherAndCount(registry, names...)
if err != nil {
t.Fatalf("testutil.GatherAndCount() error = %v", err)
}
if count != len(names) {
t.Errorf("%d of the counters %v are on the registry, want %d",
count, names, len(names))
}
}
// TestNew_FailsWhenItsCountersCannotBeRegistered makes a second proxy on
// the registry of a first. Its counters have the same names, so the
// registry refuses them, and New must return that error.
func TestNew_FailsWhenItsCountersCannotBeRegistered(t *testing.T) {
t.Parallel()
cache, err := imgcache.NewCache(setupTestDB(t),
imgcache.Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err)
}
registry := prometheus.NewRegistry()
_, err = New(&Config{Cache: cache}, registry)
if err != nil {
t.Fatalf("New() of the first proxy error = %v", err)
}
_, err = New(&Config{Cache: cache}, registry)
t.Logf("New() of the second proxy error = %v", err)
var alreadyRegistered prometheus.AlreadyRegisteredError
if !errors.As(err, &alreadyRegistered) {
t.Errorf("New() of the second proxy error = %v, want %T",
err, alreadyRegistered)
}
}
-933
View File
@@ -1,933 +0,0 @@
// Package imageproxy fetches, transforms, caches and serves images. Its
// Proxy serves an image from the cache when it can, and otherwise makes it
// from the cached or fetched source image.
package imageproxy
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"runtime/debug"
"time"
"github.com/dustin/go-humanize"
"github.com/getsentry/sentry-go"
"github.com/go-chi/chi/v5/middleware"
"github.com/prometheus/client_golang/prometheus"
"golang.org/x/sync/singleflight"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic"
)
// Proxy orchestrates the cache, fetcher, and processor.
type Proxy struct {
cache *imgcache.Cache
fetcher httpfetcher.Fetcher
processor *imageprocessor.ImageProcessor
log *slog.Logger
maxResponseSize int64
// variantsInProgress lets the requests that miss the same variant at the
// same time share one fetch and one transcode.
variantsInProgress singleflight.Group
// The metrics of upstream fetches and transforms, made and registered by
// New. They are kept in memory, so updating one waits for nothing.
upstreamFetches prometheus.Counter
upstreamFetchBytes prometheus.Counter
upstreamFetchDurations prometheus.Histogram
upstreamFetchErrors prometheus.Counter
transforms prometheus.Counter
transformDurations prometheus.Histogram
transformsInProgress prometheus.Gauge
}
// Config holds the proxy's configuration.
type Config struct {
// Cache is the cache instance
Cache *imgcache.Cache
// Fetcher fetches source images from upstream hosts
Fetcher httpfetcher.Fetcher
// Processor converts and resizes the images
Processor *imageprocessor.ImageProcessor
// MaxResponseSize is the largest source accepted, in bytes; a cached
// source larger than this is fetched again
MaxResponseSize int64
// Logger for logging
Logger *slog.Logger
}
// errCacheRequired is returned when the proxy is built without a cache.
var errCacheRequired = errors.New("cache is required")
// New creates a new image proxy and registers its metrics on
// metricsRegistry. pixad passes Prometheus' default registry, the one
// /metrics serves. It fails when metricsRegistry already holds metrics of
// the same names.
func New(
cfg *Config, metricsRegistry prometheus.Registerer,
) (*Proxy, error) {
if cfg.Cache == nil {
return nil, errCacheRequired
}
log := cfg.Logger
if log == nil {
log = slog.Default()
}
// JPEG XL is the default output format, so pixad does not start
// without it.
err := imageprocessor.CheckJPEGXLSupport()
if err != nil {
return nil, err
}
p := &Proxy{
cache: cfg.Cache,
fetcher: cfg.Fetcher,
processor: cfg.Processor,
log: log,
maxResponseSize: cfg.MaxResponseSize,
upstreamFetches: prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetches_total",
Help: "Fetches of a source image from upstream that read any bytes.",
}),
upstreamFetchBytes: prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetch_bytes_total",
Help: "Bytes read from upstream by fetches of source images.",
}),
upstreamFetchDurations: prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "pixa_upstream_fetch_duration_seconds",
Help: "Duration of the fetches pixa_upstream_fetches_total counts, " +
"until the response's last byte.",
}),
upstreamFetchErrors: prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetch_errors_total",
Help: "Fetches of a source image from upstream that failed, " +
"before or while reading the response.",
}),
transforms: prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_transforms_total",
Help: "Images the image processor transformed.",
}),
transformDurations: prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "pixa_transform_duration_seconds",
Help: "Duration of the transforms pixa_transforms_total counts, " +
"waiting for a processing slot included.",
}),
transformsInProgress: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "pixa_transforms_in_progress",
Help: "Images being transformed or waiting for a processing slot.",
}),
}
for _, metric := range []prometheus.Collector{
p.upstreamFetches, p.upstreamFetchBytes, p.upstreamFetchDurations,
p.upstreamFetchErrors, p.transforms, p.transformDurations,
p.transformsInProgress,
} {
err = metricsRegistry.Register(metric)
if err != nil {
return nil, fmt.Errorf("failed to register the proxy's metrics: %w", err)
}
}
return p, nil
}
// ErrNegativeCached is returned when a URL is in the negative cache (recently failed).
var ErrNegativeCached = errors.New("request is in negative cache (recently failed)")
// ErrProcessingTimedOut is returned when processing an image stopped because
// the downstream_timeout of the request doing it ended, to that request and to
// every request that waited for its image.
var ErrProcessingTimedOut = errors.New(
"image processing stopped at the deadline of the request doing it")
// ImageResponse represents a processed image ready to serve
type ImageResponse struct {
// Content is the image data reader
Content io.ReadCloser
// ContentLength is the size in bytes (-1 if unknown)
ContentLength int64
// ContentType is the MIME type of the response
ContentType string
// ETag is the entity tag for caching
ETag string
// CacheStatus indicates HIT or MISS
CacheStatus CacheStatus
// FetchedBytes is the number of bytes fetched from upstream (0 if cache hit)
FetchedBytes int64
}
// CacheStatus indicates whether the response was served from cache.
type CacheStatus string
// Cache status values for response headers.
const (
CacheHit CacheStatus = "HIT"
CacheMiss CacheStatus = "MISS"
)
// Get retrieves a processed image, fetching and processing if necessary.
// While its source URL is in the negative cache, a cached variant is still
// served and a variant not in the cache is still made from a cached source;
// only a request that needs the source fetched is refused.
//
// A cache hit opens the variant file and makes no database call: the cache
// hands its access time to the goroutine that writes access times, without
// waiting for it, and its hit is counted in memory. A variant whose source
// was last checked with upstream CacheTTL or more ago is a miss, which checks
// the source first (see processFromSourceOrFetch). A variant file that
// cannot be served, such as one whose first bytes are not those of an output
// format, is a miss, and the variant is made again.
func (p *Proxy) Get(
ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) {
cacheKey := imagerequest.CacheKey(req)
cached, err := p.cache.Variant(cacheKey)
if err == nil {
p.cache.IncrementStats(true)
return &ImageResponse{
Content: cached.Content,
ContentLength: cached.Size,
ContentType: cached.ContentType,
CacheStatus: CacheHit,
ETag: formatETag(cacheKey, cached.CheckedAt),
}, nil
}
if !errors.Is(err, imgcache.ErrNotFound) && !errors.Is(err, imgcache.ErrStale) {
p.log.Warn("cached variant not served, making it again",
"cache_key", cacheKey, "error", err)
}
// Cache miss - get the variant, processed once for all the requests that
// miss it at the same time, then count this request's miss, also when it
// failed or the request context has ended meanwhile
response, err := p.processOrWait(ctx, req)
p.cache.IncrementStats(false)
if err != nil {
return nil, err
}
response.CacheStatus = CacheMiss
return response, nil
}
// WaitForProcessing waits until no image is being processed, or until ctx
// ends, and returns how many images were still being processed then.
func (p *Proxy) WaitForProcessing(ctx context.Context) int {
return p.processor.WaitForProcessing(ctx)
}
// errPanicked is returned when processing a variant panicked.
var errPanicked = errors.New("panic while processing image")
// processOrWait returns the variant req asks for. The first of the requests
// that miss a variant at the same time processes it, and singleflight hands
// its result, or its error, to the others: they fetch nothing, read no source
// and take no upstream connection or processing slot. The processing ignores
// the first request's cancellation, so the others are still served if that
// client goes away, but keeps its deadline.
func (p *Proxy) processOrWait(
ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) {
// A request that has already ended starts no processing
if ctx.Err() != nil {
return nil, ctx.Err()
}
cacheKey := imagerequest.CacheKey(req)
// Closed when this request's own function runs, which singleflight does
// only when no other request is processing the variant
processing := make(chan struct{})
results := p.variantsInProgress.DoChan(string(cacheKey),
func() (_ any, err error) {
close(processing)
// singleflight would raise a panic again in a goroutine of its
// own, where no handler recovers it, and stop pixad. It is
// reported through the Sentry hub that the Sentry middleware puts
// on the request's context when sentry_dsn is set.
defer func() {
recovered := recover()
if recovered != nil {
p.log.Error("panic while processing image",
"host", req.Source.Host, "path", req.Source.Path,
"panic", recovered, "stack", string(debug.Stack()))
if hub := sentry.GetHubFromContext(ctx); hub != nil {
hub.RecoverWithContext(ctx, recovered)
}
err = fmt.Errorf("%w: %v", errPanicked, recovered)
}
}()
processingCtx, cancel := withoutCancelKeepingDeadline(ctx)
defer cancel()
return p.processFromSourceOrFetch(processingCtx, req, cacheKey)
})
var result singleflight.Result
select {
case result = <-results:
case <-ctx.Done():
select {
case <-processing:
// This request is processing the variant: it waits for the
// result, as every request did before misses were shared
result = <-results
default:
// Another request is processing the variant, or this request's
// function has not started yet; the processing goes on without it
return nil, ctx.Err()
}
}
if result.Err != nil {
return nil, result.Err
}
variant, _ := result.Val.(*processedVariant)
return &ImageResponse{
Content: io.NopCloser(bytes.NewReader(variant.data)),
ContentLength: int64(len(variant.data)),
ContentType: variant.contentType,
FetchedBytes: variant.fetchedBytes,
ETag: formatETag(cacheKey, variant.checkedAt),
}, nil
}
// withoutCancelKeepingDeadline returns ctx without its cancellation but with
// its deadline, if it has one, and the func that releases the returned
// context.
func withoutCancelKeepingDeadline(
ctx context.Context,
) (context.Context, context.CancelFunc) {
deadline, hasDeadline := ctx.Deadline()
if !hasDeadline {
return context.WithoutCancel(ctx), func() {}
}
return context.WithDeadline(context.WithoutCancel(ctx), deadline)
}
// loadCachedSource opens src's cached source, reading only the first bytes,
// which give its format, and returns it with its size and its format. It
// returns nil if the source is not in the cache, cannot be read, is in no
// source format or exceeds maxResponseSize.
func (p *Proxy) loadCachedSource(
ctx context.Context, src imagerequest.Source,
) (io.ReadCloser, int64, imagerequest.Format) {
reader, size, format, err := p.cache.Source(ctx, src)
if errors.Is(err, imgcache.ErrNotFound) {
return nil, 0, ""
}
if err != nil {
p.log.Warn("failed to load cached source, fetching", "error", err)
return nil, 0, ""
}
p.log.Debug("using cached source", "source_key", src.Key())
if size > p.maxResponseSize {
_ = reader.Close()
p.log.Warn("cached source exceeds max response size, discarding",
"source_key", src.Key(),
"max_bytes", p.maxResponseSize,
)
return nil, 0, ""
}
return reader, size, format
}
// processFromSourceOrFetch makes the variant req asks for, from the cached
// source when there is one and from a fetched source otherwise. It counts a
// source cache miss when it fetches the source because the source has no row
// in the cache or its cached file cannot be used, and a source cache hit
// otherwise.
//
// A cached source last checked with upstream CacheTTL or more ago is checked
// first: upstream is asked to send it only if it changed since. When it
// changed, the new bytes replace the cached source and the variant is made
// from them. When it did not change, or the check fails, the cached source
// is used: the variant's cached file is served again if it was made from the
// cached source's bytes, and otherwise the variant is made from them. When
// the source is not cached but the variant is, the variant is checked with
// upstream the same way (see fetchAndProcess).
func (p *Proxy) processFromSourceOrFetch(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
) (*processedVariant, error) {
cached, err := p.cache.LookupSource(ctx, req.Source)
if err != nil {
if !errors.Is(err, imgcache.ErrNotFound) {
p.log.Warn("failed to look up cached source, fetching", "error", err)
}
p.cache.IncrementSourceStats(false)
return p.fetchAndProcess(ctx, req, cacheKey)
}
if p.cache.Stale(cached.CheckedAt) {
fetched, err := p.fetchSource(ctx, req, cached.ETag, cached.LastModified)
switch {
case errors.Is(err, errSourceNotModified):
cached.CheckedAt = time.Now().UTC()
p.renewSource(ctx, req.Source, cached.CheckedAt)
case err != nil:
p.recordFailedCheck(ctx, req, err)
default:
p.cache.IncrementSourceStats(true)
return p.processFetched(ctx, req, cacheKey, fetched)
}
}
variant := p.renewVariant(ctx, cacheKey, cached)
if variant != nil {
p.cache.IncrementSourceStats(true)
return variant, nil
}
source, sourceSize, sourceFormat := p.loadCachedSource(ctx, req.Source)
p.cache.IncrementSourceStats(source != nil)
// Fetch from upstream if we don't have source data or it's empty
if source == nil {
return p.fetchAndProcess(ctx, req, cacheKey)
}
defer func() { _ = source.Close() }()
// Process using cached source; nothing was fetched from upstream. The
// image processor reads the source only once it has a processing slot,
// so a request waiting for one holds none of it in memory.
return p.processAndStore(ctx, req, cacheKey, cached, source, sourceFormat,
sourceSize)
}
// recordFailedCheck logs a failed check of req's source with upstream, after
// which the image is still served from the cache. It puts the source URL in
// the negative cache, whatever the failure, a timeout included, so upstream
// is asked at most once per NegativeTTL while it fails, however often the
// image is served. A failure that was a negative cache hit records nothing.
func (p *Proxy) recordFailedCheck(
ctx context.Context, req *imagerequest.Request, err error,
) {
if errors.Is(err, ErrNegativeCached) {
return
}
p.log.Warn("checking source with upstream failed, serving from the cache",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host, "path", req.Source.Path, "error", err)
p.storeNegative(ctx, req)
}
// renewSource records that upstream answered at checkedAt that src's cached
// source had not changed. A failure to record it is only logged: the source
// is then checked again on its next use.
func (p *Proxy) renewSource(
ctx context.Context, src imagerequest.Source, checkedAt time.Time,
) {
err := p.cache.RenewSource(ctx, src, checkedAt)
if err != nil {
p.log.Warn("failed to renew cached source", "source_key", src.Key(),
"error", err)
}
}
// renewVariant returns the cached variant of cacheKey when it was made from
// the bytes of the cached source that source describes, and renews it to
// source.CheckedAt (see imgcache.Cache.RenewVariant). It returns nil when
// there is no such variant or it cannot be read.
func (p *Proxy) renewVariant(
ctx context.Context, cacheKey imagerequest.VariantKey,
source imgcache.SourceInfo,
) *processedVariant {
cached, err := p.cache.RenewVariant(ctx, cacheKey, source)
if err != nil {
if !errors.Is(err, imgcache.ErrNotFound) {
p.log.Warn("cached variant not renewed, making it again",
"cache_key", cacheKey, "error", err)
}
return nil
}
return p.readCachedVariant(cacheKey, cached)
}
// cachedVariantIgnoringTTL returns the cached variant of cacheKey whatever
// its age, with what its row keeps about the source bytes it was made from
// (see imgcache.Cache.LookupVariant). It returns nil when the variant is not
// cached or cannot be read. The SourceInfo is empty when the variant is not
// returned, or its row is missing or cannot be read.
func (p *Proxy) cachedVariantIgnoringTTL(
ctx context.Context, cacheKey imagerequest.VariantKey,
) (*processedVariant, imgcache.SourceInfo) {
cached, err := p.cache.VariantIgnoringTTL(cacheKey)
if err != nil {
if !errors.Is(err, imgcache.ErrNotFound) {
p.log.Warn("cached variant not served", "cache_key", cacheKey,
"error", err)
}
return nil, imgcache.SourceInfo{}
}
variant := p.readCachedVariant(cacheKey, cached)
if variant == nil {
return nil, imgcache.SourceInfo{}
}
madeFrom, err := p.cache.LookupVariant(ctx, cacheKey)
if err != nil && !errors.Is(err, imgcache.ErrNotFound) {
p.log.Warn("failed to look up cached variant", "cache_key", cacheKey,
"error", err)
}
return variant, madeFrom
}
// readCachedVariant reads the cached variant of cacheKey from its opened file,
// and closes the file. It returns nil when the variant cannot be read.
func (p *Proxy) readCachedVariant(
cacheKey imagerequest.VariantKey, file imgcache.VariantFile,
) *processedVariant {
defer func() { _ = file.Content.Close() }()
data, err := io.ReadAll(file.Content)
if err != nil {
p.log.Warn("failed to read cached variant", "cache_key", cacheKey,
"error", err)
return nil
}
return &processedVariant{
data: data,
contentType: file.ContentType,
checkedAt: file.CheckedAt,
}
}
// fetchAndProcess fetches the source from upstream, stores it, and makes
// and stores the variant from it.
//
// The variant can still be cached after eviction removed its source. The
// fetch then sends the ETag and Last-Modified of the source bytes the variant
// was made from, so upstream sends the source only if it changed since. When
// it did not change, the cached variant is renewed and served, with no
// download and no conversion. When the fetch fails, the cached variant is
// served whatever its age, and the failure is recorded as a failed check (see
// recordFailedCheck).
func (p *Proxy) fetchAndProcess(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
) (*processedVariant, error) {
variant, madeFrom := p.cachedVariantIgnoringTTL(ctx, cacheKey)
fetched, err := p.fetchSource(ctx, req, madeFrom.ETag, madeFrom.LastModified)
switch {
case err == nil:
return p.processFetched(ctx, req, cacheKey, fetched)
case variant == nil:
return nil, err
case errors.Is(err, errSourceNotModified):
variant.checkedAt = time.Now().UTC()
err = p.cache.SetVariantCheckedAt(ctx, cacheKey, variant.checkedAt)
if err != nil {
p.log.Warn("failed to renew cached variant", "cache_key", cacheKey,
"error", err)
}
default:
p.recordFailedCheck(ctx, req, err)
}
return variant, nil
}
// processFetched makes and stores the variant from the source fetchSource
// fetched, then closes the fetch's response.
func (p *Proxy) processFetched(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
fetched *fetchedSource,
) (*processedVariant, error) {
// Closing the response frees the upstream connection. It is closed only
// after processing, so the fetcher's connection limit also bounds the
// fetched sources held in memory while their requests wait for a
// processing slot.
defer func() { _ = fetched.response.Close() }()
return p.processAndStore(ctx, req, cacheKey, fetched.info,
bytes.NewReader(fetched.data), fetched.format, int64(len(fetched.data)))
}
// errSourceNotModified is returned by fetchSource when upstream answers 304
// Not Modified: the cached source has not changed.
var errSourceNotModified = errors.New("upstream source not modified")
// fetchedSource is a source fetchSource fetched and stored in the cache.
type fetchedSource struct {
data []byte
format imagerequest.Format
// info is what the cache keeps about the source
info imgcache.SourceInfo
// response is the fetch's response, still open; closing it frees the
// upstream connection
response io.Closer
}
// fetchSource fetches req's source from upstream and stores it in the cache.
// A non-empty ifNoneMatch or ifModifiedSince asks upstream to send the source
// only if it changed since; errSourceNotModified says it did not. It counts
// the fetch with the bytes read from upstream and its duration, including
// when reading the response or a later step fails, and counts a fetch that
// fails or whose response cannot be read as an upstream fetch error. It
// fetches nothing while the source URL is in the negative cache. The caller
// closes the returned response.
func (p *Proxy) fetchSource(
ctx context.Context,
req *imagerequest.Request,
ifNoneMatch, ifModifiedSince string,
) (*fetchedSource, error) {
err := p.refuseNegativeCached(ctx, req)
if err != nil {
return nil, err
}
// Fetch from upstream
p.log.Debug("fetching from upstream", "host", req.Source.Host,
"path", req.Source.Path, "query", req.Source.Query)
fetchStart := time.Now()
fetchResult, err := p.fetcher.Fetch(ctx, req.Source,
ifNoneMatch, ifModifiedSince)
if err != nil {
p.upstreamFetchErrors.Inc()
// Store negative cache for certain errors
if isNegativeCacheable(err) {
p.storeNegative(ctx, req)
}
return nil, fmt.Errorf("upstream fetch failed: %w", err)
}
if fetchResult.StatusCode == http.StatusNotModified {
_ = fetchResult.Content.Close()
p.log.Info("upstream source not modified",
"request_id", middleware.GetReqID(ctx), "host", req.Source.Host,
"path", req.Source.Path, "fetch_ms", fetchResult.FetchDurationMs)
return nil, errSourceNotModified
}
// Read and validate the source content
sourceData, err := io.ReadAll(fetchResult.Content)
fetchBytes := int64(len(sourceData))
// Counted also when the request context has ended meanwhile
p.incrementUpstreamFetch(fetchBytes, time.Since(fetchStart))
if err != nil {
p.upstreamFetchErrors.Inc()
_ = fetchResult.Content.Close()
return nil, fmt.Errorf("failed to read upstream response: %w", err)
}
p.logUpstreamFetch(ctx, req, fetchResult, fetchBytes)
// The source's format, detected from its first bytes, which must match
// its content type. A body that does not is the upstream's error, so it
// gets a negative cache entry, as an upstream error status does.
sourceFormat, err := magic.ValidateMagicBytes(sourceData,
fetchResult.ContentType)
if err != nil {
p.log.Warn("source refused", "request_id", middleware.GetReqID(ctx),
"host", req.Source.Host, "path", req.Source.Path,
"content_type", fetchResult.ContentType, "error", err)
p.storeNegative(ctx, req)
_ = fetchResult.Content.Close()
return nil, fmt.Errorf("content validation failed: %w", err)
}
info := fetchedSourceInfo(fetchResult)
// Store source content
_, err = p.cache.PutSource(ctx, req.Source, bytes.NewReader(sourceData), info)
if err != nil {
p.log.Warn("failed to store source content", "error", err)
// Continue even if caching fails
}
return &fetchedSource{
data: sourceData,
format: sourceFormat,
info: info,
response: fetchResult.Content,
}, nil
}
// refuseNegativeCached returns an error wrapping ErrNegativeCached while
// req's source URL is in the negative cache, and nil otherwise. A failure to
// read the negative cache is only logged, and returns nil.
func (p *Proxy) refuseNegativeCached(
ctx context.Context, req *imagerequest.Request,
) error {
negHit, err := p.cache.Failure(ctx, req.Source)
if err != nil {
p.log.Warn("negative cache check failed", "error", err)
}
if !negHit {
return nil
}
p.log.Debug("negative cache hit",
"host", req.Source.Host,
"path", req.Source.Path,
)
return fmt.Errorf("%w: %w", httpfetcher.ErrUpstreamError, ErrNegativeCached)
}
// fetchedSourceInfo returns what the cache keeps about a source upstream
// just sent: fetched and checked now, with the ETag and Last-Modified
// headers of fetchResult.
func fetchedSourceInfo(fetchResult *httpfetcher.FetchResult) imgcache.SourceInfo {
now := time.Now().UTC()
upstreamHeaders := http.Header(fetchResult.Headers)
return imgcache.SourceInfo{
FetchedAt: now,
CheckedAt: now,
ETag: upstreamHeaders.Get("ETag"),
LastModified: upstreamHeaders.Get("Last-Modified"),
}
}
// logUpstreamFetch logs a fetch of req's source that read fetchBytes bytes,
// with its download rate and connection details.
func (p *Proxy) logUpstreamFetch(
ctx context.Context,
req *imagerequest.Request,
fetchResult *httpfetcher.FetchResult,
fetchBytes int64,
) {
var downloadRate string
if fetchResult.FetchDurationMs > 0 {
seconds := float64(fetchResult.FetchDurationMs) / 1000.0 //nolint:mnd // ms to seconds
bitsPerSecond := float64(fetchBytes*8) / seconds //nolint:mnd // bytes to bits
downloadRate = humanize.SI(bitsPerSecond, "bps")
}
p.log.Info("upstream fetched",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host,
"path", req.Source.Path,
"bytes", fetchBytes,
"fetch_ms", fetchResult.FetchDurationMs,
"rate", downloadRate,
"remote_addr", fetchResult.RemoteAddr,
"http", fetchResult.HTTPVersion,
"tls", fetchResult.TLSVersion,
"cipher", fetchResult.TLSCipherSuite,
)
}
// incrementUpstreamFetch counts one upstream fetch that read fetchBytes bytes
// and took duration. A fetch that read no bytes is not counted.
func (p *Proxy) incrementUpstreamFetch(fetchBytes int64, duration time.Duration) {
if fetchBytes <= 0 {
return
}
p.upstreamFetches.Inc()
p.upstreamFetchBytes.Add(float64(fetchBytes))
p.upstreamFetchDurations.Observe(duration.Seconds())
}
// storeNegative stores a negative cache entry for req's source, whose fetch
// failed, so the source is not fetched again until the entry expires. A
// failure to store the entry is only logged.
func (p *Proxy) storeNegative(ctx context.Context, req *imagerequest.Request) {
err := p.cache.PutFailure(ctx, req.Source)
if err != nil {
p.log.Warn("failed to store negative cache entry",
"host", req.Source.Host, "path", req.Source.Path, "error", err)
}
}
// processedVariant is a variant as processAndStore made it. Each request that
// shared its processing serves it through a reader of its own.
type processedVariant struct {
data []byte
contentType string
fetchedBytes int64
// checkedAt is its source's check time when the variant was made or last
// renewed, which its cached file has as its modification time
checkedAt time.Time
}
// processAndStore processes the image read from source, which is in
// sourceFormat, and stores the result. sourceInfo describes the source's
// bytes, and is stored with the variant (see imgcache.Cache.PutVariant).
func (p *Proxy) processAndStore(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
sourceInfo imgcache.SourceInfo,
source io.Reader,
sourceFormat imagerequest.Format,
fetchBytes int64,
) (*processedVariant, error) {
// Process the image
processStart := time.Now()
// ctx has the deadline of the request doing the processing, which can
// end while a request waiting for its image has time left
processResult, err := p.transform(ctx, source, sourceFormat, &req.Transform)
if errors.Is(err, context.DeadlineExceeded) {
return nil, fmt.Errorf("%w: %w", ErrProcessingTimedOut, err)
}
if err != nil {
return nil, fmt.Errorf("image processing failed: %w", err)
}
processDuration := time.Since(processStart)
// Counted also when the request context has ended meanwhile
p.transforms.Inc()
p.transformDurations.Observe(processDuration.Seconds())
// Read processed content
processedData, err := io.ReadAll(processResult.Content)
_ = processResult.Content.Close()
if err != nil {
return nil, fmt.Errorf("failed to read processed content: %w", err)
}
// Log conversion details
outputSize := int64(len(processedData))
var sizePercent float64
if fetchBytes > 0 {
//nolint:mnd // percentage calculation
sizePercent = float64(outputSize) / float64(fetchBytes) * 100.0
}
p.log.Info("image converted",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host,
"path", req.Source.Path,
"src_format", sourceFormat,
"dst_format", req.Transform.Format,
"src_bytes", fetchBytes,
"dst_bytes", outputSize,
"src_dimensions", fmt.Sprintf("%dx%d",
processResult.InputWidth, processResult.InputHeight),
"dst_dimensions", fmt.Sprintf("%dx%d",
processResult.Width, processResult.Height),
"size_ratio", fmt.Sprintf("%.1f%%", sizePercent),
"convert_ms", processDuration.Milliseconds(),
"quality", req.Transform.Quality,
"fit", req.Transform.FitMode,
)
// Store variant to cache
err = p.cache.PutVariant(ctx, cacheKey, sourceInfo,
bytes.NewReader(processedData))
if err != nil {
p.log.Warn("failed to store variant", "error", err)
// Continue even if caching fails
}
return &processedVariant{
data: processedData,
contentType: processResult.ContentType,
fetchedBytes: fetchBytes,
checkedAt: sourceInfo.CheckedAt,
}, nil
}
// transform has the image processor transform the image read from source
// as requested says, and counts it in pixa_transforms_in_progress until the
// processor returns or panics.
func (p *Proxy) transform(
ctx context.Context,
source io.Reader,
sourceFormat imagerequest.Format,
requested *imagerequest.Transform,
) (*imageprocessor.Result, error) {
p.transformsInProgress.Inc()
defer p.transformsInProgress.Dec()
return p.processor.Process(ctx, source, sourceFormat, requested)
}
// isNegativeCacheable returns true if the error should be cached.
func isNegativeCacheable(err error) bool {
return errors.Is(err, httpfetcher.ErrUpstreamError)
}
// etagHashLength is the number of hash characters to use for ETags.
const etagHashLength = 16
// formatETag returns the quoted ETag of the variant of key: the first
// etagHashLength characters of key, then checkedAt in Unix nanoseconds.
// checkedAt is its source's check time when the variant was made or last
// renewed, so a variant made again from new source bytes gets another ETag.
func formatETag(key imagerequest.VariantKey, checkedAt time.Time) string {
hash := string(key)
// Use first 16 characters of hash for a shorter but still unique ETag
if len(hash) > etagHashLength {
hash = hash[:etagHashLength]
}
return fmt.Sprintf(`"%s-%d"`, hash, checkedAt.UnixNano())
}
@@ -1,213 +0,0 @@
package imageproxy
import (
"os"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// metricValues holds the source cache, upstream fetch and transform metrics
// of a proxy and its cache. A histogram is given by its count of
// observations.
type metricValues struct {
sourceCacheHits uint64
sourceCacheMisses uint64
fetchDurations uint64
fetchErrors uint64
transformDuration uint64
inProgress float64
}
// readMetricValues reads the metrics of svc and its cache that
// metricValues holds from their metrics registry.
func readMetricValues(t *testing.T, svc *testProxy) metricValues {
t.Helper()
families, err := svc.registry.Gather()
if err != nil {
t.Fatalf("failed to gather the metrics: %v", err)
}
var values metricValues
for _, family := range families {
metric := family.GetMetric()[0]
switch family.GetName() {
case "pixa_source_cache_hits_total":
values.sourceCacheHits = uint64(metric.GetCounter().GetValue())
case "pixa_source_cache_misses_total":
values.sourceCacheMisses = uint64(metric.GetCounter().GetValue())
case "pixa_upstream_fetch_duration_seconds":
values.fetchDurations = metric.GetHistogram().GetSampleCount()
case "pixa_upstream_fetch_errors_total":
values.fetchErrors = uint64(metric.GetCounter().GetValue())
case "pixa_transform_duration_seconds":
values.transformDuration = metric.GetHistogram().GetSampleCount()
case "pixa_transforms_in_progress":
values.inProgress = metric.GetGauge().GetValue()
}
}
return values
}
// TestProxy_Get_CountsSourceCacheFetchesAndTransforms walks Get through a
// miss that fetches the source, a hit, a miss that uses the cached source
// and a miss whose source is not found upstream, checking the source cache,
// upstream fetch and transform metrics after each.
func TestProxy_Get_CountsSourceCacheFetchesAndTransforms(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
// want is source cache hits, source cache misses, fetch durations
// observed, fetch errors, transform durations observed, transforms in
// progress.
steps := []struct {
name string
path string
size int
wantErr bool
want metricValues
}{
{"miss that fetches the source", testPathPhoto, 50, false,
metricValues{0, 1, 1, 0, 1, 0}},
{"hit", testPathPhoto, 50, false,
metricValues{0, 1, 1, 0, 1, 0}},
{"miss that uses the cached source", testPathPhoto, 25, false,
metricValues{1, 1, 1, 0, 2, 0}},
{"miss whose source is not found", "/images/absent.jpg", 50, true,
metricValues{1, 2, 1, 1, 2, 0}},
}
for _, step := range steps {
resp, err := svc.Get(t.Context(), &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.GoodHost,
Path: step.path,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: step.size, Height: step.size},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
})
if (err != nil) != step.wantErr {
t.Fatalf("%s: Get() error = %v, want error %t", step.name, err, step.wantErr)
}
if err == nil {
_ = resp.Content.Close()
}
got := readMetricValues(t, svc)
t.Logf("after the %s: metrics = %+v", step.name, got)
if got != step.want {
t.Fatalf("after the %s: metrics = %+v, want %+v", step.name, got, step.want)
}
}
}
// TestProxy_Get_CountsMissingSourceFileAsSourceCacheMiss deletes the file of
// a cached source but keeps its row. A request for a second variant of that
// source must fetch the source again, and count a source cache miss, not a
// hit.
func TestProxy_Get_CountsMissingSourceFileAsSourceCacheMiss(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
// A first request caches the photo as a source.
resp, err := svc.Get(t.Context(), widthOnlyRequest(fixtures, 50))
if err != nil {
t.Fatalf("first Get() error = %v", err)
}
_ = resp.Content.Close()
source := widthOnlyRequest(fixtures, 50).Source
err = os.Remove(sourceFilePath(svc, source.Key()))
if err != nil {
t.Fatalf("failed to delete the cached source file: %v", err)
}
before := readMetricValues(t, svc)
resp, err = svc.Get(t.Context(), widthOnlyRequest(fixtures, 25))
if err != nil {
t.Fatalf("Get() of a second variant error = %v", err)
}
_ = resp.Content.Close()
after := readMetricValues(t, svc)
t.Logf("before the second variant: metrics = %+v; after: %+v", before, after)
if after.sourceCacheHits != before.sourceCacheHits ||
after.sourceCacheMisses != before.sourceCacheMisses+1 {
t.Errorf("source cache hits went from %d to %d and misses from %d to %d, "+
"want no new hit and one more miss", before.sourceCacheHits,
after.sourceCacheHits, before.sourceCacheMisses, after.sourceCacheMisses)
}
}
// TestProxy_Get_CountsTransformWaitingForSlotAsInProgress holds the only
// processing slot and starts a request that must transform the photo. While
// the request waits for the slot, pixa_transforms_in_progress must be 1; once
// the request returns, 0.
func TestProxy_Get_CountsTransformWaitingForSlotAsInProgress(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
svc.processor = imageprocessor.New(
imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1},
)
release := holdProcessingSlot(t, svc.processor)
result := startGet(t.Context(), svc, widthOnlyRequest(fixtures, 50))
// The request fetches the photo first, then waits for the slot.
deadline := time.Now().Add(5 * time.Second)
for readMetricValues(t, svc).inProgress != 1 && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
waiting := readMetricValues(t, svc).inProgress
t.Logf("while the request waits for the slot: transforms in progress = %v",
waiting)
select {
case got := <-result:
t.Fatalf("Get() returned while the only processing slot was held, "+
"error = %v", got.err)
default:
}
if waiting != 1 {
t.Fatalf("transforms in progress = %v while the request waits for the "+
"slot, want 1", waiting)
}
release()
got := <-result
if got.err != nil {
t.Fatalf("Get() error = %v", got.err)
}
returned := readMetricValues(t, svc).inProgress
t.Logf("once the request returned: transforms in progress = %v", returned)
if returned != 0 {
t.Errorf("transforms in progress = %v once the request returned, want 0",
returned)
}
}
@@ -1,84 +0,0 @@
package imageproxy
import (
"errors"
"os"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestProxy_Get_ServesCachedVariantOfNegativeCachedURL caches a variant of
// the test photo, then keeps a failed fetch of the photo's URL in the negative
// cache, as a failed fetch for another variant does once the source has been
// evicted. The cached variant is still served, and a variant not in the cache
// is made from the cached source. Once the source is evicted, a variant not in
// the cache needs a fetch and is refused.
func TestProxy_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
resp, err := svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
_ = resp.Content.Close()
err = svc.cache.PutFailure(t.Context(),
photoVariant(fixtures, 85, imagerequest.FitCover).Source)
if err != nil {
t.Fatalf("PutFailure() error = %v", err)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() of the cached variant error = %v", err)
}
_ = resp.Content.Close()
if resp.CacheStatus != CacheHit {
t.Errorf("cached variant CacheStatus = %q, want %q",
resp.CacheStatus, CacheHit)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
t.Logf("Get() of a variant made from the cached source: error = %v", err)
if err != nil {
t.Fatalf("Get() of a variant made from the cached source error = %v", err)
}
_ = resp.Content.Close()
source := photoVariant(fixtures, 85, imagerequest.FitCover).Source
reader, _, _, err := svc.cache.Source(t.Context(), source)
if err != nil {
t.Fatalf("Source() error = %v, want the cached source", err)
}
_ = reader.Close()
// Evicts the source as the cache's eviction does: its row, then its file
_, err = svc.db.ExecContext(t.Context(),
`DELETE FROM source_content WHERE source_key = ?`, string(source.Key()))
if err != nil {
t.Fatalf("deleting the source's row: %v", err)
}
err = os.Remove(sourceFilePath(svc, source.Key()))
if err != nil {
t.Fatalf("removing the source's file: %v", err)
}
_, err = svc.Get(t.Context(), photoVariant(fixtures, 60, imagerequest.FitCover))
t.Logf("Get() of a variant whose source was evicted: error = %v", err)
if !errors.Is(err, ErrNegativeCached) {
t.Errorf("Get() of a variant whose source was evicted error = %v, want %v",
err, ErrNegativeCached)
}
}
@@ -1,184 +0,0 @@
package imageproxy
import (
"bytes"
"io"
"os"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// getImage calls Get for req and returns the image it answered with and
// its cache status.
func getImage(
t *testing.T, svc *testProxy, req *imagerequest.Request,
) ([]byte, CacheStatus) {
t.Helper()
resp, err := svc.Get(t.Context(), req)
if err != nil {
t.Fatalf("Get() of quality %d error = %v", req.Transform.Quality, err)
}
defer func() { _ = resp.Content.Close() }()
body, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("reading the image of quality %d: %v", req.Transform.Quality, err)
}
return body, resp.CacheStatus
}
// fileSize returns the size of the file at path.
func fileSize(t *testing.T, path string) int64 {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
return info.Size()
}
// cachedSourcePath returns the path of the cached source file req uses.
func cachedSourcePath(
t *testing.T, svc *testProxy, req *imagerequest.Request,
) string {
t.Helper()
reader, _, _, err := svc.cache.Source(t.Context(), req.Source)
if err != nil {
t.Fatalf("Source() error = %v; want the cached source", err)
}
_ = reader.Close()
return sourceFilePath(svc, req.Source.Key())
}
// TestProxy_Get_FetchesAgainForRemovedCachedSource removes the cached
// source's row and file, as the reconciliation pass does when the file is
// empty or cut short. internal/imgcache tests that the pass does so. The
// test then asks for two new sizes of the source. The first must fetch it
// again and cache it whole, so the second is made from the cache without a
// third fetch.
func TestProxy_Get_FetchesAgainForRemovedCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
source := photoVariant(fixtures, 85, imagerequest.FitCover).Source
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
// Removes the source as the reconciliation pass does: its row, then its file
_, err := svc.db.ExecContext(t.Context(),
`DELETE FROM source_content WHERE source_key = ?`, string(source.Key()))
if err != nil {
t.Fatalf("deleting the source's row: %v", err)
}
err = os.Remove(path)
if err != nil {
t.Fatalf("removing the source's file: %v", err)
}
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
fetches := readCounters(t, svc).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// TestProxy_Get_FetchesAgainSourceOfNoSourceFormat replaces the cached
// source's file with text. The next request for a new size of the source
// must fetch it again and cache it whole.
func TestProxy_Get_FetchesAgainSourceOfNoSourceFormat(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
err := os.WriteFile(path, []byte("this is text, not an image"),
imgcache.StorageFilePerm)
if err != nil {
t.Fatalf("replacing the source's file: %v", err)
}
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
fetches := readCounters(t, svc).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// TestProxy_Get_ConvertsAgainForRemovedCachedVariant caches a variant of
// the photo and removes its row and file, as the reconciliation pass does
// when the file is empty or cut short. internal/imgcache tests that the
// pass does so. The test then checks that Get converts the image again, and
// that the next Get serves the whole variant from the cache.
func TestProxy_Get_ConvertsAgainForRemovedCachedVariant(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
key := imagerequest.CacheKey(req)
// Removes the variant as the reconciliation pass does: its row, then its file
_, err := svc.db.ExecContext(t.Context(),
`DELETE FROM variant_content WHERE cache_key = ?`, string(key))
if err != nil {
t.Fatalf("deleting the variant's row: %v", err)
}
err = os.Remove(variantFilePath(svc, key))
if err != nil {
t.Fatalf("removing the variant's file: %v", err)
}
got, status := getImage(t, svc, req)
t.Logf("after the removal: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the removal: Get() = %d bytes, %s; want the %d bytes "+
"converted again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
t.Logf("next request: %d bytes, %s", len(got), status)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
@@ -1,78 +0,0 @@
package imageproxy
import (
"context"
"database/sql"
"errors"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// pendingAccessTimesWait is how long a test waits for the database's
// connection, for a cache hit to be answered, for an access time to reach
// the database, or for a request to return past its deadline.
const pendingAccessTimesWait = 5 * time.Second
// holdDatabase takes the one connection of db, so that every other query
// waits for it, and returns the func that frees it.
func holdDatabase(t *testing.T, db *sql.DB) func() {
t.Helper()
ctx, cancel := context.WithTimeout(t.Context(), pendingAccessTimesWait)
defer cancel()
conn, err := db.Conn(ctx)
if err != nil {
t.Fatalf("failed to take the database connection: %v", err)
}
return func() { _ = conn.Close() }
}
// TestProxy_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy holds the
// database's one connection while a request whose fetch is held reaches its
// deadline. The request must return with the deadline's error while the
// connection is still held, as it waits for no database write past its
// deadline.
func TestProxy_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
// Room for the request to reach its fetch on a busy host
const timeout = 2 * time.Second
ctx, cancel := context.WithTimeout(t.Context(), timeout)
defer cancel()
deadline, _ := ctx.Deadline()
results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
// The request has made its database reads by the time it fetches.
select {
case <-fetcher.started:
case <-time.After(timeout):
t.Fatal("request did not reach its fetch by its deadline")
}
releaseDatabase := holdDatabase(t, svc.db)
defer releaseDatabase()
select {
case got := <-results:
t.Logf("Get() returned %v after its deadline, error = %v",
time.Since(deadline), got.err)
if !errors.Is(got.err, context.DeadlineExceeded) {
t.Errorf("Get() error = %v, want %v", got.err, context.DeadlineExceeded)
}
case <-time.After(time.Until(deadline) + pendingAccessTimesWait):
t.Fatal("request did not return while the database was busy")
}
}
@@ -1,355 +0,0 @@
package imageproxy
import (
"bytes"
"context"
"database/sql"
"image"
"image/color"
"image/gif"
"image/jpeg"
"image/png"
"io/fs"
"path/filepath"
"testing"
"testing/fstest"
"time"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// Shared test data literals, extracted as constants for goconst.
const (
testContentTypeJPEG = "image/jpeg"
)
// sqliteTimestampFormat matches the format SQLite's CURRENT_TIMESTAMP
// produces, so injected timestamps compare correctly against ones the
// cache writes.
const sqliteTimestampFormat = "2006-01-02 15:04:05"
// TestFixtures contains paths to test files in the mock filesystem.
type TestFixtures struct {
// Valid image files
GoodHostJPEG string // valid JPEG on GoodHost
GoodHostPNG string // valid PNG on GoodHost
GoodHostGIF string // valid GIF on GoodHost
OtherHostJPEG string // valid JPEG on OtherHost
OtherHostPNG string // valid PNG on OtherHost
// Invalid/edge case files
InvalidFile string // file with wrong magic bytes
EmptyFile string // zero-byte file
TextFile string // text file masquerading as image
// Hostnames
GoodHost string // a host the mock fetcher serves
OtherHost string // a second host the mock fetcher serves
}
// DefaultFixtures returns the standard test fixture paths.
func DefaultFixtures() *TestFixtures {
return &TestFixtures{
GoodHostJPEG: "goodhost.example.com/images/photo.jpg",
GoodHostPNG: "goodhost.example.com/images/logo.png",
GoodHostGIF: "goodhost.example.com/images/animation.gif",
OtherHostJPEG: "otherhost.example.com/uploads/image.jpg",
OtherHostPNG: "otherhost.example.com/uploads/icon.png",
InvalidFile: "goodhost.example.com/images/fake.jpg",
EmptyFile: "goodhost.example.com/images/empty.jpg",
TextFile: "goodhost.example.com/images/text.png",
GoodHost: "goodhost.example.com",
OtherHost: "otherhost.example.com",
}
}
// NewTestFS creates a mock filesystem with test images.
func NewTestFS(t *testing.T) (fs.FS, *TestFixtures) {
t.Helper()
fixtures := DefaultFixtures()
// Generate test images
jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
pngData := generateTestPNG(t, 100, 100, color.RGBA{0, 255, 0, 255})
gifData := generateTestGIF(t, 100, 100, color.RGBA{0, 0, 255, 255})
// Create the mock filesystem
mockFS := fstest.MapFS{
// Good host files
fixtures.GoodHostJPEG: &fstest.MapFile{Data: jpegData},
fixtures.GoodHostPNG: &fstest.MapFile{Data: pngData},
fixtures.GoodHostGIF: &fstest.MapFile{Data: gifData},
// Other host files
fixtures.OtherHostJPEG: &fstest.MapFile{Data: jpegData},
fixtures.OtherHostPNG: &fstest.MapFile{Data: pngData},
// Invalid files
fixtures.InvalidFile: &fstest.MapFile{Data: []byte("not a real image file content")},
fixtures.EmptyFile: &fstest.MapFile{Data: []byte{}},
fixtures.TextFile: &fstest.MapFile{Data: []byte("This is just text, not a PNG")},
}
return mockFS, fixtures
}
// generateTestJPEG creates a minimal valid JPEG image.
func generateTestJPEG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 85})
if err != nil {
t.Fatalf("failed to encode test JPEG: %v", err)
}
return buf.Bytes()
}
// generateTestPNG creates a minimal valid PNG image.
func generateTestPNG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
err := png.Encode(&buf, img)
if err != nil {
t.Fatalf("failed to encode test PNG: %v", err)
}
return buf.Bytes()
}
// generateTestGIF creates a minimal valid GIF image.
func generateTestGIF(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewPaletted(
image.Rect(0, 0, width, height),
[]color.Color{c, color.White},
)
for y := range height {
for x := range width {
img.SetColorIndex(x, y, 0)
}
}
var buf bytes.Buffer
err := gif.Encode(&buf, img, nil)
if err != nil {
t.Fatalf("failed to encode test GIF: %v", err)
}
return buf.Bytes()
}
// testProxy is a Proxy with the database and the state directory of its
// cache, through which a test reads and changes the cache behind the proxy.
// registry is the metrics registry the proxy and its cache share.
type testProxy struct {
*Proxy
db *sql.DB
stateDir string
registry *prometheus.Registry
}
// setupTestProxy creates a Proxy with mock fetcher for testing.
func setupTestProxy(t *testing.T) (*testProxy, *TestFixtures) {
t.Helper()
mockFS, fixtures := NewTestFS(t)
// Create temp directory for cache
tmpDir := t.TempDir()
// Create in-memory database
db := setupTestDB(t)
registry := prometheus.NewRegistry()
cache, err := imgcache.NewCache(db, imgcache.Config{
StateDir: tmpDir,
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
}, registry)
if err != nil {
t.Fatalf("failed to create cache: %v", err)
}
proxy, err := New(&Config{
Cache: cache,
Fetcher: httpfetcher.NewMock(mockFS),
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
}, registry)
if err != nil {
t.Fatalf("failed to create proxy: %v", err)
}
return &testProxy{
Proxy: proxy,
db: db,
stateDir: tmpDir,
registry: registry,
}, fixtures
}
// counterValues holds the values of the counters of a proxy and its cache.
type counterValues struct {
hitCount int64
missCount int64
upstreamFetchCount int64
upstreamFetchBytes int64
transformCount int64
}
// readCounters reads the counters of svc and its cache from their metrics
// registry.
func readCounters(t *testing.T, svc *testProxy) counterValues {
t.Helper()
families, err := svc.registry.Gather()
if err != nil {
t.Fatalf("failed to gather the counters: %v", err)
}
values := make(map[string]int64, len(families))
for _, family := range families {
values[family.GetName()] = int64(family.GetMetric()[0].GetCounter().GetValue())
}
return counterValues{
hitCount: values["pixa_cache_hits_total"],
missCount: values["pixa_cache_misses_total"],
upstreamFetchCount: values["pixa_upstream_fetches_total"],
upstreamFetchBytes: values["pixa_upstream_fetch_bytes_total"],
transformCount: values["pixa_transforms_total"],
}
}
// sourceFilePath returns the path of the file of the source key in svc's
// cache, as README.md gives it: <state_dir>/cache/sources/<ab>/<cd>/<key>.
func sourceFilePath(svc *testProxy, key imagerequest.SourceKey) string {
name := string(key)
return filepath.Join(svc.stateDir, "cache", "sources", name[0:2], name[2:4], name)
}
// variantFilePath returns the path of the file of the variant key in svc's
// cache, as README.md gives it: <state_dir>/cache/variants/<ab>/<cd>/<key>.
func variantFilePath(svc *testProxy, key imagerequest.VariantKey) string {
name := string(key)
return filepath.Join(svc.stateDir, "cache", "variants", name[0:2], name[2:4], name)
}
// setVariantLastAccessed backdates the last access time of a tracked
// variant.
func setVariantLastAccessed(
t *testing.T, db *sql.DB, key imagerequest.VariantKey, when time.Time,
) {
t.Helper()
res, err := db.ExecContext(t.Context(),
`UPDATE variant_content SET last_accessed_at = ? WHERE cache_key = ?`,
when.UTC().Format(sqliteTimestampFormat), string(key),
)
if err != nil {
t.Fatalf("failed to set variant last_accessed_at: %v", err)
}
affected, err := res.RowsAffected()
if err != nil {
t.Fatalf("failed to read affected rows: %v", err)
}
if affected != 1 {
t.Fatalf("variant %s has no accounting row (affected=%d); "+
"stores must track variants in the database", key, affected)
}
}
// setSourceLastAccessed backdates the last access time of a tracked
// source content blob.
func setSourceLastAccessed(
t *testing.T, db *sql.DB, key imagerequest.SourceKey, when time.Time,
) {
t.Helper()
res, err := db.ExecContext(t.Context(),
`UPDATE source_content SET last_accessed_at = ? WHERE source_key = ?`,
when.UTC().Format(sqliteTimestampFormat), string(key),
)
if err != nil {
t.Fatalf("failed to set source last_accessed_at: %v", err)
}
affected, err := res.RowsAffected()
if err != nil {
t.Fatalf("failed to read affected rows: %v", err)
}
if affected != 1 {
t.Fatalf("source %s has no accounting row (affected=%d)", key, affected)
}
}
// countRows returns the number of rows the given query yields.
func countRows(t *testing.T, db *sql.DB, query string, args ...any) int {
t.Helper()
var n int
err := db.QueryRowContext(t.Context(), query, args...).Scan(&n)
if err != nil {
t.Fatalf("count query %q failed: %v", query, err)
}
return n
}
// setupTestDB creates an in-memory SQLite database for testing
// using the production schema.
func setupTestDB(t *testing.T) *sql.DB {
t.Helper()
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatalf("failed to open test db: %v", err)
}
// Use the real production schema via migrations
err = database.ApplyMigrations(context.Background(), db, nil)
if err != nil {
t.Fatalf("failed to apply migrations: %v", err)
}
return db
}
+4 -34
View File
@@ -1,8 +1,7 @@
// Package imagerequest holds the image request type, its parts, its checks,
// the parsing of the /v1/image/ path, the variant key, the source key and the
// source's upstream URL. It also holds the image format type and the MIME type
// of each format. It imports no other package of this repo, so every package
// that handles an image request can import it.
// source's upstream URL. It imports no other package of this repo, so every
// package that handles an image request can import it.
package imagerequest
import (
@@ -13,12 +12,10 @@ import (
// ErrInvalidFitMode is returned when an invalid fit mode is provided.
var ErrInvalidFitMode = errors.New("invalid fit mode")
// Format is an image format. pixa reads and writes the formats from JPEG to
// GIF below. It reads SVG and writes it only as orig of an SVG source. In a
// request, orig and auto stand for another format.
// Format represents supported output image formats.
type Format string
// Supported image formats.
// Supported image output formats.
const (
FormatOriginal Format = "orig"
FormatJPEG Format = "jpeg"
@@ -27,7 +24,6 @@ const (
FormatAVIF Format = "avif"
FormatJXL Format = "jxl"
FormatGIF Format = "gif"
FormatSVG Format = "svg"
// FormatAuto stands for JPEG XL, AVIF, WebP or JPEG, chosen for each
// request from its Accept header once the URL's signature or token has
@@ -35,32 +31,6 @@ const (
FormatAuto Format = "auto"
)
// MIMEType returns the MIME type of the format, such as image/jpeg for
// FormatJPEG. orig and auto stand for another format, so they have none:
// MIMEType returns "" for them.
func (f Format) MIMEType() string {
switch f {
case FormatJPEG:
return "image/jpeg"
case FormatPNG:
return "image/png"
case FormatWebP:
return "image/webp"
case FormatAVIF:
return "image/avif"
case FormatJXL:
return "image/jxl"
case FormatGIF:
return "image/gif"
case FormatSVG:
return "image/svg+xml"
case FormatOriginal, FormatAuto:
return ""
default:
return ""
}
}
// Size represents requested image dimensions
type Size struct {
Width int
@@ -1,30 +0,0 @@
package imagerequest
import "testing"
// TestFormat_MIMEType checks the MIME type of each format pixa reads and
// writes, and that orig and auto have none.
func TestFormat_MIMEType(t *testing.T) {
t.Parallel()
tests := []struct {
format Format
want string
}{
{FormatJPEG, "image/jpeg"},
{FormatPNG, "image/png"},
{FormatWebP, "image/webp"},
{FormatAVIF, "image/avif"},
{FormatJXL, "image/jxl"},
{FormatGIF, "image/gif"},
{FormatOriginal, ""},
{FormatAuto, ""},
}
for _, tt := range tests {
got := tt.format.MIMEType()
if got != tt.want {
t.Errorf("%s MIMEType() = %q, want %q", tt.format, got, tt.want)
}
}
}
+1 -2
View File
@@ -133,8 +133,7 @@ func parseImageComponents(remainder string) (*Request, error) {
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
}
// The allowlist check and the fetcher drop the brackets, so brackets
// around anything but an IPv6 address would give a host a second spelling
// Go's URL parser refuses brackets around anything but an IPv6 address
if inBrackets := hostMatch[1]; inBrackets != "" {
addr, err := netip.ParseAddr(inBrackets)
if err != nil || !addr.Is6() {
+172 -485
View File
@@ -1,14 +1,3 @@
// Package imgcache is the image cache. It keeps:
// - the source images fetched from upstream
// - the variants made from them
// - the negative cache: the source URLs whose fetch failed recently
//
// Each image is a file with a row in the database. A source is used, and its
// variants are served, until the cache TTL has passed since upstream last sent
// the source or confirmed it unchanged. Then the source is due to be checked
// with upstream again (see Stale). Eviction removes:
// - the least recently used images, to keep the cache under its size limit
// - the images not checked with upstream for EvictionAfterLastCheck
package imgcache
import (
@@ -17,15 +6,12 @@ import (
"errors"
"fmt"
"io"
"io/fs"
"log/slog"
"os"
"path/filepath"
"time"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic"
)
// maxPendingAccessTimes is how many access times of variants served from
@@ -40,35 +26,11 @@ const maxPendingAccessTimes = 10000
// have expired.
const cleanExpiredBatchSize = 1000
// DefaultNegativeTTL is how long pixad keeps a failed fetch in the negative
// cache.
const DefaultNegativeTTL = 5 * time.Minute
// DefaultCacheTTL is how long after upstream last sent a cached source, or
// answered that it had not changed, pixad uses the source and serves its
// variants without asking upstream again.
const DefaultCacheTTL = 24 * time.Hour
// EvictionAfterLastCheck is how long after upstream last sent a cached
// source, or answered that it had not changed, the source and its variants
// are evicted (see EvictNotCheckedRecently). Until then a source whose checks
// fail is used.
const EvictionAfterLastCheck = 30 * 24 * time.Hour
// ErrStale is returned by Variant for a variant whose file's modification
// time is CacheTTL or more ago (see Variant).
var ErrStale = errors.New("cached variant is older than the cache TTL")
// Config holds cache configuration.
type Config struct {
// CacheConfig holds cache configuration.
type CacheConfig struct {
StateDir string
NegativeTTL time.Duration
// CacheTTL is how long a cached source is used, and its variants served,
// before upstream is asked again whether the source changed. Zero means
// DefaultCacheTTL.
CacheTTL time.Duration
// MaxBytes is the disk cache size limit in bytes that eviction
// enforces. Zero turns the disk cache off entirely: no cache
// directories are created, lookups always miss, stores are no-ops,
@@ -84,9 +46,9 @@ type Config struct {
// Cache implements the caching layer for the image proxy.
type Cache struct {
db *sql.DB
srcContent *FileStore // source images by source key
variants *FileStore // processed variants by cache key
config Config
srcContent *ContentStorage // source images by source key
variants *VariantStorage // processed variants by cache key
config CacheConfig
log *slog.Logger
// Eviction machinery. The channels are created in NewCache so
@@ -111,16 +73,13 @@ type Cache struct {
pendingAccessTimesDone chan struct{}
pendingAccessTimesCancel context.CancelFunc
// The cache's metrics, made and registered by registerMetrics (see
// metrics.go)
cacheHits prometheus.Counter
cacheMisses prometheus.Counter
sourceCacheHits prometheus.Counter
sourceCacheMisses prometheus.Counter
usageBytes prometheus.Gauge
maxBytes prometheus.Gauge
evictions prometheus.Counter
evictedBytes prometheus.Counter
// The counts of what the cache does, made and registered by
// registerCounters (see counters.go)
cacheHits prometheus.Counter
cacheMisses prometheus.Counter
upstreamFetches prometheus.Counter
upstreamFetchBytes prometheus.Counter
transforms prometheus.Counter
// reconciliationPageSize is the most rows one read of a content table
// returns in the reconciliation pass. NewCache sets it to
@@ -134,21 +93,17 @@ type Cache struct {
reconciliationReadTestHook func(rows int)
}
// NewCache creates a new cache instance and registers its metrics on
// NewCache creates a new cache instance and registers its counters on
// metricsRegistry. pixad passes Prometheus' default registry, the one
// /metrics serves. It makes no database call.
func NewCache(
db *sql.DB, config Config, metricsRegistry prometheus.Registerer,
db *sql.DB, config CacheConfig, metricsRegistry prometheus.Registerer,
) (*Cache, error) {
log := config.Logger
if log == nil {
log = slog.Default()
}
if config.CacheTTL == 0 {
config.CacheTTL = DefaultCacheTTL
}
c := &Cache{
db: db,
config: config,
@@ -163,7 +118,7 @@ func NewCache(
reconciliationPageSize: defaultReconciliationPageSize,
}
err := c.registerMetrics(metricsRegistry)
err := c.registerCounters(metricsRegistry)
if err != nil {
return nil, err
}
@@ -172,14 +127,14 @@ func NewCache(
return c, nil
}
srcContent, err := NewFileStore(
srcContent, err := NewContentStorage(
filepath.Join(config.StateDir, "cache", "sources"),
)
if err != nil {
return nil, fmt.Errorf("failed to create source content storage: %w", err)
}
variants, err := NewFileStore(
variants, err := NewVariantStorage(
filepath.Join(config.StateDir, "cache", "variants"),
)
if err != nil {
@@ -192,362 +147,31 @@ func NewCache(
return c, nil
}
// VariantFile is the open file of a cached variant, as Variant,
// VariantIgnoringTTL and RenewVariant return it.
type VariantFile struct {
// Content reads the file. The caller closes it.
Content io.ReadCloser
// Size is the file's size in bytes.
Size int64
// ContentType is the MIME type of the variant's format, read from the
// file's first bytes.
ContentType string
// CheckedAt is the file's modification time: its source's check time
// when the variant was made or last renewed.
CheckedAt time.Time
}
// Variant opens a cached variant and returns its file (see VariantFile). It
// opens that one file and makes no database call: the file's modification
// time is its source's check time when the variant was made or last renewed
// (see PutVariant), and a variant whose time is CacheTTL or more ago is
// ErrStale. A variant it opens has its access time handed to the background
// writer without waiting (see touchVariant). A variant not in the cache, or
// any variant of a disabled cache, is ErrNotFound; a variant file whose first
// GetVariant opens a cached variant and returns a reader, its size and its
// content type, read from the variant file's first bytes. It opens that one
// file and makes no database call. A variant not in the cache, or any
// variant of a disabled cache, is ErrNotFound; a variant file whose first
// bytes are not those of an output format is another error.
func (c *Cache) Variant(cacheKey imagerequest.VariantKey) (VariantFile, error) {
if c.config.MaxBytes == 0 {
return VariantFile{}, ErrNotFound
}
file, info, err := c.variants.Open(string(cacheKey))
if err != nil {
return VariantFile{}, err
}
if c.Stale(info.ModTime()) {
_ = file.Close()
return VariantFile{}, ErrStale
}
return c.finishOpeningVariant(cacheKey, file, info)
}
// VariantIgnoringTTL opens a cached variant whatever its age, and returns it
// as Variant does. The image proxy reads it for a variant whose source is not
// cached, before it checks the source with upstream (see LookupVariant).
func (c *Cache) VariantIgnoringTTL(
func (c *Cache) GetVariant(
cacheKey imagerequest.VariantKey,
) (VariantFile, error) {
if c.config.MaxBytes == 0 {
return VariantFile{}, ErrNotFound
}
file, info, err := c.variants.Open(string(cacheKey))
if err != nil {
return VariantFile{}, err
}
return c.finishOpeningVariant(cacheKey, file, info)
}
// RenewVariant opens a cached variant made from the source bytes that source
// describes, whatever its age, and returns it as Variant does. It moves the
// variant's check time forward to source.CheckedAt (see SetVariantCheckedAt).
// A variant made before source.FetchedAt was made from older bytes, and is
// ErrNotFound, as is a variant not in the cache.
func (c *Cache) RenewVariant(
ctx context.Context, cacheKey imagerequest.VariantKey, source SourceInfo,
) (VariantFile, error) {
if c.config.MaxBytes == 0 {
return VariantFile{}, ErrNotFound
}
file, info, err := c.variants.Open(string(cacheKey))
if err != nil {
return VariantFile{}, err
}
if info.ModTime().Before(source.FetchedAt) {
_ = file.Close()
return VariantFile{}, ErrNotFound
}
format, err := readFormat(file)
if err != nil {
_ = file.Close()
return VariantFile{}, err
}
checkedAt := info.ModTime()
if checkedAt.Before(source.CheckedAt) {
err = c.SetVariantCheckedAt(ctx, cacheKey, source.CheckedAt)
if err != nil {
_ = file.Close()
return VariantFile{}, err
}
checkedAt = source.CheckedAt
}
c.touchVariant(cacheKey)
return VariantFile{
Content: file,
Size: info.Size(),
ContentType: format.MIMEType(),
CheckedAt: checkedAt,
}, nil
}
// SetVariantCheckedAt records checkedAt as when upstream last sent the source
// bytes the cached variant of cacheKey was made from, or answered that they
// had not changed. It sets the variant file's modification time to checkedAt,
// so Variant serves the file until CacheTTL after it. It sets the checked_at
// of the variant's row too, so eviction keeps the variant until
// EvictionAfterLastCheck after it. A failure to write the row is only logged.
// On a disabled cache it is a no-op.
func (c *Cache) SetVariantCheckedAt(
ctx context.Context, cacheKey imagerequest.VariantKey, checkedAt time.Time,
) error {
if c.config.MaxBytes == 0 {
return nil
}
err := c.variants.SetModTime(string(cacheKey), checkedAt)
if err != nil {
return err
}
_, err = c.db.ExecContext(ctx,
`UPDATE variant_content SET checked_at = ? WHERE cache_key = ?`,
checkedAt.UTC(), string(cacheKey))
if err != nil {
c.log.Warn("failed to renew variant's checked_at",
"cache_key", cacheKey, "error", err)
}
return nil
}
// Stale reports whether a source last checked with upstream at checkedAt,
// and its variants, are due to be checked again.
func (c *Cache) Stale(checkedAt time.Time) bool {
return time.Since(checkedAt) >= c.config.CacheTTL
}
// PutVariant stores a processed variant by its cache key and records it in
// the size accounting. source describes the source bytes the variant was made
// from. Its CheckedAt becomes the modification time of the variant's file,
// which Variant and RenewVariant read. The variant's row keeps its CheckedAt,
// ETag and LastModified (see LookupVariant). On a disabled cache it is a
// no-op. The accounting insert is best-effort (the startup reconciliation
// pass adopts any variant file that misses its accounting row).
func (c *Cache) PutVariant(
ctx context.Context,
cacheKey imagerequest.VariantKey,
source SourceInfo,
content io.Reader,
) error {
if c.config.MaxBytes == 0 {
return nil
}
data, err := io.ReadAll(content)
if err != nil {
return fmt.Errorf("failed to read content: %w", err)
}
err = c.variants.Write(string(cacheKey), data)
if err != nil {
return err
}
// A file left with the time of its writing could be served as made from
// bytes fetched after the ones it was made from
err = c.variants.SetModTime(string(cacheKey), source.CheckedAt)
if err != nil {
deleteErr := c.variants.Delete(string(cacheKey))
if deleteErr != nil {
c.log.Warn("failed to remove variant file whose time was not set",
"cache_key", cacheKey, "error", deleteErr)
}
return err
}
size := int64(len(data))
_, err = c.db.ExecContext(ctx, `
INSERT INTO variant_content (cache_key, size_bytes, checked_at,
upstream_etag, upstream_last_modified)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(cache_key) DO UPDATE SET
size_bytes = excluded.size_bytes,
last_accessed_at = CURRENT_TIMESTAMP,
checked_at = excluded.checked_at,
upstream_etag = excluded.upstream_etag,
upstream_last_modified = excluded.upstream_last_modified
`, string(cacheKey), size, source.CheckedAt.UTC(), source.ETag,
source.LastModified)
if err != nil {
c.log.Warn("failed to record variant in size accounting",
"cache_key", cacheKey, "error", err)
}
c.notifyWritePressure()
return nil
}
// Source opens the cached source of src and returns a reader, its size in
// bytes and its format, read from the file's first bytes. A source whose row
// or file is missing is ErrNotFound, as is any source of a disabled cache; a
// file whose first bytes are not those of a source format is another error.
// A source it opens has its access time handed to the background writer
// without waiting (see touchSourceContent).
func (c *Cache) Source(
ctx context.Context, src imagerequest.Source,
) (io.ReadCloser, int64, imagerequest.Format, error) {
) (io.ReadCloser, int64, string, error) {
if c.config.MaxBytes == 0 {
return nil, 0, "", ErrNotFound
}
key := src.Key()
var rowExists bool
err := c.db.QueryRowContext(ctx, `
SELECT EXISTS(SELECT 1 FROM source_content WHERE source_key = ?)
`, string(key)).Scan(&rowExists)
if err != nil {
return nil, 0, "", fmt.Errorf("failed to lookup source: %w", err)
}
if !rowExists {
return nil, 0, "", ErrNotFound
}
file, info, err := c.srcContent.Open(string(key))
if err != nil {
return nil, 0, "", err
}
format, err := readFormat(file)
if err != nil {
_ = file.Close()
return nil, 0, "", err
}
c.touchSourceContent(key)
return file, info.Size(), format, nil
return c.variants.LoadWithContentType(cacheKey)
}
// SourceInfo is what the cache keeps in a cached source's row besides its
// size and access time.
type SourceInfo struct {
// FetchedAt is when the cached bytes were fetched.
FetchedAt time.Time
// CheckedAt is when upstream last sent the bytes or answered that they
// had not changed.
CheckedAt time.Time
// ETag is the ETag header upstream sent with the bytes, or "".
ETag string
// LastModified is the Last-Modified header upstream sent with the
// bytes, or "".
LastModified string
}
// LookupSource returns what the cache keeps about the cached source of src.
// A source with no row is ErrNotFound, as is any source of a disabled cache.
func (c *Cache) LookupSource(
ctx context.Context, src imagerequest.Source,
) (SourceInfo, error) {
if c.config.MaxBytes == 0 {
return SourceInfo{}, ErrNotFound
}
var info SourceInfo
err := c.db.QueryRowContext(ctx, `
SELECT fetched_at, checked_at, upstream_etag, upstream_last_modified
FROM source_content WHERE source_key = ?
`, string(src.Key())).Scan(
&info.FetchedAt, &info.CheckedAt, &info.ETag, &info.LastModified)
if errors.Is(err, sql.ErrNoRows) {
return SourceInfo{}, ErrNotFound
}
if err != nil {
return SourceInfo{}, fmt.Errorf("failed to look up source: %w", err)
}
return info, nil
}
// LookupVariant returns what the row of the cached variant of cacheKey keeps
// about the source bytes the variant was made from: the CheckedAt, ETag and
// LastModified they had when the variant was made or last renewed. The row
// keeps no FetchedAt, so it is zero. With the ETag and LastModified the image
// proxy checks the variant with upstream after its source is evicted. A
// variant with no row is ErrNotFound, as is any variant of a disabled cache.
func (c *Cache) LookupVariant(
ctx context.Context, cacheKey imagerequest.VariantKey,
) (SourceInfo, error) {
if c.config.MaxBytes == 0 {
return SourceInfo{}, ErrNotFound
}
var info SourceInfo
err := c.db.QueryRowContext(ctx, `
SELECT checked_at, upstream_etag, upstream_last_modified
FROM variant_content WHERE cache_key = ?
`, string(cacheKey)).Scan(&info.CheckedAt, &info.ETag, &info.LastModified)
if errors.Is(err, sql.ErrNoRows) {
return SourceInfo{}, ErrNotFound
}
if err != nil {
return SourceInfo{}, fmt.Errorf("failed to look up variant: %w", err)
}
return info, nil
}
// RenewSource records that upstream answered at checkedAt that the cached
// source of src had not changed.
func (c *Cache) RenewSource(
ctx context.Context, src imagerequest.Source, checkedAt time.Time,
) error {
_, err := c.db.ExecContext(ctx,
`UPDATE source_content SET checked_at = ? WHERE source_key = ?`,
checkedAt.UTC(), string(src.Key()))
if err != nil {
return fmt.Errorf("failed to renew source: %w", err)
}
return nil
}
// PutSource stores fetched source content with what info says about it: it
// writes the file named by the source's key, then the source's row. When the
// row cannot be written the file is removed and the store fails, as a file
// with no row is never served or evicted; if that removal fails too, the
// next reconciliation pass removes the file. On a disabled cache it is a
// no-op returning an empty key.
func (c *Cache) PutSource(
// StoreSource stores fetched source content: it writes the file named by
// the source's key, then the source's row. When the row cannot be written
// the file is removed and the store fails, as a file with no row is never
// served or evicted; if that removal fails too, the next reconciliation
// pass removes the file. On a disabled cache it is a no-op returning an
// empty key.
func (c *Cache) StoreSource(
ctx context.Context,
src imagerequest.Source,
req *imagerequest.Request,
content io.Reader,
info SourceInfo,
) (imagerequest.SourceKey, error) {
if c.config.MaxBytes == 0 {
return "", nil
@@ -558,30 +182,22 @@ func (c *Cache) PutSource(
return "", fmt.Errorf("failed to read source content: %w", err)
}
key := src.Key()
key := req.Source.Key()
err = c.srcContent.Write(string(key), data)
size, err := c.srcContent.Store(key, data)
if err != nil {
return "", fmt.Errorf("failed to store source content: %w", err)
}
size := int64(len(data))
_, err = c.db.ExecContext(ctx, `
INSERT INTO source_content (source_key, size_bytes, fetched_at,
checked_at, upstream_etag, upstream_last_modified)
VALUES (?, ?, ?, ?, ?, ?)
INSERT INTO source_content (source_key, size_bytes)
VALUES (?, ?)
ON CONFLICT(source_key) DO UPDATE SET
size_bytes = excluded.size_bytes,
last_accessed_at = CURRENT_TIMESTAMP,
fetched_at = excluded.fetched_at,
checked_at = excluded.checked_at,
upstream_etag = excluded.upstream_etag,
upstream_last_modified = excluded.upstream_last_modified
`, string(key), size, info.FetchedAt.UTC(), info.CheckedAt.UTC(), info.ETag,
info.LastModified)
last_accessed_at = CURRENT_TIMESTAMP
`, string(key), size)
if err != nil {
deleteErr := c.srcContent.Delete(string(key))
deleteErr := c.srcContent.Delete(key)
if deleteErr != nil {
c.log.Warn("failed to remove source file whose row was not written",
"source_key", key, "error", deleteErr)
@@ -595,34 +211,78 @@ func (c *Cache) PutSource(
return key, nil
}
// Failure reports whether src is in the negative cache. It only reads: a
// row past its expiry is not a hit, and the eviction goroutine deletes it
// with the other expired rows (see CleanExpired).
func (c *Cache) Failure(
ctx context.Context, src imagerequest.Source,
) (bool, error) {
var expiresAt time.Time
err := c.db.QueryRowContext(ctx, `
SELECT expires_at FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, src.Host, src.Path, src.Query).Scan(&expiresAt)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
// StoreVariant stores a processed variant by its cache key and records
// it in the size accounting. On a disabled cache it is a no-op. The
// accounting insert is best-effort (the startup reconciliation pass
// adopts any variant file that misses its accounting row).
func (c *Cache) StoreVariant(
ctx context.Context, cacheKey imagerequest.VariantKey, content io.Reader,
) error {
if c.config.MaxBytes == 0 {
return nil
}
size, err := c.variants.Store(cacheKey, content)
if err != nil {
return false, fmt.Errorf("failed to check negative cache: %w", err)
return err
}
expired := time.Now().After(expiresAt)
_, err = c.db.ExecContext(ctx, `
INSERT INTO variant_content (cache_key, size_bytes)
VALUES (?, ?)
ON CONFLICT(cache_key) DO UPDATE SET
size_bytes = excluded.size_bytes,
last_accessed_at = CURRENT_TIMESTAMP
`, string(cacheKey), size)
if err != nil {
c.log.Warn("failed to record variant in size accounting",
"cache_key", cacheKey, "error", err)
}
return !expired, nil
c.notifyWritePressure()
return nil
}
// PutFailure stores a negative cache entry for src, whose fetch failed.
func (c *Cache) PutFailure(ctx context.Context, src imagerequest.Source) error {
// LookupSource checks if we have cached source content for a request.
// Returns the source key if found, or an empty key if not; a row whose
// file is missing is not found. A source found has its access time
// handed to the background writer without waiting (see
// touchSourceContent); a disabled cache always reports no cached source.
func (c *Cache) LookupSource(
ctx context.Context, req *imagerequest.Request,
) (imagerequest.SourceKey, error) {
if c.config.MaxBytes == 0 {
return "", nil
}
key := req.Source.Key()
var rowExists bool
err := c.db.QueryRowContext(ctx, `
SELECT EXISTS(SELECT 1 FROM source_content WHERE source_key = ?)
`, string(key)).Scan(&rowExists)
if err != nil {
return "", fmt.Errorf("failed to lookup source: %w", err)
}
if !rowExists {
return "", nil
}
// Verify the content file exists
if !c.srcContent.Exists(key) {
return "", nil
}
c.touchSourceContent(key)
return key, nil
}
// StoreNegative stores a negative cache entry for a failed fetch.
func (c *Cache) StoreNegative(ctx context.Context, req *imagerequest.Request) error {
expiresAt := time.Now().UTC().Add(c.config.NegativeTTL)
_, err := c.db.ExecContext(ctx, `
@@ -631,7 +291,7 @@ func (c *Cache) PutFailure(ctx context.Context, src imagerequest.Source) error {
VALUES (?, ?, ?, ?)
ON CONFLICT(source_host, source_path, source_query) DO UPDATE SET
expires_at = excluded.expires_at
`, src.Host, src.Path, src.Query, expiresAt)
`, req.Source.Host, req.Source.Path, req.Source.Query, expiresAt)
if err != nil {
return fmt.Errorf("failed to insert negative cache: %w", err)
}
@@ -639,6 +299,18 @@ func (c *Cache) PutFailure(ctx context.Context, src imagerequest.Source) error {
return nil
}
// GetSourceContent returns a reader for cached source content by its key,
// and the content's size in bytes.
func (c *Cache) GetSourceContent(
key imagerequest.SourceKey,
) (io.ReadCloser, int64, error) {
if c.config.MaxBytes == 0 {
return nil, 0, ErrNotFound
}
return c.srcContent.LoadWithSize(key)
}
// CleanExpired removes the negative cache rows past their expiry, one
// batch per statement, until none is left. Once ctx ends it starts no
// further batch and returns ctx's error. The eviction goroutine calls it
@@ -662,6 +334,36 @@ func (c *Cache) CleanExpired(ctx context.Context) error {
}
}
// IncrementStats counts a cache hit or miss, and an upstream fetch that read
// fetchBytes bytes, as IncrementUpstreamFetch does. Like the other Increment
// methods, it only adds to the cache's Prometheus counters in memory (see
// counters.go), so it never waits for the database or fails.
func (c *Cache) IncrementStats(hit bool, fetchBytes int64) {
if hit {
c.cacheHits.Inc()
} else {
c.cacheMisses.Inc()
}
c.IncrementUpstreamFetch(fetchBytes)
}
// IncrementUpstreamFetch counts one upstream fetch that read fetchBytes bytes.
// A fetch that read no bytes is not counted.
func (c *Cache) IncrementUpstreamFetch(fetchBytes int64) {
if fetchBytes <= 0 {
return
}
c.upstreamFetches.Inc()
c.upstreamFetchBytes.Add(float64(fetchBytes))
}
// IncrementTransformCount counts one image transcoded by the image processor.
func (c *Cache) IncrementTransformCount() {
c.transforms.Inc()
}
// touchVariant hands the access time of a variant served from the cache to
// the goroutine StartPendingAccessTimeWrites starts, which writes it in its
// next batch. It never waits: while maxPendingAccessTimes access times of
@@ -686,6 +388,32 @@ func (c *Cache) touchSourceContent(key imagerequest.SourceKey) {
}
}
// checkNegativeCache checks if a request is in the negative cache. It only
// reads: a row past its expiry is not a hit, and the eviction goroutine
// deletes it with the other expired rows (see CleanExpired).
func (c *Cache) checkNegativeCache(
ctx context.Context, req *imagerequest.Request,
) (bool, error) {
var expiresAt time.Time
err := c.db.QueryRowContext(ctx, `
SELECT expires_at FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.Source.Host, req.Source.Path, req.Source.Query).Scan(&expiresAt)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("failed to check negative cache: %w", err)
}
expired := time.Now().After(expiresAt)
return !expired, nil
}
// cleanExpiredBatch deletes up to cleanExpiredBatchSize negative cache
// rows past their expiry in one statement, and returns how many it
// deleted. DELETE ... LIMIT needs a SQLite build option, so the subquery
@@ -709,44 +437,3 @@ func (c *Cache) cleanExpiredBatch(ctx context.Context) (int64, error) {
return deleted, nil
}
// finishOpeningVariant returns a variant file that FileStore.Open opened, with
// its information info, as Variant does. It reads the content type from the
// file's first bytes and hands the variant's access time to the background
// writer (see touchVariant). It closes the file when its first bytes are not
// those of an output format.
func (c *Cache) finishOpeningVariant(
cacheKey imagerequest.VariantKey, file *os.File, info fs.FileInfo,
) (VariantFile, error) {
format, err := readFormat(file)
if err != nil {
_ = file.Close()
return VariantFile{}, err
}
c.touchVariant(cacheKey)
return VariantFile{
Content: file,
Size: info.Size(),
ContentType: format.MIMEType(),
CheckedAt: info.ModTime(),
}, nil
}
// readFormat reads up to magic.MaxMagicBytes first bytes of a source or
// variant file, without moving its offset, and returns the format they show.
// Sources and variants are in the same seven formats; a variant is SVG only
// as orig of an SVG source. A file shorter than magic.MinMagicBytes is
// io.EOF.
func readFormat(content io.ReaderAt) (imagerequest.Format, error) {
firstBytes := make([]byte, magic.MaxMagicBytes)
n, err := content.ReadAt(firstBytes, 0)
if err != nil && (!errors.Is(err, io.EOF) || n < magic.MinMagicBytes) {
return "", fmt.Errorf("failed to read the first bytes: %w", err)
}
return magic.DetectFormat(firstBytes[:n])
}
+110 -27
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"image/color"
"io"
"os"
"testing"
"time"
@@ -14,10 +15,10 @@ import (
"sneak.berlin/go/pixa/internal/magic"
)
// TestVariant_ContentTypeFromFirstBytes stores an image in each output
// format as a variant and checks that Variant serves it whole with the
// TestGetVariant_ContentTypeFromFirstBytes stores an image in each output
// format as a variant and checks that GetVariant serves it whole with the
// content type of its format, read from the file.
func TestVariant_ContentTypeFromFirstBytes(t *testing.T) {
func TestGetVariant_ContentTypeFromFirstBytes(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
@@ -32,7 +33,7 @@ func TestVariant_ContentTypeFromFirstBytes(t *testing.T) {
imagerequest.FormatWebP, imagerequest.FormatAVIF, imagerequest.FormatJXL,
} {
result, err := processor.Process(t.Context(), bytes.NewReader(source),
imagerequest.FormatJPEG, &imagerequest.Transform{
&imagerequest.Transform{
Size: imagerequest.Size{Width: 20, Height: 15},
Format: format,
Quality: 85,
@@ -49,19 +50,16 @@ func TestVariant_ContentTypeFromFirstBytes(t *testing.T) {
key := imagerequest.VariantKey("aabbccdd10" + string(format))
err = cache.PutVariant(t.Context(), key, SourceInfo{CheckedAt: time.Now()},
bytes.NewReader(image))
err = cache.StoreVariant(t.Context(), key, bytes.NewReader(image))
if err != nil {
t.Fatalf("PutVariant(%s) error = %v", format, err)
t.Fatalf("StoreVariant(%s) error = %v", format, err)
}
variant, err := cache.Variant(key)
reader, size, contentType, err := cache.GetVariant(key)
if err != nil {
t.Fatalf("Variant(%s) error = %v", format, err)
t.Fatalf("GetVariant(%s) error = %v", format, err)
}
reader, size, contentType := variant.Content, variant.Size, variant.ContentType
served, err := io.ReadAll(reader)
_ = reader.Close()
@@ -71,7 +69,7 @@ func TestVariant_ContentTypeFromFirstBytes(t *testing.T) {
t.Logf("%s: %d bytes served as %s", format, size, contentType)
if want := format.MIMEType(); contentType != want {
if want := imageprocessor.FormatToMIME(format); contentType != want {
t.Errorf("%s variant content type = %q, want %q", format, contentType, want)
}
@@ -82,11 +80,11 @@ func TestVariant_ContentTypeFromFirstBytes(t *testing.T) {
}
}
// TestVariant_RefusesFileOfNoOutputFormat checks that Variant refuses
// TestGetVariant_RefusesFileOfNoOutputFormat checks that GetVariant refuses
// a variant file whose first bytes are not those of an output format, with
// an error other than ErrNotFound, and that a variant with no file, or any
// variant of a disabled cache, is ErrNotFound.
func TestVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
func TestGetVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
@@ -98,6 +96,8 @@ func TestVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
wantErr error
}{
{"text", []byte("this is text, not an image"), magic.ErrUnknownFormat},
{"SVG", []byte(`<svg xmlns="http://www.w3.org/2000/svg"></svg>`),
errNotAnOutputFormat},
{"JPEG cut to 10 bytes", jpegImage[:10], io.EOF},
{"empty", nil, io.EOF},
}
@@ -105,40 +105,123 @@ func TestVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
for i, tt := range tests {
key := imagerequest.VariantKey(fmt.Sprintf("aabbccdd20%02d", i))
err := cache.variants.Write(string(key), tt.content)
_, err := cache.variants.Store(key, bytes.NewReader(tt.content))
if err != nil {
t.Fatalf("%s: storing the variant file: %v", tt.name, err)
}
variant, err := cache.Variant(key)
t.Logf("%s: Variant() error = %v", tt.name, err)
reader, _, contentType, err := cache.GetVariant(key)
t.Logf("%s: GetVariant() error = %v", tt.name, err)
reader, contentType := variant.Content, variant.ContentType
if reader != nil || contentType != "" || !errors.Is(err, tt.wantErr) {
t.Errorf("%s: Variant() = reader %v, content type %q, error %v; "+
t.Errorf("%s: GetVariant() = reader %v, content type %q, error %v; "+
"want no reader and %v", tt.name, reader, contentType, err, tt.wantErr)
}
}
variant, err := cache.Variant(testVariantKeyFour)
reader := variant.Content
reader, _, _, err := cache.GetVariant(testVariantKeyFour)
if reader != nil || !errors.Is(err, ErrNotFound) {
t.Errorf("Variant() of a variant with no file = reader %v, error %v; "+
t.Errorf("GetVariant() of a variant with no file = reader %v, error %v; "+
"want no reader and %v", reader, err, ErrNotFound)
}
disabled, _ := newEvictionTestCache(t, 0)
variant, err = disabled.Variant(testVariantKeyOne)
reader = variant.Content
reader, _, _, err = disabled.GetVariant(testVariantKeyOne)
if reader != nil || !errors.Is(err, ErrNotFound) {
t.Errorf("Variant() on a disabled cache = reader %v, error %v; "+
t.Errorf("GetVariant() on a disabled cache = reader %v, error %v; "+
"want no reader and %v", reader, err, ErrNotFound)
}
}
// TestService_Get_MakesAgainVariantOfNoOutputFormat replaces a cached
// variant's file with text, and checks that the next request is a miss
// answered with the variant made again, and the one after a hit.
func TestService_Get_MakesAgainVariantOfNoOutputFormat(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := svc.cache.variants.keyToPath(imagerequest.CacheKey(req))
err := os.WriteFile(path, []byte("this is text, not an image"), StorageFilePerm)
if err != nil {
t.Fatalf("replacing the variant file: %v", err)
}
got, status := getImage(t, svc, req)
t.Logf("after the replacement: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the replacement: Get() = %d bytes, %s; want the %d bytes "+
"made again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestService_Get_HitWaitsForNoDatabaseWrite holds the database's one
// connection while a cached variant is requested. The hit must be answered
// while the connection is still held, and its access time must reach the
// database once it is free.
func TestService_Get_HitWaitsForNoDatabaseWrite(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
key := imagerequest.CacheKey(req)
getImage(t, svc, req)
// Each connection to an in-memory SQLite database opens a new, empty one,
// so the goroutine that writes the access times must share the
// connection that holds the schema.
svc.cache.db.SetMaxOpenConns(1)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
anHourAgo := time.Now().Add(-time.Hour)
setVariantLastAccessed(t, svc.cache, key, anHourAgo)
releaseDatabase := holdDatabase(t, svc.cache)
defer releaseDatabase()
results := startGet(t.Context(), svc, req)
select {
case got := <-results:
if got.err != nil {
t.Fatalf("Get() of the cached variant error = %v", got.err)
}
case <-time.After(pendingAccessTimesWait):
t.Fatal("the cache hit waited for the database")
}
releaseDatabase()
deadline := time.Now().Add(pendingAccessTimesWait)
for countRows(t, svc.cache, `
SELECT COUNT(*) FROM variant_content
WHERE cache_key = ? AND last_accessed_at > ?
`, string(key), anHourAgo.UTC().Format(sqliteTimestampFormat)) == 0 {
if time.Now().After(deadline) {
t.Fatal("the hit's access time was not written")
}
time.Sleep(10 * time.Millisecond)
}
}
// TestTouchVariantAndTouchSourceContent_DropAccessTimesPastTheLimit hands one
// more access time of a variant, and one more of a source, than may wait to
// be written. touchVariant and touchSourceContent must return each time
+2 -2
View File
@@ -50,7 +50,7 @@ func TestCache_CleanExpired(t *testing.T) {
tmpDir := t.TempDir()
db := setupTestDB(t)
cache, _ := NewCache(db, Config{
cache, _ := NewCache(db, CacheConfig{
StateDir: tmpDir,
NegativeTTL: 1 * time.Millisecond,
MaxBytes: 1 << 30,
@@ -103,7 +103,7 @@ func TestCache_StorageDirectoriesCreated(t *testing.T) {
tmpDir := t.TempDir()
db := setupTestDB(t)
_, err := NewCache(db, Config{
_, err := NewCache(db, CacheConfig{
StateDir: tmpDir,
MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
@@ -1,4 +1,4 @@
package imageproxy
package imgcache
import (
"bytes"
@@ -37,7 +37,6 @@ type heldFetcher struct {
func (f *heldFetcher) Fetch(
ctx context.Context, source imagerequest.Source,
ifNoneMatch, ifModifiedSince string,
) (*httpfetcher.FetchResult, error) {
f.fetches.Add(1)
@@ -49,7 +48,7 @@ func (f *heldFetcher) Fetch(
return nil, ctx.Err()
}
return f.upstream.Fetch(ctx, source, ifNoneMatch, ifModifiedSince)
return f.upstream.Fetch(ctx, source)
}
// releaseFetches lets every held fetch, and every later one, go on.
@@ -57,15 +56,15 @@ func (f *heldFetcher) releaseFetches() {
f.releaseOnce.Do(func() { close(f.release) })
}
// setupHeldFetchProxy returns a test proxy whose fetches go through a
// setupHeldFetchService returns a test service whose fetches go through a
// heldFetcher. Its database is limited to one connection: each connection to
// an in-memory SQLite database opens a new, empty one, so requests running at
// once must share the connection that holds the schema.
func setupHeldFetchProxy(t *testing.T) (*testProxy, *TestFixtures, *heldFetcher) {
func setupHeldFetchService(t *testing.T) (*Service, *TestFixtures, *heldFetcher) {
t.Helper()
svc, fixtures := setupTestProxy(t)
svc.db.SetMaxOpenConns(1)
svc, fixtures := SetupTestService(t)
svc.cache.db.SetMaxOpenConns(1)
fetcher := &heldFetcher{
upstream: svc.fetcher,
@@ -80,8 +79,8 @@ func setupHeldFetchProxy(t *testing.T) (*testProxy, *TestFixtures, *heldFetcher)
}
// photoVariant asks for the test photo, 100x100, at 50x25 as a JPEG of the
// given quality and fit mode. Each call returns a new request, which a test
// may change.
// given quality and fit mode. Each call returns a new request, as Get writes
// to the request it is given.
func photoVariant(
fixtures *TestFixtures, quality int, fit imagerequest.Fit,
) *imagerequest.Request {
@@ -108,7 +107,7 @@ type getResult struct {
// startGet calls Get in a goroutine of its own and delivers what it returned
// on the channel.
func startGet(
ctx context.Context, svc *testProxy, req *imagerequest.Request,
ctx context.Context, svc *Service, req *imagerequest.Request,
) <-chan getResult {
results := make(chan getResult, 1)
@@ -140,14 +139,14 @@ func jpegSize(data []byte) (int, int) {
return config.Width, config.Height
}
// TestProxy_Get_ConcurrentMissesShareOneFetch starts several requests for
// TestService_Get_ConcurrentMissesShareOneFetch starts several requests for
// one uncached variant while the first one's fetch is held. Between them they
// must fetch the source once and transcode it once, every one must be answered
// with the same 50x25 JPEG, and each must count one miss.
func TestProxy_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc, fixtures, fetcher := setupHeldFetchService(t)
const requests = 8
@@ -184,7 +183,7 @@ func TestProxy_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
t.Errorf("%d requests made %d upstream fetches, want 1", requests, fetches)
}
// NewTestFS builds the same files the test proxy's fetcher serves.
// NewTestFS builds the same files the test service's fetcher serves.
testFS, _ := NewTestFS(t)
photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG)
@@ -194,18 +193,18 @@ func TestProxy_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
want := counterValues{0, requests, 1, int64(len(photo)), 1}
if got := readCounters(t, svc); got != want {
if got := readCounters(svc.cache); got != want {
t.Errorf("counters = %+v, want %+v", got, want)
}
}
// TestProxy_Get_ConcurrentVariantsStayApart requests three variants of the
// TestService_Get_ConcurrentVariantsStayApart requests three variants of the
// test photo at once that differ only in quality or fit. Each must be made by
// a fetch and a transcode of its own, and each answer must be its own variant.
func TestProxy_Get_ConcurrentVariantsStayApart(t *testing.T) {
func TestService_Get_ConcurrentVariantsStayApart(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc, fixtures, fetcher := setupHeldFetchService(t)
cover := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
lowQuality := startGet(t.Context(), svc,
@@ -254,14 +253,14 @@ func TestProxy_Get_ConcurrentVariantsStayApart(t *testing.T) {
}
}
// TestProxy_Get_WaiterStopsWhenItsContextEnds has a second request for a
// TestService_Get_WaiterStopsWhenItsContextEnds has a second request for a
// variant join the first one's held fetch, then ends the second request's
// context. The second request must return at once with the context's error,
// while the fetch is still held, and the first must still be answered.
func TestProxy_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
func TestService_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc, fixtures, fetcher := setupHeldFetchService(t)
first := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
@@ -299,15 +298,15 @@ func TestProxy_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
}
}
// TestProxy_Get_FirstRequestLeavingKeepsTheWork ends the context of the
// TestService_Get_FirstRequestLeavingKeepsTheWork ends the context of the
// request whose fetch is held, after a second request has joined it. The fetch
// and transcode must go on and answer the second request. The first request
// waits for its own work, as every request did before misses were shared, and
// is answered too.
func TestProxy_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
func TestService_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc, fixtures, fetcher := setupHeldFetchService(t)
firstCtx, cancelFirst := context.WithCancel(t.Context())
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
@@ -338,11 +337,11 @@ func TestProxy_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
}
}
// TestProxy_Get_ConcurrentMissesShareAFailure has several requests for an
// TestService_Get_ConcurrentMissesShareAFailure has several requests for an
// image that cannot be served arrive while its fetch is held: the one fetch
// answers all of them with its error. The request after them is answered from
// the negative cache, where the failure is kept.
func TestProxy_Get_ConcurrentMissesShareAFailure(t *testing.T) {
func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
t.Parallel()
tests := []struct {
@@ -363,7 +362,7 @@ func TestProxy_Get_ConcurrentMissesShareAFailure(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc, fixtures, fetcher := setupHeldFetchService(t)
request := func() *imagerequest.Request {
req := photoVariant(fixtures, 85, imagerequest.FitCover)
@@ -399,12 +398,12 @@ func TestProxy_Get_ConcurrentMissesShareAFailure(t *testing.T) {
}
}
// TestProxy_Get_EndedRequestFetchesNothing checks that a request whose
// TestService_Get_EndedRequestFetchesNothing checks that a request whose
// context has already ended when it misses the cache starts no fetch.
func TestProxy_Get_EndedRequestFetchesNothing(t *testing.T) {
func TestService_Get_EndedRequestFetchesNothing(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc, fixtures, fetcher := setupHeldFetchService(t)
ctx, cancel := context.WithCancel(t.Context())
cancel()
@@ -419,14 +418,14 @@ func TestProxy_Get_EndedRequestFetchesNothing(t *testing.T) {
}
}
// TestProxy_Get_ReturnsByItsDeadline gives a request a deadline and holds
// TestService_Get_ReturnsByItsDeadline gives a request a deadline and holds
// its fetch until the fetch's context ends, as the fetcher does while it waits
// for a free connection to the host. The request must return by its deadline
// with the deadline's error.
func TestProxy_Get_ReturnsByItsDeadline(t *testing.T) {
func TestService_Get_ReturnsByItsDeadline(t *testing.T) {
t.Parallel()
svc, fixtures, _ := setupHeldFetchProxy(t)
svc, fixtures, _ := setupHeldFetchService(t)
const timeout = 200 * time.Millisecond
@@ -445,16 +444,16 @@ func TestProxy_Get_ReturnsByItsDeadline(t *testing.T) {
}
}
// TestProxy_Get_WaiterGetsProcessingTimedOut has a request with no deadline
// TestService_Get_WaiterGetsProcessingTimedOut has a request with no deadline
// join the held fetch of a first request with a short one. With the only
// processing slot held, the first request's transcode waits for it until the
// first request's deadline ends, and stops then. The waiting request, whose
// own context has not ended, must get ErrProcessingTimedOut, which the image
// routes answer 503 as when pixa is busy.
func TestProxy_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
func TestService_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc, fixtures, fetcher := setupHeldFetchService(t)
svc.processor = imageprocessor.New(
imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1},
)
@@ -490,18 +489,18 @@ func TestProxy_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
type panickingFetcher struct{}
func (panickingFetcher) Fetch(
context.Context, imagerequest.Source, string, string,
context.Context, imagerequest.Source,
) (*httpfetcher.FetchResult, error) {
panic("upstream fetcher panicked")
}
// TestProxy_Get_PanicBecomesAnError checks that a panic while a variant is
// TestService_Get_PanicBecomesAnError checks that a panic while a variant is
// being made reaches its request as an error naming the panic, instead of
// being raised again.
func TestProxy_Get_PanicBecomesAnError(t *testing.T) {
func TestService_Get_PanicBecomesAnError(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
svc, fixtures := SetupTestService(t)
svc.fetcher = panickingFetcher{}
var err error
@@ -521,13 +520,13 @@ func TestProxy_Get_PanicBecomesAnError(t *testing.T) {
}
}
// TestProxy_Get_PanicIsReportedToSentry checks that a panic while a variant
// TestService_Get_PanicIsReportedToSentry checks that a panic while a variant
// is being made is reported through the Sentry hub on the request's context,
// where the Sentry middleware puts one when sentry_dsn is set.
func TestProxy_Get_PanicIsReportedToSentry(t *testing.T) {
func TestService_Get_PanicIsReportedToSentry(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
svc, fixtures := SetupTestService(t)
svc.fetcher = panickingFetcher{}
transport := &sentry.MockTransport{}
+45
View File
@@ -0,0 +1,45 @@
package imgcache
import (
"fmt"
"github.com/prometheus/client_golang/prometheus"
)
// registerCounters makes the cache's counters and registers them on
// metricsRegistry. The counters are kept in memory, so adding to one waits
// for nothing and makes no database call. It fails when metricsRegistry
// already holds counters of the same names.
func (c *Cache) registerCounters(metricsRegistry prometheus.Registerer) error {
c.cacheHits = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_cache_hits_total",
Help: "Image requests served from the cache.",
})
c.cacheMisses = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_cache_misses_total",
Help: "Image requests not served from the cache.",
})
c.upstreamFetches = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetches_total",
Help: "Fetches of a source image from upstream that read any bytes.",
})
c.upstreamFetchBytes = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_upstream_fetch_bytes_total",
Help: "Bytes read from upstream by fetches of source images.",
})
c.transforms = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_transforms_total",
Help: "Images the image processor transformed.",
})
for _, counter := range []prometheus.Counter{
c.cacheHits, c.cacheMisses, c.upstreamFetches, c.upstreamFetchBytes, c.transforms,
} {
err := metricsRegistry.Register(counter)
if err != nil {
return fmt.Errorf("failed to register the cache's counters: %w", err)
}
}
return nil
}
+26 -3
View File
@@ -8,6 +8,26 @@ import (
"github.com/prometheus/client_golang/prometheus/testutil"
)
// counterValues holds the values of a cache's counters.
type counterValues struct {
hitCount int64
missCount int64
upstreamFetchCount int64
upstreamFetchBytes int64
transformCount int64
}
// readCounters reads cache's counters.
func readCounters(cache *Cache) counterValues {
return counterValues{
hitCount: int64(testutil.ToFloat64(cache.cacheHits)),
missCount: int64(testutil.ToFloat64(cache.cacheMisses)),
upstreamFetchCount: int64(testutil.ToFloat64(cache.upstreamFetches)),
upstreamFetchBytes: int64(testutil.ToFloat64(cache.upstreamFetchBytes)),
transformCount: int64(testutil.ToFloat64(cache.transforms)),
}
}
// TestNewCache_RegistersItsCountersOnItsRegistry checks that NewCache
// registers the cache's counters on the registry it is given, under the
// names README.md gives them.
@@ -17,7 +37,7 @@ func TestNewCache_RegistersItsCountersOnItsRegistry(t *testing.T) {
registry := prometheus.NewRegistry()
_, err := NewCache(evictionTestDB(t),
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
if err != nil {
t.Fatalf("NewCache() error = %v", err)
}
@@ -25,6 +45,9 @@ func TestNewCache_RegistersItsCountersOnItsRegistry(t *testing.T) {
names := []string{
"pixa_cache_hits_total",
"pixa_cache_misses_total",
"pixa_upstream_fetches_total",
"pixa_upstream_fetch_bytes_total",
"pixa_transforms_total",
}
count, err := testutil.GatherAndCount(registry, names...)
@@ -47,13 +70,13 @@ func TestNewCache_FailsWhenItsCountersCannotBeRegistered(t *testing.T) {
registry := prometheus.NewRegistry()
_, err := NewCache(evictionTestDB(t),
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
if err != nil {
t.Fatalf("NewCache() of the first cache error = %v", err)
}
_, err = NewCache(evictionTestDB(t),
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
t.Logf("NewCache() of the second cache error = %v", err)
var alreadyRegistered prometheus.AlreadyRegisteredError
@@ -1,4 +1,4 @@
package imageproxy
package imgcache
import (
"fmt"
@@ -1,88 +0,0 @@
package imgcache
import (
"bytes"
"os"
"slices"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestEvictNotCheckedRecently stores a source and a variant last checked with
// upstream 31 days ago and a source and a variant checked 29 days ago.
// EvictNotCheckedRecently must remove the rows and files of the first two and
// keep the other two.
func TestEvictNotCheckedRecently(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
old := time.Now().Add(-31 * 24 * time.Hour)
recent := time.Now().Add(-29 * 24 * time.Hour)
oldSource := imagerequest.Source{Host: testHostCDN, Path: "/old.jpg"}
recentSource := imagerequest.Source{Host: testHostCDN, Path: "/recent.jpg"}
for src, checkedAt := range map[imagerequest.Source]time.Time{
oldSource: old, recentSource: recent,
} {
_, err := cache.PutSource(t.Context(), src, bytes.NewReader([]byte("source")),
SourceInfo{FetchedAt: checkedAt, CheckedAt: checkedAt})
if err != nil {
t.Fatalf("PutSource(%s) error = %v", src.Path, err)
}
}
oldVariant := imagerequest.VariantKey("aabbccdd0001")
recentVariant := imagerequest.VariantKey("aabbccdd0002")
for key, checkedAt := range map[imagerequest.VariantKey]time.Time{
oldVariant: old, recentVariant: recent,
} {
err := cache.PutVariant(t.Context(), key, SourceInfo{CheckedAt: checkedAt},
bytes.NewReader([]byte("variant")))
if err != nil {
t.Fatalf("PutVariant(%s) error = %v", key, err)
}
}
err := cache.EvictNotCheckedRecently(t.Context())
if err != nil {
t.Fatalf("EvictNotCheckedRecently() error = %v", err)
}
sourceRows, err := queryStringColumn[imagerequest.SourceKey](t.Context(),
cache.db, `SELECT source_key FROM source_content`, "sources", "source")
if err != nil {
t.Fatal(err)
}
variantRows, err := queryStringColumn[imagerequest.VariantKey](t.Context(),
cache.db, `SELECT cache_key FROM variant_content`, "variants", "variant")
if err != nil {
t.Fatal(err)
}
t.Logf("source rows left = %v, variant rows left = %v", sourceRows, variantRows)
if !slices.Equal(sourceRows, []imagerequest.SourceKey{recentSource.Key()}) {
t.Errorf("source rows left = %v, want only %s", sourceRows, recentSource.Key())
}
if !slices.Equal(variantRows, []imagerequest.VariantKey{recentVariant}) {
t.Errorf("variant rows left = %v, want only %s", variantRows, recentVariant)
}
for path, wantKept := range map[string]bool{
cache.srcContent.keyToPath(string(oldSource.Key())): false,
cache.srcContent.keyToPath(string(recentSource.Key())): true,
cache.variants.keyToPath(string(oldVariant)): false,
cache.variants.keyToPath(string(recentVariant)): true,
} {
_, statErr := os.Stat(path)
if kept := statErr == nil; kept != wantKept {
t.Errorf("file %s kept = %t, want %t (%v)", path, kept, wantKept, statErr)
}
}
}
+65 -217
View File
@@ -48,7 +48,7 @@ const staleTempFileAge = time.Hour
const sqliteTimestampLayout = "2006-01-02 15:04:05"
// tempFilePrefix is the prefix os.CreateTemp uses for in-flight cache
// writes (the ".tmp-*" pattern in writeFileSynced).
// writes (".tmp-*" patterns in the storage layer).
const tempFilePrefix = ".tmp-"
// UsageBytes returns the total number of bytes of cache content
@@ -92,9 +92,8 @@ type evictionCandidate struct {
}
// EvictToLimit evicts least-recently-used cache entries until total
// tracked usage is at or below the configured MaxBytes limit. It sets the
// pixa_cache_usage_bytes gauge to each usage it reads. It is a no-op when
// the cache is disabled.
// tracked usage is at or below the configured MaxBytes limit. It is a
// no-op when the cache is disabled.
func (c *Cache) EvictToLimit(ctx context.Context) error {
if c.config.MaxBytes == 0 {
return nil
@@ -106,8 +105,6 @@ func (c *Cache) EvictToLimit(ctx context.Context) error {
return err
}
c.usageBytes.Set(float64(usage))
if usage <= c.config.MaxBytes {
return nil
}
@@ -134,56 +131,6 @@ func (c *Cache) EvictToLimit(ctx context.Context) error {
}
}
// EvictNotCheckedRecently evicts each source and variant whose checked_at is
// EvictionAfterLastCheck or more ago, evictionBatchSize of each at a time,
// until none is left. It is a no-op when the cache is disabled.
func (c *Cache) EvictNotCheckedRecently(ctx context.Context) error {
if c.config.MaxBytes == 0 {
return nil
}
cutoff := time.Now().UTC().Add(-EvictionAfterLastCheck)
for {
sources, err := queryStringColumn[imagerequest.SourceKey](ctx, c.db, `
SELECT source_key FROM source_content WHERE checked_at < ? LIMIT ?
`, "sources not checked recently", "source key", cutoff, evictionBatchSize)
if err != nil {
return err
}
for _, key := range sources {
err = c.evictSourceBlob(ctx, key)
if err != nil {
return err
}
}
variants, err := queryStringColumn[imagerequest.VariantKey](ctx, c.db, `
SELECT cache_key FROM variant_content WHERE checked_at < ? LIMIT ?
`, "variants not checked recently", "variant key", cutoff, evictionBatchSize)
if err != nil {
return err
}
for _, key := range variants {
err = c.evictVariant(ctx, key)
if err != nil {
return err
}
}
if len(sources) > 0 || len(variants) > 0 {
c.log.Info("evicted cache content not checked with upstream recently",
"sources", len(sources), "variants", len(variants))
}
if len(sources) < evictionBatchSize && len(variants) < evictionBatchSize {
return nil
}
}
}
// evictBatch fetches one batch of LRU candidates across variants and
// source blobs and evicts them oldest-first until excessBytes are
// freed or the batch is exhausted. It returns the bytes freed. A
@@ -348,95 +295,37 @@ func (c *Cache) sourceCandidates(ctx context.Context) ([]evictionCandidate, erro
return candidates, nil
}
// evictVariant removes one variant (see removeVariant) and counts the
// eviction with the size its row recorded.
// evictVariant removes one variant: accounting row first, then the
// file, so the database never references a deleted file.
func (c *Cache) evictVariant(
ctx context.Context, cacheKey imagerequest.VariantKey,
) error {
sizeBytes, err := c.removeVariant(ctx, cacheKey)
_, err := c.db.ExecContext(ctx,
`DELETE FROM variant_content WHERE cache_key = ?`, string(cacheKey))
if err != nil {
return fmt.Errorf("failed to delete variant accounting row: %w", err)
}
err = c.variants.Delete(cacheKey)
if err != nil {
return err
}
c.countEviction(sizeBytes)
return nil
}
// removeVariant removes one variant: accounting row first, then the
// file, so the database never references a deleted file. It returns the
// size the row recorded, or 0 when the variant had no row.
func (c *Cache) removeVariant(
ctx context.Context, cacheKey imagerequest.VariantKey,
) (int64, error) {
sizeBytes, err := c.deleteRowReturningSize(ctx,
`DELETE FROM variant_content WHERE cache_key = ? RETURNING size_bytes`,
string(cacheKey))
if err != nil {
return 0, fmt.Errorf("failed to delete variant accounting row: %w", err)
}
err = c.variants.Delete(string(cacheKey))
if err != nil {
return 0, err
}
return sizeBytes, nil
}
// evictSourceBlob removes one source (see removeSourceBlob) and counts the
// eviction with the size its row recorded.
// evictSourceBlob removes one source: its row first, then its file, as
// evictVariant does for a variant. It holds no lock: a store of the same
// source in between can leave the source's row with no file, which
// LookupSource treats as a miss.
func (c *Cache) evictSourceBlob(ctx context.Context, key imagerequest.SourceKey) error {
sizeBytes, err := c.removeSourceBlob(ctx, key)
_, err := c.db.ExecContext(ctx,
`DELETE FROM source_content WHERE source_key = ?`, string(key))
if err != nil {
return err
return fmt.Errorf("failed to delete source content row: %w", err)
}
c.countEviction(sizeBytes)
return nil
}
// removeSourceBlob removes one source: its row first, then its file, as
// removeVariant does for a variant, and returns the size the row recorded,
// or 0 when the source had no row. It holds no lock: a store of the same
// source in between can leave the source's row with no file, which Source
// treats as a miss.
func (c *Cache) removeSourceBlob(
ctx context.Context, key imagerequest.SourceKey,
) (int64, error) {
sizeBytes, err := c.deleteRowReturningSize(ctx,
`DELETE FROM source_content WHERE source_key = ? RETURNING size_bytes`,
string(key))
if err != nil {
return 0, fmt.Errorf("failed to delete source content row: %w", err)
}
err = c.srcContent.Delete(string(key))
if err != nil {
return 0, err
}
return sizeBytes, nil
}
// deleteRowReturningSize runs deleteQuery, which deletes the row of key and
// returns its size_bytes, and returns that size, or 0 when there was no row.
func (c *Cache) deleteRowReturningSize(
ctx context.Context, deleteQuery string, key string,
) (int64, error) {
var sizeBytes int64
err := c.db.QueryRowContext(ctx, deleteQuery, key).Scan(&sizeBytes)
if errors.Is(err, sql.ErrNoRows) {
return 0, nil
}
if err != nil {
return 0, err
}
return sizeBytes, nil
return c.srcContent.Delete(key)
}
// notifyWritePressure wakes the background evictor after a store, so
@@ -455,9 +344,8 @@ func (c *Cache) notifyWritePressure() {
}
// StartEviction launches the background eviction goroutine, which
// removes the expired negative cache rows, and the sources and variants not
// checked for EvictionAfterLastCheck, at startup and again on every periodic
// tick thereafter, reconciles the database accounting with the
// removes the expired negative cache rows at startup and again on every
// periodic tick thereafter, reconciles the database accounting with the
// cache directories at startup and again on the first periodic tick
// that comes at least ten times as long after a reconciliation pass
// ended as that pass took, and evicts to the configured limit on the given
@@ -504,7 +392,6 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
nextReconciliation := c.runReconciliationPass(ctx)
c.runCleanExpired(ctx)
c.runEvictNotCheckedRecently(ctx)
c.runEvictionPass(ctx)
ticker := time.NewTicker(interval)
@@ -530,7 +417,6 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
}
c.runCleanExpired(ctx)
c.runEvictNotCheckedRecently(ctx)
case <-c.evictionPressure:
}
@@ -588,27 +474,12 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
}
}
// runEvictNotCheckedRecently runs EvictNotCheckedRecently, logging a failure
// instead of propagating it (the loop must keep running). It does nothing
// once ctx is cancelled.
func (c *Cache) runEvictNotCheckedRecently(ctx context.Context) {
if ctx.Err() != nil {
return
}
err := c.EvictNotCheckedRecently(ctx)
if err != nil {
c.log.Warn("evicting cache content not checked recently failed",
"error", err)
}
}
// reconcileAccounting synchronizes the database size accounting with
// the actual contents of the cache directories. It runs at startup and
// then on the first periodic eviction tick that comes at least ten times
// as long after the last pass ended as that pass took, off the request
// hot path: it adopts variant files that predate the accounting table
// (or whose accounting insert failed, e.g. PutVariant's best-effort
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
// insert under transient DB contention), drops variant accounting rows
// whose files are missing, removes source blob files the database does
// not know, removes source and variant files that are empty or not the
@@ -644,15 +515,12 @@ func (c *Cache) reconcileAccounting(ctx context.Context) error {
}
// reconcileFiles walks baseDir, one of the cache's storage directories,
// sweeping stale temp files and passing each other file, with its info and
// name, to reconcileFile. It skips a directory it cannot list and a file
// whose info it cannot read, so one such entry does not stop the pass. It
// logs a warning naming the path and the error. A file renamed or removed
// after its directory was listed is skipped with no warning. Once ctx is
// cancelled it stops at the next file and returns ctx's error.
// sweeping stale temp files and passing each other file, with its name, to
// reconcileFile. Once ctx is cancelled it stops at the next file and
// returns ctx's error.
func (c *Cache) reconcileFiles(
ctx context.Context, baseDir string,
reconcileFile func(path string, info fs.FileInfo, name string) error,
reconcileFile func(path string, entry fs.DirEntry, name string) error,
) error {
return filepath.WalkDir(
baseDir,
@@ -661,38 +529,19 @@ func (c *Cache) reconcileFiles(
return ctx.Err()
}
if err != nil {
c.log.Warn("skipped unreadable entry in cache directory",
"path", path, "error", err)
return nil
}
if entry.IsDir() {
return nil
}
info, err := entry.Info()
if errors.Is(err, fs.ErrNotExist) {
return nil
}
if err != nil {
c.log.Warn("skipped unreadable entry in cache directory",
"path", path, "error", err)
return nil
if err != nil || entry.IsDir() {
return err
}
name := entry.Name()
if strings.HasPrefix(name, tempFilePrefix) {
c.sweepStaleTempFile(path, info)
c.sweepStaleTempFile(path, entry)
return nil
}
return reconcileFile(path, info, name)
return reconcileFile(path, entry, name)
},
)
}
@@ -702,8 +551,8 @@ func (c *Cache) reconcileFiles(
// and sweeping stale temp files.
func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
return c.reconcileFiles(ctx, c.variants.baseDir,
func(path string, info fs.FileInfo, name string) error {
return c.reconcileVariantFile(ctx, path, info, imagerequest.VariantKey(name))
func(path string, entry fs.DirEntry, name string) error {
return c.reconcileVariantFile(ctx, path, entry, imagerequest.VariantKey(name))
})
}
@@ -713,11 +562,16 @@ func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
// crash or a restored snapshot can leave it, is removed with its row and
// a warning naming it, so the variant is converted again.
func (c *Cache) reconcileVariantFile(
ctx context.Context, path string, info fs.FileInfo, cacheKey imagerequest.VariantKey,
ctx context.Context, path string, entry fs.DirEntry, cacheKey imagerequest.VariantKey,
) error {
info, err := entry.Info()
if err != nil {
return fmt.Errorf("failed to stat variant file: %w", err)
}
var recordedSize int64
err := c.db.QueryRowContext(ctx,
err = c.db.QueryRowContext(ctx,
`SELECT size_bytes FROM variant_content WHERE cache_key = ?`, string(cacheKey),
).Scan(&recordedSize)
@@ -734,24 +588,20 @@ func (c *Cache) reconcileVariantFile(
c.log.Warn("removing damaged cached variant file", "path", path,
"file_bytes", info.Size(), "recorded_bytes", recordedSize)
_, err = c.removeVariant(ctx, cacheKey)
return err
return c.evictVariant(ctx, cacheKey)
}
// adoptVariantFile inserts an accounting row for a variant file that
// has none, using the file's size and modification time, which is also
// the variant's checked_at.
// has none, using the file's size and modification time.
func (c *Cache) adoptVariantFile(
ctx context.Context, info fs.FileInfo, cacheKey imagerequest.VariantKey,
) error {
modTime := info.ModTime().UTC().Format(sqliteTimestampLayout)
_, err := c.db.ExecContext(ctx, `
INSERT INTO variant_content (cache_key, size_bytes, last_accessed_at,
checked_at)
VALUES (?, ?, ?, ?)
`, string(cacheKey), info.Size(), modTime, info.ModTime().UTC())
INSERT INTO variant_content (cache_key, size_bytes, last_accessed_at)
VALUES (?, ?, ?)
`, string(cacheKey), info.Size(), modTime)
if err != nil {
return fmt.Errorf("failed to adopt variant file into accounting: %w", err)
}
@@ -763,9 +613,7 @@ func (c *Cache) adoptVariantFile(
}
// reconcileVariantRows drops accounting rows whose variant files are
// missing, so the database never references deleted content. A file whose
// size cannot be read for another reason may still be on disk, so its row
// stays, with a warning naming its key and the error.
// missing, so the database never references deleted content.
func (c *Cache) reconcileVariantRows(ctx context.Context) error {
var after imagerequest.VariantKey
@@ -788,19 +636,11 @@ func (c *Cache) reconcileVariantRows(ctx context.Context) error {
return ctx.Err()
}
_, err := c.variants.Size(string(key))
if err == nil {
if c.variants.Exists(key) {
continue
}
if !errors.Is(err, ErrNotFound) {
c.log.Warn("kept accounting row for unreadable variant file",
"cache_key", key, "error", err)
continue
}
_, err = c.db.ExecContext(ctx,
_, err := c.db.ExecContext(ctx,
`DELETE FROM variant_content WHERE cache_key = ?`, string(key))
if err != nil {
return fmt.Errorf("failed to drop stale variant accounting row: %w", err)
@@ -864,8 +704,8 @@ func queryStringColumn[T ~string](
// and sweeping stale temp files.
func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
return c.reconcileFiles(ctx, c.srcContent.baseDir,
func(path string, info fs.FileInfo, name string) error {
return c.reconcileSourceFile(ctx, path, info, imagerequest.SourceKey(name))
func(path string, entry fs.DirEntry, name string) error {
return c.reconcileSourceFile(ctx, path, entry, imagerequest.SourceKey(name))
})
}
@@ -876,11 +716,16 @@ func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
// is removed with its row and a warning naming it, so the source is
// fetched again.
func (c *Cache) reconcileSourceFile(
ctx context.Context, path string, info fs.FileInfo, key imagerequest.SourceKey,
ctx context.Context, path string, entry fs.DirEntry, key imagerequest.SourceKey,
) error {
info, err := entry.Info()
if err != nil {
return fmt.Errorf("failed to stat source content file: %w", err)
}
var recordedSize int64
err := c.db.QueryRowContext(ctx,
err = c.db.QueryRowContext(ctx,
`SELECT size_bytes FROM source_content WHERE source_key = ?`,
string(key),
).Scan(&recordedSize)
@@ -898,9 +743,7 @@ func (c *Cache) reconcileSourceFile(
c.log.Warn("removing damaged cached source file", "path", path,
"file_bytes", info.Size(), "recorded_bytes", recordedSize)
_, err = c.removeSourceBlob(ctx, key)
return err
return c.evictSourceBlob(ctx, key)
}
// removeUntrackedSourceFile deletes a source blob file that has no
@@ -1047,12 +890,17 @@ func (c *Cache) correctUsageTotal(
// sweepStaleTempFile removes a temp file left behind by a crashed
// write once it is old enough that no in-flight store can own it.
func (c *Cache) sweepStaleTempFile(path string, info fs.FileInfo) {
func (c *Cache) sweepStaleTempFile(path string, entry fs.DirEntry) {
info, err := entry.Info()
if err != nil {
return
}
if time.Since(info.ModTime()) < staleTempFileAge {
return
}
err := os.Remove(path)
err = os.Remove(path)
if err != nil && !os.IsNotExist(err) {
c.log.Warn("failed to remove stale temp file", "path", path, "error", err)
+36 -330
View File
@@ -9,10 +9,8 @@ import (
"log/slog"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"syscall"
"testing"
"time"
@@ -69,7 +67,7 @@ func newEvictionTestCache(t *testing.T, maxBytes int64) (*Cache, string) {
db := evictionTestDB(t)
// maxBytes zero turns the disk cache off, as cache_max_bytes: 0 does.
cache, err := NewCache(db, Config{
cache, err := NewCache(db, CacheConfig{
StateDir: tmpDir,
NegativeTTL: 5 * time.Minute,
MaxBytes: maxBytes,
@@ -100,11 +98,11 @@ func storeEvictionTestSource(
},
}
key, err := cache.PutSource(
context.Background(), req.Source, bytes.NewReader(content), SourceInfo{},
key, err := cache.StoreSource(
context.Background(), req, bytes.NewReader(content),
)
if err != nil {
t.Fatalf("PutSource(%s%s) failed: %v", host, path, err)
t.Fatalf("StoreSource(%s%s) failed: %v", host, path, err)
}
return key
@@ -117,10 +115,9 @@ func storeEvictionTestVariant(
) {
t.Helper()
err := cache.PutVariant(t.Context(), key, SourceInfo{CheckedAt: time.Now()},
bytes.NewReader(content))
err := cache.StoreVariant(t.Context(), key, bytes.NewReader(content))
if err != nil {
t.Fatalf("PutVariant(%s) failed: %v", key, err)
t.Fatalf("StoreVariant(%s) failed: %v", key, err)
}
}
@@ -212,8 +209,7 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
t.Fatalf("failed to scan source_key: %v", err)
}
_, err = os.Stat(cache.srcContent.keyToPath(key))
if err != nil {
if !cache.srcContent.Exists(imagerequest.SourceKey(key)) {
t.Errorf("source_content references content %s but the file is missing",
key)
}
@@ -240,8 +236,7 @@ func assertNoDanglingReferences(t *testing.T, cache *Cache) {
t.Fatalf("failed to scan cache_key: %v", err)
}
_, err = os.Stat(cache.variants.keyToPath(key))
if err != nil {
if !cache.variants.Exists(imagerequest.VariantKey(key)) {
t.Errorf("variant_content references key %s but the file is missing", key)
}
}
@@ -339,8 +334,7 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
t.Errorf("usage after eviction = %d, want <= %d", usage, limit)
}
_, err = os.Stat(cache.variants.keyToPath(string(keys[0])))
if err == nil {
if cache.variants.Exists(keys[0]) {
t.Errorf("least recently used variant %s must be evicted", keys[0])
}
@@ -351,8 +345,7 @@ func TestEvictToLimitEvictsLeastRecentlyUsedFirst(t *testing.T) {
}
for _, key := range keys[1:] {
_, err = os.Stat(cache.variants.keyToPath(string(key)))
if err != nil {
if !cache.variants.Exists(key) {
t.Errorf("more recently used variant %s must survive eviction", key)
}
}
@@ -380,8 +373,7 @@ func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
}
for _, key := range keys {
_, err = os.Stat(cache.srcContent.keyToPath(string(key)))
if err != nil {
if !cache.srcContent.Exists(key) {
t.Errorf("source %s must not be evicted while usage is under the limit", key)
}
@@ -392,8 +384,7 @@ func TestEvictionKeepsEverythingWhenUnderLimit(t *testing.T) {
}
}
_, err = os.Stat(cache.variants.keyToPath(string(testVariantKeyOne)))
if err != nil {
if !cache.variants.Exists(testVariantKeyOne) {
t.Error("variant must not be evicted while usage is under the limit")
}
@@ -432,21 +423,20 @@ func assertDisabledCacheWritesAreNoOps(
ctx := t.Context()
err := cache.PutVariant(
ctx, imagerequest.CacheKey(req), SourceInfo{}, bytes.NewReader([]byte("data")),
err := cache.StoreVariant(
ctx, imagerequest.CacheKey(req), bytes.NewReader([]byte("data")),
)
if err != nil {
t.Fatalf("PutVariant on disabled cache must be a no-op, got error: %v", err)
t.Fatalf("StoreVariant on disabled cache must be a no-op, got error: %v", err)
}
key, err := cache.PutSource(ctx, req.Source, bytes.NewReader([]byte("data")),
SourceInfo{})
key, err := cache.StoreSource(ctx, req, bytes.NewReader([]byte("data")))
if err != nil {
t.Fatalf("PutSource on disabled cache must be a no-op, got error: %v", err)
t.Fatalf("StoreSource on disabled cache must be a no-op, got error: %v", err)
}
if key != "" {
t.Errorf("PutSource on disabled cache returned key %q, want empty", key)
t.Errorf("StoreSource on disabled cache returned key %q, want empty", key)
}
}
@@ -459,13 +449,13 @@ func assertDisabledCacheReadsAlwaysMiss(
ctx := t.Context()
reader, _, _, err := cache.Source(ctx, req.Source)
if !errors.Is(err, ErrNotFound) {
t.Fatalf("Source on disabled cache error = %v, want %v", err, ErrNotFound)
key, err := cache.LookupSource(ctx, req)
if err != nil {
t.Fatalf("LookupSource on disabled cache failed: %v", err)
}
if reader != nil {
t.Errorf("Source on disabled cache = reader %v, want none", reader)
if key != "" {
t.Errorf("LookupSource on disabled cache = %q, want empty", key)
}
}
@@ -554,7 +544,7 @@ func TestEvictionRunsUnderWritePressure(t *testing.T) {
}
// TestEvictionRunsOnPeriodicSchedule writes three variant files straight
// to disk, bypassing PutVariant, so they have no accounting rows and no
// to disk, bypassing StoreVariant, so they have no accounting rows and no
// write-pressure notification fires. Only a periodic reconciliation pass
// can then adopt them, and only the eviction pass that follows it can
// evict them.
@@ -596,7 +586,7 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
for i, key := range keys {
content := bytes.Repeat([]byte{fills[i]}, 1000)
err = cache.variants.Write(string(key), content)
_, err = cache.variants.Store(key, bytes.NewReader(content))
if err != nil {
t.Fatalf("failed to store variant file: %v", err)
}
@@ -615,8 +605,7 @@ func TestEvictionRunsOnPeriodicSchedule(t *testing.T) {
stored = 0
for _, key := range keys {
_, err = os.Stat(cache.variants.keyToPath(string(key)))
if err == nil {
if cache.variants.Exists(key) {
stored++
}
}
@@ -648,7 +637,9 @@ func TestStartEvictionReconcilesAccountingWithDisk(t *testing.T) {
// feature existed) must be adopted into the accounting.
untracked := bytes.Repeat([]byte{0x31}, 1000)
err := cache.variants.Write(string(testVariantKeyOne), untracked)
_, err := cache.variants.Store(
testVariantKeyOne, bytes.NewReader(untracked),
)
if err != nil {
t.Fatalf("failed to store untracked variant file: %v", err)
}
@@ -703,295 +694,10 @@ func TestStartEvictionReconcilesAccountingWithDisk(t *testing.T) {
}
}
// logLineNaming returns the first line of logs, written by slog's JSON
// handler, that is at level and names the variant key, or "" if none is.
func logLineNaming(logs, level string, key imagerequest.VariantKey) string {
for line := range strings.Lines(logs) {
if strings.Contains(line, `"level":"`+level+`"`) &&
strings.Contains(line, `"cache_key":"`+string(key)+`"`) {
return line
}
}
return ""
}
// TestReconcileVariantRowsDropsOnlyRowsOfMissingFiles gives
// reconcileVariantRows two tracked variants. The first one's file is a
// symbolic link to itself, so its size cannot be read. Its row must stay,
// with a warning naming its key and the error. The second one has no file.
// Its row must go, with the info line for a missing file.
func TestReconcileVariantRowsDropsOnlyRowsOfMissingFiles(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
storeEvictionTestVariant(t, cache, testVariantKeyOne, []byte("variant one"))
storeEvictionTestVariant(t, cache, testVariantKeyTwo, []byte("variant two"))
unreadablePath := cache.variants.keyToPath(string(testVariantKeyOne))
err := os.Remove(unreadablePath)
if err != nil {
t.Fatalf("failed to remove the first variant's file: %v", err)
}
err = os.Symlink(unreadablePath, unreadablePath)
if err != nil {
t.Fatalf("failed to link the first variant's path to itself: %v", err)
}
_, statErr := os.Stat(unreadablePath)
if statErr == nil || errors.Is(statErr, fs.ErrNotExist) {
t.Fatalf("os.Stat() of the first variant: error = %v, "+
"want an error other than a missing file", statErr)
}
err = os.Remove(cache.variants.keyToPath(string(testVariantKeyTwo)))
if err != nil {
t.Fatalf("failed to remove the second variant's file: %v", err)
}
var logBuf bytes.Buffer
cache.log = slog.New(slog.NewJSONHandler(&logBuf, nil))
err = cache.reconcileVariantRows(t.Context())
t.Logf("reconcileVariantRows() error = %v, log: %s", err, logBuf.String())
if err != nil {
t.Fatalf("reconcileVariantRows() error = %v", err)
}
rowQuery := `SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`
if n := countRows(t, cache, rowQuery, string(testVariantKeyOne)); n != 1 {
t.Errorf("the variant whose size cannot be read has %d rows, want 1", n)
}
if n := countRows(t, cache, rowQuery, string(testVariantKeyTwo)); n != 0 {
t.Errorf("the variant with no file has %d rows, want 0", n)
}
logs := logBuf.String()
warning := logLineNaming(logs, "WARN", testVariantKeyOne)
if !strings.Contains(warning, statErr.Error()) {
t.Errorf("no warning names %s and the error %q", testVariantKeyOne, statErr)
}
if line := logLineNaming(logs, "INFO", testVariantKeyOne); line != "" {
t.Errorf("the variant whose size cannot be read was logged at info: %s", line)
}
dropped := logLineNaming(logs, "INFO", testVariantKeyTwo)
if !strings.Contains(dropped, "dropped accounting row for missing variant file") {
t.Errorf("no info line says the file of %s is missing", testVariantKeyTwo)
}
}
// openDirectoryNearPathLimit makes a chain of directories inside dir, each
// named with 255 zeros, the longest name Linux allows. Linux refuses a path
// of 4096 bytes or more, and the chain goes as deep as it can below that.
// It returns the chain's deepest directory as an os.Root, closed when the
// test ends. An entry made through it with a 255-byte name has a path even
// the root user cannot read. The os.Root can make that entry because it
// names the entry only by its name, not by the whole path.
func openDirectoryNearPathLimit(t *testing.T, dir string) *os.Root {
t.Helper()
name := strings.Repeat("0", 255)
deepest := dir
for len(filepath.Join(deepest, name)) < 4096 {
deepest = filepath.Join(deepest, name)
}
err := os.MkdirAll(deepest, StorageDirPerm)
if err != nil {
t.Fatalf("failed to make a chain of directories: %v", err)
}
root, err := os.OpenRoot(deepest)
if err != nil {
t.Fatalf("os.OpenRoot(%s) error = %v", deepest, err)
}
t.Cleanup(func() { _ = root.Close() })
return root
}
// makeEntriesTooLongToRead makes a new directory and a new file in the
// deepest directory of the chain that openDirectoryNearPathLimit makes
// inside dir. Even the root user can then neither list the new directory
// nor read the new file's info. It returns the paths of the new directory
// and the new file.
func makeEntriesTooLongToRead(t *testing.T, dir string) []string {
t.Helper()
deepest := openDirectoryNearPathLimit(t, dir)
dirName := strings.Repeat("0", 255)
err := deepest.Mkdir(dirName, StorageDirPerm)
if err != nil {
t.Fatalf("failed to make a directory too deep to list: %v", err)
}
fileName := strings.Repeat("1", 255)
err = deepest.WriteFile(fileName, []byte("unreadable"), StorageFilePerm)
if err != nil {
t.Fatalf("failed to make a file too deep to read: %v", err)
}
return []string{
filepath.Join(deepest.Name(), dirName),
filepath.Join(deepest.Name(), fileName),
}
}
// makeTempFileTooLongToRead makes a temp file in the deepest directory of
// the chain that openDirectoryNearPathLimit makes inside dir. Even the
// root user then cannot read the temp file's info. It returns the temp
// file's path.
func makeTempFileTooLongToRead(t *testing.T, dir string) string {
t.Helper()
deepest := openDirectoryNearPathLimit(t, dir)
name := tempFilePrefix + strings.Repeat("1", 255-len(tempFilePrefix))
err := deepest.WriteFile(name, []byte("partial"), StorageFilePerm)
if err != nil {
t.Fatalf("failed to make a temp file too deep to read: %v", err)
}
return filepath.Join(deepest.Name(), name)
}
// assertSkippedEntryWarning checks that logs hold the warning "skipped
// unreadable entry in cache directory" naming path, with the error
// ENAMETOOLONG for that path.
func assertSkippedEntryWarning(t *testing.T, logs, path string) {
t.Helper()
for line := range strings.Lines(logs) {
if strings.Contains(line, `"level":"WARN"`) &&
strings.Contains(line, `"msg":"skipped unreadable entry in cache directory"`) &&
strings.Contains(line, `"path":"`+path+`"`) &&
strings.Contains(line, path+": "+syscall.ENAMETOOLONG.Error()) {
return
}
}
t.Errorf("no warning %q names %s with the error %q",
"skipped unreadable entry in cache directory", path, syscall.ENAMETOOLONG)
}
// TestReconciliationGoesOnPastEntriesItCannotRead puts, in each storage
// directory, a directory the pass cannot list and a file whose info it
// cannot read. The pass walks each directory in name order, so it reaches
// them before the cache's files, whose keys start with a letter. It must
// log a warning naming each one's path and the error, and still do the
// rest of its work:
// - adopt a variant file that has no row
// - drop the row of a variant whose file is missing
// - remove a source file that has no row
// - correct a wrong total cache usage
func TestReconciliationGoesOnPastEntriesItCannotRead(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
ctx := t.Context()
untrackedVariant := []byte("variant with no row")
err := cache.variants.Write(string(testVariantKeyOne), untrackedVariant)
if err != nil {
t.Fatalf("failed to write a variant file with no row: %v", err)
}
storeEvictionTestVariant(t, cache, testVariantKeyTwo, []byte("variant two"))
err = os.Remove(cache.variants.keyToPath(string(testVariantKeyTwo)))
if err != nil {
t.Fatalf("failed to remove the second variant's file: %v", err)
}
untrackedSourceKey := "ffeeddcc0001"
err = cache.srcContent.Write(untrackedSourceKey, []byte("source with no row"))
if err != nil {
t.Fatalf("failed to write a source file with no row: %v", err)
}
_, err = cache.db.ExecContext(ctx,
`UPDATE cache_usage SET total_size_bytes = 1 WHERE id = 1`)
if err != nil {
t.Fatalf("failed to set a wrong total: %v", err)
}
unreadable := slices.Concat(
makeEntriesTooLongToRead(t, cache.variants.baseDir),
makeEntriesTooLongToRead(t, cache.srcContent.baseDir),
)
logs := reconcileAsAtStartup(t, cache)
for _, path := range unreadable {
assertSkippedEntryWarning(t, logs, path)
}
rowQuery := `SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`
if n := countRows(t, cache, rowQuery, string(testVariantKeyOne)); n != 1 {
t.Errorf("the variant file with no row got %d rows, want 1", n)
}
if n := countRows(t, cache, rowQuery, string(testVariantKeyTwo)); n != 0 {
t.Errorf("the variant with no file kept %d rows, want 0", n)
}
_, err = os.Stat(cache.srcContent.keyToPath(untrackedSourceKey))
if err == nil {
t.Error("the source file with no row is still on disk")
}
usage, err := cache.UsageBytes(ctx)
if err != nil {
t.Fatalf("UsageBytes() error = %v", err)
}
if usage != int64(len(untrackedVariant)) {
t.Errorf("UsageBytes() = %d, want %d, the size of the adopted variant",
usage, len(untrackedVariant))
}
}
// TestReconciliationWarnsOfTempFilesWhoseInfoItCannotRead puts, in each
// storage directory, a temp file whose info the pass cannot read. The
// pass must log a warning naming each one's path and the error, as it
// does for any other file whose info it cannot read.
func TestReconciliationWarnsOfTempFilesWhoseInfoItCannotRead(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
unreadable := []string{
makeTempFileTooLongToRead(t, cache.variants.baseDir),
makeTempFileTooLongToRead(t, cache.srcContent.baseDir),
}
logs := reconcileAsAtStartup(t, cache)
for _, path := range unreadable {
assertSkippedEntryWarning(t, logs, path)
}
}
// TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup proves
// reconciliation is not a one-shot startup-only pass: it must also run
// on the periodic ticker, so a variant file that lands on disk with no
// accounting row well after startup (e.g. because PutVariant's
// accounting row well after startup (e.g. because StoreVariant's
// best-effort accounting insert failed under transient contention, or
// any other cause of an untracked file appearing during steady-state
// operation) is still adopted into accounting eventually, rather than
@@ -1027,12 +733,14 @@ func TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup(t *testing.T) {
// Simulate a variant whose accounting insert failed after the
// process was already running and serving requests: the content
// file is written directly, bypassing PutVariant's (and thus its
// file is written directly, bypassing StoreVariant's (and thus its
// accounting insert) entirely, exactly as would happen if that
// insert had failed and only the file write had succeeded.
untracked := bytes.Repeat([]byte{0x41}, 900)
err = cache.variants.Write("aabbccdd0099", untracked)
_, err = cache.variants.Store(
"aabbccdd0099", bytes.NewReader(untracked),
)
if err != nil {
t.Fatalf("failed to store untracked variant file: %v", err)
}
@@ -1338,15 +1046,13 @@ func assertOnlyOldestSourceEvicted(
) {
t.Helper()
_, err := os.Stat(cache.srcContent.keyToPath(string(keys[0])))
if err == nil {
if cache.srcContent.Exists(keys[0]) {
t.Errorf("source %s, evicted when the context was cancelled, "+
"is still on disk", keys[0])
}
for _, key := range keys[1:] {
_, err = os.Stat(cache.srcContent.keyToPath(string(key)))
if err != nil {
if !cache.srcContent.Exists(key) {
t.Errorf("source %s was evicted after the context was cancelled", key)
}
-198
View File
@@ -1,198 +0,0 @@
package imgcache
import (
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"time"
)
// Storage constants.
const (
// StorageDirPerm is the permission mode for storage directories.
StorageDirPerm = 0750
// StorageFilePerm is the permission mode for storage files.
StorageFilePerm = 0600
// MinHashLength is the minimum hash length for path splitting.
MinHashLength = 4
)
// Storage errors.
var (
ErrNotFound = errors.New("content not found")
)
// FileStore is a directory of files, each named by a hex key. The file for
// a key is at <basedir>/<ab>/<cd>/<key>, where <ab> and <cd> are the key's
// first and second pairs of characters. The cache keeps its source images in
// one FileStore and its variants in another.
type FileStore struct {
baseDir string
}
// NewFileStore creates the directory baseDir if it is missing and returns
// the FileStore of the files in it.
func NewFileStore(baseDir string) (*FileStore, error) {
err := os.MkdirAll(baseDir, StorageDirPerm)
if err != nil {
return nil, fmt.Errorf("failed to create storage directory: %w", err)
}
return &FileStore{baseDir: baseDir}, nil
}
// Write writes data with writeFileSynced to the file for key, replacing any
// file already there.
func (s *FileStore) Write(key string, data []byte) error {
path := s.keyToPath(key)
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm)
if err != nil {
return fmt.Errorf("failed to create directory: %w", err)
}
return writeFileSynced(path, data)
}
// Open opens the file for key and returns it with its information, which
// gives its size and its modification time. A missing file is ErrNotFound.
func (s *FileStore) Open(key string) (*os.File, fs.FileInfo, error) {
f, err := os.Open(s.keyToPath(key))
if err != nil {
if os.IsNotExist(err) {
return nil, nil, ErrNotFound
}
return nil, nil, fmt.Errorf("failed to open content: %w", err)
}
info, err := f.Stat()
if err != nil {
_ = f.Close()
return nil, nil, fmt.Errorf("failed to stat content: %w", err)
}
return f, info, nil
}
// Size returns the size in bytes of the file for key. A missing file is
// ErrNotFound.
func (s *FileStore) Size(key string) (int64, error) {
stat, err := os.Stat(s.keyToPath(key))
if err != nil {
if os.IsNotExist(err) {
return 0, ErrNotFound
}
return 0, fmt.Errorf("failed to stat content: %w", err)
}
return stat.Size(), nil
}
// SetModTime sets the modification time of the file for key to modTime,
// leaving its access time as it is.
func (s *FileStore) SetModTime(key string, modTime time.Time) error {
err := os.Chtimes(s.keyToPath(key), time.Time{}, modTime)
if err != nil {
return fmt.Errorf("failed to set modification time: %w", err)
}
return nil
}
// Delete removes the file for key. A missing file is not an error.
func (s *FileStore) Delete(key string) error {
err := os.Remove(s.keyToPath(key))
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("failed to delete content: %w", err)
}
return nil
}
// writeFileSynced writes data to path through a temp file in the same
// directory: the temp file is synced to disk, renamed to path, and the
// directory synced after, so that neither a concurrent reader nor a
// crash finds a partial file under path. When the directory sync fails,
// the file is removed from path, so that a failed write leaves no file
// there.
func writeFileSynced(path string, data []byte) error {
dir := filepath.Dir(path)
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
if err != nil {
return fmt.Errorf("failed to create temp file: %w", err)
}
tmpPath := tmpFile.Name()
_, err = tmpFile.Write(data)
if err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to write temp file: %w", err)
}
err = tmpFile.Sync()
if err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to sync temp file: %w", err)
}
err = tmpFile.Close()
if err != nil {
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to close temp file: %w", err)
}
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
if err != nil {
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to rename temp file: %w", err)
}
err = syncDirectory(dir)
if err != nil {
_ = os.Remove(path)
return err
}
return nil
}
// syncDirectory syncs the directory dir to disk, so that a file just
// renamed into it is still there after a crash.
func syncDirectory(dir string) error {
d, err := os.Open(filepath.Clean(dir))
if err != nil {
return fmt.Errorf("failed to open directory: %w", err)
}
err = d.Sync()
_ = d.Close()
if err != nil {
return fmt.Errorf("failed to sync directory: %w", err)
}
return nil
}
// keyToPath converts a key to a file path: <basedir>/<ab>/<cd>/<key>
func (s *FileStore) keyToPath(key string) string {
if len(key) < MinHashLength {
return filepath.Join(s.baseDir, key)
}
return filepath.Join(s.baseDir, key[0:2], key[2:4], key)
}
@@ -1,184 +0,0 @@
package imgcache
import (
"bytes"
"errors"
"io"
"os"
"path/filepath"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
func TestFileStore_WriteAndOpen(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewFileStore(tmpDir)
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
content := []byte("hello world")
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
err = storage.Write(key, content)
if err != nil {
t.Fatalf("Write() error = %v", err)
}
// Verify file exists at expected path
expectedPath := filepath.Join(tmpDir, key[0:2], key[2:4], key)
_, err = os.Stat(expectedPath)
if err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
// Open and verify size and content
r, info, err := storage.Open(key)
if err != nil {
t.Fatalf("Open() error = %v", err)
}
defer func() { _ = r.Close() }()
if info.Size() != int64(len(content)) {
t.Errorf("Open() size = %d, want %d", info.Size(), len(content))
}
loaded, err := io.ReadAll(r)
if err != nil {
t.Fatalf("ReadAll() error = %v", err)
}
if !bytes.Equal(loaded, content) {
t.Errorf("Open() content = %q, want %q", loaded, content)
}
}
func TestFileStore_OpenNotFound(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewFileStore(tmpDir)
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
_, _, err = storage.Open("nonexistent")
if !errors.Is(err, ErrNotFound) {
t.Errorf("Open() error = %v, want ErrNotFound", err)
}
}
// TestFileStore_Size checks that Size returns the size of a written file,
// and ErrNotFound for a key with no file.
func TestFileStore_Size(t *testing.T) {
t.Parallel()
storage, err := NewFileStore(t.TempDir())
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
content := []byte("the file's content")
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
err = storage.Write(key, content)
if err != nil {
t.Fatalf("Write() error = %v", err)
}
size, err := storage.Size(key)
if err != nil || size != int64(len(content)) {
t.Errorf("Size() = %d, error %v; want %d", size, err, len(content))
}
_, err = storage.Size("nonexistent")
if !errors.Is(err, ErrNotFound) {
t.Errorf("Size() of a missing file error = %v, want ErrNotFound", err)
}
}
func TestFileStore_Delete(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewFileStore(tmpDir)
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
content := []byte("to be deleted")
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
path := storage.keyToPath(key)
err = storage.Write(key, content)
if err != nil {
t.Fatalf("Write() error = %v", err)
}
_, err = os.Stat(path)
if err != nil {
t.Errorf("file missing before Delete(): %v", err)
}
err = storage.Delete(key)
if err != nil {
t.Fatalf("Delete() error = %v", err)
}
_, err = os.Stat(path)
if err == nil {
t.Error("file still on disk after Delete()")
}
}
func TestFileStore_DeleteNonexistent(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewFileStore(tmpDir)
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
// Should not error
err = storage.Delete("nonexistent")
if err != nil {
t.Errorf("Delete() error = %v, want nil", err)
}
}
func TestFileStore_KeyToPath(t *testing.T) {
t.Parallel()
tmpDir := t.TempDir()
storage, err := NewFileStore(tmpDir)
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
}
// Test by writing and verifying the resulting path structure
content := []byte("test content for path verification")
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
err = storage.Write(key, content)
if err != nil {
t.Fatalf("Write() error = %v", err)
}
expectedPath := filepath.Join(tmpDir, key[0:2], key[2:4], key)
_, err = os.Stat(expectedPath)
if err != nil {
t.Errorf("File not at expected path %s: %v", expectedPath, err)
}
}
+29
View File
@@ -0,0 +1,29 @@
// Package imgcache provides types for the image caching proxy.
package imgcache
import "io"
// ImageResponse represents a processed image ready to serve
type ImageResponse struct {
// Content is the image data reader
Content io.ReadCloser
// ContentLength is the size in bytes (-1 if unknown)
ContentLength int64
// ContentType is the MIME type of the response
ContentType string
// ETag is the entity tag for caching
ETag string
// CacheStatus indicates HIT or MISS
CacheStatus CacheStatus
// FetchedBytes is the number of bytes fetched from upstream (0 if cache hit)
FetchedBytes int64
}
// CacheStatus indicates whether the response was served from cache.
type CacheStatus string
// Cache status values for response headers.
const (
CacheHit CacheStatus = "HIT"
CacheMiss CacheStatus = "MISS"
)
@@ -1,4 +1,4 @@
package imageproxy
package imgcache
import (
"image/color"
@@ -41,8 +41,7 @@ func holdProcessingSlot(
input, feed := io.Pipe()
go func() {
_, _ = proc.Process(t.Context(), input, imagerequest.FormatJPEG,
&imagerequest.Transform{})
_, _ = proc.Process(t.Context(), input, &imagerequest.Transform{})
}()
_, err := feed.Write([]byte{0})
@@ -56,17 +55,16 @@ func holdProcessingSlot(
return release
}
// TestProxy_Get_WaitsForSlotBeforeReadingCachedSource checks that a
// TestService_Get_WaitsForSlotBeforeReadingCachedSource checks that a
// request whose source is cached holds none of it while it waits for a
// processing slot: past the first bytes, which give the source's format, it
// reads the cached file only once it has a slot. With the only slot held, a
// request for a new width of the cached 100x100 photo waits; the cached file
// is then rewritten as a 100x50 image before the slot is freed, so the
// request must answer with that image scaled to 40x20.
func TestProxy_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
// processing slot: it reads the cached file only once it has a slot. With
// the only slot held, a request for a new width of the cached 100x100 photo
// waits; the cached file is then rewritten as a 100x50 image before the slot
// is freed, so the request must answer with that image scaled to 40x20.
func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
svc, fixtures := SetupTestService(t)
svc.processor = imageprocessor.New(
imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1},
)
@@ -79,15 +77,13 @@ func TestProxy_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
_ = resp.Content.Close()
cachedSource := widthOnlyRequest(fixtures, 50).Source
reader, _, _, err := svc.cache.Source(t.Context(), cachedSource)
if err != nil {
t.Fatalf("Source() error = %v; want the cached source", err)
sourceKey, err := svc.cache.LookupSource(t.Context(),
widthOnlyRequest(fixtures, 50))
if err != nil || sourceKey == "" {
t.Fatalf("LookupSource() = %q, %v; want the cached source",
sourceKey, err)
}
_ = reader.Close()
release := holdProcessingSlot(t, svc.processor)
var (
@@ -107,7 +103,7 @@ func TestProxy_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
// before waiting, it would have read it by now.
time.Sleep(100 * time.Millisecond)
err = os.WriteFile(sourceFilePath(svc, cachedSource.Key()),
err = os.WriteFile(svc.cache.srcContent.keyToPath(sourceKey),
generateTestJPEG(t, 100, 50, color.RGBA{0, 0, 255, 255}), 0o600)
if err != nil {
t.Fatalf("failed to rewrite the cached source: %v", err)
-88
View File
@@ -1,88 +0,0 @@
package imgcache
import (
"fmt"
"github.com/prometheus/client_golang/prometheus"
)
// registerMetrics makes the cache's metrics and registers them on
// metricsRegistry. The metrics are kept in memory, so updating one waits
// for nothing and makes no database call. It fails when metricsRegistry
// already holds metrics of the same names.
func (c *Cache) registerMetrics(metricsRegistry prometheus.Registerer) error {
c.cacheHits = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_cache_hits_total",
Help: "Image requests served from the cache.",
})
c.cacheMisses = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_cache_misses_total",
Help: "Image requests not served from the cache.",
})
c.sourceCacheHits = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_source_cache_hits_total",
Help: "Variants made or checked whose source was in the cache.",
})
c.sourceCacheMisses = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_source_cache_misses_total",
Help: "Variants made or checked whose source was not in the cache.",
})
c.usageBytes = prometheus.NewGauge(prometheus.GaugeOpts{
Name: "pixa_cache_usage_bytes",
Help: "Bytes of sources and variants in the cache, as eviction last read them.",
})
c.maxBytes = prometheus.NewGauge(prometheus.GaugeOpts{
Name: "pixa_cache_max_bytes",
Help: "The disk cache limit in bytes, cache_max_bytes.",
})
c.evictions = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_cache_evictions_total",
Help: "Sources and variants removed by eviction.",
})
c.evictedBytes = prometheus.NewCounter(prometheus.CounterOpts{
Name: "pixa_cache_evicted_bytes_total",
Help: "Bytes of the sources and variants removed by eviction.",
})
c.maxBytes.Set(float64(c.config.MaxBytes))
for _, metric := range []prometheus.Collector{
c.cacheHits, c.cacheMisses, c.sourceCacheHits, c.sourceCacheMisses,
c.usageBytes, c.maxBytes, c.evictions, c.evictedBytes,
} {
err := metricsRegistry.Register(metric)
if err != nil {
return fmt.Errorf("failed to register the cache's metrics: %w", err)
}
}
return nil
}
// IncrementStats counts a cache hit or miss of an image request. It only
// adds to the cache's counters in memory, so it never waits for the
// database or fails.
func (c *Cache) IncrementStats(hit bool) {
if hit {
c.cacheHits.Inc()
} else {
c.cacheMisses.Inc()
}
}
// IncrementSourceStats counts whether a variant being made or checked found
// its source in the cache. Like IncrementStats, it never waits or fails.
func (c *Cache) IncrementSourceStats(hit bool) {
if hit {
c.sourceCacheHits.Inc()
} else {
c.sourceCacheMisses.Inc()
}
}
// countEviction counts one source or variant of sizeBytes bytes removed by
// eviction.
func (c *Cache) countEviction(sizeBytes int64) {
c.evictions.Inc()
c.evictedBytes.Add(float64(sizeBytes))
}
-137
View File
@@ -1,137 +0,0 @@
package imgcache
import (
"bytes"
"testing"
"time"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// gatheredValue returns the value of the metric name on registry: a
// counter's or a gauge's value.
func gatheredValue(t *testing.T, registry *prometheus.Registry, name string) float64 {
t.Helper()
families, err := registry.Gather()
if err != nil {
t.Fatalf("failed to gather the metrics: %v", err)
}
for _, family := range families {
if family.GetName() != name {
continue
}
metric := family.GetMetric()[0]
if metric.GetGauge() != nil {
return metric.GetGauge().GetValue()
}
return metric.GetCounter().GetValue()
}
t.Fatalf("the metric %s is not on the registry", name)
return 0
}
// TestEvictToLimit_CountsEvictionsAndUsage stores four variants of 1000
// bytes in a cache limited to 3000 bytes. EvictToLimit must count the one
// eviction and its bytes, and set the usage and limit gauges.
func TestEvictToLimit_CountsEvictionsAndUsage(t *testing.T) {
t.Parallel()
const limit = 3000
registry := prometheus.NewRegistry()
cache, err := NewCache(evictionTestDB(t),
Config{StateDir: t.TempDir(), MaxBytes: limit}, registry)
if err != nil {
t.Fatalf("NewCache() error = %v", err)
}
keys := []imagerequest.VariantKey{
testVariantKeyOne, testVariantKeyTwo,
testVariantKeyThree, testVariantKeyFour,
}
for i, key := range keys {
storeEvictionTestVariant(t, cache, key, bytes.Repeat([]byte{0x01}, 1000))
setVariantLastAccessed(t, cache, key,
time.Now().Add(-time.Duration(len(keys)-i)*time.Hour))
}
err = cache.EvictToLimit(t.Context())
if err != nil {
t.Fatalf("EvictToLimit() error = %v", err)
}
want := map[string]float64{
"pixa_cache_evictions_total": 1,
"pixa_cache_evicted_bytes_total": 1000,
"pixa_cache_usage_bytes": 3000,
"pixa_cache_max_bytes": limit,
}
for name, value := range want {
got := gatheredValue(t, registry, name)
t.Logf("%s = %v", name, got)
if got != value {
t.Errorf("%s = %v, want %v", name, got, value)
}
}
}
// TestEvictNotCheckedRecently_CountsEvictions stores a source of 6 bytes and
// a variant of 7 bytes last checked with upstream 31 days ago.
// EvictNotCheckedRecently must count both evictions and their 13 bytes.
func TestEvictNotCheckedRecently_CountsEvictions(t *testing.T) {
t.Parallel()
registry := prometheus.NewRegistry()
cache, err := NewCache(evictionTestDB(t),
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
if err != nil {
t.Fatalf("NewCache() error = %v", err)
}
old := time.Now().Add(-31 * 24 * time.Hour)
_, err = cache.PutSource(t.Context(),
imagerequest.Source{Host: testHostCDN, Path: "/old.jpg"},
bytes.NewReader([]byte("source")),
SourceInfo{FetchedAt: old, CheckedAt: old})
if err != nil {
t.Fatalf("PutSource() error = %v", err)
}
err = cache.PutVariant(t.Context(), testVariantKeyOne,
SourceInfo{CheckedAt: old}, bytes.NewReader([]byte("variant")))
if err != nil {
t.Fatalf("PutVariant() error = %v", err)
}
err = cache.EvictNotCheckedRecently(t.Context())
if err != nil {
t.Fatalf("EvictNotCheckedRecently() error = %v", err)
}
want := map[string]float64{
"pixa_cache_evictions_total": 2,
"pixa_cache_evicted_bytes_total": 13,
}
for name, value := range want {
got := gatheredValue(t, registry, name)
t.Logf("%s = %v", name, got)
if got != value {
t.Errorf("%s = %v, want %v", name, got, value)
}
}
}
+8
View File
@@ -0,0 +1,8 @@
package imgcache
import "time"
// CacheConfig defaults.
const (
DefaultNegativeTTL = 5 * time.Minute
)
@@ -17,7 +17,7 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
db := setupTestDB(t)
dir := t.TempDir()
cache, err := NewCache(db, Config{
cache, err := NewCache(db, CacheConfig{
StateDir: dir,
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -35,7 +35,7 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
}
// Initially should not be in negative cache
hit, err := cache.Failure(ctx, req.Source)
hit, err := cache.checkNegativeCache(ctx, req)
if err != nil {
t.Fatal(err)
}
@@ -45,13 +45,13 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
}
// Store a negative entry
err = cache.PutFailure(ctx, req.Source)
err = cache.StoreNegative(ctx, req)
if err != nil {
t.Fatal(err)
}
// Now should be in negative cache
hit, err = cache.Failure(ctx, req.Source)
hit, err = cache.checkNegativeCache(ctx, req)
if err != nil {
t.Fatal(err)
}
@@ -67,7 +67,7 @@ func TestNegativeCache_Expired(t *testing.T) {
db := setupTestDB(t)
dir := t.TempDir()
cache, err := NewCache(db, Config{
cache, err := NewCache(db, CacheConfig{
StateDir: dir,
NegativeTTL: 1 * time.Millisecond, // very short TTL
MaxBytes: 1 << 30,
@@ -85,7 +85,7 @@ func TestNegativeCache_Expired(t *testing.T) {
}
// Store a negative entry with very short TTL
err = cache.PutFailure(ctx, req.Source)
err = cache.StoreNegative(ctx, req)
if err != nil {
t.Fatal(err)
}
@@ -94,7 +94,7 @@ func TestNegativeCache_Expired(t *testing.T) {
time.Sleep(10 * time.Millisecond)
// Should no longer be in negative cache
hit, err := cache.Failure(ctx, req.Source)
hit, err := cache.checkNegativeCache(ctx, req)
if err != nil {
t.Fatal(err)
}
@@ -104,6 +104,71 @@ func TestNegativeCache_Expired(t *testing.T) {
}
}
// TestService_Get_ServesCachedVariantOfNegativeCachedURL caches a variant of
// the test photo, then keeps a failed fetch of the photo's URL in the negative
// cache, as a failed fetch for another variant does once the source has been
// evicted. The cached variant is still served, and a variant not in the cache
// is made from the cached source. Once the source is evicted, a variant not in
// the cache needs a fetch and is refused.
func TestService_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
resp, err := svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
_ = resp.Content.Close()
err = svc.cache.StoreNegative(t.Context(),
photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("StoreNegative() error = %v", err)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() of the cached variant error = %v", err)
}
_ = resp.Content.Close()
if resp.CacheStatus != CacheHit {
t.Errorf("cached variant CacheStatus = %q, want %q",
resp.CacheStatus, CacheHit)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
t.Logf("Get() of a variant made from the cached source: error = %v", err)
if err != nil {
t.Fatalf("Get() of a variant made from the cached source error = %v", err)
}
_ = resp.Content.Close()
sourceKey, err := svc.cache.LookupSource(t.Context(),
photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil || sourceKey == "" {
t.Fatalf("LookupSource() = %q, %v, want the cached source", sourceKey, err)
}
err = svc.cache.evictSourceBlob(t.Context(), sourceKey)
if err != nil {
t.Fatalf("evictSourceBlob() error = %v", err)
}
_, err = svc.Get(t.Context(), photoVariant(fixtures, 60, imagerequest.FitCover))
t.Logf("Get() of a variant whose source was evicted: error = %v", err)
if !errors.Is(err, ErrNegativeCached) {
t.Errorf("Get() of a variant whose source was evicted error = %v, want %v",
err, ErrNegativeCached)
}
}
// TestEvictionLoopRemovesExpiredNegativeCacheRows stores a failed fetch
// that expires a second later and starts the eviction loop with a short
// interval: a later tick must remove its row although its URL is never
@@ -125,7 +190,7 @@ func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t), Config{
cache, err := NewCache(evictionTestDB(t), CacheConfig{
StateDir: t.TempDir(),
NegativeTTL: time.Second,
MaxBytes: tt.maxBytes,
@@ -138,9 +203,9 @@ func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
Source: imagerequest.Source{Host: testHostExample, Path: "/gone.jpg"},
}
err = cache.PutFailure(t.Context(), expired.Source)
err = cache.StoreNegative(t.Context(), expired)
if err != nil {
t.Fatalf("PutFailure failed: %v", err)
t.Fatalf("StoreNegative failed: %v", err)
}
const notExpiredPath = "/later.jpg"
@@ -204,7 +269,7 @@ func TestCleanExpiredDeletesInBatches(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t),
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("failed to create cache: %v", err)
@@ -260,7 +325,7 @@ func TestCleanExpiredRemovesMoreThanOneBatch(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t),
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("failed to create cache: %v", err)
@@ -292,7 +357,7 @@ func TestCleanExpiredStopsOnceContextEnds(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t),
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("failed to create cache: %v", err)
@@ -345,3 +410,15 @@ func negativeCacheTableRowCount(tb testing.TB, cache *Cache) int {
return count
}
func TestService_Get_ReturnsErrorForNegativeCachedURL(t *testing.T) {
t.Parallel()
// This test verifies that Service.Get() checks the negative cache
// We can't easily test the full pipeline without vips, but we can
// verify the error type
err := ErrNegativeCached
if !errors.Is(err, ErrNegativeCached) {
t.Error("ErrNegativeCached should be identifiable with errors.Is")
}
}
@@ -12,6 +12,67 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest"
)
// getImage calls Get for req and returns the image it answered with and
// its cache status.
func getImage(
t *testing.T, svc *Service, req *imagerequest.Request,
) ([]byte, CacheStatus) {
t.Helper()
resp, err := svc.Get(t.Context(), req)
if err != nil {
t.Fatalf("Get() of quality %d error = %v", req.Transform.Quality, err)
}
defer func() { _ = resp.Content.Close() }()
body, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("reading the image of quality %d: %v", req.Transform.Quality, err)
}
return body, resp.CacheStatus
}
// fileSize returns the size of the file at path.
func fileSize(t *testing.T, path string) int64 {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
return info.Size()
}
// truncateFile cuts the file at path down to size bytes.
func truncateFile(t *testing.T, path string, size int64) {
t.Helper()
t.Logf("cutting %s from %d to %d bytes",
filepath.Base(path), fileSize(t, path), size)
err := os.Truncate(path, size)
if err != nil {
t.Fatalf("truncating %s: %v", path, err)
}
}
// cachedSourcePath returns the path of the cached source file req uses.
func cachedSourcePath(
t *testing.T, svc *Service, req *imagerequest.Request,
) string {
t.Helper()
sourceKey, err := svc.cache.LookupSource(t.Context(), req)
if err != nil || sourceKey == "" {
t.Fatalf("LookupSource() = %q, %v; want the cached source", sourceKey, err)
}
return svc.cache.srcContent.keyToPath(sourceKey)
}
// reconcileAsAtStartup runs the cache's reconciliation pass, as pixa does
// when it starts, and returns what the pass logged.
func reconcileAsAtStartup(t *testing.T, cache *Cache) string {
@@ -46,67 +107,127 @@ func assertWarningNames(t *testing.T, logs, path string) {
t.Errorf("reconciliation logged no warning naming %s", path)
}
// TestReconciliationRemovesDamagedFiles empties a cached source file and a
// cached variant file, or cuts them to half their size, as a crash or a
// restored snapshot could. It checks that the reconciliation pass removes
// each file and its row, with a warning naming the file.
func TestReconciliationRemovesDamagedFiles(t *testing.T) {
// TestService_Get_FetchesAgainForEmptyCachedSource empties the cached source
// file, as a crash or a restored snapshot could, and runs the reconciliation
// pass, which must remove it with a warning naming it. It then asks for two
// new sizes of the source: the first must fetch it again and cache it whole,
// so that the second is made from the cache without a third fetch.
func TestService_Get_FetchesAgainForEmptyCachedSource(t *testing.T) {
t.Parallel()
content := []byte("the whole cached image")
svc, fixtures := SetupTestService(t)
for name, size := range map[string]int64{
"empty": 0,
"partial": int64(len(content) / 2),
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
cache, _ := newEvictionTestCache(t, 1<<30)
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
sourceKey := storeEvictionTestSource(t, cache, testHostCDN, testPathCat, content)
storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
truncateFile(t, path, 0)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
sourcePath := cache.srcContent.keyToPath(string(sourceKey))
variantPath := cache.variants.keyToPath(string(testVariantKeyOne))
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
for _, path := range []string{sourcePath, variantPath} {
err := os.Truncate(path, size)
if err != nil {
t.Fatalf("cutting %s to %d bytes: %v", path, size, err)
}
}
fetches := readCounters(svc.cache).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
logs := reconcileAsAtStartup(t, cache)
assertWarningNames(t, logs, sourcePath)
assertWarningNames(t, logs, variantPath)
_, err := os.Stat(sourcePath)
if err == nil {
t.Error("the damaged source file is still on disk")
}
_, err = os.Stat(variantPath)
if err == nil {
t.Error("the damaged variant file is still on disk")
}
rows := countRows(t, cache,
`SELECT COUNT(*) FROM source_content WHERE source_key = ?`,
string(sourceKey))
if rows != 0 {
t.Errorf("the damaged source file kept %d rows, want 0", rows)
}
rows = countRows(t, cache,
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`,
string(testVariantKeyOne))
if rows != 0 {
t.Errorf("the damaged variant file kept %d rows, want 0", rows)
}
})
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2: the empty cached source "+
"was kept and fetched again on every new size", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// TestService_Get_FetchesAgainForPartialCachedSource cuts the cached source
// file to half its size, as a crash or a restored snapshot could, and runs
// the reconciliation pass, which must remove it with a warning naming it.
// It then asks for two new sizes of the source: the first must fetch it
// again rather than fail to decode the partial file, and the second is
// made from the cache.
func TestService_Get_FetchesAgainForPartialCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
truncateFile(t, path, wholeSize/2)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
fetches := readCounters(svc.cache).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// assertConvertedAgain caches a variant of the photo, cuts its file to size
// bytes and runs the reconciliation pass, which must remove it with a
// warning naming it. It then checks that Get converts the image again
// instead of serving the file, and that the next Get serves the whole
// variant from the cache.
func assertConvertedAgain(t *testing.T, size int64) {
t.Helper()
svc, fixtures := SetupTestService(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := svc.cache.variants.keyToPath(imagerequest.CacheKey(req))
truncateFile(t, path, size)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
got, status := getImage(t, svc, req)
t.Logf("after the cut: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the cut: Get() = %d bytes, %s; want the %d bytes "+
"converted again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
t.Logf("next request: %d bytes, %s", len(got), status)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestService_Get_ConvertsAgainForEmptyCachedVariant empties a cached
// variant file, as a crash or a restored snapshot could, and checks that it
// is converted again rather than served empty.
func TestService_Get_ConvertsAgainForEmptyCachedVariant(t *testing.T) {
t.Parallel()
assertConvertedAgain(t, 0)
}
// TestService_Get_ConvertsAgainForPartialCachedVariant cuts a cached
// variant file to ten bytes, as a crash or a restored snapshot could, and
// checks that it is converted again rather than served cut short.
func TestService_Get_ConvertsAgainForPartialCachedVariant(t *testing.T) {
t.Parallel()
assertConvertedAgain(t, 10)
}
// TestReconciliationRemovesEmptyVariantFileWithoutRow checks that the
@@ -118,19 +239,16 @@ func TestReconciliationRemovesEmptyVariantFileWithoutRow(t *testing.T) {
cache, _ := newEvictionTestCache(t, 1<<30)
err := cache.variants.Write(string(testVariantKeyOne), nil)
_, err := cache.variants.Store(testVariantKeyOne, bytes.NewReader(nil))
if err != nil {
t.Fatalf("failed to store an empty variant file: %v", err)
}
variantPath := cache.variants.keyToPath(string(testVariantKeyOne))
logs := reconcileAsAtStartup(t, cache)
assertWarningNames(t, logs, variantPath)
assertWarningNames(t, logs, cache.variants.keyToPath(testVariantKeyOne))
_, err = os.Stat(variantPath)
if err == nil {
if cache.variants.Exists(testVariantKeyOne) {
t.Error("the empty variant file is still on disk")
}
@@ -142,19 +260,19 @@ func TestReconciliationRemovesEmptyVariantFileWithoutRow(t *testing.T) {
}
}
// TestFileStore_WriteReplacesFileOfAnotherSize checks that a write over a
// file of another size at the source's path, such as a partial file a crash
// left, replaces it with the content.
func TestFileStore_WriteReplacesFileOfAnotherSize(t *testing.T) {
// TestContentStorage_StoreReplacesFileOfAnotherSize checks that a
// store over a file of another size at the source's path, such as a
// partial file a crash left, replaces it with the content.
func TestContentStorage_StoreReplacesFileOfAnotherSize(t *testing.T) {
t.Parallel()
storage, err := NewFileStore(t.TempDir())
storage, err := NewContentStorage(t.TempDir())
if err != nil {
t.Fatalf("NewFileStore() error = %v", err)
t.Fatalf("NewContentStorage() error = %v", err)
}
content := []byte("the whole source image")
key := string(imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key())
key := imagerequest.Source{Host: testHostCDN, Path: testPathCat}.Key()
path := storage.keyToPath(key)
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
@@ -167,14 +285,14 @@ func TestFileStore_WriteReplacesFileOfAnotherSize(t *testing.T) {
t.Fatalf("writing a partial file: %v", err)
}
err = storage.Write(key, content)
_, err = storage.Store(key, content)
if err != nil {
t.Fatalf("Write() error = %v", err)
t.Fatalf("Store() error = %v", err)
}
stored, _, err := storage.Open(key)
stored, _, err := storage.LoadWithSize(key)
if err != nil {
t.Fatalf("Open() error = %v", err)
t.Fatalf("LoadWithSize() error = %v", err)
}
defer func() { _ = stored.Close() }()
@@ -5,8 +5,6 @@ import (
"context"
"database/sql"
"errors"
"image/color"
"io"
"log/slog"
"strings"
"testing"
@@ -17,7 +15,8 @@ import (
// pendingAccessTimesWait is how long a test waits for the database's
// connection, for an access time to reach the database, for a write to start
// waiting for the connection, or for StopPendingAccessTimeWrites to return.
// waiting for the connection, for a request to return past its deadline, or
// for StopPendingAccessTimeWrites to return.
const pendingAccessTimesWait = 5 * time.Second
// otherWritesWait is how long a test gives other writes, once one write waits
@@ -57,6 +56,52 @@ func waitForConnectionWaits(t *testing.T, db *sql.DB, waits int64) {
}
}
// TestService_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy holds the
// database's one connection while a request whose fetch is held reaches its
// deadline. The request must return with the deadline's error while the
// connection is still held, as it waits for no database write past its
// deadline.
func TestService_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
// Room for the request to reach its fetch on a busy host
const timeout = 2 * time.Second
ctx, cancel := context.WithTimeout(t.Context(), timeout)
defer cancel()
deadline, _ := ctx.Deadline()
results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
// The request has made its database reads by the time it fetches.
select {
case <-fetcher.started:
case <-time.After(timeout):
t.Fatal("request did not reach its fetch by its deadline")
}
releaseDatabase := holdDatabase(t, svc.cache)
defer releaseDatabase()
select {
case got := <-results:
t.Logf("Get() returned %v after its deadline, error = %v",
time.Since(deadline), got.err)
if !errors.Is(got.err, context.DeadlineExceeded) {
t.Errorf("Get() error = %v, want %v", got.err, context.DeadlineExceeded)
}
case <-time.After(time.Until(deadline) + pendingAccessTimesWait):
t.Fatal("request did not return while the database was busy")
}
}
// TestStopPendingAccessTimeWrites_LetsTheWriteUnderWayFinish holds the
// database's one connection while the goroutine that writes the pending
// access times waits for it, hands over another access time, then stops that
@@ -197,16 +242,16 @@ func TestStopPendingAccessTimeWrites_ReturnsWhenItsContextEnds(t *testing.T) {
}
}
// TestSource_HandsAccessTimeToTheBackgroundWriter finds a cached source
// whose access time is an hour old. Source must not write the access
// TestLookupSource_HandsAccessTimeToTheBackgroundWriter finds a cached source
// whose access time is an hour old. LookupSource must not write the access
// time itself but hand it over to be written, and StopPendingAccessTimeWrites
// must then write it.
func TestSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) {
func TestLookupSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
content := generateTestJPEG(t, 10, 10, color.RGBA{0, 0, 255, 255})
key := storeEvictionTestSource(t, cache, "example.com", "/photo.jpg", content)
key := storeEvictionTestSource(t, cache, "example.com", "/photo.jpg",
[]byte("source bytes"))
anHourAgo := time.Now().Add(-time.Hour)
setSourceLastAccessed(t, cache, key, anHourAgo)
@@ -216,21 +261,15 @@ func TestSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) {
`
anHourAgoInDatabase := anHourAgo.UTC().Format(sqliteTimestampFormat)
reader, _, _, err := cache.Source(t.Context(),
imagerequest.Source{Host: "example.com", Path: "/photo.jpg"})
if err != nil {
t.Fatalf("Source() error = %v; want the source of %q", err, key)
}
found, err := io.ReadAll(reader)
_ = reader.Close()
if err != nil || !bytes.Equal(found, content) {
t.Fatalf("Source() reads %q, %v; want the source of %q", found, err, key)
found, err := cache.LookupSource(t.Context(), &imagerequest.Request{
Source: imagerequest.Source{Host: "example.com", Path: "/photo.jpg"},
})
if err != nil || found != key {
t.Fatalf("LookupSource() = %q, %v; want %q", found, err, key)
}
if countRows(t, cache, touchedQuery, string(key), anHourAgoInDatabase) != 0 {
t.Error("Source wrote the source's access time itself")
t.Error("LookupSource wrote the source's access time itself")
}
if waiting := len(cache.pendingSourceAccessTimes); waiting != 1 {
+528
View File
@@ -0,0 +1,528 @@
package imgcache
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"log/slog"
"runtime/debug"
"time"
"github.com/dustin/go-humanize"
"github.com/getsentry/sentry-go"
"github.com/go-chi/chi/v5/middleware"
"golang.org/x/sync/singleflight"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic"
)
// Service orchestrates the cache, fetcher, and processor.
type Service struct {
cache *Cache
fetcher httpfetcher.Fetcher
processor *imageprocessor.ImageProcessor
log *slog.Logger
maxResponseSize int64
// variantsInProgress lets the requests that miss the same variant at the
// same time share one fetch and one transcode.
variantsInProgress singleflight.Group
}
// ServiceConfig holds configuration for the image service.
type ServiceConfig struct {
// Cache is the cache instance
Cache *Cache
// Fetcher fetches source images from upstream hosts
Fetcher httpfetcher.Fetcher
// Processor converts and resizes the images
Processor *imageprocessor.ImageProcessor
// MaxResponseSize is the largest source accepted, in bytes; a cached
// source larger than this is fetched again
MaxResponseSize int64
// Logger for logging
Logger *slog.Logger
}
// errCacheRequired is returned when the service is built without a cache.
var errCacheRequired = errors.New("cache is required")
// NewService creates a new image service.
func NewService(cfg *ServiceConfig) (*Service, error) {
if cfg.Cache == nil {
return nil, errCacheRequired
}
log := cfg.Logger
if log == nil {
log = slog.Default()
}
// JPEG XL is the default output format, so pixad does not start
// without it.
err := imageprocessor.CheckJPEGXLSupport()
if err != nil {
return nil, err
}
return &Service{
cache: cfg.Cache,
fetcher: cfg.Fetcher,
processor: cfg.Processor,
log: log,
maxResponseSize: cfg.MaxResponseSize,
}, nil
}
// ErrNegativeCached is returned when a URL is in the negative cache (recently failed).
var ErrNegativeCached = errors.New("request is in negative cache (recently failed)")
// ErrProcessingTimedOut is returned when processing an image stopped because
// the downstream_timeout of the request doing it ended, to that request and to
// every request that waited for its image.
var ErrProcessingTimedOut = errors.New(
"image processing stopped at the deadline of the request doing it")
// Get retrieves a processed image, fetching and processing if necessary.
// While its source URL is in the negative cache, a cached variant is still
// served and a variant not in the cache is still made from a cached source;
// only a request that needs the source fetched is refused.
//
// A cache hit opens the variant file and makes no database call: its access
// time is handed to the goroutine that StartPendingAccessTimeWrites starts,
// without waiting for it, and its hit is counted in memory. A variant file
// that cannot be served, such as one whose first bytes are not those of an
// output format, is a miss, and the variant is made again.
func (s *Service) Get(
ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) {
cacheKey := imagerequest.CacheKey(req)
reader, size, contentType, err := s.cache.GetVariant(cacheKey)
if err == nil {
s.cache.touchVariant(cacheKey)
s.cache.IncrementStats(true, 0)
return &ImageResponse{
Content: reader,
ContentLength: size,
ContentType: contentType,
CacheStatus: CacheHit,
ETag: formatETag(cacheKey),
}, nil
}
if !errors.Is(err, ErrNotFound) {
s.log.Warn("cached variant not served, making it again",
"cache_key", cacheKey, "error", err)
}
// Cache miss - get the variant, processed once for all the requests that
// miss it at the same time, then count this request's miss, also when it
// failed or the request context has ended meanwhile
response, err := s.processOrWait(ctx, req)
s.cache.IncrementStats(false, 0)
if err != nil {
return nil, err
}
response.CacheStatus = CacheMiss
return response, nil
}
// WaitForProcessing waits until no image is being processed, or until ctx
// ends, and returns how many images were still being processed then.
func (s *Service) WaitForProcessing(ctx context.Context) int {
return s.processor.WaitForProcessing(ctx)
}
// errPanicked is returned when processing a variant panicked.
var errPanicked = errors.New("panic while processing image")
// processOrWait returns the variant req asks for. The first of the requests
// that miss a variant at the same time processes it, and singleflight hands
// its result, or its error, to the others: they fetch nothing, read no source
// and take no upstream connection or processing slot. The processing ignores
// the first request's cancellation, so the others are still served if that
// client goes away, but keeps its deadline.
func (s *Service) processOrWait(
ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) {
// A request that has already ended starts no processing
if ctx.Err() != nil {
return nil, ctx.Err()
}
cacheKey := imagerequest.CacheKey(req)
// Closed when this request's own function runs, which singleflight does
// only when no other request is processing the variant
processing := make(chan struct{})
results := s.variantsInProgress.DoChan(string(cacheKey),
func() (_ any, err error) {
close(processing)
// singleflight would raise a panic again in a goroutine of its
// own, where no handler recovers it, and stop pixad. It is
// reported through the Sentry hub that the Sentry middleware puts
// on the request's context when sentry_dsn is set.
defer func() {
recovered := recover()
if recovered != nil {
s.log.Error("panic while processing image",
"host", req.Source.Host, "path", req.Source.Path,
"panic", recovered, "stack", string(debug.Stack()))
if hub := sentry.GetHubFromContext(ctx); hub != nil {
hub.RecoverWithContext(ctx, recovered)
}
err = fmt.Errorf("%w: %v", errPanicked, recovered)
}
}()
processingCtx, cancel := withoutCancelKeepingDeadline(ctx)
defer cancel()
return s.processFromSourceOrFetch(processingCtx, req, cacheKey)
})
var result singleflight.Result
select {
case result = <-results:
case <-ctx.Done():
select {
case <-processing:
// This request is processing the variant: it waits for the
// result, as every request did before misses were shared
result = <-results
default:
// Another request is processing the variant, or this request's
// function has not started yet; the processing goes on without it
return nil, ctx.Err()
}
}
if result.Err != nil {
return nil, result.Err
}
variant, _ := result.Val.(*processedVariant)
return &ImageResponse{
Content: io.NopCloser(bytes.NewReader(variant.data)),
ContentLength: int64(len(variant.data)),
ContentType: variant.contentType,
FetchedBytes: variant.fetchedBytes,
ETag: formatETag(cacheKey),
}, nil
}
// withoutCancelKeepingDeadline returns ctx without its cancellation but with
// its deadline, if it has one, and the func that releases the returned
// context.
func withoutCancelKeepingDeadline(
ctx context.Context,
) (context.Context, context.CancelFunc) {
deadline, hasDeadline := ctx.Deadline()
if !hasDeadline {
return context.WithoutCancel(ctx), func() {}
}
return context.WithDeadline(context.WithoutCancel(ctx), deadline)
}
// loadCachedSource opens source content from cache, without reading it, and
// returns it with its size; nil if the cached data is unavailable, empty or
// exceeds maxResponseSize.
func (s *Service) loadCachedSource(
sourceKey imagerequest.SourceKey,
) (io.ReadCloser, int64) {
reader, size, err := s.cache.GetSourceContent(sourceKey)
if err != nil {
s.log.Warn("failed to load cached source, fetching", "error", err)
return nil, 0
}
if size > s.maxResponseSize {
_ = reader.Close()
s.log.Warn("cached source exceeds max response size, discarding",
"source_key", sourceKey,
"max_bytes", s.maxResponseSize,
)
return nil, 0
}
if size == 0 {
_ = reader.Close()
return nil, 0
}
return reader, size
}
// processFromSourceOrFetch processes an image, using cached source content
// if available.
func (s *Service) processFromSourceOrFetch(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
) (*processedVariant, error) {
// Check if we have cached source content
sourceKey, err := s.cache.LookupSource(ctx, req)
if err != nil {
s.log.Warn("source lookup failed", "error", err)
}
var (
source io.ReadCloser
sourceSize int64
)
if sourceKey != "" {
s.log.Debug("using cached source", "source_key", sourceKey)
source, sourceSize = s.loadCachedSource(sourceKey)
}
// Fetch from upstream if we don't have source data or it's empty
if source == nil {
return s.fetchAndProcess(ctx, req, cacheKey)
}
defer func() { _ = source.Close() }()
// Process using cached source; nothing was fetched from upstream. The
// image processor reads the source only once it has a processing slot,
// so a request waiting for one holds none of it in memory.
return s.processAndStore(ctx, req, cacheKey, source, sourceSize)
}
// fetchAndProcess fetches from upstream, processes, and caches the result.
// It counts the fetch with the bytes read from upstream, including when
// reading the response or a later step fails. It fetches nothing while the
// source URL is in the negative cache.
func (s *Service) fetchAndProcess(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
) (*processedVariant, error) {
// Check negative cache - skip fetching for recently-failed URLs
negHit, err := s.cache.checkNegativeCache(ctx, req)
if err != nil {
s.log.Warn("negative cache check failed", "error", err)
}
if negHit {
s.log.Debug("negative cache hit",
"host", req.Source.Host,
"path", req.Source.Path,
)
return nil, fmt.Errorf("%w: %w", httpfetcher.ErrUpstreamError, ErrNegativeCached)
}
// Fetch from upstream
s.log.Debug("fetching from upstream", "host", req.Source.Host,
"path", req.Source.Path, "query", req.Source.Query)
fetchResult, err := s.fetcher.Fetch(ctx, req.Source)
if err != nil {
// Store negative cache for certain errors
if isNegativeCacheable(err) {
s.storeNegative(ctx, req)
}
return nil, fmt.Errorf("upstream fetch failed: %w", err)
}
// Closing the body frees the upstream connection. It is closed only
// after processing, so the fetcher's connection limit also bounds the
// fetched sources held in memory while their requests wait for a
// processing slot.
defer func() { _ = fetchResult.Content.Close() }()
// Read and validate the source content
sourceData, err := io.ReadAll(fetchResult.Content)
fetchBytes := int64(len(sourceData))
// Counted also when the request context has ended meanwhile
s.cache.IncrementUpstreamFetch(fetchBytes)
if err != nil {
return nil, fmt.Errorf("failed to read upstream response: %w", err)
}
// Calculate download bitrate
var downloadRate string
if fetchResult.FetchDurationMs > 0 {
seconds := float64(fetchResult.FetchDurationMs) / 1000.0 //nolint:mnd // ms to seconds
bitsPerSecond := float64(fetchBytes*8) / seconds //nolint:mnd // bytes to bits
downloadRate = humanize.SI(bitsPerSecond, "bps")
}
// Log upstream fetch details
s.log.Info("upstream fetched",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host,
"path", req.Source.Path,
"bytes", fetchBytes,
"fetch_ms", fetchResult.FetchDurationMs,
"rate", downloadRate,
"remote_addr", fetchResult.RemoteAddr,
"http", fetchResult.HTTPVersion,
"tls", fetchResult.TLSVersion,
"cipher", fetchResult.TLSCipherSuite,
)
// Validate magic bytes match content type. A body that does not is the
// upstream's error, so it gets a negative cache entry, as an upstream
// error status does.
err = magic.ValidateMagicBytes(sourceData, fetchResult.ContentType)
if err != nil {
s.log.Warn("source refused", "request_id", middleware.GetReqID(ctx),
"host", req.Source.Host, "path", req.Source.Path,
"content_type", fetchResult.ContentType, "error", err)
s.storeNegative(ctx, req)
return nil, fmt.Errorf("content validation failed: %w", err)
}
// Store source content
_, err = s.cache.StoreSource(ctx, req, bytes.NewReader(sourceData))
if err != nil {
s.log.Warn("failed to store source content", "error", err)
// Continue even if caching fails
}
return s.processAndStore(
ctx, req, cacheKey, bytes.NewReader(sourceData), fetchBytes,
)
}
// storeNegative stores a negative cache entry for req's source, whose fetch
// failed, so the source is not fetched again until the entry expires. A
// failure to store the entry is only logged.
func (s *Service) storeNegative(ctx context.Context, req *imagerequest.Request) {
err := s.cache.StoreNegative(ctx, req)
if err != nil {
s.log.Warn("failed to store negative cache entry",
"host", req.Source.Host, "path", req.Source.Path, "error", err)
}
}
// processedVariant is a variant as processAndStore made it. Each request that
// shared its processing serves it through a reader of its own.
type processedVariant struct {
data []byte
contentType string
fetchedBytes int64
}
// processAndStore processes the image read from source and stores the
// result.
func (s *Service) processAndStore(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
source io.Reader,
fetchBytes int64,
) (*processedVariant, error) {
// Process the image
processStart := time.Now()
// ctx has the deadline of the request doing the processing, which can
// end while a request waiting for its image has time left
processResult, err := s.processor.Process(ctx, source, &req.Transform)
if errors.Is(err, context.DeadlineExceeded) {
return nil, fmt.Errorf("%w: %w", ErrProcessingTimedOut, err)
}
if err != nil {
return nil, fmt.Errorf("image processing failed: %w", err)
}
processDuration := time.Since(processStart)
// Counted also when the request context has ended meanwhile
s.cache.IncrementTransformCount()
// Read processed content
processedData, err := io.ReadAll(processResult.Content)
_ = processResult.Content.Close()
if err != nil {
return nil, fmt.Errorf("failed to read processed content: %w", err)
}
// Log conversion details
outputSize := int64(len(processedData))
var sizePercent float64
if fetchBytes > 0 {
//nolint:mnd // percentage calculation
sizePercent = float64(outputSize) / float64(fetchBytes) * 100.0
}
s.log.Info("image converted",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host,
"path", req.Source.Path,
"src_format", processResult.InputFormat,
"dst_format", req.Transform.Format,
"src_bytes", fetchBytes,
"dst_bytes", outputSize,
"src_dimensions", fmt.Sprintf("%dx%d",
processResult.InputWidth, processResult.InputHeight),
"dst_dimensions", fmt.Sprintf("%dx%d",
processResult.Width, processResult.Height),
"size_ratio", fmt.Sprintf("%.1f%%", sizePercent),
"convert_ms", processDuration.Milliseconds(),
"quality", req.Transform.Quality,
"fit", req.Transform.FitMode,
)
// Store variant to cache
err = s.cache.StoreVariant(ctx, cacheKey, bytes.NewReader(processedData))
if err != nil {
s.log.Warn("failed to store variant", "error", err)
// Continue even if caching fails
}
return &processedVariant{
data: processedData,
contentType: processResult.ContentType,
fetchedBytes: fetchBytes,
}, nil
}
// isNegativeCacheable returns true if the error should be cached.
func isNegativeCacheable(err error) bool {
return errors.Is(err, httpfetcher.ErrUpstreamError)
}
// etagHashLength is the number of hash characters to use for ETags.
const etagHashLength = 16
// formatETag formats a VariantKey as a quoted ETag value.
func formatETag(key imagerequest.VariantKey) string {
hash := string(key)
// Use first 16 characters of hash for a shorter but still unique ETag
if len(hash) > etagHashLength {
hash = hash[:etagHashLength]
}
return `"` + hash + `"`
}
Loaded 100 of 125 files, more files were not shown because too many files have changed in this diff. Show more