Author SHA1 Message Date
clawbot 62f46c3a49 Send CORS headers only from the image routes (closes #98)
check / check (push) Waiting to run
The CORS middleware, with the access_control_allow_origin origin, moved
from the router root onto a /v1 subrouter holding /v1/image/ and /v1/e/.
The login and URL generator pages, /metrics, the health check,
robots.txt and /static/ no longer send Access-Control-Allow-Origin, so
their safety no longer rests on the CORS options chosen for the image
routes.

It is a subrouter rather than a route group because a group's
middleware runs only for a request that matches one of its routes, and
a preflight OPTIONS request matches none. The maintenance mode group
moved inside it unchanged.

README.md and config.example.yml say the setting covers the image
routes only.

Model: opus-5-5
2026-09-29 11:00:22 +00:00
clawbot 6748bbd637 Test that only the image routes send CORS headers (closes #98)
check / check (push) Failing after 1m56s
A new server test sends requests with an Origin header through the
server's routes and expects Access-Control-Allow-Origin, set to the
configured origin, on both image routes, a preflight OPTIONS request
included, and no such header on the login and URL generator pages.
It fails for now: the CORS middleware still wraps every route.

The encrypted image path the maintenance tests use is now a named
constant, shared with the new test; what they check is unchanged.

Model: opus-5-5
2026-09-29 10:54:50 +00:00
clawbot fd7d7ed205 Container makes /var/lib/pixa usable before starting pixad (closes #159)
check / check (push) Successful in 22s
The entrypoint now creates /var/lib/pixa if it is missing. When the
directory or one of its top-level entries belongs to another user or
group, it gives the whole tree to pixad (uid and gid 65532); it then
sets the directory's mode to 750 and runs the server as pixad as
before. Data left by a run under another uid is taken over this way.
Only the top level is checked, so a normal start does not walk the
cache; the tree is changed deepest first, so an interrupted start is
finished by the next one.

"Running under upaas" in README.md no longer tells the operator to
create or chown the host directory.

Model: opus-5-5
2026-09-29 12:44:37 +02:00
clawbot bce8860c2e Keep variant content types in memory for cache hits (closes #70)
check / check (push) Successful in 14s
Cache.metaCache was declared and never used, so every cache hit read
and parsed the variant's .meta file. It is now an LRU
(github.com/hashicorp/golang-lru/v2) of up to 10,000 variants' content
types, filled by StoreVariant and by a read of a .meta file, so a hit
for a variant it holds skips the .meta read. Only a type from a .meta
file or a store ever enters memory, never the application/octet-stream
fallback, and a stored type is never replaced by an older one from
disk. The variant file itself is still opened on every hit, so nothing
is served from memory alone.

Model: opus-5-5
2026-09-29 12:00:10 +02:00
13 changed files with 566 additions and 50 deletions
+8 -4
View File
@@ -81,7 +81,10 @@ prevent abuse, and allowlisted source hosts for open access.
Multiple source paths may reference the same content blob; the Multiple source paths may reference the same content blob; the
database tracks references rather than using filesystem refcounting. database tracks references rather than using filesystem refcounting.
In-process caching of request-to-output mappings targets 1-5k r/s. Toward a target of 1-5k r/s, pixa keeps in memory the content types of
the 10,000 transformed images most recently cached or served, so a
cache hit on one of them reads only the image file from disk and not
the metadata file stored beside it.
### Routes ### Routes
@@ -235,7 +238,7 @@ variables set by the file's `env:` section are checked the same way.
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` | | `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` |
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB | | `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB |
| `PIXA_DOWNSTREAM_TIMEOUT` | `downstream_timeout` | Time allowed for answering one client request; default `60s` | | `PIXA_DOWNSTREAM_TIMEOUT` | `downstream_timeout` | Time allowed for answering one client request; default `60s` |
| `PIXA_ACCESS_CONTROL_ALLOW_ORIGIN` | `access_control_allow_origin` | CORS origin allowed to read responses: `*` or one origin; default `*` | | `PIXA_ACCESS_CONTROL_ALLOW_ORIGIN` | `access_control_allow_origin` | CORS origin allowed to read image responses: `*` or one origin; default `*` |
| `PIXA_METRICS_USERNAME` | `metrics.username` | Username for `/metrics`, which is served only when both are set | | `PIXA_METRICS_USERNAME` | `metrics.username` | Username for `/metrics`, which is served only when both are set |
| `PIXA_METRICS_PASSWORD` | `metrics.password` | Password for `/metrics`; set together with the username | | `PIXA_METRICS_PASSWORD` | `metrics.password` | Password for `/metrics`; set together with the username |
| `PIXA_SENTRY_DSN` | `sentry_dsn` | Sentry DSN for error reporting; empty disables it | | `PIXA_SENTRY_DSN` | `sentry_dsn` | Sentry DSN for error reporting; empty disables it |
@@ -244,8 +247,9 @@ variables set by the file's `env:` section are checked the same way.
Key settings in more detail: Key settings in more detail:
- `access_control_allow_origin` — the origin a browser lets read pixa's - `access_control_allow_origin` — the origin a browser lets read the responses
responses, sent as the CORS `Access-Control-Allow-Origin` header: `*`, the of the image routes, `/v1/image/` and `/v1/e/`, sent as the CORS
`Access-Control-Allow-Origin` header; no other route sends it. `*`, the
default, is any site; otherwise one `http` or `https` origin such as default, is any site; otherwise one `http` or `https` origin such as
`https://example.com`, whose host is a lowercase host name (letters, `https://example.com`, whose host is a lowercase host name (letters,
digits, hyphens and dots, with a letter in its last part) or an IP address digits, hyphens and dots, with a letter in its last part) or an IP address
+17
View File
@@ -29,6 +29,12 @@ P2: security: referer blacklist
# Completed Steps # Completed Steps
- 2026-09-29 only the image routes send CORS headers (closes #98): the CORS
middleware, with the `access_control_allow_origin` origin, moved from the
router root onto a `/v1` subrouter holding `/v1/image/` and `/v1/e/`, where it
still answers a preflight `OPTIONS` request; the login and URL generator
pages, `/metrics` and the other routes send no `Access-Control-Allow-Origin`;
documented in `README.md` and `config.example.yml`.
- 2026-09-29 the container makes `/var/lib/pixa` usable by itself (closes - 2026-09-29 the container makes `/var/lib/pixa` usable by itself (closes
#159): `deploy/docker-entrypoint.sh` creates the directory if it is missing, #159): `deploy/docker-entrypoint.sh` creates the directory if it is missing,
gives the directory and everything in it to `pixad` when the directory or one gives the directory and everything in it to `pixad` when the directory or one
@@ -36,6 +42,17 @@ P2: security: referer blacklist
`750`, then runs the server as `pixad`; data left by an earlier run under `750`, then runs the server as `pixad`; data left by an earlier run under
another uid is taken over this way; "Running under upaas" in `README.md` no another uid is taken over this way; "Running under upaas" in `README.md` no
longer tells the operator to create or chown the host directory. longer tells the operator to create or chown the host directory.
- 2026-09-29 variant content types kept in memory (closes #70):
`Cache.metaCache` holds the content types of up to 10,000 variants in an LRU
(`github.com/hashicorp/golang-lru/v2`), filled by `StoreVariant` and by
`GetVariant` after it reads a `.meta` file, where a type `StoreVariant` added
meanwhile is kept over the one read, and never with the
`application/octet-stream` served for a variant without one; for a variant it
holds, `GetVariant` skips the `.meta` read, still opening the variant file and
taking the size from it; eviction removes the entry before deleting the files,
and `GetVariant` removes it when the file will not open; the cap is a
constant, not a setting; the unused `variantMeta` type is gone; `README.md`
describes it.
- 2026-09-29 maintenance mode refuses image requests (closes #71): while - 2026-09-29 maintenance mode refuses image requests (closes #71): while
`maintenance_mode` is on, `/v1/image/` and `/v1/e/` answer 503 with a `maintenance_mode` is on, `/v1/image/` and `/v1/e/` answer 503 with a
`Retry-After` header and the JSON error body, from one middleware in `Retry-After` header and the JSON error body, from one middleware in
+3 -2
View File
@@ -103,8 +103,9 @@ upstream_max_response_size: 52428800
# longer than upstream_fetch_timeout plus 20 seconds. # longer than upstream_fetch_timeout plus 20 seconds.
downstream_timeout: 60s downstream_timeout: 60s
# The origin a browser lets read pixa's responses, sent as the CORS # The origin a browser lets read the responses of the image routes,
# Access-Control-Allow-Origin header: "*" (the default) is any site; # /v1/image/ and /v1/e/, sent as the CORS Access-Control-Allow-Origin
# header; no other route sends it. "*" (the default) is any site;
# otherwise one http or https origin such as https://example.com, whose # otherwise one http or https origin such as https://example.com, whose
# host is a lowercase host name (letters, digits, hyphens and dots, with a # host is a lowercase host name (letters, digits, hyphens and dots, with a
# letter in its last part) or an IP address (IPv6 in brackets, in its # letter in its last part) or an IP address (IPv6 in brackets, in its
+1
View File
@@ -14,6 +14,7 @@ require (
github.com/go-chi/httprate v0.16.0 github.com/go-chi/httprate v0.16.0
github.com/gorilla/csrf v1.7.3 github.com/gorilla/csrf v1.7.3
github.com/gorilla/securecookie v1.1.2 github.com/gorilla/securecookie v1.1.2
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/prometheus/client_golang v1.23.2 github.com/prometheus/client_golang v1.23.2
github.com/slok/go-http-metrics v0.13.0 github.com/slok/go-http-metrics v0.13.0
github.com/spf13/cobra v1.10.2 github.com/spf13/cobra v1.10.2
+2
View File
@@ -228,6 +228,8 @@ github.com/hashicorp/go-version v1.2.1/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc=
github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/hashicorp/hcl v1.0.1-vault-7 h1:ag5OxFVy3QYTFTJODRzTKVZ6xvdfLLCA1cy/Y6xGI0I= github.com/hashicorp/hcl v1.0.1-vault-7 h1:ag5OxFVy3QYTFTJODRzTKVZ6xvdfLLCA1cy/Y6xGI0I=
github.com/hashicorp/hcl v1.0.1-vault-7/go.mod h1:XYhtn6ijBSAj6n4YqAaf7RBPS4I06AItNorpy+MoQNM= github.com/hashicorp/hcl v1.0.1-vault-7/go.mod h1:XYhtn6ijBSAj6n4YqAaf7RBPS4I06AItNorpy+MoQNM=
github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64= github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64=
+61 -12
View File
@@ -14,6 +14,7 @@ import (
"sync" "sync"
"time" "time"
lru "github.com/hashicorp/golang-lru/v2"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
) )
@@ -26,6 +27,10 @@ var (
// HTTP status code for successful fetch. // HTTP status code for successful fetch.
const httpStatusOK = 200 const httpStatusOK = 200
// metaCacheSize is how many variants' content types metaCache holds. A
// variant not among them is served as before, reading its .meta file.
const metaCacheSize = 10000
// CacheConfig holds cache configuration. // CacheConfig holds cache configuration.
type CacheConfig struct { type CacheConfig struct {
StateDir string StateDir string
@@ -49,12 +54,6 @@ type CacheConfig struct {
Logger *slog.Logger Logger *slog.Logger
} }
// variantMeta stores content type for fast cache hits without reading .meta file.
type variantMeta struct {
ContentType string
Size int64
}
// Cache implements the caching layer for the image proxy. // Cache implements the caching layer for the image proxy.
type Cache struct { type Cache struct {
db *sql.DB db *sql.DB
@@ -76,9 +75,10 @@ type Cache struct {
evictionStarted bool evictionStarted bool
evictionStopOnce sync.Once evictionStopOnce sync.Once
// In-memory cache of variant metadata (content type, size) to avoid // metaCache holds the content types of the variants most recently
// reading .meta files // stored or served, so a hit does not read the variant's .meta file.
metaCache map[VariantKey]variantMeta // It never stands in for the variant file, which is always opened.
metaCache *lru.Cache[VariantKey, string]
// contentLocks serializes StoreSource and evictSourceBlob per // contentLocks serializes StoreSource and evictSourceBlob per
// content hash, closing the race window between an eviction's row // content hash, closing the race window between an eviction's row
@@ -101,6 +101,11 @@ func NewCache(db *sql.DB, config CacheConfig) (*Cache, error) {
log = slog.Default() log = slog.Default()
} }
metaCache, err := lru.New[VariantKey, string](metaCacheSize)
if err != nil {
return nil, fmt.Errorf("failed to create variant content type cache: %w", err)
}
c := &Cache{ c := &Cache{
db: db, db: db,
config: config, config: config,
@@ -109,7 +114,7 @@ func NewCache(db *sql.DB, config CacheConfig) (*Cache, error) {
evictionPressure: make(chan struct{}, 1), evictionPressure: make(chan struct{}, 1),
evictionStop: make(chan struct{}), evictionStop: make(chan struct{}),
evictionDone: make(chan struct{}), evictionDone: make(chan struct{}),
metaCache: make(map[VariantKey]variantMeta), metaCache: metaCache,
contentLocks: newContentLock(), contentLocks: newContentLock(),
} }
@@ -177,13 +182,30 @@ func (c *Cache) Lookup(ctx context.Context, req *ImageRequest) (*LookupResult, e
}, nil }, nil
} }
// GetVariant returns a reader, size, and content type for a cached variant. // GetVariant returns a reader, size, and content type for a cached
// variant. The content type comes from metaCache, or else from the
// variant's .meta file and is then kept in metaCache. A variant with
// no .meta file is served as application/octet-stream, which is not
// kept.
func (c *Cache) GetVariant(cacheKey VariantKey) (io.ReadCloser, int64, string, error) { func (c *Cache) GetVariant(cacheKey VariantKey) (io.ReadCloser, int64, string, error) {
if c.disabled { if c.disabled {
return nil, 0, "", ErrNotFound return nil, 0, "", ErrNotFound
} }
return c.variants.LoadWithMeta(cacheKey) contentType, known := c.metaCache.Get(cacheKey)
if !known {
return c.loadVariantWithMeta(cacheKey)
}
reader, size, err := c.variants.LoadWithSize(cacheKey)
if err != nil {
// The file is gone, e.g. deleted outside pixa
c.metaCache.Remove(cacheKey)
return nil, 0, "", err
}
return reader, size, contentType, nil
} }
// StoreSource stores fetched source content and metadata. On a // StoreSource stores fetched source content and metadata. On a
@@ -286,6 +308,8 @@ func (c *Cache) StoreVariant(
return err return err
} }
c.metaCache.Add(cacheKey, contentType)
_, err = c.db.ExecContext(ctx, ` _, err = c.db.ExecContext(ctx, `
INSERT INTO variant_content (cache_key, size_bytes, content_type) INSERT INTO variant_content (cache_key, size_bytes, content_type)
VALUES (?, ?, ?) VALUES (?, ?, ?)
@@ -499,6 +523,31 @@ func (c *Cache) IncrementTransformCount(ctx context.Context) {
} }
} }
// loadVariantWithMeta is GetVariant for a variant metaCache does not
// hold: it reads the content type from the variant's .meta file and
// keeps it in metaCache, unless a StoreVariant has put one there
// meanwhile, as the store's is newer. A read that finds no .meta file,
// as one can between a store's writing of the variant file and of its
// .meta file, serves application/octet-stream and keeps nothing, so
// metaCache only ever holds a type read from a .meta file or passed to
// StoreVariant.
func (c *Cache) loadVariantWithMeta(
cacheKey VariantKey,
) (io.ReadCloser, int64, string, error) {
reader, size, contentType, err := c.variants.LoadWithMeta(cacheKey)
if err != nil {
return nil, 0, "", err
}
if contentType == "" {
return reader, size, fallbackContentType, nil
}
c.metaCache.ContainsOrAdd(cacheKey, contentType)
return reader, size, contentType, nil
}
// writeMetadataSidecar writes the JSON metadata sidecar of a stored source. // writeMetadataSidecar writes the JSON metadata sidecar of a stored source.
// A failure is logged and is otherwise non-fatal; the metadata is in the // A failure is logged and is otherwise non-fatal; the metadata is in the
// database. // database.
+7 -3
View File
@@ -39,8 +39,8 @@ const tempFilePrefix = ".tmp-"
// to each variant file. // to each variant file.
const variantMetaSuffix = ".meta" const variantMetaSuffix = ".meta"
// fallbackContentType is recorded when a reconciled variant file has // fallbackContentType is the content type given to a variant file that
// no readable .meta sidecar. // has no readable .meta sidecar, when it is served or reconciled.
const fallbackContentType = "application/octet-stream" const fallbackContentType = "application/octet-stream"
// UsageBytes returns the total number of bytes of cache content // UsageBytes returns the total number of bytes of cache content
@@ -271,7 +271,9 @@ func (c *Cache) sourceCandidates(ctx context.Context) ([]evictionCandidate, erro
// evictVariant removes one variant: accounting row first, then the // evictVariant removes one variant: accounting row first, then the
// content and .meta files, so the database never references a deleted // content and .meta files, so the database never references a deleted
// file. // file. The metaCache entry goes before the files; a GetVariant that
// read them just before may put it back, and the next GetVariant then
// fails to open the file and removes it again.
func (c *Cache) evictVariant(ctx context.Context, cacheKey VariantKey) error { func (c *Cache) evictVariant(ctx context.Context, cacheKey VariantKey) error {
_, err := c.db.ExecContext(ctx, _, err := c.db.ExecContext(ctx,
`DELETE FROM variant_content WHERE cache_key = ?`, string(cacheKey)) `DELETE FROM variant_content WHERE cache_key = ?`, string(cacheKey))
@@ -279,6 +281,8 @@ func (c *Cache) evictVariant(ctx context.Context, cacheKey VariantKey) error {
return fmt.Errorf("failed to delete variant accounting row: %w", err) return fmt.Errorf("failed to delete variant accounting row: %w", err)
} }
c.metaCache.Remove(cacheKey)
err = c.variants.DeleteWithMeta(cacheKey) err = c.variants.DeleteWithMeta(cacheKey)
if err != nil { if err != nil {
return err return err
@@ -0,0 +1,349 @@
package imgcache
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"sync"
"testing"
"time"
)
// webpRequest returns a request for a 100x100 WebP variant of path.
func webpRequest(path string) *ImageRequest {
return &ImageRequest{
SourceHost: testHostCDN,
SourcePath: path,
Size: Size{Width: 100, Height: 100},
Format: FormatWebP,
Quality: 85,
FitMode: FitCover,
}
}
// assertVariantServed checks that GetVariant serves key with the given
// content and the image/webp content type storeEvictionTestVariant stores.
func assertVariantServed(t *testing.T, cache *Cache, key VariantKey, content []byte) {
t.Helper()
reader, size, contentType, err := cache.GetVariant(key)
if err != nil {
t.Fatalf("GetVariant(%s) error = %v", key, err)
}
defer func() { _ = reader.Close() }()
got, err := io.ReadAll(reader)
if err != nil {
t.Fatalf("reading variant %s: %v", key, err)
}
if !bytes.Equal(got, content) {
t.Errorf("GetVariant(%s) content = %q, want %q", key, got, content)
}
if size != int64(len(content)) {
t.Errorf("GetVariant(%s) size = %d, want %d", key, size, len(content))
}
if contentType != testContentTypeWebP {
t.Errorf("GetVariant(%s) content type = %q, want %q",
key, contentType, testContentTypeWebP)
}
}
// assertVariantNotFound checks that GetVariant refuses key with
// ErrNotFound.
func assertVariantNotFound(t *testing.T, cache *Cache, key VariantKey) {
t.Helper()
reader, _, _, err := cache.GetVariant(key)
if err == nil {
_ = reader.Close()
}
if !errors.Is(err, ErrNotFound) {
t.Errorf("GetVariant(%s) error = %v, want ErrNotFound", key, err)
}
}
// assertLookupMisses checks that Lookup reports request as a miss.
func assertLookupMisses(t *testing.T, cache *Cache, request *ImageRequest) {
t.Helper()
lookup, err := cache.Lookup(t.Context(), request)
if err != nil {
t.Fatalf("Lookup(%s) error = %v", request.SourcePath, err)
}
if lookup.Hit {
t.Errorf("Lookup(%s) is a hit, want a miss", request.SourcePath)
}
}
// renameFile renames the file at from to to.
func renameFile(t *testing.T, from, to string) {
t.Helper()
err := os.Rename(from, to)
if err != nil {
t.Fatalf("renaming %s: %v", from, err)
}
}
// TestSecondHitDoesNotReadMetaFile checks that once a variant has been
// stored or read, a hit takes its content type from memory: with the
// .meta file deleted, GetVariant must still return the stored content
// type rather than the application/octet-stream it uses without one.
func TestSecondHitDoesNotReadMetaFile(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<20)
content := []byte("webp variant bytes")
storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
// A second Cache on the same state directory starts with nothing in
// memory, as pixad does after a restart, so its first read uses the
// .meta file.
restarted, err := NewCache(cache.db, cache.config)
if err != nil {
t.Fatalf("NewCache() error = %v", err)
}
assertVariantServed(t, restarted, testVariantKeyOne, content)
err = os.Remove(cache.variants.keyToPath(testVariantKeyOne) + ".meta")
if err != nil {
t.Fatalf("removing .meta file: %v", err)
}
assertVariantServed(t, cache, testVariantKeyOne, content)
assertVariantServed(t, restarted, testVariantKeyOne, content)
}
// TestReadDuringStoreKeepsStoredContentType checks that a GetVariant
// which began before StoreVariant finished cannot replace the content
// type the store kept in memory. Such a read can find the variant file
// but not yet its .meta file, and so gets application/octet-stream. The
// test deletes the .meta file after the store, then runs the part of
// GetVariant that comes after its check of memory.
func TestReadDuringStoreKeepsStoredContentType(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<20)
content := []byte("webp variant bytes")
storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
err := os.Remove(cache.variants.keyToPath(testVariantKeyOne) + ".meta")
if err != nil {
t.Fatalf("removing .meta file: %v", err)
}
reader, _, contentType, err := cache.loadVariantWithMeta(testVariantKeyOne)
if err != nil {
t.Fatalf("loadVariantWithMeta(%s) error = %v", testVariantKeyOne, err)
}
_ = reader.Close()
t.Logf("the read without a .meta file got content type %q", contentType)
assertVariantServed(t, cache, testVariantKeyOne, content)
}
// TestReadOfOlderMetaFileKeepsStoredContentType checks that a read
// which got its content type from a .meta file that StoreVariant had
// not yet rewritten cannot replace the type the store kept in memory.
// The test writes such a .meta file, with a different content type,
// after the store, then runs the part of GetVariant that comes after
// its check of memory.
func TestReadOfOlderMetaFileKeepsStoredContentType(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<20)
content := []byte("webp variant bytes")
storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
olderMeta, err := json.Marshal(VariantMeta{
ContentType: testContentTypeJPEG,
Size: int64(len(content)),
})
if err != nil {
t.Fatalf("encoding .meta file: %v", err)
}
metaPath := cache.variants.keyToPath(testVariantKeyOne) + ".meta"
err = os.WriteFile(metaPath, olderMeta, StorageFilePerm)
if err != nil {
t.Fatalf("writing .meta file: %v", err)
}
reader, _, contentType, err := cache.loadVariantWithMeta(testVariantKeyOne)
if err != nil {
t.Fatalf("loadVariantWithMeta(%s) error = %v", testVariantKeyOne, err)
}
_ = reader.Close()
if contentType != testContentTypeJPEG {
t.Fatalf("loadVariantWithMeta(%s) content type = %q, want %q from the .meta file",
testVariantKeyOne, contentType, testContentTypeJPEG)
}
kept, _ := cache.metaCache.Get(testVariantKeyOne)
if kept != testContentTypeWebP {
t.Errorf("content type in memory = %q, want the stored %q",
kept, testContentTypeWebP)
}
}
// TestFailedReadDuringStoreKeepsStoredContentType checks that a read
// which found no .meta file cannot leave application/octet-stream in
// memory, even when another read has removed the content type
// StoreVariant kept there. In this order: the store; a read that finds
// the variant in memory but cannot open its file, and so removes it
// from memory; a read that opened the variant file before the store
// wrote its .meta file. Later hits must get the stored content type.
func TestFailedReadDuringStoreKeepsStoredContentType(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<20)
content := []byte("webp variant bytes")
variantPath := cache.variants.keyToPath(testVariantKeyOne)
metaPath := variantPath + ".meta"
storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
renameFile(t, variantPath, variantPath+".hidden")
assertVariantNotFound(t, cache, testVariantKeyOne)
renameFile(t, variantPath+".hidden", variantPath)
renameFile(t, metaPath, metaPath+".hidden")
reader, _, contentType, err := cache.GetVariant(testVariantKeyOne)
if err != nil {
t.Fatalf("GetVariant(%s) error = %v", testVariantKeyOne, err)
}
_ = reader.Close()
t.Logf("the read without a .meta file got content type %q", contentType)
renameFile(t, metaPath+".hidden", metaPath)
assertVariantServed(t, cache, testVariantKeyOne, content)
}
// TestEvictedVariantIsNotServed checks that a variant the evictor
// removed is a miss and cannot be read, although it had been stored
// and served before.
func TestEvictedVariantIsNotServed(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1500)
oldRequest := webpRequest("/old.jpg")
newRequest := webpRequest("/new.jpg")
oldKey := CacheKey(oldRequest)
newKey := CacheKey(newRequest)
oldContent := bytes.Repeat([]byte{0x01}, 1000)
newContent := bytes.Repeat([]byte{0x02}, 1000)
storeEvictionTestVariant(t, cache, oldKey, oldContent)
storeEvictionTestVariant(t, cache, newKey, newContent)
assertVariantServed(t, cache, oldKey, oldContent)
setVariantLastAccessed(t, cache, oldKey, time.Now().Add(-time.Hour))
err := cache.EvictToLimit(t.Context())
if err != nil {
t.Fatalf("EvictToLimit() error = %v", err)
}
assertLookupMisses(t, cache, oldRequest)
assertVariantNotFound(t, cache, oldKey)
assertVariantServed(t, cache, newKey, newContent)
}
// TestVariantDeletedFromDiskIsNotServed checks that a variant whose
// file was deleted by something other than the evictor cannot be read,
// and is a miss afterwards.
func TestVariantDeletedFromDiskIsNotServed(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<20)
request := webpRequest("/deleted.jpg")
key := CacheKey(request)
content := []byte("webp variant bytes")
storeEvictionTestVariant(t, cache, key, content)
assertVariantServed(t, cache, key, content)
err := os.Remove(cache.variants.keyToPath(key))
if err != nil {
t.Fatalf("removing variant file: %v", err)
}
assertVariantNotFound(t, cache, key)
assertLookupMisses(t, cache, request)
}
// TestConcurrentVariantStoreReadAndEvict stores, reads and evicts
// variants from several goroutines at once, for the race detector.
func TestConcurrentVariantStoreReadAndEvict(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<20)
ctx := t.Context()
var wg sync.WaitGroup
for goroutine := range 8 {
wg.Go(func() {
key := VariantKey(fmt.Sprintf("aabbccdd01%02d", goroutine))
content := []byte(key)
for range 20 {
err := cache.StoreVariant(
ctx, key, bytes.NewReader(content), testContentTypeWebP)
if err != nil {
t.Errorf("StoreVariant(%s) error = %v", key, err)
return
}
reader, _, contentType, err := cache.GetVariant(key)
if err != nil {
t.Errorf("GetVariant(%s) error = %v", key, err)
return
}
_ = reader.Close()
if contentType != testContentTypeWebP {
t.Errorf("GetVariant(%s) content type = %q, want %q",
key, contentType, testContentTypeWebP)
}
err = cache.evictVariant(ctx, key)
if err != nil {
t.Errorf("evictVariant(%s) error = %v", key, err)
return
}
assertVariantNotFound(t, cache, key)
}
})
}
wg.Wait()
}
+24 -12
View File
@@ -506,32 +506,44 @@ func (s *VariantStorage) Load(key VariantKey) (io.ReadCloser, error) {
return f, nil return f, nil
} }
// LoadWithMeta returns a reader, size, and content type for the content at // LoadWithSize returns a reader and file size for the content at the
// the given key. // given key.
func (s *VariantStorage) LoadWithMeta( func (s *VariantStorage) LoadWithSize(key VariantKey) (io.ReadCloser, int64, error) {
key VariantKey,
) (io.ReadCloser, int64, string, error) {
path := s.keyToPath(key) path := s.keyToPath(key)
metaPath := path + ".meta"
f, err := os.Open(path) //nolint:gosec // path derived from cache key f, err := os.Open(path) //nolint:gosec // path derived from cache key
if err != nil { if err != nil {
if os.IsNotExist(err) { if os.IsNotExist(err) {
return nil, 0, "", ErrNotFound return nil, 0, ErrNotFound
} }
return nil, 0, "", fmt.Errorf("failed to open content: %w", err) return nil, 0, fmt.Errorf("failed to open content: %w", err)
} }
stat, err := f.Stat() stat, err := f.Stat()
if err != nil { if err != nil {
_ = f.Close() _ = f.Close()
return nil, 0, "", fmt.Errorf("failed to stat content: %w", err) return nil, 0, fmt.Errorf("failed to stat content: %w", err)
} }
// Load metadata for content type return f, stat.Size(), nil
contentType := "application/octet-stream" // fallback }
// LoadWithMeta returns a reader, size, and content type for the content at
// the given key. The content type is read from the .meta file, and is
// empty when that file is missing or unreadable.
func (s *VariantStorage) LoadWithMeta(
key VariantKey,
) (io.ReadCloser, int64, string, error) {
f, size, err := s.LoadWithSize(key)
if err != nil {
return nil, 0, "", err
}
var contentType string
metaPath := s.keyToPath(key) + ".meta"
metaData, err := os.ReadFile(metaPath) //nolint:gosec // path derived from cache key metaData, err := os.ReadFile(metaPath) //nolint:gosec // path derived from cache key
if err == nil { if err == nil {
@@ -541,7 +553,7 @@ func (s *VariantStorage) LoadWithMeta(
} }
} }
return f, stat.Size(), contentType, nil return f, size, contentType, nil
} }
// Exists checks if content exists at the given key. // Exists checks if content exists at the given key.
@@ -24,6 +24,7 @@ const (
testHostExample = "example.com" testHostExample = "example.com"
testPathCat = "/photos/cat.jpg" testPathCat = "/photos/cat.jpg"
testContentTypeJPEG = "image/jpeg" testContentTypeJPEG = "image/jpeg"
testContentTypeWebP = "image/webp"
testHeaderContentType = "Content-Type" testHeaderContentType = "Content-Type"
) )
+64
View File
@@ -0,0 +1,64 @@
package server
import (
"net/http"
"net/http/httptest"
"testing"
)
// TestCORSOnlyOnImageRoutes verifies that the image routes answer with the
// configured access_control_allow_origin, a preflight request included, and
// that the login and URL generator pages send no Access-Control-Allow-Origin,
// so no other site can read them. /metrics is left out: its middleware
// registers with the process-wide Prometheus registry, which only one test
// in this package can do.
func TestCORSOnlyOnImageRoutes(t *testing.T) {
t.Parallel()
const appOrigin = "https://app.example.com"
s := newTestServer(t)
s.config.AccessControlAllowOrigin = appOrigin
s.SetupRoutes()
requests := []struct {
method string
path string
want string
}{
{http.MethodGet, unsignedImagePath, appOrigin},
{http.MethodHead, unsignedImagePath, appOrigin},
{http.MethodOptions, unsignedImagePath, appOrigin},
{http.MethodGet, encryptedImagePath, appOrigin},
{http.MethodGet, "/", ""},
{http.MethodOptions, "/", ""},
{http.MethodPost, "/generate", ""},
{http.MethodGet, "/logout", ""},
}
for _, tc := range requests {
t.Run(tc.method+" "+tc.path, func(t *testing.T) {
t.Parallel()
req := httptest.NewRequestWithContext(
t.Context(), tc.method, tc.path, nil)
req.Header.Set("Origin", appOrigin)
// An OPTIONS request naming the method it asks about is the
// preflight a browser sends before some cross-origin requests.
if tc.method == http.MethodOptions {
req.Header.Set("Access-Control-Request-Method", http.MethodGet)
}
rec := httptest.NewRecorder()
s.ServeHTTP(rec, req)
t.Logf("status %d", rec.Code)
got := rec.Header().Get("Access-Control-Allow-Origin")
if got != tc.want {
t.Errorf("Access-Control-Allow-Origin = %q, want %q",
got, tc.want)
}
})
}
}
+6 -2
View File
@@ -13,6 +13,10 @@ import (
// unsignedImagePath is an image URL that carries no signature. // unsignedImagePath is an image URL that carries no signature.
const unsignedImagePath = "/v1/image/cdn.example.com/cat.jpg/100x100.jpeg" const unsignedImagePath = "/v1/image/cdn.example.com/cat.jpg/100x100.jpeg"
// encryptedImagePath is an encrypted image URL whose token cannot be
// decrypted.
const encryptedImagePath = "/v1/e/token/cat.jpg"
// TestMaintenanceModeRefusesImageRequests verifies that while maintenance // TestMaintenanceModeRefusesImageRequests verifies that while maintenance
// mode is on, both image routes answer 503 Service Unavailable with a // mode is on, both image routes answer 503 Service Unavailable with a
// Retry-After header and the JSON error body the image handlers send. // Retry-After header and the JSON error body the image handlers send.
@@ -28,7 +32,7 @@ func TestMaintenanceModeRefusesImageRequests(t *testing.T) {
}{ }{
{http.MethodGet, unsignedImagePath}, {http.MethodGet, unsignedImagePath},
{http.MethodHead, unsignedImagePath}, {http.MethodHead, unsignedImagePath},
{http.MethodGet, "/v1/e/token/cat.jpg"}, {http.MethodGet, encryptedImagePath},
} }
for _, tc := range requests { for _, tc := range requests {
@@ -95,7 +99,7 @@ func TestImageRequestsServedWithoutMaintenanceMode(t *testing.T) {
}{ }{
{http.MethodGet, unsignedImagePath, http.StatusUnauthorized}, {http.MethodGet, unsignedImagePath, http.StatusUnauthorized},
{http.MethodHead, unsignedImagePath, http.StatusUnauthorized}, {http.MethodHead, unsignedImagePath, http.StatusUnauthorized},
{http.MethodGet, "/v1/e/token/cat.jpg", http.StatusBadRequest}, {http.MethodGet, encryptedImagePath, http.StatusBadRequest},
} }
for _, tc := range requests { for _, tc := range requests {
+23 -15
View File
@@ -38,7 +38,6 @@ func (s *Server) SetupRoutes() {
s.router.Use(s.mw.Metrics()) s.router.Use(s.mw.Metrics())
} }
s.router.Use(s.mw.CORS())
s.router.Use(middleware.Timeout(s.config.DownstreamTimeout)) s.router.Use(middleware.Timeout(s.config.DownstreamTimeout))
if s.sentryEnabled { if s.sentryEnabled {
@@ -74,22 +73,31 @@ func (s *Server) SetupRoutes() {
s.router.Get("/logout", s.h.HandleLogout()) s.router.Get("/logout", s.h.HandleLogout())
// Image routes, refused while maintenance mode is on. Only these: the // Image routes, the only ones that send CORS headers, as pages on other
// image's Docker HEALTHCHECK requests the health check, a 503 there // sites read them. They are a subrouter rather than a group: a group's
// would make the container unhealthy, and upaas marks a deploy failed // middleware runs only for a request that matches one of its routes,
// when its container is unhealthy. // and a browser's preflight OPTIONS request matches none, so the CORS
s.router.Group(func(r chi.Router) { // middleware could not answer it.
r.Use(s.refuseDuringMaintenance) s.router.Route("/v1", func(r chi.Router) {
r.Use(s.mw.CORS())
// Main image proxy route // Refused while maintenance mode is on. Only these: the image's
// /v1/image/<host>/<path>/<width>x<height>.<format> // Docker HEALTHCHECK requests the health check, a 503 there would
r.Get("/v1/image/*", s.h.HandleImage()) // make the container unhealthy, and upaas marks a deploy failed
r.Head("/v1/image/*", s.h.HandleImage()) // when its container is unhealthy.
r.Group(func(r chi.Router) {
r.Use(s.refuseDuringMaintenance)
// Encrypted image URL route // Main image proxy route
// The trailing filename (e.g., /img.jpg) is ignored but helps // /v1/image/<host>/<path>/<width>x<height>.<format>
// browsers with content type r.Get("/image/*", s.h.HandleImage())
r.Get("/v1/e/{token}/*", s.h.HandleImageEnc()) r.Head("/image/*", s.h.HandleImage())
// Encrypted image URL route
// The trailing filename (e.g., /img.jpg) is ignored but helps
// browsers with content type
r.Get("/e/{token}/*", s.h.HandleImageEnc())
})
}) })
// Metrics endpoint with auth // Metrics endpoint with auth