9 Commits
Author SHA1 Message Date
clawbot c6e66bf45a Test that a read of an older .meta file keeps the stored content type (closes #70)
check / check (push) Successful in 3m23s
A read that finds no .meta file keeps nothing in memory, so the
existing test no longer reaches the step that keeps a type StoreVariant
put there. This one writes a .meta file with a different content type
after the store, as one not yet rewritten by it would hold, and checks
that the read leaves the stored type in memory.

Model: opus-5-5
2026-09-29 09:31:47 +00:00
clawbot 314fbd186d Keep application/octet-stream out of memory for variants (closes #70)
VariantStorage.LoadWithMeta now returns an empty content type when the
.meta file is missing or unreadable, and GetVariant serves
application/octet-stream in that case without keeping it in memory.
Memory now only ever holds a content type read from a .meta file or
passed to StoreVariant, so a read that ran before a store wrote the
.meta file can no longer leave the wrong type in memory, however it
interleaves with the store, other reads and eviction.

Model: opus-5-5
2026-09-29 09:28:49 +00:00
clawbot 8be21c005f Test that a failed read during a store keeps the stored content type (closes #70)
A read that finds a variant in memory but cannot open its file removes
it from memory. When that happens after StoreVariant has added the
variant's content type, a read that opened the variant file before the
store wrote its .meta file finds memory empty and keeps
application/octet-stream there, so every later hit serves the image
with it. The test runs the store, the failed read and the read without
a .meta file in that order, then checks a later hit. It fails now.

Model: opus-5-5
2026-09-29 09:28:38 +00:00
clawbot c7e4eacd87 Have Lookup check the disk for variants held in memory too (closes #70)
Lookup again counts a variant as present only when its file exists.
Treating a variant held in memory as present saved one check of the
disk, but no test covered it, and GetVariant opens the file anyway.

Model: opus-5-5
2026-09-29 09:28:38 +00:00
clawbot 77e9c9cd7f Keep the content type a store put in memory over one read from disk (closes #70)
GetVariant now adds the content type it read from a .meta file only
when memory holds none for the variant, so a type StoreVariant added
meanwhile is not replaced. A read that found no .meta file yet still
gets application/octet-stream for that one request, as before this
change, but no longer leaves it in memory for later hits.

Model: opus-5-5
2026-09-29 09:28:28 +00:00
clawbot 220fb3ee4e Test that a read during a store keeps the stored content type (closes #70)
A GetVariant that begins before StoreVariant finishes can find the
variant file but not yet its .meta file, and so reads
application/octet-stream. When it adds that to memory after the store
added the real type, the wrong type is served to every later hit. The
part of GetVariant that runs after its check of memory moves, unchanged,
into loadVariantWithMeta, so the test can run it after a store. The
test fails now.

Model: opus-5-5
2026-09-29 09:28:28 +00:00
clawbot ffe64d5fe0 Keep variant content types in memory for cache hits (closes #70)
Cache.metaCache was declared and never used, so every hit read and
parsed the variant's .meta file. It is now an LRU of up to 10,000
content types (hashicorp/golang-lru/v2), filled by StoreVariant and by
GetVariant after it reads a .meta file. For a variant it holds, Lookup
skips the disk check and GetVariant skips the .meta read; the variant
file is still opened and its size taken from it. Eviction removes the
entry before deleting the files, and GetVariant removes it when the
file will not open, so a missing variant is never served. The cap is a
constant, not a setting. README.md describes it.

Model: opus-5-5
2026-09-29 09:28:28 +00:00
clawbot 33c958ad1c Test that a cache hit takes the content type from memory (closes #70)
Tests for keeping each variant's content type in memory. A second hit
must still get the stored content type after the variant's .meta file
is deleted, whether the first came from storing the variant or from
reading it after a restart; this fails now. A variant removed by
EvictToLimit, or whose file was deleted from disk, must be a miss and
must not be served, and concurrent stores, reads and evictions run
under the race detector; these pass now and guard the change.

Model: opus-5-5
2026-09-29 09:28:16 +00:00
clawbot 99735f479b Answer image requests with 503 in maintenance mode (closes #71)
check / check (push) Successful in 13s
maintenance_mode was only reported by the health check; every request
was still served. One middleware in routes.go, applied to /v1/image/
and /v1/e/ only, now answers them with 503, a Retry-After of
MaintenanceRetryAfterSeconds and the JSON error body while it is on.
It calls Server.MaintenanceMode(), which had no caller.

The health check stays 200 and reports maintenance_mode: the image's
Docker HEALTHCHECK requests it, a 503 there would make the container
unhealthy, and upaas marks a deploy failed when its container is
unhealthy. The login and URL generator pages and /metrics keep working.
Documented in README.md and config.example.yml.

Model: opus-5-5
2026-09-29 11:01:30 +02:00
7 changed files with 298 additions and 10 deletions
+8 -1
View File
@@ -248,7 +248,7 @@ variables set by the file's `env:` section are checked the same way.
| `PIXA_METRICS_PASSWORD` | `metrics.password` | Password for `/metrics`; set together with the username |
| `PIXA_SENTRY_DSN` | `sentry_dsn` | Sentry DSN for error reporting; empty disables it |
| `PIXA_DEBUG` | `debug` | Debug logging and plain-HTTP local development; default `false` |
| `PIXA_MAINTENANCE_MODE` | `maintenance_mode` | Maintenance flag reported by the health check; default `false` |
| `PIXA_MAINTENANCE_MODE` | `maintenance_mode` | Answer image requests with 503; the health check stays 200; default `false` |
Key settings in more detail:
@@ -309,6 +309,13 @@ Key settings in more detail:
container's CPU limit. A request that finds all of them in use waits up to 10
seconds for one to free up; if none does, and `downstream_timeout` has not
ended first, it is answered 503 the same way
- `maintenance_mode` — while `true`, the image routes (`/v1/image/` and
`/v1/e/`) answer every request with 503, a `Retry-After` header and a JSON
error body. The health check (`/.well-known/healthcheck.json`) still answers
200 and reports `"maintenance_mode": true`. It stays 200 because the image's
Docker `HEALTHCHECK` requests it: a 503 there would make the container
unhealthy, and upaas marks a deploy failed when its container is unhealthy.
The login and URL generator pages and `/metrics` keep working
See `config.example.yml` for all options with defaults.
+8
View File
@@ -40,6 +40,14 @@ P2: security: referer blacklist
and `GetVariant` removes it when the file will not open; the cap is a
constant, not a setting; the unused `variantMeta` type is gone; `README.md`
describes it.
- 2026-09-29 maintenance mode refuses image requests (closes #71): while
`maintenance_mode` is on, `/v1/image/` and `/v1/e/` answer 503 with a
`Retry-After` header and the JSON error body, from one middleware in
`internal/server/routes.go`; the health check stays 200 and reports
`maintenance_mode`, as the image's Docker `HEALTHCHECK` requests it and upaas
marks a deploy failed when its container is unhealthy; the login and URL
generator pages and `/metrics` keep working; documented in `README.md` and
`config.example.yml`.
- 2026-09-29 bound concurrent image processing and upstream fetches (closes
#64): `max_concurrent_processing` (default the number of CPUs pixa can use)
limits the images decoded and encoded at once, and `upstream_connections`
+6
View File
@@ -16,6 +16,12 @@
# Server settings
port: 8080
debug: false
# While true, the image routes (/v1/image/ and /v1/e/) answer every request
# with 503 and a Retry-After header. The health check keeps answering 200 and
# reports maintenance_mode as true. It stays 200 because the image's Docker
# HEALTHCHECK requests it: a 503 there would make the container unhealthy, and
# upaas marks a deploy failed when its container is unhealthy.
maintenance_mode: false
# Data directory for SQLite database and cache files
@@ -2,6 +2,7 @@ package imgcache
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
@@ -155,6 +156,54 @@ func TestReadDuringStoreKeepsStoredContentType(t *testing.T) {
assertVariantServed(t, cache, testVariantKeyOne, content)
}
// TestReadOfOlderMetaFileKeepsStoredContentType checks that a read
// which got its content type from a .meta file that StoreVariant had
// not yet rewritten cannot replace the type the store kept in memory.
// The test writes such a .meta file, with a different content type,
// after the store, then runs the part of GetVariant that comes after
// its check of memory.
func TestReadOfOlderMetaFileKeepsStoredContentType(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<20)
content := []byte("webp variant bytes")
storeEvictionTestVariant(t, cache, testVariantKeyOne, content)
olderMeta, err := json.Marshal(VariantMeta{
ContentType: testContentTypeJPEG,
Size: int64(len(content)),
})
if err != nil {
t.Fatalf("encoding .meta file: %v", err)
}
metaPath := cache.variants.keyToPath(testVariantKeyOne) + ".meta"
err = os.WriteFile(metaPath, olderMeta, StorageFilePerm)
if err != nil {
t.Fatalf("writing .meta file: %v", err)
}
reader, _, contentType, err := cache.loadVariantWithMeta(testVariantKeyOne)
if err != nil {
t.Fatalf("loadVariantWithMeta(%s) error = %v", testVariantKeyOne, err)
}
_ = reader.Close()
if contentType != testContentTypeJPEG {
t.Fatalf("loadVariantWithMeta(%s) content type = %q, want %q from the .meta file",
testVariantKeyOne, contentType, testContentTypeJPEG)
}
kept, _ := cache.metaCache.Get(testVariantKeyOne)
if kept != testContentTypeWebP {
t.Errorf("content type in memory = %q, want the stored %q",
kept, testContentTypeWebP)
}
}
// TestFailedReadDuringStoreKeepsStoredContentType checks that a read
// which found no .meta file cannot leave application/octet-stream in
// memory, even when another read has removed the content type
@@ -18,6 +18,7 @@ import (
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/globals"
"sneak.berlin/go/pixa/internal/handlers"
"sneak.berlin/go/pixa/internal/healthcheck"
"sneak.berlin/go/pixa/internal/logger"
"sneak.berlin/go/pixa/internal/middleware"
)
@@ -71,8 +72,15 @@ func newTestServer(t *testing.T) *Server {
t.Fatalf("database.New() error = %v", err)
}
hc, err := healthcheck.New(lc, healthcheck.Params{
Globals: &globals.Globals{}, Config: cfg, Logger: log, Database: db,
})
if err != nil {
t.Fatalf("healthcheck.New() error = %v", err)
}
h, err := handlers.New(lc, handlers.Params{
Logger: log, Database: db, Config: cfg,
Logger: log, Healthcheck: hc, Database: db, Config: cfg,
})
if err != nil {
t.Fatalf("handlers.New() error = %v", err)
@@ -0,0 +1,169 @@
package server
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strconv"
"testing"
"sneak.berlin/go/pixa/internal/healthcheck"
)
// unsignedImagePath is an image URL that carries no signature.
const unsignedImagePath = "/v1/image/cdn.example.com/cat.jpg/100x100.jpeg"
// TestMaintenanceModeRefusesImageRequests verifies that while maintenance
// mode is on, both image routes answer 503 Service Unavailable with a
// Retry-After header and the JSON error body the image handlers send.
func TestMaintenanceModeRefusesImageRequests(t *testing.T) {
t.Parallel()
s := newTestServer(t)
s.config.MaintenanceMode = true
requests := []struct {
method string
path string
}{
{http.MethodGet, unsignedImagePath},
{http.MethodHead, unsignedImagePath},
{http.MethodGet, "/v1/e/token/cat.jpg"},
}
for _, tc := range requests {
t.Run(tc.method+" "+tc.path, func(t *testing.T) {
t.Parallel()
rec := httptest.NewRecorder()
s.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), tc.method, tc.path, nil))
t.Logf("status %d, body %s", rec.Code, rec.Body.String())
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("status = %d, want %d",
rec.Code, http.StatusServiceUnavailable)
}
retryAfter := rec.Header().Get("Retry-After")
seconds, err := strconv.Atoi(retryAfter)
if err != nil || seconds <= 0 {
t.Errorf("Retry-After = %q, want a positive number of seconds",
retryAfter)
}
// A HEAD response carries no body.
if tc.method == http.MethodHead {
return
}
var body struct {
Error string `json:"error"`
Status int `json:"status"`
Timestamp string `json:"timestamp"`
}
err = json.NewDecoder(rec.Body).Decode(&body)
if err != nil {
t.Fatalf("body is not JSON: %v", err)
}
if body.Error == "" || body.Status != http.StatusServiceUnavailable ||
body.Timestamp == "" {
t.Errorf("body = %+v, want an error, status %d and a timestamp",
body, http.StatusServiceUnavailable)
}
})
}
}
// TestImageRequestsServedWithoutMaintenanceMode verifies that while
// maintenance mode is off, image requests reach the image handlers instead
// of the 503. The handlers refuse an unsigned image URL with 401 and a token
// they cannot decrypt with 400, so either status shows a request got through.
func TestImageRequestsServedWithoutMaintenanceMode(t *testing.T) {
t.Parallel()
s := newTestServer(t)
s.config.MaintenanceMode = false
requests := []struct {
method string
path string
want int
}{
{http.MethodGet, unsignedImagePath, http.StatusUnauthorized},
{http.MethodHead, unsignedImagePath, http.StatusUnauthorized},
{http.MethodGet, "/v1/e/token/cat.jpg", http.StatusBadRequest},
}
for _, tc := range requests {
t.Run(tc.method+" "+tc.path, func(t *testing.T) {
t.Parallel()
rec := httptest.NewRecorder()
s.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), tc.method, tc.path, nil))
t.Logf("status %d, body %s", rec.Code, rec.Body.String())
if rec.Code != tc.want {
t.Errorf("status = %d, want %d from the image handler",
rec.Code, tc.want)
}
})
}
}
// TestMaintenanceModeKeepsOtherRoutes verifies that while maintenance mode
// is on, the health check still answers 200 and reports it, and the login
// page and /metrics still answer 200. The image's Docker HEALTHCHECK
// requests the health check: a 503 there would make the container
// unhealthy, and upaas marks a deploy failed when its container is
// unhealthy.
func TestMaintenanceModeKeepsOtherRoutes(t *testing.T) {
t.Parallel()
s := newTestServer(t)
s.config.MaintenanceMode = true
// /metrics is routed only when its username is set.
s.config.MetricsUsername = "metrics"
s.config.MetricsPassword = "metrics-password"
s.SetupRoutes()
rec := httptest.NewRecorder()
s.ServeHTTP(rec, httptest.NewRequestWithContext(t.Context(),
http.MethodGet, "/.well-known/healthcheck.json", nil))
t.Logf("health check status %d, body %s", rec.Code, rec.Body.String())
if rec.Code != http.StatusOK {
t.Fatalf("health check status = %d, want %d", rec.Code, http.StatusOK)
}
var health healthcheck.Response
err := json.NewDecoder(rec.Body).Decode(&health)
if err != nil || !health.Maintenance {
t.Errorf("health check maintenance_mode = %v (error %v), want true",
health.Maintenance, err)
}
rec = httptest.NewRecorder()
s.ServeHTTP(rec, clientRequest(t, http.MethodGet, nil, firstClient, ""))
if rec.Code != http.StatusOK {
t.Errorf("login page status = %d, want %d", rec.Code, http.StatusOK)
}
req := httptest.NewRequestWithContext(t.Context(),
http.MethodGet, "/metrics", nil)
req.SetBasicAuth(s.config.MetricsUsername, s.config.MetricsPassword)
rec = httptest.NewRecorder()
s.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("/metrics status = %d, want %d", rec.Code, http.StatusOK)
}
}
+49 -8
View File
@@ -1,7 +1,9 @@
package server
import (
"encoding/json"
"net/http"
"strconv"
"time"
sentryhttp "github.com/getsentry/sentry-go/http"
@@ -17,6 +19,10 @@ import (
// make per minute; the next is refused with 429 Too Many Requests.
const LoginAttemptsPerMinute = 5
// MaintenanceRetryAfterSeconds is the Retry-After, in seconds, sent with
// the 503 that the image routes answer while maintenance mode is on.
const MaintenanceRetryAfterSeconds = 300
// SetupRoutes configures all HTTP routes.
func (s *Server) SetupRoutes() {
s.router = chi.NewRouter()
@@ -68,15 +74,23 @@ func (s *Server) SetupRoutes() {
s.router.Get("/logout", s.h.HandleLogout())
// Main image proxy route
// /v1/image/<host>/<path>/<width>x<height>.<format>
s.router.Get("/v1/image/*", s.h.HandleImage())
s.router.Head("/v1/image/*", s.h.HandleImage())
// Image routes, refused while maintenance mode is on. Only these: the
// image's Docker HEALTHCHECK requests the health check, a 503 there
// would make the container unhealthy, and upaas marks a deploy failed
// when its container is unhealthy.
s.router.Group(func(r chi.Router) {
r.Use(s.refuseDuringMaintenance)
// Encrypted image URL route
// The trailing filename (e.g., /img.jpg) is ignored but helps
// browsers with content type
s.router.Get("/v1/e/{token}/*", s.h.HandleImageEnc())
// Main image proxy route
// /v1/image/<host>/<path>/<width>x<height>.<format>
r.Get("/v1/image/*", s.h.HandleImage())
r.Head("/v1/image/*", s.h.HandleImage())
// Encrypted image URL route
// The trailing filename (e.g., /img.jpg) is ignored but helps
// browsers with content type
r.Get("/v1/e/{token}/*", s.h.HandleImageEnc())
})
// Metrics endpoint with auth
if s.config.MetricsUsername != "" {
@@ -86,3 +100,30 @@ func (s *Server) SetupRoutes() {
})
}
}
// refuseDuringMaintenance answers a request with 503 Service Unavailable,
// a Retry-After header and a JSON error body while maintenance mode is on,
// and passes it on otherwise. The body has the fields of the JSON errors
// the image handlers send.
func (s *Server) refuseDuringMaintenance(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !s.MaintenanceMode() {
next.ServeHTTP(w, r)
return
}
w.Header().Set("Retry-After", strconv.Itoa(MaintenanceRetryAfterSeconds))
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusServiceUnavailable)
err := json.NewEncoder(w).Encode(map[string]any{
"error": "down for maintenance, try again later",
"status": http.StatusServiceUnavailable,
"timestamp": time.Now().UTC().Format(time.RFC3339),
})
if err != nil {
s.log.Error("json encode error", "error", err)
}
})
}