Author SHA1 Message Date
clawbot fc080147af Queue SQLite writes on one connection so none fails as locked (closes #223)
check / check (push) Waiting to run
internal/database now opens the database with one connection. pixa's
own reads and writes run on it one at a time instead of competing for
SQLite's lock, where a write that kept losing could wait past the
five-second busy timeout and fail with "database is locked". The busy
timeout stays, for another program writing to the same file.

With one connection, a query run while rows or a transaction are still
open would wait forever. No code in internal/database or
internal/imgcache does that; the comment on DB() tells callers.
README.md says pixa uses one connection and that requests wait while
eviction runs one of its queries.

Model: opus-5-5
2026-10-08 02:00:17 +00:00
11 changed files with 77 additions and 225 deletions
+3 -6
View File
@@ -22,9 +22,8 @@ FROM golang:1.25.4-alpine@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66
WORKDIR /src
# script/bootstrap --cgo installs the build dependencies (a C compiler,
# the libvips and libheif headers, and libvips' JPEG XL support, which
# the tests need) and downloads the Go modules.
# script/bootstrap --cgo installs the build dependencies (a C compiler
# and the libvips and libheif headers) and downloads the Go modules.
COPY script/ ./script/
COPY go.mod go.sum ./
RUN script/bootstrap --cgo
@@ -81,11 +80,9 @@ RUN version="${VERSION:-$(git describe --tags --always)}"; \
# alpine:3.21, 2026-02-25
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
# Install runtime dependencies only. vips-jxl is libvips' JPEG XL
# support, without which pixad does not start.
# Install runtime dependencies only
RUN apk add --no-cache \
vips \
vips-jxl \
libheif \
ca-certificates \
tzdata \
+10 -10
View File
@@ -89,10 +89,7 @@ another part of pixa failed to stop. A request not finished by then is cut off.
`docker stop` waits 10 seconds before it kills the container.
Outside Docker, pixa needs libvips (the image has 8.15) and libheif to run, as
it uses libvips through CGO. pixad does not start unless libvips has its JPEG XL
support, which on Alpine is the `vips-jxl` package and which the nix and brew
packages of libvips include, as do the apt ones from Debian 12 and Ubuntu 24.04
on. Building pixa also needs the development files of libvips and libheif,
it uses libvips through CGO; building it also needs their development files,
`pkg-config` and a C compiler. `script/bootstrap --cgo` installs all of these,
as the `Dockerfile` does where it compiles pixa. Plain `script/bootstrap`, which
`script/setup` and `script/cibuild` run, installs git, make and Go, and Node,
@@ -532,10 +529,13 @@ Key settings in more detail:
- `signing_key` — HMAC secret for URL signatures
- `db_url` — the SQLite database to open; omitted, it is
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
database in WAL mode. pixa adds `_pragma=busy_timeout(5000)` to any `db_url`,
so a write that finds another in progress waits up to five seconds for it
instead of failing. WAL mode comes only from the URL: keep
`_pragma=journal_mode(WAL)` in one you set
database in WAL mode. pixa opens one connection to it, so its own reads and
writes run one at a time. Requests wait while eviction runs one of its
queries, some of which read a whole table. pixa adds
`_pragma=busy_timeout(5000)` to any `db_url`, so a write that finds another
program writing to the file waits up to five seconds for it instead of
failing. WAL mode comes only from the URL: keep `_pragma=journal_mode(WAL)` in
one you set
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
cache entirely; omitted defaults to 75% of the sum of the free space on the
filesystem containing `<state_dir>/cache/` and the bytes of source and
@@ -585,8 +585,8 @@ provide:
- `script/bootstrap` — install git, make, Go, Node, Yarn and prettier and
download the Go modules (idempotent); with `--cgo`, the C compiler and the
libvips (with its JPEG XL support) and libheif libraries that compiling and
testing pixa need instead of Node, Yarn and prettier
libvips and libheif libraries that compiling pixa needs instead of Node, Yarn
and prettier
- `script/setup` — make a fresh clone ready for development (bootstrap, then
install-precommit)
- `script/projectname` — output the project name ("pixa")
+7 -6
View File
@@ -30,12 +30,13 @@ P2: security: per-IP rate limiting on the image routes
# Completed Steps
- 2026-10-08 libvips' JPEG XL support is installed and required (part of #222):
`script/bootstrap --cgo` installs `vips-jxl` on Alpine, whose `vips` package
lacks it, and the runtime stage of the `Dockerfile` installs it too.
`imageprocessor.New` fails when libvips cannot load and save JPEG XL, so pixad
does not start without it. A test saves an image as JPEG XL with govips and
loads it back. JPEG XL is not yet a format pixa serves.
- 2026-10-08 SQLite writes no longer fail with "database is locked" under load
(closes #223): `internal/database` opens the database with one connection, so
pixa's own reads and writes run on it one at a time instead of competing for
SQLite's lock, where a write that kept losing could wait past the five-second
busy timeout and be lost. The busy timeout stays, for another program writing
to the same file. No code in pixa keeps rows or a transaction open while it
runs another query, which with one connection would wait forever.
- 2026-10-05 the format `auto` (closes #88): a format in the `/v1/image/` path,
an encrypted URL's token and the generator page's format choice, chosen for
each request from `Accept` once the signature or token is checked: AVIF when
@@ -13,11 +13,10 @@ import (
)
// TestConcurrentWritesAllSucceed opens a database the way pixad does and
// writes to it from several goroutines at once, so the writes run on
// separate connections, as one request's writes and the background eviction
// pass do. Every write must succeed, none failing with "database is locked",
// whether or not db_url already has parameters, and the parameters it has
// must still apply.
// writes to it from several goroutines at once, as one request's writes and
// the background eviction pass do. Every write must succeed, none failing
// with "database is locked", whether or not db_url already has parameters,
// and the parameters it has must still apply.
func TestConcurrentWritesAllSucceed(t *testing.T) {
t.Parallel()
+12 -6
View File
@@ -237,17 +237,18 @@ func ApplyMigrations(ctx context.Context, db *sql.DB, log *slog.Logger) error {
return nil
}
// DB returns the underlying sql.DB.
// DB returns the underlying sql.DB. It has one connection, so close any
// rows and end any transaction before running another query on it; a
// query run while they are open waits forever.
func (s *Database) DB() *sql.DB {
return s.db
}
func (s *Database) connect(ctx context.Context) error {
// Requests and the eviction pass write on separate connections. With
// a busy timeout, a write that finds another one in progress waits up
// to five seconds for it instead of failing at once with "database is
// locked". The driver runs each _pragma parameter on every connection
// it opens.
// With a busy timeout, a write that finds another program writing to
// the same database file waits up to five seconds for it instead of
// failing at once with "database is locked". The driver runs each
// _pragma parameter on every connection it opens.
separator := "?"
if strings.Contains(s.config.DBURL, "?") {
separator = "&"
@@ -264,6 +265,11 @@ func (s *Database) connect(ctx context.Context) error {
return err
}
// One connection: pixa's own reads and writes run on it one at a
// time instead of competing for SQLite's lock, where a write that
// keeps losing can wait past the busy timeout and be lost.
db.SetMaxOpenConns(1)
err = db.PingContext(ctx)
if err != nil {
s.log.Error("failed to ping database", "error", err)
+5 -21
View File
@@ -19,17 +19,13 @@ import (
//nolint:gochecknoglobals // package-level sync.Once for one-time vips init
var vipsOnce sync.Once
// errNoJPEGXL is returned by New when libvips cannot load and save JPEG XL.
var errNoJPEGXL = errors.New("libvips lacks JPEG XL support")
// initVips initializes libvips with quiet logging, one worker thread per
// image and no operation cache. Process already works on one image per CPU
// by default, so more threads per image would only compete for the CPUs.
// Each request decodes different source bytes, so the operation cache
// would rarely be hit and would hold memory outside MaxConcurrentProcessing;
// repeated requests are served from pixa's disk cache instead.
// It returns errNoJPEGXL when libvips cannot load and save JPEG XL.
func initVips() error {
func initVips() {
vipsOnce.Do(func() {
vips.LoggingSettings(nil, vips.LogLevelError)
vips.Startup(&vips.Config{
@@ -39,14 +35,6 @@ func initVips() error {
MaxCacheFiles: 0,
})
})
// govips counts a format as supported when libvips has its loader;
// libvips builds the JPEG XL loader and saver together.
if !vips.IsTypeSupported(vips.ImageTypeJXL) {
return errNoJPEGXL
}
return nil
}
// Format represents supported output image formats.
@@ -161,13 +149,9 @@ type Params struct {
}
// New creates a new image processor with the given parameters.
// A zero-value Params{} uses sensible defaults. It fails when libvips
// cannot load and save JPEG XL.
func New(params Params) (*ImageProcessor, error) {
err := initVips()
if err != nil {
return nil, err
}
// A zero-value Params{} uses sensible defaults.
func New(params Params) *ImageProcessor {
initVips()
maxInputBytes := params.MaxInputBytes
if maxInputBytes <= 0 {
@@ -183,7 +167,7 @@ func New(params Params) (*ImageProcessor, error) {
maxInputBytes: maxInputBytes,
processingSemaphore: make(chan struct{}, maxConcurrentProcessing),
processingWaitTimeout: ProcessingWaitTimeout,
}, nil
}
}
// Process transforms an image according to the request. When
@@ -18,10 +18,7 @@ import (
)
func TestMain(m *testing.M) {
err := initVips()
if err != nil {
panic(err)
}
initVips()
code := m.Run()
@@ -121,11 +118,7 @@ func detectMIME(data []byte) string {
func TestImageProcessor_ResizeJPEG(t *testing.T) {
t.Parallel()
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{})
ctx := context.Background()
input := createTestJPEG(t, 800, 600)
@@ -171,11 +164,7 @@ func TestImageProcessor_ResizeJPEG(t *testing.T) {
func TestImageProcessor_ConvertToPNG(t *testing.T) {
t.Parallel()
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{})
ctx := context.Background()
input := createTestJPEG(t, 200, 150)
@@ -211,11 +200,7 @@ func processAndCheckSize(
) {
t.Helper()
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{})
ctx := context.Background()
input := createTestJPEG(t, inputW, inputH)
@@ -253,11 +238,7 @@ func TestImageProcessor_OriginalSize(t *testing.T) {
func TestImageProcessor_FitContain(t *testing.T) {
t.Parallel()
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{})
ctx := context.Background()
// 800x400 image (2:1 aspect) into 400x400 box with contain
@@ -303,11 +284,7 @@ func TestImageProcessor_ProportionalScale_HeightOnly(t *testing.T) {
func TestImageProcessor_ProcessPNG(t *testing.T) {
t.Parallel()
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{})
ctx := context.Background()
input := createTestPNG(t, 400, 300)
@@ -337,10 +314,7 @@ func TestImageProcessor_ProcessPNG(t *testing.T) {
func TestImageProcessor_SupportedFormats(t *testing.T) {
t.Parallel()
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{})
inputFormats := proc.SupportedInputFormats()
if len(inputFormats) == 0 {
@@ -371,11 +345,7 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{})
ctx := context.Background()
input := createTestJPEG(t, tt.width, tt.height)
@@ -386,7 +356,7 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
FitMode: FitCover,
}
_, err = proc.Process(ctx, bytes.NewReader(input), req)
_, err := proc.Process(ctx, bytes.NewReader(input), req)
if err == nil {
t.Error("Process() should reject oversized input images")
}
@@ -401,11 +371,7 @@ func TestImageProcessor_RejectsOversizedInput(t *testing.T) {
func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
t.Parallel()
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{})
ctx := context.Background()
// Create an image at exactly MaxInputDimension - should be accepted
@@ -434,11 +400,7 @@ func TestImageProcessor_AcceptsMaxDimensionInput(t *testing.T) {
func encodeAndCheck(t *testing.T, format Format, quality int, wantMIME string) {
t.Helper()
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{})
ctx := context.Background()
input := createTestJPEG(t, 200, 150)
@@ -487,11 +449,7 @@ func TestImageProcessor_EncodeWebP(t *testing.T) {
func TestImageProcessor_DecodeAVIF(t *testing.T) {
t.Parallel()
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{})
ctx := context.Background()
// Load test AVIF file
@@ -533,11 +491,7 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
// Create a processor with a very small byte limit
const limit = 1024
proc, err := New(Params{MaxInputBytes: limit})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{MaxInputBytes: limit})
ctx := context.Background()
// Create a valid JPEG that exceeds the byte limit
@@ -553,7 +507,7 @@ func TestImageProcessor_RejectsOversizedInputData(t *testing.T) {
FitMode: FitCover,
}
_, err = proc.Process(ctx, bytes.NewReader(input), req)
_, err := proc.Process(ctx, bytes.NewReader(input), req)
if err == nil {
t.Fatal("Process() should reject input exceeding maxInputBytes")
}
@@ -570,11 +524,7 @@ func TestImageProcessor_AcceptsInputWithinLimit(t *testing.T) {
input := createTestJPEG(t, 10, 10)
limit := int64(len(input)) * 10 // 10× headroom
proc, err := New(Params{MaxInputBytes: limit})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{MaxInputBytes: limit})
ctx := context.Background()
req := &Request{
@@ -596,21 +546,13 @@ func TestImageProcessor_DefaultMaxInputBytes(t *testing.T) {
t.Parallel()
// Passing 0 should use the default
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{})
if proc.maxInputBytes != DefaultMaxInputBytes {
t.Errorf("maxInputBytes = %d, want %d", proc.maxInputBytes, DefaultMaxInputBytes)
}
// Passing negative should also use the default
proc, err = New(Params{MaxInputBytes: -1})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc = New(Params{MaxInputBytes: -1})
if proc.maxInputBytes != DefaultMaxInputBytes {
t.Errorf("maxInputBytes = %d, want %d", proc.maxInputBytes, DefaultMaxInputBytes)
}
@@ -622,51 +564,14 @@ func TestImageProcessor_EncodeAVIF(t *testing.T) {
encodeAndCheck(t, FormatAVIF, 85, mimeAVIF)
}
// TestLibvipsSavesAndLoadsJPEGXL saves an image as JPEG XL with govips and
// loads it back. It fails when libvips lacks JPEG XL support, as on Alpine
// without the vips-jxl package.
func TestLibvipsSavesAndLoadsJPEGXL(t *testing.T) {
t.Parallel()
img, err := vips.NewImageFromBuffer(createTestJPEG(t, 64, 48))
if err != nil {
t.Fatalf("failed to load test JPEG: %v", err)
}
defer img.Close()
jxl, _, err := img.ExportJxl(vips.NewJxlExportParams())
if err != nil {
t.Fatalf("ExportJxl() error = %v", err)
}
loaded, err := vips.NewImageFromBuffer(jxl)
if err != nil {
t.Fatalf("failed to load the JPEG XL image: %v", err)
}
defer loaded.Close()
if loaded.Format() != vips.ImageTypeJXL {
t.Errorf("loaded format = %s, want jxl", vips.ImageTypes[loaded.Format()])
}
if loaded.Width() != 64 || loaded.Height() != 48 {
t.Errorf("loaded size = %dx%d, want 64x48", loaded.Width(), loaded.Height())
}
}
// processAndDecode runs input through Process and decodes the output with
// vips, so a test can inspect the image a client would receive.
func processAndDecode(t *testing.T, input []byte, req *Request) *vips.ImageRef {
t.Helper()
proc, err := New(Params{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
result, err := proc.Process(context.Background(), bytes.NewReader(input), req)
result, err := New(Params{}).Process(
context.Background(), bytes.NewReader(input), req,
)
if err != nil {
t.Fatalf("Process() error = %v", err)
}
@@ -119,22 +119,14 @@ func TestNewDefaultsMaxConcurrentProcessingToCPUs(t *testing.T) {
t.Parallel()
for _, limit := range []int{0, -1} {
proc, err := New(Params{MaxConcurrentProcessing: limit})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{MaxConcurrentProcessing: limit})
if got := cap(proc.processingSemaphore); got != runtime.GOMAXPROCS(0) {
t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, one per CPU",
limit, got, runtime.GOMAXPROCS(0))
}
}
proc, err := New(Params{MaxConcurrentProcessing: 3})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{MaxConcurrentProcessing: 3})
if got := cap(proc.processingSemaphore); got != 3 {
t.Errorf("MaxConcurrentProcessing 3: %d slots, want 3", got)
}
@@ -153,11 +145,7 @@ func TestProcessNeverExceedsMaxConcurrentProcessing(t *testing.T) {
calls = 6
)
proc, err := New(Params{MaxConcurrentProcessing: limit})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{MaxConcurrentProcessing: limit})
input := createTestJPEG(t, 50, 50)
counter := &readingCounter{}
@@ -204,11 +192,7 @@ func TestProcessNeverExceedsMaxConcurrentProcessing(t *testing.T) {
func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) {
t.Parallel()
proc, err := New(Params{MaxConcurrentProcessing: 1})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{MaxConcurrentProcessing: 1})
proc.processingWaitTimeout = 100 * time.Millisecond
input := createTestJPEG(t, 10, 10)
@@ -228,7 +212,7 @@ func TestProcessWaitsThenFailsWhenNoSlotFrees(t *testing.T) {
start := time.Now()
_, err = proc.Process(context.Background(), bytes.NewReader(input),
_, err := proc.Process(context.Background(), bytes.NewReader(input),
smallJPEGRequest())
if !errors.Is(err, ErrTooManyImages) {
t.Fatalf("Process() error = %v, want ErrTooManyImages", err)
@@ -294,14 +278,10 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
proc, err := New(Params{MaxInputBytes: 4096, MaxConcurrentProcessing: 1})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{MaxInputBytes: 4096, MaxConcurrentProcessing: 1})
proc.processingWaitTimeout = 100 * time.Millisecond
_, err = proc.Process(context.Background(), tc.input, tc.req)
_, err := proc.Process(context.Background(), tc.input, tc.req)
if err == nil || (tc.want != nil && !errors.Is(err, tc.want)) {
t.Fatalf("Process() error = %v, want %v", err, tc.want)
}
@@ -328,10 +308,7 @@ func TestProcessReleasesSlotOnError(t *testing.T) {
func TestWaitForProcessing(t *testing.T) {
t.Parallel()
proc, err := New(Params{MaxConcurrentProcessing: 2})
if err != nil {
t.Fatalf("New() error = %v", err)
}
proc := New(Params{MaxConcurrentProcessing: 2})
gate := make(chan struct{})
entered := make(chan struct{}, 1)
@@ -370,7 +347,7 @@ func TestWaitForProcessing(t *testing.T) {
openGate()
err = <-results
err := <-results
if err != nil {
t.Errorf("Process() error = %v, want nil", err)
}
@@ -60,15 +60,9 @@ func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
processor, err := imageprocessor.New(
svc.processor = imageprocessor.New(
imageprocessor.Params{MaxConcurrentProcessing: 1},
)
if err != nil {
t.Fatalf("imageprocessor.New() error = %v", err)
}
svc.processor = processor
// A first request caches the photo as a source.
resp, err := svc.Get(t.Context(), widthOnlyRequest(fixtures, 50))
+1 -5
View File
@@ -101,14 +101,10 @@ func NewService(cfg *ServiceConfig) (*Service, error) {
}
maxResponseSize := fetcherCfg.MaxResponseSize
processor, err := imageprocessor.New(imageprocessor.Params{
processor := imageprocessor.New(imageprocessor.Params{
MaxInputBytes: maxResponseSize,
MaxConcurrentProcessing: cfg.MaxConcurrentProcessing,
})
if err != nil {
return nil, err
}
return &Service{
cache: cfg.Cache,
+5 -12
View File
@@ -14,12 +14,11 @@
# script/fmt-check: all the host needs, as
# the checks compile pixa in Docker
# script/bootstrap --cgo git, make, Go, and a C compiler and the
# CGO image libraries (pkg-config, vips
# with its JPEG XL support, libheif) for
# the govips bindings instead of Node: to
# compile pixa, in the Dockerfile's test
# phase and build stage, which format
# nothing
# CGO image libraries (pkg-config, vips,
# libheif) for the govips bindings instead
# of Node: to compile pixa, in the
# Dockerfile's test phase and build stage,
# which format nothing
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
@@ -151,12 +150,6 @@ ensure_cgo_deps() {
if ! pkg-config --exists vips; then
pkg_install vips libvips-dev vips vips-dev
fi
# libvips' JPEG XL loader and saver are in the nix and brew vips
# packages, and in apt's from Debian 12 and Ubuntu 24.04 on, but in a
# package of their own on Alpine.
if command -v apk >/dev/null 2>&1 && ! apk info -e vips-jxl >/dev/null; then
apk add --no-cache vips-jxl
fi
if ! pkg-config --exists libheif; then
pkg_install libheif libheif-dev libheif libheif-dev
fi