Default max_concurrent_processing to half the CPUs pixa sees (closes #242) #261

Merged
clawbot merged 10 commits from issue-242-default-concurrency into next 2026-10-15 23:07:23 +02:00
10 Commits
Author SHA1 Message Date
sneak a9e4d76b5f Keep this issue's TODO.md entry at the top of Completed Steps
check / check (push) Waiting to run
The rebase onto next put the newer entry above it.

Model: opus-5-5
2026-10-15 20:41:27 +00:00
sneak cead3d57bf Say the libvips temporary file leaves /tmp but keeps its space
libvips removes the file's name from /tmp as soon as it has written the
decoded pixels, and the file keeps its space, on disk or in memory where
/tmp is a tmpfs, until the image is done. "Running under upaas" said the
file is deleted when the image is done, so an operator looking in /tmp
for it would find nothing.

Model: opus-5-5
2026-10-15 20:41:02 +00:00
sneak b12cb4565c Give the largest measured per-image memory in "Running under upaas"
The per-image figure is now the largest peak measured over one
8192x8192 source of each source format, with 16-bit PNG, 12-bit AVIF,
and 16-bit and 32-bit floating-point JPEG XL sources too, each saved at
full size to every output format with pixad's libvips settings: 4.2 GiB,
for the 16-bit JPEG XL saved as WebP. Temporary files says that for a
GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory
as well as written to /tmp.

Model: opus-5-5
2026-10-15 20:41:02 +00:00
sneak 80e336c937 Say half the CPUs plainly in DefaultMaxConcurrentProcessing
The constant cpusPerImage gave a reason nothing else states, that each
image uses two CPUs. The function now divides by 2, with the nolint
comment the repo uses for such numbers.

Model: opus-5-5
2026-10-15 20:41:02 +00:00
sneak 64d96aab87 Give the size of libvips' temporary file in "Running under upaas"
While pixa processes a large source, libvips writes its decoded pixels
to a file in /tmp, outside /var/lib/pixa and cache_max_bytes, and not
in the 1.6 GiB of resident memory. Measured on pixad for 8192x8192
sources, it is 192 MiB for an 8-bit RGB JPEG, 512 MiB for 16-bit RGBA,
and 1 GiB for a floating-point RGBA JPEG XL. The section now says so,
and that the file is memory where /tmp is a tmpfs.

Model: opus-5-5
2026-10-15 20:41:02 +00:00
sneak e8fdd49c80 Claim only one libvips worker thread per image in initVips
The comment said the images processed at once use no more CPUs than
max_concurrent_processing, which the review found does not hold for GIF
output. It now says only that libvips runs one worker thread per image.

Model: opus-5-5
2026-10-15 20:41:02 +00:00
sneak 5b89f928d7 Define the default max_concurrent_processing in internal/config
The default is configuration, so it now sits with the other defaults as
config.DefaultMaxConcurrentProcessing, and imageprocessor.New reads it
from there for a value below 1. internal/config no longer imports
internal/imageprocessor, so the packages that import internal/config do
not build against libvips.

Model: opus-5-5
2026-10-15 20:41:02 +00:00
sneak ebdecf9aba Default max_concurrent_processing to half the CPUs pixa sees (closes #242)
The default was one image per CPU Go uses, and a container with no CPU
limit sees every CPU of the host. It is now half of runtime.GOMAXPROCS(0),
rounded down and at least 1, from imageprocessor.DefaultMaxConcurrentProcessing,
which both the config and imageprocessor.New use. The two existing tests of
the default expect the new one, as the owner allowed on the issue.

"Running under upaas" gives the worst-case memory, with a per-image peak of
about 1.6 GiB measured on pixad for an 8192x8192 source in each output
format, and tells the operator to set PIXA_MAX_CONCURRENT_PROCESSING and
PIXA_UPSTREAM_CONNECTIONS to fit the container's memory.

Model: opus-5-5
2026-10-15 20:41:02 +00:00
sneak de7ecfdb5e Test that max_concurrent_processing defaults to half the CPUs
An omitted max_concurrent_processing, and New given none, must be half
the CPUs Go uses, rounded down, and at least 1. The tests set GOMAXPROCS
to 1, 2, 3 and 8 and fail until the default changes
(#242).

Model: opus-5-5
2026-10-15 20:41:02 +00:00
sneak ae7564b769 Test that an invalid PIXA_MAX_CONCURRENT_PROCESSING aborts startup
An empty, fractional, negative or non-numeric value must abort startup
naming the variable and the key, never fall back to the default. The
config loader already refuses these; this pins it before the default
changes (#242).

Model: opus-5-5
2026-10-15 20:41:02 +00:00