Give the largest measured per-image memory in "Running under upaas"
check / check (push) Waiting to run

The per-image figure is now the largest peak measured over one
8192x8192 source of each source format, with 16-bit PNG, 12-bit AVIF,
and 16-bit and 32-bit floating-point JPEG XL sources too, each saved at
full size to every output format with pixad's libvips settings: 4.2 GiB,
for the 16-bit JPEG XL saved as WebP. Temporary files says that for a
GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory
as well as written to /tmp.

Model: opus-5-5
This commit is contained in:
2026-10-15 17:44:29 +00:00
parent 0abbeaf215
commit d9febdd87c
2 changed files with 31 additions and 26 deletions
+19 -14
View File
@@ -130,25 +130,30 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
the cache can grow to three quarters or more of the space they all have
left, and to all of it when that is 500 MiB or less.
- the rest are in the table under Configuration below
- **Memory:** pixa can use up to `max_concurrent_processing` times 1.6 GiB for
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
the images it processes at once, plus `upstream_connections` times
`upstream_max_response_size` for the source images it has fetched that wait
for processing. 1.6 GiB is the most pixad's resident memory was measured to
grow while it processed one 8192x8192 source, the largest it accepts, into
each output format in turn, one at a time; it grew most for a 40 MB JPEG of
random pixels saved as AVIF. At the defaults that is half the CPUs pixa sees
times 1.6 GiB, plus 64 times 50 MiB: on a host where pixa sees 48 CPUs, 24
times 1.6 GiB plus 3.1 GiB, about 42 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING`
and `PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
for processing. 4.2 GiB is the most resident memory was measured to grow while
pixa, with pixad's libvips settings, processed one 8192x8192 source, the
largest it accepts, saving it at full size to each output format in turn, one
at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG,
an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point
JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as
WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64
times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
use.
- **Temporary files:** while pixa processes a large source, libvips writes its
decoded pixels to a file in `/tmp` and deletes the file when the image is
done. For an 8192x8192 source the file was measured on pixad at 192 MiB for an
8-bit RGB JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG
XL of 32-bit floating-point RGBA, the largest measured: up to 1 GiB per image
processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs.
This `/tmp` is in the container's own filesystem, not on the volume at
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 1.6 GiB above
done. For a GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in
memory as well as written to that file, and the 4.2 GiB above includes that
memory. For an 8192x8192 source the file was measured on pixad at 192 MiB for
an 8-bit RGB JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a
JPEG XL of 32-bit floating-point RGBA, the largest measured: up to 1 GiB per
image processed at once, 24 GiB at the defaults on a host where pixa sees 48
CPUs. This `/tmp` is in the container's own filesystem, not on the volume at
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
does not include these files; where `/tmp` is a tmpfs, they are memory on top
of it.
- **Health check:** the image's `HEALTHCHECK` requests
+12 -12
View File
@@ -30,6 +30,18 @@ P2: security: per-origin rate limiting
# Completed Steps
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
host pixa processed 48 images at once, each of which can take about 4.2 GiB.
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
"Running under upaas" in `README.md` gives the worst-case memory, measured,
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
`/tmp` is a tmpfs.
- 2026-10-15 a cached file left empty or partial is no longer served, or fetched
again on every request (closes #252). Source and transformed image files, and
each transformed image's `.meta` file, are synced to disk before they are
@@ -76,18 +88,6 @@ P2: security: per-origin rate limiting
405, or a CORS preflight. The metrics library takes the label before the
request is routed, so the metrics middleware asks the router which route
matches.
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
host pixa processed 48 images at once, each of which can take about 1.6 GiB.
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
"Running under upaas" in `README.md` gives the worst-case memory, measured,
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
`/tmp` is a tmpfs.
- 2026-10-15 expired negative cache rows are removed on a schedule (closes
#244): the row of a failed fetch, which makes pixa answer 502 for that source
URL for 5 minutes, was deleted only when the same URL was requested again