Give the largest measured per-image memory in "Running under upaas"
check / check (push) Waiting to run
check / check (push) Waiting to run
The per-image figure is now the largest peak measured over one 8192x8192 source of each source format, with 16-bit PNG, 12-bit AVIF, and 16-bit and 32-bit floating-point JPEG XL sources too, each saved at full size to every output format with pixad's libvips settings: 4.2 GiB, for the 16-bit JPEG XL saved as WebP. Temporary files says that for a GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory as well as written to /tmp. Model: opus-5-5
This commit is contained in:
@@ -130,25 +130,30 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
|
||||
the cache can grow to three quarters or more of the space they all have
|
||||
left, and to all of it when that is 500 MiB or less.
|
||||
- the rest are in the table under Configuration below
|
||||
- **Memory:** pixa can use up to `max_concurrent_processing` times 1.6 GiB for
|
||||
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
|
||||
the images it processes at once, plus `upstream_connections` times
|
||||
`upstream_max_response_size` for the source images it has fetched that wait
|
||||
for processing. 1.6 GiB is the most pixad's resident memory was measured to
|
||||
grow while it processed one 8192x8192 source, the largest it accepts, into
|
||||
each output format in turn, one at a time; it grew most for a 40 MB JPEG of
|
||||
random pixels saved as AVIF. At the defaults that is half the CPUs pixa sees
|
||||
times 1.6 GiB, plus 64 times 50 MiB: on a host where pixa sees 48 CPUs, 24
|
||||
times 1.6 GiB plus 3.1 GiB, about 42 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING`
|
||||
and `PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
|
||||
for processing. 4.2 GiB is the most resident memory was measured to grow while
|
||||
pixa, with pixad's libvips settings, processed one 8192x8192 source, the
|
||||
largest it accepts, saving it at full size to each output format in turn, one
|
||||
at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG,
|
||||
an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point
|
||||
JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as
|
||||
WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64
|
||||
times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
|
||||
GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
|
||||
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
|
||||
use.
|
||||
- **Temporary files:** while pixa processes a large source, libvips writes its
|
||||
decoded pixels to a file in `/tmp` and deletes the file when the image is
|
||||
done. For an 8192x8192 source the file was measured on pixad at 192 MiB for an
|
||||
8-bit RGB JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG
|
||||
XL of 32-bit floating-point RGBA, the largest measured: up to 1 GiB per image
|
||||
processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs.
|
||||
This `/tmp` is in the container's own filesystem, not on the volume at
|
||||
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 1.6 GiB above
|
||||
done. For a GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in
|
||||
memory as well as written to that file, and the 4.2 GiB above includes that
|
||||
memory. For an 8192x8192 source the file was measured on pixad at 192 MiB for
|
||||
an 8-bit RGB JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a
|
||||
JPEG XL of 32-bit floating-point RGBA, the largest measured: up to 1 GiB per
|
||||
image processed at once, 24 GiB at the defaults on a host where pixa sees 48
|
||||
CPUs. This `/tmp` is in the container's own filesystem, not on the volume at
|
||||
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
|
||||
does not include these files; where `/tmp` is a tmpfs, they are memory on top
|
||||
of it.
|
||||
- **Health check:** the image's `HEALTHCHECK` requests
|
||||
|
||||
@@ -30,6 +30,18 @@ P2: security: per-origin rate limiting
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
|
||||
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
|
||||
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
|
||||
host pixa processed 48 images at once, each of which can take about 4.2 GiB.
|
||||
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
|
||||
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
|
||||
"Running under upaas" in `README.md` gives the worst-case memory, measured,
|
||||
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
|
||||
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
|
||||
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
|
||||
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
|
||||
`/tmp` is a tmpfs.
|
||||
- 2026-10-15 a cached file left empty or partial is no longer served, or fetched
|
||||
again on every request (closes #252). Source and transformed image files, and
|
||||
each transformed image's `.meta` file, are synced to disk before they are
|
||||
@@ -76,18 +88,6 @@ P2: security: per-origin rate limiting
|
||||
405, or a CORS preflight. The metrics library takes the label before the
|
||||
request is routed, so the metrics middleware asks the router which route
|
||||
matches.
|
||||
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
|
||||
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
|
||||
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
|
||||
host pixa processed 48 images at once, each of which can take about 1.6 GiB.
|
||||
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
|
||||
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
|
||||
"Running under upaas" in `README.md` gives the worst-case memory, measured,
|
||||
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
|
||||
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
|
||||
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
|
||||
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
|
||||
`/tmp` is a tmpfs.
|
||||
- 2026-10-15 expired negative cache rows are removed on a schedule (closes
|
||||
#244): the row of a failed fetch, which makes pixa answer 502 for that source
|
||||
URL for 5 minutes, was deleted only when the same URL was requested again
|
||||
|
||||
Reference in New Issue
Block a user