Give the largest measured per-image memory in "Running under upaas"

The per-image figure is now the largest peak measured over one
8192x8192 source of each source format, with 16-bit PNG, 12-bit AVIF,
and 16-bit and 32-bit floating-point JPEG XL sources too, each saved at
full size to every output format with pixad's libvips settings: 4.2 GiB,
for the 16-bit JPEG XL saved as WebP. Temporary files says that for a
GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory
as well as written to /tmp.

Model: opus-5-5
This commit is contained in:
2026-10-15 19:15:46 +00:00
parent db3d875251
commit e32ad30afd
2 changed files with 31 additions and 26 deletions
+19 -14
View File
@@ -131,25 +131,30 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
the cache can grow to three quarters or more of the space they all have the cache can grow to three quarters or more of the space they all have
left, and to all of it when that is 500 MiB or less. left, and to all of it when that is 500 MiB or less.
- the rest are in the table under Configuration below - the rest are in the table under Configuration below
- **Memory:** pixa can use up to `max_concurrent_processing` times 1.6 GiB for - **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
the images it processes at once, plus `upstream_connections` times the images it processes at once, plus `upstream_connections` times
`upstream_max_response_size` for the source images it has fetched that wait `upstream_max_response_size` for the source images it has fetched that wait
for processing. 1.6 GiB is the most pixad's resident memory was measured to for processing. 4.2 GiB is the most resident memory was measured to grow while
grow while it processed one 8192x8192 source, the largest it accepts, into pixa, with pixad's libvips settings, processed one 8192x8192 source, the
each output format in turn, one at a time; it grew most for a 40 MB JPEG of largest it accepts, saving it at full size to each output format in turn, one
random pixels saved as AVIF. At the defaults that is half the CPUs pixa sees at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG,
times 1.6 GiB, plus 64 times 50 MiB: on a host where pixa sees 48 CPUs, 24 an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point
times 1.6 GiB plus 3.1 GiB, about 42 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as
and `PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64
times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
use. use.
- **Temporary files:** while pixa processes a large source, libvips writes its - **Temporary files:** while pixa processes a large source, libvips writes its
decoded pixels to a file in `/tmp` and deletes the file when the image is decoded pixels to a file in `/tmp` and deletes the file when the image is
done. For an 8192x8192 source the file was measured on pixad at 192 MiB for an done. For a GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in
8-bit RGB JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG memory as well as written to that file, and the 4.2 GiB above includes that
XL of 32-bit floating-point RGBA, the largest measured: up to 1 GiB per image memory. For an 8192x8192 source the file was measured on pixad at 192 MiB for
processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs. an 8-bit RGB JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a
This `/tmp` is in the container's own filesystem, not on the volume at JPEG XL of 32-bit floating-point RGBA, the largest measured: up to 1 GiB per
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 1.6 GiB above image processed at once, 24 GiB at the defaults on a host where pixa sees 48
CPUs. This `/tmp` is in the container's own filesystem, not on the volume at
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
does not include these files; where `/tmp` is a tmpfs, they are memory on top does not include these files; where `/tmp` is a tmpfs, they are memory on top
of it. of it.
- **Health check:** the image's `HEALTHCHECK` requests - **Health check:** the image's `HEALTHCHECK` requests
+12 -12
View File
@@ -47,6 +47,18 @@ P2: security: per-origin rate limiting
the cookie, as anyone can take a CSRF token from the login page. A logout that the cookie, as anyone can take a CSRF token from the login page. A logout that
cannot be saved answers 500. `session.NewManager`, which the tests use, keeps cannot be saved answers 500. `session.NewManager`, which the tests use, keeps
the value in memory only; pixad uses `session.NewManagerWithDatabase`. the value in memory only; pixad uses `session.NewManagerWithDatabase`.
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
host pixa processed 48 images at once, each of which can take about 4.2 GiB.
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
"Running under upaas" in `README.md` gives the worst-case memory, measured,
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
`/tmp` is a tmpfs.
- 2026-10-15 a cached file left empty or partial is no longer served, or fetched - 2026-10-15 a cached file left empty or partial is no longer served, or fetched
again on every request (closes #252). Source and transformed image files, and again on every request (closes #252). Source and transformed image files, and
each transformed image's `.meta` file, are synced to disk before they are each transformed image's `.meta` file, are synced to disk before they are
@@ -93,18 +105,6 @@ P2: security: per-origin rate limiting
405, or a CORS preflight. The metrics library takes the label before the 405, or a CORS preflight. The metrics library takes the label before the
request is routed, so the metrics middleware asks the router which route request is routed, so the metrics middleware asks the router which route
matches. matches.
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
host pixa processed 48 images at once, each of which can take about 1.6 GiB.
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
"Running under upaas" in `README.md` gives the worst-case memory, measured,
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
`/tmp` is a tmpfs.
- 2026-10-15 expired negative cache rows are removed on a schedule (closes - 2026-10-15 expired negative cache rows are removed on a schedule (closes
#244): the row of a failed fetch, which makes pixa answer 502 for that source #244): the row of a failed fetch, which makes pixa answer 502 for that source
URL for 5 minutes, was deleted only when the same URL was requested again URL for 5 minutes, was deleted only when the same URL was requested again