Give the largest measured per-image memory in "Running under upaas"
check / check (push) Waiting to run
check / check (push) Waiting to run
The per-image figure is now the largest peak measured over one 8192x8192 source of each source format, with 16-bit PNG, 12-bit AVIF, and 16-bit and 32-bit floating-point JPEG XL sources too, each saved at full size to every output format with pixad's libvips settings: 4.2 GiB, for the 16-bit JPEG XL saved as WebP. Temporary files says that for a GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory as well as written to /tmp. Model: opus-5-5
This commit is contained in:
@@ -130,25 +130,30 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
|
|||||||
the cache can grow to three quarters or more of the space they all have
|
the cache can grow to three quarters or more of the space they all have
|
||||||
left, and to all of it when that is 500 MiB or less.
|
left, and to all of it when that is 500 MiB or less.
|
||||||
- the rest are in the table under Configuration below
|
- the rest are in the table under Configuration below
|
||||||
- **Memory:** pixa can use up to `max_concurrent_processing` times 1.6 GiB for
|
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
|
||||||
the images it processes at once, plus `upstream_connections` times
|
the images it processes at once, plus `upstream_connections` times
|
||||||
`upstream_max_response_size` for the source images it has fetched that wait
|
`upstream_max_response_size` for the source images it has fetched that wait
|
||||||
for processing. 1.6 GiB is the most pixad's resident memory was measured to
|
for processing. 4.2 GiB is the most resident memory was measured to grow while
|
||||||
grow while it processed one 8192x8192 source, the largest it accepts, into
|
pixa, with pixad's libvips settings, processed one 8192x8192 source, the
|
||||||
each output format in turn, one at a time; it grew most for a 40 MB JPEG of
|
largest it accepts, saving it at full size to each output format in turn, one
|
||||||
random pixels saved as AVIF. At the defaults that is half the CPUs pixa sees
|
at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG,
|
||||||
times 1.6 GiB, plus 64 times 50 MiB: on a host where pixa sees 48 CPUs, 24
|
an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point
|
||||||
times 1.6 GiB plus 3.1 GiB, about 42 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING`
|
JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as
|
||||||
and `PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
|
WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64
|
||||||
|
times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
|
||||||
|
GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
|
||||||
|
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
|
||||||
use.
|
use.
|
||||||
- **Temporary files:** while pixa processes a large source, libvips writes its
|
- **Temporary files:** while pixa processes a large source, libvips writes its
|
||||||
decoded pixels to a file in `/tmp` and deletes the file when the image is
|
decoded pixels to a file in `/tmp` and deletes the file when the image is
|
||||||
done. For an 8192x8192 source the file was measured on pixad at 192 MiB for an
|
done. For a GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in
|
||||||
8-bit RGB JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG
|
memory as well as written to that file, and the 4.2 GiB above includes that
|
||||||
XL of 32-bit floating-point RGBA, the largest measured: up to 1 GiB per image
|
memory. For an 8192x8192 source the file was measured on pixad at 192 MiB for
|
||||||
processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs.
|
an 8-bit RGB JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a
|
||||||
This `/tmp` is in the container's own filesystem, not on the volume at
|
JPEG XL of 32-bit floating-point RGBA, the largest measured: up to 1 GiB per
|
||||||
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 1.6 GiB above
|
image processed at once, 24 GiB at the defaults on a host where pixa sees 48
|
||||||
|
CPUs. This `/tmp` is in the container's own filesystem, not on the volume at
|
||||||
|
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
|
||||||
does not include these files; where `/tmp` is a tmpfs, they are memory on top
|
does not include these files; where `/tmp` is a tmpfs, they are memory on top
|
||||||
of it.
|
of it.
|
||||||
- **Health check:** the image's `HEALTHCHECK` requests
|
- **Health check:** the image's `HEALTHCHECK` requests
|
||||||
|
|||||||
@@ -30,6 +30,18 @@ P2: security: per-origin rate limiting
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
|
||||||
|
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
|
||||||
|
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
|
||||||
|
host pixa processed 48 images at once, each of which can take about 4.2 GiB.
|
||||||
|
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
|
||||||
|
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
|
||||||
|
"Running under upaas" in `README.md` gives the worst-case memory, measured,
|
||||||
|
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
|
||||||
|
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
|
||||||
|
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
|
||||||
|
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
|
||||||
|
`/tmp` is a tmpfs.
|
||||||
- 2026-10-15 a cached file left empty or partial is no longer served, or fetched
|
- 2026-10-15 a cached file left empty or partial is no longer served, or fetched
|
||||||
again on every request (closes #252). Source and transformed image files, and
|
again on every request (closes #252). Source and transformed image files, and
|
||||||
each transformed image's `.meta` file, are synced to disk before they are
|
each transformed image's `.meta` file, are synced to disk before they are
|
||||||
@@ -76,18 +88,6 @@ P2: security: per-origin rate limiting
|
|||||||
405, or a CORS preflight. The metrics library takes the label before the
|
405, or a CORS preflight. The metrics library takes the label before the
|
||||||
request is routed, so the metrics middleware asks the router which route
|
request is routed, so the metrics middleware asks the router which route
|
||||||
matches.
|
matches.
|
||||||
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
|
|
||||||
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
|
|
||||||
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
|
|
||||||
host pixa processed 48 images at once, each of which can take about 1.6 GiB.
|
|
||||||
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
|
|
||||||
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
|
|
||||||
"Running under upaas" in `README.md` gives the worst-case memory, measured,
|
|
||||||
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
|
|
||||||
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
|
|
||||||
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
|
|
||||||
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
|
|
||||||
`/tmp` is a tmpfs.
|
|
||||||
- 2026-10-15 expired negative cache rows are removed on a schedule (closes
|
- 2026-10-15 expired negative cache rows are removed on a schedule (closes
|
||||||
#244): the row of a failed fetch, which makes pixa answer 502 for that source
|
#244): the row of a failed fetch, which makes pixa answer 502 for that source
|
||||||
URL for 5 minutes, was deleted only when the same URL was requested again
|
URL for 5 minutes, was deleted only when the same URL was requested again
|
||||||
|
|||||||
Reference in New Issue
Block a user