From d9febdd87c9f419d0b505886acceff09bf1335e5 Mon Sep 17 00:00:00 2001 From: sneak Date: Thu, 15 Oct 2026 17:19:47 +0000 Subject: [PATCH] Give the largest measured per-image memory in "Running under upaas" The per-image figure is now the largest peak measured over one 8192x8192 source of each source format, with 16-bit PNG, 12-bit AVIF, and 16-bit and 32-bit floating-point JPEG XL sources too, each saved at full size to every output format with pixad's libvips settings: 4.2 GiB, for the 16-bit JPEG XL saved as WebP. Temporary files says that for a GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory as well as written to /tmp. Model: opus-5-5 --- README.md | 33 +++++++++++++++++++-------------- TODO.md | 24 ++++++++++++------------ 2 files changed, 31 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index ff0d577..bf70fe0 100644 --- a/README.md +++ b/README.md @@ -130,25 +130,30 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs: the cache can grow to three quarters or more of the space they all have left, and to all of it when that is 500 MiB or less. - the rest are in the table under Configuration below -- **Memory:** pixa can use up to `max_concurrent_processing` times 1.6 GiB for +- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for the images it processes at once, plus `upstream_connections` times `upstream_max_response_size` for the source images it has fetched that wait - for processing. 1.6 GiB is the most pixad's resident memory was measured to - grow while it processed one 8192x8192 source, the largest it accepts, into - each output format in turn, one at a time; it grew most for a 40 MB JPEG of - random pixels saved as AVIF. At the defaults that is half the CPUs pixa sees - times 1.6 GiB, plus 64 times 50 MiB: on a host where pixa sees 48 CPUs, 24 - times 1.6 GiB plus 3.1 GiB, about 42 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` - and `PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may + for processing. 4.2 GiB is the most resident memory was measured to grow while + pixa, with pixad's libvips settings, processed one 8192x8192 source, the + largest it accepts, saving it at full size to each output format in turn, one + at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG, + an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point + JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as + WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64 + times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1 + GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and + `PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may use. - **Temporary files:** while pixa processes a large source, libvips writes its decoded pixels to a file in `/tmp` and deletes the file when the image is - done. For an 8192x8192 source the file was measured on pixad at 192 MiB for an - 8-bit RGB JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG - XL of 32-bit floating-point RGBA, the largest measured: up to 1 GiB per image - processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs. - This `/tmp` is in the container's own filesystem, not on the volume at - `/var/lib/pixa`, and `cache_max_bytes` does not count it. The 1.6 GiB above + done. For a GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in + memory as well as written to that file, and the 4.2 GiB above includes that + memory. For an 8192x8192 source the file was measured on pixad at 192 MiB for + an 8-bit RGB JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a + JPEG XL of 32-bit floating-point RGBA, the largest measured: up to 1 GiB per + image processed at once, 24 GiB at the defaults on a host where pixa sees 48 + CPUs. This `/tmp` is in the container's own filesystem, not on the volume at + `/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above does not include these files; where `/tmp` is a tmpfs, they are memory on top of it. - **Health check:** the image's `HEALTHCHECK` requests diff --git a/TODO.md b/TODO.md index 578cd65..0eca9a5 100644 --- a/TODO.md +++ b/TODO.md @@ -30,6 +30,18 @@ P2: security: per-origin rate limiting # Completed Steps +- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees, + rounded down, and at least 1 (closes #242): it defaulted to one image per CPU, + and a container with no CPU limit sees every CPU of the host, so on a 48-CPU + host pixa processed 48 images at once, each of which can take about 4.2 GiB. + `config.DefaultMaxConcurrentProcessing` gives the default to the config and to + `imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it. + "Running under upaas" in `README.md` gives the worst-case memory, measured, + and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and + `PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the + file libvips writes to `/tmp` for each large source, up to 1 GiB, measured, + which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where + `/tmp` is a tmpfs. - 2026-10-15 a cached file left empty or partial is no longer served, or fetched again on every request (closes #252). Source and transformed image files, and each transformed image's `.meta` file, are synced to disk before they are @@ -76,18 +88,6 @@ P2: security: per-origin rate limiting 405, or a CORS preflight. The metrics library takes the label before the request is routed, so the metrics middleware asks the router which route matches. -- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees, - rounded down, and at least 1 (closes #242): it defaulted to one image per CPU, - and a container with no CPU limit sees every CPU of the host, so on a 48-CPU - host pixa processed 48 images at once, each of which can take about 1.6 GiB. - `config.DefaultMaxConcurrentProcessing` gives the default to the config and to - `imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it. - "Running under upaas" in `README.md` gives the worst-case memory, measured, - and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and - `PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the - file libvips writes to `/tmp` for each large source, up to 1 GiB, measured, - which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where - `/tmp` is a tmpfs. - 2026-10-15 expired negative cache rows are removed on a schedule (closes #244): the row of a failed fetch, which makes pixa answer 502 for that source URL for 5 minutes, was deleted only when the same URL was requested again