Bound concurrent image processing and upstream fetches (closes #64)

max_concurrent_processing (default: the number of CPUs Go uses) bounds
the images processed at once, and upstream_connections (default 64) the
fetches from all upstream hosts together, beside the per-host limit. A
request that finds either full waits up to 10 seconds, then gets 503
"server busy, try again later". The processor holds its slot from before
it reads the input until it returns, and takes a free slot even after the
request context has ended; a fetch holds its connection until the
response body is closed, after its image is processed. libvips now starts
with one worker thread per image and no operation cache. Both settings
have PIXA_ variables and are in README.md and config.example.yml.

Model: opus-5-5
This commit is contained in:
2026-09-29 06:10:38 +00:00
parent 4a27ace44f
commit 070718a80b
10 changed files with 262 additions and 49 deletions
+10 -3
View File
@@ -25,11 +25,19 @@ The disk cache is now size-bounded with LRU eviction
# Next Step
P1: rate limit global concurrent upstream fetches to prevent resource
exhaustion
P2: security: referer blacklist
# Completed Steps
- 2026-09-29 bound concurrent image processing and upstream fetches (closes
#64): `max_concurrent_processing` (default the number of CPUs pixa can use)
limits the images decoded and encoded at once, and `upstream_connections`
(default 64) the connections to all upstream hosts together, on top of
`upstream_connections_per_host`; a fetch holds its connection until its image
has been processed; a request that finds either limit reached waits up to 10
seconds for a free one, then gets 503 `server busy, try again later`; libvips
runs one worker thread per image with its operation cache off; documented in
`README.md` and `config.example.yml`.
- 2026-09-29 migrations at the path `REPO_POLICIES.md` sets (closes #96): the
migration files moved, contents unchanged, from `internal/database/schema/`
to `internal/db/migrations/` as `000_migration.sql` and `001_schema.sql`; the
@@ -297,7 +305,6 @@ exhaustion
# Future Steps
- P2: security
- referer blacklist
- per-IP rate limiting on the image routes
- per-origin rate limiting
- P2: HTTP response handling