Queue SQLite writes on one connection so none fails as locked (closes #223) #225
@@ -529,10 +529,13 @@ Key settings in more detail:
|
||||
- `signing_key` — HMAC secret for URL signatures
|
||||
- `db_url` — the SQLite database to open; omitted, it is
|
||||
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
|
||||
database in WAL mode. pixa adds `_pragma=busy_timeout(5000)` to any `db_url`,
|
||||
so a write that finds another in progress waits up to five seconds for it
|
||||
instead of failing. WAL mode comes only from the URL: keep
|
||||
`_pragma=journal_mode(WAL)` in one you set
|
||||
database in WAL mode. pixa opens one connection to it, so its own reads and
|
||||
writes run one at a time. Requests wait while eviction runs one of its
|
||||
queries, some of which read a whole table. pixa adds
|
||||
`_pragma=busy_timeout(5000)` to any `db_url`, so a write that finds another
|
||||
program writing to the file waits up to five seconds for it instead of
|
||||
failing. WAL mode comes only from the URL: keep `_pragma=journal_mode(WAL)` in
|
||||
one you set
|
||||
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
|
||||
cache entirely; omitted defaults to 75% of the sum of the free space on the
|
||||
filesystem containing `<state_dir>/cache/` and the bytes of source and
|
||||
|
||||
@@ -30,6 +30,13 @@ P2: security: per-IP rate limiting on the image routes
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-08 SQLite writes no longer fail with "database is locked" under load
|
||||
(closes #223): `internal/database` opens the database with one connection, so
|
||||
pixa's own reads and writes run on it one at a time instead of competing for
|
||||
SQLite's lock, where a write that kept losing could wait past the five-second
|
||||
busy timeout and be lost. The busy timeout stays, for another program writing
|
||||
to the same file. No code in pixa keeps rows or a transaction open while it
|
||||
runs another query, which with one connection would wait forever.
|
||||
- 2026-10-05 the format `auto` (closes #88): a format in the `/v1/image/` path,
|
||||
an encrypted URL's token and the generator page's format choice, chosen for
|
||||
each request from `Accept` once the signature or token is checked: AVIF when
|
||||
|
||||
@@ -13,11 +13,10 @@ import (
|
||||
)
|
||||
|
||||
// TestConcurrentWritesAllSucceed opens a database the way pixad does and
|
||||
// writes to it from several goroutines at once, so the writes run on
|
||||
// separate connections, as one request's writes and the background eviction
|
||||
// pass do. Every write must succeed, none failing with "database is locked",
|
||||
// whether or not db_url already has parameters, and the parameters it has
|
||||
// must still apply.
|
||||
// writes to it from several goroutines at once, as one request's writes and
|
||||
// the background eviction pass do. Every write must succeed, none failing
|
||||
// with "database is locked", whether or not db_url already has parameters,
|
||||
// and the parameters it has must still apply.
|
||||
func TestConcurrentWritesAllSucceed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -237,17 +237,18 @@ func ApplyMigrations(ctx context.Context, db *sql.DB, log *slog.Logger) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// DB returns the underlying sql.DB.
|
||||
// DB returns the underlying sql.DB. It has one connection, so close any
|
||||
// rows and end any transaction before running another query on it; a
|
||||
// query run while they are open waits forever.
|
||||
func (s *Database) DB() *sql.DB {
|
||||
return s.db
|
||||
}
|
||||
|
||||
func (s *Database) connect(ctx context.Context) error {
|
||||
// Requests and the eviction pass write on separate connections. With
|
||||
// a busy timeout, a write that finds another one in progress waits up
|
||||
// to five seconds for it instead of failing at once with "database is
|
||||
// locked". The driver runs each _pragma parameter on every connection
|
||||
// it opens.
|
||||
// With a busy timeout, a write that finds another program writing to
|
||||
// the same database file waits up to five seconds for it instead of
|
||||
// failing at once with "database is locked". The driver runs each
|
||||
// _pragma parameter on every connection it opens.
|
||||
separator := "?"
|
||||
if strings.Contains(s.config.DBURL, "?") {
|
||||
separator = "&"
|
||||
@@ -264,6 +265,11 @@ func (s *Database) connect(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// One connection: pixa's own reads and writes run on it one at a
|
||||
// time instead of competing for SQLite's lock, where a write that
|
||||
// keeps losing can wait past the busy timeout and be lost.
|
||||
db.SetMaxOpenConns(1)
|
||||
|
||||
err = db.PingContext(ctx)
|
||||
if err != nil {
|
||||
s.log.Error("failed to ping database", "error", err)
|
||||
|
||||
Reference in New Issue
Block a user