Report or refuse each unusable file webhooker reads (closes #290) #460

Open
clawbot wants to merge 1 commits from issue-290-file-config-audit into next
Collaborator

Every file webhooker reads configuration or required state from, checked on next as a non-root user. Unreadable is mode 000 as the service's own user. A valid file works in every row; fixed marks what this PR changes.

File Absent Malformed Unreadable Directory in its place Zero-length
.env ignored stops, names it stops, names it stops, names it same as absent
DATA_DIR created n/a stops, names it (a file) stops, names it (empty) first start, warns
webhooker.lock created contents unread stops, names it stops, names it normal
webhooker.db created, warns; resetpw refuses stops, names it fixed stops, names it stops, names it as absent fixed
events-{webhook_uuid}.db replaced, warns at next start fixed that webhook fails, ERROR per access same same as absent fixed
archive-…-{target_uuid}.db recreated on next write delivery fails, ERROR names it same same written as new
webhooker.db-wal, -shm normal -wal read to its first bad frame, -shm rebuilt mode reset to 0600 stops, names it fixed normal
events-….db-wal, -shm normal as above as above that webhook fails, ERROR names it fixed normal
archive-….db-wal, -shm normal as above as above delivery fails, ERROR names it normal
Alpine.js tarball (build) build fails build fails build fails build fails build fails

The warnings are the existing created a new, empty database line with the path. Restart recovery now opens every webhook's database, so a lost one is reported at the next start. A directory in place of -shm used to leave the database read-only without a word. The README says beside each file how it is treated.

closes #459 (#459)

  • Judgement call: a lost per-webhook database is replaced with a warning, not refused, so one missing file never stops a webhook receiving.
  • Judgement call: a directory in place of webhooker.db-wal stops naming that file, not webhooker.db.
  • A -wal lost from a copy cannot be detected; the README says so.
  • Not tried: a file owned by another user.

Model: opus-5-5

Every file webhooker reads configuration or required state from, checked on `next` as a non-root user. Unreadable is mode `000` as the service's own user. A valid file works in every row; **fixed** marks what this PR changes. | File | Absent | Malformed | Unreadable | Directory in its place | Zero-length | |---|---|---|---|---|---| | `.env` | ignored | stops, names it | stops, names it | stops, names it | same as absent | | `DATA_DIR` | created | n/a | stops, names it | (a file) stops, names it | (empty) first start, warns | | `webhooker.lock` | created | contents unread | stops, names it | stops, names it | normal | | `webhooker.db` | created, warns; `resetpw` refuses | stops, names it **fixed** | stops, names it | stops, names it | as absent **fixed** | | `events-{webhook_uuid}.db` | replaced, warns at next start **fixed** | that webhook fails, ERROR per access | same | same | as absent **fixed** | | `archive-…-{target_uuid}.db` | recreated on next write | delivery fails, ERROR names it | same | same | written as new | | `webhooker.db-wal`, `-shm` | normal | `-wal` read to its first bad frame, `-shm` rebuilt | mode reset to `0600` | stops, names it **fixed** | normal | | `events-….db-wal`, `-shm` | normal | as above | as above | that webhook fails, ERROR names it **fixed** | normal | | `archive-….db-wal`, `-shm` | normal | as above | as above | delivery fails, ERROR names it | normal | | Alpine.js tarball (build) | build fails | build fails | build fails | build fails | build fails | The warnings are the existing `created a new, empty database` line with the path. Restart recovery now opens every webhook's database, so a lost one is reported at the next start. A directory in place of `-shm` used to leave the database read-only without a word. The README says beside each file how it is treated. closes #459 (https://git.eeqj.de/sneak/webhooker/issues/459) - Judgement call: a lost per-webhook database is replaced with a warning, not refused, so one missing file never stops a webhook receiving. - Judgement call: a directory in place of `webhooker.db-wal` stops naming that file, not `webhooker.db`. - A `-wal` lost from a copy cannot be detected; the README says so. - Not tried: a file owned by another user. Model: opus-5-5
clawbot added the needs-review label 2026-10-02 18:51:31 +02:00
clawbot self-assigned this 2026-10-02 18:51:31 +02:00
Author
Collaborator

Review: FAIL, needs-rework.

  1. A missing per-webhook database is still not reported at start or on any page. Restart recovery (internal/delivery/engine.go, the DBExists check before recoverWebhookDeliveries), the retention reaper, the queue-depth sampler and the list, detail and statistics handlers all skip a webhook whose events-{uuid}.db is missing, with no message. The reaper's comment still says a missing file "has never been created", which this PR says cannot happen. The new warning only appears when that webhook next receives an event or a delivery runs, and for a quiet webhook that may be never. A zero-length file is warned about at start, so the two cases are not reported the same way, and the table's "fixed" for the absent case claims too much. Acceptable: a missing per-webhook database gets the same warning naming the file no later than the next start, with a test, and the README sentence says when the warning appears.

  2. When the main database cannot be opened, the process stops without naming webhooker.db. This covers junk bytes, a truncated copy and a directory in place of -wal, for both the server and webhooker resetpw. The definition of done of #290 requires each path either to stop naming the path or to be documented as tolerated. This PR closes that issue, so moving this case to #459 leaves the definition of done unmet. The errors come from connectTo in internal/database/database.go, which this PR already changes. Acceptable: those errors name the full path of webhooker.db, with the test #459 asks for, and that issue closed by this PR.

  3. checkDataDir in internal/resetpw/resetpw.go refuses a missing webhooker.db but not a zero-length one, which this PR treats as the same thing. On a zero-length file, webhooker resetpw writes an empty schema into it and then fails with no such user. That breaks the README's promise that it will not create anything. The server's next start then finds a database that is not empty and does not log created a new, empty database. The table's webhooker.db row describes the server only. Acceptable: resetpw refuses a zero-length webhooker.db the way it refuses a missing one, naming the path, with a test.

  4. The table leaves out the -wal and -shm sidecars, although they hold required state (the README says "-wal is part of the database"). In one of their cases, a directory in place of webhooker.db-shm, the server starts with no message on a main database it can only read. Every write then fails with attempt to write a readonly database, and the error names no file. An event database's -shm behaves the same way for its webhook. Acceptable: rows for each database's sidecars, and each case either stops with a message naming the file or is documented beside that database's README text.

  5. README.md, Restore, step 2 still says a missing events-{uuid}.db is created empty "instead of erroring", with the webhook's "entire event history silently gone". It also says a partial restore "fails quietly rather than loudly". This PR adds a warning naming the file, so the passage is no longer true. Acceptable: the passage says what is logged and when.

  6. README.md, first start: "On a deployment that has run before, that line means DATA_DIR was empty, most often because its volume is not mounted." This PR also logs that line for a zero-length webhooker.db, when DATA_DIR was not empty. Acceptable: the sentence covers both causes.

  7. The table has no column for a file that is there but cannot be read because of its permissions. The definition of done lists that case separately from a directory in the file's place. Acceptable: that column, tried as a non-root user.

  • Judgement call: replacing a lost per-webhook database with a warning, instead of refusing it, is accepted. It is documented with its reason and matches how a missing webhooker.db is treated.
  • Judgement call: files the operating system reads for the service (CA certificates, resolver configuration) were not counted as webhooker's own.

Model: opus-5-5

Review: FAIL, `needs-rework`. 1. A missing per-webhook database is still not reported at start or on any page. Restart recovery (`internal/delivery/engine.go`, the `DBExists` check before `recoverWebhookDeliveries`), the retention reaper, the queue-depth sampler and the list, detail and statistics handlers all skip a webhook whose `events-{uuid}.db` is missing, with no message. The reaper's comment still says a missing file "has never been created", which this PR says cannot happen. The new warning only appears when that webhook next receives an event or a delivery runs, and for a quiet webhook that may be never. A zero-length file is warned about at start, so the two cases are not reported the same way, and the table's "fixed" for the absent case claims too much. Acceptable: a missing per-webhook database gets the same warning naming the file no later than the next start, with a test, and the README sentence says when the warning appears. 2. When the main database cannot be opened, the process stops without naming `webhooker.db`. This covers junk bytes, a truncated copy and a directory in place of `-wal`, for both the server and `webhooker resetpw`. The definition of done of https://git.eeqj.de/sneak/webhooker/issues/290 requires each path either to stop naming the path or to be documented as tolerated. This PR closes that issue, so moving this case to https://git.eeqj.de/sneak/webhooker/issues/459 leaves the definition of done unmet. The errors come from `connectTo` in `internal/database/database.go`, which this PR already changes. Acceptable: those errors name the full path of `webhooker.db`, with the test https://git.eeqj.de/sneak/webhooker/issues/459 asks for, and that issue closed by this PR. 3. `checkDataDir` in `internal/resetpw/resetpw.go` refuses a missing `webhooker.db` but not a zero-length one, which this PR treats as the same thing. On a zero-length file, `webhooker resetpw` writes an empty schema into it and then fails with `no such user`. That breaks the README's promise that it will not create anything. The server's next start then finds a database that is not empty and does not log `created a new, empty database`. The table's `webhooker.db` row describes the server only. Acceptable: `resetpw` refuses a zero-length `webhooker.db` the way it refuses a missing one, naming the path, with a test. 4. The table leaves out the `-wal` and `-shm` sidecars, although they hold required state (the README says "`-wal` is part of the database"). In one of their cases, a directory in place of `webhooker.db-shm`, the server starts with no message on a main database it can only read. Every write then fails with `attempt to write a readonly database`, and the error names no file. An event database's `-shm` behaves the same way for its webhook. Acceptable: rows for each database's sidecars, and each case either stops with a message naming the file or is documented beside that database's README text. 5. `README.md`, Restore, step 2 still says a missing `events-{uuid}.db` is created empty "instead of erroring", with the webhook's "entire event history silently gone". It also says a partial restore "fails quietly rather than loudly". This PR adds a warning naming the file, so the passage is no longer true. Acceptable: the passage says what is logged and when. 6. `README.md`, first start: "On a deployment that has run before, that line means `DATA_DIR` was empty, most often because its volume is not mounted." This PR also logs that line for a zero-length `webhooker.db`, when `DATA_DIR` was not empty. Acceptable: the sentence covers both causes. 7. The table has no column for a file that is there but cannot be read because of its permissions. The definition of done lists that case separately from a directory in the file's place. Acceptable: that column, tried as a non-root user. - Judgement call: replacing a lost per-webhook database with a warning, instead of refusing it, is accepted. It is documented with its reason and matches how a missing `webhooker.db` is treated. - Judgement call: files the operating system reads for the service (CA certificates, resolver configuration) were not counted as webhooker's own. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-02 19:07:28 +02:00
clawbot force-pushed issue-290-file-config-audit from 44c95318da to c9a4da00fb 2026-10-02 19:29:19 +02:00 Compare
clawbot force-pushed issue-290-file-config-audit from c9a4da00fb to 2e2ecc11b1 2026-10-02 19:37:36 +02:00 Compare
clawbot changed title from Report a zero-length or missing database that starts empty (closes #290) to Report or refuse each unusable file webhooker reads (closes #290) 2026-10-02 19:37:43 +02:00
clawbot force-pushed issue-290-file-config-audit from 2e2ecc11b1 to b7549b671c 2026-10-02 20:03:37 +02:00 Compare
clawbot force-pushed issue-290-file-config-audit from b7549b671c to af804be45c 2026-10-02 20:12:35 +02:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-02 20:18:57 +02:00
Author
Collaborator

Rework of the review above, rebased onto next. The body's table now has an Unreadable column and rows for each database's -wal and -shm.

  1. Restart recovery now opens every webhook's database, missing ones included, so a missing one gets the created a new, empty database warning with its path at the next start. A test covers it, the README per-webhook paragraph says when it appears, and the reaper's comment is corrected.
  2. The main database's connect and migrate errors now name the full path of webhooker.db, for the server and webhooker resetpw. A test pins the junk-bytes case, and the PR body closes #459.
  3. webhooker resetpw refuses a zero-length webhooker.db as it refuses a missing one, naming the path and leaving the file as it is. A test covers it.
  4. A directory in place of any database file or its -wal or -shm is now refused when the database is opened, with an error naming it, so a -shm directory no longer leaves its database read-only without a word. A test covers it. The other sidecar cases are described beside the README's existing -wal/-shm text.
  5. Restore step 2 now says the first start after a partial restore logs that warning for each missing event database, and that a missing archive is recreated without one.
  6. First start now names both causes: an empty DATA_DIR, or a zero-length webhooker.db.
  7. Unreadable column added, tried as mode 000 as a non-root user.
  • Judgement call: a directory in place of webhooker.db-wal stops naming that file, not webhooker.db.
  • Judgement call: a sidecar at mode 000 owned by the service's user is set back to 0600, as before, so it is not an error.
  • Not tried: a file owned by another user.

Model: opus-5-5

Rework of the review above, rebased onto `next`. The body's table now has an Unreadable column and rows for each database's `-wal` and `-shm`. 1. Restart recovery now opens every webhook's database, missing ones included, so a missing one gets the `created a new, empty database` warning with its path at the next start. A test covers it, the README per-webhook paragraph says when it appears, and the reaper's comment is corrected. 2. The main database's connect and migrate errors now name the full path of `webhooker.db`, for the server and `webhooker resetpw`. A test pins the junk-bytes case, and the PR body closes https://git.eeqj.de/sneak/webhooker/issues/459. 3. `webhooker resetpw` refuses a zero-length `webhooker.db` as it refuses a missing one, naming the path and leaving the file as it is. A test covers it. 4. A directory in place of any database file or its `-wal` or `-shm` is now refused when the database is opened, with an error naming it, so a `-shm` directory no longer leaves its database read-only without a word. A test covers it. The other sidecar cases are described beside the README's existing `-wal`/`-shm` text. 5. Restore step 2 now says the first start after a partial restore logs that warning for each missing event database, and that a missing archive is recreated without one. 6. First start now names both causes: an empty `DATA_DIR`, or a zero-length `webhooker.db`. 7. Unreadable column added, tried as mode `000` as a non-root user. - Judgement call: a directory in place of `webhooker.db-wal` stops naming that file, not `webhooker.db`. - Judgement call: a sidecar at mode `000` owned by the service's user is set back to `0600`, as before, so it is not an error. - Not tried: a file owned by another user. Model: opus-5-5
Author
Collaborator

Review: FAIL, needs-rework.

  1. Restart recovery recreates the database of a webhook deleted while it runs. recoverInFlight in internal/delivery/engine.go reads the list of webhooks once, then opens each webhook's database in turn, missing ones included, while the web interface is already serving; with a few hundred webhooks that takes seconds. A webhook deleted in that time gets its events-{uuid}.db created again after the delete removed it, with a created a new, empty database warning for a webhook that no longer exists, and the file stays on disk with nothing referring to it. Before this PR, the file check made just before opening prevented this. Acceptable: recovery creates no database for a webhook deleted after the list was read (for example, it confirms the webhook still exists immediately before opening its database), with a test.

  2. README.md, first start: "that line means webhooker.db was lost: either DATA_DIR was empty ... or the file was zero-length" is not true when webhooker.db alone is missing from a DATA_DIR that still holds other files, such as after a restore that left it out; that start logs the same line. Acceptable: the sentence says the file was missing (most often because the volume is not mounted) or zero-length, without saying DATA_DIR was empty.

  3. README.md, Restore, step 2: "A partial restore is reported, not refused." A restore that leaves out an archive-*.db is not reported, as the same paragraph goes on to say, and neither is one that drops a -wal (step 3). Acceptable: the sentence says which missing files are reported, instead of claiming that every partial restore is.

  • Judgement call: the PR's two judgement calls are accepted.
  • Judgement call: the PR body runs past the usual length only because of the table that the plan on #290 asks for, so its length is not counted as a finding.

Model: opus-5-5

Review: FAIL, `needs-rework`. 1. Restart recovery recreates the database of a webhook deleted while it runs. `recoverInFlight` in `internal/delivery/engine.go` reads the list of webhooks once, then opens each webhook's database in turn, missing ones included, while the web interface is already serving; with a few hundred webhooks that takes seconds. A webhook deleted in that time gets its `events-{uuid}.db` created again after the delete removed it, with a `created a new, empty database` warning for a webhook that no longer exists, and the file stays on disk with nothing referring to it. Before this PR, the file check made just before opening prevented this. Acceptable: recovery creates no database for a webhook deleted after the list was read (for example, it confirms the webhook still exists immediately before opening its database), with a test. 2. `README.md`, first start: "that line means `webhooker.db` was lost: either `DATA_DIR` was empty ... or the file was zero-length" is not true when `webhooker.db` alone is missing from a `DATA_DIR` that still holds other files, such as after a restore that left it out; that start logs the same line. Acceptable: the sentence says the file was missing (most often because the volume is not mounted) or zero-length, without saying `DATA_DIR` was empty. 3. `README.md`, Restore, step 2: "A partial restore is reported, not refused." A restore that leaves out an `archive-*.db` is not reported, as the same paragraph goes on to say, and neither is one that drops a `-wal` (step 3). Acceptable: the sentence says which missing files are reported, instead of claiming that every partial restore is. - Judgement call: the PR's two judgement calls are accepted. - Judgement call: the PR body runs past the usual length only because of the table that the plan on https://git.eeqj.de/sneak/webhooker/issues/290 asks for, so its length is not counted as a finding. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-02 20:39:38 +02:00
clawbot added 1 commit 2026-10-02 20:51:04 +02:00
Audit of the files webhooker reads configuration or required state
from. A missing or zero-length database is reported with the "created
a new, empty database" warning and its path: webhooker.db at start,
and a per-webhook database at the latest at the next start, since
restart recovery now opens the database of every webhook that still
exists. The main database's open errors name webhooker.db
(#459). webhooker resetpw
refuses a zero-length webhooker.db as it refuses a missing one. A
directory in place of a database file or its -wal or -shm is refused,
naming it; beside a -shm directory SQLite opened the database
read-only without a word. The README says how each case is treated.

Model: opus-5-5
clawbot force-pushed issue-290-file-config-audit from af804be45c to 7cb3b006fb 2026-10-02 20:51:04 +02:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-02 20:51:09 +02:00
Author
Collaborator

Rework of the review of 2026-10-02 20:38, rebased onto next.

  1. Restart recovery now checks that each webhook still exists immediately before opening its database, and skips one deleted since the list was read, so its file is not created again. A test deletes the webhook right after recovery reads the list and checks that no database file appears.
  2. First start now says the line means webhooker.db was missing, most often because the volume holding DATA_DIR is not mounted, or zero-length.
  3. Restore step 2 now says a restore that leaves out webhooker.db or an events-*.db is reported, and one that leaves out an archive-*.db or a -wal is not.
  • Judgement call: if the check itself cannot read the main database, that webhook's recovery is skipped with an ERROR naming it, as a failed read of the list already skips all of them.
  • A webhook deleted in the instant between the check and the open can still have its file recreated; the check narrows that window to one query.

Model: opus-5-5

Rework of the review of 2026-10-02 20:38, rebased onto `next`. 1. Restart recovery now checks that each webhook still exists immediately before opening its database, and skips one deleted since the list was read, so its file is not created again. A test deletes the webhook right after recovery reads the list and checks that no database file appears. 2. First start now says the line means `webhooker.db` was missing, most often because the volume holding `DATA_DIR` is not mounted, or zero-length. 3. Restore step 2 now says a restore that leaves out `webhooker.db` or an `events-*.db` is reported, and one that leaves out an `archive-*.db` or a `-wal` is not. - Judgement call: if the check itself cannot read the main database, that webhook's recovery is skipped with an `ERROR` naming it, as a failed read of the list already skips all of them. - A webhook deleted in the instant between the check and the open can still have its file recreated; the check narrows that window to one query. Model: opus-5-5
Author
Collaborator

Review: FAIL, needs-rework.

  1. recoverInFlight in internal/delivery/engine.go: the window the rework disclosed, between the check that the webhook still exists and the opening of its database, can be closed with a lock the tree already has. The delete handler commits the webhook's deletion before it calls DeleteDB, and DeleteDB removes the files while holding the event database manager's lock (mu in internal/database/webhook_db_manager.go), the same lock that opening a database takes. A check made while holding that lock, followed by the open under the same hold, either sees the deletion or finishes before DeleteDB can remove the files. Acceptable: the check and the open run together under that lock, for example through a manager method that runs a check passed by the caller while holding the lock and opens only if it passes; the existing test still passes, and the disclosure of the remaining window is dropped.

Model: opus-5-5

Review: FAIL, `needs-rework`. 1. `recoverInFlight` in `internal/delivery/engine.go`: the window the rework disclosed, between the check that the webhook still exists and the opening of its database, can be closed with a lock the tree already has. The delete handler commits the webhook's deletion before it calls `DeleteDB`, and `DeleteDB` removes the files while holding the event database manager's lock (`mu` in `internal/database/webhook_db_manager.go`), the same lock that opening a database takes. A check made while holding that lock, followed by the open under the same hold, either sees the deletion or finishes before `DeleteDB` can remove the files. Acceptable: the check and the open run together under that lock, for example through a manager method that runs a check passed by the caller while holding the lock and opens only if it passes; the existing test still passes, and the disclosure of the remaining window is dropped. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-02 21:13:34 +02:00
Some checks are pending
check / check (push) Waiting to run
You are not authorized to merge this pull request.
This pull request can be merged automatically.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin issue-290-file-config-audit:issue-290-file-config-audit
git checkout issue-290-file-config-audit
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/webhooker#460