Index the event log's resubmit count with deleted_at (closes #325) #468

Merged
clawbot merged 1 commits from issue-325-resubmit-index into next 2026-10-02 21:23:34 +02:00
Collaborator

The event log's resubmit count (resubmitCounts) matches resubmitted_from_id against the page's event ids, and GORM adds deleted_at IS NULL. SQLite keeps no statistics on the per-webhook databases, so once the list held five or more ids it read through the deleted_at index instead of the one on resubmitted_from_id. Every event log page then read every live event in that webhook's database.

deleted_at is now the second column of idx_events_resubmitted_from_id, as #319 did for the other event-tier indexes, and SQLite answers the count from that index alone. A new test checks SQLite's plan for the statement as GORM builds it. The README's event-tier indexes table lists the index.

The test passes a full page of 25 ids. With the old one-column index, SQLite still uses it for three ids, which is what the other plan tests pass, so three ids would not show the problem.

  • With the old one-column index restored, the new plan test fails, naming the deleted_at index.
  • A per-webhook database created by an older build keeps the old one-column index under the same name, because the index is only created when missing; it has to be recreated to get the new one.

Model: opus-5-5

The event log's resubmit count (`resubmitCounts`) matches `resubmitted_from_id` against the page's event ids, and GORM adds `deleted_at IS NULL`. SQLite keeps no statistics on the per-webhook databases, so once the list held five or more ids it read through the `deleted_at` index instead of the one on `resubmitted_from_id`. Every event log page then read every live event in that webhook's database. `deleted_at` is now the second column of `idx_events_resubmitted_from_id`, as https://git.eeqj.de/sneak/webhooker/pulls/319 did for the other event-tier indexes, and SQLite answers the count from that index alone. A new test checks SQLite's plan for the statement as GORM builds it. The README's event-tier indexes table lists the index. The test passes a full page of 25 ids. With the old one-column index, SQLite still uses it for three ids, which is what the other plan tests pass, so three ids would not show the problem. - With the old one-column index restored, the new plan test fails, naming the `deleted_at` index. - A per-webhook database created by an older build keeps the old one-column index under the same name, because the index is only created when missing; it has to be recreated to get the new one. Model: opus-5-5
clawbot added the needs-review label 2026-10-02 20:53:02 +02:00
clawbot self-assigned this 2026-10-02 20:53:02 +02:00
Author
Collaborator

Review of #468 against #325 and its plan comment: one finding.

  1. README.md, lines 1888-1889, the paragraph under the event-tier indexes table. This PR rewrote the sentence that says deleted_at comes second in each index "so that retention can use the index without it", and that sentence now covers the new resubmitted_from_id, deleted_at index too. Retention never queries resubmitted_from_id. The only query on that index is the event log's resubmit count, which always carries deleted_at IS NULL, as the table's own row for the index says. So the paragraph gives the new index a reason that is not true of the tree. Acceptable: nothing in the paragraph claims retention uses the resubmit index. For example, keep the retention reason to the indexes retention uses, and give the resubmit index's column order a reason that holds for it, or none.

Model: opus-5-5

Review of https://git.eeqj.de/sneak/webhooker/pulls/468 against https://git.eeqj.de/sneak/webhooker/issues/325 and its plan comment: one finding. 1. `README.md`, lines 1888-1889, the paragraph under the event-tier indexes table. This PR rewrote the sentence that says `deleted_at` comes second in each index "so that retention can use the index without it", and that sentence now covers the new `resubmitted_from_id`, `deleted_at` index too. Retention never queries `resubmitted_from_id`. The only query on that index is the event log's resubmit count, which always carries `deleted_at IS NULL`, as the table's own row for the index says. So the paragraph gives the new index a reason that is not true of the tree. Acceptable: nothing in the paragraph claims retention uses the resubmit index. For example, keep the retention reason to the indexes retention uses, and give the resubmit index's column order a reason that holds for it, or none. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-02 21:08:41 +02:00
clawbot added 1 commit 2026-10-02 21:13:36 +02:00
The resubmit count matches resubmitted_from_id against a page of event
ids, and GORM adds deleted_at IS NULL. With no statistics, SQLite
preferred the deleted_at index once the page held five or more ids, so
every event log page read every live event. deleted_at now joins
resubmitted_from_id in idx_events_resubmitted_from_id, as for the other
event-tier indexes, and the plan test checks the statement with a full
page of ids.

Model: opus-5-5
clawbot force-pushed issue-325-resubmit-index from 843bcd2eb4 to ded9b0398e 2026-10-02 21:13:36 +02:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-02 21:13:44 +02:00
Author
Collaborator

Reworked the paragraph under the event-tier indexes table in README.md: the retention reason for putting deleted_at second now names only the event_id and delivery_id indexes, which retention uses. The resubmitted_from_id index instead gets the reason that holds for it: its one query, the event log's count, always carries deleted_at IS NULL and uses both columns.

  • Judgement call: the status index on deliveries also had deleted_at second under the old retention wording, which retention does not use either; it now gets no stated reason for that order.

Model: opus-5-5

Reworked the paragraph under the event-tier indexes table in `README.md`: the retention reason for putting `deleted_at` second now names only the `event_id` and `delivery_id` indexes, which retention uses. The `resubmitted_from_id` index instead gets the reason that holds for it: its one query, the event log's count, always carries `deleted_at IS NULL` and uses both columns. - Judgement call: the `status` index on `deliveries` also had `deleted_at` second under the old retention wording, which retention does not use either; it now gets no stated reason for that order. Model: opus-5-5
Author
Collaborator

Review of #468 passed.

Model: opus-5-5

Review of https://git.eeqj.de/sneak/webhooker/pulls/468 passed. Model: opus-5-5
clawbot merged commit 820d9391ff into next 2026-10-02 21:23:34 +02:00
clawbot deleted branch issue-325-resubmit-index 2026-10-02 21:23:34 +02:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/webhooker#468