Run the reconciliation pass at startup and once a day (closes #250) #279

Open
clawbot wants to merge 2 commits from issue-250-daily-reconciliation into next
Collaborator

Closes #250.

The reconciliation pass makes a database query and a stat for every cached file, and it ran on every eviction tick. On a large cache a pass could take longer than the 5-minute tick, and the tick waiting in the ticker then started the next pass at once.

The pass now runs at startup and then on a timer of its own, started with one day each time a pass ends, so passes never run back to back. The eviction tick still removes the expired negative cache rows and evicts to the limit. Eviction also runs after each reconciliation pass, so a file a pass adopts can be evicted right away. The comments that bound the drift now say about one day, and so do README.md and TODO.md.

What the diff does not show: a variant file whose database row failed to be written now stays out of the cache size for up to a day, not 5 minutes.

TestEvictionTicksDoNotReconcile counts reads through reconciliationReadTestHook, and checks that three eviction ticks after the startup pass make none.

Red: TestEvictionRunsOnPeriodicSchedule and TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup fail, because each waits for a reconciliation pass on a 100 ms eviction tick. Existing tests may not change without approval, so the question for sneak is in a comment on this PR.

Model: opus-5-5

Closes https://git.eeqj.de/sneak/pixa/issues/250. The reconciliation pass makes a database query and a stat for every cached file, and it ran on every eviction tick. On a large cache a pass could take longer than the 5-minute tick, and the tick waiting in the ticker then started the next pass at once. The pass now runs at startup and then on a timer of its own, started with one day each time a pass ends, so passes never run back to back. The eviction tick still removes the expired negative cache rows and evicts to the limit. Eviction also runs after each reconciliation pass, so a file a pass adopts can be evicted right away. The comments that bound the drift now say about one day, and so do `README.md` and `TODO.md`. What the diff does not show: a variant file whose database row failed to be written now stays out of the cache size for up to a day, not 5 minutes. `TestEvictionTicksDoNotReconcile` counts reads through `reconciliationReadTestHook`, and checks that three eviction ticks after the startup pass make none. Red: `TestEvictionRunsOnPeriodicSchedule` and `TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup` fail, because each waits for a reconciliation pass on a 100 ms eviction tick. Existing tests may not change without approval, so the question for sneak is in a comment on this PR. Model: opus-5-5
clawbot added the needs-review label 2026-10-15 22:51:47 +02:00
clawbot added 2 commits 2026-10-15 22:51:47 +02:00
The eviction loop runs the reconciliation pass on every periodic tick,
so this test fails until the pass runs at startup and then once a day.

Model: opus-5-5
The pass makes a database query for every cached file and ran on every
eviction tick, so on a large cache one pass could outlast the tick and
the next started at once. It now runs on a timer of its own that starts
when a pass ends, a day long. Eviction and the removal of expired
negative cache rows keep the eviction interval.

Model: opus-5-5
clawbot self-assigned this 2026-10-15 22:51:51 +02:00
Author
Collaborator

Question for sneak: TestEvictionRunsOnPeriodicSchedule and TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup write a file after the startup pass and wait 5 seconds for a reconciliation pass on a 100 ms eviction tick to adopt it. Now that the pass runs once a day, both fail, and changing them needs your approval. What should happen to them?

  1. Give Cache a reconciliation interval that newCache sets to one day, the way it sets reconciliationPageSize, and add one line to each test setting it to 100 ms. Both tests then still check that a pass after startup adopts a file written later.
  2. Delete both tests. The new test checks the schedule, and other tests check what a pass does.
  3. Leave both as they are, and make a zero reconciliation interval mean "reconcile on every eviction tick". Only tests would use that.

Recommendation: 1.

Reading taken: neither test is changed, so the branch fails on those two tests until you decide.

Model: opus-5-5

Question for sneak: `TestEvictionRunsOnPeriodicSchedule` and `TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup` write a file after the startup pass and wait 5 seconds for a reconciliation pass on a 100 ms eviction tick to adopt it. Now that the pass runs once a day, both fail, and changing them needs your approval. What should happen to them? 1. Give `Cache` a reconciliation interval that `newCache` sets to one day, the way it sets `reconciliationPageSize`, and add one line to each test setting it to 100 ms. Both tests then still check that a pass after startup adopts a file written later. 2. Delete both tests. The new test checks the schedule, and other tests check what a pass does. 3. Leave both as they are, and make a zero reconciliation interval mean "reconcile on every eviction tick". Only tests would use that. Recommendation: 1. Reading taken: neither test is changed, so the branch fails on those two tests until you decide. Model: opus-5-5
clawbot added needs-checks and removed needs-review labels 2026-10-15 22:53:35 +02:00
Some checks are pending
check / check (push) Waiting to run
This pull request has changes conflicting with the target branch.
  • TODO.md
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin issue-250-daily-reconciliation:issue-250-daily-reconciliation
git checkout issue-250-daily-reconciliation
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/pixa#279