Go on past cache entries the reconciliation pass cannot read (closes #343)

The walk of each storage directory returned the error of an entry it
could not read. The pass then stopped before its later checks, and the
same entry stopped every later pass. Now reconcileFiles skips a
directory it cannot list and a file whose info it cannot read, temp
files included. It logs a warning naming the path and the error, and the
walk goes on. It reads each file's info once, before the temp-file
check, and hands it to sweepStaleTempFile, reconcileVariantFile or
reconcileSourceFile. A file renamed or removed after its directory was
listed is skipped with no warning. A cancelled context still stops the
walk with its error. README.md says so under Storage.

Model: opus-5-5

AGENTS.md

@@ -1,76 +1,75 @@
 # Repository Rules
 
-Last Updated 2026-10-27
+Last Updated 2026-01-08
 
-These rules MUST be followed at all times. This is very important.
+These rules MUST be followed at all times, it is very important.
 
-- Never use `git add -A`. Stage specific changes for a deliberate commit. A
+- Never use `git add -A` - add specific changes to a deliberate commit. A commit
-  commit holds one change. After each change, make a commit with a good one-line
+  should contain one change. After each change, make a commit with a good
-  summary.
+  one-line summary.
 
 - NEVER modify the linter config without asking first.
 
-- NEVER modify a test to exclude special cases or otherwise make it pass without
+- NEVER modify tests to exclude special cases or otherwise get them to pass
-  asking first. In almost all cases, change the code, NOT the test. You need
+  without asking first. In almost all cases, the code should be changed, NOT the
-  explicit user approval to modify an existing test. If you think a test needs
+  tests. If you think the test needs to be changed, make your case for that and
-  to change, make your case and ask for permission to proceed. Then stop. A bug
+  ask for permission to proceed, then stop. You need explicit user approval to
-  fix may change an existing test without asking only when that test pins
+  modify existing tests. A bug fix may change an existing test without asking
-  exactly the behavior the filed bug calls wrong. The change must then move the
+  when that test pins exactly the behavior the filed bug calls wrong, and the
-  test to the fixed behavior and nothing else. Any other change to an existing
+  change moves it to the fixed behavior and nothing else; any other change to an
-  test needs explicit user approval. Writing NEW tests needs no approval.
+  existing test still needs explicit user approval. (You do not need user
+  approval for writing NEW tests.)
 
-- When linting, assume the linter config is CORRECT. Treat each item the linter
+- When linting, assume the linter config is CORRECT, and that each item output
-  reports as something that legitimately needs fixing in the code.
+  by the linter is something that legitimately needs fixing in the code.
 
 - When running tests, use `make test`.
 
-- Before committing, run `make check`. It runs `make test`, `make lint` and
+- Before commits, run `make check`. This runs `make lint` and `make test` and
-  `make fmt-check`. Any issue it finds MUST be resolved before committing,
+  `make check-fmt`. Any issues discovered MUST be resolved before committing
-  unless you are explicitly told otherwise.
+  unless explicitly told otherwise.
 
 - When fixing a bug, write a failing test for the bug FIRST. Add appropriate
-  logging to the test, to make sure the test is written correctly. Commit that.
+  logging to the test to ensure it is written correctly. Commit that. Then go
-  Then work on the fix until the test passes, without changing the test further.
+  about fixing the bug until the test passes (without modifying the test
-  Then commit the fix.
+  further). Then commit that.
 
-- When adding a new feature, do the same: write a test first (test-driven
+- When adding a new feature, do the same - implement a test first (TDD). It
-  development, TDD). The test does not need to be very complex. Commit the test,
+  doesn't have to be super complex. Commit the test, then commit the feature.
-  then commit the feature.
 
-- When adding a new feature, use a feature branch. The feature branch reaches
+- When adding a new feature, use a feature branch. When the feature is
-  `next` through a PR that passes review. It is merged only when the feature is
+  completely finished and the code is up to standards (passes `make check`) then
-  completely finished and the code is up to standards. The code is up to
+  and only then can the feature branch be merged into `main` and the branch
-  standards when it passes `make check`. The branch is deleted after the merge.
+  deleted.
-  Only the user merges `next` into `main`.
 
-- Write `godoc` comments for all exported types and functions as you go along.
+- Write godoc documentation comments for all exported types and functions as you
+  go along.
 
-- ALWAYS be consistent in naming. If you name something one way in one place,
+- ALWAYS be consistent in naming. If you name something one thing in one place,
-  use the EXACT SAME name in every other place.
+  name it the EXACT SAME THING in another place.
 
-- Be descriptive and specific in naming. `al` is bad; `SourceHostAllowlist` is
+- Be descriptive and specific in naming. `wl` is bad; `SourceHostWhitelist` is
   good. `ConnsPerHost` is bad; `MaxConnectionsPerHost` is good.
 
-- This is not prototype or teaching code. It is designed for production. Any
+- This is not prototype or teaching code - this is designed for production. Any
-  security issue (such as denial of service) or other web vulnerability is a P1
+  security issues (such as denial of service) or other web vulnerabilities are
-  bug. It must be added at the top of `TODO.md`.
+  P1 bugs and must be added to TODO.md at the top.
 
-- Do not stub an implementation unless you are specifically instructed to. This
+- As this is production code, no stubbing of implementations unless specifically
-  is production code, and we need working implementations.
+  instructed. We need working implementations.
 
 - NEVER silently fall back to a different setting when a user's parameter
-  explicitly specifies a value. For example, if a user requests `format=webp`
+  explicitly specifies a value. If a user requests format=webp and WebP encoding
-  and WebP encoding is not supported, return an error. Do NOT silently output
+  is not supported, return an error - do NOT silently output PNG instead. If a
-  PNG instead. If a user specifies `fit=invalid` and that fit mode does not
+  user specifies fit=invalid and that fit mode doesn't exist, return an error -
-  exist, return an error. Do NOT silently default to `cover`. A silent fallback
+  do NOT silently default to "cover". Silent fallbacks violate the principle of
-  violates the principle of least surprise and masks bugs. Defaults are
+  least surprise and mask bugs. The only acceptable defaults are for OMITTED
-  acceptable only for OMITTED parameters, never for INVALID explicit values.
+  parameters, never for INVALID explicit values.
 
-- Avoid vendoring dependencies unless specifically instructed to. NEVER commit
+- Avoid vendoring deps unless specifically instructed to. NEVER commit the
-  the `vendor/` directory. NEVER commit compiled binaries. If such a directory
+  vendor directory, NEVER commit compiled binaries. If these directories or
-  or file exists and `.gitignore` does not list it, add it to `.gitignore`.
+  files exist, add them to .gitignore (and commit the .gitignore) if they are
-  Commit that `.gitignore` change. Keep the entire git repository, history
+  not already in there. Keep the entire git repository (with history) small -
-  included, under 20 MiB. Go over 20 MiB only when you specifically must commit
+  under 20MiB, unless you specifically must commit larger files (e.g. test
-  a large file, such as a media file used as a test fixture. Only OUR source
+  fixture example media files). Only OUR source code and immediately supporting
-  code and its immediately supporting files, such as test examples, go into the
+  files (such as test examples) goes into the repo/history.
-  repository and its history.

README.md

@@ -256,10 +256,10 @@ Each look also drops the rows of variants whose files are missing. When pixa
 cannot read a variant file's size for another reason, it keeps the variant's
 row. It logs a warning naming the variant key and the error.
 
-During a look, pixa skips a directory it cannot list and a file whose size it
+During a look, pixa skips any file or directory in the cache that it cannot
-cannot read. It logs a warning naming the path and the error, and goes on with
+read. It logs a warning naming the path and the error, and goes on with the rest
-the rest of the look. A file renamed or removed after pixa listed its directory
+of the look. A file renamed or removed after pixa listed its directory gets no
-gets no warning.
+warning.
 
 pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
 Test). Toward it, a cache hit opens the variant's file and nothing else, and

TODO.md

@@ -31,9 +31,8 @@ P2: security: per-origin rate limiting
 # Completed Steps
 
 - 2026-10-27 the reconciliation pass skips a directory it cannot list and a file
-  whose info it cannot read, and goes on (closes #343). It stopped before its
+  whose info it cannot read, and goes on (closes #343). It returned the error,
-  later checks at a directory it could not list, or at a source or variant file
+  so one such entry stopped every pass before its later checks.
-  whose info it could not read.
     - It logs the warning "skipped unreadable entry in cache directory", with
       the path and the error.
     - A file renamed or removed after its directory was listed gets no warning.

internal/imgcache/eviction_internal_test.go

@@ -819,11 +819,10 @@ func openDirectoryNearPathLimit(t *testing.T, dir string) *os.Root {
 	return root
 }
 
-// makeEntriesTooLongToRead makes a new directory and a new file in the
+// makeEntriesTooLongToRead makes a directory and a file in the directory
-// deepest directory of the chain that openDirectoryNearPathLimit makes
+// that openDirectoryNearPathLimit opens inside dir. Even the root user can
-// inside dir. Even the root user can then neither list the new directory
+// then neither list that directory nor read the file's info. It returns
-// nor read the new file's info. It returns the paths of the new directory
+// their two paths.
-// and the new file.
 func makeEntriesTooLongToRead(t *testing.T, dir string) []string {
 	t.Helper()
 
@@ -848,10 +847,9 @@ func makeEntriesTooLongToRead(t *testing.T, dir string) []string {
 	}
 }
 
-// makeTempFileTooLongToRead makes a temp file in the deepest directory of
+// makeTempFileTooLongToRead makes a temp file in the directory that
-// the chain that openDirectoryNearPathLimit makes inside dir. Even the
+// openDirectoryNearPathLimit opens inside dir. Even the root user then
-// root user then cannot read the temp file's info. It returns the temp
+// cannot read its info. It returns its path.
-// file's path.
 func makeTempFileTooLongToRead(t *testing.T, dir string) string {
 	t.Helper()
 
@@ -866,25 +864,6 @@ func makeTempFileTooLongToRead(t *testing.T, dir string) string {
 	return filepath.Join(deepest.Name(), name)
 }
 
-// assertSkippedEntryWarning checks that logs hold the warning "skipped
-// unreadable entry in cache directory" naming path, with the error
-// ENAMETOOLONG for that path.
-func assertSkippedEntryWarning(t *testing.T, logs, path string) {
-	t.Helper()
-
-	for line := range strings.Lines(logs) {
-		if strings.Contains(line, `"level":"WARN"`) &&
-			strings.Contains(line, `"msg":"skipped unreadable entry in cache directory"`) &&
-			strings.Contains(line, `"path":"`+path+`"`) &&
-			strings.Contains(line, path+": "+syscall.ENAMETOOLONG.Error()) {
-			return
-		}
-	}
-
-	t.Errorf("no warning %q names %s with the error %q",
-		"skipped unreadable entry in cache directory", path, syscall.ENAMETOOLONG)
-}
-
 // TestReconciliationGoesOnPastEntriesItCannotRead puts, in each storage
 // directory, a directory the pass cannot list and a file whose info it
 // cannot read. The pass walks each directory in name order, so it reaches
@@ -936,7 +915,11 @@ func TestReconciliationGoesOnPastEntriesItCannotRead(t *testing.T) {
 	logs := reconcileAsAtStartup(t, cache)
 
 	for _, path := range unreadable {
-		assertSkippedEntryWarning(t, logs, path)
+		assertWarningNames(t, logs, path)
+
+		if !strings.Contains(logs, path+": "+syscall.ENAMETOOLONG.Error()) {
+			t.Errorf("no log line gives the error reading %s", path)
+		}
 	}
 
 	rowQuery := `SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`
@@ -965,11 +948,11 @@ func TestReconciliationGoesOnPastEntriesItCannotRead(t *testing.T) {
 	}
 }
 
-// TestReconciliationWarnsOfTempFilesWhoseInfoItCannotRead puts, in each
+// TestReconciliationWarnsOfTempFilesItCannotRead puts, in each storage
-// storage directory, a temp file whose info the pass cannot read. The
+// directory, a temp file whose info the pass cannot read. The pass must
-// pass must log a warning naming each one's path and the error, as it
+// log a warning naming each one's path and the error, as it does for any
-// does for any other file whose info it cannot read.
+// other file whose info it cannot read.
-func TestReconciliationWarnsOfTempFilesWhoseInfoItCannotRead(t *testing.T) {
+func TestReconciliationWarnsOfTempFilesItCannotRead(t *testing.T) {
 	t.Parallel()
 
 	cache, _ := newEvictionTestCache(t, 1<<30)
@@ -982,7 +965,11 @@ func TestReconciliationWarnsOfTempFilesWhoseInfoItCannotRead(t *testing.T) {
 	logs := reconcileAsAtStartup(t, cache)
 
 	for _, path := range unreadable {
-		assertSkippedEntryWarning(t, logs, path)
+		assertWarningNames(t, logs, path)
+
+		if !strings.Contains(logs, path+": "+syscall.ENAMETOOLONG.Error()) {
+			t.Errorf("no log line gives the error reading %s", path)
+		}
 	}
 }