3 Commits
Author SHA1 Message Date
clawbot 9f1e9f5253 Rewrite AGENTS.md in short plain sentences (#327)
check / check (push) Canceled after 0s
Every rule keeps its meaning and emphasis. Sentences are split so each
holds one idea, with the main rule before its exception and every
command, file name and identifier in backticks.

Three statements were false against the repo or the standing rulings
on record and are corrected:

- the format check is `make fmt-check`, not `make check-fmt`
- a feature branch reaches `next` through a PR that passes review, is
  merged only when the feature is completely finished and the code is
  up to standards, and is deleted after the merge; only the user
  merges `next` into `main`
- the naming example uses the code's term: `al` is bad,
  `SourceHostAllowlist` is good

The `Last Updated` line takes today's date.

Model: fable-5-1
2026-10-27 05:06:36 +01:00
clawbot 2c90144e7b Go on past cache entries the reconciliation pass cannot read (closes #343)
check / check (push) Canceled after 0s
The walk of each storage directory returned the error of a directory it
could not list, or of a source or variant file whose info it could not
read. The pass then stopped before its later checks, every time it ran.
Now reconcileFiles skips a directory it cannot list and a file whose
info it cannot read, temp files included. It logs a warning naming the
path and the error, and the walk goes on. It reads each file's info
once and hands it to sweepStaleTempFile, reconcileVariantFile or
reconcileSourceFile. A file renamed or removed after its directory was
listed is skipped with no warning. A cancelled context still stops the
walk with its error. README.md says so under Storage.

Model: opus-5-5
2026-10-27 01:13:55 +00:00
clawbot 26b5160aa8 Test that reconciliation goes on past entries it cannot read
Each storage directory gets a directory the pass cannot list and a file
whose info it cannot read. Their paths are longer than Linux accepts,
even for root. The test wants a warning naming each path and the error.
It also wants the rest of the pass done: a variant file with no row
adopted, the row of a missing variant file dropped, a source file with
no row removed, and a wrong usage total corrected. A second test puts a
temp file whose info cannot be read in each storage directory and wants
the same warning. On next the first test's pass stops at the first such
directory and returns its error. The second test gets no warning.

Model: opus-5-5
2026-10-27 01:12:34 +00:00
4 changed files with 94 additions and 79 deletions
+52 -51
View File
@@ -1,75 +1,76 @@
# Repository Rules # Repository Rules
Last Updated 2026-01-08 Last Updated 2026-10-27
These rules MUST be followed at all times, it is very important. These rules MUST be followed at all times. This is very important.
- Never use `git add -A` - add specific changes to a deliberate commit. A commit - Never use `git add -A`. Stage specific changes for a deliberate commit. A
should contain one change. After each change, make a commit with a good commit holds one change. After each change, make a commit with a good one-line
one-line summary. summary.
- NEVER modify the linter config without asking first. - NEVER modify the linter config without asking first.
- NEVER modify tests to exclude special cases or otherwise get them to pass - NEVER modify a test to exclude special cases or otherwise make it pass without
without asking first. In almost all cases, the code should be changed, NOT the asking first. In almost all cases, change the code, NOT the test. You need
tests. If you think the test needs to be changed, make your case for that and explicit user approval to modify an existing test. If you think a test needs
ask for permission to proceed, then stop. You need explicit user approval to to change, make your case and ask for permission to proceed. Then stop. A bug
modify existing tests. A bug fix may change an existing test without asking fix may change an existing test without asking only when that test pins
when that test pins exactly the behavior the filed bug calls wrong, and the exactly the behavior the filed bug calls wrong. The change must then move the
change moves it to the fixed behavior and nothing else; any other change to an test to the fixed behavior and nothing else. Any other change to an existing
existing test still needs explicit user approval. (You do not need user test needs explicit user approval. Writing NEW tests needs no approval.
approval for writing NEW tests.)
- When linting, assume the linter config is CORRECT, and that each item output - When linting, assume the linter config is CORRECT. Treat each item the linter
by the linter is something that legitimately needs fixing in the code. reports as something that legitimately needs fixing in the code.
- When running tests, use `make test`. - When running tests, use `make test`.
- Before commits, run `make check`. This runs `make lint` and `make test` and - Before committing, run `make check`. It runs `make test`, `make lint` and
`make check-fmt`. Any issues discovered MUST be resolved before committing `make fmt-check`. Any issue it finds MUST be resolved before committing,
unless explicitly told otherwise. unless you are explicitly told otherwise.
- When fixing a bug, write a failing test for the bug FIRST. Add appropriate - When fixing a bug, write a failing test for the bug FIRST. Add appropriate
logging to the test to ensure it is written correctly. Commit that. Then go logging to the test, to make sure the test is written correctly. Commit that.
about fixing the bug until the test passes (without modifying the test Then work on the fix until the test passes, without changing the test further.
further). Then commit that. Then commit the fix.
- When adding a new feature, do the same - implement a test first (TDD). It - When adding a new feature, do the same: write a test first (test-driven
doesn't have to be super complex. Commit the test, then commit the feature. development, TDD). The test does not need to be very complex. Commit the test,
then commit the feature.
- When adding a new feature, use a feature branch. When the feature is - When adding a new feature, use a feature branch. The feature branch reaches
completely finished and the code is up to standards (passes `make check`) then `next` through a PR that passes review. It is merged only when the feature is
and only then can the feature branch be merged into `main` and the branch completely finished and the code is up to standards. The code is up to
deleted. standards when it passes `make check`. The branch is deleted after the merge.
Only the user merges `next` into `main`.
- Write godoc documentation comments for all exported types and functions as you - Write `godoc` comments for all exported types and functions as you go along.
go along.
- ALWAYS be consistent in naming. If you name something one thing in one place, - ALWAYS be consistent in naming. If you name something one way in one place,
name it the EXACT SAME THING in another place. use the EXACT SAME name in every other place.
- Be descriptive and specific in naming. `wl` is bad; `SourceHostWhitelist` is - Be descriptive and specific in naming. `al` is bad; `SourceHostAllowlist` is
good. `ConnsPerHost` is bad; `MaxConnectionsPerHost` is good. good. `ConnsPerHost` is bad; `MaxConnectionsPerHost` is good.
- This is not prototype or teaching code - this is designed for production. Any - This is not prototype or teaching code. It is designed for production. Any
security issues (such as denial of service) or other web vulnerabilities are security issue (such as denial of service) or other web vulnerability is a P1
P1 bugs and must be added to TODO.md at the top. bug. It must be added at the top of `TODO.md`.
- As this is production code, no stubbing of implementations unless specifically - Do not stub an implementation unless you are specifically instructed to. This
instructed. We need working implementations. is production code, and we need working implementations.
- NEVER silently fall back to a different setting when a user's parameter - NEVER silently fall back to a different setting when a user's parameter
explicitly specifies a value. If a user requests format=webp and WebP encoding explicitly specifies a value. For example, if a user requests `format=webp`
is not supported, return an error - do NOT silently output PNG instead. If a and WebP encoding is not supported, return an error. Do NOT silently output
user specifies fit=invalid and that fit mode doesn't exist, return an error - PNG instead. If a user specifies `fit=invalid` and that fit mode does not
do NOT silently default to "cover". Silent fallbacks violate the principle of exist, return an error. Do NOT silently default to `cover`. A silent fallback
least surprise and mask bugs. The only acceptable defaults are for OMITTED violates the principle of least surprise and masks bugs. Defaults are
parameters, never for INVALID explicit values. acceptable only for OMITTED parameters, never for INVALID explicit values.
- Avoid vendoring deps unless specifically instructed to. NEVER commit the - Avoid vendoring dependencies unless specifically instructed to. NEVER commit
vendor directory, NEVER commit compiled binaries. If these directories or the `vendor/` directory. NEVER commit compiled binaries. If such a directory
files exist, add them to .gitignore (and commit the .gitignore) if they are or file exists and `.gitignore` does not list it, add it to `.gitignore`.
not already in there. Keep the entire git repository (with history) small - Commit that `.gitignore` change. Keep the entire git repository, history
under 20MiB, unless you specifically must commit larger files (e.g. test included, under 20 MiB. Go over 20 MiB only when you specifically must commit
fixture example media files). Only OUR source code and immediately supporting a large file, such as a media file used as a test fixture. Only OUR source
files (such as test examples) goes into the repo/history. code and its immediately supporting files, such as test examples, go into the
repository and its history.
+4 -4
View File
@@ -256,10 +256,10 @@ Each look also drops the rows of variants whose files are missing. When pixa
cannot read a variant file's size for another reason, it keeps the variant's cannot read a variant file's size for another reason, it keeps the variant's
row. It logs a warning naming the variant key and the error. row. It logs a warning naming the variant key and the error.
During a look, pixa skips any file or directory in the cache that it cannot During a look, pixa skips a directory it cannot list and a file whose size it
read. It logs a warning naming the path and the error, and goes on with the rest cannot read. It logs a warning naming the path and the error, and goes on with
of the look. A file renamed or removed after pixa listed its directory gets no the rest of the look. A file renamed or removed after pixa listed its directory
warning. gets no warning.
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
Test). Toward it, a cache hit opens the variant's file and nothing else, and Test). Toward it, a cache hit opens the variant's file and nothing else, and
+3 -2
View File
@@ -31,8 +31,9 @@ P2: security: per-origin rate limiting
# Completed Steps # Completed Steps
- 2026-10-27 the reconciliation pass skips a directory it cannot list and a file - 2026-10-27 the reconciliation pass skips a directory it cannot list and a file
whose info it cannot read, and goes on (closes #343). It returned the error, whose info it cannot read, and goes on (closes #343). It stopped before its
so one such entry stopped every pass before its later checks. later checks at a directory it could not list, or at a source or variant file
whose info it could not read.
- It logs the warning "skipped unreadable entry in cache directory", with - It logs the warning "skipped unreadable entry in cache directory", with
the path and the error. the path and the error.
- A file renamed or removed after its directory was listed gets no warning. - A file renamed or removed after its directory was listed gets no warning.
+35 -22
View File
@@ -819,10 +819,11 @@ func openDirectoryNearPathLimit(t *testing.T, dir string) *os.Root {
return root return root
} }
// makeEntriesTooLongToRead makes a directory and a file in the directory // makeEntriesTooLongToRead makes a new directory and a new file in the
// that openDirectoryNearPathLimit opens inside dir. Even the root user can // deepest directory of the chain that openDirectoryNearPathLimit makes
// then neither list that directory nor read the file's info. It returns // inside dir. Even the root user can then neither list the new directory
// their two paths. // nor read the new file's info. It returns the paths of the new directory
// and the new file.
func makeEntriesTooLongToRead(t *testing.T, dir string) []string { func makeEntriesTooLongToRead(t *testing.T, dir string) []string {
t.Helper() t.Helper()
@@ -847,9 +848,10 @@ func makeEntriesTooLongToRead(t *testing.T, dir string) []string {
} }
} }
// makeTempFileTooLongToRead makes a temp file in the directory that // makeTempFileTooLongToRead makes a temp file in the deepest directory of
// openDirectoryNearPathLimit opens inside dir. Even the root user then // the chain that openDirectoryNearPathLimit makes inside dir. Even the
// cannot read its info. It returns its path. // root user then cannot read the temp file's info. It returns the temp
// file's path.
func makeTempFileTooLongToRead(t *testing.T, dir string) string { func makeTempFileTooLongToRead(t *testing.T, dir string) string {
t.Helper() t.Helper()
@@ -864,6 +866,25 @@ func makeTempFileTooLongToRead(t *testing.T, dir string) string {
return filepath.Join(deepest.Name(), name) return filepath.Join(deepest.Name(), name)
} }
// assertSkippedEntryWarning checks that logs hold the warning "skipped
// unreadable entry in cache directory" naming path, with the error
// ENAMETOOLONG for that path.
func assertSkippedEntryWarning(t *testing.T, logs, path string) {
t.Helper()
for line := range strings.Lines(logs) {
if strings.Contains(line, `"level":"WARN"`) &&
strings.Contains(line, `"msg":"skipped unreadable entry in cache directory"`) &&
strings.Contains(line, `"path":"`+path+`"`) &&
strings.Contains(line, path+": "+syscall.ENAMETOOLONG.Error()) {
return
}
}
t.Errorf("no warning %q names %s with the error %q",
"skipped unreadable entry in cache directory", path, syscall.ENAMETOOLONG)
}
// TestReconciliationGoesOnPastEntriesItCannotRead puts, in each storage // TestReconciliationGoesOnPastEntriesItCannotRead puts, in each storage
// directory, a directory the pass cannot list and a file whose info it // directory, a directory the pass cannot list and a file whose info it
// cannot read. The pass walks each directory in name order, so it reaches // cannot read. The pass walks each directory in name order, so it reaches
@@ -915,11 +936,7 @@ func TestReconciliationGoesOnPastEntriesItCannotRead(t *testing.T) {
logs := reconcileAsAtStartup(t, cache) logs := reconcileAsAtStartup(t, cache)
for _, path := range unreadable { for _, path := range unreadable {
assertWarningNames(t, logs, path) assertSkippedEntryWarning(t, logs, path)
if !strings.Contains(logs, path+": "+syscall.ENAMETOOLONG.Error()) {
t.Errorf("no log line gives the error reading %s", path)
}
} }
rowQuery := `SELECT COUNT(*) FROM variant_content WHERE cache_key = ?` rowQuery := `SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`
@@ -948,11 +965,11 @@ func TestReconciliationGoesOnPastEntriesItCannotRead(t *testing.T) {
} }
} }
// TestReconciliationWarnsOfTempFilesItCannotRead puts, in each storage // TestReconciliationWarnsOfTempFilesWhoseInfoItCannotRead puts, in each
// directory, a temp file whose info the pass cannot read. The pass must // storage directory, a temp file whose info the pass cannot read. The
// log a warning naming each one's path and the error, as it does for any // pass must log a warning naming each one's path and the error, as it
// other file whose info it cannot read. // does for any other file whose info it cannot read.
func TestReconciliationWarnsOfTempFilesItCannotRead(t *testing.T) { func TestReconciliationWarnsOfTempFilesWhoseInfoItCannotRead(t *testing.T) {
t.Parallel() t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30) cache, _ := newEvictionTestCache(t, 1<<30)
@@ -965,11 +982,7 @@ func TestReconciliationWarnsOfTempFilesItCannotRead(t *testing.T) {
logs := reconcileAsAtStartup(t, cache) logs := reconcileAsAtStartup(t, cache)
for _, path := range unreadable { for _, path := range unreadable {
assertWarningNames(t, logs, path) assertSkippedEntryWarning(t, logs, path)
if !strings.Contains(logs, path+": "+syscall.ENAMETOOLONG.Error()) {
t.Errorf("no log line gives the error reading %s", path)
}
} }
} }