report and trees hold every record, including both hex hashes, in memory #14

New Issue

clawbot · 2026-08-09T03:45:49+02:00

clawbot commented

2026-08-09 03:45:49 +02:00

report.go:32-48 calls loadFileRows (db.go:207-239), which materializes the entire files table as a []scanRec. trees.go:38-43 does the same and then builds the full node graph on top of it.

Each record carries two 64-character hex hash strings plus the path plus slice and string headers — call it 210 bytes at minimum, before Go's allocator overhead. At the stated scale of ~10M files that is over 2 GB resident for report alone, on a machine that may be the same storage server running the scan.

This is the one design constraint the scan phase goes out of its way to honour and the reporting phase ignores. README §Design goal 2 is explicit: "Holding one small record (path, size, mtime) per file in memory during a scan is acceptable; holding every file's hashes is not (they stay in the database)."

Note the two commands have different ceilings. report never needs more than one duplicate group at a time and can stream. trees genuinely needs the whole hierarchy to compute Merkle digests bottom-up, but it does not need to retain the hex hashes once a leaf's signature has been folded into its parent's digest.

Definition of done

report streams: group in SQL with ORDER BY size DESC, head, tail, path over rows where head is non-empty, add the supporting index, and emit each group as its boundary is crossed. It must never hold more than one group. Output must remain byte-identical to today's for a given database — the existing report tests must pass unchanged.
trees stops retaining the record slice and the hex hash strings; leaf signatures are folded into parent digests as rows stream in, so peak memory is proportional to the directory count, not the file count.
Measured, not asserted: build a synthetic database of 10M rows, record peak RSS for both commands before and after, and put the numbers in the PR description. README §Design gets a sentence stating the reporting commands' memory behaviour at scale.
Determinism is preserved exactly — README §report mode and §trees mode both require identical output for identical database contents regardless of insertion order.
make check green, and make test still finishes in under 20 seconds (build the 10M-row database in a throwaway benchmark, not in the test suite).

`report.go:32-48` calls `loadFileRows` (`db.go:207-239`), which materializes the entire `files` table as a `[]scanRec`. `trees.go:38-43` does the same and then builds the full node graph on top of it. Each record carries two 64-character hex hash strings plus the path plus slice and string headers — call it 210 bytes at minimum, before Go's allocator overhead. At the stated scale of ~10M files that is over 2 GB resident for `report` alone, on a machine that may be the same storage server running the scan. This is the one design constraint the scan phase goes out of its way to honour and the reporting phase ignores. README §Design goal 2 is explicit: "Holding one small record (path, size, mtime) per file in memory during a scan is acceptable; holding every file's hashes is not (they stay in the database)." Note the two commands have different ceilings. `report` never needs more than one duplicate group at a time and can stream. `trees` genuinely needs the whole hierarchy to compute Merkle digests bottom-up, but it does not need to retain the hex hashes once a leaf's signature has been folded into its parent's digest. ## Definition of done 1. `report` streams: group in SQL with `ORDER BY size DESC, head, tail, path` over rows where `head` is non-empty, add the supporting index, and emit each group as its boundary is crossed. It must never hold more than one group. Output must remain byte-identical to today's for a given database — the existing report tests must pass unchanged. 2. `trees` stops retaining the record slice and the hex hash strings; leaf signatures are folded into parent digests as rows stream in, so peak memory is proportional to the directory count, not the file count. 3. Measured, not asserted: build a synthetic database of 10M rows, record peak RSS for both commands before and after, and put the numbers in the PR description. README §Design gets a sentence stating the reporting commands' memory behaviour at scale. 4. Determinism is preserved exactly — README §report mode and §trees mode both require identical output for identical database contents regardless of insertion order. 5. `make check` green, and `make test` still finishes in under 20 seconds (build the 10M-row database in a throwaway benchmark, not in the test suite).

clawbot added this to the 1.0.0 milestone 2026-08-09 03:45:49 +02:00

clawbot referenced this issue

2026-08-09 03:47:54 +02:00

Land golangci-lint v2.12.2 and the canonical .golangci.yml on main #3

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/sfdupes#14