sneak
e0d578a707
scan will maintain a persistent database of file signatures (default /var/lib/sfdupes/db.sqlite, overridable via SFDUPES_DATABASE) that survives between runs; rescans hash only new or changed files (mtime/size) and remove records for vanished files, so scan can be cronned daily. report and trees will read the database instead of a scan stream.
117 lines
5.5 KiB
Markdown
117 lines
5.5 KiB
Markdown
# Workflow
|
|
|
|
- branch (from `main`)
|
|
- do the work in Next Step
|
|
- move Next Step to the top of Completed Steps
|
|
- move the top item of Future Steps into Next Step
|
|
- commit (`TODO.md` changes in the same commit as the work)
|
|
- merge to `main` if the branch is not protected, otherwise open a PR
|
|
- push
|
|
|
|
# Status
|
|
|
|
- pre-1.0
|
|
|
|
# Next Step
|
|
|
|
- persistent scan database (branch `persistent-database`): `scan`
|
|
maintains a SQLite database that survives between runs so it can be
|
|
cronned daily; `report` and `trees` read the database instead of a
|
|
scan stream. Database path defaults to `/var/lib/sfdupes/db.sqlite`,
|
|
overridable via `SFDUPES_DATABASE`. Plan:
|
|
- [x] update `README.md` (the authoritative spec) for the database:
|
|
new Database section, revised `scan`/`report`/`trees` modes,
|
|
dependency list (`modernc.org/sqlite`, pure Go, cgo stays
|
|
disabled), exit codes, smoke test with incremental steps
|
|
- [ ] add `modernc.org/sqlite` (hash-pinned via `go.sum`)
|
|
- [ ] `db.go`: path resolution (`SFDUPES_DATABASE` env, default
|
|
`/var/lib/sfdupes/db.sqlite`), open/create with WAL +
|
|
busy-timeout pragmas and `PRAGMA user_version` schema check,
|
|
`files` table (path BLOB primary key, size, mtime, head,
|
|
tail), load-all-rows, and single-transaction apply of
|
|
upserts/deletes
|
|
- [ ] `scan.go`: resolve operands to absolute paths; diff stat
|
|
results against loaded rows (reuse hashes when size matches
|
|
and mtime is not newer); hash only new/changed files; delete
|
|
rows under the scanned operands not successfully processed;
|
|
leave rows outside the operands untouched; new summary line
|
|
(added/updated/removed/unchanged/skipped); "update" progress
|
|
pass for the database write
|
|
- [ ] `report.go`/`trees.go`: read records from the database (no
|
|
positional args, no stream parsing); drop the NUL-stream
|
|
parser and malformed-record handling
|
|
- [ ] `main.go`: updated usage strings; `report`/`trees` take no
|
|
args (usage error otherwise)
|
|
- [ ] tests: db open/env-override/schema tests; incremental scan
|
|
tests (add, mtime update, delete, unchanged-hash-reuse,
|
|
out-of-scope rows untouched, error paths); rework pipeline
|
|
test to go through the database; keep walk/stat/hash/grouping
|
|
unit tests
|
|
- [ ] `.gitignore`: local `*.sqlite` artifacts
|
|
- [ ] `make check`, `make docker`, and the README smoke test pass
|
|
|
|
# Completed Steps
|
|
|
|
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
|
|
`v0.0.1`, and push `main` plus tags (2026-07-23)
|
|
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
|
|
`PATH...` operands via cobra flags replacing the `/srv` `-root`
|
|
default; new `-x`/`--one-file-system` flag (GNU convention) to stop
|
|
at filesystem boundaries, which are crossed by default
|
|
- bring the repo into full policy compliance (2026-07-23, branch
|
|
`repo-policy-compliance`; checklist below)
|
|
- `git init` with README-only first commit; code baseline committed on
|
|
`main` (2026-07-22)
|
|
- implement `scan`, `report`, and `trees` subcommands (pre-git history)
|
|
|
|
# Future Steps
|
|
|
|
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
|
entrypoints like the other managed repos
|
|
- possible later features (explicitly out of scope per README):
|
|
full-content verification of candidates, removal-script helpers
|
|
|
|
# Repo Policy Compliance
|
|
|
|
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing
|
|
repo checklist, and the Go styleguide. Code is already gofmt-clean, so no
|
|
standalone formatting commit is needed.
|
|
|
|
- [x] `.gitignore` missing — the compiled `sfdupes` binary and
|
|
`files.dat` sit untracked in the tree; needs OS/editor/Go
|
|
artifacts plus secrets patterns
|
|
- [x] `.editorconfig` missing
|
|
- [x] `LICENSE` missing and README has no License section (MIT assumed
|
|
from house convention — user to confirm)
|
|
- [x] `REPO_POLICIES.md` missing from repo root
|
|
- [x] `.golangci.yml` missing (install canonical copy); code must then
|
|
pass `make lint` (150 findings fixed; `make lint` is clean)
|
|
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`,
|
|
`fmt-check`, `docker`, `hooks`; `check` currently depends on
|
|
`build`, which writes the binary (`make check` must not modify
|
|
files)
|
|
- [x] no tests — `go test ./...` has nothing to run; policy requires
|
|
real tests with a 30-second timeout and the conditional `-v`
|
|
rerun pattern (suite covers parsing, grouping, digests,
|
|
suppression, hashing, and the scan pipeline; 64% coverage)
|
|
- [x] `Dockerfile` missing — Go multistage with hash-pinned images:
|
|
fail-fast lint stage, build stage running `make check`
|
|
- [x] `.dockerignore` missing
|
|
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push,
|
|
checkout action pinned by commit SHA)
|
|
- [x] README lacks required sections: Description first line
|
|
(name/purpose/category/license/author), Getting Started,
|
|
Rationale, TODO, License, Author
|
|
- [x] README non-goal "no git repository setup and no CI" is stale now
|
|
that the repo is under git with CI
|
|
- [x] pre-commit hook not installed (`make hooks` once the target
|
|
exists)
|
|
|
|
Accepted divergences (no action):
|
|
|
|
- flat single-package layout with `.go` files in the repo root — fine
|
|
for a small single-binary tool per the Go styleguide; the tracker
|
|
audit agrees
|
|
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0`
|
|
(pure-Go builds) and the race detector requires cgo
|