Make phases scan-wide, walk operands concurrently, batch updates
All checks were successful
check / check (push) Successful in 5s
All checks were successful
check / check (push) Successful in 5s
All PATH operands belong to a single scan: every operand seeds the shared walk worker pool, and each pass (walk, stat, hash, update) runs exactly once over the whole scan, so pass totals, percentages, and ETAs are scan-global. The per-operand walk/hash/update cycles and their stderr operand announcements are gone; duplicate paths from overlapping operands are deduplicated before stat. The update pass now commits in batched transactions (10k changes per batch) instead of one scan-wide transaction: the filesystem is authoritative and the database is an eventually-consistent reflection of it, so scan-level atomicity buys nothing, while batches keep the WAL small and let concurrent reports observe progress.
This commit is contained in:
35
db.go
35
db.go
@@ -8,6 +8,7 @@ import (
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
|
||||
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
|
||||
@@ -237,12 +238,40 @@ func loadFileRows(db *sql.DB) ([]scanRec, error) {
|
||||
return recs, nil
|
||||
}
|
||||
|
||||
// updateBatchSize is the number of record changes committed per
|
||||
// transaction during the update pass. The filesystem is authoritative
|
||||
// and the database an eventually-consistent reflection of it, so
|
||||
// scan-level atomicity is not required; smaller transactions keep the
|
||||
// WAL small and let concurrent reports observe progress.
|
||||
const updateBatchSize = 10000
|
||||
|
||||
// applyChanges writes one scan's database changes — upserts for new and
|
||||
// changed files, deletes for vanished ones — in a single transaction,
|
||||
// so a concurrent report never observes a half-finished scan. Progress
|
||||
// is rendered on prog (one increment per change).
|
||||
// changed files, deletes for vanished ones — in batched transactions.
|
||||
// Progress is rendered on prog (one increment per change).
|
||||
func applyChanges(db *sql.DB, upserts []scanRec, deletes []string,
|
||||
prog *progress,
|
||||
) error {
|
||||
for batch := range slices.Chunk(upserts, updateBatchSize) {
|
||||
err := applyBatch(db, batch, nil, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
for batch := range slices.Chunk(deletes, updateBatchSize) {
|
||||
err := applyBatch(db, nil, batch, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// applyBatch commits one batch of upserts and deletes in a single
|
||||
// transaction.
|
||||
func applyBatch(db *sql.DB, upserts []scanRec, deletes []string,
|
||||
prog *progress,
|
||||
) error {
|
||||
ctx := context.Background()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user