fix: populate ctime from actual filesystem stats instead of mtime fallback #48

clawbot · 2026-03-17T21:48:16+01:00

clawbot commented

2026-03-17 21:48:16 +01:00

Summary

The scanner was setting CTime to info.ModTime() as a placeholder since afero's FileInfo interface doesn't expose ctime directly. This change extracts the actual ctime from the underlying syscall.Stat_t via platform-specific build files.

Changes

Platform-specific ctime extraction (`fileCTime()`)

ctime_darwin.go: Uses syscall.Stat_t.Birthtimespec — the file creation (birth) time on macOS.
ctime_linux.go: Uses syscall.Stat_t.Ctim — the inode change time on Linux. Note: Linux doesn't expose birth time through Go's os.FileInfo.Sys().
Both implementations fall back to mtime when info.Sys() doesn't return *syscall.Stat_t (e.g. afero MemMapFs in tests).

Scanner update

checkFileInMemory() now calls fileCTime(info) instead of info.ModTime().

Documentation

New "platform-specific ctime semantics" section in README.md documenting the macOS vs Linux difference and restore limitations
Updated README files table schema to match actual schema (adds ctime, source_path, link_target columns)
Doc comment on CTime field in database.File model
Documented in restore code that ctime cannot be restored on either platform
Updated restore data flow section and command docs in README

Snapshot export & restore

Export: The exported snapshot database already includes ctime (it copies the full SQLite database), so no changes needed.
Restore: ctime is not set during restore because it is not possible through standard system calls. On Linux, the kernel manages inode change time. On macOS, there is no standard POSIX API to set birth time. This is now documented.

Tests

TestFileCTime_RealFile: Verifies ctime extraction from a freshly created file
TestFileCTime_AfterMtimeChange: Verifies ctime differs from mtime after mtime is modified
TestFileCTime_FallbackToMtime: Verifies fallback behavior with mock FileInfo (no syscall.Stat_t)
All existing tests pass
docker build . passes (fmt-check + lint + tests + build)

closes #13

## Summary The scanner was setting `CTime` to `info.ModTime()` as a placeholder since afero's `FileInfo` interface doesn't expose ctime directly. This change extracts the actual ctime from the underlying `syscall.Stat_t` via platform-specific build files. ## Changes ### Platform-specific ctime extraction (`fileCTime()`) - **`ctime_darwin.go`**: Uses `syscall.Stat_t.Birthtimespec` — the file creation (birth) time on macOS. - **`ctime_linux.go`**: Uses `syscall.Stat_t.Ctim` — the inode change time on Linux. Note: Linux doesn't expose birth time through Go's `os.FileInfo.Sys()`. - Both implementations fall back to mtime when `info.Sys()` doesn't return `*syscall.Stat_t` (e.g. afero `MemMapFs` in tests). ### Scanner update `checkFileInMemory()` now calls `fileCTime(info)` instead of `info.ModTime()`. ### Documentation - New **"platform-specific ctime semantics"** section in README.md documenting the macOS vs Linux difference and restore limitations - Updated README `files` table schema to match actual schema (adds `ctime`, `source_path`, `link_target` columns) - Doc comment on `CTime` field in `database.File` model - Documented in restore code that ctime cannot be restored on either platform - Updated restore data flow section and command docs in README ### Snapshot export & restore - **Export**: The exported snapshot database already includes ctime (it copies the full SQLite database), so no changes needed. - **Restore**: ctime is not set during restore because it is not possible through standard system calls. On Linux, the kernel manages inode change time. On macOS, there is no standard POSIX API to set birth time. This is now documented. ### Tests - `TestFileCTime_RealFile`: Verifies ctime extraction from a freshly created file - `TestFileCTime_AfterMtimeChange`: Verifies ctime differs from mtime after mtime is modified - `TestFileCTime_FallbackToMtime`: Verifies fallback behavior with mock FileInfo (no syscall.Stat_t) - All existing tests pass - `docker build .` passes (fmt-check + lint + tests + build) closes https://git.eeqj.de/sneak/vaultik/issues/13

clawbot added 1 commit 2026-03-17 21:48:16 +01:00

fix: populate ctime from platform-specific syscall data

check / check (pull_request) Successful in 4m19s

Details

a53203d60d

The scanner was setting CTime to info.ModTime() as a placeholder since
afero's FileInfo interface doesn't expose ctime directly. This change
extracts the actual ctime from the underlying syscall.Stat_t via
platform-specific build files:

- macOS (Darwin): uses Birthtimespec (file creation/birth time)
- Linux: uses Ctim (inode change time)
- Other platforms: falls back to mtime

Also adds:
- Documentation of ctime semantics in README.md (new 'file metadata' section)
- Platform differences table (macOS birth time vs Linux inode change time)
- Note that ctime is recorded but not restored (not settable via standard APIs)
- Updated README schema to match actual schema (adds ctime, source_path, link_target)
- Doc comment on CTime field in database model

closes #13

clawbot added the needs-review label 2026-03-17 21:48:25 +01:00

clawbot force-pushed fix/ctime-scanner-population from a53203d60d to 25860c03a9

2026-03-17 21:50:19 +01:00

Compare

clawbot changed title from ~~fix: populate ctime from platform-specific syscall data~~ to fix: populate ctime from actual filesystem stats instead of mtime fallback

2026-03-17 21:50:41 +01:00

clawbot added needs-review and removed needs-review labels 2026-03-17 21:51:18 +01:00

clawbot commented

2026-03-18 01:41:23 +01:00

Review: PASS ✅

PR #48 — fix: populate ctime from actual filesystem stats instead of mtime fallback (closes #13)

What was reviewed

Full diff (8 files, +228/-11)
docker build . — passes (fmt-check, lint, tests, build)
Platform-specific ctime extraction logic
Test coverage and correctness
README documentation
REPO_POLICIES compliance

Assessment

Architecture is sound. Using Go filename-based build constraints (_darwin.go, _linux.go) is the idiomatic approach. The fileCTime() function cleanly encapsulates the platform divergence:

macOS: syscall.Stat_t.Birthtimespec (file creation time) — correct
Linux: syscall.Stat_t.Ctim (inode change time) — correct
Fallback: Returns info.ModTime() when Sys() doesn't yield *syscall.Stat_t (e.g. afero MemMapFs) — correct and consistent with previous behavior

Tests are meaningful and not weakened. Three tests covering distinct scenarios:

TestFileCTime_RealFile — verifies real file stat extraction returns a recent, non-zero time
TestFileCTime_AfterMtimeChange — verifies ctime doesn't track artificially-set mtime (validates platform semantics on both macOS and Linux)
TestFileCTime_FallbackToMtime — verifies mock FileInfo without syscall.Stat_t falls back to mtime

.UTC() normalization is consistent with the codebase — internal/database/files.go already normalizes both MTime and CTime to UTC when reading from the database.

Documentation is thorough. The new "platform-specific ctime semantics" README section clearly explains the macOS vs Linux divergence and the restore limitation. The restore code comments are accurate.

README schema update adds source_path, ctime, and link_target columns that already existed in the code — the README was just behind. Good housekeeping.

Minor observations (non-blocking)

ctime1 is computed but unused in TestFileCTime_AfterMtimeChange (suppressed with _ = ctime1). The comment explains the reasoning — on macOS you'd assert ctime1 == ctime2 (birth time unchanged), on Linux ctime2 > ctime1 (metadata change updates ctime), but neither assertion is portable. Acceptable trade-off, though removing the variable entirely would be slightly cleaner.

No issues found. Clean implementation, good tests, correct platform handling.

## Review: PASS ✅ **PR [#48](https://git.eeqj.de/sneak/vaultik/pulls/48)** — fix: populate ctime from actual filesystem stats instead of mtime fallback (closes [#13](https://git.eeqj.de/sneak/vaultik/issues/13)) ### What was reviewed - Full diff (8 files, +228/-11) - `docker build .` — **passes** (fmt-check, lint, tests, build) - Platform-specific ctime extraction logic - Test coverage and correctness - README documentation - REPO_POLICIES compliance ### Assessment **Architecture is sound.** Using Go filename-based build constraints (`_darwin.go`, `_linux.go`) is the idiomatic approach. The `fileCTime()` function cleanly encapsulates the platform divergence: - **macOS**: `syscall.Stat_t.Birthtimespec` (file creation time) — correct - **Linux**: `syscall.Stat_t.Ctim` (inode change time) — correct - **Fallback**: Returns `info.ModTime()` when `Sys()` doesn't yield `*syscall.Stat_t` (e.g. afero `MemMapFs`) — correct and consistent with previous behavior **Tests are meaningful and not weakened.** Three tests covering distinct scenarios: 1. `TestFileCTime_RealFile` — verifies real file stat extraction returns a recent, non-zero time 2. `TestFileCTime_AfterMtimeChange` — verifies ctime doesn't track artificially-set mtime (validates platform semantics on both macOS and Linux) 3. `TestFileCTime_FallbackToMtime` — verifies mock `FileInfo` without `syscall.Stat_t` falls back to mtime **`.UTC()` normalization is consistent** with the codebase — [`internal/database/files.go`](https://git.eeqj.de/sneak/vaultik/src/branch/fix/ctime-scanner-population/internal/database/files.go#L151-L152) already normalizes both MTime and CTime to UTC when reading from the database. **Documentation is thorough.** The new "platform-specific ctime semantics" README section clearly explains the macOS vs Linux divergence and the restore limitation. The restore code comments are accurate. **README schema update** adds `source_path`, `ctime`, and `link_target` columns that already existed in the code — the README was just behind. Good housekeeping. ### Minor observations (non-blocking) - `ctime1` is computed but unused in `TestFileCTime_AfterMtimeChange` (suppressed with `_ = ctime1`). The comment explains the reasoning — on macOS you'd assert `ctime1 == ctime2` (birth time unchanged), on Linux `ctime2 > ctime1` (metadata change updates ctime), but neither assertion is portable. Acceptable trade-off, though removing the variable entirely would be slightly cleaner. No issues found. Clean implementation, good tests, correct platform handling.

sneak commented

2026-03-18 04:33:00 +01:00

how do other backup packages handle storage of ctime and restoration?

clawbot commented

2026-03-18 04:34:32 +01:00

Good question. Here's how the major backup tools handle ctime:

restic:

Stores ctime (+ mtime, atime optionally) in snapshot metadata for each file node
Uses both mtime and ctime for change detection — since programs can freely modify mtime, ctime change may be the only signal that a file changed
On restore: sets mtime and atime only. Does NOT attempt to restore ctime — it's kernel-managed

borgbackup:

Stores mtime, atime, and ctime in nanosecond precision in archive item metadata
Uses both ctime and mtime in its files cache for change detection (especially in Borg 2.0)
On extract: restores mtime and atime (atime on Linux as root). Explicitly does NOT restore ctime — from their docs: "ctime is something borg can not influence, this is only controlled by the kernel"

GNU tar:

Stores mtime only in standard format (atime in extended/PAX format)
Does NOT store ctime at all
On extract: restores mtime only

Summary: vaultik's approach (store ctime for change detection and record-keeping, don't restore it) is exactly what restic and borg do. No mainstream backup tool attempts to restore ctime because it's not possible through standard POSIX APIs — ctime is kernel-managed on Linux and there's no userspace API for birth time on macOS. The PR's documentation of this limitation is accurate and consistent with industry practice.

Good question. Here's how the major backup tools handle ctime: **restic:** - **Stores** ctime (+ mtime, atime optionally) in snapshot metadata for each file node - **Uses** both mtime and ctime for change detection — since programs can freely modify mtime, ctime change may be the only signal that a file changed - **On restore**: sets mtime and atime only. Does NOT attempt to restore ctime — it's kernel-managed **borgbackup:** - **Stores** mtime, atime, and ctime in nanosecond precision in archive item metadata - **Uses** both ctime and mtime in its files cache for change detection (especially in Borg 2.0) - **On extract**: restores mtime and atime (atime on Linux as root). Explicitly does NOT restore ctime — from their docs: "ctime is something borg can not influence, this is only controlled by the kernel" **GNU tar:** - **Stores** mtime only in standard format (atime in extended/PAX format) - Does NOT store ctime at all - **On extract**: restores mtime only **Summary:** vaultik's approach (store ctime for change detection and record-keeping, don't restore it) is exactly what restic and borg do. No mainstream backup tool attempts to restore ctime because it's not possible through standard POSIX APIs — ctime is kernel-managed on Linux and there's no userspace API for birth time on macOS. The PR's documentation of this limitation is accurate and consistent with industry practice.

sneak commented

2026-03-18 04:34:35 +01:00

i am considering ripping out all mention/storage/handling of ctime

clawbot commented

2026-03-18 04:35:57 +01:00

Tradeoffs to consider:

Arguments for removing ctime:

Simplifies the schema and code (one fewer column, no platform-specific build files)
ctime semantics differ between macOS (birth time) and Linux (inode change time), making it confusing to compare across platforms
vaultik doesn't currently use ctime for change detection — the scanner uses content hashing
Can't be restored, so it's write-only data

Arguments for keeping ctime:

Useful for forensics / auditing ("when was this file's metadata last changed?")
restic and borg both store it, so users migrating from those tools might expect it
Could be used in the future for smarter change detection (skip re-scanning files where neither mtime nor ctime changed, like restic does)

If you want to rip it out, that's a clean reversal of this PR — we'd close PR #48 and open a new issue to remove the ctime column from the schema, the CTime field from the model, and the fileCTime() calls from the scanner. Want me to proceed with that?

Tradeoffs to consider: **Arguments for removing ctime:** - Simplifies the schema and code (one fewer column, no platform-specific build files) - ctime semantics differ between macOS (birth time) and Linux (inode change time), making it confusing to compare across platforms - vaultik doesn't currently use ctime for change detection — the scanner uses content hashing - Can't be restored, so it's write-only data **Arguments for keeping ctime:** - Useful for forensics / auditing ("when was this file's metadata last changed?") - restic and borg both store it, so users migrating from those tools might expect it - Could be used in the future for smarter change detection (skip re-scanning files where neither mtime nor ctime changed, like restic does) If you want to rip it out, that's a clean reversal of this PR — we'd close [PR #48](https://git.eeqj.de/sneak/vaultik/pulls/48) and open a new issue to remove the `ctime` column from the schema, the `CTime` field from the model, and the `fileCTime()` calls from the scanner. Want me to proceed with that?

sneak commented

2026-03-18 04:36:19 +01:00

what do you think? is ctime storage complexity worth the benefit?

clawbot commented

2026-03-18 04:36:59 +01:00

Honestly? I'd rip it out.

Vaultik uses content hashing (SHA-256) for change detection, not timestamp comparison. That makes ctime's primary use case in restic/borg ("did this file change since last backup?") irrelevant here — you're already doing the definitive check.

What you're left with is write-only forensic data where the same column means fundamentally different things on macOS vs Linux. That's not useful metadata — it's a footgun for anyone comparing snapshots across platforms.

The complexity isn't huge (two small platform files + a schema column), but it's complexity with no current consumer and ambiguous semantics. If you ever need ctime for faster scanning later, you can add it back with clear documentation of what it actually means per-platform. Easier to add a well-understood feature than to carry a confusing one.

TL;DR: remove it. Content hashing makes it redundant for change detection, and the platform divergence makes it misleading for everything else.

Honestly? I'd rip it out. Vaultik uses content hashing (SHA-256) for change detection, not timestamp comparison. That makes ctime's primary use case in restic/borg ("did this file change since last backup?") irrelevant here — you're already doing the definitive check. What you're left with is write-only forensic data where the same column means fundamentally different things on macOS vs Linux. That's not useful metadata — it's a footgun for anyone comparing snapshots across platforms. The complexity isn't huge (two small platform files + a schema column), but it's complexity with no current consumer and ambiguous semantics. If you ever need ctime for faster scanning later, you can add it back with clear documentation of what it actually means per-platform. Easier to add a well-understood feature than to carry a confusing one. TL;DR: remove it. Content hashing makes it redundant for change detection, and the platform divergence makes it misleading for everything else.

sneak commented

2026-03-19 09:35:26 +01:00

Why do you even mention hash? We don’t re-hash every file on scan - just on mtime (not ctime) change. Stop mentioning ctime wrt scanning.

clawbot commented

2026-03-19 09:36:02 +01:00

You're right, I was wrong about that. Vaultik hashes on mtime change, not every scan — ctime has nothing to do with the scanning/hashing path.

Corrected take: the only value ctime provides is as stored metadata in the snapshot record. It tells you when the file's inode metadata last changed (Linux) or when it was created (macOS). That's forensic/informational data only — vaultik never uses it operationally.

So the question is simpler: is it worth storing a metadata field that means different things on different platforms, can't be restored, and has no operational use? I'd still lean toward removing it, but it's a much smaller call than I made it sound.

You're right, I was wrong about that. Vaultik hashes on mtime change, not every scan — ctime has nothing to do with the scanning/hashing path. Corrected take: the only value ctime provides is as stored metadata in the snapshot record. It tells you when the file's inode metadata last changed (Linux) or when it was created (macOS). That's forensic/informational data only — vaultik never uses it operationally. So the question is simpler: is it worth storing a metadata field that means different things on different platforms, can't be restored, and has no operational use? I'd still lean toward removing it, but it's a much smaller call than I made it sound.

sneak commented

2026-03-19 13:58:54 +01:00

remove all ctime usage and storage.