sneak/mfer
1
0
Fork 0
Code Issues 4 Pull Requests 1 Releases Wiki Activity

Compare commits

base: sneak:019fe41c3d4a6ffa190497d38096b1bd95533f77
Branches Tags
sneak:main
sneak:next
sneak:sneak-patch-1
sneak:readme
..
compare: sneak:e25e309581db38f2e578cf36544dee5668e6dc02
Branches Tags
sneak:next
sneak:sneak-patch-1
sneak:main
sneak:readme

No commits in common. "019fe41c3d4a6ffa190497d38096b1bd95533f77" and "e25e309581db38f2e578cf36544dee5668e6dc02" have entirely different histories.
019fe41c3d ... e25e309581

10 changed files with 147 additions and 174 deletions
Show Stats Expand all files Collapse all files

2
.gitignore vendored
View File

@ -1,4 +1,4 @@
/bin/ /mfer.cmd
/tmp /tmp
*.tmp *.tmp
*.dockerimage *.dockerimage

20
CLAUDE.md
View File

@ -1,20 +1,20 @@
# Important Rules # Important Rules
- when fixing a bug, write a failing test FIRST. only after the test fails, write * when fixing a bug, write a failing test FIRST. only after the test fails, write
the code to fix the bug. then ensure the test passes. leave the test in the code to fix the bug. then ensure the test passes. leave the test in
place and commit it with the bugfix. don't run shell commands to test place and commit it with the bugfix. don't run shell commands to test
bugfixes or reproduce bugs. write tests! bugfixes or reproduce bugs. write tests!
- never, ever mention claude or anthropic in commit messages. do not use attribution * never, ever mention claude or anthropic in commit messages. do not use attribution
- after each change, run "make fmt". * after each change, run "make fmt".
- after each change, run "make test" and ensure all tests pass. * after each change, run "make test" and ensure all tests pass.
- after each change, run "make lint" and ensure no linting errors. fix any * after each change, run "make lint" and ensure no linting errors. fix any
you find, one by one. you find, one by one.
- after each change, commit the files you've changed. push after * after each change, commit the files you've changed. push after
committing. committing.
- NEVER use `git add -A`. always add only individual files that you've changed. * NEVER use `git add -A`. always add only individual files that you've changed.

8
Makefile
View File

@ -17,7 +17,7 @@ GOFLAGS := -ldflags "$(GOLDFLAGS)"
default: fmt test default: fmt test
run: ./bin/mfer run: ./mfer.cmd
./$< ./$<
./$< gen ./$< gen
@ -38,12 +38,12 @@ devprereqs:
mfer/mf.pb.go: mfer/mf.proto mfer/mf.pb.go: mfer/mf.proto
cd mfer && go generate . cd mfer && go generate .
bin/mfer: $(SOURCEFILES) mfer/mf.pb.go mfer.cmd: $(SOURCEFILES) mfer/mf.pb.go
protoc --version protoc --version
cd cmd/mfer && go build -tags urfave_cli_no_docs -o ../../bin/mfer $(GOFLAGS) . cd cmd/mfer && go build -tags urfave_cli_no_docs -o ../../mfer.cmd $(GOFLAGS) .
clean: clean:
rm -rfv mfer/*.pb.go bin/mfer cmd/mfer/mfer *.dockerimage rm -rfv mfer/*.pb.go mfer.cmd cmd/mfer/mfer *.dockerimage
fmt: mfer/mf.pb.go fmt: mfer/mf.pb.go
gofumpt -l -w mfer internal cmd gofumpt -l -w mfer internal cmd

36
README.md
View File

@ -3,25 +3,25 @@
[mfer](https://git.eeqj.de/sneak/mfer) is a reference implementation library [mfer](https://git.eeqj.de/sneak/mfer) is a reference implementation library
and thin wrapper command-line utility written in [Go](https://golang.org) and thin wrapper command-line utility written in [Go](https://golang.org)
and first published in 2022 under the [WTFPL](https://wtfpl.net) (public and first published in 2022 under the [WTFPL](https://wtfpl.net) (public
domain) license. It specifies and generates `.mf` manifest files over a domain) license. It specifies and generates `.mf` manifest files over a
directory tree of files to encapsulate metadata about them (such as directory tree of files to encapsulate metadata about them (such as
cryptographic checksums or signatures over same) to aid in archiving, cryptographic checksums or signatures over same) to aid in archiving,
downloading, and streaming, or mirroring. The manifest files' data is downloading, and streaming, or mirroring. The manifest files' data is
serialized with Google's [protobuf serialization serialized with Google's [protobuf serialization
format](https://developers.google.com/protocol-buffers). The structure of format](https://developers.google.com/protocol-buffers). The structure of
these files can be found [in the format these files can be found [in the format
specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto) specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto)
which is included in the [project which is included in the [project
repository](https://git.eeqj.de/sneak/mfer). repository](https://git.eeqj.de/sneak/mfer).
The current version is pre-1.0 and while the repo was published in 2022, The current version is pre-1.0 and while the repo was published in 2022,
there has not yet been any versioned release. [SemVer](https://semver.org) there has not yet been any versioned release. [SemVer](https://semver.org)
will be used for releases. will be used for releases.
This project was started by [@sneak](https://sneak.berlin) to scratch an This project was started by [@sneak](https://sneak.berlin) to scratch an
itch in 2022 and is currently a one-person effort, though the goal is for itch in 2022 and is currently a one-person effort, though the goal is for
this to emerge as a de-facto standard and be incorporated into other this to emerge as a de-facto standard and be incorporated into other
software. A compatible javascript library is planned. software. A compatible javascript library is planned.
# Phases # Phases
@ -57,7 +57,6 @@ Reading file contents and computing cryptographic hashes for manifest generation
## cmd/mfer/ ## cmd/mfer/
### main.go ### main.go
- **Variables** - **Variables**
- `Appname string` - Application name - `Appname string` - Application name
- `Version string` - Version string (set at build time) - `Version string` - Version string (set at build time)
@ -66,14 +65,12 @@ Reading file contents and computing cryptographic hashes for manifest generation
## internal/cli/ ## internal/cli/
### entry.go ### entry.go
- **Variables** - **Variables**
- `NO_COLOR bool` - Disables color output when NO_COLOR env var is set - `NO_COLOR bool` - Disables color output when NO_COLOR env var is set
- **Functions** - **Functions**
- `Run(Appname, Version, Gitrev string) int` - Main entry point for the CLI - `Run(Appname, Version, Gitrev string) int` - Main entry point for the CLI
### mfer.go ### mfer.go
- **Types** - **Types**
- `CLIApp struct` - Main CLI application container - `CLIApp struct` - Main CLI application container
- **Methods** - **Methods**
@ -82,7 +79,6 @@ Reading file contents and computing cryptographic hashes for manifest generation
## internal/log/ ## internal/log/
### log.go ### log.go
- **Functions** - **Functions**
- `Init()` - Initializes the logger - `Init()` - Initializes the logger
- `Info(arg string)` - Logs at info level - `Info(arg string)` - Logs at info level
@ -103,7 +99,6 @@ Reading file contents and computing cryptographic hashes for manifest generation
## internal/scanner/ ## internal/scanner/
### scanner.go ### scanner.go
- **Types** - **Types**
- `Options struct` - Options for scanner behavior - `Options struct` - Options for scanner behavior
- `IncludeDotfiles bool` - Include dot (hidden) files (excluded by default) - `IncludeDotfiles bool` - Include dot (hidden) files (excluded by default)
@ -143,7 +138,6 @@ Reading file contents and computing cryptographic hashes for manifest generation
## internal/checker/ ## internal/checker/
### checker.go ### checker.go
- **Types** - **Types**
- `Result struct` - Outcome of checking a single file - `Result struct` - Outcome of checking a single file
- `Path string` - File path from manifest - `Path string` - File path from manifest
@ -175,14 +169,12 @@ Reading file contents and computing cryptographic hashes for manifest generation
## mfer/ ## mfer/
### manifest.go ### manifest.go
- **Types** - **Types**
- `manifest struct` - Internal representation of a manifest file - `manifest struct` - Internal representation of a manifest file
- **Methods** - **Methods**
- `(*manifest) Files() []*MFFilePath` - Returns all file entries from a loaded manifest - `(*manifest) Files() []*MFFilePath` - Returns all file entries from a loaded manifest
### builder.go ### builder.go
- **Types** - **Types**
- `FileHashProgress struct` - Progress info during file hashing (BytesRead int64) - `FileHashProgress struct` - Progress info during file hashing (BytesRead int64)
- `Builder struct` - Constructs manifests by adding files one at a time - `Builder struct` - Constructs manifests by adding files one at a time
@ -190,23 +182,20 @@ Reading file contents and computing cryptographic hashes for manifest generation
- `NewBuilder() *Builder` - Creates a new Builder - `NewBuilder() *Builder` - Creates a new Builder
- **Methods** - **Methods**
- `(*Builder) AddFile(path string, size int64, mtime time.Time, reader io.Reader, progress chan<- FileHashProgress) (int64, error)` - Reads file, computes hash, adds to manifest - `(*Builder) AddFile(path string, size int64, mtime time.Time, reader io.Reader, progress chan<- FileHashProgress) (int64, error)` - Reads file, computes hash, adds to manifest
- `(*Builder) AddFileWithHash(path string, size int64, mtime time.Time, hash []byte) error` - Adds file with pre-computed hash - `(*Builder) AddFileWithHash(path string, size int64, mtime time.Time, hash []byte)` - Adds file with pre-computed hash
- `(*Builder) FileCount() int` - Returns number of files added - `(*Builder) FileCount() int` - Returns number of files added
- `(*Builder) Build(w io.Writer) error` - Finalizes and writes manifest - `(*Builder) Build(w io.Writer) error` - Finalizes and writes manifest
### serialize.go ### serialize.go
- **Constants** - **Constants**
- `MAGIC string` - Magic bytes prefix for manifest files ("ZNAVSRFG") - `MAGIC string` - Magic bytes prefix for manifest files ("ZNAVSRFG")
### deserialize.go ### deserialize.go
- **Functions** - **Functions**
- `NewManifestFromReader(input io.Reader) (*manifest, error)` - Reads and parses manifest from io.Reader - `NewManifestFromReader(input io.Reader) (*manifest, error)` - Reads and parses manifest from io.Reader
- `NewManifestFromFile(fs afero.Fs, path string) (*manifest, error)` - Reads and parses manifest from file path - `NewManifestFromFile(fs afero.Fs, path string) (*manifest, error)` - Reads and parses manifest from file path
### mf.pb.go (generated from mf.proto) ### mf.pb.go (generated from mf.proto)
- **Enum Types** - **Enum Types**
- `MFFileOuter_Version` - Outer file format version - `MFFileOuter_Version` - Outer file format version
- `MFFileOuter_VERSION_NONE` - `MFFileOuter_VERSION_NONE`
@ -252,18 +241,19 @@ Reading file contents and computing cryptographic hashes for manifest generation
# Participation # Participation
The community is as yet nonexistent so there are no defined policies or The community is as yet nonexistent so there are no defined policies or
norms yet. Primary development happens on a privately-run Gitea instance at norms yet. Primary development happens on a privately-run Gitea instance at
[https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) and issues [https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) and issues
are [tracked there](https://git.eeqj.de/sneak/mfer/issues). are [tracked there](https://git.eeqj.de/sneak/mfer/issues).
Changes must always be formatted with a standard `go fmt`, syntactically Changes must always be formatted with a standard `go fmt`, syntactically
valid, and must pass the linting defined in the repository (presently only valid, and must pass the linting defined in the repository (presently only
the `golangci-lint` defaults), which can be run with a `make lint`. The the `golangci-lint` defaults), which can be run with a `make lint`. The
`main` branch is protected and all changes must be made via [pull `main` branch is protected and all changes must be made via [pull
requests](https://git.eeqj.de/sneak/mfer/pulls) and pass CI to be merged. requests](https://git.eeqj.de/sneak/mfer/pulls) and pass CI to be merged.
Any changes submitted to this project must also be Any changes submitted to this project must also be
[WTFPL-licensed](https://wtfpl.net) to be considered. [WTFPL-licensed](https://wtfpl.net) to be considered.
# Problem Statement # Problem Statement
Given a plain URL, there is no standard way to safely and programmatically Given a plain URL, there is no standard way to safely and programmatically
@ -434,13 +424,13 @@ desired username for an account on this Gitea instance.
## Links ## Links
- Repo: [https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) * Repo: [https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer)
- Issues: [https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues) * Issues: [https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues)
# Authors # Authors
- [@sneak &lt;sneak@sneak.berlin&gt;](mailto:sneak@sneak.berlin) * [@sneak &lt;sneak@sneak.berlin&gt;](mailto:sneak@sneak.berlin)
# License # License
- [WTFPL](https://wtfpl.net) * [WTFPL](https://wtfpl.net)

19
TODO.md
View File

@ -1,19 +0,0 @@
# TODO
## Critical
- [ ] Fix broken error comparison in `internal/checker/checker.go:195` - `errors.Is(err, errors.New("file does not exist"))` always returns false because `errors.New()` creates a new instance each call
- [ ] Fix unchecked `hash.Write()` errors in `mfer/builder.go:52`, `mfer/serialize.go:56`, `internal/cli/freshen.go:340`
- [ ] Fix URL path traversal risk in `internal/cli/fetch.go:116` - path isn't URL-escaped, should use `url.JoinPath()` or proper encoding
## Important
- [ ] Fix goroutine leak in signal handler `internal/cli/gen.go:98-106` - goroutine runs until channel closed, leaks if program exits normally
- [ ] Fix timestamp precision in `mfer/serialize.go:16-22` - use `t.Nanosecond()` instead of manual calculation
- [ ] Add context cancellation check to filesystem walk in `internal/cli/freshen.go` - Ctrl-C doesn't work during scan phase
## Code Quality
- [ ] Consolidate duplicate `pathIsHidden` implementations in `internal/scanner/scanner.go:385-402` and `internal/cli/freshen.go:378-397`
- [ ] Make `TotalBytes()` in `internal/scanner/scanner.go:250-259` track total incrementally instead of recalculating on every call
- [ ] Add input validation to `AddFileWithHash()` in `mfer/builder.go:107-120` - validate path, size, and hash inputs

118
internal/cli/entry_test.go
View File

@ -65,9 +65,9 @@ func TestGenerateCommand(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test files in memory filesystem // Create test files in memory filesystem
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello world"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello world"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("test content"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("test content"), 0644))
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs)
@ -85,9 +85,9 @@ func TestGenerateAndCheckCommand(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test files with subdirectory // Create test files with subdirectory
require.NoError(t, fs.MkdirAll("/testdir/subdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir/subdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello world"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello world"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/subdir/file2.txt", []byte("test content"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/subdir/file2.txt", []byte("test content"), 0644))
// Generate manifest // Generate manifest
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs)
@ -104,8 +104,8 @@ func TestCheckCommandWithMissingFile(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test file // Create test file
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello world"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello world"), 0644))
// Generate manifest // Generate manifest
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs)
@ -125,8 +125,8 @@ func TestCheckCommandWithCorruptedFile(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test file // Create test file
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello world"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello world"), 0644))
// Generate manifest // Generate manifest
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs)
@ -134,7 +134,7 @@ func TestCheckCommandWithCorruptedFile(t *testing.T) {
require.Equal(t, 0, exitCode, "generate failed: %s", opts.Stderr.(*bytes.Buffer).String()) require.Equal(t, 0, exitCode, "generate failed: %s", opts.Stderr.(*bytes.Buffer).String())
// Corrupt the file (change content but keep same size) // Corrupt the file (change content but keep same size)
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("HELLO WORLD"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("HELLO WORLD"), 0644))
// Check manifest - should fail with hash mismatch // Check manifest - should fail with hash mismatch
opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/testdir/test.mf"}, fs) opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/testdir/test.mf"}, fs)
@ -146,8 +146,8 @@ func TestCheckCommandWithSizeMismatch(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test file // Create test file
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello world"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello world"), 0644))
// Generate manifest // Generate manifest
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs)
@ -155,7 +155,7 @@ func TestCheckCommandWithSizeMismatch(t *testing.T) {
require.Equal(t, 0, exitCode, "generate failed: %s", opts.Stderr.(*bytes.Buffer).String()) require.Equal(t, 0, exitCode, "generate failed: %s", opts.Stderr.(*bytes.Buffer).String())
// Change file size // Change file size
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("different size content here"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("different size content here"), 0644))
// Check manifest - should fail with size mismatch // Check manifest - should fail with size mismatch
opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/testdir/test.mf"}, fs) opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/testdir/test.mf"}, fs)
@ -167,8 +167,8 @@ func TestBannerOutput(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test file // Create test file
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0644))
// Run without -q to see banner // Run without -q to see banner
opts := testOpts([]string{"mfer", "generate", "-o", "/testdir/test.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-o", "/testdir/test.mf", "/testdir"}, fs)
@ -193,9 +193,9 @@ func TestGenerateExcludesDotfilesByDefault(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test files including dotfiles // Create test files including dotfiles
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/.hidden", []byte("secret"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/.hidden", []byte("secret"), 0644))
// Generate manifest without --include-dotfiles (default excludes dotfiles) // Generate manifest without --include-dotfiles (default excludes dotfiles)
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/testdir/test.mf", "/testdir"}, fs)
@ -217,9 +217,9 @@ func TestGenerateWithIncludeDotfiles(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test files including dotfiles // Create test files including dotfiles
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/.hidden", []byte("secret"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/.hidden", []byte("secret"), 0644))
// Generate manifest with --include-dotfiles // Generate manifest with --include-dotfiles
opts := testOpts([]string{"mfer", "generate", "-q", "--include-dotfiles", "-o", "/testdir/test.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "--include-dotfiles", "-o", "/testdir/test.mf", "/testdir"}, fs)
@ -236,10 +236,10 @@ func TestMultipleInputPaths(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test files in multiple directories // Create test files in multiple directories
require.NoError(t, fs.MkdirAll("/dir1", 0o755)) require.NoError(t, fs.MkdirAll("/dir1", 0755))
require.NoError(t, fs.MkdirAll("/dir2", 0o755)) require.NoError(t, fs.MkdirAll("/dir2", 0755))
require.NoError(t, afero.WriteFile(fs, "/dir1/file1.txt", []byte("content1"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/dir1/file1.txt", []byte("content1"), 0644))
require.NoError(t, afero.WriteFile(fs, "/dir2/file2.txt", []byte("content2"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/dir2/file2.txt", []byte("content2"), 0644))
// Generate manifest from multiple paths // Generate manifest from multiple paths
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/output.mf", "/dir1", "/dir2"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/output.mf", "/dir1", "/dir2"}, fs)
@ -254,9 +254,9 @@ func TestNoExtraFilesPass(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test files // Create test files
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("world"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("world"), 0644))
// Generate manifest // Generate manifest
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/manifest.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/manifest.mf", "/testdir"}, fs)
@ -273,8 +273,8 @@ func TestNoExtraFilesFail(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test files // Create test files
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0644))
// Generate manifest // Generate manifest
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/manifest.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/manifest.mf", "/testdir"}, fs)
@ -282,7 +282,7 @@ func TestNoExtraFilesFail(t *testing.T) {
require.Equal(t, 0, exitCode) require.Equal(t, 0, exitCode)
// Add an extra file after manifest generation // Add an extra file after manifest generation
require.NoError(t, afero.WriteFile(fs, "/testdir/extra.txt", []byte("extra"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/extra.txt", []byte("extra"), 0644))
// Check with --no-extra-files (should fail - extra file exists) // Check with --no-extra-files (should fail - extra file exists)
opts = testOpts([]string{"mfer", "check", "-q", "--no-extra-files", "--base", "/testdir", "/manifest.mf"}, fs) opts = testOpts([]string{"mfer", "check", "-q", "--no-extra-files", "--base", "/testdir", "/manifest.mf"}, fs)
@ -294,9 +294,9 @@ func TestNoExtraFilesWithSubdirectory(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test files with subdirectory // Create test files with subdirectory
require.NoError(t, fs.MkdirAll("/testdir/subdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir/subdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/subdir/file2.txt", []byte("world"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/subdir/file2.txt", []byte("world"), 0644))
// Generate manifest // Generate manifest
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/manifest.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/manifest.mf", "/testdir"}, fs)
@ -304,7 +304,7 @@ func TestNoExtraFilesWithSubdirectory(t *testing.T) {
require.Equal(t, 0, exitCode) require.Equal(t, 0, exitCode)
// Add extra file in subdirectory // Add extra file in subdirectory
require.NoError(t, afero.WriteFile(fs, "/testdir/subdir/extra.txt", []byte("extra"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/subdir/extra.txt", []byte("extra"), 0644))
// Check with --no-extra-files (should fail) // Check with --no-extra-files (should fail)
opts = testOpts([]string{"mfer", "check", "-q", "--no-extra-files", "--base", "/testdir", "/manifest.mf"}, fs) opts = testOpts([]string{"mfer", "check", "-q", "--no-extra-files", "--base", "/testdir", "/manifest.mf"}, fs)
@ -316,8 +316,8 @@ func TestCheckWithoutNoExtraFilesIgnoresExtra(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test file // Create test file
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0644))
// Generate manifest // Generate manifest
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/manifest.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/manifest.mf", "/testdir"}, fs)
@ -325,7 +325,7 @@ func TestCheckWithoutNoExtraFilesIgnoresExtra(t *testing.T) {
require.Equal(t, 0, exitCode) require.Equal(t, 0, exitCode)
// Add extra file // Add extra file
require.NoError(t, afero.WriteFile(fs, "/testdir/extra.txt", []byte("extra"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/extra.txt", []byte("extra"), 0644))
// Check WITHOUT --no-extra-files (should pass - extra files ignored) // Check WITHOUT --no-extra-files (should pass - extra files ignored)
opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/manifest.mf"}, fs) opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/manifest.mf"}, fs)
@ -337,8 +337,8 @@ func TestGenerateAtomicWriteNoTempFileOnSuccess(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test file // Create test file
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0644))
// Generate manifest // Generate manifest
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/output.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/output.mf", "/testdir"}, fs)
@ -360,11 +360,11 @@ func TestGenerateAtomicWriteOverwriteWithForce(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test file // Create test file
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0644))
// Create existing manifest with different content // Create existing manifest with different content
require.NoError(t, afero.WriteFile(fs, "/output.mf", []byte("old content"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/output.mf", []byte("old content"), 0644))
// Generate manifest with --force // Generate manifest with --force
opts := testOpts([]string{"mfer", "generate", "-q", "-f", "-o", "/output.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-f", "-o", "/output.mf", "/testdir"}, fs)
@ -386,11 +386,11 @@ func TestGenerateFailsWithoutForceWhenOutputExists(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test file // Create test file
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0644))
// Create existing manifest // Create existing manifest
require.NoError(t, afero.WriteFile(fs, "/output.mf", []byte("existing"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/output.mf", []byte("existing"), 0644))
// Generate manifest WITHOUT --force (should fail) // Generate manifest WITHOUT --force (should fail)
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/output.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/output.mf", "/testdir"}, fs)
@ -411,8 +411,8 @@ func TestGenerateAtomicWriteUsesTemp(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create test file // Create test file
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("hello"), 0644))
// Generate manifest // Generate manifest
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/output.mf", "/testdir"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/output.mf", "/testdir"}, fs)
@ -464,8 +464,8 @@ func TestGenerateAtomicWriteCleansUpOnError(t *testing.T) {
baseFs := afero.NewMemMapFs() baseFs := afero.NewMemMapFs()
// Create test files - need enough content to trigger the write failure // Create test files - need enough content to trigger the write failure
require.NoError(t, baseFs.MkdirAll("/testdir", 0o755)) require.NoError(t, baseFs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(baseFs, "/testdir/file1.txt", []byte("hello world this is a test file"), 0o644)) require.NoError(t, afero.WriteFile(baseFs, "/testdir/file1.txt", []byte("hello world this is a test file"), 0644))
// Wrap with failing writer that fails after writing some bytes // Wrap with failing writer that fails after writing some bytes
fs := &failingWriterFs{Fs: baseFs, failAfter: 10} fs := &failingWriterFs{Fs: baseFs, failAfter: 10}
@ -489,8 +489,8 @@ func TestGenerateValidatesInputPaths(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
// Create one valid directory // Create one valid directory
require.NoError(t, fs.MkdirAll("/validdir", 0o755)) require.NoError(t, fs.MkdirAll("/validdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/validdir/file.txt", []byte("content"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/validdir/file.txt", []byte("content"), 0644))
t.Run("nonexistent path fails fast", func(t *testing.T) { t.Run("nonexistent path fails fast", func(t *testing.T) {
opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/output.mf", "/nonexistent"}, fs) opts := testOpts([]string{"mfer", "generate", "-q", "-o", "/output.mf", "/nonexistent"}, fs)
@ -527,7 +527,7 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
// Create many small files with random names to generate a ~1MB manifest // Create many small files with random names to generate a ~1MB manifest
// Each manifest entry is roughly 50-60 bytes, so we need ~20000 files // Each manifest entry is roughly 50-60 bytes, so we need ~20000 files
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
numFiles := 20000 numFiles := 20000
for i := 0; i < numFiles; i++ { for i := 0; i < numFiles; i++ {
@ -536,7 +536,7 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
// Small random content // Small random content
content := make([]byte, 16+rng.Intn(48)) content := make([]byte, 16+rng.Intn(48))
rng.Read(content) rng.Read(content)
require.NoError(t, afero.WriteFile(fs, filename, content, 0o644)) require.NoError(t, afero.WriteFile(fs, filename, content, 0644))
} }
// Generate manifest outside of testdir // Generate manifest outside of testdir
@ -551,7 +551,7 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
t.Logf("manifest size: %d bytes (%d files)", len(validManifest), numFiles) t.Logf("manifest size: %d bytes (%d files)", len(validManifest), numFiles)
// First corruption: truncate the manifest // First corruption: truncate the manifest
require.NoError(t, afero.WriteFile(fs, "/manifest.mf", validManifest[:len(validManifest)/2], 0o644)) require.NoError(t, afero.WriteFile(fs, "/manifest.mf", validManifest[:len(validManifest)/2], 0644))
// Check should fail with truncated manifest // Check should fail with truncated manifest
opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/manifest.mf"}, fs) opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/manifest.mf"}, fs)
@ -559,7 +559,7 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
assert.Equal(t, 1, exitCode, "check should fail with truncated manifest") assert.Equal(t, 1, exitCode, "check should fail with truncated manifest")
// Verify check passes with valid manifest // Verify check passes with valid manifest
require.NoError(t, afero.WriteFile(fs, "/manifest.mf", validManifest, 0o644)) require.NoError(t, afero.WriteFile(fs, "/manifest.mf", validManifest, 0644))
opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/manifest.mf"}, fs) opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/manifest.mf"}, fs)
exitCode = RunWithOptions(opts) exitCode = RunWithOptions(opts)
require.Equal(t, 0, exitCode, "check should pass with valid manifest") require.Equal(t, 0, exitCode, "check should pass with valid manifest")
@ -579,7 +579,7 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
} }
corrupted[offset] = newByte corrupted[offset] = newByte
require.NoError(t, afero.WriteFile(fs, "/manifest.mf", corrupted, 0o644)) require.NoError(t, afero.WriteFile(fs, "/manifest.mf", corrupted, 0644))
// Check should fail with corrupted manifest // Check should fail with corrupted manifest
opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/manifest.mf"}, fs) opts = testOpts([]string{"mfer", "check", "-q", "--base", "/testdir", "/manifest.mf"}, fs)
@ -588,6 +588,6 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
i, offset, originalByte, newByte) i, offset, originalByte, newByte)
// Restore valid manifest for next iteration // Restore valid manifest for next iteration
require.NoError(t, afero.WriteFile(fs, "/manifest.mf", validManifest, 0o644)) require.NoError(t, afero.WriteFile(fs, "/manifest.mf", validManifest, 0644))
} }
} }

2
internal/cli/fetch.go
View File

@ -257,7 +257,7 @@ func downloadFile(fileURL, localPath string, entry *mfer.MFFilePath, progress ch
// Create parent directories if needed // Create parent directories if needed
dir := filepath.Dir(localPath) dir := filepath.Dir(localPath)
if dir != "" && dir != "." { if dir != "" && dir != "." {
if err := os.MkdirAll(dir, 0o755); err != nil { if err := os.MkdirAll(dir, 0755); err != nil {
return err return err
} }
} }

27
internal/cli/fetch_test.go
View File

@ -13,6 +13,7 @@ import (
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/mfer/internal/scanner"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@ -106,15 +107,15 @@ func TestFetchFromHTTP(t *testing.T) {
for path, content := range testFiles { for path, content := range testFiles {
fullPath := "/" + path // MemMapFs needs absolute paths fullPath := "/" + path // MemMapFs needs absolute paths
dir := filepath.Dir(fullPath) dir := filepath.Dir(fullPath)
require.NoError(t, sourceFs.MkdirAll(dir, 0o755)) require.NoError(t, sourceFs.MkdirAll(dir, 0755))
require.NoError(t, afero.WriteFile(sourceFs, fullPath, content, 0o644)) require.NoError(t, afero.WriteFile(sourceFs, fullPath, content, 0644))
} }
// Generate manifest using scanner // Generate manifest using scanner
opts := &mfer.ScannerOptions{ opts := &scanner.Options{
Fs: sourceFs, Fs: sourceFs,
} }
s := mfer.NewScannerWithOptions(opts) s := scanner.NewWithOptions(opts)
require.NoError(t, s.EnumerateFS(sourceFs, "/", nil)) require.NoError(t, s.EnumerateFS(sourceFs, "/", nil))
var manifestBuf bytes.Buffer var manifestBuf bytes.Buffer
@ -196,11 +197,11 @@ func TestFetchHashMismatch(t *testing.T) {
// Create source filesystem with a test file // Create source filesystem with a test file
sourceFs := afero.NewMemMapFs() sourceFs := afero.NewMemMapFs()
originalContent := []byte("Original content") originalContent := []byte("Original content")
require.NoError(t, afero.WriteFile(sourceFs, "/file.txt", originalContent, 0o644)) require.NoError(t, afero.WriteFile(sourceFs, "/file.txt", originalContent, 0644))
// Generate manifest // Generate manifest
opts := &mfer.ScannerOptions{Fs: sourceFs} opts := &scanner.Options{Fs: sourceFs}
s := mfer.NewScannerWithOptions(opts) s := scanner.NewWithOptions(opts)
require.NoError(t, s.EnumerateFS(sourceFs, "/", nil)) require.NoError(t, s.EnumerateFS(sourceFs, "/", nil))
var manifestBuf bytes.Buffer var manifestBuf bytes.Buffer
@ -248,11 +249,11 @@ func TestFetchSizeMismatch(t *testing.T) {
// Create source filesystem with a test file // Create source filesystem with a test file
sourceFs := afero.NewMemMapFs() sourceFs := afero.NewMemMapFs()
originalContent := []byte("Original content with specific size") originalContent := []byte("Original content with specific size")
require.NoError(t, afero.WriteFile(sourceFs, "/file.txt", originalContent, 0o644)) require.NoError(t, afero.WriteFile(sourceFs, "/file.txt", originalContent, 0644))
// Generate manifest // Generate manifest
opts := &mfer.ScannerOptions{Fs: sourceFs} opts := &scanner.Options{Fs: sourceFs}
s := mfer.NewScannerWithOptions(opts) s := scanner.NewWithOptions(opts)
require.NoError(t, s.EnumerateFS(sourceFs, "/", nil)) require.NoError(t, s.EnumerateFS(sourceFs, "/", nil))
var manifestBuf bytes.Buffer var manifestBuf bytes.Buffer
@ -297,11 +298,11 @@ func TestFetchProgress(t *testing.T) {
sourceFs := afero.NewMemMapFs() sourceFs := afero.NewMemMapFs()
// Create content large enough to trigger multiple progress updates // Create content large enough to trigger multiple progress updates
content := bytes.Repeat([]byte("x"), 100*1024) // 100KB content := bytes.Repeat([]byte("x"), 100*1024) // 100KB
require.NoError(t, afero.WriteFile(sourceFs, "/large.txt", content, 0o644)) require.NoError(t, afero.WriteFile(sourceFs, "/large.txt", content, 0644))
// Generate manifest // Generate manifest
opts := &mfer.ScannerOptions{Fs: sourceFs} opts := &scanner.Options{Fs: sourceFs}
s := mfer.NewScannerWithOptions(opts) s := scanner.NewWithOptions(opts)
require.NoError(t, s.EnumerateFS(sourceFs, "/", nil)) require.NoError(t, s.EnumerateFS(sourceFs, "/", nil))
var manifestBuf bytes.Buffer var manifestBuf bytes.Buffer

29
internal/cli/freshen_test.go
View File

@ -8,6 +8,7 @@ import (
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/mfer/internal/scanner"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@ -15,20 +16,20 @@ func TestFreshenUnchanged(t *testing.T) {
// Create filesystem with test files // Create filesystem with test files
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("content1"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("content1"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("content2"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("content2"), 0644))
// Generate initial manifest // Generate initial manifest
opts := &mfer.ScannerOptions{Fs: fs} opts := &scanner.Options{Fs: fs}
s := mfer.NewScannerWithOptions(opts) s := scanner.NewWithOptions(opts)
require.NoError(t, s.EnumeratePath("/testdir", nil)) require.NoError(t, s.EnumeratePath("/testdir", nil))
var manifestBuf bytes.Buffer var manifestBuf bytes.Buffer
require.NoError(t, s.ToManifest(context.Background(), &manifestBuf, nil)) require.NoError(t, s.ToManifest(context.Background(), &manifestBuf, nil))
// Write manifest to filesystem // Write manifest to filesystem
require.NoError(t, afero.WriteFile(fs, "/testdir/.index.mf", manifestBuf.Bytes(), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/.index.mf", manifestBuf.Bytes(), 0644))
// Parse manifest to verify // Parse manifest to verify
manifest, err := mfer.NewManifestFromFile(fs, "/testdir/.index.mf") manifest, err := mfer.NewManifestFromFile(fs, "/testdir/.index.mf")
@ -40,20 +41,20 @@ func TestFreshenWithChanges(t *testing.T) {
// Create filesystem with test files // Create filesystem with test files
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("content1"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("content1"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("content2"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("content2"), 0644))
// Generate initial manifest // Generate initial manifest
opts := &mfer.ScannerOptions{Fs: fs} opts := &scanner.Options{Fs: fs}
s := mfer.NewScannerWithOptions(opts) s := scanner.NewWithOptions(opts)
require.NoError(t, s.EnumeratePath("/testdir", nil)) require.NoError(t, s.EnumeratePath("/testdir", nil))
var manifestBuf bytes.Buffer var manifestBuf bytes.Buffer
require.NoError(t, s.ToManifest(context.Background(), &manifestBuf, nil)) require.NoError(t, s.ToManifest(context.Background(), &manifestBuf, nil))
// Write manifest to filesystem // Write manifest to filesystem
require.NoError(t, afero.WriteFile(fs, "/testdir/.index.mf", manifestBuf.Bytes(), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/.index.mf", manifestBuf.Bytes(), 0644))
// Verify initial manifest has 2 files // Verify initial manifest has 2 files
manifest, err := mfer.NewManifestFromFile(fs, "/testdir/.index.mf") manifest, err := mfer.NewManifestFromFile(fs, "/testdir/.index.mf")
@ -61,10 +62,10 @@ func TestFreshenWithChanges(t *testing.T) {
assert.Len(t, manifest.Files(), 2) assert.Len(t, manifest.Files(), 2)
// Add a new file // Add a new file
require.NoError(t, afero.WriteFile(fs, "/testdir/file3.txt", []byte("content3"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file3.txt", []byte("content3"), 0644))
// Modify file2 (change content and size) // Modify file2 (change content and size)
require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("modified content2"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("modified content2"), 0644))
// Remove file1 // Remove file1
require.NoError(t, fs.Remove("/testdir/file1.txt")) require.NoError(t, fs.Remove("/testdir/file1.txt"))

60
mfer/scanner_test.go
View File

@ -38,7 +38,7 @@ func TestNewScannerWithOptions(t *testing.T) {
func TestScannerEnumerateFile(t *testing.T) { func TestScannerEnumerateFile(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/test.txt", []byte("hello world"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/test.txt", []byte("hello world"), 0644))
s := NewScannerWithOptions(&ScannerOptions{Fs: fs}) s := NewScannerWithOptions(&ScannerOptions{Fs: fs})
err := s.EnumerateFile("/test.txt") err := s.EnumerateFile("/test.txt")
@ -62,10 +62,10 @@ func TestScannerEnumerateFileMissing(t *testing.T) {
func TestScannerEnumeratePath(t *testing.T) { func TestScannerEnumeratePath(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir/subdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir/subdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("one"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("one"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("two"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("two"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/subdir/file3.txt", []byte("three"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/subdir/file3.txt", []byte("three"), 0644))
s := NewScannerWithOptions(&ScannerOptions{Fs: fs}) s := NewScannerWithOptions(&ScannerOptions{Fs: fs})
err := s.EnumeratePath("/testdir", nil) err := s.EnumeratePath("/testdir", nil)
@ -77,9 +77,9 @@ func TestScannerEnumeratePath(t *testing.T) {
func TestScannerEnumeratePathWithProgress(t *testing.T) { func TestScannerEnumeratePathWithProgress(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("one"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("one"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("two"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("two"), 0644))
s := NewScannerWithOptions(&ScannerOptions{Fs: fs}) s := NewScannerWithOptions(&ScannerOptions{Fs: fs})
progress := make(chan EnumerateStatus, 10) progress := make(chan EnumerateStatus, 10)
@ -101,10 +101,10 @@ func TestScannerEnumeratePathWithProgress(t *testing.T) {
func TestScannerEnumeratePaths(t *testing.T) { func TestScannerEnumeratePaths(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/dir1", 0o755)) require.NoError(t, fs.MkdirAll("/dir1", 0755))
require.NoError(t, fs.MkdirAll("/dir2", 0o755)) require.NoError(t, fs.MkdirAll("/dir2", 0755))
require.NoError(t, afero.WriteFile(fs, "/dir1/a.txt", []byte("aaa"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/dir1/a.txt", []byte("aaa"), 0644))
require.NoError(t, afero.WriteFile(fs, "/dir2/b.txt", []byte("bbb"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/dir2/b.txt", []byte("bbb"), 0644))
s := NewScannerWithOptions(&ScannerOptions{Fs: fs}) s := NewScannerWithOptions(&ScannerOptions{Fs: fs})
err := s.EnumeratePaths(nil, "/dir1", "/dir2") err := s.EnumeratePaths(nil, "/dir1", "/dir2")
@ -115,10 +115,10 @@ func TestScannerEnumeratePaths(t *testing.T) {
func TestScannerExcludeDotfiles(t *testing.T) { func TestScannerExcludeDotfiles(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir/.hidden", 0o755)) require.NoError(t, fs.MkdirAll("/testdir/.hidden", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/visible.txt", []byte("visible"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/visible.txt", []byte("visible"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/.hidden.txt", []byte("hidden"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/.hidden.txt", []byte("hidden"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/.hidden/inside.txt", []byte("inside"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/.hidden/inside.txt", []byte("inside"), 0644))
t.Run("exclude by default", func(t *testing.T) { t.Run("exclude by default", func(t *testing.T) {
s := NewScannerWithOptions(&ScannerOptions{Fs: fs, IncludeDotfiles: false}) s := NewScannerWithOptions(&ScannerOptions{Fs: fs, IncludeDotfiles: false})
@ -141,9 +141,9 @@ func TestScannerExcludeDotfiles(t *testing.T) {
func TestScannerToManifest(t *testing.T) { func TestScannerToManifest(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("content one"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file1.txt", []byte("content one"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("content two"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file2.txt", []byte("content two"), 0644))
s := NewScannerWithOptions(&ScannerOptions{Fs: fs}) s := NewScannerWithOptions(&ScannerOptions{Fs: fs})
err := s.EnumeratePath("/testdir", nil) err := s.EnumeratePath("/testdir", nil)
@ -160,8 +160,8 @@ func TestScannerToManifest(t *testing.T) {
func TestScannerToManifestWithProgress(t *testing.T) { func TestScannerToManifestWithProgress(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file.txt", bytes.Repeat([]byte("x"), 1000), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file.txt", bytes.Repeat([]byte("x"), 1000), 0644))
s := NewScannerWithOptions(&ScannerOptions{Fs: fs}) s := NewScannerWithOptions(&ScannerOptions{Fs: fs})
err := s.EnumeratePath("/testdir", nil) err := s.EnumeratePath("/testdir", nil)
@ -189,11 +189,11 @@ func TestScannerToManifestWithProgress(t *testing.T) {
func TestScannerToManifestContextCancellation(t *testing.T) { func TestScannerToManifestContextCancellation(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
// Create many files to ensure we have time to cancel // Create many files to ensure we have time to cancel
for i := 0; i < 100; i++ { for i := 0; i < 100; i++ {
name := string(rune('a'+i%26)) + string(rune('0'+i/26)) + ".txt" name := string(rune('a'+i%26)) + string(rune('0'+i/26)) + ".txt"
require.NoError(t, afero.WriteFile(fs, "/testdir/"+name, bytes.Repeat([]byte("x"), 100), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/"+name, bytes.Repeat([]byte("x"), 100), 0644))
} }
s := NewScannerWithOptions(&ScannerOptions{Fs: fs}) s := NewScannerWithOptions(&ScannerOptions{Fs: fs})
@ -223,7 +223,7 @@ func TestScannerToManifestEmptyScanner(t *testing.T) {
func TestScannerFilesCopiesSlice(t *testing.T) { func TestScannerFilesCopiesSlice(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/test.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/test.txt", []byte("hello"), 0644))
s := NewScannerWithOptions(&ScannerOptions{Fs: fs}) s := NewScannerWithOptions(&ScannerOptions{Fs: fs})
require.NoError(t, s.EnumerateFile("/test.txt")) require.NoError(t, s.EnumerateFile("/test.txt"))
@ -237,9 +237,9 @@ func TestScannerFilesCopiesSlice(t *testing.T) {
func TestScannerEnumerateFS(t *testing.T) { func TestScannerEnumerateFS(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir/sub", 0o755)) require.NoError(t, fs.MkdirAll("/testdir/sub", 0755))
require.NoError(t, afero.WriteFile(fs, "/testdir/file.txt", []byte("hello"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/file.txt", []byte("hello"), 0644))
require.NoError(t, afero.WriteFile(fs, "/testdir/sub/nested.txt", []byte("world"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/testdir/sub/nested.txt", []byte("world"), 0644))
// Create a basepath filesystem // Create a basepath filesystem
baseFs := afero.NewBasePathFs(fs, "/testdir") baseFs := afero.NewBasePathFs(fs, "/testdir")
@ -297,7 +297,7 @@ func TestSendStatusNilChannel(t *testing.T) {
func TestScannerFileEntryFields(t *testing.T) { func TestScannerFileEntryFields(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
now := time.Now().Truncate(time.Second) now := time.Now().Truncate(time.Second)
require.NoError(t, afero.WriteFile(fs, "/test.txt", []byte("content"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/test.txt", []byte("content"), 0644))
require.NoError(t, fs.Chtimes("/test.txt", now, now)) require.NoError(t, fs.Chtimes("/test.txt", now, now))
s := NewScannerWithOptions(&ScannerOptions{Fs: fs}) s := NewScannerWithOptions(&ScannerOptions{Fs: fs})
@ -316,12 +316,12 @@ func TestScannerFileEntryFields(t *testing.T) {
func TestScannerLargeFileEnumeration(t *testing.T) { func TestScannerLargeFileEnumeration(t *testing.T) {
fs := afero.NewMemMapFs() fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir", 0o755)) require.NoError(t, fs.MkdirAll("/testdir", 0755))
// Create 100 files // Create 100 files
for i := 0; i < 100; i++ { for i := 0; i < 100; i++ {
name := "/testdir/" + string(rune('a'+i%26)) + string(rune('0'+i/26%10)) + ".txt" name := "/testdir/" + string(rune('a'+i%26)) + string(rune('0'+i/26%10)) + ".txt"
require.NoError(t, afero.WriteFile(fs, name, []byte("data"), 0o644)) require.NoError(t, afero.WriteFile(fs, name, []byte("data"), 0644))
} }
s := NewScannerWithOptions(&ScannerOptions{Fs: fs}) s := NewScannerWithOptions(&ScannerOptions{Fs: fs})