Record file mode in the manifest, 0000 unless asked (closes #161)
check / check (push) Failing after 6s

MFFilePath gains mode (field 304): a file's permission bits, 0777 at
most, or 0000, meaning none recorded. gen and freshen record real modes
only with --include-permissions (ScannerOptions.IncludePermissions); the
builder keeps only mode.Perm(), so setuid, setgid and sticky are never
written. list -l and export show the mode in octal. check reports
MODE_MISMATCH for a recorded mode other than 0000 the file lacks. fetch
refuses a manifest with a mode above 0777 before requesting any file,
sets only the permission bits of each recorded mode on the files it
writes, and downloads again a present file whose mode differs. The decoding-cost bound counts a file
entry at 176 bytes, up from 160.

Model: opus-5-5
This commit is contained in:
2026-10-06 06:27:56 +00:00
parent ce66f7c1c1
commit c5df0e3139
26 changed files with 652 additions and 106 deletions
+16 -11
View File
@@ -261,9 +261,12 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
- `mfer gen` / `mfer gen .` - `mfer gen` / `mfer gen .`
- recurses under current directory and writes out an `index.mf` - recurses under current directory and writes out an `index.mf`
- records every file's mode as `0000` unless given `--include-permissions`,
which records each file's permission bits (`0777` at most)
- `mfer check` / `mfer check .` - `mfer check` / `mfer check .`
- verifies checksums of all files in manifest, displaying error and exiting - verifies checksums of all files in manifest, displaying error and exiting
nonzero if any files are missing or corrupted nonzero if any files are missing or corrupted, or have permission bits
other than the mode the manifest records, unless that is `0000`
- warns about each file under the base directory that the manifest does not - warns about each file under the base directory that the manifest does not
list, hidden files included; with `--no-extra-files` each one is a failure list, hidden files included; with `--no-extra-files` each one is a failure
instead instead
@@ -271,16 +274,18 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
- fetches `/stuff/index.mf` and downloads all files listed in manifest into - fetches `/stuff/index.mf` and downloads all files listed in manifest into
the current directory, or the one given with `--dest`, and assures the current directory, or the one given with `--dest`, and assures
cryptographic integrity of downloaded files. A file already there with the cryptographic integrity of downloaded files. A file already there with the
size and hash the manifest lists is skipped. Once every file is in place, size, hash and recorded mode the manifest lists is skipped. Once every
the manifest is saved there as `index.mf`, so `mfer check` can verify the file is in place, the manifest is saved there as `index.mf`, so
tree later. Each file is downloaded to a temp file beside it, such as `mfer check` can verify the tree later. Each file is downloaded to a temp
`.a.txt.tmp` for `a.txt`, then moved into place. A manifest is refused file beside it, such as `.a.txt.tmp` for `a.txt`, given the mode the
before any file is downloaded if it lists a file where fetch writes manifest records unless that is `0000`, then moved into place. A manifest
another: at another listed file or a directory one is in, at the temp file is refused before any file is downloaded if it records a mode above
of a listed file, or at `index.mf` or `.index.mf.tmp` at the top of the `0777`, or lists a file where fetch writes another: at another listed file
tree. Names are compared in any letter case, on every filesystem, since on or a directory one is in, at the temp file of a listed file, or at
a case-insensitive one `A.txt` and `a.txt` are one file; a directory two `index.mf` or `.index.mf.tmp` at the top of the tree. Names are compared
listed files are in must be spelled alike in both. in any letter case, on every filesystem, since on a case-insensitive one
`A.txt` and `a.txt` are one file; a directory two listed files are in must
be spelled alike in both.
- `mfer fetch --require-signature <fingerprint> https://example.com/stuff/` - `mfer fetch --require-signature <fingerprint> https://example.com/stuff/`
- as above, but first refuses a manifest not signed by the key with that - as above, but first refuses a manifest not signed by the key with that
fingerprint, as `mfer check --require-signature` does, before downloading fingerprint, as `mfer check --require-signature` does, before downloading
+22 -3
View File
@@ -6,8 +6,11 @@ Version 1.0
An `.mf` file is a binary manifest that describes a directory tree of files, An `.mf` file is a binary manifest that describes a directory tree of files,
including their paths, sizes, and cryptographic checksums. It supports optional including their paths, sizes, and cryptographic checksums. It supports optional
GPG signatures for integrity verification and optional timestamps for metadata GPG signatures for integrity verification and optional timestamps and file
preservation. permissions for metadata preservation.
Nothing goes in the 1.0 manifest that 1.0 does not read or write: no field is
reserved or kept for later use.
## File Structure ## File Structure
@@ -50,7 +53,7 @@ enforce a decompression size limit to prevent decompression bombs. The reference
implementation limits decompressed size to 256 MB. It writes zstd frames with a implementation limits decompressed size to 256 MB. It writes zstd frames with a
window of at most 8 MiB, the largest window the zstd format recommends decoders window of at most 8 MiB, the largest window the zstd format recommends decoders
support, and refuses frames that ask for a larger one. It also refuses an inner support, and refuses frames that ask for a larger one. It also refuses an inner
message whose file entries, hashes, timestamps and MIME types, counted at 160, message whose file entries, hashes, timestamps and MIME types, counted at 176,
112, 64 and 16 bytes each, add up to more than 8 times its size. 112, 64 and 16 bytes each, add up to more than 8 times its size.
## Inner Message (`MFFile`) ## Inner Message (`MFFile`)
@@ -77,6 +80,21 @@ Each file entry contains:
| `mimeType` | 301 | string (optional) | MIME type | | `mimeType` | 301 | string (optional) | MIME type |
| `mtime` | 302 | Timestamp (optional) | Modification time | | `mtime` | 302 | Timestamp (optional) | Modification time |
| `ctime` | 303 | Timestamp (optional) | Change time (inode metadata change) | | `ctime` | 303 | Timestamp (optional) | Change time (inode metadata change) |
| `mode` | 304 | uint32 | Permission bits (see File Mode) |
## File Mode
`mode` holds a file's Unix permission bits, the nine `rwx` bits, so it is never
above `0777` (octal); the setuid, setgid and sticky bits are never recorded.
Writers record `0000` unless whoever creates the manifest asks for permissions.
`0000`, the proto3 default, means no mode was recorded: readers never check or
apply it.
The reference implementation records modes when `gen` or `freshen` is given
`--include-permissions`. `check` fails a file whose permission bits differ from
a recorded mode other than `0000`. `fetch` sets a recorded mode other than
`0000` on each file it writes, and refuses a manifest that records a mode above
`0777` before it requests any file.
## Path Rules ## Path Rules
@@ -127,6 +145,7 @@ By default, manifests are generated deterministically:
- File entries are sorted by `path` in **lexicographic byte order** - File entries are sorted by `path` in **lexicographic byte order**
- `createdAt` is omitted unless explicitly requested - `createdAt` is omitted unless explicitly requested
- `mode` is `0000` unless explicitly requested
This ensures that two independent runs over the same directory tree produce This ensures that two independent runs over the same directory tree produce
byte-identical `.mf` files (assuming file contents and metadata have not byte-identical `.mf` files (assuming file contents and metadata have not
+93
View File
@@ -3,6 +3,7 @@ package cli
import ( import (
"bytes" "bytes"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -374,6 +375,98 @@ func TestGenerateAndCheckCommand(t *testing.T) {
assert.Equal(t, 0, exitCode, "check failed: %s", testStderr(t, opts)) assert.Equal(t, 0, exitCode, "check failed: %s", testStderr(t, opts))
} }
// TestGenerateRecordsModeOnlyWhenAsked runs gen with and without
// --include-permissions: without it every mode is recorded as 0000, with
// it each file's permission bits. list -l and export show the recorded
// modes.
func TestGenerateRecordsModeOnlyWhenAsked(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, "/testdir/notes.txt", "hello world")
writeTestFile(t, fs, "/testdir/run.sh", "#!/bin/sh\n")
require.NoError(t, fs.Chmod("/testdir/run.sh", 0o755))
for _, tc := range []struct {
flags []string
want map[string]string // recorded mode by path
}{
{nil, map[string]string{"notes.txt": "0000", "run.sh": "0000"}},
{
[]string{"--" + flagIncludePermissions},
map[string]string{"notes.txt": "0644", "run.sh": "0755"},
},
} {
opts := testOpts(slices.Concat(
[]string{testApp, cmdGenerate, "-q", "-f", "-o", testOutput}, tc.flags,
[]string{testDir},
), fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
opts = testOpts([]string{testApp, cmdList, "-l", testOutput}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
listed := map[string]string{}
out := strings.TrimSuffix(testStdout(t, opts), "\n")
for _, line := range strings.Split(out, "\n") {
fields := strings.Split(line, "\t") // mode, size, mtime, path
require.Len(t, fields, 4, line)
listed[fields[3]] = fields[0]
}
assert.Equal(t, tc.want, listed, "list -l %v", tc.flags)
opts = testOpts([]string{testApp, cmdExport, testOutput}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
var entries []ExportEntry
require.NoError(t, json.Unmarshal([]byte(testStdout(t, opts)), &entries))
exported := map[string]string{}
for _, e := range entries {
exported[e.Path] = e.Mode
}
assert.Equal(t, tc.want, exported, "export %v", tc.flags)
}
}
// TestCheckComparesRecordedMode changes a file's mode after gen: check
// must fail on it when gen recorded the file's mode, and pass when gen
// recorded 0000.
func TestCheckComparesRecordedMode(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
flags []string
exitCode int
}{
{nil, 0},
{[]string{"--" + flagIncludePermissions}, 1},
} {
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, testFile1, "hello world")
opts := testOpts(slices.Concat(
[]string{testApp, cmdGenerate, "-q", "-o", testMF}, tc.flags,
[]string{testDir},
), fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
require.NoError(t, fs.Chmod(testFile1, 0o600))
opts = testOpts([]string{testApp, cmdCheck, testFlagBase, testDir, testMF}, fs)
assert.Equal(t, tc.exitCode, runCLI(opts), "%v: %s", tc.flags, testStderr(t, opts))
if tc.exitCode != 0 {
assert.Contains(t, testStderr(t, opts), "MODE_MISMATCH: file1.txt")
}
}
}
// sharedWriter appends to a buffer shared with other sharedWriters, so // sharedWriter appends to a buffer shared with other sharedWriters, so
// output written to stdout and stderr is kept in the order it was written. // output written to stdout and stderr is kept in the order it was written.
// Each write first waits for delay. // Each write first waits for delay.
+1 -1
View File
@@ -162,7 +162,7 @@ func signedManifest(t *testing.T, files map[string][]byte) []byte {
for path, content := range files { for path, content := range files {
_, err = b.AddFile(mfer.RelFilePath(path), mfer.FileSize(len(content)), _, err = b.AddFile(mfer.RelFilePath(path), mfer.FileSize(len(content)),
mfer.ModTime{}, bytes.NewReader(content), nil) mfer.ModTime{}, 0, bytes.NewReader(content), nil)
require.NoError(t, err) require.NoError(t, err)
} }
+2
View File
@@ -18,6 +18,7 @@ type ExportEntry struct {
Hashes []string `json:"hashes"` Hashes []string `json:"hashes"`
Mtime *string `json:"mtime,omitempty"` Mtime *string `json:"mtime,omitempty"`
Ctime *string `json:"ctime,omitempty"` Ctime *string `json:"ctime,omitempty"`
Mode string `json:"mode"` // octal, "0000" when none was recorded
} }
func (mfa *CLIApp) exportManifestOperation( func (mfa *CLIApp) exportManifestOperation(
@@ -49,6 +50,7 @@ func (mfa *CLIApp) exportManifestOperation(
Path: f.GetPath(), Path: f.GetPath(),
Size: f.GetSize(), Size: f.GetSize(),
Hashes: make([]string, 0, len(f.GetHashes())), Hashes: make([]string, 0, len(f.GetHashes())),
Mode: fmt.Sprintf("%04o", f.GetMode()),
} }
for _, h := range f.GetHashes() { for _, h := range f.GetHashes() {
+1 -1
View File
@@ -132,7 +132,7 @@ func TestListFromHTTPURL(t *testing.T) {
exitCode := runCLI(&RunOptions{ exitCode := runCLI(&RunOptions{
Appname: testApp, Appname: testApp,
Args: []string{testApp, "list", server.URL + "/index.mf"}, Args: []string{testApp, cmdList, server.URL + "/index.mf"},
Stdin: &bytes.Buffer{}, Stdin: &bytes.Buffer{},
Stdout: &stdout, Stdout: &stdout,
Stderr: &stderr, Stderr: &stderr,
+40 -8
View File
@@ -95,6 +95,9 @@ var (
// writes another file. // writes another file.
errNameClash = errors.New( errNameClash = errors.New(
"manifest lists a file where fetch writes another file") "manifest lists a file where fetch writes another file")
// errModeOutOfRange indicates a manifest that lists a mode with more
// than the permission bits, such as setuid.
errModeOutOfRange = errors.New("manifest lists a mode outside 0777")
) )
// DownloadProgress reports the progress of a single file download. // DownloadProgress reports the progress of a single file download.
@@ -292,11 +295,11 @@ func downloadManifestFiles(
} }
// alreadyPresent reports whether localPath under dest is a regular file // alreadyPresent reports whether localPath under dest is a regular file
// with the size and one of the hashes the manifest lists for entry. It // with the size, the recorded mode if any, and one of the hashes the
// hashes the whole file, since a matching size alone would accept a // manifest lists for entry. It hashes the whole file, since a matching
// corrupted or partly written one. A file it cannot read, or reaches only // size alone would accept a corrupted or partly written one. A file it
// through a symlink, is not present: fetch downloads it, and the download // cannot read, or reaches only through a symlink, is not present: fetch
// reports the problem. // downloads it, and the download reports the problem.
func alreadyPresent(dest, localPath string, entry *mfer.MFFilePath) bool { func alreadyPresent(dest, localPath string, entry *mfer.MFFilePath) bool {
if checkNoSymlinks(dest, localPath) != nil { if checkNoSymlinks(dest, localPath) != nil {
return false return false
@@ -309,6 +312,12 @@ func alreadyPresent(dest, localPath string, entry *mfer.MFFilePath) bool {
return false return false
} }
// A recorded mode of 0 means none was recorded.
if entry.GetMode() != 0 &&
info.Mode().Perm() != os.FileMode(entry.GetMode()).Perm() {
return false
}
// G304: localPath is a relative path that sanitizePath keeps inside // G304: localPath is a relative path that sanitizePath keeps inside
// dest as text, and checkNoSymlinks just found no symlink in it. // dest as text, and checkNoSymlinks just found no symlink in it.
f, err := os.Open(path) //nolint:gosec // G304: see comment above f, err := os.Open(path) //nolint:gosec // G304: see comment above
@@ -414,9 +423,9 @@ func (mfa *CLIApp) fetchManifestOperation(
// fetchManifest downloads the manifest at manifestURL and parses it, // fetchManifest downloads the manifest at manifestURL and parses it,
// enforcing --require-signature if it is given and refusing a manifest // enforcing --require-signature if it is given and refusing a manifest
// that lists a file where fetch writes another. It returns the manifest as // that lists a file where fetch writes another or a mode outside 0777. It
// downloaded, to be saved once the files are in place, and the files it // returns the manifest as downloaded, to be saved once the files are in
// lists. // place, and the files it lists.
func fetchManifest( func fetchManifest(
ctx context.Context, cmd *cli.Command, client retryingClient, manifestURL string, ctx context.Context, cmd *cli.Command, client retryingClient, manifestURL string,
) ([]byte, []*mfer.MFFilePath, error) { ) ([]byte, []*mfer.MFFilePath, error) {
@@ -460,6 +469,16 @@ func fetchManifest(
return nil, nil, err return nil, nil, err
} }
// fetch sets each recorded mode on the file it writes, so a mode above
// 0777, which could carry setuid, setgid or sticky bits, is refused
// before any file is requested.
for _, f := range files {
if f.GetMode() > uint32(os.ModePerm) {
return nil, nil, fmt.Errorf("%w: %s (%#o)",
errModeOutOfRange, f.GetPath(), f.GetMode())
}
}
log.Infof("manifest contains %d files", len(files)) log.Infof("manifest contains %d files", len(files))
return manifestData, files, nil return manifestData, files, nil
@@ -866,6 +885,19 @@ func saveResponse(
return err return err
} }
// A recorded mode of 0 means none was recorded. Of any other, only the
// permission bits are set, whatever the umask, so setuid, setgid and
// sticky never are, whichever caller passed the entry.
if entry.GetMode() != 0 {
err = out.Chmod(os.FileMode(entry.GetMode()).Perm())
if err != nil {
_ = out.Close()
_ = os.Remove(filepath.Join(dest, tmpPath))
return fmt.Errorf("failed to set mode: %w", err)
}
}
// Set up hash computation // Set up hash computation
h := sha256.New() h := sha256.New()
+193 -1
View File
@@ -4,6 +4,7 @@ package cli
import ( import (
"bytes" "bytes"
"context" "context"
"crypto/sha256"
"fmt" "fmt"
"io" "io"
"maps" "maps"
@@ -19,9 +20,13 @@ import (
"testing" "testing"
"time" "time"
"github.com/google/uuid"
"github.com/klauspost/compress/zstd"
"github.com/multiformats/go-multihash"
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"google.golang.org/protobuf/proto"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@@ -1299,6 +1304,193 @@ func TestFetchRefusesNamesEqualIgnoringCase(t *testing.T) {
}) })
} }
// TestFetchSetsRecordedMode fetches a tree whose manifest records the
// modes 0640 and 0755. Each file must get its mode whatever the umask, and
// check must pass on the result. After one file's mode is changed, a
// second fetch must download that file again, and only it, to restore its
// mode.
func TestFetchSetsRecordedMode(t *testing.T) {
t.Parallel()
files := map[string][]byte{
testFileTxt: []byte("a file"),
"tool.sh": []byte("#!/bin/sh\n"),
}
modes := map[string]os.FileMode{testFileTxt: 0o640, "tool.sh": 0o755}
sourceFs := afero.NewMemMapFs()
for p, content := range files {
require.NoError(t, afero.WriteFile(sourceFs, "/"+p, content, modes[p]))
}
scanner := mfer.NewScannerWithOptions(&mfer.ScannerOptions{
Fs: sourceFs,
IncludePermissions: true,
})
require.NoError(t, scanner.EnumerateFS(sourceFs, "/", nil))
var manifest bytes.Buffer
require.NoError(t, scanner.ToManifest(context.Background(), &manifest, nil))
tree := fetchTestHandler(manifest.Bytes(), files)
var (
mu sync.Mutex
requested []string
)
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
requested = append(requested, r.URL.Path)
mu.Unlock()
tree.ServeHTTP(w, r)
}))
defer server.Close()
dest := t.TempDir()
fetch := []string{testApp, cmdFetch, "-q", "--" + flagDest, dest, server.URL}
opts := testOpts(fetch, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
for p, mode := range modes {
info, err := os.Stat(filepath.Join(dest, p))
require.NoError(t, err)
assert.Equal(t, mode, info.Mode().Perm(), p)
}
opts = testOpts([]string{
testApp, cmdCheck, "-q", testFlagBase, dest,
filepath.Join(dest, defaultManifestName),
}, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
require.NoError(t, os.Chmod(filepath.Join(dest, testFileTxt), 0o600))
mu.Lock()
requested = nil
mu.Unlock()
opts = testOpts(fetch, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
info, err := os.Stat(filepath.Join(dest, testFileTxt))
require.NoError(t, err)
assert.Equal(t, os.FileMode(0o640), info.Mode().Perm())
mu.Lock()
defer mu.Unlock()
assert.ElementsMatch(t,
[]string{"/" + defaultManifestName, "/" + testFileTxt}, requested)
}
// TestFetchRefusesModeOutsidePermissionBits fetches manifests that record
// a mode with the setuid bit, once as Unix writes it (04755) and once as
// Go keeps it in os.FileMode. fetch must refuse both before it creates the
// destination or requests any file.
func TestFetchRefusesModeOutsidePermissionBits(t *testing.T) {
t.Parallel()
files := map[string][]byte{testFileTxt: []byte("a file")}
assertFetchRefused(t, manifestWithMode(t, testFileTxt, files[testFileTxt], 0o4755),
files, "manifest lists a mode outside 0777: file.txt (04755)")
assertFetchRefused(t,
manifestWithMode(t, testFileTxt, files[testFileTxt],
uint32(os.ModeSetuid|0o755)),
files, "manifest lists a mode outside 0777: file.txt (040000755)")
}
// TestDownloadFileSetsOnlyPermissionBits downloads a file whose entry
// records mode 0755 together with setuid, setgid or sticky, as Go keeps
// them in os.FileMode. fetchManifest would refuse such an entry; called
// directly, downloadFile must still set only the permission bits.
func TestDownloadFileSetsOnlyPermissionBits(t *testing.T) {
t.Parallel()
content := []byte("#!/bin/sh\n")
digest := sha256.Sum256(content)
hash, err := multihash.Encode(digest[:], multihash.SHA2_256)
require.NoError(t, err)
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write(content)
}))
defer server.Close()
for _, special := range []os.FileMode{os.ModeSetuid, os.ModeSetgid, os.ModeSticky} {
entry := &mfer.MFFilePath{
Path: testFileTxt,
Size: int64(len(content)),
Hashes: []*mfer.MFFileChecksum{{MultiHash: hash}},
Mode: uint32(special | 0o755),
}
dest := t.TempDir()
err := downloadFile(context.Background(), testClient(),
server.URL+"/"+testFileTxt, dest, testFileTxt, entry, nil)
require.NoError(t, err, special)
info, err := os.Stat(filepath.Join(dest, testFileTxt))
require.NoError(t, err)
assert.Equal(t, os.FileMode(0o755), info.Mode(), special)
}
}
// manifestWithMode returns a manifest listing one file, path, with content
// and the given recorded mode. It is assembled by hand, since the builder
// never records a mode outside 0777.
func manifestWithMode(t *testing.T, path string, content []byte, mode uint32) []byte {
t.Helper()
digest := sha256.Sum256(content)
hash, err := multihash.Encode(digest[:], multihash.SHA2_256)
require.NoError(t, err)
id := uuid.New()
inner, err := proto.Marshal(&mfer.MFFile{
Version: mfer.MFFile_VERSION_ONE,
Files: []*mfer.MFFilePath{{
Path: path,
Size: int64(len(content)),
Hashes: []*mfer.MFFileChecksum{{MultiHash: hash}},
Mode: mode,
}},
Uuid: id[:],
})
require.NoError(t, err)
encoder, err := zstd.NewWriter(nil)
require.NoError(t, err)
compressed := encoder.EncodeAll(inner, nil)
require.NoError(t, encoder.Close())
sum := sha256.Sum256(compressed)
outer, err := proto.Marshal(&mfer.MFFileOuter{
Version: mfer.MFFileOuter_VERSION_ONE,
CompressionType: mfer.MFFileOuter_COMPRESSION_ZSTD,
Size: int64(len(inner)),
Sha256: sum[:],
Uuid: id[:],
InnerMessage: compressed,
})
require.NoError(t, err)
return append([]byte(mfer.MAGIC), outer...)
}
// builtManifest returns a manifest of files, built directly rather than // builtManifest returns a manifest of files, built directly rather than
// scanned, since a scan lists no hidden files and never a path starting // scanned, since a scan lists no hidden files and never a path starting
// with "./". // with "./".
@@ -1309,7 +1501,7 @@ func builtManifest(t *testing.T, files map[string][]byte) []byte {
for p, content := range files { for p, content := range files {
_, err := builder.AddFile(mfer.RelFilePath(p), mfer.FileSize(len(content)), _, err := builder.AddFile(mfer.RelFilePath(p), mfer.FileSize(len(content)),
mfer.ModTime(time.Now()), bytes.NewReader(content), nil) mfer.ModTime(time.Now()), 0, bytes.NewReader(content), nil)
require.NoError(t, err) require.NoError(t, err)
} }
+38 -23
View File
@@ -48,6 +48,7 @@ type freshenEntry struct {
path string path string
size int64 size int64
mtime time.Time mtime time.Time
mode fs.FileMode // mode to record, 0 for none
needsHash bool // true if new or changed needsHash bool // true if new or changed
existing *mfer.MFFilePath // existing manifest entry if unchanged existing *mfer.MFFilePath // existing manifest entry if unchanged
} }
@@ -55,13 +56,14 @@ type freshenEntry struct {
// freshenScanner walks the filesystem and compares it against the // freshenScanner walks the filesystem and compares it against the
// entries of an existing manifest. // entries of an existing manifest.
type freshenScanner struct { type freshenScanner struct {
fs afero.Fs fs afero.Fs
absBase string absBase string
excluded []fs.FileInfo // files left out of the listing excluded []fs.FileInfo // files left out of the listing
includeDotfiles bool includeDotfiles bool
followSymlinks bool followSymlinks bool
showProgress bool includePermissions bool
existingByPath map[string]*mfer.MFFilePath showProgress bool
existingByPath map[string]*mfer.MFFilePath
entries []*freshenEntry entries []*freshenEntry
scanCount int64 scanCount int64
@@ -93,6 +95,12 @@ func (s *freshenScanner) resolveSymlink(path string) (fs.FileInfo, bool) {
// recordEntry classifies a scanned file as changed, unchanged, or added // recordEntry classifies a scanned file as changed, unchanged, or added
// relative to the existing manifest. // relative to the existing manifest.
func (s *freshenScanner) recordEntry(relPath string, info fs.FileInfo) { func (s *freshenScanner) recordEntry(relPath string, info fs.FileInfo) {
// A mode of 0 records 0000, which means none was recorded.
var mode fs.FileMode
if s.includePermissions {
mode = info.Mode().Perm()
}
existing, inManifest := s.existingByPath[relPath] existing, inManifest := s.existingByPath[relPath]
if !inManifest { if !inManifest {
s.added++ s.added++
@@ -102,16 +110,17 @@ func (s *freshenScanner) recordEntry(relPath string, info fs.FileInfo) {
path: relPath, path: relPath,
size: info.Size(), size: info.Size(),
mtime: info.ModTime(), mtime: info.ModTime(),
mode: mode,
needsHash: true, needsHash: true,
}) })
return return
} }
// Check if changed (size or mtime). An entry with no recorded mtime // Check if changed (size, mtime, or the mode to record). An entry
// cannot be compared, so it counts as changed and gets re-hashed; // with no recorded mtime cannot be compared, so it counts as changed
// silently treating the absent mtime as the Unix epoch would classify // and gets re-hashed; silently treating the absent mtime as the Unix
// every such entry as changed without saying why. // epoch would classify every such entry as changed without saying why.
existingMtime, haveMtime := entryMtime(existing) existingMtime, haveMtime := entryMtime(existing)
if !haveMtime { if !haveMtime {
log.Debugf("%s: manifest entry has no mtime, treating as changed", log.Debugf("%s: manifest entry has no mtime, treating as changed",
@@ -119,7 +128,8 @@ func (s *freshenScanner) recordEntry(relPath string, info fs.FileInfo) {
} }
if !haveMtime || existing.GetSize() != info.Size() || if !haveMtime || existing.GetSize() != info.Size() ||
!existingMtime.Equal(info.ModTime()) { !existingMtime.Equal(info.ModTime()) ||
fs.FileMode(existing.GetMode()) != mode {
s.changed++ s.changed++
log.Verbosef("M %s", relPath) log.Verbosef("M %s", relPath)
@@ -127,6 +137,7 @@ func (s *freshenScanner) recordEntry(relPath string, info fs.FileInfo) {
path: relPath, path: relPath,
size: info.Size(), size: info.Size(),
mtime: info.ModTime(), mtime: info.ModTime(),
mode: mode,
needsHash: true, needsHash: true,
}) })
} else { } else {
@@ -136,6 +147,7 @@ func (s *freshenScanner) recordEntry(relPath string, info fs.FileInfo) {
path: relPath, path: relPath,
size: info.Size(), size: info.Size(),
mtime: info.ModTime(), mtime: info.ModTime(),
mode: mode,
needsHash: false, needsHash: false,
existing: existing, existing: existing,
}) })
@@ -296,7 +308,7 @@ func (h *freshenHasher) processEntry(e *freshenEntry) error {
h.hashedFiles++ h.hashedFiles++
// Add to builder with computed hash // Add to builder with computed hash
err = addFileToBuilder(h.builder, e.path, e.size, e.mtime, hash) err = addFileToBuilder(h.builder, e.path, e.size, e.mtime, e.mode, hash)
if err != nil { if err != nil {
return fmt.Errorf("failed to add %s: %w", e.path, err) return fmt.Errorf("failed to add %s: %w", e.path, err)
} }
@@ -379,13 +391,14 @@ func (mfa *CLIApp) freshenScan(
} }
scanner := &freshenScanner{ scanner := &freshenScanner{
fs: mfa.Fs, fs: mfa.Fs,
absBase: absBase, absBase: absBase,
excluded: excluded, excluded: excluded,
includeDotfiles: cmd.Bool("include-dotfiles"), includeDotfiles: cmd.Bool("include-dotfiles"),
followSymlinks: cmd.Bool("follow-symlinks"), followSymlinks: cmd.Bool("follow-symlinks"),
showProgress: showProgress, includePermissions: cmd.Bool(flagIncludePermissions),
existingByPath: existingByPath, showProgress: showProgress,
existingByPath: existingByPath,
} }
err := afero.Walk(mfa.Fs, absBase, scanner.walk) err := afero.Walk(mfa.Fs, absBase, scanner.walk)
@@ -611,10 +624,12 @@ func hashFile(r io.Reader, progress func(int64)) ([]byte, int64, error) {
// addFileToBuilder adds a new file entry to the builder // addFileToBuilder adds a new file entry to the builder
func addFileToBuilder( func addFileToBuilder(
b *mfer.Builder, path string, size int64, mtime time.Time, hash []byte, b *mfer.Builder, path string, size int64, mtime time.Time, mode fs.FileMode,
hash []byte,
) error { ) error {
return b.AddFileWithHash( return b.AddFileWithHash(
mfer.RelFilePath(path), mfer.FileSize(size), mfer.ModTime(mtime), hash) mfer.RelFilePath(path), mfer.FileSize(size), mfer.ModTime(mtime), mode,
hash)
} }
// addExistingToBuilder adds an existing manifest entry to the builder. // addExistingToBuilder adds an existing manifest entry to the builder.
@@ -634,7 +649,7 @@ func addExistingToBuilder(b *mfer.Builder, entry *mfer.MFFilePath) error {
err := b.AddFileWithHash(mfer.RelFilePath(entry.GetPath()), err := b.AddFileWithHash(mfer.RelFilePath(entry.GetPath()),
mfer.FileSize(entry.GetSize()), mfer.ModTime(mtime), mfer.FileSize(entry.GetSize()), mfer.ModTime(mtime),
entry.GetHashes()[0].GetMultiHash()) fs.FileMode(entry.GetMode()), entry.GetHashes()[0].GetMultiHash())
if err != nil { if err != nil {
return fmt.Errorf( return fmt.Errorf(
"manifest entry %s: %w (regenerate the manifest with mfer generate)", "manifest entry %s: %w (regenerate the manifest with mfer generate)",
+36
View File
@@ -99,6 +99,42 @@ func TestFreshenUnchanged(t *testing.T) {
} }
} }
// TestFreshenRecordsModeOnlyWhenAsked changes only the modes of a tree
// after gen made its manifest, which recorded every mode as 0000. freshen
// --include-permissions must record each file's permission bits, and a
// later freshen without it must record 0000 again, although no file's
// content or mtime changed.
func TestFreshenRecordsModeOnlyWhenAsked(t *testing.T) {
t.Parallel()
fs := afero.NewOsFs()
root, manifestPath := setupFreshenDir(t, fs,
map[string]string{testFileTxt: "a file", testDirFile: "a file in dir"})
require.NoError(t, fs.Chmod(filepath.Join(root, testFileTxt), 0o640))
require.NoError(t, fs.Chmod(filepath.Join(root, testDirFile), 0o755))
recordedModes := func() map[string]uint32 {
modes := map[string]uint32{}
for _, f := range manifestFiles(t, fs, manifestPath) {
modes[f.GetPath()] = f.GetMode()
}
return modes
}
opts := testOpts([]string{
testApp, cmdFreshen, "-q", "--" + flagIncludePermissions,
testFlagBase, root, manifestPath,
}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.Equal(t, map[string]uint32{testFileTxt: 0o640, testDirFile: 0o755},
recordedModes())
runFreshen(t, fs, root, manifestPath)
assert.Equal(t, map[string]uint32{testFileTxt: 0, testDirFile: 0},
recordedModes())
}
// assertManifestLists asserts that the manifest at manifestPath lists // assertManifestLists asserts that the manifest at manifestPath lists
// exactly the files in want, each with the size and SHA-256 hash of its // exactly the files in want, each with the size and SHA-256 hash of its
// content in want and the mtime of the file of that name under root. // content in want and the mtime of the file of that name under root.
+5 -4
View File
@@ -92,10 +92,11 @@ func (mfa *CLIApp) collectInputPaths(args cli.Args) ([]string, error) {
func (mfa *CLIApp) buildScannerOptions(cmd *cli.Command) *mfer.ScannerOptions { func (mfa *CLIApp) buildScannerOptions(cmd *cli.Command) *mfer.ScannerOptions {
output := cmd.String("output") output := cmd.String("output")
opts := &mfer.ScannerOptions{ opts := &mfer.ScannerOptions{
IncludeDotfiles: cmd.Bool("include-dotfiles"), IncludeDotfiles: cmd.Bool("include-dotfiles"),
FollowSymLinks: cmd.Bool("follow-symlinks"), FollowSymLinks: cmd.Bool("follow-symlinks"),
IncludeTimestamps: cmd.Bool("include-timestamps"), IncludeTimestamps: cmd.Bool("include-timestamps"),
Fs: mfa.Fs, IncludePermissions: cmd.Bool(flagIncludePermissions),
Fs: mfa.Fs,
// Neither a manifest being replaced nor a temp file left by an // Neither a manifest being replaced nor a temp file left by an
// interrupted run belongs in the new manifest. // interrupted run belongs in the new manifest.
ExcludePaths: []string{output, manifestTempPath(output)}, ExcludePaths: []string{output, manifestTempPath(output)},
+2 -2
View File
@@ -52,8 +52,8 @@ func (mfa *CLIApp) listManifestOperation(ctx context.Context, cmd *cli.Command)
mtimeStr = mtime.Format(time.RFC3339) mtimeStr = mtime.Format(time.RFC3339)
} }
_, _ = fmt.Fprintf(mfa.Stdout, "%d\t%s\t%s%s", _, _ = fmt.Fprintf(mfa.Stdout, "%04o\t%d\t%s\t%s%s",
f.GetSize(), mtimeStr, f.GetPath(), lineEnd) f.GetMode(), f.GetSize(), mtimeStr, f.GetPath(), lineEnd)
} else { } else {
_, _ = fmt.Fprintf(mfa.Stdout, "%s%s", f.GetPath(), lineEnd) _, _ = fmt.Fprintf(mfa.Stdout, "%s%s", f.GetPath(), lineEnd)
} }
+20 -6
View File
@@ -21,12 +21,14 @@ const (
cmdFreshen = "freshen" cmdFreshen = "freshen"
cmdExport = "export" cmdExport = "export"
cmdFetch = "fetch" cmdFetch = "fetch"
cmdList = "list"
cmdVersion = "version" cmdVersion = "version"
flagProgress = "progress" flagProgress = "progress"
flagTimeout = "timeout" flagTimeout = "timeout"
flagDest = "dest" flagDest = "dest"
flagRequireSignature = "require-signature" flagRequireSignature = "require-signature"
flagIncludePermissions = "include-permissions"
manifestArgsUsage = "[manifest file]" manifestArgsUsage = "[manifest file]"
@@ -167,6 +169,16 @@ func requireSignatureFlag() *cli.StringFlag {
} }
} }
// includePermissionsFlag returns the --include-permissions flag taken by the
// generate and freshen subcommands.
func includePermissionsFlag() *cli.BoolFlag {
return &cli.BoolFlag{
Name: flagIncludePermissions,
Usage: "Record each file's permission bits in manifest " +
"(recorded as 0000 by default)",
}
}
func (mfa *CLIApp) generateCommand() *cli.Command { func (mfa *CLIApp) generateCommand() *cli.Command {
return &cli.Command{ return &cli.Command{
Name: cmdGenerate, Name: cmdGenerate,
@@ -224,6 +236,7 @@ func (mfa *CLIApp) generateCommand() *cli.Command {
Usage: "Include createdAt timestamp in manifest " + Usage: "Include createdAt timestamp in manifest " +
"(omitted by default for determinism)", "(omitted by default for determinism)",
}, },
includePermissionsFlag(),
), ),
} }
} }
@@ -307,6 +320,7 @@ func (mfa *CLIApp) freshenCommand() *cli.Command {
Usage: "Include createdAt timestamp in manifest " + Usage: "Include createdAt timestamp in manifest " +
"(omitted by default for determinism)", "(omitted by default for determinism)",
}, },
includePermissionsFlag(),
), ),
} }
} }
@@ -340,7 +354,7 @@ func (mfa *CLIApp) versionCommand() *cli.Command {
func (mfa *CLIApp) listCommand() *cli.Command { func (mfa *CLIApp) listCommand() *cli.Command {
return &cli.Command{ return &cli.Command{
Name: "list", Name: cmdList,
Aliases: []string{"ls"}, Aliases: []string{"ls"},
Usage: "List files in manifest", Usage: "List files in manifest",
ArgsUsage: manifestArgsUsage, ArgsUsage: manifestArgsUsage,
@@ -350,7 +364,7 @@ func (mfa *CLIApp) listCommand() *cli.Command {
&cli.BoolFlag{ &cli.BoolFlag{
Name: "long", Name: "long",
Aliases: []string{"l"}, Aliases: []string{"l"},
Usage: "Show size and mtime", Usage: "Show mode, size and mtime",
}, },
&cli.BoolFlag{ &cli.BoolFlag{
Name: "print0", Name: "print0",
+7
View File
@@ -8,6 +8,7 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"io/fs"
"sort" "sort"
"strings" "strings"
"sync" "sync"
@@ -137,12 +138,14 @@ func (b *Builder) SetSeed(seed string) {
} }
// AddFile reads file content from reader, computes hashes, and adds to manifest. // AddFile reads file content from reader, computes hashes, and adds to manifest.
// Only mode's permission bits (mode.Perm()) are recorded; 0 records none.
// Progress updates are sent to the progress channel (if non-nil) without blocking. // Progress updates are sent to the progress channel (if non-nil) without blocking.
// Returns the number of bytes read. // Returns the number of bytes read.
func (b *Builder) AddFile( func (b *Builder) AddFile(
path RelFilePath, path RelFilePath,
size FileSize, size FileSize,
mtime ModTime, mtime ModTime,
mode fs.FileMode,
reader io.Reader, reader io.Reader,
progress chan<- FileHashProgress, progress chan<- FileHashProgress,
) (FileSize, error) { ) (FileSize, error) {
@@ -198,6 +201,7 @@ func (b *Builder) AddFile(
{MultiHash: mh}, {MultiHash: mh},
}, },
Mtime: mtime.Timestamp(), Mtime: mtime.Timestamp(),
Mode: uint32(mode.Perm()),
} }
b.mu.Lock() b.mu.Lock()
@@ -229,12 +233,14 @@ func (b *Builder) FileCount() int {
// AddFileWithHash adds a file entry with a pre-computed hash. // AddFileWithHash adds a file entry with a pre-computed hash.
// This is useful when the hash is already known (e.g., from an existing manifest). // This is useful when the hash is already known (e.g., from an existing manifest).
// Only mode's permission bits (mode.Perm()) are recorded; 0 records none.
// Returns an error if path is invalid, size is negative, or hash is not a // Returns an error if path is invalid, size is negative, or hash is not a
// multihash with a digest of at least 32 bytes, as long as SHA-256's. // multihash with a digest of at least 32 bytes, as long as SHA-256's.
func (b *Builder) AddFileWithHash( func (b *Builder) AddFileWithHash(
path RelFilePath, path RelFilePath,
size FileSize, size FileSize,
mtime ModTime, mtime ModTime,
mode fs.FileMode,
hash Multihash, hash Multihash,
) error { ) error {
err := ValidatePath(string(path)) err := ValidatePath(string(path))
@@ -268,6 +274,7 @@ func (b *Builder) AddFileWithHash(
{MultiHash: hash}, {MultiHash: hash},
}, },
Mtime: mtime.Timestamp(), Mtime: mtime.Timestamp(),
Mode: uint32(mode.Perm()),
} }
b.mu.Lock() b.mu.Lock()
+18 -18
View File
@@ -35,7 +35,7 @@ func TestBuilderAddFile(t *testing.T) {
reader := bytes.NewReader(content) reader := bytes.NewReader(content)
bytesRead, err := b.AddFile( bytesRead, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()), reader, nil, "test.txt", FileSize(len(content)), ModTime(time.Now()), 0, reader, nil,
) )
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, FileSize(len(content)), bytesRead) assert.Equal(t, FileSize(len(content)), bytesRead)
@@ -49,7 +49,7 @@ func TestBuilderAddFileWithHash(t *testing.T) {
hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256) hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
require.NoError(t, err) require.NoError(t, err)
err = b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), hash) err = b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), 0, hash)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, 1, b.FileCount()) assert.Equal(t, 1, b.FileCount())
} }
@@ -64,7 +64,7 @@ func TestBuilderAddFileWithHashValidation(t *testing.T) {
t.Parallel() t.Parallel()
b := NewBuilder() b := NewBuilder()
err := b.AddFileWithHash("", 100, ModTime(time.Now()), sha256Hash) err := b.AddFileWithHash("", 100, ModTime(time.Now()), 0, sha256Hash)
require.Error(t, err) require.Error(t, err)
assert.Contains(t, err.Error(), "path") assert.Contains(t, err.Error(), "path")
}) })
@@ -73,7 +73,7 @@ func TestBuilderAddFileWithHashValidation(t *testing.T) {
t.Parallel() t.Parallel()
b := NewBuilder() b := NewBuilder()
err := b.AddFileWithHash("test.txt", -1, ModTime(time.Now()), sha256Hash) err := b.AddFileWithHash("test.txt", -1, ModTime(time.Now()), 0, sha256Hash)
require.Error(t, err) require.Error(t, err)
assert.Contains(t, err.Error(), "size") assert.Contains(t, err.Error(), "size")
}) })
@@ -82,7 +82,7 @@ func TestBuilderAddFileWithHashValidation(t *testing.T) {
t.Parallel() t.Parallel()
b := NewBuilder() b := NewBuilder()
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), sha256Hash) err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), 0, sha256Hash)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, 1, b.FileCount()) assert.Equal(t, 1, b.FileCount())
}) })
@@ -118,7 +118,7 @@ func TestBuilderAddFileWithHashRejectsBadHashes(t *testing.T) {
t.Parallel() t.Parallel()
b := NewBuilder() b := NewBuilder()
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), tt.hash) err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), 0, tt.hash)
require.ErrorIs(t, err, tt.want) require.ErrorIs(t, err, tt.want)
assert.Equal(t, 0, b.FileCount()) assert.Equal(t, 0, b.FileCount())
}) })
@@ -133,7 +133,7 @@ func TestBuilderBuild(t *testing.T) {
reader := bytes.NewReader(content) reader := bytes.NewReader(content)
_, err := b.AddFile( _, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()), reader, nil, "test.txt", FileSize(len(content)), ModTime(time.Now()), 0, reader, nil,
) )
require.NoError(t, err) require.NoError(t, err)
@@ -196,7 +196,7 @@ func TestBuilderDeterministicOutput(t *testing.T) {
for _, f := range files { for _, f := range files {
r := bytes.NewReader([]byte(f.content)) r := bytes.NewReader([]byte(f.content))
_, err := b.AddFile( _, err := b.AddFile(
RelFilePath(f.path), FileSize(len(f.content)), mtime, r, nil, RelFilePath(f.path), FileSize(len(f.content)), mtime, 0, r, nil,
) )
require.NoError(t, err) require.NoError(t, err)
} }
@@ -279,7 +279,7 @@ func TestBuilderAddFileSizeMismatch(t *testing.T) {
reader := bytes.NewReader(content) reader := bytes.NewReader(content)
// Declare wrong size // Declare wrong size
_, err := b.AddFile("test.txt", FileSize(100), ModTime(time.Now()), reader, nil) _, err := b.AddFile("test.txt", FileSize(100), ModTime(time.Now()), 0, reader, nil)
require.Error(t, err) require.Error(t, err)
assert.Contains(t, err.Error(), "size mismatch") assert.Contains(t, err.Error(), "size mismatch")
} }
@@ -291,12 +291,12 @@ func TestBuilderAddFileInvalidPath(t *testing.T) {
content := []byte("data") content := []byte("data")
reader := bytes.NewReader(content) reader := bytes.NewReader(content)
_, err := b.AddFile("", FileSize(len(content)), ModTime(time.Now()), reader, nil) _, err := b.AddFile("", FileSize(len(content)), ModTime(time.Now()), 0, reader, nil)
require.Error(t, err) require.Error(t, err)
reader.Reset(content) reader.Reset(content)
_, err = b.AddFile( _, err = b.AddFile(
"/absolute", FileSize(len(content)), ModTime(time.Now()), reader, nil, "/absolute", FileSize(len(content)), ModTime(time.Now()), 0, reader, nil,
) )
assert.Error(t, err) assert.Error(t, err)
} }
@@ -310,7 +310,7 @@ func TestBuilderAddFileWithProgress(t *testing.T) {
progress := make(chan FileHashProgress, 100) progress := make(chan FileHashProgress, 100)
bytesRead, err := b.AddFile( bytesRead, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()), reader, progress, "test.txt", FileSize(len(content)), ModTime(time.Now()), 0, reader, progress,
) )
close(progress) close(progress)
require.NoError(t, err) require.NoError(t, err)
@@ -345,7 +345,7 @@ func TestBuilderBuildRoundTrip(t *testing.T) {
for _, f := range files { for _, f := range files {
reader := bytes.NewReader(f.content) reader := bytes.NewReader(f.content)
_, err := b.AddFile( _, err := b.AddFile(
RelFilePath(f.path), FileSize(len(f.content)), ModTime(now), reader, nil, RelFilePath(f.path), FileSize(len(f.content)), ModTime(now), 0, reader, nil,
) )
require.NoError(t, err) require.NoError(t, err)
} }
@@ -387,7 +387,7 @@ func TestBuilderBuildRoundTripLargeManifest(t *testing.T) {
for i := range 4000 { for i := range 4000 {
path := RelFilePath(fmt.Sprintf("dir/file-%05d.txt", i)) path := RelFilePath(fmt.Sprintf("dir/file-%05d.txt", i))
require.NoError(t, b.AddFileWithHash(path, FileSize(i), ModTime{}, hash)) require.NoError(t, b.AddFileWithHash(path, FileSize(i), ModTime{}, 0, hash))
} }
var buf bytes.Buffer var buf bytes.Buffer
@@ -452,7 +452,7 @@ func TestManifestString(t *testing.T) {
content := []byte("test") content := []byte("test")
reader := bytes.NewReader(content) reader := bytes.NewReader(content)
_, err := b.AddFile( _, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()), reader, nil, "test.txt", FileSize(len(content)), ModTime(time.Now()), 0, reader, nil,
) )
require.NoError(t, err) require.NoError(t, err)
@@ -484,7 +484,7 @@ func TestBuilderOmitsCreatedAtByDefault(t *testing.T) {
b := NewBuilder() b := NewBuilder()
content := []byte("hello") content := []byte("hello")
_, err := b.AddFile( _, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()), "test.txt", FileSize(len(content)), ModTime(time.Now()), 0,
bytes.NewReader(content), nil, bytes.NewReader(content), nil,
) )
require.NoError(t, err) require.NoError(t, err)
@@ -506,7 +506,7 @@ func TestBuilderIncludesCreatedAtWhenRequested(t *testing.T) {
content := []byte("hello") content := []byte("hello")
_, err := b.AddFile( _, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()), "test.txt", FileSize(len(content)), ModTime(time.Now()), 0,
bytes.NewReader(content), nil, bytes.NewReader(content), nil,
) )
require.NoError(t, err) require.NoError(t, err)
@@ -532,7 +532,7 @@ func TestBuilderDeterministicFileOrder(t *testing.T) {
content := []byte("content of " + name) content := []byte("content of " + name)
_, err := b.AddFile( _, err := b.AddFile(
RelFilePath(name), FileSize(len(content)), RelFilePath(name), FileSize(len(content)),
ModTime(time.Unix(1000, 0)), bytes.NewReader(content), nil, ModTime(time.Unix(1000, 0)), 0, bytes.NewReader(content), nil,
) )
require.NoError(t, err) require.NoError(t, err)
} }
+17 -3
View File
@@ -36,6 +36,7 @@ const (
StatusMissing // File not found on disk StatusMissing // File not found on disk
StatusSizeMismatch // File size differs from manifest StatusSizeMismatch // File size differs from manifest
StatusHashMismatch // File hash differs from manifest StatusHashMismatch // File hash differs from manifest
StatusModeMismatch // File permission bits differ from a recorded mode
StatusExtra // File exists on disk but not in manifest StatusExtra // File exists on disk but not in manifest
StatusError // Error occurred during verification StatusError // Error occurred during verification
) )
@@ -50,6 +51,8 @@ func (s Status) String() string {
return "SIZE_MISMATCH" return "SIZE_MISMATCH"
case StatusHashMismatch: case StatusHashMismatch:
return "HASH_MISMATCH" return "HASH_MISMATCH"
case StatusModeMismatch:
return "MODE_MISMATCH"
case StatusExtra: case StatusExtra:
return "EXTRA" return "EXTRA"
case StatusError: case StatusError:
@@ -408,11 +411,22 @@ func (c *Checker) checkFile(entry *MFFilePath, checkedBytes *FileSize) Result {
return Result{Path: relPath, Status: StatusError, Message: err.Error()} return Result{Path: relPath, Status: StatusError, Message: err.Error()}
} }
// Check against all hashes in manifest (at least one must match) // Check against all hashes in manifest (at least one must match),
// then against the recorded mode, where one is: 0 means none was.
for _, hash := range entry.GetHashes() { for _, hash := range entry.GetHashes() {
if bytes.Equal(computed, hash.GetMultiHash()) { if !bytes.Equal(computed, hash.GetMultiHash()) {
return Result{Path: relPath, Status: StatusOK} continue
} }
if entry.GetMode() != 0 && info.Mode().Perm() != os.FileMode(entry.GetMode()) {
return Result{
Path: relPath,
Status: StatusModeMismatch,
Message: "mode mismatch",
}
}
return Result{Path: relPath, Status: StatusOK}
} }
return Result{ return Result{
+52 -2
View File
@@ -34,6 +34,7 @@ func TestStatusString(t *testing.T) {
{StatusMissing, "MISSING"}, {StatusMissing, "MISSING"},
{StatusSizeMismatch, "SIZE_MISMATCH"}, {StatusSizeMismatch, "SIZE_MISMATCH"},
{StatusHashMismatch, "HASH_MISMATCH"}, {StatusHashMismatch, "HASH_MISMATCH"},
{StatusModeMismatch, "MODE_MISMATCH"},
{StatusExtra, "EXTRA"}, {StatusExtra, "EXTRA"},
{StatusError, "ERROR"}, {StatusError, "ERROR"},
{Status(99), "UNKNOWN"}, {Status(99), "UNKNOWN"},
@@ -59,7 +60,7 @@ func createTestManifest(
for path, content := range files { for path, content := range files {
reader := bytes.NewReader(content) reader := bytes.NewReader(content)
_, err := builder.AddFile( _, err := builder.AddFile(
RelFilePath(path), FileSize(len(content)), ModTime(time.Now()), reader, nil, RelFilePath(path), FileSize(len(content)), ModTime(time.Now()), 0, reader, nil,
) )
require.NoError(t, err) require.NoError(t, err)
} }
@@ -279,7 +280,8 @@ func TestCheckMissingFile(t *testing.T) {
missingCount++ missingCount++
assert.Equal(t, RelFilePath("missing.txt"), r.Path) assert.Equal(t, RelFilePath("missing.txt"), r.Path)
case StatusSizeMismatch, StatusHashMismatch, StatusExtra, StatusError: case StatusSizeMismatch, StatusHashMismatch, StatusModeMismatch,
StatusExtra, StatusError:
// Not expected in this test; counted assertions below will fail. // Not expected in this test; counted assertions below will fail.
} }
} }
@@ -349,6 +351,54 @@ func TestCheckHashMismatch(t *testing.T) {
assert.Equal(t, RelFilePath(testFileName), r.Path) assert.Equal(t, RelFilePath(testFileName), r.Path)
} }
// A recorded mode other than 0000 that differs from the file's permission
// bits fails the check; a recorded 0000 is never checked.
func TestCheckMode(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
recorded os.FileMode
onDisk os.FileMode
want Status
}{
{"recorded mode matches", 0o640, 0o640, StatusOK},
{"recorded mode differs", 0o640, 0o600, StatusModeMismatch},
{"0000 is not checked", 0, 0o600, StatusOK},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
content := []byte("content")
b := NewBuilder()
_, err := b.AddFile(testFileName, FileSize(len(content)), ModTime{},
tc.recorded, bytes.NewReader(content), nil)
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
require.NoError(t, afero.WriteFile(fs, testManifestPath, buf.Bytes(), 0o644))
require.NoError(t, fs.MkdirAll(testDataDir, 0o755))
require.NoError(t, afero.WriteFile(fs,
testDataDir+"/"+testFileName, content, tc.onDisk))
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
require.NoError(t, err)
results := make(chan Result, 1)
require.NoError(t, chk.Check(context.Background(), results, nil))
assert.Equal(t, tc.want, (<-results).Status)
})
}
}
func TestCheckWithProgress(t *testing.T) { func TestCheckWithProgress(t *testing.T) {
t.Parallel() t.Parallel()
+4 -4
View File
@@ -29,8 +29,8 @@ const (
// Bytes decoding sets aside for each file entry, hash, timestamp and // Bytes decoding sets aside for each file entry, hash, timestamp and
// MIME type, however short its encoding. checkDecodedSize refuses an // MIME type, however short its encoding. checkDecodedSize refuses an
// inner message for which these add up to more than maxDecodedGrowth // inner message for which these add up to more than maxDecodedGrowth
// times its size. // times its size. The mode is held in the file entry itself.
decodedFileEntrySize = 160 decodedFileEntrySize = 176
decodedHashSize = 112 decodedHashSize = 112
decodedTimestampSize = 64 decodedTimestampSize = 64
decodedMIMETypeSize = 16 decodedMIMETypeSize = 16
@@ -38,7 +38,7 @@ const (
// Each file entry mfer writes holds a path of at least one byte, a // Each file entry mfer writes holds a path of at least one byte, a
// multihash at least as long as SHA-256's 34 bytes (AddFileWithHash // multihash at least as long as SHA-256's 34 bytes (AddFileWithHash
// refuses shorter ones) and a modification time: at least 47 bytes, // refuses shorter ones) and a modification time: at least 47 bytes,
// counted at 336. So its manifests add up to at most about 7.15 times // counted at 352. So its manifests add up to at most about 7.49 times
// their size, and this limit is about 12% above that. // their size, and this limit is about 7% above that.
maxDecodedGrowth = 8 maxDecodedGrowth = 8
) )
+9 -9
View File
@@ -119,11 +119,11 @@ func TestDeserializeRejectsInvalidEntryPaths(t *testing.T) {
} }
// Entries of a path, an empty hash, an empty MIME type and empty modification // Entries of a path, an empty hash, an empty MIME type and empty modification
// and change times are counted at 416 bytes each (160 + 112 + 16 + 64 + 64) // and change times are counted at 432 bytes each (176 + 112 + 16 + 64 + 64)
// and take 16 bytes plus the path to encode. A 35-character path makes that // and take 16 bytes plus the path to encode. A 37-character path makes that
// 51 bytes, about 8.2 times: refused, and leaving any one of the five // 53 bytes, about 8.2 times: refused, and leaving any one of the five
// uncounted, even the MIME type, brings it under 8. A 37-character path makes // uncounted, even the MIME type, brings it under 8. A 39-character path makes
// it 53 bytes, about 7.8 times: loaded. // it 55 bytes, about 7.9 times: loaded.
func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) { func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
t.Parallel() t.Parallel()
@@ -131,8 +131,8 @@ func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
pathLen int pathLen int
refused bool refused bool
}{ }{
{35, true}, {37, true},
{37, false}, {39, false},
} }
for _, tt := range tests { for _, tt := range tests {
@@ -215,7 +215,7 @@ func TestDeserializeLoadsDensestManifest(t *testing.T) {
const files = 10000 const files = 10000
for i := range files { for i := range files {
name := RelFilePath(strconv.FormatInt(int64(i), 36)) name := RelFilePath(strconv.FormatInt(int64(i), 36))
require.NoError(t, b.AddFileWithHash(name, 0, ModTime(time.Unix(0, 0)), hash)) require.NoError(t, b.AddFileWithHash(name, 0, ModTime(time.Unix(0, 0)), 0, hash))
} }
var buf bytes.Buffer var buf bytes.Buffer
@@ -233,7 +233,7 @@ func TestDeserializeValidManifestRoundTrips(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
b := NewBuilder() b := NewBuilder()
require.NoError(t, b.AddFileWithHash("dir/file.txt", 123, ModTime{}, hash)) require.NoError(t, b.AddFileWithHash("dir/file.txt", 123, ModTime{}, 0, hash))
var buf bytes.Buffer var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf)) require.NoError(t, b.Build(context.Background(), &buf))
+4 -4
View File
@@ -187,7 +187,7 @@ func TestBuilderWithSigning(t *testing.T) {
// Add a test file // Add a test file
content := []byte("test file content") content := []byte("test file content")
reader := bytes.NewReader(content) reader := bytes.NewReader(content)
_, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, reader, nil) _, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, 0, reader, nil)
require.NoError(t, err) require.NoError(t, err)
// Build the manifest // Build the manifest
@@ -314,7 +314,7 @@ func TestManifestSignatureVerification(t *testing.T) {
// Add a test file // Add a test file
content := []byte("test file content for verification") content := []byte("test file content for verification")
reader := bytes.NewReader(content) reader := bytes.NewReader(content)
_, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, reader, nil) _, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, 0, reader, nil)
require.NoError(t, err) require.NoError(t, err)
// Build the manifest // Build the manifest
@@ -344,7 +344,7 @@ func TestManifestTamperedSignatureFails(t *testing.T) {
content := []byte("test file content") content := []byte("test file content")
reader := bytes.NewReader(content) reader := bytes.NewReader(content)
_, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, reader, nil) _, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, 0, reader, nil)
require.NoError(t, err) require.NoError(t, err)
var buf bytes.Buffer var buf bytes.Buffer
@@ -377,7 +377,7 @@ func TestBuilderWithoutSigning(t *testing.T) {
// Add a test file // Add a test file
content := []byte("test file content") content := []byte("test file content")
reader := bytes.NewReader(content) reader := bytes.NewReader(content)
_, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, reader, nil) _, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, 0, reader, nil)
require.NoError(t, err) require.NoError(t, err)
// Build the manifest // Build the manifest
+15 -5
View File
@@ -337,9 +337,11 @@ type MFFilePath struct {
// gotta have at least one: // gotta have at least one:
Hashes []*MFFileChecksum `protobuf:"bytes,3,rep,name=hashes,proto3" json:"hashes,omitempty"` Hashes []*MFFileChecksum `protobuf:"bytes,3,rep,name=hashes,proto3" json:"hashes,omitempty"`
// optional per-file metadata // optional per-file metadata
MimeType *string `protobuf:"bytes,301,opt,name=mimeType,proto3,oneof" json:"mimeType,omitempty"` MimeType *string `protobuf:"bytes,301,opt,name=mimeType,proto3,oneof" json:"mimeType,omitempty"`
Mtime *Timestamp `protobuf:"bytes,302,opt,name=mtime,proto3,oneof" json:"mtime,omitempty"` Mtime *Timestamp `protobuf:"bytes,302,opt,name=mtime,proto3,oneof" json:"mtime,omitempty"`
Ctime *Timestamp `protobuf:"bytes,303,opt,name=ctime,proto3,oneof" json:"ctime,omitempty"` Ctime *Timestamp `protobuf:"bytes,303,opt,name=ctime,proto3,oneof" json:"ctime,omitempty"`
// permission bits, at most 0777; 0 when not recorded
Mode uint32 `protobuf:"varint,304,opt,name=mode,proto3" json:"mode,omitempty"`
unknownFields protoimpl.UnknownFields unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache sizeCache protoimpl.SizeCache
} }
@@ -416,6 +418,13 @@ func (x *MFFilePath) GetCtime() *Timestamp {
return nil return nil
} }
func (x *MFFilePath) GetMode() uint32 {
if x != nil {
return x.Mode
}
return 0
}
type MFFileChecksum struct { type MFFileChecksum struct {
state protoimpl.MessageState `protogen:"open.v1"` state protoimpl.MessageState `protogen:"open.v1"`
// 1.0 golang implementation must write a multihash here // 1.0 golang implementation must write a multihash here
@@ -561,7 +570,7 @@ const file_mf_proto_rawDesc = "" +
"\n" + "\n" +
"_signatureB\t\n" + "_signatureB\t\n" +
"\a_signerB\x10\n" + "\a_signerB\x10\n" +
"\x0e_signingPubKey\"\xf0\x01\n" + "\x0e_signingPubKey\"\x85\x02\n" +
"\n" + "\n" +
"MFFilePath\x12\x12\n" + "MFFilePath\x12\x12\n" +
"\x04path\x18\x01 \x01(\tR\x04path\x12\x12\n" + "\x04path\x18\x01 \x01(\tR\x04path\x12\x12\n" +
@@ -571,7 +580,8 @@ const file_mf_proto_rawDesc = "" +
"\x05mtime\x18\xae\x02 \x01(\v2\n" + "\x05mtime\x18\xae\x02 \x01(\v2\n" +
".TimestampH\x01R\x05mtime\x88\x01\x01\x12&\n" + ".TimestampH\x01R\x05mtime\x88\x01\x01\x12&\n" +
"\x05ctime\x18\xaf\x02 \x01(\v2\n" + "\x05ctime\x18\xaf\x02 \x01(\v2\n" +
".TimestampH\x02R\x05ctime\x88\x01\x01B\v\n" + ".TimestampH\x02R\x05ctime\x88\x01\x01\x12\x13\n" +
"\x04mode\x18\xb0\x02 \x01(\rR\x04modeB\v\n" +
"\t_mimeTypeB\b\n" + "\t_mimeTypeB\b\n" +
"\x06_mtimeB\b\n" + "\x06_mtimeB\b\n" +
"\x06_ctime\".\n" + "\x06_ctime\".\n" +
+2
View File
@@ -59,6 +59,8 @@ message MFFilePath {
optional string mimeType = 301; optional string mimeType = 301;
optional Timestamp mtime = 302; optional Timestamp mtime = 302;
optional Timestamp ctime = 303; optional Timestamp ctime = 303;
// permission bits, at most 0777; 0 when not recorded
uint32 mode = 304;
} }
message MFFileChecksum { message MFFileChecksum {
+1 -1
View File
@@ -1 +1 @@
fa6fceaba5553c8667c631994535fe5c307c8adb19f3ad289babf79a0f438650 mf.proto 3d4dcb0b2f4640dd3ae6bb48b833e9f26ae9771e2d3e88bd646e7f1724dda654 mf.proto
+1
View File
@@ -41,6 +41,7 @@ func TestFileEntryFieldsMatchSpec(t *testing.T) {
"mimeType": 301, "mimeType": 301,
"mtime": 302, "mtime": 302,
"ctime": 303, "ctime": 303,
"mode": 304,
} }
got := map[string]protoreflect.FieldNumber{} got := map[string]protoreflect.FieldNumber{}
+12
View File
@@ -52,6 +52,9 @@ type ScannerOptions struct {
// IncludeTimestamps includes a createdAt timestamp in the manifest // IncludeTimestamps includes a createdAt timestamp in the manifest
// (default: omit for determinism). // (default: omit for determinism).
IncludeTimestamps bool IncludeTimestamps bool
// IncludePermissions records each file's permission bits, 0777 at
// most, in the manifest (default: record 0000).
IncludePermissions bool
// Fs is the filesystem to use, defaults to OsFs if nil. // Fs is the filesystem to use, defaults to OsFs if nil.
Fs afero.Fs Fs afero.Fs
// SigningOptions holds GPG signing options (nil = no signing). // SigningOptions holds GPG signing options (nil = no signing).
@@ -69,6 +72,7 @@ type FileEntry struct {
Size FileSize // File size in bytes Size FileSize // File size in bytes
Mtime ModTime // Last modification time Mtime ModTime // Last modification time
Ctime time.Time // Creation time (platform-dependent) Ctime time.Time // Creation time (platform-dependent)
Mode fs.FileMode // Permission bits (Perm() of the file's mode)
} }
// Scanner accumulates files and generates manifests from them. // Scanner accumulates files and generates manifests from them.
@@ -353,11 +357,18 @@ func (s *Scanner) scanFile(
}(scannedBytes, scannedFiles) }(scannedBytes, scannedFiles)
} }
// A mode of 0 records 0000, which means none was recorded.
var mode fs.FileMode
if s.options.IncludePermissions {
mode = entry.Mode
}
// Add to manifest with progress channel // Add to manifest with progress channel
bytesRead, err := builder.AddFile( bytesRead, err := builder.AddFile(
entry.Path, entry.Path,
entry.Size, entry.Size,
entry.Mtime, entry.Mtime,
mode,
f, f,
fileProgress, fileProgress,
) )
@@ -461,6 +472,7 @@ func (s *Scanner) enumerateFileWithInfo(
AbsPath: AbsFilePath(absPath), AbsPath: AbsFilePath(absPath),
Size: FileSize(info.Size()), Size: FileSize(info.Size()),
Mtime: ModTime(info.ModTime()), Mtime: ModTime(info.ModTime()),
Mode: info.Mode().Perm(),
// Note: Ctime not available from fs.FileInfo on all platforms // Note: Ctime not available from fs.FileInfo on all platforms
// Will need platform-specific code to extract it // Will need platform-specific code to extract it
} }
+41
View File
@@ -4,6 +4,7 @@ package mfer
import ( import (
"bytes" "bytes"
"context" "context"
"os"
"testing" "testing"
"time" "time"
@@ -350,6 +351,46 @@ func TestScannerFileEntryFields(t *testing.T) {
assert.WithinDuration(t, now, time.Time(entry.Mtime), 2*time.Second) assert.WithinDuration(t, now, time.Time(entry.Mtime), 2*time.Second)
} }
// A manifest records every mode as 0000 unless the creator asks for
// permissions; then it records each file's permission bits and never the
// setuid, setgid or sticky bits.
func TestScannerRecordsModeOnlyWhenAsked(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/"+testFile1, []byte("a"), 0o640))
require.NoError(t, afero.WriteFile(fs, "/run.sh", []byte("b"), 0o755))
require.NoError(t, afero.WriteFile(fs, "/su", []byte("c"), 0o755))
require.NoError(t, fs.Chmod("/su", 0o755|os.ModeSetuid|os.ModeSetgid|os.ModeSticky))
for _, tc := range []struct {
includePermissions bool
want map[string]uint32
}{
{false, map[string]uint32{testFile1: 0, "run.sh": 0, "su": 0}},
{true, map[string]uint32{testFile1: 0o640, "run.sh": 0o755, "su": 0o755}},
} {
s := NewScannerWithOptions(&ScannerOptions{
Fs: fs,
IncludePermissions: tc.includePermissions,
})
require.NoError(t, s.EnumerateFS(fs, "/", nil))
var buf bytes.Buffer
require.NoError(t, s.ToManifest(context.Background(), &buf, nil))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
got := map[string]uint32{}
for _, f := range m.Files() {
got[f.GetPath()] = f.GetMode()
}
assert.Equal(t, tc.want, got, "IncludePermissions: %v", tc.includePermissions)
}
}
func TestScannerLargeFileEnumeration(t *testing.T) { func TestScannerLargeFileEnumeration(t *testing.T) {
t.Parallel() t.Parallel()