Load the manifest check downloads from memory, not a temp file (closes #223)

check given a URL wrote the manifest it downloaded to a file in the temp
directory before loading it, so it failed wherever that directory is
missing or cannot be written, such as the scratch image script/cibuild
builds. It now keeps the manifest in memory, as fetch, list and export
do. NewChecker reads a manifest handed to it as a reader through the new
CheckerOptions.ManifestReader field, in place of ManifestPath, because it
otherwise reads the manifest from the filesystem holding the files it
checks. Only check uses that field. verifyRequiredSigner takes the
signer's fingerprint instead of a Checker, so fetch passes the signer of
the manifest it has already loaded and builds no Checker.

Model: opus-5-5

README.md

@@ -13,11 +13,14 @@ The structure of these files can be found
 schema `mfer/mf.proto` for exact field numbers and types. Both are included in
 the [project repository](https://git.eeqj.de/sneak/mfer).
 
-Releases are git tags named by [SemVer](https://semver.org), starting with
+The current version is pre-1.0 and while the repo was published in 2022, there
-`v1.0.0`. No file in the tree records a version number. `mfer version` prints
+has not yet been any versioned release. [SemVer](https://semver.org) will be
-the revision the build stamped into the binary, which for `make build` and the
+used for releases.
-Docker image is the output of `git describe` on the checkout they build. A
+
-binary built without that stamp, as
+A version is a git tag, and no file in the tree records a version number.
+`mfer version` prints the revision the build stamped into the binary, which for
+`make build` and the Docker image is the output of `git describe` on the
+checkout they build. A binary built without that stamp, as
 `go install sneak.berlin/go/mfer/cmd/mfer@<tag>` builds it, prints the module
 version Go records in it instead. A program that imports the `mfer` package
 finds the version it was built against in its own build information
@@ -196,7 +199,8 @@ The manifest file would do several important things:
   a signature and a public key alongside in a single file
 - contain a list of files, each with a relative path to the manifest
 - contain manifest timestamp
-- contain mtime information for files so that file metadata can be preserved
+- contain ctime/mtime information for files so that file metadata can be
+  preserved
 - contain cryptographic checksums in several different algorithms for each file
     - probably encoded with multihash to indicate algo + hash
     - sha256 at the minimum
@@ -286,9 +290,6 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
       only on the paths it is given and the files under them, so with the same
       `--seed` and an unchanged tree it writes the same bytes wherever the
       manifest goes. The file it writes to is never listed
-    - never lists `index.mf` at the top of a directory it is given, or a file it
-      is given by that name, wherever it writes the manifest, since `fetch`
-      saves the manifest there; `index.mf` in a subdirectory is listed
 - `mfer check` / `mfer check .`
     - verifies checksums of all files in manifest, displaying error and exiting
       nonzero if any files are missing or corrupted, or have permission bits
@@ -303,15 +304,11 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
       `--no-extra-files` each one is a failure instead. A named pipe, socket or
       device, a symlink to a directory and a broken symlink are never reported,
       since gen cannot list them
-    - never reports `index.mf` at the top of the base directory as a file the
-      manifest does not list, even when given another manifest or a URL, since
-      gen and fetch write the manifest there
 - `mfer freshen` / `mfer freshen .`
     - rewrites `index.mf` to list the files now under the directory holding it,
       or under the one `--base` names, hashing only the files that are new or
       changed
-    - lists regular files only, and never `index.mf` at the top of that
+    - lists regular files only, as gen does
-      directory, as gen does
     - leaves out hidden files unless given `--include-dotfiles`, and symlinks
       unless given `--follow-symlinks`, which lists each symlink to a regular
       file under its own name with the contents of the file it points to
@@ -368,8 +365,8 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
     - (fetches `https://example.com/manifestdirectory/index.mf`, downloads and
       checksums all files, resumes any that exist locally already)
 - `mfer fetch https://example.com/manifestdirectory/`
-- a command line option to zero/omit mtime, as well as manifest timestamp, and
+- a command line option to zero/omit mtime/ctime, as well as manifest timestamp,
-  sort all directory listings so that manifest file generation is
+  and sort all directory listings so that manifest file generation is
   deterministic/reproducible
 - URL format
   `mfer fetch https://exmaple.com/manifestdirectory/?key=5539AD00DE4C42F3AFE11575052443F4DF2A55C2`

docs/FORMAT.md

@@ -9,11 +9,8 @@ including their paths, sizes, and cryptographic checksums. It supports optional
 OpenPGP signatures for integrity verification and optional timestamps and file
 permissions for metadata preservation.
 
-Nothing goes in the 1.0 manifest that 1.0 does not read or write, and no field
+Nothing goes in the 1.0 manifest that 1.0 does not read or write: no field is
-is kept for later use. [`mfer/mf.proto`](../mfer/mf.proto) reserves the field
+reserved or kept for later use.
-numbers 301 and 303 and the names `mimeType` and `ctime` in `MFFilePath`, so
-that a later version cannot reuse them by accident. No field uses them, and
-readers ignore them like any field they do not know.
 
 ## File Structure
 
@@ -61,11 +58,11 @@ enforce a decompression size limit to prevent decompression bombs. The reference
 implementation limits decompressed size to 256 MiB. It writes zstd frames with a
 window of at most 8 MiB, the largest window the zstd format recommends decoders
 support, and refuses frames that ask for a larger one. It also refuses an inner
-message whose file entries, hashes and timestamps, counted at 160, 112 and 64
+message whose file entries, hashes, timestamps and MIME types, counted at 176,
-bytes each, add up to more than 8 times its size. It refuses a manifest file
+112, 64 and 16 bytes each, add up to more than 8 times its size. It refuses a
-larger than 258 MiB without reading the rest of it: zstd's worst case grows a
+manifest file larger than 258 MiB without reading the rest of it: zstd's worst
-256 MiB inner message by 1/256 to 257 MiB, and the last MiB is room for the
+case grows a 256 MiB inner message by 1/256 to 257 MiB, and the last MiB is room
-signature, the signing key and the other outer fields.
+for the signature, the signing key and the other outer fields.
 
 ## Inner Message (`MFFile`)
 
@@ -84,25 +81,28 @@ After decompressing `innerMessage`, the result is a serialized `MFFile`
 Each file entry contains:
 
 | Field      | Number | Type                      | Description                                                  |
-| -------- | ------ | ------------------------- | ------------------------------------------------------------ |
+| ---------- | ------ | ------------------------- | ------------------------------------------------------------ |
 | `path`     | 1      | string                    | Relative file path (see Path Rules)                          |
 | `size`     | 2      | int64                     | File size in bytes                                           |
 | `hashes`   | 3      | repeated `MFFileChecksum` | Must include exactly one SHA-256 multihash (see Hash Format) |
+| `mimeType` | 301    | string (optional)         | MIME type                                                    |
 | `mtime`    | 302    | Timestamp (optional)      | Modification time                                            |
+| `ctime`    | 303    | Timestamp (optional)      | Change time (inode metadata change)                          |
 | `mode`     | 304    | uint32                    | Permission bits (see File Mode)                              |
 
 ## File Mode
 
 `mode` holds a file's Unix permission bits, the nine `rwx` bits, so it is never
-above `0777` (octal); the setuid, setgid and sticky bits are never recorded. A
+above `0777` (octal); the setuid, setgid and sticky bits are never recorded.
-reader must reject a manifest with an entry whose mode is above `0777`. Writers
+Writers record `0000` unless whoever creates the manifest asks for permissions.
-record `0000` unless whoever creates the manifest asks for permissions. `0000`,
+`0000`, the proto3 default, means no mode was recorded: readers never check or
-the proto3 default, means no mode was recorded: readers never check or apply it.
+apply it.
 
 The reference implementation records modes when `gen` or `freshen` is given
 `--include-permissions`. `check` fails a file whose permission bits differ from
 a recorded mode other than `0000`. `fetch` sets a recorded mode other than
-`0000` on each file it writes.
+`0000` on each file it writes, and refuses a manifest that records a mode above
+`0777` before it requests any file.
 
 ## Path Rules
 

internal/cli/check.go

@@ -195,10 +195,7 @@ func countCheckFailures(
 
 // findExtraFiles reports files present on disk but absent from the
 // manifest, and anything the search cannot read: each is a failure, logged
-// at error level, under --no-extra-files, otherwise a warning. index.mf at
+// at error level, under --no-extra-files, otherwise a warning.
-// the top of the base directory is never reported as absent from the
-// manifest, even when check is given another manifest or a URL, since gen
-// and fetch write the manifest there.
 func findExtraFiles(
 	ctx context.Context, cmd *cli.Command, chk *mfer.Checker, failures *int64,
 ) error {
@@ -207,10 +204,6 @@ func findExtraFiles(
 
 	go func() {
 		for result := range extraResults {
-			if result.Status == mfer.StatusExtra && result.Path == defaultManifestName {
-				continue
-			}
-
 			if cmd.Bool("no-extra-files") {
 				*failures++
 

internal/cli/entry_test.go

@@ -36,7 +36,6 @@ const (
 	testManifest    = "/manifest.mf"
 	testFlagBase    = "--base"
 	testFlagNoExtra = "--no-extra-files"
-	testFlagFollowSymlinks = "--follow-symlinks"
 	testFlagVersion = "--version"
 	testFlagVerbose = "--verbose"
 	testFlagHelp    = "--help"
@@ -1003,7 +1002,7 @@ func TestCheckReportsUnlistedSymlinkToFileOnly(t *testing.T) {
 		exitCode int
 		extras   []string // what check reports as not in the manifest
 	}{
-		"gen --follow-symlinks": {[]string{testFlagFollowSymlinks}, 0, nil},
+		"gen --follow-symlinks": {[]string{"--follow-symlinks"}, 0, nil},
 		"plain gen":             {nil, 1, []string{"filelink"}},
 	} {
 		t.Run(name, func(t *testing.T) {
@@ -1191,99 +1190,6 @@ func TestCheckURLManifestWithoutTempDirectory(t *testing.T) {
 	assert.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
 }
 
-// TestCheckNeverReportsIndexMFAtTopOfBase checks a tree holding index.mf at
-// its top, where gen and fetch write the manifest, against a manifest that
-// does not list it, given by URL and by a path outside the tree: check never
-// reports it, and passes with and without --no-extra-files. index.mf in a
-// subdirectory, and the temp file fetch saves index.mf through, are still
-// reported.
-func TestCheckNeverReportsIndexMFAtTopOfBase(t *testing.T) {
-	t.Parallel()
-
-	files := map[string][]byte{testFileTxt: []byte("hello")}
-	manifest := manifestOf(t, files)
-
-	server := httptest.NewServer(fetchTestHandler(manifest, files))
-	t.Cleanup(server.Close)
-
-	manifestOutsideTree := filepath.Join(t.TempDir(), "other.mf")
-	require.NoError(t, os.WriteFile(manifestOutsideTree, manifest, 0o600))
-
-	for name, manifestArg := range map[string]string{
-		"manifest by URL":  server.URL + "/" + defaultManifestName,
-		"manifest by path": manifestOutsideTree,
-	} {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			base := t.TempDir()
-			require.NoError(t, os.WriteFile(
-				filepath.Join(base, testFileTxt), files[testFileTxt], 0o600))
-			require.NoError(t, os.WriteFile(
-				filepath.Join(base, defaultManifestName), manifest, 0o600))
-
-			check := func(flags ...string) (int, string) {
-				opts := testOpts(slices.Concat(
-					[]string{testApp, cmdCheck}, flags,
-					[]string{testFlagBase, base, manifestArg},
-				), afero.NewOsFs())
-
-				return runCLI(opts), testStderr(t, opts)
-			}
-
-			exitCode, stderr := check()
-			assert.Equal(t, 0, exitCode, "stderr: %s", stderr)
-			assert.NotContains(t, stderr, "EXTRA: "+defaultManifestName)
-
-			exitCode, stderr = check(testFlagNoExtra)
-			assert.Equal(t, 0, exitCode, "stderr: %s", stderr)
-			assert.NotContains(t, stderr, "EXTRA: "+defaultManifestName)
-
-			reported := []string{
-				testSubdirManifest,
-				tempPathFor(defaultManifestName),
-			}
-
-			require.NoError(t, os.Mkdir(filepath.Join(base, testSubdir), 0o750))
-
-			for _, path := range reported {
-				require.NoError(t,
-					os.WriteFile(filepath.Join(base, path), manifest, 0o600))
-			}
-
-			exitCode, stderr = check(testFlagNoExtra)
-			assert.Equal(t, 1, exitCode, "stderr: %s", stderr)
-			assert.NotContains(t, stderr, "EXTRA: "+defaultManifestName)
-
-			for _, path := range reported {
-				assert.Contains(t, stderr, "EXTRA: "+path)
-			}
-		})
-	}
-}
-
-// TestCheckChecksListedIndexMF checks a tree against a manifest that lists
-// index.mf at the top of the tree, where that file has changed since: check
-// reports it like any other changed file.
-func TestCheckChecksListedIndexMF(t *testing.T) {
-	t.Parallel()
-
-	manifestPath := filepath.Join(t.TempDir(), "other.mf")
-	require.NoError(t, os.WriteFile(manifestPath, manifestOf(t, map[string][]byte{
-		defaultManifestName: []byte("listed"),
-	}), 0o600))
-
-	base := t.TempDir()
-	require.NoError(t, os.WriteFile(
-		filepath.Join(base, defaultManifestName), []byte("edited"), 0o600))
-
-	opts := testOpts([]string{
-		testApp, cmdCheck, testFlagBase, base, manifestPath,
-	}, afero.NewOsFs())
-	assert.Equal(t, 1, runCLI(opts), "stderr: %s", testStderr(t, opts))
-	assert.Contains(t, testStderr(t, opts), "HASH_MISMATCH: "+defaultManifestName)
-}
-
 // unlistableDirFs is a filesystem on which one directory cannot be listed.
 type unlistableDirFs struct {
 	afero.Fs
@@ -1332,60 +1238,6 @@ func TestCheckWithUnlistableDirectory(t *testing.T) {
 	assert.Contains(t, testStderr(t, opts), "unlisted.txt")
 }
 
-// TestCheckReportsUnreadableIndexMFAtTopOfBase has a directory named
-// index.mf at the top of the base that cannot be listed. check leaves out
-// only the report that index.mf is not in the manifest, so it reports the
-// error and fails under --no-extra-files.
-func TestCheckReportsUnreadableIndexMFAtTopOfBase(t *testing.T) {
-	t.Parallel()
-
-	unreadable := filepath.Join(testDir, defaultManifestName)
-
-	mem := afero.NewMemMapFs()
-	require.NoError(t, mem.MkdirAll(unreadable, 0o755))
-	writeTestFile(t, mem, testFile1, "hello")
-
-	opts := testOpts([]string{
-		testApp, cmdGenerate, "-q", "-o", testManifest, testDir,
-	}, mem)
-	require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-	fs := unlistableDirFs{Fs: mem, dir: unreadable}
-
-	opts = testOpts([]string{
-		testApp, cmdCheck, testFlagNoExtra, testFlagBase, testDir, testManifest,
-	}, fs)
-	assert.Equal(t, 1, runCLI(opts), "stderr: %s", testStderr(t, opts))
-	assert.Contains(t, testStderr(t, opts), "ERROR: "+defaultManifestName)
-}
-
-// TestCheckReportsUpperCaseIndexMFAtTopOfBase checks a tree holding
-// INDEX.MF at its top against a manifest that does not list it. Only the
-// name index.mf, in exactly that letter case, is left out, so check reports
-// INDEX.MF. The tree is in memory, where the two names are two files
-// whatever the letter case rules of the host's filesystem.
-func TestCheckReportsUpperCaseIndexMFAtTopOfBase(t *testing.T) {
-	t.Parallel()
-
-	upperCase := strings.ToUpper(defaultManifestName)
-
-	fs := afero.NewMemMapFs()
-	writeTestFile(t, fs, testFile1, "hello")
-
-	opts := testOpts([]string{
-		testApp, cmdGenerate, "-q", "-o", testManifest, testDir,
-	}, fs)
-	require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-	writeTestFile(t, fs, filepath.Join(testDir, upperCase), "not listed")
-
-	opts = testOpts([]string{
-		testApp, cmdCheck, testFlagNoExtra, testFlagBase, testDir, testManifest,
-	}, fs)
-	assert.Equal(t, 1, runCLI(opts), "stderr: %s", testStderr(t, opts))
-	assert.Contains(t, testStderr(t, opts), "EXTRA: "+upperCase)
-}
-
 func TestGenerateAtomicWriteNoTempFileOnSuccess(t *testing.T) {
 	t.Parallel()
 
@@ -1568,141 +1420,6 @@ func TestGenerateLeavesLeftoverTempFileOutOfListing(t *testing.T) {
 	assert.Equal(t, []string{testFileTxt}, manifestPaths(t, fs, output))
 }
 
-// TestGenerateLeavesTopIndexMFOutOfListing runs gen with --output outside
-// the paths it is given: index.mf at the top of each directory given, and
-// a file given by that name, are not listed, since fetch saves the manifest
-// as index.mf at the top of the tree, while sub/index.mf is listed.
-func TestGenerateLeavesTopIndexMFOutOfListing(t *testing.T) {
-	t.Parallel()
-
-	// Paths are relative to a temp dir holding tree, which holds file.txt,
-	// index.mf and sub/index.mf, and second, which holds second.txt and
-	// index.mf.
-	for name, tc := range map[string]struct {
-		paths  []string
-		listed []string
-	}{
-		"directory": {
-			[]string{"tree"},
-			[]string{testFileTxt, testSubdirManifest},
-		},
-		"two directories": {
-			[]string{"tree", "second"},
-			[]string{testFileTxt, testSubdirManifest, "second.txt"},
-		},
-		"files": {
-			[]string{"tree/" + testFileTxt, "tree/" + defaultManifestName},
-			[]string{testFileTxt},
-		},
-	} {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			root := t.TempDir()
-			tree := filepath.Join(root, "tree")
-			second := filepath.Join(root, "second")
-
-			fs := afero.NewOsFs()
-			require.NoError(t, fs.MkdirAll(filepath.Join(tree, testSubdir), 0o750))
-			require.NoError(t, fs.MkdirAll(second, 0o750))
-			writeTestFile(t, fs, filepath.Join(tree, testFileTxt), "hello")
-			writeTestFile(t, fs, filepath.Join(tree, defaultManifestName), "an ordinary file")
-			writeTestFile(t, fs, filepath.Join(tree, testSubdirManifest), "an ordinary file")
-			writeTestFile(t, fs, filepath.Join(second, "second.txt"), "second")
-			writeTestFile(t, fs, filepath.Join(second, defaultManifestName), "an ordinary file")
-
-			paths := make([]string, 0, len(tc.paths))
-			for _, path := range tc.paths {
-				paths = append(paths, filepath.Join(root, path))
-			}
-
-			output := filepath.Join(root, "other.mf")
-			opts := testOpts(slices.Concat(
-				[]string{testApp, cmdGenerate, "-q", "-o", output}, paths,
-			), fs)
-			require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-			assert.ElementsMatch(t, tc.listed, manifestPaths(t, fs, output))
-		})
-	}
-}
-
-// TestGenerateListsTargetOfTopIndexMFSymlink makes index.mf at the top of
-// the tree a symlink to another file in it, and runs gen with --output
-// outside the tree: the file it points to is listed under its own name,
-// and the symlink, which gen lists only with --follow-symlinks, is left out
-// like any index.mf at the top of the tree.
-func TestGenerateListsTargetOfTopIndexMFSymlink(t *testing.T) {
-	t.Parallel()
-
-	// Paths are relative to tree, which holds file.txt, sub/index.mf and
-	// index.mf, a symlink to target.
-	for name, tc := range map[string]struct {
-		target string
-		flags  []string
-		path   string // the path gen is given
-		listed []string
-	}{
-		"directory given": {
-			testSubdirManifest, nil, ".",
-			[]string{testFileTxt, testSubdirManifest},
-		},
-		"directory given, --follow-symlinks": {
-			testSubdirManifest,
-			[]string{testFlagFollowSymlinks},
-			".",
-			[]string{testFileTxt, testSubdirManifest},
-		},
-		"file given": {
-			testFileTxt, nil, testFileTxt,
-			[]string{testFileTxt},
-		},
-	} {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			root := t.TempDir()
-			tree := filepath.Join(root, "tree")
-
-			fs := afero.NewOsFs()
-			require.NoError(t, fs.MkdirAll(filepath.Join(tree, testSubdir), 0o750))
-			writeTestFile(t, fs, filepath.Join(tree, testFileTxt), "hello")
-			writeTestFile(t, fs, filepath.Join(tree, testSubdirManifest), "an ordinary file")
-			require.NoError(t,
-				os.Symlink(tc.target, filepath.Join(tree, defaultManifestName)))
-
-			output := filepath.Join(root, "other.mf")
-			opts := testOpts(slices.Concat(
-				[]string{testApp, cmdGenerate, "-q", "-o", output}, tc.flags,
-				[]string{filepath.Join(tree, tc.path)},
-			), fs)
-			require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-			assert.ElementsMatch(t, tc.listed, manifestPaths(t, fs, output))
-		})
-	}
-}
-
-// TestGenerateListsIndexMFBesideFileGiven runs gen with --output outside
-// the tree, given sub/b.txt and the tree: sub/index.mf, beside the file
-// given, is listed, since a file given is left out only when its own name
-// is index.mf.
-//
-//nolint:paralleltest // changes the process-global working directory
-func TestGenerateListsIndexMFBesideFileGiven(t *testing.T) {
-	setupManifestInSubdir(t)
-
-	fs := afero.NewOsFs()
-	output := filepath.Join(t.TempDir(), "other.mf")
-
-	opts := testOpts([]string{
-		testApp, cmdGenerate, "-q", "-o", output, filepath.Join(testSubdir, "b.txt"), ".",
-	}, fs)
-	require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-	assert.Contains(t, manifestPaths(t, fs, output), testSubdirManifest)
-}
-
 // writeTestTree writes file.txt and sub/nested.txt under dir.
 func writeTestTree(t *testing.T, fs afero.Fs, dir string) {
 	t.Helper()

internal/cli/export.go

@@ -17,6 +17,7 @@ type ExportEntry struct {
 	Size   int64    `json:"size"`
 	Hashes []string `json:"hashes"`
 	Mtime  *string  `json:"mtime,omitempty"`
+	Ctime  *string  `json:"ctime,omitempty"`
 	Mode   string   `json:"mode"` // octal, "0000" when none was recorded
 }
 
@@ -60,6 +61,12 @@ func (mfa *CLIApp) exportManifestOperation(
 			entry.Mtime = &t
 		}
 
+		if f.GetCtime() != nil {
+			t := time.Unix(f.GetCtime().GetSeconds(), int64(f.GetCtime().GetNanos())).
+				UTC().Format(time.RFC3339Nano)
+			entry.Ctime = &t
+		}
+
 		entries = append(entries, entry)
 	}
 

internal/cli/export_test.go

@@ -4,10 +4,8 @@ import (
 	"bytes"
 	"context"
 	"encoding/json"
-	"maps"
 	"net/http"
 	"net/http/httptest"
-	"slices"
 	"testing"
 
 	"github.com/spf13/afero"
@@ -86,36 +84,6 @@ func TestExportManifestOperation(t *testing.T) {
 	assert.True(t, pathSet["sub/file.txt"])
 }
 
-// export prints the fields of a file entry and nothing else, so no ctime.
-func TestExportPrintsOnlyFileEntryFields(t *testing.T) {
-	t.Parallel()
-
-	manifestData := buildTestManifest(t, map[string][]byte{"entry.txt": []byte("aaa")})
-
-	fs := afero.NewMemMapFs()
-	require.NoError(t, afero.WriteFile(fs, "/test.mf", manifestData, 0o644))
-
-	var stdout, stderr bytes.Buffer
-
-	exitCode := runCLI(&RunOptions{
-		Appname: testApp,
-		Args:    []string{testApp, testCmdExport, "/test.mf"},
-		Stdin:   &bytes.Buffer{},
-		Stdout:  &stdout,
-		Stderr:  &stderr,
-		Fs:      fs,
-	})
-
-	require.Equal(t, 0, exitCode, "stderr: %s", stderr.String())
-
-	var entries []map[string]json.RawMessage
-	require.NoError(t, json.Unmarshal(stdout.Bytes(), &entries))
-	require.Len(t, entries, 1)
-	assert.Equal(t,
-		[]string{"hashes", "mode", "mtime", "path", "size"},
-		slices.Sorted(maps.Keys(entries[0])))
-}
-
 func TestExportFromHTTPURL(t *testing.T) {
 	t.Parallel()
 

internal/cli/fetch.go

@@ -95,6 +95,9 @@ var (
 	// writes another file.
 	errNameClash = errors.New(
 		"manifest lists a file where fetch writes another file")
+	// errModeOutOfRange indicates a manifest that lists a mode with more
+	// than the permission bits, such as setuid.
+	errModeOutOfRange = errors.New("manifest lists a mode outside 0777")
 )
 
 // DownloadProgress reports the progress of a single file download.
@@ -425,9 +428,9 @@ func (mfa *CLIApp) fetchManifestOperation(
 
 // fetchManifest downloads the manifest at manifestURL and parses it,
 // enforcing --require-signature if it is given and refusing a manifest
-// that lists a file where fetch writes another. It returns the manifest as
+// that lists a file where fetch writes another or a mode outside 0777. It
-// downloaded, to be saved once the files are in place, and the files it
+// returns the manifest as downloaded, to be saved once the files are in
-// lists.
+// place, and the files it lists.
 func (mfa *CLIApp) fetchManifest(
 	ctx context.Context, cmd *cli.Command, client retryingClient, manifestURL string,
 ) ([]byte, []*mfer.MFFilePath, error) {
@@ -484,6 +487,16 @@ func (mfa *CLIApp) fetchManifest(
 		return nil, nil, err
 	}
 
+	// fetch sets each recorded mode on the file it writes, so a mode above
+	// 0777, which could carry setuid, setgid or sticky bits, is refused
+	// before any file is requested.
+	for _, f := range files {
+		if f.GetMode() > uint32(os.ModePerm) {
+			return nil, nil, fmt.Errorf("%w: %s (%#o)",
+				errModeOutOfRange, f.GetPath(), f.GetMode())
+		}
+	}
+
 	log.Infof("manifest contains %d files", len(files))
 
 	return manifestData, files, nil

internal/cli/fetch_test.go

@@ -19,12 +19,15 @@ import (
 	"sync/atomic"
 	"testing"
 	"time"
+	"uuid"
 
+	"github.com/klauspost/compress/zstd"
 	"github.com/multiformats/go-multihash"
 	"github.com/spf13/afero"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 	urfcli "github.com/urfave/cli/v3"
+	"google.golang.org/protobuf/proto"
 	"sneak.berlin/go/mfer/mfer"
 )
 
@@ -1532,9 +1535,27 @@ func TestFetchSetsRecordedMode(t *testing.T) {
 		[]string{"/" + defaultManifestName, "/" + testFileTxt}, requested)
 }
 
+// TestFetchRefusesModeOutsidePermissionBits fetches manifests that record
+// a mode with the setuid bit, once as Unix writes it (04755) and once as
+// Go keeps it in os.FileMode. fetch must refuse both before it creates the
+// destination or requests any file.
+func TestFetchRefusesModeOutsidePermissionBits(t *testing.T) {
+	t.Parallel()
+
+	files := map[string][]byte{testFileTxt: []byte("a file")}
+
+	assertFetchRefused(t, manifestWithMode(t, testFileTxt, files[testFileTxt], 0o4755),
+		files, "manifest lists a mode outside 0777: file.txt (04755)")
+
+	assertFetchRefused(t,
+		manifestWithMode(t, testFileTxt, files[testFileTxt],
+			uint32(os.ModeSetuid|0o755)),
+		files, "manifest lists a mode outside 0777: file.txt (040000755)")
+}
+
 // TestDownloadFileSetsOnlyPermissionBits downloads a file whose entry
 // records mode 0755 together with setuid, setgid or sticky, as Go keeps
-// them in os.FileMode. Loading a manifest refuses such an entry; called
+// them in os.FileMode. fetchManifest would refuse such an entry; called
 // directly, downloadFile must still set only the permission bits.
 func TestDownloadFileSetsOnlyPermissionBits(t *testing.T) {
 	t.Parallel()
@@ -1571,6 +1592,51 @@ func TestDownloadFileSetsOnlyPermissionBits(t *testing.T) {
 	}
 }
 
+// manifestWithMode returns a manifest listing one file, path, with content
+// and the given recorded mode. It is assembled by hand, since the builder
+// never records a mode outside 0777.
+func manifestWithMode(t *testing.T, path string, content []byte, mode uint32) []byte {
+	t.Helper()
+
+	digest := sha256.Sum256(content)
+	hash, err := multihash.Encode(digest[:], multihash.SHA2_256)
+	require.NoError(t, err)
+
+	id := uuid.NewV4()
+
+	inner, err := proto.Marshal(&mfer.MFFile{
+		Version: mfer.MFFile_VERSION_ONE,
+		Files: []*mfer.MFFilePath{{
+			Path:   path,
+			Size:   int64(len(content)),
+			Hashes: []*mfer.MFFileChecksum{{MultiHash: hash}},
+			Mode:   mode,
+		}},
+		Uuid: id[:],
+	})
+	require.NoError(t, err)
+
+	encoder, err := zstd.NewWriter(nil)
+	require.NoError(t, err)
+
+	compressed := encoder.EncodeAll(inner, nil)
+	require.NoError(t, encoder.Close())
+
+	sum := sha256.Sum256(compressed)
+
+	outer, err := proto.Marshal(&mfer.MFFileOuter{
+		Version:         mfer.MFFileOuter_VERSION_ONE,
+		CompressionType: mfer.MFFileOuter_COMPRESSION_ZSTD,
+		Size:            int64(len(inner)),
+		Sha256:          sum[:],
+		Uuid:            id[:],
+		InnerMessage:    compressed,
+	})
+	require.NoError(t, err)
+
+	return append([]byte(mfer.MAGIC), outer...)
+}
+
 // builtManifest returns a manifest of files, built directly rather than
 // scanned, since a scan lists no hidden files.
 func builtManifest(t *testing.T, files map[string][]byte) []byte {

internal/cli/freshen.go

@@ -192,12 +192,6 @@ func (s *freshenScanner) walk(path string, info fs.FileInfo, walkErr error) erro
 		return nil
 	}
 
-	// Leave out a file listed as index.mf, at the top of the base
-	// directory, since fetch saves the manifest there, as gen does.
-	if relPath == defaultManifestName {
-		return nil
-	}
-
 	// Handle symlinks
 	if info.Mode()&fs.ModeSymlink != 0 {
 		realInfo, keep := s.resolveSymlink(path)

internal/cli/freshen_test.go

@@ -413,77 +413,6 @@ func TestFreshenLeavesLeftoverTempFileOutOfListing(t *testing.T) {
 		manifestPaths(t, fs, manifestPath))
 }
 
-// TestFreshenLeavesTopIndexMFOutOfListing freshens a manifest kept outside
-// the directory --base names, after index.mf and sub/index.mf are added
-// there: index.mf at the top is not listed, since fetch saves the manifest
-// there, while sub/index.mf is listed.
-func TestFreshenLeavesTopIndexMFOutOfListing(t *testing.T) {
-	t.Parallel()
-
-	root := t.TempDir()
-	tree := filepath.Join(root, "tree")
-	manifestPath := filepath.Join(root, "other.mf")
-
-	fs := afero.NewOsFs()
-	require.NoError(t, fs.MkdirAll(filepath.Join(tree, testSubdir), 0o750))
-	writeTestFile(t, fs, filepath.Join(tree, testFileTxt), "hello")
-
-	opts := testOpts([]string{testApp, cmdGenerate, "-q", "-o", manifestPath, tree}, fs)
-	require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-	writeTestFile(t, fs, filepath.Join(tree, defaultManifestName), "an ordinary file")
-	writeTestFile(t, fs, filepath.Join(tree, testSubdirManifest), "an ordinary file")
-
-	runFreshen(t, fs, tree, manifestPath)
-
-	assert.ElementsMatch(t, []string{testFileTxt, testSubdirManifest},
-		manifestPaths(t, fs, manifestPath))
-}
-
-// TestFreshenListsTargetOfTopIndexMFSymlink freshens a manifest kept
-// outside the directory --base names, after sub/index.mf is added there
-// and index.mf at the top is made a symlink to it: sub/index.mf is listed,
-// and the symlink, which freshen lists only with --follow-symlinks, is left
-// out like any index.mf at the top.
-func TestFreshenListsTargetOfTopIndexMFSymlink(t *testing.T) {
-	t.Parallel()
-
-	for name, flags := range map[string][]string{
-		"plain freshen":             nil,
-		"freshen --follow-symlinks": {testFlagFollowSymlinks},
-	} {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			root := t.TempDir()
-			tree := filepath.Join(root, "tree")
-			manifestPath := filepath.Join(root, "other.mf")
-
-			fs := afero.NewOsFs()
-			require.NoError(t, fs.MkdirAll(filepath.Join(tree, testSubdir), 0o750))
-			writeTestFile(t, fs, filepath.Join(tree, testFileTxt), "hello")
-
-			opts := testOpts([]string{
-				testApp, cmdGenerate, "-q", "-o", manifestPath, tree,
-			}, fs)
-			require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-			writeTestFile(t, fs, filepath.Join(tree, testSubdirManifest), "an ordinary file")
-			require.NoError(t, os.Symlink(testSubdirManifest,
-				filepath.Join(tree, defaultManifestName)))
-
-			opts = testOpts(slices.Concat(
-				[]string{testApp, cmdFreshen, "-q"}, flags,
-				[]string{testFlagBase, tree, manifestPath},
-			), fs)
-			require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-			assert.ElementsMatch(t, []string{testFileTxt, testSubdirManifest},
-				manifestPaths(t, fs, manifestPath))
-		})
-	}
-}
-
 // TestFreshenResolvesEntriesAgainstManifestDirectory adds sub/c.txt, then
 // runs freshen from the directory above sub, on the manifest in sub.
 // Without --base, the manifest then lists the files in sub, whether freshen

internal/cli/gen.go

@@ -137,9 +137,6 @@ func (mfa *CLIApp) buildScannerOptions(
 		// Neither a manifest being replaced nor a temp file left by an
 		// interrupted run belongs in the new manifest.
 		ExcludePaths: []string{output, manifestTempPath(output)},
-		// Nor does a file listed as index.mf, at the top of the tree, since
-		// fetch saves the manifest there and refuses a manifest that lists it.
-		ExcludeRelPaths: []mfer.RelFilePath{defaultManifestName},
 	}
 
 	// Set seed for deterministic UUID if provided

internal/cli/pipe_test.go

@@ -81,7 +81,7 @@ func TestFreshenSkipsPipes(t *testing.T) {
 	require.NoError(t, os.Symlink(target, filepath.Join(root, testLink)))
 
 	opts := testOpts([]string{
-		testApp, cmdFreshen, "-v", testFlagFollowSymlinks,
+		testApp, cmdFreshen, "-v", "--follow-symlinks",
 		testFlagBase, root, manifestPath,
 	}, fs)
 	require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))

mfer/constants.go

@@ -23,20 +23,23 @@ const (
 	// that decoding sets aside a fixed amount of memory for.
 	filesFieldNumber    = 101
 	hashesFieldNumber   = 3
+	mimeTypeFieldNumber = 301
 	mtimeFieldNumber    = 302
+	ctimeFieldNumber    = 303
 
-	// Bytes decoding sets aside for each file entry, hash and timestamp,
+	// Bytes decoding sets aside for each file entry, hash, timestamp and
-	// however short its encoding. checkDecodedSize refuses an inner
+	// MIME type, however short its encoding. checkDecodedSize refuses an
-	// message for which these add up to more than maxDecodedGrowth times
+	// inner message for which these add up to more than maxDecodedGrowth
-	// its size. The mode is held in the file entry itself.
+	// times its size. The mode is held in the file entry itself.
-	decodedFileEntrySize = 160
+	decodedFileEntrySize = 176
 	decodedHashSize      = 112
 	decodedTimestampSize = 64
+	decodedMIMETypeSize  = 16
 
 	// Each file entry mfer writes holds a path of at least one byte, a
 	// multihash at least as long as SHA-256's 34 bytes (AddFileWithHash
 	// refuses shorter ones) and a modification time: at least 47 bytes,
-	// counted at 336. So its manifests add up to at most about 7.15 times
+	// counted at 352. So its manifests add up to at most about 7.49 times
-	// their size, and this limit is about 12% above that.
+	// their size, and this limit is about 7% above that.
 	maxDecodedGrowth = 8
 )

mfer/deserialize.go

@@ -6,7 +6,6 @@ import (
 	"errors"
 	"fmt"
 	"io"
-	"os"
 	"strings"
 
 	"github.com/klauspost/compress/zstd"
@@ -30,7 +29,6 @@ var (
 	errInvalidManifestPath  = errors.New("invalid file entry")
 	errEntryMissingHash     = errors.New("file entry has no SHA-256 hash")
 	errEntryExtraSHA256Hash = errors.New("file entry has more than one SHA-256 hash")
-	errModeOutOfRange       = errors.New("file entry has a mode above 0777")
 	errDecodedTooLarge      = errors.New("too much memory needed")
 	errSignerNotSigningKey  = errors.New(
 		"signer is not the fingerprint of the key that made the signature")
@@ -161,10 +159,10 @@ func (m *manifest) decompressInner() ([]byte, error) {
 }
 
 // checkDecodedSize refuses an encoded inner message whose file entries,
-// hashes and timestamps would take more than maxDecodedGrowth times its
+// hashes, timestamps and MIME types would take more than maxDecodedGrowth
-// size to decode. Decoding sets aside a fixed amount for each, however
+// times its size to decode. Decoding sets aside a fixed amount for each,
-// short its encoding, so a message of empty ones would take about 50 times
+// however short its encoding, so a message of empty ones would take about
-// its size.
+// 50 times its size.
 func checkDecodedSize(inner []byte) error {
 	limit := maxDecodedGrowth * int64(len(inner))
 
@@ -194,10 +192,14 @@ func checkDecodedSize(inner []byte) error {
 				return add(decodedHashSize)
 			}
 
-			if num == mtimeFieldNumber {
+			if num == mtimeFieldNumber || num == ctimeFieldNumber {
 				return add(decodedTimestampSize)
 			}
 
+			if num == mimeTypeFieldNumber {
+				return add(decodedMIMETypeSize)
+			}
+
 			return nil
 		})
 	})
@@ -318,9 +320,7 @@ func (m *manifest) deserializeInner() error {
 	// dropping entries, which would let a hostile manifest hide files from a
 	// check. A path listed twice is refused too: check would check the one
 	// file against both entries. So is an entry that does not list exactly
-	// one SHA-256 hash, and one that records a mode above 0777, such as one
+	// one SHA-256 hash.
-	// with setuid: a mode holds the permission bits alone, which are all
-	// check compares and fetch sets.
 	seen := make(map[string]bool, len(m.pbInner.GetFiles()))
 
 	for _, f := range m.pbInner.GetFiles() {
@@ -334,10 +334,6 @@ func (m *manifest) deserializeInner() error {
 			return err
 		}
 
-		if f.GetMode() > uint32(os.ModePerm) {
-			return fmt.Errorf("%w: %q (%#o)", errModeOutOfRange, f.GetPath(), f.GetMode())
-		}
-
 		if seen[f.GetPath()] {
 			return fmt.Errorf("%w %q", errDuplicatePath, f.GetPath())
 		}

mfer/deserialize_fuzz_test.go

@@ -48,8 +48,8 @@ func FuzzNewManifestFromReader(f *testing.F) {
 		// It also keeps a few copies of its input. Buffers grow by
 		// copying, so reaching those sizes allocates up to about six times
 		// them in total. Decoding the decompressed data takes up to
-		// maxDecodedGrowth times its size for file entries, hashes and
+		// maxDecodedGrowth times its size for file entries, hashes,
-		// timestamps, and drops fields it does not know.
+		// timestamps and MIME types, and drops fields it does not know.
 		// The strings and bytes it copies out of it, such as many one-byte
 		// values in one hash, take up to about five times more under the
 		// race detector, which pads every small copy to 16 bytes, and about

mfer/deserialize_path_test.go

@@ -7,7 +7,6 @@ import (
 	"crypto/sha256"
 	"crypto/sha512"
 	"fmt"
-	"os"
 	"strconv"
 	"strings"
 	"testing"
@@ -235,77 +234,14 @@ func TestDeserializeRequiresExactlyOneSHA256Hash(t *testing.T) {
 	}
 }
 
-// A file entry that records a mode with more than the permission bits is
+// Entries of a path, a SHA-256 hash, an empty MIME type and empty
-// refused as it is loaded, naming the path and the mode: once with setuid as
+// modification and change times are counted at 432 bytes each (176 + 112 +
-// Unix writes it (04755) and once as Go keeps it in os.FileMode. An entry
+// 16 + 64 + 64) and take 52 bytes plus the path to encode. A one-character
-// recording 0777 loads.
+// path makes that 53 bytes, about 8.2 times: refused, and leaving any one of
-func TestDeserializeRefusesModeOutsidePermissionBits(t *testing.T) {
+// the five uncounted, even the MIME type, brings it under 8. A two-character
-	t.Parallel()
+// path makes it 54 bytes, 8 times exactly: loaded. There is one entry for
-
+// each base-36 digit, and each entry's path is its number in base 36, padded
-	hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
+// with zeros to that length, since a manifest lists a path only once.
-	require.NoError(t, err)
-
-	tests := []struct {
-		name string
-		mode uint32
-		want string // "" for an entry that loads
-	}{
-		{
-			"setuid as Unix writes it",
-			0o4755,
-			`file entry has a mode above 0777: "x" (04755)`,
-		},
-		{
-			"setuid as Go keeps it",
-			uint32(os.ModeSetuid | 0o755),
-			`file entry has a mode above 0777: "x" (040000755)`,
-		},
-		{"permission bits only", 0o777, ""},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			t.Parallel()
-
-			id := uuid.NewV4()
-			inner, err := proto.Marshal(&MFFile{
-				Version: MFFile_VERSION_ONE,
-				Files: []*MFFilePath{{
-					Path:   "x",
-					Size:   1,
-					Hashes: []*MFFileChecksum{{MultiHash: hash}},
-					Mode:   tt.mode,
-				}},
-				Uuid: id[:],
-			})
-			require.NoError(t, err)
-
-			m, err := NewManifestFromReader(bytes.NewReader(wrapInner(t, id, inner)))
-			if tt.want == "" {
-				require.NoError(t, err)
-				require.Len(t, m.Files(), 1)
-				assert.Equal(t, tt.mode, m.Files()[0].GetMode())
-
-				return
-			}
-
-			require.ErrorIs(t, err, errModeOutOfRange)
-			require.EqualError(t, err, tt.want)
-		})
-	}
-}
-
-// Entries of a path, a SHA-256 hash, an empty second hash and an empty
-// modification time are counted at 448 bytes each (160 + 112 + 112 + 64)
-// and take 48 bytes plus the path to encode. A 7-character path makes that
-// 55 bytes, about 8.1 times: refused, and leaving any one of the four
-// uncounted, even the modification time, brings it under 8. An 8-character
-// path makes it 56 bytes, 8 times exactly: loaded. Without the second hash
-// an entry is never refused: it takes at least 47 bytes and is counted at
-// 336. There is one entry for each base-36 digit, enough that the 22 bytes
-// of the version and UUID change neither outcome, and each entry's path is
-// its number in base 36, padded with zeros to that length, since a manifest
-// lists a path only once.
 func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
 	t.Parallel()
 
@@ -319,8 +255,8 @@ func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
 		pathLen int
 		refused bool
 	}{
-		{7, true},
+		{1, true},
-		{8, false},
+		{2, false},
 	}
 
 	for _, tt := range tests {
@@ -341,10 +277,12 @@ func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
 				entry = protowire.AppendString(entry, path)
 				entry = protowire.AppendTag(entry, 3, protowire.BytesType) // MFFilePath.hashes
 				entry = protowire.AppendBytes(entry, checksum)
-				entry = protowire.AppendTag(entry, 3, protowire.BytesType) // MFFilePath.hashes
+				entry = protowire.AppendTag(entry, 301, protowire.BytesType) // MFFilePath.mimeType
 				entry = protowire.AppendBytes(entry, nil)
 				entry = protowire.AppendTag(entry, 302, protowire.BytesType) // MFFilePath.mtime
 				entry = protowire.AppendBytes(entry, nil)
+				entry = protowire.AppendTag(entry, 303, protowire.BytesType) // MFFilePath.ctime
+				entry = protowire.AppendBytes(entry, nil)
 
 				inner = protowire.AppendTag(inner, 101, protowire.BytesType) // MFFile.files
 				inner = protowire.AppendBytes(inner, entry)
@@ -400,44 +338,6 @@ func TestDeserializeDropsUnknownFields(t *testing.T) {
 	assert.Empty(t, m.Files()[0].ProtoReflect().GetUnknown())
 }
 
-// A file entry holding fields 301 and 303, which mf.proto reserves, loads,
-// and the loaded entry keeps neither.
-func TestDeserializeDropsReservedFields(t *testing.T) {
-	t.Parallel()
-
-	hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
-	require.NoError(t, err)
-
-	checksum, err := proto.Marshal(&MFFileChecksum{MultiHash: hash})
-	require.NoError(t, err)
-
-	timestamp, err := proto.Marshal(&Timestamp{Seconds: 1})
-	require.NoError(t, err)
-
-	entry := protowire.AppendTag(nil, 1, protowire.BytesType) // MFFilePath.path
-	entry = protowire.AppendString(entry, "a")
-	entry = protowire.AppendTag(entry, 3, protowire.BytesType) // MFFilePath.hashes
-	entry = protowire.AppendBytes(entry, checksum)
-	entry = protowire.AppendTag(entry, 301, protowire.BytesType) // reserved
-	entry = protowire.AppendString(entry, "text/plain")
-	entry = protowire.AppendTag(entry, 303, protowire.BytesType) // reserved
-	entry = protowire.AppendBytes(entry, timestamp)
-
-	id := uuid.NewV4()
-	inner := protowire.AppendTag(nil, 100, protowire.VarintType) // MFFile.version
-	inner = protowire.AppendVarint(inner, uint64(MFFile_VERSION_ONE))
-	inner = protowire.AppendTag(inner, 101, protowire.BytesType) // MFFile.files
-	inner = protowire.AppendBytes(inner, entry)
-	inner = protowire.AppendTag(inner, 102, protowire.BytesType) // MFFile.uuid
-	inner = protowire.AppendBytes(inner, id[:])
-
-	m, err := NewManifestFromReader(bytes.NewReader(wrapInner(t, id, inner)))
-	require.NoError(t, err)
-	require.Len(t, m.Files(), 1)
-	assert.Equal(t, "a", m.Files()[0].GetPath())
-	assert.Empty(t, m.Files()[0].ProtoReflect().GetUnknown())
-}
-
 // Many empty files with names of at most three characters and modification
 // times at the epoch make about the densest manifest mfer writes: it takes
 // about 7 times its size to decode, and still loads. A signature would not

mfer/mf.pb.go

@@ -340,7 +340,9 @@ type MFFilePath struct {
 	// gotta have at least one:
 	Hashes []*MFFileChecksum `protobuf:"bytes,3,rep,name=hashes,proto3" json:"hashes,omitempty"`
 	// optional per-file metadata
+	MimeType *string    `protobuf:"bytes,301,opt,name=mimeType,proto3,oneof" json:"mimeType,omitempty"`
 	Mtime    *Timestamp `protobuf:"bytes,302,opt,name=mtime,proto3,oneof" json:"mtime,omitempty"`
+	Ctime    *Timestamp `protobuf:"bytes,303,opt,name=ctime,proto3,oneof" json:"ctime,omitempty"`
 	// permission bits, at most 0777; 0 when not recorded
 	Mode          uint32 `protobuf:"varint,304,opt,name=mode,proto3" json:"mode,omitempty"`
 	unknownFields protoimpl.UnknownFields
@@ -398,6 +400,13 @@ func (x *MFFilePath) GetHashes() []*MFFileChecksum {
 	return nil
 }
 
+func (x *MFFilePath) GetMimeType() string {
+	if x != nil && x.MimeType != nil {
+		return *x.MimeType
+	}
+	return ""
+}
+
 func (x *MFFilePath) GetMtime() *Timestamp {
 	if x != nil {
 		return x.Mtime
@@ -405,6 +414,13 @@ func (x *MFFilePath) GetMtime() *Timestamp {
 	return nil
 }
 
+func (x *MFFilePath) GetCtime() *Timestamp {
+	if x != nil {
+		return x.Ctime
+	}
+	return nil
+}
+
 func (x *MFFilePath) GetMode() uint32 {
 	if x != nil {
 		return x.Mode
@@ -557,16 +573,21 @@ const file_mf_proto_rawDesc = "" +
 	"\n" +
 	"_signatureB\t\n" +
 	"\a_signerB\x10\n" +
-	"\x0e_signingPubKey\"\xc5\x01\n" +
+	"\x0e_signingPubKey\"\x85\x02\n" +
 	"\n" +
 	"MFFilePath\x12\x12\n" +
 	"\x04path\x18\x01 \x01(\tR\x04path\x12\x12\n" +
 	"\x04size\x18\x02 \x01(\x03R\x04size\x12'\n" +
-	"\x06hashes\x18\x03 \x03(\v2\x0f.MFFileChecksumR\x06hashes\x12&\n" +
+	"\x06hashes\x18\x03 \x03(\v2\x0f.MFFileChecksumR\x06hashes\x12 \n" +
+	"\bmimeType\x18\xad\x02 \x01(\tH\x00R\bmimeType\x88\x01\x01\x12&\n" +
 	"\x05mtime\x18\xae\x02 \x01(\v2\n" +
-	".TimestampH\x00R\x05mtime\x88\x01\x01\x12\x13\n" +
+	".TimestampH\x01R\x05mtime\x88\x01\x01\x12&\n" +
-	"\x04mode\x18\xb0\x02 \x01(\rR\x04modeB\b\n" +
+	"\x05ctime\x18\xaf\x02 \x01(\v2\n" +
-	"\x06_mtimeJ\x06\b\xad\x02\x10\xae\x02J\x06\b\xaf\x02\x10\xb0\x02R\bmimeTypeR\x05ctime\".\n" +
+	".TimestampH\x02R\x05ctime\x88\x01\x01\x12\x13\n" +
+	"\x04mode\x18\xb0\x02 \x01(\rR\x04modeB\v\n" +
+	"\t_mimeTypeB\b\n" +
+	"\x06_mtimeB\b\n" +
+	"\x06_ctime\".\n" +
 	"\x0eMFFileChecksum\x12\x1c\n" +
 	"\tmultiHash\x18\x01 \x01(\fR\tmultiHash\"\xd6\x01\n" +
 	"\x06MFFile\x12)\n" +
@@ -610,14 +631,15 @@ var file_mf_proto_depIdxs = []int32{
 	1, // 1: MFFileOuter.compressionType:type_name -> MFFileOuter.CompressionType
 	6, // 2: MFFilePath.hashes:type_name -> MFFileChecksum
 	3, // 3: MFFilePath.mtime:type_name -> Timestamp
-	2, // 4: MFFile.version:type_name -> MFFile.Version
+	3, // 4: MFFilePath.ctime:type_name -> Timestamp
-	5, // 5: MFFile.files:type_name -> MFFilePath
+	2, // 5: MFFile.version:type_name -> MFFile.Version
-	3, // 6: MFFile.createdAt:type_name -> Timestamp
+	5, // 6: MFFile.files:type_name -> MFFilePath
-	7, // [7:7] is the sub-list for method output_type
+	3, // 7: MFFile.createdAt:type_name -> Timestamp
-	7, // [7:7] is the sub-list for method input_type
+	8, // [8:8] is the sub-list for method output_type
-	7, // [7:7] is the sub-list for extension type_name
+	8, // [8:8] is the sub-list for method input_type
-	7, // [7:7] is the sub-list for extension extendee
+	8, // [8:8] is the sub-list for extension type_name
-	0, // [0:7] is the sub-list for field type_name
+	8, // [8:8] is the sub-list for extension extendee
+	0, // [0:8] is the sub-list for field type_name
 }
 
 func init() { file_mf_proto_init() }

mfer/mf.proto

@@ -58,13 +58,11 @@ message MFFilePath {
     repeated MFFileChecksum hashes = 3;
 
     // optional per-file metadata
+    optional string mimeType = 301;
     optional Timestamp mtime = 302;
+    optional Timestamp ctime = 303;
     // permission bits, at most 0777; 0 when not recorded
     uint32 mode = 304;
-
-    // used by no field, so that a later version cannot reuse them by accident
-    reserved 301, 303;
-    reserved "mimeType", "ctime";
 }
 
 message MFFileChecksum {

mfer/mf.proto.sha256

@@ -1 +1 @@
-16a2e6b39986d02aa1c7c01dc6b97c9d4c30db7a9d000056bd5985a3fa7c8089  mf.proto
+1668b2f6b8639a9213873474e58dbacb3a4ef77cc27c42253cad8dd076e5f206  mf.proto

mfer/mf_test.go

@@ -7,7 +7,6 @@ import (
 	"strings"
 	"testing"
 
-	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 	"google.golang.org/protobuf/reflect/protoreflect"
 	"sneak.berlin/go/mfer/mfer"
@@ -39,7 +38,9 @@ func TestFileEntryFieldsMatchSpec(t *testing.T) {
 		"path":     1,
 		"size":     2,
 		"hashes":   3,
+		"mimeType": 301,
 		"mtime":    302,
+		"ctime":    303,
 		"mode":     304,
 	}
 
@@ -52,16 +53,3 @@ func TestFileEntryFieldsMatchSpec(t *testing.T) {
 
 	require.Equal(t, want, got)
 }
-
-// A file entry reserves the numbers 301 and 303 and the names mimeType and
-// ctime, as docs/FORMAT.md says, so that no field can take them.
-func TestFileEntryReservesMIMETypeAndCtime(t *testing.T) {
-	t.Parallel()
-
-	desc := (&mfer.MFFilePath{}).ProtoReflect().Descriptor()
-
-	assert.True(t, desc.ReservedRanges().Has(301))
-	assert.True(t, desc.ReservedRanges().Has(303))
-	assert.True(t, desc.ReservedNames().Has("mimeType"))
-	assert.True(t, desc.ReservedNames().Has("ctime"))
-}

mfer/scanner.go

@@ -8,7 +8,6 @@ import (
 	"os"
 	"path"
 	"path/filepath"
-	"slices"
 	"strings"
 	"sync"
 	"time"
@@ -65,9 +64,6 @@ type ScannerOptions struct {
 	Seed string
 	// ExcludePaths lists files to leave out of the listing, matched with os.SameFile.
 	ExcludePaths []string
-	// ExcludeRelPaths lists paths to leave out of the listing, matched
-	// against the path a file would be listed by, with no file looked up.
-	ExcludeRelPaths []RelFilePath
 }
 
 // FileEntry represents a file that has been enumerated.
@@ -76,6 +72,7 @@ type FileEntry struct {
 	AbsPath AbsFilePath // Absolute path (used for reading file content)
 	Size    FileSize    // File size in bytes
 	Mtime   ModTime     // Last modification time
+	Ctime   time.Time   // Creation time (platform-dependent)
 	Mode    fs.FileMode // Permission bits (Perm() of the file's mode)
 }
 
@@ -442,10 +439,9 @@ func (s *Scanner) enumerateFileWithInfo(
 	}
 
 	// Clean the path - remove leading slash if present
-	cleanPath := strings.TrimPrefix(filePath, "/")
+	cleanPath := filePath
-
+	if len(cleanPath) > 0 && cleanPath[0] == '/' {
-	if slices.Contains(s.options.ExcludeRelPaths, RelFilePath(cleanPath)) {
+		cleanPath = cleanPath[1:]
-		return nil
 	}
 
 	// Compute absolute path for file reading
@@ -497,6 +493,8 @@ func (s *Scanner) enumerateFileWithInfo(
 		Size:    FileSize(info.Size()),
 		Mtime:   ModTime(info.ModTime()),
 		Mode:    info.Mode().Perm(),
+		// Note: Ctime not available from fs.FileInfo on all platforms
+		// Will need platform-specific code to extract it
 	}
 
 	s.mu.Lock()