Never report index.mf at the top of the base as unlisted in check (closes #229)

gen and fetch write the manifest as index.mf at the top of the tree.
check left out only the manifest file it was given, so given a URL, or a
manifest stored elsewhere, it reported that index.mf as not in the
manifest, and --no-extra-files failed a tree fetch had just verified.
check now drops that one report for index.mf at the top of the base
directory. A manifest that lists index.mf still has it checked, and
index.mf in a subdirectory or fetch's leftover .index.mf.tmp are still
reported. The change is in package cli, which owns the default manifest
name; the Checker in package mfer is unchanged.

Model: opus-5-5

README.md

@@ -196,7 +196,8 @@ The manifest file would do several important things:
   a signature and a public key alongside in a single file
 - contain a list of files, each with a relative path to the manifest
 - contain manifest timestamp
-- contain mtime information for files so that file metadata can be preserved
+- contain ctime/mtime information for files so that file metadata can be
+  preserved
 - contain cryptographic checksums in several different algorithms for each file
     - probably encoded with multihash to indicate algo + hash
     - sha256 at the minimum
@@ -286,9 +287,6 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
       only on the paths it is given and the files under them, so with the same
       `--seed` and an unchanged tree it writes the same bytes wherever the
       manifest goes. The file it writes to is never listed
-    - never lists `index.mf` at the top of a directory it is given, or a file it
-      is given by that name, wherever it writes the manifest, since `fetch`
-      saves the manifest there; `index.mf` in a subdirectory is listed
 - `mfer check` / `mfer check .`
     - verifies checksums of all files in manifest, displaying error and exiting
       nonzero if any files are missing or corrupted, or have permission bits
@@ -310,8 +308,7 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
     - rewrites `index.mf` to list the files now under the directory holding it,
       or under the one `--base` names, hashing only the files that are new or
       changed
-    - lists regular files only, and never `index.mf` at the top of that
+    - lists regular files only, as gen does
-      directory, as gen does
     - leaves out hidden files unless given `--include-dotfiles`, and symlinks
       unless given `--follow-symlinks`, which lists each symlink to a regular
       file under its own name with the contents of the file it points to
@@ -368,8 +365,8 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
     - (fetches `https://example.com/manifestdirectory/index.mf`, downloads and
       checksums all files, resumes any that exist locally already)
 - `mfer fetch https://example.com/manifestdirectory/`
-- a command line option to zero/omit mtime, as well as manifest timestamp, and
+- a command line option to zero/omit mtime/ctime, as well as manifest timestamp,
-  sort all directory listings so that manifest file generation is
+  and sort all directory listings so that manifest file generation is
   deterministic/reproducible
 - URL format
   `mfer fetch https://exmaple.com/manifestdirectory/?key=5539AD00DE4C42F3AFE11575052443F4DF2A55C2`

docs/FORMAT.md

@@ -9,11 +9,8 @@ including their paths, sizes, and cryptographic checksums. It supports optional
 OpenPGP signatures for integrity verification and optional timestamps and file
 permissions for metadata preservation.
 
-Nothing goes in the 1.0 manifest that 1.0 does not read or write, and no field
+Nothing goes in the 1.0 manifest that 1.0 does not read or write: no field is
-is kept for later use. [`mfer/mf.proto`](../mfer/mf.proto) reserves the field
+reserved or kept for later use.
-numbers 301 and 303 and the names `mimeType` and `ctime` in `MFFilePath`, so
-that a later version cannot reuse them by accident. No field uses them, and
-readers ignore them like any field they do not know.
 
 ## File Structure
 
@@ -61,11 +58,11 @@ enforce a decompression size limit to prevent decompression bombs. The reference
 implementation limits decompressed size to 256 MiB. It writes zstd frames with a
 window of at most 8 MiB, the largest window the zstd format recommends decoders
 support, and refuses frames that ask for a larger one. It also refuses an inner
-message whose file entries, hashes and timestamps, counted at 160, 112 and 64
+message whose file entries, hashes, timestamps and MIME types, counted at 176,
-bytes each, add up to more than 8 times its size. It refuses a manifest file
+112, 64 and 16 bytes each, add up to more than 8 times its size. It refuses a
-larger than 258 MiB without reading the rest of it: zstd's worst case grows a
+manifest file larger than 258 MiB without reading the rest of it: zstd's worst
-256 MiB inner message by 1/256 to 257 MiB, and the last MiB is room for the
+case grows a 256 MiB inner message by 1/256 to 257 MiB, and the last MiB is room
-signature, the signing key and the other outer fields.
+for the signature, the signing key and the other outer fields.
 
 ## Inner Message (`MFFile`)
 
@@ -84,11 +81,13 @@ After decompressing `innerMessage`, the result is a serialized `MFFile`
 Each file entry contains:
 
 | Field      | Number | Type                      | Description                                                  |
-| -------- | ------ | ------------------------- | ------------------------------------------------------------ |
+| ---------- | ------ | ------------------------- | ------------------------------------------------------------ |
 | `path`     | 1      | string                    | Relative file path (see Path Rules)                          |
 | `size`     | 2      | int64                     | File size in bytes                                           |
 | `hashes`   | 3      | repeated `MFFileChecksum` | Must include exactly one SHA-256 multihash (see Hash Format) |
+| `mimeType` | 301    | string (optional)         | MIME type                                                    |
 | `mtime`    | 302    | Timestamp (optional)      | Modification time                                            |
+| `ctime`    | 303    | Timestamp (optional)      | Change time (inode metadata change)                          |
 | `mode`     | 304    | uint32                    | Permission bits (see File Mode)                              |
 
 ## File Mode

internal/cli/entry_test.go

@@ -36,7 +36,6 @@ const (
 	testManifest    = "/manifest.mf"
 	testFlagBase    = "--base"
 	testFlagNoExtra = "--no-extra-files"
-	testFlagFollowSymlinks = "--follow-symlinks"
 	testFlagVersion = "--version"
 	testFlagVerbose = "--verbose"
 	testFlagHelp    = "--help"
@@ -1003,7 +1002,7 @@ func TestCheckReportsUnlistedSymlinkToFileOnly(t *testing.T) {
 		exitCode int
 		extras   []string // what check reports as not in the manifest
 	}{
-		"gen --follow-symlinks": {[]string{testFlagFollowSymlinks}, 0, nil},
+		"gen --follow-symlinks": {[]string{"--follow-symlinks"}, 0, nil},
 		"plain gen":             {nil, 1, []string{"filelink"}},
 	} {
 		t.Run(name, func(t *testing.T) {
@@ -1191,13 +1190,13 @@ func TestCheckURLManifestWithoutTempDirectory(t *testing.T) {
 	assert.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
 }
 
-// TestCheckNeverReportsIndexMFAtTopOfBase checks a tree holding index.mf at
+// TestCheckNeverReportsIndexMfAtTopOfBase checks a tree holding index.mf at
 // its top, where gen and fetch write the manifest, against a manifest that
 // does not list it, given by URL and by a path outside the tree: check never
 // reports it, and passes with and without --no-extra-files. index.mf in a
 // subdirectory, and the temp file fetch saves index.mf through, are still
 // reported.
-func TestCheckNeverReportsIndexMFAtTopOfBase(t *testing.T) {
+func TestCheckNeverReportsIndexMfAtTopOfBase(t *testing.T) {
 	t.Parallel()
 
 	files := map[string][]byte{testFileTxt: []byte("hello")}
@@ -1240,7 +1239,7 @@ func TestCheckNeverReportsIndexMFAtTopOfBase(t *testing.T) {
 			assert.NotContains(t, stderr, "EXTRA: "+defaultManifestName)
 
 			reported := []string{
-				testSubdirManifest,
+				testSubdir + "/" + defaultManifestName,
 				tempPathFor(defaultManifestName),
 			}
 
@@ -1262,10 +1261,10 @@ func TestCheckNeverReportsIndexMFAtTopOfBase(t *testing.T) {
 	}
 }
 
-// TestCheckChecksListedIndexMF checks a tree against a manifest that lists
+// TestCheckChecksListedIndexMf checks a tree against a manifest that lists
 // index.mf at the top of the tree, where that file has changed since: check
 // reports it like any other changed file.
-func TestCheckChecksListedIndexMF(t *testing.T) {
+func TestCheckChecksListedIndexMf(t *testing.T) {
 	t.Parallel()
 
 	manifestPath := filepath.Join(t.TempDir(), "other.mf")
@@ -1332,11 +1331,11 @@ func TestCheckWithUnlistableDirectory(t *testing.T) {
 	assert.Contains(t, testStderr(t, opts), "unlisted.txt")
 }
 
-// TestCheckReportsUnreadableIndexMFAtTopOfBase has a directory named
+// TestCheckReportsUnreadableIndexMfAtTopOfBase has a directory named
 // index.mf at the top of the base that cannot be listed. check leaves out
 // only the report that index.mf is not in the manifest, so it reports the
 // error and fails under --no-extra-files.
-func TestCheckReportsUnreadableIndexMFAtTopOfBase(t *testing.T) {
+func TestCheckReportsUnreadableIndexMfAtTopOfBase(t *testing.T) {
 	t.Parallel()
 
 	unreadable := filepath.Join(testDir, defaultManifestName)
@@ -1359,33 +1358,6 @@ func TestCheckReportsUnreadableIndexMFAtTopOfBase(t *testing.T) {
 	assert.Contains(t, testStderr(t, opts), "ERROR: "+defaultManifestName)
 }
 
-// TestCheckReportsUpperCaseIndexMFAtTopOfBase checks a tree holding
-// INDEX.MF at its top against a manifest that does not list it. Only the
-// name index.mf, in exactly that letter case, is left out, so check reports
-// INDEX.MF. The tree is in memory, where the two names are two files
-// whatever the letter case rules of the host's filesystem.
-func TestCheckReportsUpperCaseIndexMFAtTopOfBase(t *testing.T) {
-	t.Parallel()
-
-	upperCase := strings.ToUpper(defaultManifestName)
-
-	fs := afero.NewMemMapFs()
-	writeTestFile(t, fs, testFile1, "hello")
-
-	opts := testOpts([]string{
-		testApp, cmdGenerate, "-q", "-o", testManifest, testDir,
-	}, fs)
-	require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-	writeTestFile(t, fs, filepath.Join(testDir, upperCase), "not listed")
-
-	opts = testOpts([]string{
-		testApp, cmdCheck, testFlagNoExtra, testFlagBase, testDir, testManifest,
-	}, fs)
-	assert.Equal(t, 1, runCLI(opts), "stderr: %s", testStderr(t, opts))
-	assert.Contains(t, testStderr(t, opts), "EXTRA: "+upperCase)
-}
-
 func TestGenerateAtomicWriteNoTempFileOnSuccess(t *testing.T) {
 	t.Parallel()
 
@@ -1568,141 +1540,6 @@ func TestGenerateLeavesLeftoverTempFileOutOfListing(t *testing.T) {
 	assert.Equal(t, []string{testFileTxt}, manifestPaths(t, fs, output))
 }
 
-// TestGenerateLeavesTopIndexMFOutOfListing runs gen with --output outside
-// the paths it is given: index.mf at the top of each directory given, and
-// a file given by that name, are not listed, since fetch saves the manifest
-// as index.mf at the top of the tree, while sub/index.mf is listed.
-func TestGenerateLeavesTopIndexMFOutOfListing(t *testing.T) {
-	t.Parallel()
-
-	// Paths are relative to a temp dir holding tree, which holds file.txt,
-	// index.mf and sub/index.mf, and second, which holds second.txt and
-	// index.mf.
-	for name, tc := range map[string]struct {
-		paths  []string
-		listed []string
-	}{
-		"directory": {
-			[]string{"tree"},
-			[]string{testFileTxt, testSubdirManifest},
-		},
-		"two directories": {
-			[]string{"tree", "second"},
-			[]string{testFileTxt, testSubdirManifest, "second.txt"},
-		},
-		"files": {
-			[]string{"tree/" + testFileTxt, "tree/" + defaultManifestName},
-			[]string{testFileTxt},
-		},
-	} {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			root := t.TempDir()
-			tree := filepath.Join(root, "tree")
-			second := filepath.Join(root, "second")
-
-			fs := afero.NewOsFs()
-			require.NoError(t, fs.MkdirAll(filepath.Join(tree, testSubdir), 0o750))
-			require.NoError(t, fs.MkdirAll(second, 0o750))
-			writeTestFile(t, fs, filepath.Join(tree, testFileTxt), "hello")
-			writeTestFile(t, fs, filepath.Join(tree, defaultManifestName), "an ordinary file")
-			writeTestFile(t, fs, filepath.Join(tree, testSubdirManifest), "an ordinary file")
-			writeTestFile(t, fs, filepath.Join(second, "second.txt"), "second")
-			writeTestFile(t, fs, filepath.Join(second, defaultManifestName), "an ordinary file")
-
-			paths := make([]string, 0, len(tc.paths))
-			for _, path := range tc.paths {
-				paths = append(paths, filepath.Join(root, path))
-			}
-
-			output := filepath.Join(root, "other.mf")
-			opts := testOpts(slices.Concat(
-				[]string{testApp, cmdGenerate, "-q", "-o", output}, paths,
-			), fs)
-			require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-			assert.ElementsMatch(t, tc.listed, manifestPaths(t, fs, output))
-		})
-	}
-}
-
-// TestGenerateListsTargetOfTopIndexMFSymlink makes index.mf at the top of
-// the tree a symlink to another file in it, and runs gen with --output
-// outside the tree: the file it points to is listed under its own name,
-// and the symlink, which gen lists only with --follow-symlinks, is left out
-// like any index.mf at the top of the tree.
-func TestGenerateListsTargetOfTopIndexMFSymlink(t *testing.T) {
-	t.Parallel()
-
-	// Paths are relative to tree, which holds file.txt, sub/index.mf and
-	// index.mf, a symlink to target.
-	for name, tc := range map[string]struct {
-		target string
-		flags  []string
-		path   string // the path gen is given
-		listed []string
-	}{
-		"directory given": {
-			testSubdirManifest, nil, ".",
-			[]string{testFileTxt, testSubdirManifest},
-		},
-		"directory given, --follow-symlinks": {
-			testSubdirManifest,
-			[]string{testFlagFollowSymlinks},
-			".",
-			[]string{testFileTxt, testSubdirManifest},
-		},
-		"file given": {
-			testFileTxt, nil, testFileTxt,
-			[]string{testFileTxt},
-		},
-	} {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			root := t.TempDir()
-			tree := filepath.Join(root, "tree")
-
-			fs := afero.NewOsFs()
-			require.NoError(t, fs.MkdirAll(filepath.Join(tree, testSubdir), 0o750))
-			writeTestFile(t, fs, filepath.Join(tree, testFileTxt), "hello")
-			writeTestFile(t, fs, filepath.Join(tree, testSubdirManifest), "an ordinary file")
-			require.NoError(t,
-				os.Symlink(tc.target, filepath.Join(tree, defaultManifestName)))
-
-			output := filepath.Join(root, "other.mf")
-			opts := testOpts(slices.Concat(
-				[]string{testApp, cmdGenerate, "-q", "-o", output}, tc.flags,
-				[]string{filepath.Join(tree, tc.path)},
-			), fs)
-			require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-			assert.ElementsMatch(t, tc.listed, manifestPaths(t, fs, output))
-		})
-	}
-}
-
-// TestGenerateListsIndexMFBesideFileGiven runs gen with --output outside
-// the tree, given sub/b.txt and the tree: sub/index.mf, beside the file
-// given, is listed, since a file given is left out only when its own name
-// is index.mf.
-//
-//nolint:paralleltest // changes the process-global working directory
-func TestGenerateListsIndexMFBesideFileGiven(t *testing.T) {
-	setupManifestInSubdir(t)
-
-	fs := afero.NewOsFs()
-	output := filepath.Join(t.TempDir(), "other.mf")
-
-	opts := testOpts([]string{
-		testApp, cmdGenerate, "-q", "-o", output, filepath.Join(testSubdir, "b.txt"), ".",
-	}, fs)
-	require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-	assert.Contains(t, manifestPaths(t, fs, output), testSubdirManifest)
-}
-
 // writeTestTree writes file.txt and sub/nested.txt under dir.
 func writeTestTree(t *testing.T, fs afero.Fs, dir string) {
 	t.Helper()

internal/cli/export.go

@@ -17,6 +17,7 @@ type ExportEntry struct {
 	Size   int64    `json:"size"`
 	Hashes []string `json:"hashes"`
 	Mtime  *string  `json:"mtime,omitempty"`
+	Ctime  *string  `json:"ctime,omitempty"`
 	Mode   string   `json:"mode"` // octal, "0000" when none was recorded
 }
 
@@ -60,6 +61,12 @@ func (mfa *CLIApp) exportManifestOperation(
 			entry.Mtime = &t
 		}
 
+		if f.GetCtime() != nil {
+			t := time.Unix(f.GetCtime().GetSeconds(), int64(f.GetCtime().GetNanos())).
+				UTC().Format(time.RFC3339Nano)
+			entry.Ctime = &t
+		}
+
 		entries = append(entries, entry)
 	}
 

internal/cli/export_test.go

@@ -4,10 +4,8 @@ import (
 	"bytes"
 	"context"
 	"encoding/json"
-	"maps"
 	"net/http"
 	"net/http/httptest"
-	"slices"
 	"testing"
 
 	"github.com/spf13/afero"
@@ -86,36 +84,6 @@ func TestExportManifestOperation(t *testing.T) {
 	assert.True(t, pathSet["sub/file.txt"])
 }
 
-// export prints the fields of a file entry and nothing else, so no ctime.
-func TestExportPrintsOnlyFileEntryFields(t *testing.T) {
-	t.Parallel()
-
-	manifestData := buildTestManifest(t, map[string][]byte{"entry.txt": []byte("aaa")})
-
-	fs := afero.NewMemMapFs()
-	require.NoError(t, afero.WriteFile(fs, "/test.mf", manifestData, 0o644))
-
-	var stdout, stderr bytes.Buffer
-
-	exitCode := runCLI(&RunOptions{
-		Appname: testApp,
-		Args:    []string{testApp, testCmdExport, "/test.mf"},
-		Stdin:   &bytes.Buffer{},
-		Stdout:  &stdout,
-		Stderr:  &stderr,
-		Fs:      fs,
-	})
-
-	require.Equal(t, 0, exitCode, "stderr: %s", stderr.String())
-
-	var entries []map[string]json.RawMessage
-	require.NoError(t, json.Unmarshal(stdout.Bytes(), &entries))
-	require.Len(t, entries, 1)
-	assert.Equal(t,
-		[]string{"hashes", "mode", "mtime", "path", "size"},
-		slices.Sorted(maps.Keys(entries[0])))
-}
-
 func TestExportFromHTTPURL(t *testing.T) {
 	t.Parallel()
 

internal/cli/freshen.go

@@ -192,12 +192,6 @@ func (s *freshenScanner) walk(path string, info fs.FileInfo, walkErr error) erro
 		return nil
 	}
 
-	// Leave out a file listed as index.mf, at the top of the base
-	// directory, since fetch saves the manifest there, as gen does.
-	if relPath == defaultManifestName {
-		return nil
-	}
-
 	// Handle symlinks
 	if info.Mode()&fs.ModeSymlink != 0 {
 		realInfo, keep := s.resolveSymlink(path)

internal/cli/freshen_test.go

@@ -413,77 +413,6 @@ func TestFreshenLeavesLeftoverTempFileOutOfListing(t *testing.T) {
 		manifestPaths(t, fs, manifestPath))
 }
 
-// TestFreshenLeavesTopIndexMFOutOfListing freshens a manifest kept outside
-// the directory --base names, after index.mf and sub/index.mf are added
-// there: index.mf at the top is not listed, since fetch saves the manifest
-// there, while sub/index.mf is listed.
-func TestFreshenLeavesTopIndexMFOutOfListing(t *testing.T) {
-	t.Parallel()
-
-	root := t.TempDir()
-	tree := filepath.Join(root, "tree")
-	manifestPath := filepath.Join(root, "other.mf")
-
-	fs := afero.NewOsFs()
-	require.NoError(t, fs.MkdirAll(filepath.Join(tree, testSubdir), 0o750))
-	writeTestFile(t, fs, filepath.Join(tree, testFileTxt), "hello")
-
-	opts := testOpts([]string{testApp, cmdGenerate, "-q", "-o", manifestPath, tree}, fs)
-	require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-	writeTestFile(t, fs, filepath.Join(tree, defaultManifestName), "an ordinary file")
-	writeTestFile(t, fs, filepath.Join(tree, testSubdirManifest), "an ordinary file")
-
-	runFreshen(t, fs, tree, manifestPath)
-
-	assert.ElementsMatch(t, []string{testFileTxt, testSubdirManifest},
-		manifestPaths(t, fs, manifestPath))
-}
-
-// TestFreshenListsTargetOfTopIndexMFSymlink freshens a manifest kept
-// outside the directory --base names, after sub/index.mf is added there
-// and index.mf at the top is made a symlink to it: sub/index.mf is listed,
-// and the symlink, which freshen lists only with --follow-symlinks, is left
-// out like any index.mf at the top.
-func TestFreshenListsTargetOfTopIndexMFSymlink(t *testing.T) {
-	t.Parallel()
-
-	for name, flags := range map[string][]string{
-		"plain freshen":             nil,
-		"freshen --follow-symlinks": {testFlagFollowSymlinks},
-	} {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			root := t.TempDir()
-			tree := filepath.Join(root, "tree")
-			manifestPath := filepath.Join(root, "other.mf")
-
-			fs := afero.NewOsFs()
-			require.NoError(t, fs.MkdirAll(filepath.Join(tree, testSubdir), 0o750))
-			writeTestFile(t, fs, filepath.Join(tree, testFileTxt), "hello")
-
-			opts := testOpts([]string{
-				testApp, cmdGenerate, "-q", "-o", manifestPath, tree,
-			}, fs)
-			require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-			writeTestFile(t, fs, filepath.Join(tree, testSubdirManifest), "an ordinary file")
-			require.NoError(t, os.Symlink(testSubdirManifest,
-				filepath.Join(tree, defaultManifestName)))
-
-			opts = testOpts(slices.Concat(
-				[]string{testApp, cmdFreshen, "-q"}, flags,
-				[]string{testFlagBase, tree, manifestPath},
-			), fs)
-			require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
-
-			assert.ElementsMatch(t, []string{testFileTxt, testSubdirManifest},
-				manifestPaths(t, fs, manifestPath))
-		})
-	}
-}
-
 // TestFreshenResolvesEntriesAgainstManifestDirectory adds sub/c.txt, then
 // runs freshen from the directory above sub, on the manifest in sub.
 // Without --base, the manifest then lists the files in sub, whether freshen

internal/cli/gen.go

@@ -137,9 +137,6 @@ func (mfa *CLIApp) buildScannerOptions(
 		// Neither a manifest being replaced nor a temp file left by an
 		// interrupted run belongs in the new manifest.
 		ExcludePaths: []string{output, manifestTempPath(output)},
-		// Nor does a file listed as index.mf, at the top of the tree, since
-		// fetch saves the manifest there and refuses a manifest that lists it.
-		ExcludeRelPaths: []mfer.RelFilePath{defaultManifestName},
 	}
 
 	// Set seed for deterministic UUID if provided

internal/cli/pipe_test.go

@@ -81,7 +81,7 @@ func TestFreshenSkipsPipes(t *testing.T) {
 	require.NoError(t, os.Symlink(target, filepath.Join(root, testLink)))
 
 	opts := testOpts([]string{
-		testApp, cmdFreshen, "-v", testFlagFollowSymlinks,
+		testApp, cmdFreshen, "-v", "--follow-symlinks",
 		testFlagBase, root, manifestPath,
 	}, fs)
 	require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))

mfer/constants.go

@@ -23,20 +23,23 @@ const (
 	// that decoding sets aside a fixed amount of memory for.
 	filesFieldNumber    = 101
 	hashesFieldNumber   = 3
+	mimeTypeFieldNumber = 301
 	mtimeFieldNumber    = 302
+	ctimeFieldNumber    = 303
 
-	// Bytes decoding sets aside for each file entry, hash and timestamp,
+	// Bytes decoding sets aside for each file entry, hash, timestamp and
-	// however short its encoding. checkDecodedSize refuses an inner
+	// MIME type, however short its encoding. checkDecodedSize refuses an
-	// message for which these add up to more than maxDecodedGrowth times
+	// inner message for which these add up to more than maxDecodedGrowth
-	// its size. The mode is held in the file entry itself.
+	// times its size. The mode is held in the file entry itself.
-	decodedFileEntrySize = 160
+	decodedFileEntrySize = 176
 	decodedHashSize      = 112
 	decodedTimestampSize = 64
+	decodedMIMETypeSize  = 16
 
 	// Each file entry mfer writes holds a path of at least one byte, a
 	// multihash at least as long as SHA-256's 34 bytes (AddFileWithHash
 	// refuses shorter ones) and a modification time: at least 47 bytes,
-	// counted at 336. So its manifests add up to at most about 7.15 times
+	// counted at 352. So its manifests add up to at most about 7.49 times
-	// their size, and this limit is about 12% above that.
+	// their size, and this limit is about 7% above that.
 	maxDecodedGrowth = 8
 )

mfer/deserialize.go

@@ -161,10 +161,10 @@ func (m *manifest) decompressInner() ([]byte, error) {
 }
 
 // checkDecodedSize refuses an encoded inner message whose file entries,
-// hashes and timestamps would take more than maxDecodedGrowth times its
+// hashes, timestamps and MIME types would take more than maxDecodedGrowth
-// size to decode. Decoding sets aside a fixed amount for each, however
+// times its size to decode. Decoding sets aside a fixed amount for each,
-// short its encoding, so a message of empty ones would take about 50 times
+// however short its encoding, so a message of empty ones would take about
-// its size.
+// 50 times its size.
 func checkDecodedSize(inner []byte) error {
 	limit := maxDecodedGrowth * int64(len(inner))
 
@@ -194,10 +194,14 @@ func checkDecodedSize(inner []byte) error {
 				return add(decodedHashSize)
 			}
 
-			if num == mtimeFieldNumber {
+			if num == mtimeFieldNumber || num == ctimeFieldNumber {
 				return add(decodedTimestampSize)
 			}
 
+			if num == mimeTypeFieldNumber {
+				return add(decodedMIMETypeSize)
+			}
+
 			return nil
 		})
 	})

mfer/deserialize_fuzz_test.go

@@ -48,8 +48,8 @@ func FuzzNewManifestFromReader(f *testing.F) {
 		// It also keeps a few copies of its input. Buffers grow by
 		// copying, so reaching those sizes allocates up to about six times
 		// them in total. Decoding the decompressed data takes up to
-		// maxDecodedGrowth times its size for file entries, hashes and
+		// maxDecodedGrowth times its size for file entries, hashes,
-		// timestamps, and drops fields it does not know.
+		// timestamps and MIME types, and drops fields it does not know.
 		// The strings and bytes it copies out of it, such as many one-byte
 		// values in one hash, take up to about five times more under the
 		// race detector, which pads every small copy to 16 bytes, and about

mfer/deserialize_path_test.go

@@ -295,17 +295,14 @@ func TestDeserializeRefusesModeOutsidePermissionBits(t *testing.T) {
 	}
 }
 
-// Entries of a path, a SHA-256 hash, an empty second hash and an empty
+// Entries of a path, a SHA-256 hash, an empty MIME type and empty
-// modification time are counted at 448 bytes each (160 + 112 + 112 + 64)
+// modification and change times are counted at 432 bytes each (176 + 112 +
-// and take 48 bytes plus the path to encode. A 7-character path makes that
+// 16 + 64 + 64) and take 52 bytes plus the path to encode. A one-character
-// 55 bytes, about 8.1 times: refused, and leaving any one of the four
+// path makes that 53 bytes, about 8.2 times: refused, and leaving any one of
-// uncounted, even the modification time, brings it under 8. An 8-character
+// the five uncounted, even the MIME type, brings it under 8. A two-character
-// path makes it 56 bytes, 8 times exactly: loaded. Without the second hash
+// path makes it 54 bytes, 8 times exactly: loaded. There is one entry for
-// an entry is never refused: it takes at least 47 bytes and is counted at
+// each base-36 digit, and each entry's path is its number in base 36, padded
-// 336. There is one entry for each base-36 digit, enough that the 22 bytes
+// with zeros to that length, since a manifest lists a path only once.
-// of the version and UUID change neither outcome, and each entry's path is
-// its number in base 36, padded with zeros to that length, since a manifest
-// lists a path only once.
 func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
 	t.Parallel()
 
@@ -319,8 +316,8 @@ func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
 		pathLen int
 		refused bool
 	}{
-		{7, true},
+		{1, true},
-		{8, false},
+		{2, false},
 	}
 
 	for _, tt := range tests {
@@ -341,10 +338,12 @@ func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
 				entry = protowire.AppendString(entry, path)
 				entry = protowire.AppendTag(entry, 3, protowire.BytesType) // MFFilePath.hashes
 				entry = protowire.AppendBytes(entry, checksum)
-				entry = protowire.AppendTag(entry, 3, protowire.BytesType) // MFFilePath.hashes
+				entry = protowire.AppendTag(entry, 301, protowire.BytesType) // MFFilePath.mimeType
 				entry = protowire.AppendBytes(entry, nil)
 				entry = protowire.AppendTag(entry, 302, protowire.BytesType) // MFFilePath.mtime
 				entry = protowire.AppendBytes(entry, nil)
+				entry = protowire.AppendTag(entry, 303, protowire.BytesType) // MFFilePath.ctime
+				entry = protowire.AppendBytes(entry, nil)
 
 				inner = protowire.AppendTag(inner, 101, protowire.BytesType) // MFFile.files
 				inner = protowire.AppendBytes(inner, entry)
@@ -400,44 +399,6 @@ func TestDeserializeDropsUnknownFields(t *testing.T) {
 	assert.Empty(t, m.Files()[0].ProtoReflect().GetUnknown())
 }
 
-// A file entry holding fields 301 and 303, which mf.proto reserves, loads,
-// and the loaded entry keeps neither.
-func TestDeserializeDropsReservedFields(t *testing.T) {
-	t.Parallel()
-
-	hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
-	require.NoError(t, err)
-
-	checksum, err := proto.Marshal(&MFFileChecksum{MultiHash: hash})
-	require.NoError(t, err)
-
-	timestamp, err := proto.Marshal(&Timestamp{Seconds: 1})
-	require.NoError(t, err)
-
-	entry := protowire.AppendTag(nil, 1, protowire.BytesType) // MFFilePath.path
-	entry = protowire.AppendString(entry, "a")
-	entry = protowire.AppendTag(entry, 3, protowire.BytesType) // MFFilePath.hashes
-	entry = protowire.AppendBytes(entry, checksum)
-	entry = protowire.AppendTag(entry, 301, protowire.BytesType) // reserved
-	entry = protowire.AppendString(entry, "text/plain")
-	entry = protowire.AppendTag(entry, 303, protowire.BytesType) // reserved
-	entry = protowire.AppendBytes(entry, timestamp)
-
-	id := uuid.NewV4()
-	inner := protowire.AppendTag(nil, 100, protowire.VarintType) // MFFile.version
-	inner = protowire.AppendVarint(inner, uint64(MFFile_VERSION_ONE))
-	inner = protowire.AppendTag(inner, 101, protowire.BytesType) // MFFile.files
-	inner = protowire.AppendBytes(inner, entry)
-	inner = protowire.AppendTag(inner, 102, protowire.BytesType) // MFFile.uuid
-	inner = protowire.AppendBytes(inner, id[:])
-
-	m, err := NewManifestFromReader(bytes.NewReader(wrapInner(t, id, inner)))
-	require.NoError(t, err)
-	require.Len(t, m.Files(), 1)
-	assert.Equal(t, "a", m.Files()[0].GetPath())
-	assert.Empty(t, m.Files()[0].ProtoReflect().GetUnknown())
-}
-
 // Many empty files with names of at most three characters and modification
 // times at the epoch make about the densest manifest mfer writes: it takes
 // about 7 times its size to decode, and still loads. A signature would not

mfer/mf.pb.go

@@ -340,7 +340,9 @@ type MFFilePath struct {
 	// gotta have at least one:
 	Hashes []*MFFileChecksum `protobuf:"bytes,3,rep,name=hashes,proto3" json:"hashes,omitempty"`
 	// optional per-file metadata
+	MimeType *string    `protobuf:"bytes,301,opt,name=mimeType,proto3,oneof" json:"mimeType,omitempty"`
 	Mtime    *Timestamp `protobuf:"bytes,302,opt,name=mtime,proto3,oneof" json:"mtime,omitempty"`
+	Ctime    *Timestamp `protobuf:"bytes,303,opt,name=ctime,proto3,oneof" json:"ctime,omitempty"`
 	// permission bits, at most 0777; 0 when not recorded
 	Mode          uint32 `protobuf:"varint,304,opt,name=mode,proto3" json:"mode,omitempty"`
 	unknownFields protoimpl.UnknownFields
@@ -398,6 +400,13 @@ func (x *MFFilePath) GetHashes() []*MFFileChecksum {
 	return nil
 }
 
+func (x *MFFilePath) GetMimeType() string {
+	if x != nil && x.MimeType != nil {
+		return *x.MimeType
+	}
+	return ""
+}
+
 func (x *MFFilePath) GetMtime() *Timestamp {
 	if x != nil {
 		return x.Mtime
@@ -405,6 +414,13 @@ func (x *MFFilePath) GetMtime() *Timestamp {
 	return nil
 }
 
+func (x *MFFilePath) GetCtime() *Timestamp {
+	if x != nil {
+		return x.Ctime
+	}
+	return nil
+}
+
 func (x *MFFilePath) GetMode() uint32 {
 	if x != nil {
 		return x.Mode
@@ -557,16 +573,21 @@ const file_mf_proto_rawDesc = "" +
 	"\n" +
 	"_signatureB\t\n" +
 	"\a_signerB\x10\n" +
-	"\x0e_signingPubKey\"\xc5\x01\n" +
+	"\x0e_signingPubKey\"\x85\x02\n" +
 	"\n" +
 	"MFFilePath\x12\x12\n" +
 	"\x04path\x18\x01 \x01(\tR\x04path\x12\x12\n" +
 	"\x04size\x18\x02 \x01(\x03R\x04size\x12'\n" +
-	"\x06hashes\x18\x03 \x03(\v2\x0f.MFFileChecksumR\x06hashes\x12&\n" +
+	"\x06hashes\x18\x03 \x03(\v2\x0f.MFFileChecksumR\x06hashes\x12 \n" +
+	"\bmimeType\x18\xad\x02 \x01(\tH\x00R\bmimeType\x88\x01\x01\x12&\n" +
 	"\x05mtime\x18\xae\x02 \x01(\v2\n" +
-	".TimestampH\x00R\x05mtime\x88\x01\x01\x12\x13\n" +
+	".TimestampH\x01R\x05mtime\x88\x01\x01\x12&\n" +
-	"\x04mode\x18\xb0\x02 \x01(\rR\x04modeB\b\n" +
+	"\x05ctime\x18\xaf\x02 \x01(\v2\n" +
-	"\x06_mtimeJ\x06\b\xad\x02\x10\xae\x02J\x06\b\xaf\x02\x10\xb0\x02R\bmimeTypeR\x05ctime\".\n" +
+	".TimestampH\x02R\x05ctime\x88\x01\x01\x12\x13\n" +
+	"\x04mode\x18\xb0\x02 \x01(\rR\x04modeB\v\n" +
+	"\t_mimeTypeB\b\n" +
+	"\x06_mtimeB\b\n" +
+	"\x06_ctime\".\n" +
 	"\x0eMFFileChecksum\x12\x1c\n" +
 	"\tmultiHash\x18\x01 \x01(\fR\tmultiHash\"\xd6\x01\n" +
 	"\x06MFFile\x12)\n" +
@@ -610,14 +631,15 @@ var file_mf_proto_depIdxs = []int32{
 	1, // 1: MFFileOuter.compressionType:type_name -> MFFileOuter.CompressionType
 	6, // 2: MFFilePath.hashes:type_name -> MFFileChecksum
 	3, // 3: MFFilePath.mtime:type_name -> Timestamp
-	2, // 4: MFFile.version:type_name -> MFFile.Version
+	3, // 4: MFFilePath.ctime:type_name -> Timestamp
-	5, // 5: MFFile.files:type_name -> MFFilePath
+	2, // 5: MFFile.version:type_name -> MFFile.Version
-	3, // 6: MFFile.createdAt:type_name -> Timestamp
+	5, // 6: MFFile.files:type_name -> MFFilePath
-	7, // [7:7] is the sub-list for method output_type
+	3, // 7: MFFile.createdAt:type_name -> Timestamp
-	7, // [7:7] is the sub-list for method input_type
+	8, // [8:8] is the sub-list for method output_type
-	7, // [7:7] is the sub-list for extension type_name
+	8, // [8:8] is the sub-list for method input_type
-	7, // [7:7] is the sub-list for extension extendee
+	8, // [8:8] is the sub-list for extension type_name
-	0, // [0:7] is the sub-list for field type_name
+	8, // [8:8] is the sub-list for extension extendee
+	0, // [0:8] is the sub-list for field type_name
 }
 
 func init() { file_mf_proto_init() }

mfer/mf.proto

@@ -58,13 +58,11 @@ message MFFilePath {
     repeated MFFileChecksum hashes = 3;
 
     // optional per-file metadata
+    optional string mimeType = 301;
     optional Timestamp mtime = 302;
+    optional Timestamp ctime = 303;
     // permission bits, at most 0777; 0 when not recorded
     uint32 mode = 304;
-
-    // used by no field, so that a later version cannot reuse them by accident
-    reserved 301, 303;
-    reserved "mimeType", "ctime";
 }
 
 message MFFileChecksum {

mfer/mf.proto.sha256

@@ -1 +1 @@
-16a2e6b39986d02aa1c7c01dc6b97c9d4c30db7a9d000056bd5985a3fa7c8089  mf.proto
+1668b2f6b8639a9213873474e58dbacb3a4ef77cc27c42253cad8dd076e5f206  mf.proto

mfer/mf_test.go

@@ -7,7 +7,6 @@ import (
 	"strings"
 	"testing"
 
-	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 	"google.golang.org/protobuf/reflect/protoreflect"
 	"sneak.berlin/go/mfer/mfer"
@@ -39,7 +38,9 @@ func TestFileEntryFieldsMatchSpec(t *testing.T) {
 		"path":     1,
 		"size":     2,
 		"hashes":   3,
+		"mimeType": 301,
 		"mtime":    302,
+		"ctime":    303,
 		"mode":     304,
 	}
 
@@ -52,16 +53,3 @@ func TestFileEntryFieldsMatchSpec(t *testing.T) {
 
 	require.Equal(t, want, got)
 }
-
-// A file entry reserves the numbers 301 and 303 and the names mimeType and
-// ctime, as docs/FORMAT.md says, so that no field can take them.
-func TestFileEntryReservesMIMETypeAndCtime(t *testing.T) {
-	t.Parallel()
-
-	desc := (&mfer.MFFilePath{}).ProtoReflect().Descriptor()
-
-	assert.True(t, desc.ReservedRanges().Has(301))
-	assert.True(t, desc.ReservedRanges().Has(303))
-	assert.True(t, desc.ReservedNames().Has("mimeType"))
-	assert.True(t, desc.ReservedNames().Has("ctime"))
-}

mfer/scanner.go

@@ -8,7 +8,6 @@ import (
 	"os"
 	"path"
 	"path/filepath"
-	"slices"
 	"strings"
 	"sync"
 	"time"
@@ -65,9 +64,6 @@ type ScannerOptions struct {
 	Seed string
 	// ExcludePaths lists files to leave out of the listing, matched with os.SameFile.
 	ExcludePaths []string
-	// ExcludeRelPaths lists paths to leave out of the listing, matched
-	// against the path a file would be listed by, with no file looked up.
-	ExcludeRelPaths []RelFilePath
 }
 
 // FileEntry represents a file that has been enumerated.
@@ -76,6 +72,7 @@ type FileEntry struct {
 	AbsPath AbsFilePath // Absolute path (used for reading file content)
 	Size    FileSize    // File size in bytes
 	Mtime   ModTime     // Last modification time
+	Ctime   time.Time   // Creation time (platform-dependent)
 	Mode    fs.FileMode // Permission bits (Perm() of the file's mode)
 }
 
@@ -442,10 +439,9 @@ func (s *Scanner) enumerateFileWithInfo(
 	}
 
 	// Clean the path - remove leading slash if present
-	cleanPath := strings.TrimPrefix(filePath, "/")
+	cleanPath := filePath
-
+	if len(cleanPath) > 0 && cleanPath[0] == '/' {
-	if slices.Contains(s.options.ExcludeRelPaths, RelFilePath(cleanPath)) {
+		cleanPath = cleanPath[1:]
-		return nil
 	}
 
 	// Compute absolute path for file reading
@@ -497,6 +493,8 @@ func (s *Scanner) enumerateFileWithInfo(
 		Size:    FileSize(info.Size()),
 		Mtime:   ModTime(info.ModTime()),
 		Mode:    info.Mode().Perm(),
+		// Note: Ctime not available from fs.FileInfo on all platforms
+		// Will need platform-specific code to extract it
 	}
 
 	s.mu.Lock()