Compare commits
3
Commits
438b73eddf
...
df985e44dc
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
df985e44dc | ||
|
|
acff23d92b | ||
|
|
8fe0244291 |
+2
-1
@@ -9,7 +9,8 @@ RUN go mod download
|
|||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# Go half of fmt-check only: this image has no node, so no prettier. The
|
# Go half of fmt-check only: this image has no node, so no prettier. The
|
||||||
# markdown half runs in the mdfmt stage below.
|
# markdown half runs in the mdfmt stage below. The image has no gofumpt
|
||||||
|
# either; script/gofumpt builds the version it pins with `go run`.
|
||||||
RUN make fmt-check-go
|
RUN make fmt-check-go
|
||||||
# The linter directly, not `make lint`: script/lint builds this stage, and
|
# The linter directly, not `make lint`: script/lint builds this stage, and
|
||||||
# there is no docker inside this build.
|
# there is no docker inside this build.
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ fmt-check:
|
|||||||
# Halves of fmt-check, for environments that have only one toolchain:
|
# Halves of fmt-check, for environments that have only one toolchain:
|
||||||
# the Docker lint stage has Go but no node, the markdown stage the reverse.
|
# the Docker lint stage has Go but no node, the markdown stage the reverse.
|
||||||
fmt-check-go:
|
fmt-check-go:
|
||||||
@script/fmt-check-go
|
@script/gofumpt --check
|
||||||
|
|
||||||
fmt-check-md:
|
fmt-check-md:
|
||||||
@script/prettier --check
|
@script/prettier --check
|
||||||
|
|||||||
@@ -40,12 +40,14 @@ tree by URL:
|
|||||||
bin/mfer gen .
|
bin/mfer gen .
|
||||||
|
|
||||||
# Verify the files on disk against the manifest. Exits nonzero if any file
|
# Verify the files on disk against the manifest. Exits nonzero if any file
|
||||||
# is missing or corrupted.
|
# it lists is missing or corrupted; warns about files it does not list.
|
||||||
bin/mfer check index.mf
|
bin/mfer check index.mf
|
||||||
|
|
||||||
# Download and cryptographically verify a tree published over HTTP: mfer
|
# Download and cryptographically verify a tree published over HTTP into
|
||||||
# fetches <url>/index.mf, then downloads every file it lists.
|
# ./mirror: mfer fetches <url>/index.mf, downloads every file it lists,
|
||||||
bin/mfer fetch https://example.com/tree/
|
# skipping any already there with the right hash, then saves the manifest as
|
||||||
|
# mirror/index.mf.
|
||||||
|
bin/mfer fetch --dest mirror https://example.com/tree/
|
||||||
```
|
```
|
||||||
|
|
||||||
Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
|
Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
|
||||||
@@ -91,16 +93,19 @@ provide:
|
|||||||
- `script/lint` — run `golangci-lint` in Docker: builds only the `lint` stage of
|
- `script/lint` — run `golangci-lint` in Docker: builds only the `lint` stage of
|
||||||
the `Dockerfile` (the Go format check, then the linter), uncached so it runs
|
the `Dockerfile` (the Go format check, then the linter), uncached so it runs
|
||||||
every time, then removes the image
|
every time, then removes the image
|
||||||
- `script/fmt` — format all code and docs (writes): `gofumpt` and
|
- `script/fmt` — format all code and docs (writes): `script/gofumpt --write` and
|
||||||
`script/prettier --write`
|
`script/prettier --write`
|
||||||
|
- `script/gofumpt` — run `gofumpt` over every Go file in the repository in the
|
||||||
|
given mode, `--write` or `--check`, at the one version it pins (built on
|
||||||
|
demand by `go run`, so nothing installs it); `script/fmt`, `script/fmt-check`
|
||||||
|
and the Docker lint stage (`make fmt-check-go`) all go through it, so they
|
||||||
|
cannot disagree about Go formatting
|
||||||
- `script/prettier` — run prettier over the repository's canonical file set
|
- `script/prettier` — run prettier over the repository's canonical file set
|
||||||
(Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or
|
(Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or
|
||||||
`--check`; the single definition of that file set, so `script/fmt` and
|
`--check`; the single definition of that file set, so `script/fmt` and
|
||||||
`script/fmt-check` cannot disagree about it
|
`script/fmt-check` cannot disagree about it
|
||||||
- `script/fmt-check` — check formatting without writing: `script/fmt-check-go`
|
- `script/fmt-check` — check formatting without writing:
|
||||||
plus `script/prettier --check`
|
`script/gofumpt --check` plus `script/prettier --check`
|
||||||
- `script/fmt-check-go` — the Go half of `script/fmt-check`, on its own, for the
|
|
||||||
Docker lint stage, whose image has no node
|
|
||||||
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
|
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
|
||||||
- `script/docker` — build the Docker image tagged with the project name
|
- `script/docker` — build the Docker image tagged with the project name
|
||||||
- `script/cibuild` — CI entrypoint: builds the image with the same command as
|
- `script/cibuild` — CI entrypoint: builds the image with the same command as
|
||||||
@@ -251,10 +256,22 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
|
|||||||
- `mfer check` / `mfer check .`
|
- `mfer check` / `mfer check .`
|
||||||
- verifies checksums of all files in manifest, displaying error and exiting
|
- verifies checksums of all files in manifest, displaying error and exiting
|
||||||
nonzero if any files are missing or corrupted
|
nonzero if any files are missing or corrupted
|
||||||
|
- warns about each file under the base directory that the manifest does not
|
||||||
|
list, hidden files included; with `--no-extra-files` each one is a failure
|
||||||
|
instead
|
||||||
- `mfer fetch https://example.com/stuff/`
|
- `mfer fetch https://example.com/stuff/`
|
||||||
- fetches `/stuff/index.mf` and downloads all files listed in manifest,
|
- fetches `/stuff/index.mf` and downloads all files listed in manifest into
|
||||||
optionally resuming any that already exist locally, and assures
|
the current directory, or the one given with `--dest`, and assures
|
||||||
cryptographic integrity of downloaded files.
|
cryptographic integrity of downloaded files. A file already there with the
|
||||||
|
size and hash the manifest lists is skipped. Once every file is in place,
|
||||||
|
the manifest is saved there as `index.mf`, so `mfer check` can verify the
|
||||||
|
tree later. A manifest that lists `index.mf` (in any letter case) or
|
||||||
|
`.index.mf.tmp` at the top of the tree is refused before any file is
|
||||||
|
downloaded, since saving the manifest would replace it.
|
||||||
|
- `mfer fetch --require-signature <fingerprint> https://example.com/stuff/`
|
||||||
|
- as above, but first refuses a manifest not signed by the key with that
|
||||||
|
fingerprint, as `mfer check --require-signature` does, before downloading
|
||||||
|
any file.
|
||||||
|
|
||||||
# Implementation Plan
|
# Implementation Plan
|
||||||
|
|
||||||
|
|||||||
+12
-10
@@ -206,16 +206,21 @@ func countCheckFailures(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// findExtraFiles reports files present on disk but absent from the
|
// findExtraFiles reports files present on disk but absent from the
|
||||||
// manifest, counting each as a failure.
|
// manifest, and anything the search cannot read: each is a failure under
|
||||||
|
// --no-extra-files, otherwise a warning.
|
||||||
func findExtraFiles(ctx *cli.Context, chk *mfer.Checker, failures *int64) error {
|
func findExtraFiles(ctx *cli.Context, chk *mfer.Checker, failures *int64) error {
|
||||||
extraResults := make(chan mfer.Result, 1)
|
extraResults := make(chan mfer.Result, 1)
|
||||||
extraDone := make(chan struct{})
|
extraDone := make(chan struct{})
|
||||||
|
|
||||||
go func() {
|
go func() {
|
||||||
for result := range extraResults {
|
for result := range extraResults {
|
||||||
*failures++
|
if ctx.Bool("no-extra-files") {
|
||||||
|
*failures++
|
||||||
|
|
||||||
log.Infof("%s: %s (%s)", result.Status, result.Path, result.Message)
|
log.Infof("%s: %s (%s)", result.Status, result.Path, result.Message)
|
||||||
|
} else {
|
||||||
|
log.Warnf("%s: %s (%s)", result.Status, result.Path, result.Message)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
close(extraDone)
|
close(extraDone)
|
||||||
@@ -270,12 +275,9 @@ func runCheck(ctx *cli.Context, chk *mfer.Checker, showProgress bool) (int64, er
|
|||||||
// Wait for results processing to complete
|
// Wait for results processing to complete
|
||||||
<-done
|
<-done
|
||||||
|
|
||||||
// Check for extra files if requested
|
err = findExtraFiles(ctx, chk, &failures)
|
||||||
if ctx.Bool("no-extra-files") {
|
if err != nil {
|
||||||
err = findExtraFiles(ctx, chk, &failures)
|
return 0, err
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return failures, nil
|
return failures, nil
|
||||||
@@ -317,7 +319,7 @@ func (mfa *CLIApp) checkManifestOperation(ctx *cli.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Check signature requirement
|
// Check signature requirement
|
||||||
requiredSigner := ctx.String("require-signature")
|
requiredSigner := ctx.String(flagRequireSignature)
|
||||||
if requiredSigner != "" {
|
if requiredSigner != "" {
|
||||||
err = verifyRequiredSigner(ctx.Context, chk, requiredSigner)
|
err = verifyRequiredSigner(ctx.Context, chk, requiredSigner)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
+141
-16
@@ -705,30 +705,155 @@ func TestNoExtraFilesWithSubdirectory(t *testing.T) {
|
|||||||
"check should fail when extra files exist in subdirectory")
|
"check should fail when extra files exist in subdirectory")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestCheckWithoutNoExtraFilesIgnoresExtra(t *testing.T) {
|
// TestCheckWarnsAboutUnlistedFiles adds one file the manifest does not list
|
||||||
|
// to a tree that had none: it gets one warning and the check passes, unless
|
||||||
|
// --no-extra-files makes it a failure. --quiet hides the warning, not the
|
||||||
|
// failure.
|
||||||
|
func TestCheckWarnsAboutUnlistedFiles(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
fs := afero.NewMemMapFs()
|
for name, unlisted := range map[string]string{
|
||||||
|
"regular file": "extra.txt",
|
||||||
|
"dotfile": ".hidden",
|
||||||
|
"file in hidden directory": ".git/config",
|
||||||
|
} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
// Create test file
|
fs := afero.NewMemMapFs()
|
||||||
require.NoError(t, fs.MkdirAll(testDir, 0o755))
|
require.NoError(t, fs.MkdirAll(testDir, 0o755))
|
||||||
writeTestFile(t, fs, testFile1, "hello")
|
writeTestFile(t, fs, testFile1, "hello")
|
||||||
|
|
||||||
// Generate manifest
|
opts := testOpts([]string{
|
||||||
opts := testOpts([]string{testApp, cmdGenerate, "-q", "-o", testManifest, testDir}, fs)
|
testApp, cmdGenerate, "-q", "-o", testManifest, testDir,
|
||||||
exitCode := runCLI(opts)
|
}, fs)
|
||||||
require.Equal(t, 0, exitCode)
|
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
|
||||||
|
|
||||||
// Add extra file
|
check := func(flags ...string) (int, string) {
|
||||||
writeTestFile(t, fs, "/testdir/extra.txt", "extra")
|
args := append([]string{testApp, cmdCheck, testFlagBase, testDir}, flags...)
|
||||||
|
opts := testOpts(append(args, testManifest), fs)
|
||||||
|
|
||||||
|
return runCLI(opts), testStderr(t, opts)
|
||||||
|
}
|
||||||
|
|
||||||
|
exitCode, stderr := check()
|
||||||
|
assert.Equal(t, 0, exitCode, "stderr: %s", stderr)
|
||||||
|
assert.NotContains(t, stderr, "not in manifest")
|
||||||
|
|
||||||
|
writeTestFile(t, fs, filepath.Join(testDir, unlisted), "unlisted")
|
||||||
|
|
||||||
|
exitCode, stderr = check()
|
||||||
|
assert.Equal(t, 0, exitCode, "stderr: %s", stderr)
|
||||||
|
assert.Equal(t, 1, strings.Count(stderr, "not in manifest"), stderr)
|
||||||
|
assert.Contains(t, stderr, unlisted)
|
||||||
|
|
||||||
|
exitCode, stderr = check("-q")
|
||||||
|
assert.Equal(t, 0, exitCode, "stderr: %s", stderr)
|
||||||
|
assert.NotContains(t, stderr, "not in manifest")
|
||||||
|
|
||||||
|
exitCode, stderr = check(testFlagNoExtra)
|
||||||
|
assert.Equal(t, 1, exitCode, "stderr: %s", stderr)
|
||||||
|
assert.Contains(t, stderr, unlisted)
|
||||||
|
|
||||||
|
exitCode, _ = check("-q", testFlagNoExtra)
|
||||||
|
assert.Equal(t, 1, exitCode)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCheckNeverReportsManifest keeps the manifest inside the checked tree,
|
||||||
|
// under several names and path spellings, and names the tree both directly
|
||||||
|
// and through a symlink: the manifest is never reported, even under
|
||||||
|
// --no-extra-files. The manifest is recognized by file identity, which needs
|
||||||
|
// the real filesystem.
|
||||||
|
func TestCheckNeverReportsManifest(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// Manifest paths are relative to the checked tree.
|
||||||
|
for name, manifest := range map[string]string{
|
||||||
|
"default name": defaultManifestName,
|
||||||
|
"hidden name": ".index.mf",
|
||||||
|
"other name in a subdirectory": "sub/listing.mf",
|
||||||
|
"path spelled through ..": "sub/../index.mf",
|
||||||
|
} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// A temp dir holding data/tree and link, a symlink to data.
|
||||||
|
root := t.TempDir()
|
||||||
|
tree := filepath.Join(root, "data", "tree")
|
||||||
|
// Not filepath.Join, which would clean away a "..".
|
||||||
|
manifestPath := tree + "/" + manifest
|
||||||
|
|
||||||
|
fs := afero.NewOsFs()
|
||||||
|
require.NoError(t, fs.MkdirAll(filepath.Join(tree, "sub"), 0o750))
|
||||||
|
require.NoError(t,
|
||||||
|
os.Symlink(filepath.Join(root, "data"), filepath.Join(root, "link")))
|
||||||
|
writeTestFile(t, fs, filepath.Join(tree, testFileTxt), "hello")
|
||||||
|
|
||||||
|
opts := testOpts([]string{
|
||||||
|
testApp, cmdGenerate, "-q", "-o", manifestPath, tree,
|
||||||
|
}, fs)
|
||||||
|
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
|
||||||
|
|
||||||
|
for _, base := range []string{tree, filepath.Join(root, "link", "tree")} {
|
||||||
|
opts = testOpts([]string{
|
||||||
|
testApp, cmdCheck, testFlagNoExtra, testFlagBase, base, manifestPath,
|
||||||
|
}, fs)
|
||||||
|
assert.Equal(t, 0, runCLI(opts),
|
||||||
|
"base %s, stderr: %s", base, testStderr(t, opts))
|
||||||
|
assert.NotContains(t, testStderr(t, opts), "not in manifest")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// unlistableDirFs is a filesystem on which one directory cannot be listed.
|
||||||
|
type unlistableDirFs struct {
|
||||||
|
afero.Fs
|
||||||
|
|
||||||
|
dir string
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:ireturn // Open must return afero.File to satisfy afero.Fs.
|
||||||
|
func (f unlistableDirFs) Open(name string) (afero.File, error) {
|
||||||
|
if name == f.dir {
|
||||||
|
return nil, os.ErrPermission
|
||||||
|
}
|
||||||
|
|
||||||
|
return f.Fs.Open(name)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCheckWithUnlistableDirectory has a directory under the base that
|
||||||
|
// cannot be listed, followed by an unlisted file: both are warned about and
|
||||||
|
// the check still passes, unless --no-extra-files is given.
|
||||||
|
func TestCheckWithUnlistableDirectory(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mem := afero.NewMemMapFs()
|
||||||
|
require.NoError(t, mem.MkdirAll("/testdir/locked", 0o755))
|
||||||
|
writeTestFile(t, mem, testFile1, "hello")
|
||||||
|
|
||||||
|
opts := testOpts([]string{
|
||||||
|
testApp, cmdGenerate, "-q", "-o", testManifest, testDir,
|
||||||
|
}, mem)
|
||||||
|
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
|
||||||
|
|
||||||
|
// Directories are searched in name order, so this comes after "locked".
|
||||||
|
writeTestFile(t, mem, "/testdir/unlisted.txt", "unlisted")
|
||||||
|
|
||||||
|
fs := unlistableDirFs{Fs: mem, dir: "/testdir/locked"}
|
||||||
|
|
||||||
|
opts = testOpts([]string{testApp, cmdCheck, testFlagBase, testDir, testManifest}, fs)
|
||||||
|
assert.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
|
||||||
|
assert.Contains(t, testStderr(t, opts), os.ErrPermission.Error())
|
||||||
|
assert.Contains(t, testStderr(t, opts), "unlisted.txt")
|
||||||
|
|
||||||
// Check WITHOUT --no-extra-files (should pass - extra files ignored)
|
|
||||||
opts = testOpts([]string{
|
opts = testOpts([]string{
|
||||||
testApp, cmdCheck, "-q", testFlagBase, testDir, testManifest,
|
testApp, cmdCheck, testFlagNoExtra, testFlagBase, testDir, testManifest,
|
||||||
}, fs)
|
}, fs)
|
||||||
exitCode = runCLI(opts)
|
assert.Equal(t, 1, runCLI(opts), "stderr: %s", testStderr(t, opts))
|
||||||
assert.Equal(t, 0, exitCode,
|
assert.Contains(t, testStderr(t, opts), "unlisted.txt")
|
||||||
"check without --no-extra-files should ignore extra files")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestGenerateAtomicWriteNoTempFileOnSuccess(t *testing.T) {
|
func TestGenerateAtomicWriteNoTempFileOnSuccess(t *testing.T) {
|
||||||
|
|||||||
+22
-12
@@ -111,9 +111,10 @@ func TestVerifyRequiredSignerMessages(t *testing.T) {
|
|||||||
// string; the required signer is a fixed value that cannot match it. Requires
|
// string; the required signer is a fixed value that cannot match it. Requires
|
||||||
// gpg and is skipped where it is absent, as the other signing tests are.
|
// gpg and is skipped where it is absent, as the other signing tests are.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // signedChecker calls t.Setenv, which bars t.Parallel
|
//nolint:paralleltest // signedManifest calls t.Setenv, which bars t.Parallel
|
||||||
func TestSignerMismatchMessage(t *testing.T) {
|
func TestSignerMismatchMessage(t *testing.T) {
|
||||||
chk := signedChecker(t)
|
chk := signedChecker(t,
|
||||||
|
signedManifest(t, map[string][]byte{"f.txt": []byte("signed file")}))
|
||||||
|
|
||||||
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP(context.Background())
|
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP(context.Background())
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
@@ -125,9 +126,10 @@ func TestSignerMismatchMessage(t *testing.T) {
|
|||||||
" does not match required "+msgFpB)
|
" does not match required "+msgFpB)
|
||||||
}
|
}
|
||||||
|
|
||||||
// signedChecker builds a Checker over a manifest signed by a throwaway GPG
|
// signedManifest returns a manifest of files signed by a throwaway GPG key
|
||||||
// key generated in a temporary GNUPGHOME.
|
// generated in a temporary GNUPGHOME, which it leaves set for the rest of
|
||||||
func signedChecker(t *testing.T) *mfer.Checker {
|
// the test.
|
||||||
|
func signedManifest(t *testing.T, files map[string][]byte) []byte {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
_, err := exec.LookPath("gpg")
|
_, err := exec.LookPath("gpg")
|
||||||
@@ -159,17 +161,25 @@ func signedChecker(t *testing.T) *mfer.Checker {
|
|||||||
b := mfer.NewBuilder()
|
b := mfer.NewBuilder()
|
||||||
b.SetSigningOptions(&mfer.SigningOptions{KeyID: mfer.GPGKeyID("test@mfer.test")})
|
b.SetSigningOptions(&mfer.SigningOptions{KeyID: mfer.GPGKeyID("test@mfer.test")})
|
||||||
|
|
||||||
content := []byte("signed file")
|
for path, content := range files {
|
||||||
_, err = b.AddFile("f.txt", mfer.FileSize(len(content)), mfer.ModTime{},
|
_, err = b.AddFile(mfer.RelFilePath(path), mfer.FileSize(len(content)),
|
||||||
bytes.NewReader(content), nil)
|
mfer.ModTime{}, bytes.NewReader(content), nil)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
}
|
||||||
|
|
||||||
var buf bytes.Buffer
|
var buf bytes.Buffer
|
||||||
|
|
||||||
require.NoError(t, b.Build(context.Background(), &buf))
|
require.NoError(t, b.Build(context.Background(), &buf))
|
||||||
|
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
// signedChecker builds a Checker over manifest, a signed manifest.
|
||||||
|
func signedChecker(t *testing.T, manifest []byte) *mfer.Checker {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
fs := afero.NewMemMapFs()
|
fs := afero.NewMemMapFs()
|
||||||
require.NoError(t, afero.WriteFile(fs, "/index.mf", buf.Bytes(), 0o644))
|
require.NoError(t, afero.WriteFile(fs, "/index.mf", manifest, 0o644))
|
||||||
|
|
||||||
chk, err := mfer.NewChecker(&mfer.CheckerOptions{
|
chk, err := mfer.NewChecker(&mfer.CheckerOptions{
|
||||||
ManifestPath: "/index.mf",
|
ManifestPath: "/index.mf",
|
||||||
@@ -300,7 +310,7 @@ func TestFetchFileHTTPStatusMessage(t *testing.T) {
|
|||||||
|
|
||||||
// downloadFile logs each retry of the 500 to the process-global logger.
|
// downloadFile logs each retry of the 500 to the process-global logger.
|
||||||
err := runLocked(func() error {
|
err := runLocked(func() error {
|
||||||
return downloadFile(context.Background(), testClient(), server.URL+"/x", "x",
|
return downloadFile(context.Background(), testClient(), server.URL+"/x", ".", "x",
|
||||||
&mfer.MFFilePath{}, nil)
|
&mfer.MFFilePath{}, nil)
|
||||||
})
|
})
|
||||||
require.ErrorIs(t, err, errHTTPStatus)
|
require.ErrorIs(t, err, errHTTPStatus)
|
||||||
@@ -355,7 +365,7 @@ func TestSizeMismatchMessage(t *testing.T) {
|
|||||||
|
|
||||||
// finishDownload returns the size-mismatch error before it touches the
|
// finishDownload returns the size-mismatch error before it touches the
|
||||||
// paths, digest, or entry, so those can be zero here.
|
// paths, digest, or entry, so those can be zero here.
|
||||||
err := finishDownload("", "", 9, 10, nil, nil, nil, nil)
|
err := finishDownload("", "", "", 9, 10, nil, nil, nil, nil)
|
||||||
require.ErrorIs(t, err, errSizeMismatch)
|
require.ErrorIs(t, err, errSizeMismatch)
|
||||||
assert.EqualError(t, err, "size mismatch: expected 10 bytes, got 9")
|
assert.EqualError(t, err, "size mismatch: expected 10 bytes, got 9")
|
||||||
}
|
}
|
||||||
|
|||||||
+286
-93
@@ -19,6 +19,7 @@ import (
|
|||||||
|
|
||||||
"github.com/dustin/go-humanize"
|
"github.com/dustin/go-humanize"
|
||||||
"github.com/multiformats/go-multihash"
|
"github.com/multiformats/go-multihash"
|
||||||
|
"github.com/spf13/afero"
|
||||||
"github.com/urfave/cli/v2"
|
"github.com/urfave/cli/v2"
|
||||||
"sneak.berlin/go/mfer/internal/log"
|
"sneak.berlin/go/mfer/internal/log"
|
||||||
"sneak.berlin/go/mfer/mfer"
|
"sneak.berlin/go/mfer/mfer"
|
||||||
@@ -90,6 +91,10 @@ var (
|
|||||||
// errHashMismatch indicates a downloaded file whose hash matches no
|
// errHashMismatch indicates a downloaded file whose hash matches no
|
||||||
// manifest hash.
|
// manifest hash.
|
||||||
errHashMismatch = errors.New("hash mismatch")
|
errHashMismatch = errors.New("hash mismatch")
|
||||||
|
// errManifestNameListed indicates a manifest that lists a file where
|
||||||
|
// fetch saves the manifest.
|
||||||
|
errManifestNameListed = errors.New(
|
||||||
|
"manifest lists a file where fetch saves the manifest")
|
||||||
)
|
)
|
||||||
|
|
||||||
// DownloadProgress reports the progress of a single file download.
|
// DownloadProgress reports the progress of a single file download.
|
||||||
@@ -239,20 +244,34 @@ func manifestBaseURL(manifestURL string) (*url.URL, error) {
|
|||||||
return parsed.JoinPath(".."), nil
|
return parsed.JoinPath(".."), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// downloadManifestFiles downloads every file in the manifest, reporting
|
// downloadManifestFiles downloads every file in the manifest into dest,
|
||||||
// progress on the progress channel.
|
// reporting progress on the progress channel. A file already present in
|
||||||
|
// dest is skipped. It returns how many files it downloaded and their
|
||||||
|
// total size.
|
||||||
func downloadManifestFiles(
|
func downloadManifestFiles(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
client retryingClient,
|
client retryingClient,
|
||||||
baseURL *url.URL,
|
baseURL *url.URL,
|
||||||
|
dest string,
|
||||||
files []*mfer.MFFilePath,
|
files []*mfer.MFFilePath,
|
||||||
progress chan<- DownloadProgress,
|
progress chan<- DownloadProgress,
|
||||||
) error {
|
) (int, int64, error) {
|
||||||
|
var (
|
||||||
|
downloaded int
|
||||||
|
downloadedBytes int64
|
||||||
|
)
|
||||||
|
|
||||||
for _, f := range files {
|
for _, f := range files {
|
||||||
// Sanitize the path to prevent path traversal attacks
|
// Sanitize the path to prevent path traversal attacks
|
||||||
localPath, err := sanitizePath(f.GetPath())
|
localPath, err := sanitizePath(f.GetPath())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("invalid path in manifest: %w", err)
|
return 0, 0, fmt.Errorf("invalid path in manifest: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if alreadyPresent(dest, localPath, f) {
|
||||||
|
log.Infof("skipping %s: already present", f.GetPath())
|
||||||
|
|
||||||
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
// JoinPath takes escaped path text, so a name such as "100%.txt"
|
// JoinPath takes escaped path text, so a name such as "100%.txt"
|
||||||
@@ -260,13 +279,53 @@ func downloadManifestFiles(
|
|||||||
fileURL := baseURL.JoinPath(encodeFilePath(f.GetPath())).String()
|
fileURL := baseURL.JoinPath(encodeFilePath(f.GetPath())).String()
|
||||||
log.Infof("fetching %s", f.GetPath())
|
log.Infof("fetching %s", f.GetPath())
|
||||||
|
|
||||||
err = downloadFile(ctx, client, fileURL, localPath, f, progress)
|
err = downloadFile(ctx, client, fileURL, dest, localPath, f, progress)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to download %s: %w", f.GetPath(), err)
|
return 0, 0, fmt.Errorf("failed to download %s: %w", f.GetPath(), err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
downloaded++
|
||||||
|
downloadedBytes += f.GetSize()
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return downloaded, downloadedBytes, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// alreadyPresent reports whether localPath under dest is a regular file
|
||||||
|
// with the size and one of the hashes the manifest lists for entry. It
|
||||||
|
// hashes the whole file, since a matching size alone would accept a
|
||||||
|
// corrupted or partly written one. A file it cannot read, or reaches only
|
||||||
|
// through a symlink, is not present: fetch downloads it, and the download
|
||||||
|
// reports the problem.
|
||||||
|
func alreadyPresent(dest, localPath string, entry *mfer.MFFilePath) bool {
|
||||||
|
if checkNoSymlinks(dest, localPath) != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
path := filepath.Join(dest, localPath)
|
||||||
|
|
||||||
|
info, err := os.Lstat(path)
|
||||||
|
if err != nil || !info.Mode().IsRegular() || info.Size() != entry.GetSize() {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// G304: localPath is a relative path that sanitizePath keeps inside
|
||||||
|
// dest as text, and checkNoSymlinks just found no symlink in it.
|
||||||
|
f, err := os.Open(path) //nolint:gosec // G304: see comment above
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = f.Close() }()
|
||||||
|
|
||||||
|
h := sha256.New()
|
||||||
|
|
||||||
|
_, err = io.Copy(h, f)
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
return verifyDownloadedHash(h.Sum(nil), entry) == nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
|
func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
|
||||||
@@ -291,43 +350,22 @@ func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
|
|||||||
firstDelay: firstRetryDelay,
|
firstDelay: firstRetryDelay,
|
||||||
}
|
}
|
||||||
|
|
||||||
log.Infof("fetching manifest from %s", manifestURL)
|
manifestData, files, err := fetchManifest(ctx, client, manifestURL)
|
||||||
|
|
||||||
// Read the whole manifest before parsing it, so that a connection
|
|
||||||
// lost partway through is retried rather than reported as a bad
|
|
||||||
// manifest.
|
|
||||||
var manifestData []byte
|
|
||||||
|
|
||||||
err = client.get(ctx.Context, manifestURL, func(resp *http.Response) error {
|
|
||||||
var readErr error
|
|
||||||
|
|
||||||
manifestData, readErr = io.ReadAll(resp.Body)
|
|
||||||
|
|
||||||
return readErr
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to fetch manifest: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Parse manifest
|
|
||||||
manifest, err := mfer.NewManifestFromReader(bytes.NewReader(manifestData))
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to parse manifest: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
files := manifest.Files()
|
|
||||||
log.Infof("manifest contains %d files", len(files))
|
|
||||||
|
|
||||||
// Compute base URL (directory containing manifest)
|
// Compute base URL (directory containing manifest)
|
||||||
baseURL, err := manifestBaseURL(manifestURL)
|
baseURL, err := manifestBaseURL(manifestURL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Calculate total bytes to download
|
dest := ctx.String(flagDest)
|
||||||
var totalBytes int64
|
|
||||||
for _, f := range files {
|
err = os.MkdirAll(dest, dirPerms)
|
||||||
totalBytes += f.GetSize()
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to create destination directory %s: %w", dest, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create progress channel and start progress reporter goroutine
|
// Create progress channel and start progress reporter goroutine
|
||||||
@@ -340,7 +378,8 @@ func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
|
|||||||
startTime := time.Now()
|
startTime := time.Now()
|
||||||
|
|
||||||
// Download each file
|
// Download each file
|
||||||
dlErr := downloadManifestFiles(ctx.Context, client, baseURL, files, progress)
|
downloaded, downloadedBytes, dlErr := downloadManifestFiles(
|
||||||
|
ctx.Context, client, baseURL, dest, files, progress)
|
||||||
|
|
||||||
close(progress)
|
close(progress)
|
||||||
<-done
|
<-done
|
||||||
@@ -349,15 +388,150 @@ func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
|
|||||||
return dlErr
|
return dlErr
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Saved only now that every file is in place and verified, so that
|
||||||
|
// "mfer check" can verify the tree later.
|
||||||
|
err = saveManifest(dest, manifestData)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to save manifest: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
// Print summary
|
// Print summary
|
||||||
elapsed := time.Since(startTime)
|
elapsed := time.Since(startTime)
|
||||||
avgBytesPerSec := float64(totalBytes) / elapsed.Seconds()
|
avgBytesPerSec := float64(downloadedBytes) / elapsed.Seconds()
|
||||||
avgRate := formatBitrate(avgBytesPerSec * bitsPerByte)
|
avgRate := formatBitrate(avgBytesPerSec * bitsPerByte)
|
||||||
log.Infof("downloaded %d files (%s) in %.1fs (%s avg)",
|
log.Infof("downloaded %d files (%s) in %.1fs (%s avg), skipped %d already present",
|
||||||
len(files),
|
downloaded,
|
||||||
humanize.IBytes(safeUint64(totalBytes)),
|
humanize.IBytes(safeUint64(downloadedBytes)),
|
||||||
elapsed.Seconds(),
|
elapsed.Seconds(),
|
||||||
avgRate)
|
avgRate,
|
||||||
|
len(files)-downloaded)
|
||||||
|
log.Infof("saved manifest to %s", filepath.Join(dest, defaultManifestName))
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// fetchManifest downloads the manifest at manifestURL and parses it,
|
||||||
|
// enforcing --require-signature if it is given and refusing a manifest
|
||||||
|
// that lists a file where it will be saved. It returns the manifest as
|
||||||
|
// downloaded, to be saved once the files are in place, and the files it
|
||||||
|
// lists.
|
||||||
|
func fetchManifest(
|
||||||
|
ctx *cli.Context, client retryingClient, manifestURL string,
|
||||||
|
) ([]byte, []*mfer.MFFilePath, error) {
|
||||||
|
log.Infof("fetching manifest from %s", manifestURL)
|
||||||
|
|
||||||
|
// Read the whole manifest before parsing it, so that a connection
|
||||||
|
// lost partway through is retried rather than reported as a bad
|
||||||
|
// manifest.
|
||||||
|
var manifestData []byte
|
||||||
|
|
||||||
|
err := client.get(ctx.Context, manifestURL, func(resp *http.Response) error {
|
||||||
|
var readErr error
|
||||||
|
|
||||||
|
manifestData, readErr = io.ReadAll(resp.Body)
|
||||||
|
|
||||||
|
return readErr
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("failed to fetch manifest: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse manifest
|
||||||
|
manifest, err := mfer.NewManifestFromReader(bytes.NewReader(manifestData))
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("failed to parse manifest: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
requiredSigner := ctx.String(flagRequireSignature)
|
||||||
|
if requiredSigner != "" {
|
||||||
|
err = verifyFetchedSigner(ctx, manifestData, requiredSigner)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
files := manifest.Files()
|
||||||
|
|
||||||
|
err = checkManifestNameUnlisted(files)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Infof("manifest contains %d files", len(files))
|
||||||
|
|
||||||
|
return manifestData, files, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkManifestNameUnlisted returns an error if files lists a file or
|
||||||
|
// directory at the top of the tree under the name fetch saves the
|
||||||
|
// manifest as, or under that name's temp file. Saving the manifest would
|
||||||
|
// replace or remove it, or fail once every file was downloaded, leaving a
|
||||||
|
// tree check rejects. Names are compared ignoring case, since on a
|
||||||
|
// case-insensitive filesystem INDEX.MF and index.mf are one file.
|
||||||
|
func checkManifestNameUnlisted(files []*mfer.MFFilePath) error {
|
||||||
|
for _, f := range files {
|
||||||
|
top, _, _ := strings.Cut(filepath.Clean(f.GetPath()), string(filepath.Separator))
|
||||||
|
|
||||||
|
if strings.EqualFold(top, defaultManifestName) ||
|
||||||
|
strings.EqualFold(top, tempPathFor(defaultManifestName)) {
|
||||||
|
return fmt.Errorf("%w: %s", errManifestNameListed, f.GetPath())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// verifyFetchedSigner enforces --require-signature on the fetched manifest
|
||||||
|
// exactly as check does. verifyRequiredSigner takes a Checker, which loads
|
||||||
|
// its manifest from a file, so the manifest is handed to it as a file in
|
||||||
|
// memory.
|
||||||
|
func verifyFetchedSigner(
|
||||||
|
ctx *cli.Context, manifestData []byte, requiredSigner string,
|
||||||
|
) error {
|
||||||
|
memFs := afero.NewMemMapFs()
|
||||||
|
manifestPath := "/" + defaultManifestName
|
||||||
|
|
||||||
|
err := afero.WriteFile(memFs, manifestPath, manifestData, filePerms)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
chk, err := mfer.NewChecker(&mfer.CheckerOptions{
|
||||||
|
ManifestPath: manifestPath,
|
||||||
|
BasePath: "/",
|
||||||
|
Fs: memFs,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to load manifest: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return verifyRequiredSigner(ctx.Context, chk, requiredSigner)
|
||||||
|
}
|
||||||
|
|
||||||
|
// saveManifest writes the fetched manifest into dest under the default
|
||||||
|
// manifest name, the way fetch writes every file: to a new temp file that
|
||||||
|
// is then renamed into place.
|
||||||
|
func saveManifest(dest string, manifestData []byte) error {
|
||||||
|
tmpPath := tempPathFor(defaultManifestName)
|
||||||
|
|
||||||
|
out, err := createTempFile(dest, tmpPath)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
_, writeErr := out.Write(manifestData)
|
||||||
|
closeErr := out.Close()
|
||||||
|
|
||||||
|
err = errors.Join(writeErr, closeErr)
|
||||||
|
if err == nil {
|
||||||
|
err = moveIntoPlace(dest, tmpPath, defaultManifestName)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
_ = os.Remove(filepath.Join(dest, tmpPath))
|
||||||
|
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -401,14 +575,15 @@ func sanitizePath(p string) (string, error) {
|
|||||||
return cleaned, nil
|
return cleaned, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkNoSymlinks returns an error if any part of the relative path p
|
// checkNoSymlinks returns an error if any part of p, a path relative to
|
||||||
// already exists as a symlink. sanitizePath checks p only as text, so
|
// dest, already exists under dest as a symlink. dest itself is the user's
|
||||||
// without this a symlink inside the target directory could send a write
|
// choice and may be one. sanitizePath checks p only as text, so without
|
||||||
// to p outside of it. Parts that do not exist yet are fine: fetch creates
|
// this a symlink inside dest could send a write to p outside of it. Parts
|
||||||
// them as plain directories and files. Call it immediately before each
|
// that do not exist yet are fine: fetch creates them as plain directories
|
||||||
// write: a symlink created after it returns is not caught.
|
// and files. Call it immediately before each write: a symlink created
|
||||||
func checkNoSymlinks(p string) error {
|
// after it returns is not caught.
|
||||||
current := ""
|
func checkNoSymlinks(dest, p string) error {
|
||||||
|
current := dest
|
||||||
|
|
||||||
for _, part := range strings.Split(p, string(filepath.Separator)) {
|
for _, part := range strings.Split(p, string(filepath.Separator)) {
|
||||||
current = filepath.Join(current, part)
|
current = filepath.Join(current, part)
|
||||||
@@ -548,13 +723,14 @@ func verifyDownloadedHash(digest []byte, entry *mfer.MFFilePath) error {
|
|||||||
return errHashMismatch
|
return errHashMismatch
|
||||||
}
|
}
|
||||||
|
|
||||||
// downloadFile downloads a URL to a local file path with hash verification.
|
// downloadFile downloads a URL to localPath, a path relative to dest, with
|
||||||
// It downloads to a temporary file, verifies the hash, then renames to the final path.
|
// hash verification. It downloads to a temporary file, verifies the hash,
|
||||||
// Progress is reported via the progress channel.
|
// then renames to the final path. Progress is reported via the progress
|
||||||
|
// channel.
|
||||||
func downloadFile(
|
func downloadFile(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
client retryingClient,
|
client retryingClient,
|
||||||
fileURL, localPath string,
|
fileURL, dest, localPath string,
|
||||||
entry *mfer.MFFilePath,
|
entry *mfer.MFFilePath,
|
||||||
progress chan<- DownloadProgress,
|
progress chan<- DownloadProgress,
|
||||||
) error {
|
) error {
|
||||||
@@ -568,11 +744,13 @@ func downloadFile(
|
|||||||
// Create parent directories if needed
|
// Create parent directories if needed
|
||||||
dir := filepath.Dir(localPath)
|
dir := filepath.Dir(localPath)
|
||||||
if dir != "" && dir != "." {
|
if dir != "" && dir != "." {
|
||||||
err = checkNoSymlinks(dir)
|
err = checkNoSymlinks(dest, dir)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
dir = filepath.Join(dest, dir)
|
||||||
|
|
||||||
err = os.MkdirAll(dir, dirPerms)
|
err = os.MkdirAll(dir, dirPerms)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to create directory %s: %w", dir, err)
|
return fmt.Errorf("failed to create directory %s: %w", dir, err)
|
||||||
@@ -582,17 +760,61 @@ func downloadFile(
|
|||||||
tmpPath := tempPathFor(localPath)
|
tmpPath := tempPathFor(localPath)
|
||||||
|
|
||||||
return client.get(ctx, fileURL, func(resp *http.Response) error {
|
return client.get(ctx, fileURL, func(resp *http.Response) error {
|
||||||
return saveResponse(resp, tmpPath, localPath, entry, progress)
|
return saveResponse(resp, dest, tmpPath, localPath, entry, progress)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// createTempFile creates tmpPath, a path relative to dest, as a new empty
|
||||||
|
// file.
|
||||||
|
func createTempFile(dest, tmpPath string) (*os.File, error) {
|
||||||
|
err := checkNoSymlinks(dest, tmpPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
path := filepath.Join(dest, tmpPath)
|
||||||
|
|
||||||
|
// Remove whatever is at tmpPath, such as a leftover from an
|
||||||
|
// interrupted run, rather than write into it: it may be a hard link
|
||||||
|
// to a file outside dest, and removing a hard link removes only this
|
||||||
|
// name. If the removal fails, O_EXCL below makes the create fail.
|
||||||
|
_ = os.Remove(path)
|
||||||
|
|
||||||
|
// Create the temp file only if nothing is at tmpPath (O_EXCL).
|
||||||
|
//
|
||||||
|
// G304: tmpPath is a relative path that sanitizePath keeps inside dest
|
||||||
|
// as text, and checkNoSymlinks just found no symlink in it.
|
||||||
|
out, err := os.OpenFile( //nolint:gosec // G304: see comment above
|
||||||
|
path, os.O_RDWR|os.O_CREATE|os.O_EXCL, filePerms)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to create temp file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// moveIntoPlace renames tmpPath to localPath, both relative to dest.
|
||||||
|
func moveIntoPlace(dest, tmpPath, localPath string) error {
|
||||||
|
err := checkNoSymlinks(dest, localPath)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
err = os.Rename(filepath.Join(dest, tmpPath), filepath.Join(dest, localPath))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to rename temp file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// saveResponse writes resp's body to tmpPath, verifies it against entry,
|
// saveResponse writes resp's body to tmpPath, verifies it against entry,
|
||||||
// and renames it to localPath. It starts a new temp file each time and
|
// and renames it to localPath, both paths relative to dest. It starts a
|
||||||
// removes it on failure, so a retry after a failed try never appends to
|
// new temp file each time and removes it on failure, so a retry after a
|
||||||
// or keeps a partial file.
|
// failed try never appends to or keeps a partial file.
|
||||||
func saveResponse(
|
func saveResponse(
|
||||||
resp *http.Response,
|
resp *http.Response,
|
||||||
tmpPath, localPath string,
|
dest, tmpPath, localPath string,
|
||||||
entry *mfer.MFFilePath,
|
entry *mfer.MFFilePath,
|
||||||
progress chan<- DownloadProgress,
|
progress chan<- DownloadProgress,
|
||||||
) error {
|
) error {
|
||||||
@@ -604,29 +826,11 @@ func saveResponse(
|
|||||||
totalBytes = expectedSize
|
totalBytes = expectedSize
|
||||||
}
|
}
|
||||||
|
|
||||||
err := checkNoSymlinks(tmpPath)
|
out, err := createTempFile(dest, tmpPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Remove whatever is at tmpPath, such as a leftover from an
|
|
||||||
// interrupted run, rather than write into it: it may be a hard link
|
|
||||||
// to a file outside the target directory, and removing a hard link
|
|
||||||
// removes only this name. If the removal fails, O_EXCL below makes
|
|
||||||
// the create fail.
|
|
||||||
_ = os.Remove(tmpPath)
|
|
||||||
|
|
||||||
// Create the temp file only if nothing is at tmpPath (O_EXCL).
|
|
||||||
//
|
|
||||||
// G304: tmpPath is a relative path that sanitizePath keeps inside the
|
|
||||||
// target directory as text, and checkNoSymlinks just found no symlink
|
|
||||||
// in it.
|
|
||||||
out, err := os.OpenFile( //nolint:gosec // G304: see comment above
|
|
||||||
tmpPath, os.O_RDWR|os.O_CREATE|os.O_EXCL, filePerms)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to create temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Set up hash computation
|
// Set up hash computation
|
||||||
h := sha256.New()
|
h := sha256.New()
|
||||||
|
|
||||||
@@ -646,10 +850,10 @@ func saveResponse(
|
|||||||
closeErr := out.Close()
|
closeErr := out.Close()
|
||||||
|
|
||||||
err = finishDownload(
|
err = finishDownload(
|
||||||
tmpPath, localPath, written, expectedSize, h.Sum(nil), entry,
|
dest, tmpPath, localPath, written, expectedSize, h.Sum(nil), entry,
|
||||||
copyErr, closeErr)
|
copyErr, closeErr)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = os.Remove(tmpPath)
|
_ = os.Remove(filepath.Join(dest, tmpPath))
|
||||||
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -660,7 +864,7 @@ func saveResponse(
|
|||||||
// finishDownload validates the copy result, verifies size and hash, and
|
// finishDownload validates the copy result, verifies size and hash, and
|
||||||
// moves the temp file into place. On error the caller removes tmpPath.
|
// moves the temp file into place. On error the caller removes tmpPath.
|
||||||
func finishDownload(
|
func finishDownload(
|
||||||
tmpPath, localPath string,
|
dest, tmpPath, localPath string,
|
||||||
written, expectedSize int64,
|
written, expectedSize int64,
|
||||||
digest []byte,
|
digest []byte,
|
||||||
entry *mfer.MFFilePath,
|
entry *mfer.MFFilePath,
|
||||||
@@ -686,16 +890,5 @@ func finishDownload(
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
err = checkNoSymlinks(localPath)
|
return moveIntoPlace(dest, tmpPath, localPath)
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
// Rename temp file to final path
|
|
||||||
err = os.Rename(tmpPath, localPath)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to rename temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
|
|||||||
+292
-33
@@ -13,6 +13,7 @@ import (
|
|||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"sync"
|
"sync"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
@@ -338,7 +339,7 @@ func TestFetchFromHTTP(t *testing.T) {
|
|||||||
|
|
||||||
fileURL := baseURL + f.GetPath()
|
fileURL := baseURL + f.GetPath()
|
||||||
err = downloadFile(context.Background(), testClient(),
|
err = downloadFile(context.Background(), testClient(),
|
||||||
fileURL, localPath, f, progress)
|
fileURL, ".", localPath, f, progress)
|
||||||
require.NoError(t, err, "failed to download %s", f.GetPath())
|
require.NoError(t, err, "failed to download %s", f.GetPath())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -386,7 +387,7 @@ func TestFetchHashMismatch(t *testing.T) {
|
|||||||
|
|
||||||
// Try to download - should fail with hash mismatch
|
// Try to download - should fail with hash mismatch
|
||||||
err = downloadFile(context.Background(), testClient(),
|
err = downloadFile(context.Background(), testClient(),
|
||||||
server.URL+"/file.txt", testFileTxt, files[0], nil)
|
server.URL+"/file.txt", ".", testFileTxt, files[0], nil)
|
||||||
require.Error(t, err)
|
require.Error(t, err)
|
||||||
assert.Contains(t, err.Error(), "mismatch")
|
assert.Contains(t, err.Error(), "mismatch")
|
||||||
|
|
||||||
@@ -432,7 +433,7 @@ func TestFetchSizeMismatch(t *testing.T) {
|
|||||||
|
|
||||||
// Try to download - should fail with size mismatch
|
// Try to download - should fail with size mismatch
|
||||||
err = downloadFile(context.Background(), testClient(),
|
err = downloadFile(context.Background(), testClient(),
|
||||||
server.URL+"/file.txt", testFileTxt, files[0], nil)
|
server.URL+"/file.txt", ".", testFileTxt, files[0], nil)
|
||||||
require.Error(t, err)
|
require.Error(t, err)
|
||||||
assert.Contains(t, err.Error(), "size mismatch")
|
assert.Contains(t, err.Error(), "size mismatch")
|
||||||
|
|
||||||
@@ -490,7 +491,7 @@ func TestFetchProgress(t *testing.T) {
|
|||||||
|
|
||||||
// Download
|
// Download
|
||||||
err = downloadFile(context.Background(), testClient(),
|
err = downloadFile(context.Background(), testClient(),
|
||||||
server.URL+"/large.txt", "large.txt", files[0], progress)
|
server.URL+"/large.txt", ".", "large.txt", files[0], progress)
|
||||||
close(progress)
|
close(progress)
|
||||||
<-done
|
<-done
|
||||||
|
|
||||||
@@ -513,24 +514,51 @@ func TestFetchProgress(t *testing.T) {
|
|||||||
assert.Equal(t, content, downloaded)
|
assert.Equal(t, content, downloaded)
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestFetchRefusesSymlinks runs fetch into a destination directory that
|
// TestFetchRefusesSymlinks runs fetch with --dest naming a directory other
|
||||||
// holds a symlink pointing outside it, in each of the three places fetch
|
// than the current one, which holds a symlink pointing outside it, in each
|
||||||
// writes: a parent directory, the temp file, and the file itself, which
|
// place fetch writes: a parent directory, the temp file, the file itself,
|
||||||
// the temp file is renamed onto; and once as a directory inside a plain
|
// which the temp file is renamed onto, and the saved manifest's temp file
|
||||||
// directory. The fetch must fail and nothing outside may change.
|
// and final name; and once as a directory inside a plain directory. The
|
||||||
|
// fetch must fail, and neither the outside directory nor the current one
|
||||||
|
// may change.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // changes the process-global working directory
|
//nolint:paralleltest // changes the process-global working directory
|
||||||
func TestFetchRefusesSymlinks(t *testing.T) {
|
func TestFetchRefusesSymlinks(t *testing.T) {
|
||||||
|
// What a link standing for a file points to: a file outside that does
|
||||||
|
// not exist yet.
|
||||||
|
const newFile = "new.txt"
|
||||||
|
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
entry string // the manifest's only file
|
entry string // the manifest's only file
|
||||||
link string // symlink placed in the destination directory
|
link string // symlink placed in the destination directory
|
||||||
target string // what link points to, relative to the outside directory
|
target string // what link points to, relative to the outside directory
|
||||||
|
failure string // what fetch reports it was doing when it found link
|
||||||
}{
|
}{
|
||||||
{"parent directory", "sub/deeper/file.txt", "sub", "."},
|
{
|
||||||
{"directory inside a plain directory", "docs/data/passwd", "docs/data", "."},
|
"parent directory", "sub/deeper/file.txt", "sub", ".",
|
||||||
{"temp file", testFileTxt, ".file.txt.tmp", "new.txt"},
|
"failed to download sub/deeper/file.txt",
|
||||||
{"file", testFileTxt, testFileTxt, "new.txt"},
|
},
|
||||||
|
{
|
||||||
|
"directory inside a plain directory", "docs/data/passwd", "docs/data", ".",
|
||||||
|
"failed to download docs/data/passwd",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"temp file", testFileTxt, ".file.txt.tmp", newFile,
|
||||||
|
"failed to download " + testFileTxt,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file", testFileTxt, testFileTxt, newFile,
|
||||||
|
"failed to download " + testFileTxt,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"manifest temp file", testFileTxt, tempPathFor(defaultManifestName), newFile,
|
||||||
|
"failed to save manifest",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"manifest", testFileTxt, defaultManifestName, newFile,
|
||||||
|
"failed to save manifest",
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
@@ -545,23 +573,61 @@ func TestFetchRefusesSymlinks(t *testing.T) {
|
|||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
outside := t.TempDir()
|
outside := t.TempDir()
|
||||||
|
cwd := chdirTemp(t)
|
||||||
|
dest := t.TempDir()
|
||||||
|
link := filepath.Join(dest, tt.link)
|
||||||
|
|
||||||
chdirTemp(t)
|
require.NoError(t, os.MkdirAll(filepath.Dir(link), 0o750))
|
||||||
require.NoError(t, os.MkdirAll(filepath.Dir(tt.link), 0o750))
|
require.NoError(t, os.Symlink(filepath.Join(outside, tt.target), link))
|
||||||
require.NoError(t, os.Symlink(filepath.Join(outside, tt.target), tt.link))
|
|
||||||
|
|
||||||
opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
|
opts := testOpts([]string{
|
||||||
|
testApp, cmdFetch, "-q", "--" + flagDest, dest, server.URL,
|
||||||
|
}, afero.NewOsFs())
|
||||||
assert.Equal(t, 1, runCLI(opts))
|
assert.Equal(t, 1, runCLI(opts))
|
||||||
assert.Contains(t, testStderr(t, opts), "failed to download "+tt.entry+
|
assert.Contains(t, testStderr(t, opts),
|
||||||
": symlink in path not allowed: "+tt.link)
|
tt.failure+": symlink in path not allowed: "+link)
|
||||||
|
|
||||||
written, err := os.ReadDir(outside)
|
written, err := os.ReadDir(outside)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Empty(t, written, "fetch wrote outside the destination")
|
assert.Empty(t, written, "fetch wrote outside the destination")
|
||||||
|
|
||||||
|
written, err = os.ReadDir(cwd)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Empty(t, written, "fetch wrote to the current directory")
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestFetchDoesNotSkipThroughSymlink runs fetch with --dest holding a
|
||||||
|
// symlink to a directory outside it, where the file the manifest lists
|
||||||
|
// through that symlink already sits with the listed content. fetch must
|
||||||
|
// not take that file as already present: it must fail on the symlink, as
|
||||||
|
// the download would, and leave the outside file alone.
|
||||||
|
func TestFetchDoesNotSkipThroughSymlink(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
content := []byte("fetched")
|
||||||
|
files := map[string][]byte{"sub/" + testFileTxt: content}
|
||||||
|
|
||||||
|
server := httptest.NewServer(fetchTestHandler(manifestOf(t, files), files))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
outside := t.TempDir()
|
||||||
|
require.NoError(t, os.WriteFile(filepath.Join(outside, testFileTxt), content, 0o600))
|
||||||
|
|
||||||
|
dest := t.TempDir()
|
||||||
|
link := filepath.Join(dest, "sub")
|
||||||
|
require.NoError(t, os.Symlink(outside, link))
|
||||||
|
|
||||||
|
opts := testOpts([]string{
|
||||||
|
testApp, cmdFetch, "-q", "--" + flagDest, dest, server.URL,
|
||||||
|
}, afero.NewOsFs())
|
||||||
|
assert.Equal(t, 1, runCLI(opts))
|
||||||
|
assert.Contains(t, testStderr(t, opts),
|
||||||
|
"failed to download sub/"+testFileTxt+": symlink in path not allowed: "+link)
|
||||||
|
assert.Equal(t, map[string][]byte{testFileTxt: content}, filesUnder(t, outside))
|
||||||
|
}
|
||||||
|
|
||||||
// TestFetchReplacesHardLinkAtTempName runs fetch into a destination
|
// TestFetchReplacesHardLinkAtTempName runs fetch into a destination
|
||||||
// directory that holds, at the temp file's name, a hard link to a file
|
// directory that holds, at the temp file's name, a hard link to a file
|
||||||
// outside it. To fetch that is an ordinary leftover from an interrupted
|
// outside it. To fetch that is an ordinary leftover from an interrupted
|
||||||
@@ -787,7 +853,7 @@ func TestDownloadFileRetriesToSuccess(t *testing.T) {
|
|||||||
chdirTemp(t)
|
chdirTemp(t)
|
||||||
|
|
||||||
err = downloadFile(context.Background(), testClient(),
|
err = downloadFile(context.Background(), testClient(),
|
||||||
server.URL+"/"+testFileTxt, testFileTxt, manifest.Files()[0], nil)
|
server.URL+"/"+testFileTxt, ".", testFileTxt, manifest.Files()[0], nil)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Equal(t, int32(3), requests.Load())
|
assert.Equal(t, int32(3), requests.Load())
|
||||||
|
|
||||||
@@ -878,7 +944,7 @@ func TestFetchEscapedPaths(t *testing.T) {
|
|||||||
|
|
||||||
// TestFetchTree runs fetch on a tree with nested directories. Every file
|
// TestFetchTree runs fetch on a tree with nested directories. Every file
|
||||||
// the manifest lists must land under its own path with its own content,
|
// the manifest lists must land under its own path with its own content,
|
||||||
// and nothing else may be left in the destination.
|
// beside the manifest, and nothing else may be left in the destination.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // changes the process-global working directory
|
//nolint:paralleltest // changes the process-global working directory
|
||||||
func TestFetchTree(t *testing.T) {
|
func TestFetchTree(t *testing.T) {
|
||||||
@@ -889,7 +955,9 @@ func TestFetchTree(t *testing.T) {
|
|||||||
"other/deep/est.txt": []byte("in a second directory"),
|
"other/deep/est.txt": []byte("in a second directory"),
|
||||||
}
|
}
|
||||||
|
|
||||||
server := httptest.NewServer(fetchTestHandler(manifestOf(t, files), files))
|
manifest := manifestOf(t, files)
|
||||||
|
|
||||||
|
server := httptest.NewServer(fetchTestHandler(manifest, files))
|
||||||
defer server.Close()
|
defer server.Close()
|
||||||
|
|
||||||
dest := chdirTemp(t)
|
dest := chdirTemp(t)
|
||||||
@@ -897,7 +965,9 @@ func TestFetchTree(t *testing.T) {
|
|||||||
opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
|
opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
|
||||||
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
|
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
|
||||||
|
|
||||||
assert.Equal(t, files, filesUnder(t, dest))
|
want := maps.Clone(files)
|
||||||
|
want[defaultManifestName] = manifest
|
||||||
|
assert.Equal(t, want, filesUnder(t, dest))
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestFetchFailsOnHashMismatch runs fetch against a server that serves a
|
// TestFetchFailsOnHashMismatch runs fetch against a server that serves a
|
||||||
@@ -922,9 +992,10 @@ func TestFetchFailsOnHashMismatch(t *testing.T) {
|
|||||||
// TestFetchIntoPartlyFilledDestination runs fetch where an interrupted
|
// TestFetchIntoPartlyFilledDestination runs fetch where an interrupted
|
||||||
// fetch of an older version of the tree left one file current, one file
|
// fetch of an older version of the tree left one file current, one file
|
||||||
// out of date and one half written to its temp file, beside a file the
|
// out of date and one half written to its temp file, beside a file the
|
||||||
// manifest does not list. fetch downloads every file the manifest lists,
|
// manifest does not list. fetch skips the current file and downloads the
|
||||||
// those already present included, and replaces what is there; the file
|
// other two. The out-of-date file has the same size as the new version,
|
||||||
// the manifest does not list is left alone.
|
// so only its hash shows that it must be replaced. The file the manifest
|
||||||
|
// does not list is left alone, and the manifest is saved beside the files.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // changes the process-global working directory
|
//nolint:paralleltest // changes the process-global working directory
|
||||||
func TestFetchIntoPartlyFilledDestination(t *testing.T) {
|
func TestFetchIntoPartlyFilledDestination(t *testing.T) {
|
||||||
@@ -935,7 +1006,8 @@ func TestFetchIntoPartlyFilledDestination(t *testing.T) {
|
|||||||
}
|
}
|
||||||
unlisted := []byte("not in the manifest")
|
unlisted := []byte("not in the manifest")
|
||||||
|
|
||||||
tree := fetchTestHandler(manifestOf(t, files), files)
|
manifest := manifestOf(t, files)
|
||||||
|
tree := fetchTestHandler(manifest, files)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
@@ -963,21 +1035,208 @@ func TestFetchIntoPartlyFilledDestination(t *testing.T) {
|
|||||||
os.WriteFile(tempPathFor("sub/partial.txt"), []byte("cut off"), 0o600))
|
os.WriteFile(tempPathFor("sub/partial.txt"), []byte("cut off"), 0o600))
|
||||||
require.NoError(t, os.WriteFile("unlisted.txt", unlisted, 0o600))
|
require.NoError(t, os.WriteFile("unlisted.txt", unlisted, 0o600))
|
||||||
|
|
||||||
opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
|
opts := testOpts([]string{testApp, cmdFetch, server.URL}, afero.NewOsFs())
|
||||||
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
|
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
|
||||||
|
assert.Contains(t, testStderr(t, opts), "skipping current.txt: already present")
|
||||||
|
|
||||||
want := maps.Clone(files)
|
want := maps.Clone(files)
|
||||||
want["unlisted.txt"] = unlisted
|
want["unlisted.txt"] = unlisted
|
||||||
|
want[defaultManifestName] = manifest
|
||||||
assert.Equal(t, want, filesUnder(t, dest))
|
assert.Equal(t, want, filesUnder(t, dest))
|
||||||
|
|
||||||
mu.Lock()
|
mu.Lock()
|
||||||
defer mu.Unlock()
|
defer mu.Unlock()
|
||||||
|
|
||||||
assert.ElementsMatch(t, []string{
|
assert.ElementsMatch(t, []string{
|
||||||
"/" + defaultManifestName, "/current.txt", "/sub/changed.txt", "/sub/partial.txt",
|
"/" + defaultManifestName, "/sub/changed.txt", "/sub/partial.txt",
|
||||||
}, requested)
|
}, requested)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestFetchIntoDest fetches a tree with --dest into a directory that does
|
||||||
|
// not exist yet. The files and the manifest must land there and nowhere
|
||||||
|
// else, and check must pass on the result with no extra files. A second
|
||||||
|
// fetch into the same directory must download nothing but the manifest.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // changes the process-global working directory
|
||||||
|
func TestFetchIntoDest(t *testing.T) {
|
||||||
|
files := map[string][]byte{
|
||||||
|
"top.txt": []byte("at the top"),
|
||||||
|
"sub/one.txt": []byte("one level down"),
|
||||||
|
}
|
||||||
|
|
||||||
|
manifest := manifestOf(t, files)
|
||||||
|
tree := fetchTestHandler(manifest, files)
|
||||||
|
|
||||||
|
var (
|
||||||
|
mu sync.Mutex
|
||||||
|
requested []string
|
||||||
|
)
|
||||||
|
|
||||||
|
server := httptest.NewServer(
|
||||||
|
http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
mu.Lock()
|
||||||
|
|
||||||
|
requested = append(requested, r.URL.Path)
|
||||||
|
|
||||||
|
mu.Unlock()
|
||||||
|
|
||||||
|
tree.ServeHTTP(w, r)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
cwd := chdirTemp(t)
|
||||||
|
dest := filepath.Join(t.TempDir(), "mirror")
|
||||||
|
fetch := []string{testApp, cmdFetch, "-q", "--" + flagDest, dest, server.URL}
|
||||||
|
|
||||||
|
opts := testOpts(fetch, afero.NewOsFs())
|
||||||
|
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
|
||||||
|
|
||||||
|
want := maps.Clone(files)
|
||||||
|
want[defaultManifestName] = manifest
|
||||||
|
assert.Equal(t, want, filesUnder(t, dest))
|
||||||
|
assert.Empty(t, filesUnder(t, cwd), "fetch wrote outside --dest")
|
||||||
|
|
||||||
|
check := testOpts([]string{
|
||||||
|
testApp, cmdCheck, "-q", testFlagBase, dest, testFlagNoExtra,
|
||||||
|
filepath.Join(dest, defaultManifestName),
|
||||||
|
}, afero.NewOsFs())
|
||||||
|
require.Equal(t, 0, runCLI(check), testStderr(t, check))
|
||||||
|
|
||||||
|
mu.Lock()
|
||||||
|
requested = nil
|
||||||
|
mu.Unlock()
|
||||||
|
|
||||||
|
opts = testOpts(fetch, afero.NewOsFs())
|
||||||
|
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
|
||||||
|
assert.Equal(t, want, filesUnder(t, dest))
|
||||||
|
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
|
||||||
|
assert.Equal(t, []string{"/" + defaultManifestName}, requested)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFetchRequireSignature runs fetch with --require-signature. A
|
||||||
|
// manifest that is unsigned, or signed by another key, must stop fetch
|
||||||
|
// with check's message before it downloads or writes anything; the
|
||||||
|
// required key lets it through. The signed cases need gpg and are skipped
|
||||||
|
// without it, as the other signing tests are.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // signedManifest calls t.Setenv, which bars t.Parallel
|
||||||
|
func TestFetchRequireSignature(t *testing.T) {
|
||||||
|
files := map[string][]byte{testFileTxt: []byte("signed file")}
|
||||||
|
|
||||||
|
t.Run("unsigned", func(t *testing.T) {
|
||||||
|
assertFetchRefused(t, manifestOf(t, files), files,
|
||||||
|
"manifest is not signed, but signature from "+msgFpA+" is required",
|
||||||
|
"--"+flagRequireSignature, msgFpA)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("signed", func(t *testing.T) {
|
||||||
|
manifest := signedManifest(t, files)
|
||||||
|
|
||||||
|
signer, err := signedChecker(t, manifest).
|
||||||
|
ExtractEmbeddedSigningKeyFP(context.Background())
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
assertFetchRefused(t, manifest, files,
|
||||||
|
"embedded signing key fingerprint "+signer+" does not match required "+msgFpB,
|
||||||
|
"--"+flagRequireSignature, msgFpB)
|
||||||
|
|
||||||
|
server := httptest.NewServer(fetchTestHandler(manifest, files))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
dest := t.TempDir()
|
||||||
|
|
||||||
|
opts := testOpts([]string{
|
||||||
|
testApp, cmdFetch, "-q", "--" + flagDest, dest,
|
||||||
|
"--" + flagRequireSignature, signer, server.URL,
|
||||||
|
}, afero.NewOsFs())
|
||||||
|
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
|
||||||
|
assert.Equal(t, files[testFileTxt], filesUnder(t, dest)[testFileTxt])
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFetchRefusesListedManifestName fetches manifests that list, at the
|
||||||
|
// top of the tree, the name fetch saves the manifest under or that name's
|
||||||
|
// temp file: as a file, as a directory, in capitals, and with a leading
|
||||||
|
// "./". Saving the manifest would replace or remove what is listed there,
|
||||||
|
// so fetch must refuse the manifest before it creates the destination or
|
||||||
|
// requests any file.
|
||||||
|
func TestFetchRefusesListedManifestName(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, listed := range []string{
|
||||||
|
defaultManifestName,
|
||||||
|
tempPathFor(defaultManifestName),
|
||||||
|
defaultManifestName + "/" + testFileTxt,
|
||||||
|
"INDEX.MF",
|
||||||
|
"./" + defaultManifestName,
|
||||||
|
} {
|
||||||
|
t.Run(listed, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// Built directly rather than scanned, since a scan lists no
|
||||||
|
// hidden files and never a path starting with "./".
|
||||||
|
content := []byte("listed")
|
||||||
|
builder := mfer.NewBuilder()
|
||||||
|
_, err := builder.AddFile(mfer.RelFilePath(listed), mfer.FileSize(len(content)),
|
||||||
|
mfer.ModTime(time.Now()), bytes.NewReader(content), nil)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
var manifest bytes.Buffer
|
||||||
|
require.NoError(t, builder.Build(context.Background(), &manifest))
|
||||||
|
|
||||||
|
assertFetchRefused(t, manifest.Bytes(), map[string][]byte{listed: content},
|
||||||
|
"manifest lists a file where fetch saves the manifest: "+listed)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertFetchRefused serves manifest, a manifest of files, and fetches it
|
||||||
|
// with flags into a directory that does not exist yet. fetch must fail
|
||||||
|
// with message after requesting only the manifest, and must not create
|
||||||
|
// the directory.
|
||||||
|
func assertFetchRefused(
|
||||||
|
t *testing.T, manifest []byte, files map[string][]byte,
|
||||||
|
message string, flags ...string,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
tree := fetchTestHandler(manifest, files)
|
||||||
|
|
||||||
|
var (
|
||||||
|
mu sync.Mutex
|
||||||
|
requested []string
|
||||||
|
)
|
||||||
|
|
||||||
|
server := httptest.NewServer(
|
||||||
|
http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
mu.Lock()
|
||||||
|
|
||||||
|
requested = append(requested, r.URL.Path)
|
||||||
|
|
||||||
|
mu.Unlock()
|
||||||
|
|
||||||
|
tree.ServeHTTP(w, r)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
dest := filepath.Join(t.TempDir(), "mirror")
|
||||||
|
|
||||||
|
opts := testOpts(slices.Concat(
|
||||||
|
[]string{testApp, cmdFetch, "-q", "--" + flagDest, dest}, flags, []string{server.URL},
|
||||||
|
), afero.NewOsFs())
|
||||||
|
assert.Equal(t, 1, runCLI(opts))
|
||||||
|
assert.Contains(t, testStderr(t, opts), message)
|
||||||
|
assert.NoDirExists(t, dest, "fetch created the destination before refusing")
|
||||||
|
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
|
||||||
|
assert.Equal(t, []string{"/" + defaultManifestName}, requested)
|
||||||
|
}
|
||||||
|
|
||||||
// TestFetchTimeoutFlag runs fetch with --timeout against a server that
|
// TestFetchTimeoutFlag runs fetch with --timeout against a server that
|
||||||
// never answers. Without the flag's limit the request would wait forever;
|
// never answers. Without the flag's limit the request would wait forever;
|
||||||
// once fetch gives up on it, the server cancels fetch's context so that
|
// once fetch gives up on it, the server cancels fetch's context so that
|
||||||
|
|||||||
+24
-9
@@ -22,8 +22,10 @@ const (
|
|||||||
cmdFetch = "fetch"
|
cmdFetch = "fetch"
|
||||||
cmdVersion = "version"
|
cmdVersion = "version"
|
||||||
|
|
||||||
flagProgress = "progress"
|
flagProgress = "progress"
|
||||||
flagTimeout = "timeout"
|
flagTimeout = "timeout"
|
||||||
|
flagDest = "dest"
|
||||||
|
flagRequireSignature = "require-signature"
|
||||||
|
|
||||||
manifestArgsUsage = "[manifest file]"
|
manifestArgsUsage = "[manifest file]"
|
||||||
|
|
||||||
@@ -142,6 +144,17 @@ func commonFlags() []cli.Flag {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// requireSignatureFlag returns the --require-signature flag taken by the
|
||||||
|
// check and fetch subcommands.
|
||||||
|
func requireSignatureFlag() *cli.StringFlag {
|
||||||
|
return &cli.StringFlag{
|
||||||
|
Name: flagRequireSignature,
|
||||||
|
Aliases: []string{"S"},
|
||||||
|
Usage: "Require manifest to be signed by the specified GPG key ID",
|
||||||
|
EnvVars: []string{"MFER_REQUIRE_SIGNATURE"},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func (mfa *CLIApp) generateCommand() *cli.Command {
|
func (mfa *CLIApp) generateCommand() *cli.Command {
|
||||||
return &cli.Command{
|
return &cli.Command{
|
||||||
Name: cmdGenerate,
|
Name: cmdGenerate,
|
||||||
@@ -227,14 +240,9 @@ func (mfa *CLIApp) checkCommand() *cli.Command {
|
|||||||
},
|
},
|
||||||
&cli.BoolFlag{
|
&cli.BoolFlag{
|
||||||
Name: "no-extra-files",
|
Name: "no-extra-files",
|
||||||
Usage: "Fail if files exist in base directory that are not in manifest",
|
Usage: "Fail, instead of warning, if files in base directory are not in manifest",
|
||||||
},
|
|
||||||
&cli.StringFlag{
|
|
||||||
Name: "require-signature",
|
|
||||||
Aliases: []string{"S"},
|
|
||||||
Usage: "Require manifest to be signed by the specified GPG key ID",
|
|
||||||
EnvVars: []string{"MFER_REQUIRE_SIGNATURE"},
|
|
||||||
},
|
},
|
||||||
|
requireSignatureFlag(),
|
||||||
),
|
),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -354,6 +362,13 @@ func (mfa *CLIApp) fetchCommand() *cli.Command {
|
|||||||
Usage: "Time limit for each HTTP request, including the download " +
|
Usage: "Time limit for each HTTP request, including the download " +
|
||||||
"of its body",
|
"of its body",
|
||||||
},
|
},
|
||||||
|
&cli.StringFlag{
|
||||||
|
Name: flagDest,
|
||||||
|
Aliases: []string{"d"},
|
||||||
|
Value: ".",
|
||||||
|
Usage: "Directory to download the files and the manifest into",
|
||||||
|
},
|
||||||
|
requireSignatureFlag(),
|
||||||
),
|
),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+50
-35
@@ -77,9 +77,9 @@ type Checker struct {
|
|||||||
fs afero.Fs
|
fs afero.Fs
|
||||||
// manifestPaths is a set of paths in the manifest for quick lookup
|
// manifestPaths is a set of paths in the manifest for quick lookup
|
||||||
manifestPaths map[RelFilePath]struct{}
|
manifestPaths map[RelFilePath]struct{}
|
||||||
// manifestRelPath is the relative path of the manifest file from
|
// manifestInfo is the manifest file, which FindExtraFiles leaves out,
|
||||||
// basePath (for exclusion)
|
// matched with os.SameFile.
|
||||||
manifestRelPath RelFilePath
|
manifestInfo os.FileInfo
|
||||||
// signature info from the manifest
|
// signature info from the manifest
|
||||||
signature []byte
|
signature []byte
|
||||||
signer []byte
|
signer []byte
|
||||||
@@ -133,26 +133,20 @@ func NewChecker(opts *CheckerOptions) (*Checker, error) {
|
|||||||
manifestPaths[RelFilePath(f.GetPath())] = struct{}{}
|
manifestPaths[RelFilePath(f.GetPath())] = struct{}{}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Compute manifest's relative path from basePath for exclusion in FindExtraFiles
|
manifestInfo, err := fs.Stat(opts.ManifestPath)
|
||||||
absManifest, err := filepath.Abs(opts.ManifestPath)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
manifestRel, err := filepath.Rel(abs, absManifest)
|
|
||||||
if err != nil {
|
|
||||||
manifestRel = ""
|
|
||||||
}
|
|
||||||
|
|
||||||
return &Checker{
|
return &Checker{
|
||||||
basePath: AbsFilePath(abs),
|
basePath: AbsFilePath(abs),
|
||||||
files: files,
|
files: files,
|
||||||
fs: fs,
|
fs: fs,
|
||||||
manifestPaths: manifestPaths,
|
manifestPaths: manifestPaths,
|
||||||
manifestRelPath: RelFilePath(manifestRel),
|
manifestInfo: manifestInfo,
|
||||||
signature: m.pbOuter.GetSignature(),
|
signature: m.pbOuter.GetSignature(),
|
||||||
signer: m.pbOuter.GetSigner(),
|
signer: m.pbOuter.GetSigner(),
|
||||||
signingPubKey: m.pbOuter.GetSigningPubKey(),
|
signingPubKey: m.pbOuter.GetSigningPubKey(),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -273,20 +267,27 @@ func (c *Checker) Check(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// FindExtraFiles walks the filesystem and reports files not in the manifest.
|
// FindExtraFiles walks the filesystem and reports files not in the manifest,
|
||||||
// Results are sent to the results channel. The channel is closed when done.
|
// hidden files and directories included. The manifest file itself is not
|
||||||
// Hidden files/directories (starting with .) are skipped, as they are excluded
|
// reported. Anything the search cannot read, such as a directory that cannot
|
||||||
// from manifests by default. The manifest file itself is also skipped.
|
// be listed, is reported with StatusError and the search goes on. Results are
|
||||||
|
// sent to the results channel. The channel is closed when done.
|
||||||
func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) error {
|
func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) error {
|
||||||
if results != nil {
|
if results != nil {
|
||||||
defer close(results)
|
defer close(results)
|
||||||
}
|
}
|
||||||
|
|
||||||
walkFn := func(walkPath string, info os.FileInfo, err error) error {
|
// The search does not follow symlinks, so a base directory named
|
||||||
if err != nil {
|
// through one is resolved first. If that fails, the base is searched as
|
||||||
return err
|
// named and the search reports the problem.
|
||||||
}
|
root := string(c.basePath)
|
||||||
|
|
||||||
|
resolved, err := filepath.EvalSymlinks(root)
|
||||||
|
if err == nil {
|
||||||
|
root = resolved
|
||||||
|
}
|
||||||
|
|
||||||
|
walkFn := func(walkPath string, info os.FileInfo, walkErr error) error {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return ctx.Err()
|
return ctx.Err()
|
||||||
@@ -294,15 +295,22 @@ func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) err
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Get relative path
|
// Get relative path
|
||||||
rel, err := filepath.Rel(string(c.basePath), walkPath)
|
rel, err := filepath.Rel(root, walkPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Skip hidden files and directories (dotfiles)
|
relPath := RelFilePath(rel)
|
||||||
if IsHiddenPath(filepath.ToSlash(rel)) {
|
|
||||||
if info.IsDir() {
|
// Report what cannot be read, such as a directory that cannot be
|
||||||
return filepath.SkipDir
|
// listed, and go on with the rest.
|
||||||
|
if walkErr != nil {
|
||||||
|
if results != nil {
|
||||||
|
results <- Result{
|
||||||
|
Path: relPath,
|
||||||
|
Status: StatusError,
|
||||||
|
Message: walkErr.Error(),
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
@@ -313,10 +321,17 @@ func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) err
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
relPath := RelFilePath(rel)
|
// A symlink is compared by what it points to, so a manifest reached
|
||||||
|
// through one is not reported either.
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
target, statErr := c.fs.Stat(walkPath)
|
||||||
|
if statErr == nil {
|
||||||
|
info = target
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Skip the manifest file itself
|
// Skip the manifest file itself, however its path is spelled
|
||||||
if relPath == c.manifestRelPath {
|
if os.SameFile(info, c.manifestInfo) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -334,7 +349,7 @@ func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) err
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
return afero.Walk(c.fs, string(c.basePath), walkFn)
|
return afero.Walk(c.fs, root, walkFn)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Checker) checkFile(entry *MFFilePath, checkedBytes *FileSize) Result {
|
func (c *Checker) checkFile(entry *MFFilePath, checkedBytes *FileSize) Result {
|
||||||
|
|||||||
+94
-97
@@ -21,8 +21,6 @@ const (
|
|||||||
testExistsFile = "exists.txt"
|
testExistsFile = "exists.txt"
|
||||||
testManifestPath = "/manifest.mf"
|
testManifestPath = "/manifest.mf"
|
||||||
testDataDir = "/data"
|
testDataDir = "/data"
|
||||||
// testDataManifestPath is a manifest kept inside the checked tree.
|
|
||||||
testDataManifestPath = testDataDir + "/index.mf"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestStatusString(t *testing.T) {
|
func TestStatusString(t *testing.T) {
|
||||||
@@ -459,50 +457,120 @@ func TestFindExtraFiles(t *testing.T) {
|
|||||||
assert.Equal(t, "not in manifest", extras[0].Message)
|
assert.Equal(t, "not in manifest", extras[0].Message)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFindExtraFilesSkipsManifestAndDotfiles(t *testing.T) {
|
// TestFindExtraFilesReportsHiddenFilesButNotManifest keeps the manifest
|
||||||
|
// inside the checked tree: hidden files and directories are reported, the
|
||||||
|
// manifest is not. The manifest is recognized by file identity, which needs
|
||||||
|
// the real filesystem.
|
||||||
|
func TestFindExtraFilesReportsHiddenFilesButNotManifest(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
fs := afero.NewMemMapFs()
|
dir := t.TempDir()
|
||||||
manifestFiles := map[string][]byte{
|
manifestPath := filepath.Join(dir, "index.mf")
|
||||||
testFile1: []byte("in manifest"),
|
|
||||||
}
|
fs := afero.NewOsFs()
|
||||||
createTestManifest(t, fs, testDataManifestPath, manifestFiles)
|
createTestManifest(t, fs, manifestPath, map[string][]byte{
|
||||||
createFilesOnDisk(t, fs, map[string][]byte{
|
|
||||||
testFile1: []byte("in manifest"),
|
testFile1: []byte("in manifest"),
|
||||||
})
|
})
|
||||||
// Create dotfile and manifest that should be skipped
|
|
||||||
require.NoError(t, afero.WriteFile(fs, "/data/.hidden", []byte("hidden"), 0o644))
|
unlisted := []RelFilePath{"extra.txt", ".hidden", ".git/config"}
|
||||||
require.NoError(t, afero.WriteFile(fs, "/data/.config/settings", []byte("cfg"), 0o644))
|
for _, p := range append([]RelFilePath{testFile1}, unlisted...) {
|
||||||
// Create a real extra file
|
path := filepath.Join(dir, string(p))
|
||||||
require.NoError(t, fs.MkdirAll(testDataDir, 0o755))
|
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o750))
|
||||||
require.NoError(t, afero.WriteFile(fs, "/data/extra.txt", []byte("extra"), 0o644))
|
require.NoError(t, afero.WriteFile(fs, path, []byte("x"), 0o600))
|
||||||
|
}
|
||||||
|
|
||||||
chk, err := NewChecker(&CheckerOptions{
|
chk, err := NewChecker(&CheckerOptions{
|
||||||
ManifestPath: testDataManifestPath,
|
ManifestPath: manifestPath,
|
||||||
BasePath: testDataDir,
|
BasePath: dir,
|
||||||
Fs: fs,
|
Fs: fs,
|
||||||
})
|
})
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
results := make(chan Result, 10)
|
results := make(chan Result, 10)
|
||||||
err = chk.FindExtraFiles(context.Background(), results)
|
require.NoError(t, chk.FindExtraFiles(context.Background(), results))
|
||||||
|
|
||||||
|
var extras []RelFilePath
|
||||||
|
for r := range results {
|
||||||
|
extras = append(extras, r.Path)
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.ElementsMatch(t, unlisted, extras)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFindExtraFilesSkipsManifestReachedThroughSymlink checks a tree whose
|
||||||
|
// index.mf is a symlink to the manifest kept outside the tree: the symlink is
|
||||||
|
// not reported.
|
||||||
|
func TestFindExtraFilesSkipsManifestReachedThroughSymlink(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
tree := filepath.Join(dir, "tree")
|
||||||
|
manifestPath := filepath.Join(dir, "real.mf")
|
||||||
|
linkPath := filepath.Join(tree, "index.mf")
|
||||||
|
|
||||||
|
fs := afero.NewOsFs()
|
||||||
|
createTestManifest(t, fs, manifestPath, map[string][]byte{testFile1: []byte("x")})
|
||||||
|
require.NoError(t, fs.MkdirAll(tree, 0o750))
|
||||||
|
require.NoError(t,
|
||||||
|
afero.WriteFile(fs, filepath.Join(tree, testFile1), []byte("x"), 0o600))
|
||||||
|
require.NoError(t, os.Symlink(manifestPath, linkPath))
|
||||||
|
|
||||||
|
chk, err := NewChecker(&CheckerOptions{
|
||||||
|
ManifestPath: linkPath,
|
||||||
|
BasePath: tree,
|
||||||
|
Fs: fs,
|
||||||
|
})
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
var extras []Result
|
results := make(chan Result, 10)
|
||||||
|
require.NoError(t, chk.FindExtraFiles(context.Background(), results))
|
||||||
|
|
||||||
|
var extras []RelFilePath
|
||||||
for r := range results {
|
for r := range results {
|
||||||
extras = append(extras, r)
|
extras = append(extras, r.Path)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Should only report extra.txt, not .hidden, .config/settings, or index.mf
|
assert.Empty(t, extras)
|
||||||
for _, e := range extras {
|
}
|
||||||
t.Logf("extra: %s", e.Path)
|
|
||||||
|
// TestFindExtraFilesSearchesBaseNamedThroughSymlink names the checked tree
|
||||||
|
// through a symlink to it: the files in the tree are searched, and the
|
||||||
|
// symlink itself is not reported.
|
||||||
|
func TestFindExtraFilesSearchesBaseNamedThroughSymlink(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
tree := filepath.Join(dir, "tree")
|
||||||
|
link := filepath.Join(dir, "link")
|
||||||
|
manifestPath := filepath.Join(dir, "index.mf")
|
||||||
|
|
||||||
|
fs := afero.NewOsFs()
|
||||||
|
createTestManifest(t, fs, manifestPath, map[string][]byte{testFile1: []byte("x")})
|
||||||
|
require.NoError(t, fs.MkdirAll(tree, 0o750))
|
||||||
|
|
||||||
|
for _, name := range []string{testFile1, testFile2} {
|
||||||
|
require.NoError(t,
|
||||||
|
afero.WriteFile(fs, filepath.Join(tree, name), []byte("x"), 0o600))
|
||||||
}
|
}
|
||||||
|
|
||||||
assert.Len(t, extras, 1)
|
require.NoError(t, os.Symlink(tree, link))
|
||||||
|
|
||||||
if len(extras) > 0 {
|
chk, err := NewChecker(&CheckerOptions{
|
||||||
assert.Equal(t, RelFilePath("extra.txt"), extras[0].Path)
|
ManifestPath: manifestPath,
|
||||||
|
BasePath: link,
|
||||||
|
Fs: fs,
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
results := make(chan Result, 10)
|
||||||
|
require.NoError(t, chk.FindExtraFiles(context.Background(), results))
|
||||||
|
|
||||||
|
var extras []RelFilePath
|
||||||
|
for r := range results {
|
||||||
|
extras = append(extras, r.Path)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
assert.Equal(t, []RelFilePath{testFile2}, extras)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFindExtraFilesContextCancellation(t *testing.T) {
|
func TestFindExtraFilesContextCancellation(t *testing.T) {
|
||||||
@@ -649,77 +717,6 @@ func TestCheckMissingFileDetectedWithoutFallback(t *testing.T) {
|
|||||||
assert.Equal(t, 0, statusCounts[StatusError], "no files should be ERROR")
|
assert.Equal(t, 0, statusCounts[StatusError], "no files should be ERROR")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFindExtraFilesSkipsDotfiles(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// Regression test for #16: FindExtraFiles should not report dotfiles
|
|
||||||
// or the manifest file itself as extra files.
|
|
||||||
fs := afero.NewMemMapFs()
|
|
||||||
files := map[string][]byte{
|
|
||||||
testFile1: []byte("in manifest"),
|
|
||||||
}
|
|
||||||
createTestManifest(t, fs, testDataManifestPath, files)
|
|
||||||
createFilesOnDisk(t, fs, files)
|
|
||||||
|
|
||||||
// Add dotfiles and manifest file on disk
|
|
||||||
require.NoError(t, afero.WriteFile(fs, "/data/.hidden", []byte("dotfile"), 0o644))
|
|
||||||
require.NoError(t, fs.MkdirAll("/data/.git", 0o755))
|
|
||||||
require.NoError(t,
|
|
||||||
afero.WriteFile(fs, "/data/.git/config", []byte("git config"), 0o644))
|
|
||||||
|
|
||||||
chk, err := NewChecker(&CheckerOptions{
|
|
||||||
ManifestPath: testDataManifestPath,
|
|
||||||
BasePath: testDataDir,
|
|
||||||
Fs: fs,
|
|
||||||
})
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
results := make(chan Result, 10)
|
|
||||||
err = chk.FindExtraFiles(context.Background(), results)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
var extras []Result
|
|
||||||
for r := range results {
|
|
||||||
extras = append(extras, r)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Should report NO extra files — dotfiles and manifest should be skipped
|
|
||||||
assert.Empty(t, extras,
|
|
||||||
"FindExtraFiles should not report dotfiles or manifest file as extra; got: %v",
|
|
||||||
extras)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFindExtraFilesSkipsManifestFile(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// The manifest file itself should never be reported as extra
|
|
||||||
fs := afero.NewMemMapFs()
|
|
||||||
files := map[string][]byte{
|
|
||||||
testFile1: []byte("content"),
|
|
||||||
}
|
|
||||||
createTestManifest(t, fs, testDataManifestPath, files)
|
|
||||||
createFilesOnDisk(t, fs, files)
|
|
||||||
|
|
||||||
chk, err := NewChecker(&CheckerOptions{
|
|
||||||
ManifestPath: testDataManifestPath,
|
|
||||||
BasePath: testDataDir,
|
|
||||||
Fs: fs,
|
|
||||||
})
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
results := make(chan Result, 10)
|
|
||||||
err = chk.FindExtraFiles(context.Background(), results)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
var extras []Result
|
|
||||||
for r := range results {
|
|
||||||
extras = append(extras, r)
|
|
||||||
}
|
|
||||||
|
|
||||||
assert.Empty(t, extras,
|
|
||||||
"manifest file should not be reported as extra; got: %v", extras)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCheckEmptyManifest(t *testing.T) {
|
func TestCheckEmptyManifest(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
+3
-2
@@ -7,8 +7,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
gofumpt -l -w mfer internal cmd
|
# Go, then Markdown and JSON, through the same scripts script/fmt-check
|
||||||
# Markdown and JSON, over the same file set script/fmt-check verifies.
|
# uses, so both see the same files and the same tool versions.
|
||||||
|
"$SCRIPT_DIR/gofumpt" --write
|
||||||
"$SCRIPT_DIR/prettier" --write
|
"$SCRIPT_DIR/prettier" --write
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -1,13 +1,13 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/fmt-check: check formatting (read-only). Same scope as
|
# script/fmt-check: check formatting (read-only). Same scope as
|
||||||
# script/fmt, but fails instead of writing: Go via script/fmt-check-go,
|
# script/fmt, but fails instead of writing: Go via script/gofumpt,
|
||||||
# Markdown and JSON via script/prettier.
|
# Markdown and JSON via script/prettier.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
"$SCRIPT_DIR/fmt-check-go"
|
"$SCRIPT_DIR/gofumpt" --check
|
||||||
"$SCRIPT_DIR/prettier" --check
|
"$SCRIPT_DIR/prettier" --check
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,18 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
# script/fmt-check-go: check Go formatting (read-only). Split out from
|
|
||||||
# script/fmt-check so the Docker lint stage, whose image has no node and
|
|
||||||
# therefore no prettier, can run the Go half on its own.
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|
||||||
|
|
||||||
main() {
|
|
||||||
cd "$ROOT"
|
|
||||||
if [ -n "$(gofmt -l .)" ]; then
|
|
||||||
echo "gofmt: files need formatting:" >&2
|
|
||||||
gofmt -l . >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
Executable
+44
@@ -0,0 +1,44 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# script/gofumpt: run gofumpt over this repo's Go files.
|
||||||
|
#
|
||||||
|
# Takes exactly one mode argument, --write or --check, and runs the same
|
||||||
|
# gofumpt version over the same files in both modes. script/fmt and
|
||||||
|
# script/fmt-check both go through here, and so does the Docker lint
|
||||||
|
# stage (make fmt-check-go), so what gets formatted and what gets
|
||||||
|
# verified cannot drift apart.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
|
# gofumpt v0.12.0, 2026-10-04. `go run` fetches and builds exactly this
|
||||||
|
# version, so neither a developer machine nor the lint image needs
|
||||||
|
# gofumpt installed.
|
||||||
|
GOFUMPT="mvdan.cc/gofumpt@v0.12.0"
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
echo "usage: script/gofumpt --write|--check" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
[ "$#" -eq 1 ] || usage
|
||||||
|
cd "$ROOT"
|
||||||
|
# Every Go file in the repo. gofumpt holds generated files, such as
|
||||||
|
# mfer/mf.pb.go, to gofmt's rules only.
|
||||||
|
case "$1" in
|
||||||
|
--write) go run "$GOFUMPT" -l -w . ;;
|
||||||
|
--check)
|
||||||
|
# Own line: a failing command inside `[ -n "$(...)" ]` does
|
||||||
|
# not trip `set -e`, so a gofumpt that never ran would pass.
|
||||||
|
unformatted="$(go run "$GOFUMPT" -l .)"
|
||||||
|
if [ -n "$unformatted" ]; then
|
||||||
|
echo "gofumpt: files need formatting (run make fmt):" >&2
|
||||||
|
echo "$unformatted" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
*) usage ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
Reference in New Issue
Block a user