Compare commits

..
Author SHA1 Message Date
clawbot d2728b1cea copy files off a card, verified by SHA256 (closes #5)
Store.Import copies every member of every file group and the images
and videos under PRIVATE/M4ROOT to dest/YYYY-MM-DD/<file name>. The day
comes from the new Object.ModTime. A file group stays in the folder of
its primary member.

Each copy is read back and compared by SHA256. A file with the same
hash already there is skipped, and anything else there is a conflict
and left alone. One failed file does not stop the others. A cancelled
context stops at once and removes the partial copy.

Object.Hash now shares its reading code with the import.

Model: opus-5-5
2026-11-03 20:06:00 +00:00
clawbot acb108a82e get cards with GetStores and GetStore, read them through fs.FS (closes #4)
GetStores replaces GetDCFStores: it takes a context, has no count
argument and returns a plain slice. GetStore returns the one mounted
card, or ErrNoStore or ErrMultipleStores.

Each Object keeps the fs.FS its card was read through, and Hash reads
through it, so tests can use fstest.MapFS. A cancelled context stops
the read before the next directory. Library errors wrap with %w.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
2026-11-03 20:54:19 +01:00
clawbot cf7611ea1b group DCF files that share a file number (closes #3)
Files in one DCF directory that share a file number are one file group,
as section 4.3.2.2 of the specification says. `Directory.FileGroups`
lists them, each with its numbers, its members and its primary member. A
lone file is a group of one. A group reports its member count and total
size.

Section 6.2.1.3 gives a DCF basic file priority over a DCF extended image
file. It says nothing about a DCF optional file, so ranking every `JPG`
first is a choice. A `THM` comes last.

`isImageFile` now accepts `.tif`, so a `TIF` counts as an image.
`Store.Images`, `Store.Videos` and the totals still count each file on
its own.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
2026-11-03 20:05:44 +01:00
clawbot 2c61b85a53 parse the DCF structure under DCIM (closes #2)
A card is now read by the naming rules in CIPA DC-009-2010, not by
looking for media extensions anywhere on the volume.

Store.Directories holds the DCF directories and their objects, with
directory and file numbers. Store.NonConforming lists everything else
under DCIM, including symlinks and names fs.FS refuses. Media under
PRIVATE/M4ROOT stays, with no DCF numbers. Media anywhere else is no
longer part of the store. A symlink is never followed, and a read error
names the card's mount point.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
2026-11-03 19:25:32 +01:00
clawbot 445424ba3e detect PRIVATE/M4ROOT and return the requested store count (closes #6)
A card was detected only with M4ROOT at its root, and asking for two or
more stores returned one too few. A mount point now counts when its root
holds a DCIM or PRIVATE/M4ROOT directory, and the count is exact.

The check and the counting are split from the partition listing so tests
can hand them temporary directories.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
2026-11-02 21:01:29 +01:00
clawbot 623bfe9a43 bring the repo up to repo standards (closes #1)
Adopt the standard tooling from sneak/prompts: the Makefile and script/
entrypoints, lint and test stages in one Dockerfile, the lint config, the
CI workflow and the formatter settings. make fmt also formats Go, with a
pinned Go from script/bootstrap. The module path becomes
sneak.berlin/go/dcf.

Fix the lint findings in pkg/dcf. DCFStore becomes Store and DCFObject
becomes Object. Three behaviours change. An error listing mount points is
now returned, where a typo used to drop it. Store.String() prints Store{
instead of DCFStore{, and a negative total as 0 B instead of a huge size.

Replace the placeholder test with real tests. Move the README into the
required sections. make build compiles the library; it pointed at a
command that does not exist.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
2026-11-02 20:30:17 +01:00
17 changed files with 3031 additions and 170 deletions

No files matched your search

+12 -14
View File
@@ -1,10 +1,7 @@
# Lint phase. The tools are invoked directly rather than through `make # Lint phase. The linter is invoked directly rather than through `make
# lint` or `script/lint`, which are themselves a docker build and would # lint` or `script/lint`, which are themselves a docker build and would
# recurse into a daemon that does not exist in a build step. # recurse into a daemon that does not exist in a build step.
# #
# golangci-lint does not check formatting under the canonical config, so
# gofmt runs here too.
#
# golangci/golangci-lint:v2.12.2, 2026-11-02 # golangci/golangci-lint:v2.12.2, 2026-11-02
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
@@ -15,14 +12,15 @@ RUN go mod download
COPY . . COPY . .
RUN test -z "$(gofmt -l .)" || { echo "gofmt needed on:"; gofmt -l .; exit 1; }
RUN golangci-lint run --config .golangci.yml ./... RUN golangci-lint run --config .golangci.yml ./...
# Test phase, invoked directly for the same reason. The Debian image is # Test phase, invoked directly for the same reason. The Debian image has
# used because the race detector needs glibc. # the C compiler the race detector needs.
# #
# golang:1.26.2-trixie, 2026-11-02 # golang:1.27.3-trixie, 2026-11-02. It carries the Go that
FROM golang@sha256:4a7137ea573f79c86ae451ff05817ed762ef5597fcf732259e97abeb3108d873 AS test # script/bootstrap installs on the host; change both together, and the
# same image in the last stage.
FROM golang@sha256:c65ecfe613b539ac74c62a4c3b2c27e8f8a6076384c8f27e8a3c0f3be17cc385 AS test
WORKDIR /src WORKDIR /src
@@ -41,20 +39,20 @@ RUN go test -count=1 -timeout 90s -race -cover ./... || \
# this image cannot be produced unless lint and test passed. A stage # this image cannot be produced unless lint and test passed. A stage
# appended after this one would drop all three out of a plain build. # appended after this one would drop all three out of a plain build.
# #
# golang:1.26.2-trixie, 2026-11-02 # golang:1.27.3-trixie, 2026-11-02
FROM golang@sha256:4a7137ea573f79c86ae451ff05817ed762ef5597fcf732259e97abeb3108d873 FROM golang@sha256:c65ecfe613b539ac74c62a4c3b2c27e8f8a6076384c8f27e8a3c0f3be17cc385
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
WORKDIR /src WORKDIR /src
# script/bootstrap installs the tools for formatting Markdown. Debian is # script/bootstrap finds its pinned Go already in this image. It installs
# used here because the node it installs needs glibc. # the Go modules and the tools for formatting Markdown. Debian is used
# here because the node it installs needs glibc.
COPY script/ script/ COPY script/ script/
COPY package.json yarn.lock go.mod go.sum ./ COPY package.json yarn.lock go.mod go.sum ./
RUN script/bootstrap RUN script/bootstrap
RUN go mod download
COPY . . COPY . .
+172 -30
View File
@@ -3,7 +3,7 @@
## Description ## Description
`dcf` is a WTFPL-licensed Go library by [@sneak](https://sneak.berlin) that `dcf` is a WTFPL-licensed Go library by [@sneak](https://sneak.berlin) that
reads DCF memory cards from digital cameras. reads DCF memory cards from digital cameras and copies the files off them.
DCF is the "Design rule for Camera File system". It is also JEITA specification DCF is the "Design rule for Camera File system". It is also JEITA specification
number CP-3461. JEITA is the Japan Electronics and Information Technology number CP-3461. JEITA is the Japan Electronics and Information Technology
@@ -29,6 +29,7 @@ Then list the DCF stores on the mounted filesystems:
package main package main
import ( import (
"context"
"fmt" "fmt"
"log" "log"
@@ -36,17 +37,53 @@ import (
) )
func main() { func main() {
stores, err := dcf.GetDCFStores(0) stores, err := dcf.GetStores(context.Background())
if err != nil { if err != nil {
log.Fatal(err) log.Fatal(err)
} }
for _, store := range *stores { for _, store := range stores {
fmt.Println(store.String()) fmt.Println(store.String())
} }
} }
``` ```
Call `dcf.GetStore` instead when you expect exactly one DCF store. It returns
`dcf.ErrNoStore` when none is mounted, and `dcf.ErrMultipleStores` when more
than one is.
To copy the files off a card, call `Import` with a destination directory:
```go
package main
import (
"context"
"fmt"
"log"
"sneak.berlin/go/dcf/pkg/dcf"
)
func main() {
ctx := context.Background()
store, err := dcf.GetStore(ctx)
if err != nil {
log.Fatal(err)
}
result, err := store.Import(ctx, "/path/to/photos")
if err != nil {
log.Fatal(err)
}
fmt.Println("copied:", result.Copied)
fmt.Println("skipped:", result.Skipped)
fmt.Println("conflicts:", result.Conflicts)
}
```
To work on the library itself, you need docker and git: To work on the library itself, you need docker and git:
```sh ```sh
@@ -64,20 +101,27 @@ standard. Each entrypoint is a POSIX sh script in `script/`. Use these instead
of running the underlying tools directly. of running the underlying tools directly.
- `script/bootstrap` (`make bootstrap`) installs the tools the host needs. It - `script/bootstrap` (`make bootstrap`) installs the tools the host needs. It
does not install Go or the linter. does not install docker or the linter. It installs:
- `make` and git
- Go into `~/.local/go`, at the version in the `Dockerfile`, unless the `go`
on the `PATH` is already that version
- the Go modules
- node and yarn
- `prettier`
- `script/setup` (`make setup`) runs `script/bootstrap` and then installs the - `script/setup` (`make setup`) runs `script/bootstrap` and then installs the
git pre-commit hook. git pre-commit hook.
- `script/test` (`make test`) runs the tests by building the `test` stage of the - `script/test` (`make test`) runs the tests by building the `test` stage of the
`Dockerfile`. `Dockerfile`.
- `script/lint` (`make lint`) checks the Go code by building the `lint` stage of - `script/lint` (`make lint`) checks the Go code by building the `lint` stage of
the `Dockerfile`. The stage fails on Go code that `gofmt` would change. the `Dockerfile`.
- `script/fmt` (`make fmt`) rewrites the Markdown files with `prettier`. It does - `script/fmt` (`make fmt`) rewrites the Go code with `go fmt` and the Markdown
not format Go code. files with `prettier`.
- `script/fmt-check` (`make fmt-check`) checks the Markdown formatting and - `script/fmt-check` (`make fmt-check`) checks the same formatting and changes
changes nothing. nothing.
- `script/check` (`make check`) runs the tests, then the linter, then the - `script/check` (`make check`) runs the tests, then the linter, then the
Markdown check. formatting check.
- `script/build` (`make build`) compiles every package. It needs Go on the host. - `script/build` (`make build`) compiles every package. It needs the Go that
`script/bootstrap` installs.
- `script/docker` (`make docker`) builds the development image, tagged `dcf`. - `script/docker` (`make docker`) builds the development image, tagged `dcf`.
- `script/cibuild` runs the whole CI build. The Gitea workflow runs it on every - `script/cibuild` runs the whole CI build. The Gitea workflow runs it on every
push. push.
@@ -98,35 +142,133 @@ overengineering and reusable code.
All the code is in the `pkg/dcf` package. The repo has no commands. All the code is in the `pkg/dcf` package. The repo has no commands.
- `dcf.go` holds the public API. `GetDCFStores` finds the DCF stores and lists - `dcf.go` holds the public API for reading a card. `GetStores` finds the DCF
the files on them. stores and reads each one. `GetStore` returns the one DCF store when exactly
one is mounted.
- `helpers.go` finds the mount points that hold a DCF store. - `helpers.go` finds the mount points that hold a DCF store.
- `parse.go` reads one DCF store through an `fs.FS` rooted at its mount point.
- `filegroup.go` sorts the files of one DCF directory into file groups.
- `import.go` holds `Store.Import`, which copies files off a card.
The main types are: The main types are:
- `Store` is one DCF store on one mounted filesystem. - `Store` is one DCF store on one mounted filesystem.
- `Object` is one file in a store. - `Directory` is one DCF directory. It has a directory number, a name, the
objects inside it and their file groups.
- `Object` is one file in a store. A file in a DCF directory also has a
directory number and a file number. `Object.ModTime` is its modification time.
`Object.Hash` reads the file through the `fs.FS` of its store. An `Object` you
build yourself has none, so its `Hash` returns an error.
- `FileGroup` is one file group in a DCF directory.
- `Image` and `Video` embed `Object`. The file extension decides which one a - `Image` and `Video` embed `Object`. The file extension decides which one a
file becomes. file becomes. A file with any other extension, such as a `THM` thumbnail, is
neither.
- `ImportResult` lists what `Store.Import` did with each file.
A mounted filesystem is a DCF store when its root holds a `DCIM` or `M4ROOT` A mounted filesystem is a DCF store when its root holds a `DCIM` or
directory. `dcf` never mounts a filesystem. `PRIVATE/M4ROOT` directory. It needs only one of them. `dcf` never mounts a
filesystem. Media anywhere else on the filesystem is not part of the store.
`dcf` never follows a symlink.
`GetStores` and `GetStore` take a `context.Context`. When it is cancelled, the
read stops before the next directory. The returned error matches
`context.Canceled`.
`dcf` applies the naming rules of the DCF specification, CIPA DC-009-2010, under
`DCIM`:
- A DCF directory sits directly under `DCIM`. Its name is a directory number
from 100 to 999, then five characters, such as `100MSDCF`.
- A DCF file sits directly in a DCF directory. Its name is four characters, then
a file number from 0001 to 9999, then a dot and a three-character extension,
such as `DSC00001.ARW`.
- The characters allowed are `A` to `Z`, `0` to `9` and `_`. Lower-case letters
count as upper-case.
- Directories that share a directory number are not DCF directories.
- `JPG` files that share a file number in one directory are not DCF files.
`Store.NonConforming` lists the path of everything else under `DCIM`. It
includes everything inside a directory that breaks the rules. It also includes
every symlink, whatever its name. These entries have no DCF numbers, and they
are not counted as images or videos.
The naming rules do not apply under `PRIVATE/M4ROOT`. The images and videos
there are in `Store.Images` and `Store.Videos`, and their DCF numbers are 0.
`dcf` skips any symlink there.
A file group is made of the DCF files in one DCF directory that share a file
number. The specification calls it a DCF object. A RAW file and its JPEG form a
file group, and so do a video and its `THM` thumbnail.
- The first four characters and the extension can differ within a group.
- A DCF file that shares its number with no other file is a group of one.
- Files in different DCF directories are never in the same group.
- Only DCF files are grouped. Files in `Store.NonConforming` and under
`PRIVATE/M4ROOT` are in no group.
`Directory.FileGroups` lists the groups in file number order. Each `FileGroup`
has:
- `DirectoryNumber` and `FileNumber`
- `Members`, which is every file in the group
- `Primary`, which is the member that stands for the whole group
- `MembersCount`, which returns the number of members
- `TotalSize`, which returns the total size of the members in bytes
The primary member is the first of these that the group holds:
1. a `JPG` file
2. any other file in `Store.Images`
3. a file in `Store.Videos`
4. any other file except a `THM`
5. a `THM`
`Store.Images` and `Store.Videos` list each file on its own, whatever its group.
`Store.Import` copies files off a card to a destination directory. It verifies
each copy against the SHA256 hash of its source. It only reads the card, so it
never deletes anything there.
It copies:
- every member of every file group
- the images and videos under `PRIVATE/M4ROOT`
It does not copy the files in `Store.NonConforming`.
Each file goes to `YYYY-MM-DD/<file name>` under the destination directory. The
date is the day of the file's modification time, in the computer's time zone.
Every member of a file group goes to the day of its primary member, so a group
never splits across two folders. Each copy keeps the modification time of its
source. Only the user who ran the import can read the copies.
`ImportResult` lists each destination path in one of three lists:
- `Copied` holds the copies this import wrote and verified.
- `Skipped` holds the paths that already held a file with the same SHA256 hash.
- `Conflicts` holds the paths that already held anything else. `dcf` leaves that
alone and does not copy the file.
A conflict is not an error, so a caller must check `Conflicts`. These are the
usual causes:
- Two different files have the same name and the same day. Two cards can reuse a
file number, and so can two DCF directories on one card. The first file is
copied, and the second is a conflict.
- A partial file remains because the process died during a copy. Delete it and
import again.
A copy that fails or does not match the hash of its source is removed. One file
that fails does not stop the others. The returned error joins the failures, each
naming its file. The result still lists what was done.
When the context is cancelled, the import stops at once, even inside a file, and
removes the partial copy. The returned error matches `context.Canceled`.
## TODO ## TODO
- Handle DCF file groups, where multiple objects share the same index number. No open items.
`dcf` does not handle these yet. If you need this, send me a link to a zip or
tar of a memory card that uses them, and I'll see what I can do.
- Parse the DCF directory and file names, rather than listing every media file
on the filesystem.
- Look for `PRIVATE/M4ROOT` rather than `M4ROOT` when detecting a DCF store.
- Return exactly the number of stores asked for. `GetDCFStores` returns one too
few when asked for two or more.
- Read stores through `fs.FS`, with a `context.Context` to cancel a walk.
- Return a plain slice from `GetDCFStores`, not a pointer to one.
- Test the walk of a store and the detection of mount points. They only run
against real mounted media today.
- Copy files off a card, and verify each copy against its SHA256 hash.
## License ## License
+128 -55
View File
@@ -1,27 +1,78 @@
// Package dcf reads DCF cards: memory cards laid out to the "Design rule for // Package dcf reads DCF cards: memory cards laid out to the "Design rule for
// Camera File system" specification (JEITA CP-3461). It finds the cards among // Camera File system" specification (JEITA CP-3461). It finds the cards among
// the mounted filesystems and lists the images and videos on each one. // the mounted filesystems and parses the DCF directories and files on each one
// into their directory numbers and file numbers. It lists the images and videos
// on each card, and everything under DCIM that is not a DCF directory or file. It
// groups the files in each DCF directory that share a file number. It copies the
// files off a card, and verifies each copy against its SHA256 hash.
package dcf package dcf
import ( import (
"crypto/sha256" "context"
"encoding/hex" "errors"
"fmt" "fmt"
"io" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"slices" "slices"
"strings" "strings"
"time"
"github.com/dustin/go-humanize" "github.com/dustin/go-humanize"
) )
var (
// ErrNoStore is the error from GetStore when no DCF store is mounted.
ErrNoStore = errors.New("no DCF store is mounted")
// ErrMultipleStores is the error from GetStore when more than one DCF store
// is mounted.
ErrMultipleStores = errors.New("more than one DCF store is mounted")
)
// Store represents a DCF store, which is a directory structure that contains // Store represents a DCF store, which is a directory structure that contains
// images and videos in the root of a mounted filesystem. // images and videos in the root of a mounted filesystem.
type Store struct { type Store struct {
RootDirectory string RootDirectory string
// Directories holds the DCF directories under DCIM.
Directories []*Directory
// Images and Videos hold the images and videos among the objects in
// Directories, and those under PRIVATE/M4ROOT.
Images []*Image Images []*Image
Videos []*Video Videos []*Video
// NonConforming holds the path, relative to RootDirectory, of everything
// under DCIM that is not a DCF directory or a DCF file. It includes every
// symlink, and everything inside a directory that breaks the DCF naming
// rules. A directory whose name is not valid UTF-8 is listed without what is
// inside it, because fs.FS refuses to open one.
NonConforming []string
}
// Directory represents a DCF directory: a directory directly under DCIM whose
// name follows the DCF naming rules.
type Directory struct {
// Number is the directory number, from 100 to 999.
Number int
// Name is the name of the directory as it appears on the card.
Name string
// Objects holds every DCF file in the directory, including those that are
// neither images nor videos.
Objects []*Object
// FileGroups holds the same files, grouped by file number, in file number
// order.
FileGroups []*FileGroup
}
// FileGroup represents one file group: the files in one DCF directory that share
// a file number. The DCF specification calls this a DCF object. A file that
// shares its number with no other file is a group of one.
type FileGroup struct {
DirectoryNumber int
FileNumber int
// Members holds every file in the group, including Primary, in name order.
// Two members can have the same extension.
Members []*Object
// Primary is the member that stands for the whole group.
Primary *Object
} }
// Object represents a file in a DCF store. It is embedded in Image and Video // Object represents a file in a DCF store. It is embedded in Image and Video
@@ -31,6 +82,15 @@ type Object struct {
Path string Path string
Size int64 Size int64
Extension string Extension string
// ModTime is the modification time of the file.
ModTime time.Time
// DirectoryNumber and FileNumber are the DCF numbers of a file in a DCF
// directory. Both are 0 for a file under PRIVATE/M4ROOT.
DirectoryNumber int
FileNumber int
// fsys is the fs.FS the store was read through. Hash reads the file through
// it.
fsys fs.FS
} }
// Image represents an image file on a DCF store. // Image represents an image file on a DCF store.
@@ -86,6 +146,21 @@ func (d *Store) TotalSize() int64 {
return d.TotalImageSize() + d.TotalVideoSize() return d.TotalImageSize() + d.TotalVideoSize()
} }
// MembersCount returns the number of files in the file group.
func (g *FileGroup) MembersCount() int {
return len(g.Members)
}
// TotalSize returns the total size in bytes of the files in the file group.
func (g *FileGroup) TotalSize() int64 {
totalSize := int64(0)
for _, member := range g.Members {
totalSize += member.Size
}
return totalSize
}
func (d *Store) String() string { func (d *Store) String() string {
// A caller can set a negative Size, and uint64 would make that huge. // A caller can set a negative Size, and uint64 would make that huge.
totalSize := uint64(max(d.TotalSize(), 0)) totalSize := uint64(max(d.TotalSize(), 0))
@@ -113,77 +188,75 @@ func (i *Image) String() string {
) )
} }
// Hash returns the SHA256 hash of the file as a hex string. // Hash returns the SHA256 hash of the file as a hex string. It reads the file
// through the fs.FS of its store. An Object built by hand has none, so Hash
// returns an error that matches fs.ErrInvalid.
func (d *Object) Hash() (string, error) { func (d *Object) Hash() (string, error) {
return pathToSHA256(d.FullFilePath()) hash, err := d.hash(context.Background())
}
func pathToSHA256(path string) (string, error) {
file, err := os.Open(filepath.Clean(path))
if err != nil { if err != nil {
return "", err return "", fmt.Errorf("hashing %s: %w", d.FullFilePath(), err)
} }
defer func() { _ = file.Close() }() return hash, nil
hash := sha256.New()
_, err = io.Copy(hash, file)
if err != nil {
return "", err
} }
return hex.EncodeToString(hash.Sum(nil)), nil // hash returns the SHA256 hash of the file as a hex string, and stops when ctx is
// cancelled.
func (d *Object) hash(ctx context.Context) (string, error) {
if d.fsys == nil {
return "", fs.ErrInvalid
} }
// GetDCFStores returns a list of DCF stores found on the system in the root of return hashFile(ctx, d.fsys, d.Path)
// any mounted filesystems. The integer argument is the number of DCF stores to }
// return. If the argument is 0, all DCF stores are returned. It does not mount
// filesystems or search outside of filesystem root directories. It does, // GetStores finds the DCF stores in the root of the mounted filesystems and reads
// however, walk the filesystems to find images and videos and populate the // each one. It does not mount filesystems or search outside of filesystem root
// returned Stores. // directories. It reads the DCIM and PRIVATE/M4ROOT directories of each
func GetDCFStores(requestedCount int) (*[]Store, error) { // filesystem to populate the returned Stores. It never follows a symlink. If a
dcfStorePaths, err := findDCFMountPoints(requestedCount) // card cannot be read, it returns an error that names the card's mount point.
// When ctx is cancelled, the read stops before the next directory, and the error
// matches ctx.Err().
func GetStores(ctx context.Context) ([]*Store, error) {
dcfStorePaths, err := findDCFMountPoints(ctx)
if err != nil { if err != nil {
return nil, err return nil, err
} }
dcfStores := []Store{} return parseStores(ctx, dcfStorePaths)
for _, dcfStorePath := range dcfStorePaths {
dcfStore := Store{
RootDirectory: dcfStorePath,
Images: make([]*Image, 0),
Videos: make([]*Video, 0),
} }
addFile := func(path string, info os.FileInfo, err error) error { // GetStore returns the one DCF store when exactly one is mounted. It returns
// ErrNoStore when none is mounted, and ErrMultipleStores when more than one is.
// It reads every card first, the way GetStores does, so a card that cannot be
// read returns that error instead.
func GetStore(ctx context.Context) (*Store, error) {
stores, err := GetStores(ctx)
if err != nil { if err != nil {
return err return nil, err
} }
if !info.IsDir() && isImageFile(path) { return onlyStore(stores)
image := Image{}
image.Path = strings.TrimPrefix(path, dcfStorePath+"/")
image.Size = info.Size()
image.Extension = filepath.Ext(path)
image.DCFStoreRoot = dcfStorePath
dcfStore.Images = append(dcfStore.Images, &image)
} }
if !info.IsDir() && isVideoFile(path) { // onlyStore returns the store when stores holds exactly one.
video := Video{} func onlyStore(stores []*Store) (*Store, error) {
video.Path = strings.TrimPrefix(path, dcfStorePath+"/") switch len(stores) {
video.Size = info.Size() case 0:
video.Extension = filepath.Ext(path) return nil, ErrNoStore
video.DCFStoreRoot = dcfStorePath case 1:
dcfStore.Videos = append(dcfStore.Videos, &video) return stores[0], nil
default:
return nil, ErrMultipleStores
}
} }
return nil // parseStores reads the card at each of the given mount points.
} func parseStores(ctx context.Context, mountpoints []string) ([]*Store, error) {
dcfStores := []*Store{}
err := filepath.Walk(dcfStorePath, addFile) for _, mountpoint := range mountpoints {
dcfStore, err := parseStore(ctx, os.DirFS(mountpoint), mountpoint)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -191,12 +264,12 @@ func GetDCFStores(requestedCount int) (*[]Store, error) {
dcfStores = append(dcfStores, dcfStore) dcfStores = append(dcfStores, dcfStore)
} }
return &dcfStores, nil return dcfStores, nil
} }
func isImageFile(path string) bool { func isImageFile(path string) bool {
imageExtensions := []string{ imageExtensions := []string{
".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tiff", ".raw", ".arw", ".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tif", ".tiff", ".raw", ".arw",
} }
return slices.Contains(imageExtensions, strings.ToLower(filepath.Ext(path))) return slices.Contains(imageExtensions, strings.ToLower(filepath.Ext(path)))
+141 -1
View File
@@ -1,6 +1,144 @@
package dcf package dcf
import "testing" import (
"context"
"errors"
"io/fs"
"path/filepath"
"strings"
"testing"
"testing/fstest"
)
func TestParseStoresReadsEachMountPoint(t *testing.T) {
t.Parallel()
first := newCardOnDisk(t, "DCIM/100ABCDE/ABCD0001.JPG")
second := newCardOnDisk(t, "PRIVATE/M4ROOT/CLIP/C0001.MP4")
stores, err := parseStores(t.Context(), []string{first, second})
if err != nil {
t.Fatalf("parseStores: %v", err)
}
if len(stores) != 2 {
t.Fatalf("parseStores returned %d stores, want 2", len(stores))
}
cases := map[string]Object{
filepath.Join(first, "DCIM/100ABCDE/ABCD0001.JPG"): stores[0].Images[0].Object,
filepath.Join(second, "PRIVATE/M4ROOT/CLIP/C0001.MP4"): stores[1].Videos[0].Object,
}
for want, object := range cases {
if got := object.FullFilePath(); got != want {
t.Errorf("FullFilePath() = %q, want %q", got, want)
}
got, err := object.Hash()
if err != nil {
t.Fatalf("Hash() returned error: %v", err)
}
// The published SHA-256 test vector for "abc".
wantHash := "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"
if got != wantHash {
t.Errorf("Hash() = %q, want %q", got, wantHash)
}
}
}
func TestParseStoresReturnsErrorNamingTheCard(t *testing.T) {
t.Parallel()
// Every lookup under a plain file fails.
mountpoint := filepath.Join(newCardOnDisk(t, "FILE"), "FILE")
_, err := parseStores(t.Context(), []string{mountpoint})
if err == nil || !strings.Contains(err.Error(), mountpoint) {
t.Fatalf("parseStores error = %v, want one naming %s", err, mountpoint)
}
}
func TestParseStoresReturnsErrorWhenCancelled(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(t.Context())
cancel()
mountpoint := newCardOnDisk(t, "DCIM/100ABCDE/ABCD0001.JPG")
_, err := parseStores(ctx, []string{mountpoint})
if !errors.Is(err, context.Canceled) {
t.Fatalf("parseStores error = %v, want %v", err, context.Canceled)
}
}
func TestOnlyStore(t *testing.T) {
t.Parallel()
first := newMountPoint(t, dcimDirectory)
second := newMountPoint(t, m4RootDirectory)
stores, err := parseStores(t.Context(), []string{first, second})
if err != nil {
t.Fatalf("parseStores: %v", err)
}
_, err = onlyStore(stores[:0])
if !errors.Is(err, ErrNoStore) {
t.Errorf("onlyStore of no cards: error = %v, want %v", err, ErrNoStore)
}
store, err := onlyStore(stores[:1])
if err != nil {
t.Fatalf("onlyStore of one card: %v", err)
}
if store.RootDirectory != first {
t.Errorf("onlyStore returned the card at %s, want %s",
store.RootDirectory, first)
}
_, err = onlyStore(stores)
if !errors.Is(err, ErrMultipleStores) {
t.Errorf("onlyStore of two cards: error = %v, want %v",
err, ErrMultipleStores)
}
}
func TestObjectHashReadsThroughTheCard(t *testing.T) {
t.Parallel()
card := fstest.MapFS{"DCIM/100ABCDE/ABCD0001.JPG": {Data: []byte("abc")}}
got, err := mustParseStore(t, card).Images[0].Hash()
if err != nil {
t.Fatalf("Hash() returned error: %v", err)
}
// The published SHA-256 test vector for "abc".
want := "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"
if got != want {
t.Fatalf("Hash() = %q, want %q", got, want)
}
}
func TestObjectHashOfMissingFileReturnsError(t *testing.T) {
t.Parallel()
const filePath = "DCIM/100ABCDE/ABCD0001.JPG"
card := newCard(filePath)
image := mustParseStore(t, card).Images[0]
delete(card, filePath)
_, err := image.Hash()
if !errors.Is(err, fs.ErrNotExist) {
t.Fatalf("Hash() error = %v, want %v", err, fs.ErrNotExist)
}
}
func TestIsImageFile(t *testing.T) { func TestIsImageFile(t *testing.T) {
t.Parallel() t.Parallel()
@@ -8,6 +146,8 @@ func TestIsImageFile(t *testing.T) {
cases := map[string]bool{ cases := map[string]bool{
"DCIM/100MSDCF/DSC00001.jpg": true, "DCIM/100MSDCF/DSC00001.jpg": true,
"DCIM/100MSDCF/DSC00001.ARW": true, "DCIM/100MSDCF/DSC00001.ARW": true,
"DCIM/100MSDCF/DSC00001.TIF": true,
"DCIM/100MSDCF/DSC00001.tif": true,
"PRIVATE/M4ROOT/CLIP/C0001.MP4": false, "PRIVATE/M4ROOT/CLIP/C0001.MP4": false,
"DCIM/100MSDCF/notes.txt": false, "DCIM/100MSDCF/notes.txt": false,
"DCIM/100MSDCF/noextension": false, "DCIM/100MSDCF/noextension": false,
+24 -30
View File
@@ -1,8 +1,9 @@
package dcf_test package dcf_test
import ( import (
"os" "context"
"path/filepath" "errors"
"io/fs"
"testing" "testing"
"sneak.berlin/go/dcf/pkg/dcf" "sneak.berlin/go/dcf/pkg/dcf"
@@ -125,37 +126,30 @@ func TestVideoString(t *testing.T) {
} }
} }
func TestObjectHashIsSHA256OfFile(t *testing.T) { func TestObjectHashOfObjectBuiltByHandReturnsError(t *testing.T) {
t.Parallel() t.Parallel()
root := t.TempDir() object := dcf.Object{DCFStoreRoot: cardRoot, Path: "a.jpg"}
err := os.WriteFile(filepath.Join(root, "a.jpg"), []byte("abc"), 0o600)
if err != nil {
t.Fatalf("WriteFile: %v", err)
}
object := dcf.Object{DCFStoreRoot: root, Path: "a.jpg"}
got, err := object.Hash()
if err != nil {
t.Fatalf("Hash() returned error: %v", err)
}
// The published SHA-256 test vector for "abc".
want := "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"
if got != want {
t.Fatalf("Hash() = %q, want %q", got, want)
}
}
func TestObjectHashOfMissingFileReturnsError(t *testing.T) {
t.Parallel()
object := dcf.Object{DCFStoreRoot: t.TempDir(), Path: "absent.jpg"}
_, err := object.Hash() _, err := object.Hash()
if err == nil { if !errors.Is(err, fs.ErrInvalid) {
t.Fatal("Hash() of a missing file returned no error") t.Fatalf("Hash() error = %v, want %v", err, fs.ErrInvalid)
}
}
func TestGetStoresAndGetStoreReturnCancellation(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(t.Context())
cancel()
_, err := dcf.GetStores(ctx)
if !errors.Is(err, context.Canceled) {
t.Errorf("GetStores error = %v, want %v", err, context.Canceled)
}
_, err = dcf.GetStore(ctx)
if !errors.Is(err, context.Canceled) {
t.Errorf("GetStore error = %v, want %v", err, context.Canceled)
} }
} }
+67
View File
@@ -0,0 +1,67 @@
package dcf
import (
"cmp"
"path"
"slices"
"strings"
)
// newFileGroups sorts the objects of one DCF directory into file groups, in file
// number order. Section 4.3.2.2 of the specification says the files in a DCF
// directory that share a file number are one group. Their first four characters
// and their extensions can differ.
func newFileGroups(objects []*Object) []*FileGroup {
var groups []*FileGroup
groupWithNumber := map[int]*FileGroup{}
for _, object := range objects {
group, found := groupWithNumber[object.FileNumber]
if !found {
group = &FileGroup{
DirectoryNumber: object.DirectoryNumber,
FileNumber: object.FileNumber,
}
groupWithNumber[object.FileNumber] = group
groups = append(groups, group)
}
group.Members = append(group.Members, object)
}
for _, group := range groups {
group.Primary = primaryMember(group.Members)
}
slices.SortFunc(groups, func(a, b *FileGroup) int {
return cmp.Compare(a.FileNumber, b.FileNumber)
})
return groups
}
// primaryMember returns the member that stands for the whole group. Section
// 6.2.1.3 of the specification gives a DCF basic file priority over a DCF extended
// image file. It says nothing about a DCF optional file, the other kind of JPG, so
// ranking every JPG first is a choice. Section 4.6 makes a THM only the thumbnail
// of another file, so it comes last. The specification leaves the rest open, so an
// image comes before a video, and a video before any other file. Among equals, the
// first in name order wins.
func primaryMember(members []*Object) *Object {
choices := []func(string) bool{isJPG, isImageFile, isVideoFile, isNotTHM}
for _, choice := range choices {
for _, member := range members {
if choice(member.Path) {
return member
}
}
}
return members[0]
}
func isNotTHM(name string) bool {
return !strings.EqualFold(path.Ext(name), ".THM")
}
+232
View File
@@ -0,0 +1,232 @@
package dcf
import (
"slices"
"testing"
"testing/fstest"
)
// mustFileGroups returns the file groups of directory, and stops the test unless
// there are exactly count of them.
func mustFileGroups(t *testing.T, directory *Directory, count int) []*FileGroup {
t.Helper()
if len(directory.FileGroups) != count {
t.Fatalf("%s has %d file groups, want %d",
directory.Name, len(directory.FileGroups), count)
}
return directory.FileGroups
}
// assertFileGroup checks the numbers, the primary member and the members of group.
func assertFileGroup(
t *testing.T,
group *FileGroup,
directoryNumber, fileNumber int,
primary string,
members ...string,
) {
t.Helper()
if group.DirectoryNumber != directoryNumber || group.FileNumber != fileNumber {
t.Errorf("group numbers = %d and %d, want %d and %d",
group.DirectoryNumber, group.FileNumber, directoryNumber, fileNumber)
}
if group.Primary.Path != primary {
t.Errorf("Primary = %q, want %q", group.Primary.Path, primary)
}
paths := make([]string, 0, len(group.Members))
for _, member := range group.Members {
paths = append(paths, member.Path)
}
assertPaths(t, "members", paths, members...)
if got := group.MembersCount(); got != len(members) {
t.Errorf("MembersCount() = %d, want %d", got, len(members))
}
}
// assertSizes checks the total size of group and the size of each member.
func assertSizes(t *testing.T, group *FileGroup, total int64, sizes ...int64) {
t.Helper()
if got := group.TotalSize(); got != total {
t.Errorf("TotalSize() = %d, want %d", got, total)
}
got := make([]int64, 0, len(group.Members))
for _, member := range group.Members {
got = append(got, member.Size)
}
if !slices.Equal(got, sizes) {
t.Errorf("member sizes = %v, want %v", got, sizes)
}
}
// assertPrimaryMember checks which of the named members primaryMember picks.
func assertPrimaryMember(t *testing.T, want string, names ...string) {
t.Helper()
members := make([]*Object, 0, len(names))
for _, name := range names {
members = append(members, &Object{Path: name})
}
if got := primaryMember(members).Path; got != want {
t.Errorf("primaryMember of %q = %q, want %q", names, got, want)
}
}
func TestParseStoreGroupsFilesSharingAFileNumber(t *testing.T) {
t.Parallel()
store := mustParseStore(t, fstest.MapFS{
"DCIM/100ABCDE/DSC00001.ARW": {Data: make([]byte, 2000)},
"DCIM/100ABCDE/DSC00001.JPG": {Data: make([]byte, 500)},
"DCIM/100ABCDE/MVI_0002.MP4": {Data: make([]byte, 9000)},
"DCIM/100ABCDE/MVI_0002.THM": {Data: make([]byte, 100)},
"DCIM/100ABCDE/DSC00003.JPG": {Data: make([]byte, 700)},
"DCIM/101ABCDE/DSC00001.JPG": {Data: make([]byte, 300)},
})
first := mustFileGroups(t, store.Directories[0], 3)
second := mustFileGroups(t, store.Directories[1], 1)
assertFileGroup(t, first[0], 100, 1, "DCIM/100ABCDE/DSC00001.JPG",
"DCIM/100ABCDE/DSC00001.ARW",
"DCIM/100ABCDE/DSC00001.JPG",
)
assertSizes(t, first[0], 2500, 2000, 500)
assertFileGroup(t, first[1], 100, 2, "DCIM/100ABCDE/MVI_0002.MP4",
"DCIM/100ABCDE/MVI_0002.MP4",
"DCIM/100ABCDE/MVI_0002.THM",
)
assertSizes(t, first[1], 9100, 9000, 100)
assertFileGroup(t, first[2], 100, 3, "DCIM/100ABCDE/DSC00003.JPG",
"DCIM/100ABCDE/DSC00003.JPG",
)
assertSizes(t, first[2], 700, 700)
assertFileGroup(t, second[0], 101, 1, "DCIM/101ABCDE/DSC00001.JPG",
"DCIM/101ABCDE/DSC00001.JPG",
)
assertSizes(t, second[0], 300, 300)
// The images, the videos and the totals still count each file on its own.
if got := store.ImagesCount(); got != 4 {
t.Errorf("ImagesCount() = %d, want 4", got)
}
if got := store.VideosCount(); got != 1 {
t.Errorf("VideosCount() = %d, want 1", got)
}
if got := store.TotalSize(); got != 12500 {
t.Errorf("TotalSize() = %d, want 12500", got)
}
}
func TestParseStoreGroupsFilesWhoseFirstFourCharactersDiffer(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard(
"DCIM/100ABCDE/ABCD0004.TIF",
"DCIM/100ABCDE/THMB0004.THM",
))
groups := mustFileGroups(t, store.Directories[0], 1)
assertFileGroup(t, groups[0], 100, 4, "DCIM/100ABCDE/ABCD0004.TIF",
"DCIM/100ABCDE/ABCD0004.TIF",
"DCIM/100ABCDE/THMB0004.THM",
)
}
func TestParseStoreOrdersFileGroupsByFileNumber(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard(
"DCIM/100ABCDE/ABCD0002.JPG",
"DCIM/100ABCDE/WXYZ0001.JPG",
))
groups := mustFileGroups(t, store.Directories[0], 2)
assertFileGroup(t, groups[0], 100, 1, "DCIM/100ABCDE/WXYZ0001.JPG",
"DCIM/100ABCDE/WXYZ0001.JPG",
)
assertFileGroup(t, groups[1], 100, 2, "DCIM/100ABCDE/ABCD0002.JPG",
"DCIM/100ABCDE/ABCD0002.JPG",
)
}
func TestParseStoreLeavesOtherFilesOutOfFileGroups(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard(
"DCIM/100ABCDE/ABCD0004.ARW",
"DCIM/100ABCDE/ABCD0004.JPG",
"DCIM/100ABCDE/PQRS0004.JPG",
"DCIM/100ABCDE/README.TXT",
"DCIM/MISC/ABCD0005.JPG",
"PRIVATE/M4ROOT/CLIP/C0006.MP4",
))
if len(store.Directories) != 1 {
t.Fatalf("got %d directories, want 1", len(store.Directories))
}
groups := mustFileGroups(t, store.Directories[0], 1)
assertFileGroup(t, groups[0], 100, 4, "DCIM/100ABCDE/ABCD0004.ARW",
"DCIM/100ABCDE/ABCD0004.ARW",
)
}
func TestParseStoreGivesDirectoryWithoutDCFFilesNoFileGroups(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard("DCIM/100ABCDE/README.TXT"))
mustFileGroups(t, store.Directories[0], 0)
}
func TestPrimaryMember(t *testing.T) {
t.Parallel()
// A JPG comes first, in either case.
assertPrimaryMember(t, "A0010001.JPG", "A0010001.ARW", "A0010001.JPG")
assertPrimaryMember(t, "A0010001.JPG", "A0010001.JPG", "A0010001.MOV")
assertPrimaryMember(t, "a0010001.jpg", "a0010001.arw", "a0010001.jpg")
// Then any other image.
assertPrimaryMember(t, "Z0010001.ARW", "A0010001.MP4", "Z0010001.ARW")
assertPrimaryMember(t, "Z0010001.ARW", "A0010001.WAV", "Z0010001.ARW")
assertPrimaryMember(t, "A0010001.TIF", "A0010001.MOV", "A0010001.TIF")
assertPrimaryMember(t, "Z0010001.TIF", "A0010001.WAV", "Z0010001.TIF")
// Then a video.
assertPrimaryMember(t, "A0010001.MP4", "A0010001.AAC", "A0010001.MP4")
assertPrimaryMember(t, "Z0010001.MP4", "A0010001.THM", "Z0010001.MP4")
// Then any other file except a THM, in either case.
assertPrimaryMember(t, "Z0010001.WAV", "A0010001.THM", "Z0010001.WAV")
assertPrimaryMember(t, "a0010001.wav", "a0010001.thm", "a0010001.wav")
// A THM is primary only when the group holds nothing else.
assertPrimaryMember(t, "A0010001.THM", "A0010001.THM")
// Among equals, the first in name order wins.
assertPrimaryMember(t, "A0010001.THM", "A0010001.THM", "Z0010001.THM")
assertPrimaryMember(t, "A0010001.WAV", "A0010001.WAV", "Z0010001.XML")
}
+39 -32
View File
@@ -1,21 +1,28 @@
package dcf package dcf
import ( import (
"context"
"fmt" "fmt"
"log/slog" "log/slog"
"os" "os"
"path/filepath"
"slices" "slices"
"github.com/shirou/gopsutil/disk" "github.com/shirou/gopsutil/disk"
) )
func findAllMountPoints() ([]string, error) { func findAllMountPoints(ctx context.Context) ([]string, error) {
mountpoints := []string{} mountpoints := []string{}
partitions, err := disk.Partitions(false) // physical devices only, so false // disk.PartitionsWithContext does not check ctx on Linux.
err := ctx.Err()
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("listing the mounted filesystems: %w", err)
}
// physical devices only, so false
partitions, err := disk.PartitionsWithContext(ctx, false)
if err != nil {
return nil, fmt.Errorf("listing the mounted filesystems: %w", err)
} }
for _, partition := range partitions { for _, partition := range partitions {
@@ -27,46 +34,46 @@ func findAllMountPoints() ([]string, error) {
return mountpoints, nil return mountpoints, nil
} }
// findDCFMountPoints returns a list of strings of the paths of mountpoints // findDCFMountPoints returns the paths of the mountpoints that hold a DCF store.
// that contain a DCIM or M4ROOT directory, which is how we detect DCF stores. func findDCFMountPoints(ctx context.Context) ([]string, error) {
// Its only argument is an integer of how many mountpoints to return. If the mountpoints, err := findAllMountPoints(ctx)
// argument is 0, all mountpoints are returned.
func findDCFMountPoints(requestedCount int) ([]string, error) {
filteredMountpoints := []string{}
mountpoints, err := findAllMountPoints()
if err != nil { if err != nil {
return nil, err return nil, err
} }
return filterDCFMountPoints(mountpoints), nil
}
// filterDCFMountPoints returns the mountpoints that hold a DCF store.
func filterDCFMountPoints(mountpoints []string) []string {
filteredMountpoints := []string{}
for _, mountpoint := range mountpoints { for _, mountpoint := range mountpoints {
dcimPath := filepath.Join(mountpoint, "DCIM") if !isDCFMountPoint(mountpoint) {
m4rootPath := filepath.Join(mountpoint, "M4ROOT") continue
shouldKeep := false
_, err = os.Stat(dcimPath)
if err == nil {
shouldKeep = true
slog.Debug(fmt.Sprintf("Found DCIM directory at %s\n", dcimPath))
} }
_, err = os.Stat(m4rootPath)
if err == nil {
shouldKeep = true
slog.Debug(fmt.Sprintf("Found M4ROOT directory at %s\n", m4rootPath))
}
if shouldKeep {
slog.Debug(fmt.Sprintf("Keeping mountpoint %s\n", mountpoint)) slog.Debug(fmt.Sprintf("Keeping mountpoint %s\n", mountpoint))
filteredMountpoints = append(filteredMountpoints, mountpoint) filteredMountpoints = append(filteredMountpoints, mountpoint)
if (requestedCount > 0) && (len(filteredMountpoints)+1 >= requestedCount) {
break
} }
return filteredMountpoints
}
// isDCFMountPoint reports whether the root of mountpoint holds a DCIM or
// PRIVATE/M4ROOT directory, which is how we detect DCF stores.
func isDCFMountPoint(mountpoint string) bool {
fsys := os.DirFS(mountpoint)
for _, directory := range []string{dcimDirectory, m4RootDirectory} {
found, err := hasDirectory(fsys, directory)
if err == nil && found {
slog.Debug(fmt.Sprintf("Found %s directory at %s\n", directory, mountpoint))
return true
} }
} }
return filteredMountpoints, nil return false
} }
+87
View File
@@ -0,0 +1,87 @@
package dcf
import (
"os"
"path/filepath"
"slices"
"testing"
)
// newMountPoint returns a temporary directory holding the given directories.
func newMountPoint(t *testing.T, directories ...string) string {
t.Helper()
root := t.TempDir()
for _, directory := range directories {
err := os.MkdirAll(filepath.Join(root, directory), 0o750)
if err != nil {
t.Fatalf("MkdirAll: %v", err)
}
}
return root
}
func TestIsDCFMountPoint(t *testing.T) {
t.Parallel()
cases := map[string]bool{
"DCIM": true,
"PRIVATE/M4ROOT": true,
"M4ROOT": false,
"PRIVATE": false,
}
for directory, want := range cases {
got := isDCFMountPoint(newMountPoint(t, directory))
if got != want {
t.Errorf("isDCFMountPoint with only %q = %v, want %v",
directory, got, want)
}
}
}
func TestIsDCFMountPointIgnoresPlainFile(t *testing.T) {
t.Parallel()
root := t.TempDir()
err := os.WriteFile(filepath.Join(root, "DCIM"), nil, 0o600)
if err != nil {
t.Fatalf("WriteFile: %v", err)
}
if isDCFMountPoint(root) {
t.Fatal("isDCFMountPoint counted a plain file named DCIM as a directory")
}
}
func TestIsDCFMountPointIgnoresSymlink(t *testing.T) {
t.Parallel()
outside := newMountPoint(t, "M4ROOT")
for _, link := range []string{dcimDirectory, privateDirectory, m4RootDirectory} {
root := newMountPoint(t, filepath.Dir(link))
addSymlink(t, root, link, outside)
if isDCFMountPoint(root) {
t.Errorf("isDCFMountPoint counted a symlink at %s as a directory", link)
}
}
}
func TestFilterDCFMountPointsKeepsThoseHoldingAStore(t *testing.T) {
t.Parallel()
first := newMountPoint(t, "DCIM")
second := newMountPoint(t, "PRIVATE/M4ROOT")
got := filterDCFMountPoints([]string{newMountPoint(t), first, second})
want := []string{first, second}
if !slices.Equal(got, want) {
t.Errorf("filterDCFMountPoints = %v, want %v", got, want)
}
}
+279
View File
@@ -0,0 +1,279 @@
package dcf
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path"
"path/filepath"
"time"
)
// The modes of what Import creates. Only the user who ran it can read the copies.
const (
importDirectoryMode = 0o750
importFileMode = 0o600
)
var errHashMismatch = errors.New("the SHA256 hash of the copy differs from the source")
// ImportResult lists what Store.Import did with each file, by destination path.
type ImportResult struct {
// Copied holds the copies this import wrote and verified.
Copied []string
// Skipped holds the paths that already held a file with the same SHA256 hash.
Skipped []string
// Conflicts holds the paths that already held anything else. Import left it
// alone and did not copy the file.
Conflicts []string
}
// importFile is one file for Import to copy, and the path to copy it to.
type importFile struct {
source *Object
target string
}
// Import copies the files of the store to dest, and verifies each copy against
// the SHA256 hash of its source. It copies every member of every file group, and
// the images and videos under PRIVATE/M4ROOT. It only reads the card.
//
// Each file goes to dest/YYYY-MM-DD/<file name>, from its modification time. Every
// member of a file group goes to the day of its primary member, so a group never
// splits across two folders.
//
// A file with the same hash already at the path is skipped. Anything else there is
// a conflict, which is not an error. Import never overwrites anything.
//
// One file that fails does not stop the others. The returned error joins the
// failures, each naming its file, and the result still lists what was done. When
// ctx is cancelled, the import stops at once and removes the partial copy, and the
// error matches ctx.Err().
func (d *Store) Import(ctx context.Context, dest string) (ImportResult, error) {
var (
result ImportResult
failures []error
)
for _, file := range d.importFiles(dest) {
err := ctx.Err()
if err != nil {
failures = append(failures, fmt.Errorf("import stopped: %w", err))
break
}
err = copyFile(ctx, file, &result)
if err != nil {
failures = append(failures,
fmt.Errorf("importing %s: %w", file.source.FullFilePath(), err))
}
}
return result, errors.Join(failures...)
}
// importFiles lists the files of the store that Import copies to dest.
func (d *Store) importFiles(dest string) []importFile {
var files []importFile
add := func(source *Object, day time.Time) {
name := path.Base(source.Path)
files = append(files, importFile{
source: source,
target: filepath.Join(dest, day.Format(time.DateOnly), name),
})
}
for _, directory := range d.Directories {
for _, group := range directory.FileGroups {
for _, member := range group.Members {
add(member, group.Primary.ModTime)
}
}
}
// The files under PRIVATE/M4ROOT are the ones with no DCF numbers.
for _, image := range d.Images {
if image.DirectoryNumber == 0 {
add(&image.Object, image.ModTime)
}
}
for _, video := range d.Videos {
if video.DirectoryNumber == 0 {
add(&video.Object, video.ModTime)
}
}
return files
}
// copyFile copies one file, and adds its target to the list in result that says
// what happened. It adds the target to no list when it returns an error.
func copyFile(ctx context.Context, file importFile, result *ImportResult) error {
sourceHash, err := file.source.hash(ctx)
if err != nil {
return err
}
err = os.MkdirAll(filepath.Dir(file.target), importDirectoryMode)
if err != nil {
return err
}
// O_EXCL fails on anything already at the path, even a symlink, so nothing
// there can be overwritten.
target, err := os.OpenFile(
file.target, os.O_WRONLY|os.O_CREATE|os.O_EXCL, importFileMode,
)
if errors.Is(err, fs.ErrExist) {
return checkExisting(ctx, file.target, sourceHash, result)
}
if err != nil {
return err
}
err = writeCopy(ctx, file, target, sourceHash)
if err != nil {
return errors.Join(err, os.Remove(file.target))
}
result.Copied = append(result.Copied, file.target)
return nil
}
// checkExisting adds target to the skipped files if it is a file with the given
// hash, and to the conflicts if it is anything else.
func checkExisting(
ctx context.Context,
target, sourceHash string,
result *ImportResult,
) error {
info, err := os.Lstat(target)
if err != nil {
return err
}
if !info.Mode().IsRegular() {
result.Conflicts = append(result.Conflicts, target)
return nil
}
targetHash, err := hashOnDisk(ctx, target)
if err != nil {
return err
}
if targetHash == sourceHash {
result.Skipped = append(result.Skipped, target)
} else {
result.Conflicts = append(result.Conflicts, target)
}
return nil
}
// writeCopy fills the new file target from the source, and closes it. It then
// reads the written file back to compare its hash with the hash of the source.
func writeCopy(
ctx context.Context,
file importFile,
target *os.File,
sourceHash string,
) error {
err := writeAndClose(ctx, file.source, target)
if err != nil {
return err
}
targetHash, err := hashOnDisk(ctx, file.target)
if err != nil {
return err
}
if targetHash != sourceHash {
return errHashMismatch
}
// The zero time leaves the access time as it is.
return os.Chtimes(file.target, time.Time{}, file.source.ModTime)
}
// writeAndClose writes the bytes of source to target, syncs target and closes it.
func writeAndClose(ctx context.Context, source *Object, target *os.File) error {
defer func() { _ = target.Close() }()
sourceFile, err := source.fsys.Open(source.Path)
if err != nil {
return err
}
defer func() { _ = sourceFile.Close() }()
_, err = io.Copy(target, contextReader(ctx, sourceFile))
if err != nil {
return err
}
err = target.Sync()
if err != nil {
return err
}
return target.Close()
}
// hashOnDisk returns the SHA256 hash of the file at filePath on the local
// filesystem.
func hashOnDisk(ctx context.Context, filePath string) (string, error) {
return hashFile(ctx, os.DirFS(filepath.Dir(filePath)), filepath.Base(filePath))
}
// hashFile returns the SHA256 hash of the named file as a hex string.
func hashFile(ctx context.Context, fsys fs.FS, name string) (string, error) {
file, err := fsys.Open(name)
if err != nil {
return "", err
}
defer func() { _ = file.Close() }()
hash := sha256.New()
_, err = io.Copy(hash, contextReader(ctx, file))
if err != nil {
return "", err
}
return hex.EncodeToString(hash.Sum(nil)), nil
}
// readerFunc is a function that works as an io.Reader.
type readerFunc func([]byte) (int, error)
func (f readerFunc) Read(p []byte) (int, error) {
return f(p)
}
// contextReader returns a reader that fails with ctx.Err() once ctx is cancelled,
// so a copy stops inside a file.
func contextReader(ctx context.Context, reader io.Reader) io.Reader {
return readerFunc(func(p []byte) (int, error) {
err := ctx.Err()
if err != nil {
return 0, err
}
return reader.Read(p)
})
}
+521
View File
@@ -0,0 +1,521 @@
package dcf
import (
"context"
"errors"
"io/fs"
"maps"
"os"
"path/filepath"
"strings"
"testing"
"testing/fstest"
"time"
)
// testDay is the folder that a file from cardFile goes to. The other constants
// are the paths and contents of the files on most cards in these tests.
const (
testDay = "2026-10-03"
firstJPG = "DCIM/100ABCDE/DSC00011.JPG"
firstJPGData = "first jpeg"
secondJPG = "DCIM/100ABCDE/DSC00012.JPG"
secondJPGData = "second jpeg"
testClip = "PRIVATE/M4ROOT/CLIP/C0011.MP4"
testClipData = "clip"
)
// cardFile returns a file on a card, modified at noon on testDay.
func cardFile(contents string) *fstest.MapFile {
return &fstest.MapFile{
Data: []byte(contents),
ModTime: time.Date(2026, 10, 3, 12, 0, 0, 0, time.UTC),
}
}
// inTestDay returns the paths of the named files in the testDay folder of dest.
func inTestDay(dest string, names ...string) []string {
paths := make([]string, 0, len(names))
for _, name := range names {
paths = append(paths, filepath.Join(dest, testDay, name))
}
return paths
}
// assertDest checks that dest holds exactly the given files, as a map from each
// file's path under dest to its contents.
func assertDest(t *testing.T, dest string, want map[string]string) {
t.Helper()
fsys := os.DirFS(dest)
got := map[string]string{}
err := fs.WalkDir(fsys, ".",
func(name string, entry fs.DirEntry, err error) error {
if err != nil || !entry.Type().IsRegular() {
return err
}
data, err := fs.ReadFile(fsys, name)
got[name] = string(data)
return err
})
if err != nil {
t.Fatalf("reading %s: %v", dest, err)
}
if !maps.Equal(got, want) {
t.Errorf("destination holds %q, want %q", got, want)
}
}
func assertEmptyDirectory(t *testing.T, directory string) {
t.Helper()
entries, err := os.ReadDir(directory)
if err != nil {
t.Fatalf("ReadDir: %v", err)
}
if len(entries) != 0 {
t.Errorf("%s holds %v, want nothing", directory, entries)
}
}
// addDestFile writes a file at the named path under dest.
func addDestFile(t *testing.T, dest, name, contents string) {
t.Helper()
fullPath := filepath.Join(dest, name)
err := os.MkdirAll(filepath.Dir(fullPath), 0o750)
if err != nil {
t.Fatalf("MkdirAll: %v", err)
}
err = os.WriteFile(fullPath, []byte(contents), 0o600)
if err != nil {
t.Fatalf("WriteFile: %v", err)
}
}
func TestImportCopiesEveryFileAndKeepsItsModificationTime(t *testing.T) {
t.Parallel()
dest := t.TempDir()
store := mustParseStore(t, fstest.MapFS{
"DCIM/100ABCDE/DSC00011.ARW": cardFile("raw"),
firstJPG: cardFile(firstJPGData),
"DCIM/101ABCDE/MVI_0012.MP4": cardFile("video"),
"DCIM/101ABCDE/MVI_0012.THM": cardFile("thumbnail"),
"DCIM/MISC/DSC00013.JPG": cardFile("not conforming"),
testClip: cardFile(testClipData),
"PRIVATE/M4ROOT/CLIP/C0011M01.XML": cardFile("neither image nor video"),
"PRIVATE/M4ROOT/THMBNL/C0011T01.JPG": cardFile("clip thumbnail"),
})
result, err := store.Import(t.Context(), dest)
if err != nil {
t.Fatalf("Import: %v", err)
}
assertPaths(t, "Copied", result.Copied, inTestDay(dest,
"DSC00011.ARW",
"DSC00011.JPG",
"MVI_0012.MP4",
"MVI_0012.THM",
"C0011T01.JPG",
"C0011.MP4",
)...)
assertPaths(t, "Skipped", result.Skipped)
assertPaths(t, "Conflicts", result.Conflicts)
assertDest(t, dest, map[string]string{
testDay + "/DSC00011.ARW": "raw",
testDay + "/DSC00011.JPG": firstJPGData,
testDay + "/MVI_0012.MP4": "video",
testDay + "/MVI_0012.THM": "thumbnail",
testDay + "/C0011.MP4": testClipData,
testDay + "/C0011T01.JPG": "clip thumbnail",
})
for _, copied := range result.Copied {
info, err := os.Stat(copied)
if err != nil {
t.Fatalf("Stat: %v", err)
}
if want := cardFile("").ModTime; !info.ModTime().Equal(want) {
t.Errorf("%s modified at %v, want %v", copied, info.ModTime(), want)
}
}
}
func TestImportTwiceSkipsEveryFileTheSecondTime(t *testing.T) {
t.Parallel()
dest := t.TempDir()
store := mustParseStore(t, fstest.MapFS{
firstJPG: cardFile(firstJPGData),
testClip: cardFile(testClipData),
})
_, err := store.Import(t.Context(), dest)
if err != nil {
t.Fatalf("first Import: %v", err)
}
result, err := store.Import(t.Context(), dest)
if err != nil {
t.Fatalf("second Import: %v", err)
}
assertPaths(t, "Copied", result.Copied)
assertPaths(t, "Skipped", result.Skipped,
inTestDay(dest, "DSC00011.JPG", "C0011.MP4")...)
assertPaths(t, "Conflicts", result.Conflicts)
assertDest(t, dest, map[string]string{
testDay + "/DSC00011.JPG": firstJPGData,
testDay + "/C0011.MP4": testClipData,
})
}
func TestImportLeavesADifferentFileAloneAndReportsAConflict(t *testing.T) {
t.Parallel()
dest := t.TempDir()
addDestFile(t, dest, testDay+"/DSC00011.JPG", "different")
store := mustParseStore(t, fstest.MapFS{
firstJPG: cardFile(firstJPGData),
secondJPG: cardFile(secondJPGData),
})
result, err := store.Import(t.Context(), dest)
if err != nil {
t.Fatalf("Import: %v", err)
}
assertPaths(t, "Copied", result.Copied, inTestDay(dest, "DSC00012.JPG")...)
assertPaths(t, "Skipped", result.Skipped)
assertPaths(t, "Conflicts", result.Conflicts, inTestDay(dest, "DSC00011.JPG")...)
assertDest(t, dest, map[string]string{
testDay + "/DSC00011.JPG": "different",
testDay + "/DSC00012.JPG": secondJPGData,
})
}
func TestImportReportsAConflictBetweenTwoFilesWithOneName(t *testing.T) {
t.Parallel()
dest := t.TempDir()
store := mustParseStore(t, fstest.MapFS{
firstJPG: cardFile(firstJPGData),
"DCIM/101ABCDE/DSC00011.JPG": cardFile(secondJPGData),
})
result, err := store.Import(t.Context(), dest)
if err != nil {
t.Fatalf("Import: %v", err)
}
assertPaths(t, "Copied", result.Copied, inTestDay(dest, "DSC00011.JPG")...)
assertPaths(t, "Conflicts", result.Conflicts, inTestDay(dest, "DSC00011.JPG")...)
assertDest(t, dest, map[string]string{testDay + "/DSC00011.JPG": firstJPGData})
}
func TestImportReportsAConflictAtADirectoryOrASymlink(t *testing.T) {
t.Parallel()
dest := t.TempDir()
// The symlink points at a file with the same bytes, which must not count.
outside := newCardOnDisk(t, "OUTSIDE.JPG")
err := os.MkdirAll(filepath.Join(dest, testDay, "DSC00011.JPG"), 0o750)
if err != nil {
t.Fatalf("MkdirAll: %v", err)
}
addSymlink(t, dest, testDay+"/DSC00012.JPG", filepath.Join(outside, "OUTSIDE.JPG"))
store := mustParseStore(t, fstest.MapFS{
firstJPG: cardFile("abc"),
secondJPG: cardFile("abc"),
})
result, err := store.Import(t.Context(), dest)
if err != nil {
t.Fatalf("Import: %v", err)
}
assertPaths(t, "Copied", result.Copied)
assertPaths(t, "Skipped", result.Skipped)
assertPaths(t, "Conflicts", result.Conflicts,
inTestDay(dest, "DSC00011.JPG", "DSC00012.JPG")...)
}
func TestImportPutsAFileGroupInTheFolderOfItsPrimaryMember(t *testing.T) {
t.Parallel()
dest := t.TempDir()
beforeMidnight := time.Date(2026, 10, 3, 23, 59, 59, 0, time.UTC)
afterMidnight := time.Date(2026, 10, 4, 0, 0, 1, 0, time.UTC)
store := mustParseStore(t, fstest.MapFS{
"DCIM/100ABCDE/DSC00011.ARW": {Data: []byte("raw"), ModTime: afterMidnight},
firstJPG: {Data: []byte("jpeg"), ModTime: beforeMidnight},
secondJPG: {Data: []byte("next"), ModTime: afterMidnight},
})
_, err := store.Import(t.Context(), dest)
if err != nil {
t.Fatalf("Import: %v", err)
}
assertDest(t, dest, map[string]string{
"2026-10-03/DSC00011.ARW": "raw",
"2026-10-03/DSC00011.JPG": "jpeg",
"2026-10-04/DSC00012.JPG": "next",
})
// The copy keeps its own modification time, whichever folder it is in.
info, err := os.Stat(filepath.Join(dest, "2026-10-03/DSC00011.ARW"))
if err != nil {
t.Fatalf("Stat: %v", err)
}
if !info.ModTime().Equal(afterMidnight) {
t.Errorf("modified at %v, want %v", info.ModTime(), afterMidnight)
}
}
// changingFS is a card whose file at name holds other bytes from its second read
// on. It counts the reads in opens.
type changingFS struct {
fstest.MapFS
name string
opens *int
}
func (f changingFS) Open(name string) (fs.File, error) {
if name == f.name {
*f.opens++
if *f.opens > 1 {
return fstest.MapFS{name: cardFile("other bytes")}.Open(name)
}
}
return f.MapFS.Open(name)
}
func TestImportRemovesACopyThatDoesNotMatchItsSource(t *testing.T) {
t.Parallel()
dest := t.TempDir()
opens := 0
store := mustParseStore(t, changingFS{
MapFS: fstest.MapFS{
firstJPG: cardFile(firstJPGData),
secondJPG: cardFile(secondJPGData),
},
name: firstJPG,
opens: &opens,
})
result, err := store.Import(t.Context(), dest)
if !errors.Is(err, errHashMismatch) {
t.Fatalf("Import error = %v, want %v", err, errHashMismatch)
}
if !strings.Contains(err.Error(), firstJPG) {
t.Errorf("Import error = %v, want one naming %s", err, firstJPG)
}
assertPaths(t, "Copied", result.Copied, inTestDay(dest, "DSC00012.JPG")...)
assertDest(t, dest, map[string]string{testDay + "/DSC00012.JPG": secondJPGData})
}
func TestImportCopiesTheOtherFilesWhenOneCannotBeRead(t *testing.T) {
t.Parallel()
dest := t.TempDir()
store := mustParseStore(t, failingFS{
MapFS: fstest.MapFS{
firstJPG: cardFile(firstJPGData),
secondJPG: cardFile(secondJPGData),
testClip: cardFile(testClipData),
},
call: "Open",
name: firstJPG,
})
result, err := store.Import(t.Context(), dest)
if !errors.Is(err, fs.ErrPermission) {
t.Fatalf("Import error = %v, want %v", err, fs.ErrPermission)
}
if !strings.Contains(err.Error(), firstJPG) {
t.Errorf("Import error = %v, want one naming %s", err, firstJPG)
}
assertPaths(t, "Copied", result.Copied,
inTestDay(dest, "DSC00012.JPG", "C0011.MP4")...)
assertDest(t, dest, map[string]string{
testDay + "/DSC00012.JPG": secondJPGData,
testDay + "/C0011.MP4": testClipData,
})
}
func TestImportNamesEveryFileThatFails(t *testing.T) {
t.Parallel()
dest := t.TempDir()
// A plain file where the day folder belongs fails every file of that day.
addDestFile(t, dest, testDay, "not a folder")
store := mustParseStore(t, fstest.MapFS{
firstJPG: cardFile(firstJPGData),
secondJPG: cardFile(secondJPGData),
})
result, err := store.Import(t.Context(), dest)
if err == nil {
t.Fatal("Import returned no error")
}
for _, name := range []string{firstJPG, secondJPG} {
if !strings.Contains(err.Error(), name) {
t.Errorf("Import error = %v, want one naming %s", err, name)
}
}
assertPaths(t, "Copied", result.Copied)
}
func TestImportOfObjectBuiltByHandReturnsError(t *testing.T) {
t.Parallel()
dest := t.TempDir()
store := &Store{Images: []*Image{{Object: Object{Path: "DSC00011.JPG"}}}}
_, err := store.Import(t.Context(), dest)
if !errors.Is(err, fs.ErrInvalid) {
t.Fatalf("Import error = %v, want %v", err, fs.ErrInvalid)
}
assertEmptyDirectory(t, dest)
}
// cancelingFile is an open file that cancels the import after each read.
type cancelingFile struct {
fs.File
cancel context.CancelFunc
}
func (f cancelingFile) Read(p []byte) (int, error) {
n, err := f.File.Read(p)
f.cancel()
return n, err
}
// cancelingReadFS is a card that cancels the import during the read of the named
// file, the time that file is opened for the openCount-th time. It counts the
// reads in opens.
type cancelingReadFS struct {
fstest.MapFS
name string
openCount int
opens *int
cancel context.CancelFunc
}
func (f cancelingReadFS) Open(name string) (fs.File, error) {
file, err := f.MapFS.Open(name)
if err != nil || name != f.name {
return file, err
}
*f.opens++
if *f.opens != f.openCount {
return file, nil
}
return cancelingFile{File: file, cancel: f.cancel}, nil
}
func TestImportStopsInsideAFileWhenCancelledAndRemovesThePartialCopy(t *testing.T) {
t.Parallel()
// The first read of a file is hashing it, and the second is copying it.
for _, openCount := range []int{1, 2} {
ctx, cancel := context.WithCancel(t.Context())
dest := t.TempDir()
opens := 0
store := mustParseStore(t, cancelingReadFS{
MapFS: fstest.MapFS{
// Larger than one read, so the cancellation falls inside the file.
firstJPG: {Data: make([]byte, 1<<20)},
secondJPG: cardFile(secondJPGData),
},
name: firstJPG,
openCount: openCount,
opens: &opens,
cancel: cancel,
})
result, err := store.Import(ctx, dest)
if !errors.Is(err, context.Canceled) {
t.Errorf("cancelled in read %d: error = %v, want %v",
openCount, err, context.Canceled)
}
assertPaths(t, "Copied", result.Copied)
assertDest(t, dest, map[string]string{})
if opens != openCount {
t.Errorf("cancelled in read %d: file opened %d times", openCount, opens)
}
cancel()
}
}
func TestImportCopiesNothingWhenAlreadyCancelled(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(t.Context())
cancel()
dest := t.TempDir()
store := mustParseStore(t, fstest.MapFS{firstJPG: cardFile(firstJPGData)})
result, err := store.Import(ctx, dest)
if !errors.Is(err, context.Canceled) {
t.Fatalf("Import error = %v, want %v", err, context.Canceled)
}
assertPaths(t, "Copied", result.Copied)
assertEmptyDirectory(t, dest)
}
+311
View File
@@ -0,0 +1,311 @@
package dcf
import (
"context"
"errors"
"fmt"
"io/fs"
"path"
"regexp"
"strconv"
"strings"
)
// The two directories at the root of a card that hold its files.
const (
dcimDirectory = "DCIM"
m4RootDirectory = "PRIVATE/M4ROOT"
)
// The naming rules of the DCF specification, CIPA DC-009-2010, sections 2.3,
// 4.2.2 and 4.3.1. Names use A-Z, 0-9 and "_", and lower-case letters count as
// upper-case. Each pattern captures the number in the name.
var (
dcfDirectoryName = regexp.MustCompile(`^([1-9][0-9]{2})[0-9A-Za-z_]{5}$`)
dcfFileName = regexp.MustCompile(`^[0-9A-Za-z_]{4}([0-9]{4})\.[0-9A-Za-z_]{3}$`)
)
// parseStore reads one card from fsys, which is rooted at the card's mount
// point. The root argument is the path of that mount point. Only DCIM and
// PRIVATE/M4ROOT are read. A card can lack either one. A symlink is never
// followed. An error names the mount point, because fsys reports only paths
// inside it. When ctx is cancelled, the read stops before the next directory.
func parseStore(ctx context.Context, fsys fs.FS, root string) (*Store, error) {
store := &Store{RootDirectory: root}
err := parseDCIM(ctx, fsys, store)
if err == nil {
err = parseM4Root(ctx, fsys, store)
}
if err != nil {
return nil, fmt.Errorf("reading the card at %s: %w", root, err)
}
return store, nil
}
// hasDirectory reports whether the card holds the named directory. A plain file
// or a symlink at any step of the path counts as missing. Any other failure to
// look is an error.
func hasDirectory(fsys fs.FS, name string) (bool, error) {
parent := path.Dir(name)
if parent != "." {
found, err := hasDirectory(fsys, parent)
if err != nil || !found {
return false, err
}
}
info, err := fs.Lstat(fsys, name)
if errors.Is(err, fs.ErrNotExist) {
return false, nil
}
if err != nil {
return false, err
}
return info.IsDir(), nil
}
// readDir returns the entries of the named directory, unless ctx is cancelled.
func readDir(ctx context.Context, fsys fs.FS, name string) ([]fs.DirEntry, error) {
err := ctx.Err()
if err != nil {
return nil, err
}
return fs.ReadDir(fsys, name)
}
// parseDCIM adds the DCF directories under DCIM to store. Everything else there
// is not conforming. That includes directories that share a directory number,
// which section 7.1.2 of the specification treats as non-DCF directories.
func parseDCIM(ctx context.Context, fsys fs.FS, store *Store) error {
found, err := hasDirectory(fsys, dcimDirectory)
if err != nil || !found {
return err
}
entries, err := readDir(ctx, fsys, dcimDirectory)
if err != nil {
return err
}
directoriesWithNumber := map[int]int{}
for _, entry := range entries {
if entry.IsDir() {
directoriesWithNumber[parseNumber(dcfDirectoryName, entry.Name())]++
}
}
for _, entry := range entries {
number := parseNumber(dcfDirectoryName, entry.Name())
if entry.IsDir() && number != 0 && directoriesWithNumber[number] == 1 {
err = parseDirectory(ctx, fsys, store, entry.Name(), number)
} else {
err = addNonConforming(ctx, fsys, store, dcimDirectory, entry)
}
if err != nil {
return err
}
}
return nil
}
// parseDirectory adds one DCF directory and the DCF files directly inside it to
// store. A DCF file is a regular file, so a symlink with a DCF name is not one.
// Everything else there is not conforming. That includes JPG files that share a
// file number, which section 7.2.2 of the specification says must not show DCF
// numbers. Other files that share a file number do conform.
func parseDirectory(
ctx context.Context,
fsys fs.FS,
store *Store,
name string,
number int,
) error {
directoryPath := path.Join(dcimDirectory, name)
entries, err := readDir(ctx, fsys, directoryPath)
if err != nil {
return err
}
jpgsWithNumber := map[int]int{}
for _, entry := range entries {
if entry.Type().IsRegular() && isJPG(entry.Name()) {
jpgsWithNumber[parseNumber(dcfFileName, entry.Name())]++
}
}
directory := &Directory{Number: number, Name: name}
for _, entry := range entries {
fileNumber := parseNumber(dcfFileName, entry.Name())
sharedJPG := isJPG(entry.Name()) && jpgsWithNumber[fileNumber] > 1
if !entry.Type().IsRegular() || fileNumber == 0 || sharedJPG {
err = addNonConforming(ctx, fsys, store, directoryPath, entry)
if err != nil {
return err
}
continue
}
entryPath := path.Join(directoryPath, entry.Name())
object, err := newObject(fsys, store, entryPath, entry)
if err != nil {
return err
}
object.DirectoryNumber = number
object.FileNumber = fileNumber
directory.Objects = append(directory.Objects, &object)
addMedia(store, object)
}
directory.FileGroups = newFileGroups(directory.Objects)
store.Directories = append(store.Directories, directory)
return nil
}
// parseM4Root adds the images and videos under PRIVATE/M4ROOT to store. The DCF
// naming rules do not apply there, so these files have no DCF numbers. It skips
// a symlink. It also skips a name that is not valid UTF-8, because fs.FS refuses
// to open one.
func parseM4Root(ctx context.Context, fsys fs.FS, store *Store) error {
found, err := hasDirectory(fsys, m4RootDirectory)
if err != nil || !found {
return err
}
return fs.WalkDir(fsys, m4RootDirectory,
func(filePath string, entry fs.DirEntry, err error) error {
if err != nil {
return err
}
validName := fs.ValidPath(filePath)
if entry.IsDir() && !validName {
return fs.SkipDir
}
// fs.WalkDir reads a directory right after this call for it.
if entry.IsDir() {
return ctx.Err()
}
if !entry.Type().IsRegular() || !validName {
return nil
}
object, err := newObject(fsys, store, filePath, entry)
if err != nil {
return err
}
addMedia(store, object)
return nil
})
}
// addNonConforming lists one entry of the directory dir on store as not
// conforming. If the entry is a directory, it lists everything inside it too. It
// does not follow a symlink. It does not read a directory whose name is not
// valid UTF-8, because fs.FS refuses to open one.
func addNonConforming(
ctx context.Context,
fsys fs.FS,
store *Store,
dir string,
entry fs.DirEntry,
) error {
entryPath := path.Join(dir, entry.Name())
store.NonConforming = append(store.NonConforming, entryPath)
if !entry.IsDir() || !fs.ValidPath(entryPath) {
return nil
}
entries, err := readDir(ctx, fsys, entryPath)
if err != nil {
return err
}
for _, inner := range entries {
err = addNonConforming(ctx, fsys, store, entryPath, inner)
if err != nil {
return err
}
}
return nil
}
// newObject returns the Object for the file at filePath, with no DCF numbers.
func newObject(
fsys fs.FS,
store *Store,
filePath string,
entry fs.DirEntry,
) (Object, error) {
info, err := entry.Info()
if err != nil {
return Object{}, err
}
return Object{
DCFStoreRoot: store.RootDirectory,
Path: filePath,
Size: info.Size(),
Extension: path.Ext(filePath),
ModTime: info.ModTime(),
fsys: fsys,
}, nil
}
// addMedia adds object to the images or videos of store. A file with any other
// extension is neither, and is not added.
func addMedia(store *Store, object Object) {
if isImageFile(object.Path) {
store.Images = append(store.Images, &Image{Object: object})
}
if isVideoFile(object.Path) {
store.Videos = append(store.Videos, &Video{Object: object})
}
}
// parseNumber returns the number that pattern captures in name. It returns 0
// when name does not match. No DCF number is 0, so this also rejects the file
// number 0000, which the specification forbids.
func parseNumber(pattern *regexp.Regexp, name string) int {
match := pattern.FindStringSubmatch(name)
if match == nil {
return 0
}
number, err := strconv.Atoi(match[1])
if err != nil {
return 0
}
return number
}
func isJPG(name string) bool {
return strings.EqualFold(path.Ext(name), ".JPG")
}
+915
View File
@@ -0,0 +1,915 @@
package dcf
import (
"context"
"errors"
"io/fs"
"os"
"path"
"path/filepath"
"reflect"
"slices"
"testing"
"testing/fstest"
)
const (
testRoot = "/mnt/card"
privateDirectory = "PRIVATE"
)
// newCard returns a card holding an empty file at each of the given paths.
func newCard(paths ...string) fstest.MapFS {
card := fstest.MapFS{}
for _, filePath := range paths {
card[filePath] = &fstest.MapFile{}
}
return card
}
// newCardOnDisk returns a temporary directory holding the three bytes "abc" in a
// file at each of the given paths. It is for tests that fstest.MapFS cannot do.
func newCardOnDisk(t *testing.T, paths ...string) string {
t.Helper()
root := t.TempDir()
for _, filePath := range paths {
fullPath := filepath.Join(root, filePath)
err := os.MkdirAll(filepath.Dir(fullPath), 0o750)
if err != nil {
t.Fatalf("MkdirAll: %v", err)
}
err = os.WriteFile(fullPath, []byte("abc"), 0o600)
if err != nil {
t.Fatalf("WriteFile: %v", err)
}
}
return root
}
// addSymlink creates a symlink to target at the named path under root.
func addSymlink(t *testing.T, root, name, target string) {
t.Helper()
err := os.Symlink(target, filepath.Join(root, name))
if err != nil {
t.Fatalf("Symlink: %v", err)
}
}
func mustParseStore(t *testing.T, fsys fs.FS) *Store {
t.Helper()
store, err := parseStore(t.Context(), fsys, testRoot)
if err != nil {
t.Fatalf("parseStore: %v", err)
}
return store
}
func imagePaths(store *Store) []string {
paths := make([]string, 0, len(store.Images))
for _, image := range store.Images {
paths = append(paths, image.Path)
}
return paths
}
func videoPaths(store *Store) []string {
paths := make([]string, 0, len(store.Videos))
for _, video := range store.Videos {
paths = append(paths, video.Path)
}
return paths
}
func objectPaths(directory *Directory) []string {
paths := make([]string, 0, len(directory.Objects))
for _, object := range directory.Objects {
paths = append(paths, object.Path)
}
return paths
}
func directoryNumbers(store *Store) []int {
numbers := make([]int, 0, len(store.Directories))
for _, directory := range store.Directories {
numbers = append(numbers, directory.Number)
}
return numbers
}
// wantDirectory returns the Directory that parseStore should build for objects.
func wantDirectory(number int, name string, objects ...*Object) *Directory {
return &Directory{
Number: number,
Name: name,
Objects: objects,
FileGroups: newFileGroups(objects),
}
}
func assertPaths(t *testing.T, what string, got []string, want ...string) {
t.Helper()
if !slices.Equal(got, want) {
t.Errorf("%s = %q, want %q", what, got, want)
}
}
func TestParseStorePutsEachFileWhereItBelongs(t *testing.T) {
t.Parallel()
card := fstest.MapFS{
"DCIM/100MSDCF/ABCD0001.ARW": {Data: make([]byte, 2000)},
"DCIM/100MSDCF/readme.txt": {Data: make([]byte, 7)},
"DCIM/101CANON/MVI_0002.MP4": {Data: make([]byte, 9000)},
"DCIM/101CANON/MVI_0002.THM": {Data: make([]byte, 100)},
"DCIM/MISC/DSC00003.JPG": {Data: make([]byte, 7)},
"DCIM/DSC00004.JPG": {Data: make([]byte, 7)},
"PRIVATE/M4ROOT/CLIP/C0002.MP4": {Data: make([]byte, 400)},
"PRIVATE/M4ROOT/CLIP/C0002M01.XML": {Data: make([]byte, 7)},
"PRIVATE/OTHER/C0003.MP4": {Data: make([]byte, 7)},
"BACKUP/DSC00005.JPG": {Data: make([]byte, 7)},
"DSC00006.JPG": {Data: make([]byte, 7)},
}
store := mustParseStore(t, card)
arw := Object{
DCFStoreRoot: testRoot,
Path: "DCIM/100MSDCF/ABCD0001.ARW",
Size: 2000,
Extension: ".ARW",
DirectoryNumber: 100,
FileNumber: 1,
fsys: card,
}
mp4 := Object{
DCFStoreRoot: testRoot,
Path: "DCIM/101CANON/MVI_0002.MP4",
Size: 9000,
Extension: ".MP4",
DirectoryNumber: 101,
FileNumber: 2,
fsys: card,
}
thm := Object{
DCFStoreRoot: testRoot,
Path: "DCIM/101CANON/MVI_0002.THM",
Size: 100,
Extension: ".THM",
DirectoryNumber: 101,
FileNumber: 2,
fsys: card,
}
clip := Object{
DCFStoreRoot: testRoot,
Path: "PRIVATE/M4ROOT/CLIP/C0002.MP4",
Size: 400,
Extension: ".MP4",
fsys: card,
}
wantDirectories := []*Directory{
wantDirectory(100, "100MSDCF", &arw),
wantDirectory(101, "101CANON", &mp4, &thm),
}
if !reflect.DeepEqual(store.Directories, wantDirectories) {
t.Errorf("Directories = %+v, want %+v", store.Directories, wantDirectories)
}
wantImages := []*Image{{Object: arw}}
if !reflect.DeepEqual(store.Images, wantImages) {
t.Errorf("Images = %+v, want %+v", store.Images, wantImages)
}
wantVideos := []*Video{{Object: mp4}, {Object: clip}}
if !reflect.DeepEqual(store.Videos, wantVideos) {
t.Errorf("Videos = %+v, want %+v", store.Videos, wantVideos)
}
assertPaths(t, "NonConforming", store.NonConforming,
"DCIM/100MSDCF/readme.txt",
"DCIM/DSC00004.JPG",
"DCIM/MISC",
"DCIM/MISC/DSC00003.JPG",
)
}
func TestParseStoreCountsOnlyImagesAndVideos(t *testing.T) {
t.Parallel()
card := fstest.MapFS{
"DCIM/100ABCDE/ABCD0001.ARW": {Data: make([]byte, 2000)},
"DCIM/100ABCDE/ABCD0002.THM": {Data: make([]byte, 100)},
"DCIM/100ABCDE/ABCD0003.MP4": {Data: make([]byte, 9000)},
"DCIM/MISC/ABCD0004.JPG": {Data: make([]byte, 50)},
"PRIVATE/M4ROOT/CLIP/C0004.MP4": {Data: make([]byte, 400)},
"ABCD0005.JPG": {Data: make([]byte, 30)},
}
store := mustParseStore(t, card)
if got := store.ImagesCount(); got != 1 {
t.Errorf("ImagesCount() = %d, want 1", got)
}
if got := store.VideosCount(); got != 2 {
t.Errorf("VideosCount() = %d, want 2", got)
}
if got := store.TotalSize(); got != 11400 {
t.Errorf("TotalSize() = %d, want 11400", got)
}
}
func TestParseStoreSetsFullFilePath(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard("DCIM/100MSDCF/DSC00001.JPG"))
want := testRoot + "/DCIM/100MSDCF/DSC00001.JPG"
if got := store.Images[0].FullFilePath(); got != want {
t.Fatalf("FullFilePath() = %q, want %q", got, want)
}
}
func TestParseNumberOfDirectoryName(t *testing.T) {
t.Parallel()
cases := map[string]int{
"100MSDCF": 100,
"999VWXYZ": 999,
"100D3500": 100,
"101abcde": 101,
"102A_B_C": 102,
"099ABCDE": 0,
"000ABCDE": 0,
"103A": 0,
"IM104ABC": 0,
"105_abcdefg": 0,
"107@-%+]": 0,
"IM08ABCD": 0,
"100ABC-E": 0,
"100ABCD": 0,
"100ABCDEF": 0,
"1000ABCDE": 0,
"X100ABCDE": 0,
"TEMP": 0,
}
for name, want := range cases {
if got := parseNumber(dcfDirectoryName, name); got != want {
t.Errorf("directory %q parsed to %d, want %d", name, got, want)
}
}
}
func TestParseNumberOfFileName(t *testing.T) {
t.Parallel()
cases := map[string]int{
"DSC00001.ARW": 1,
"ABCD9999.JPG": 9999,
"abcd0002.jpg": 2,
"_ABC0001.JPG": 1,
"IMG_0001.JPG": 1,
"12340005.JPG": 5,
"ABCD0000.JPG": 0,
"A0004.JPG": 0,
"ABCDE0005.JPG": 0,
"ABCD10000.JPG": 0,
"abcdefg0006.JPG": 0,
"+-@]0007.JPG": 0,
"ABCD00~1.JPG": 0,
"ABCDEFGH.JPG": 0,
"ABCD0001": 0,
"ABCD0001.": 0,
"ABCD0001.JP": 0,
"ABCD0001.JPEG": 0,
"ABCD0001.J-G": 0,
"._DSC0001.JPG": 0,
".DS_Store": 0,
}
for name, want := range cases {
if got := parseNumber(dcfFileName, name); got != want {
t.Errorf("file %q parsed to %d, want %d", name, got, want)
}
}
}
func TestParseStoreTreatsLowerCaseNamesAsDCFNames(t *testing.T) {
t.Parallel()
card := newCard("DCIM/101abcde/abcd0002.jpg")
store := mustParseStore(t, card)
want := []*Directory{wantDirectory(101, "101abcde", &Object{
DCFStoreRoot: testRoot,
Path: "DCIM/101abcde/abcd0002.jpg",
Extension: ".jpg",
DirectoryNumber: 101,
FileNumber: 2,
fsys: card,
})}
if !reflect.DeepEqual(store.Directories, want) {
t.Errorf("Directories = %+v, want %+v", store.Directories, want)
}
assertPaths(t, "Images", imagePaths(store), "DCIM/101abcde/abcd0002.jpg")
assertPaths(t, "NonConforming", store.NonConforming)
}
func TestParseStoreAcceptsUnderscoreInNames(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard("DCIM/100A_B_C/_ABC0001.JPG"))
assertPaths(t, "Images", imagePaths(store), "DCIM/100A_B_C/_ABC0001.JPG")
assertPaths(t, "NonConforming", store.NonConforming)
}
func TestParseStoreRejectsDirectoriesSharingANumber(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard(
"DCIM/102ABCDE/ABCD0001.JPG",
"DCIM/102PQRST/ABCD0002.JPG",
"DCIM/103ABCDE/ABCD0003.JPG",
))
if got, want := directoryNumbers(store), []int{103}; !slices.Equal(got, want) {
t.Errorf("directory numbers = %v, want %v", got, want)
}
assertPaths(t, "Images", imagePaths(store), "DCIM/103ABCDE/ABCD0003.JPG")
assertPaths(t, "NonConforming", store.NonConforming,
"DCIM/102ABCDE",
"DCIM/102ABCDE/ABCD0001.JPG",
"DCIM/102PQRST",
"DCIM/102PQRST/ABCD0002.JPG",
)
}
func TestParseStoreDoesNotCountPlainFileAsSharingADirectoryNumber(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard(
"DCIM/100ABCDE",
"DCIM/100PQRST/ABCD0001.JPG",
))
if got, want := directoryNumbers(store), []int{100}; !slices.Equal(got, want) {
t.Errorf("directory numbers = %v, want %v", got, want)
}
assertPaths(t, "NonConforming", store.NonConforming, "DCIM/100ABCDE")
}
func TestParseStoreRejectsJPGsSharingAFileNumber(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard(
"DCIM/100ABCDE/ABCD0003.JPG",
"DCIM/100ABCDE/PQRS0003.jpg",
"DCIM/101ABCDE/ABCD0003.JPG",
))
assertPaths(t, "objects", objectPaths(store.Directories[0]))
assertPaths(t, "objects", objectPaths(store.Directories[1]),
"DCIM/101ABCDE/ABCD0003.JPG",
)
assertPaths(t, "Images", imagePaths(store), "DCIM/101ABCDE/ABCD0003.JPG")
assertPaths(t, "NonConforming", store.NonConforming,
"DCIM/100ABCDE/ABCD0003.JPG",
"DCIM/100ABCDE/PQRS0003.jpg",
)
}
func TestParseStoreKeepsOtherFilesSharingAFileNumber(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard(
"DCIM/100ABCDE/DSC00004.ARW",
"DCIM/100ABCDE/DSC00004.JPG",
"DCIM/100ABCDE/MVI_0005.MP4",
"DCIM/100ABCDE/MVI_0005.THM",
))
assertPaths(t, "objects", objectPaths(store.Directories[0]),
"DCIM/100ABCDE/DSC00004.ARW",
"DCIM/100ABCDE/DSC00004.JPG",
"DCIM/100ABCDE/MVI_0005.MP4",
"DCIM/100ABCDE/MVI_0005.THM",
)
assertPaths(t, "Images", imagePaths(store),
"DCIM/100ABCDE/DSC00004.ARW",
"DCIM/100ABCDE/DSC00004.JPG",
)
assertPaths(t, "Videos", videoPaths(store), "DCIM/100ABCDE/MVI_0005.MP4")
assertPaths(t, "NonConforming", store.NonConforming)
}
func TestParseStoreKeepsARWSharingANumberWithTwoJPGs(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard(
"DCIM/100ABCDE/ABCD0004.ARW",
"DCIM/100ABCDE/ABCD0004.JPG",
"DCIM/100ABCDE/PQRS0004.JPG",
))
assertPaths(t, "objects", objectPaths(store.Directories[0]),
"DCIM/100ABCDE/ABCD0004.ARW",
)
assertPaths(t, "NonConforming", store.NonConforming,
"DCIM/100ABCDE/ABCD0004.JPG",
"DCIM/100ABCDE/PQRS0004.JPG",
)
}
func TestParseStoreDoesNotCountDirectoryAsSharingAFileNumber(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard(
"DCIM/100ABCDE/ABCD0003.JPG",
"DCIM/100ABCDE/PQRS0003.JPG/README.TXT",
))
assertPaths(t, "objects", objectPaths(store.Directories[0]),
"DCIM/100ABCDE/ABCD0003.JPG",
)
assertPaths(t, "NonConforming", store.NonConforming,
"DCIM/100ABCDE/PQRS0003.JPG",
"DCIM/100ABCDE/PQRS0003.JPG/README.TXT",
)
}
func TestParseStoreDoesNotCountDirectoryNamedLikeAVideo(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard("PRIVATE/M4ROOT/CLIP.MP4/C0001.XML"))
assertPaths(t, "Videos", videoPaths(store))
}
func TestParseStoreRejectsSubdirectoryOfDCFDirectory(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard(
"DCIM/100ABCDE/SUB/ABCD0001.JPG",
"DCIM/100ABCDE/ABCD0002.JPG/ABCD0003.JPG",
))
assertPaths(t, "objects", objectPaths(store.Directories[0]))
assertPaths(t, "Images", imagePaths(store))
assertPaths(t, "NonConforming", store.NonConforming,
"DCIM/100ABCDE/ABCD0002.JPG",
"DCIM/100ABCDE/ABCD0002.JPG/ABCD0003.JPG",
"DCIM/100ABCDE/SUB",
"DCIM/100ABCDE/SUB/ABCD0001.JPG",
)
}
func TestParseStoreListsEmptyDirectories(t *testing.T) {
t.Parallel()
store := mustParseStore(t, fstest.MapFS{
"DCIM/100EMPTY": {Mode: fs.ModeDir},
"DCIM/MISC": {Mode: fs.ModeDir},
})
if got, want := directoryNumbers(store), []int{100}; !slices.Equal(got, want) {
t.Errorf("directory numbers = %v, want %v", got, want)
}
assertPaths(t, "NonConforming", store.NonConforming, "DCIM/MISC")
}
func TestParseStoreWithoutDCIM(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard("PRIVATE/M4ROOT/CLIP/C0001.MP4"))
if len(store.Directories) != 0 {
t.Errorf("Directories = %+v, want none", store.Directories)
}
assertPaths(t, "Videos", videoPaths(store), "PRIVATE/M4ROOT/CLIP/C0001.MP4")
}
func TestParseStoreWithoutM4Root(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard("DCIM/100ABCDE/ABCD0001.JPG"))
assertPaths(t, "Images", imagePaths(store), "DCIM/100ABCDE/ABCD0001.JPG")
assertPaths(t, "Videos", videoPaths(store))
}
func TestParseStoreTreatsPlainFilesAsMissingDirectories(t *testing.T) {
t.Parallel()
store := mustParseStore(t, newCard("DCIM", "PRIVATE/M4ROOT"))
assertPaths(t, "Images", imagePaths(store))
assertPaths(t, "Videos", videoPaths(store))
assertPaths(t, "NonConforming", store.NonConforming)
}
// A real filesystem reports a lookup under a plain file differently from a
// missing path, which fstest.MapFS does not reproduce.
func TestParseStoreReadsRealFilesystemWithPlainFileNamedPrivate(t *testing.T) {
t.Parallel()
root := newCardOnDisk(t, "DCIM/100ABCDE/ABCD0001.JPG", "PRIVATE")
store := mustParseStore(t, os.DirFS(root))
assertPaths(t, "Images", imagePaths(store), "DCIM/100ABCDE/ABCD0001.JPG")
if got := store.TotalImageSize(); got != 3 {
t.Errorf("TotalImageSize() = %d, want 3", got)
}
}
// A real filesystem can hold a name with the byte 0xff, which is not valid UTF-8.
// fs.FS refuses to open such a name.
func TestParseStoreListsNamesThatFSRefuses(t *testing.T) {
t.Parallel()
root := newCardOnDisk(t,
"DCIM/100ABCDE/ABCD0001.JPG",
"DCIM/100ABCDE/A\xffDIR/INNER",
"DCIM/100ABCDE/A\xffFILE",
"DCIM/A\xffDIR/INNER",
"DCIM/A\xffFILE",
"DCIM/MISC/A\xffDIR/INNER",
"DCIM/MISC/A\xffFILE",
)
store := mustParseStore(t, os.DirFS(root))
assertPaths(t, "Images", imagePaths(store), "DCIM/100ABCDE/ABCD0001.JPG")
assertPaths(t, "NonConforming", store.NonConforming,
"DCIM/100ABCDE/A\xffDIR",
"DCIM/100ABCDE/A\xffFILE",
"DCIM/A\xffDIR",
"DCIM/A\xffFILE",
"DCIM/MISC",
"DCIM/MISC/A\xffDIR",
"DCIM/MISC/A\xffFILE",
)
}
func TestParseStoreListsDanglingSymlinks(t *testing.T) {
t.Parallel()
root := newCardOnDisk(t, "DCIM/100ABCDE/ABCD0001.JPG", "DCIM/MISC/README.TXT")
addSymlink(t, root, "DCIM/100ABCDE/ABCD0002.JPG", "absent")
addSymlink(t, root, "DCIM/100ABCDE/LINK", "absent")
addSymlink(t, root, "DCIM/101ABCDE", "absent")
addSymlink(t, root, "DCIM/LINK", "absent")
addSymlink(t, root, "DCIM/MISC/LINK", "absent")
store := mustParseStore(t, os.DirFS(root))
if got, want := directoryNumbers(store), []int{100}; !slices.Equal(got, want) {
t.Errorf("directory numbers = %v, want %v", got, want)
}
assertPaths(t, "Images", imagePaths(store), "DCIM/100ABCDE/ABCD0001.JPG")
assertPaths(t, "NonConforming", store.NonConforming,
"DCIM/100ABCDE/ABCD0002.JPG",
"DCIM/100ABCDE/LINK",
"DCIM/101ABCDE",
"DCIM/LINK",
"DCIM/MISC",
"DCIM/MISC/LINK",
"DCIM/MISC/README.TXT",
)
}
func TestParseStoreDoesNotFollowSymlinksUnderDCIM(t *testing.T) {
t.Parallel()
outside := newCardOnDisk(t, "OUTSIDE.JPG", "DIRECTORY/ABCD0009.JPG")
outsideFile := filepath.Join(outside, "OUTSIDE.JPG")
outsideDirectory := filepath.Join(outside, "DIRECTORY")
root := newCardOnDisk(t,
"DCIM/100ABCDE/ABCD0001.JPG",
"DCIM/102ABCDE/ABCD0002.JPG",
"DCIM/MISC/README.TXT",
)
// These two have DCF file names. The second has the number of a real JPG,
// which must not make that JPG shared.
addSymlink(t, root, "DCIM/100ABCDE/ABCD0003.JPG", outsideFile)
addSymlink(t, root, "DCIM/100ABCDE/PQRS0001.JPG", outsideFile)
// These two have DCF directory names. The second has the number of a real
// directory, which must not make that directory shared.
addSymlink(t, root, "DCIM/101ABCDE", outsideDirectory)
addSymlink(t, root, "DCIM/102PQRST", outsideDirectory)
addSymlink(t, root, "DCIM/LINK", outsideDirectory)
addSymlink(t, root, "DCIM/MISC/LINK", outsideDirectory)
store := mustParseStore(t, os.DirFS(root))
if got, want := directoryNumbers(store), []int{100, 102}; !slices.Equal(got, want) {
t.Errorf("directory numbers = %v, want %v", got, want)
}
assertPaths(t, "Images", imagePaths(store),
"DCIM/100ABCDE/ABCD0001.JPG",
"DCIM/102ABCDE/ABCD0002.JPG",
)
if got := store.TotalImageSize(); got != 6 {
t.Errorf("TotalImageSize() = %d, want 6", got)
}
assertPaths(t, "NonConforming", store.NonConforming,
"DCIM/100ABCDE/ABCD0003.JPG",
"DCIM/100ABCDE/PQRS0001.JPG",
"DCIM/101ABCDE",
"DCIM/102PQRST",
"DCIM/LINK",
"DCIM/MISC",
"DCIM/MISC/LINK",
"DCIM/MISC/README.TXT",
)
}
func TestParseStoreSkipsSymlinksAndNamesThatFSRefusesUnderM4Root(t *testing.T) {
t.Parallel()
outside := newCardOnDisk(t, "OUTSIDE.MP4")
// The names with the byte 0xff sort before CLIP, so a walk that stops at one
// of them does not reach it.
root := newCardOnDisk(t,
"PRIVATE/M4ROOT/A\xffDIR/C0002.MP4",
"PRIVATE/M4ROOT/A\xffFILE.MP4",
"PRIVATE/M4ROOT/CLIP/C0001.MP4",
)
addSymlink(t, root, "PRIVATE/M4ROOT/CLIP/C0003.MP4",
filepath.Join(outside, "OUTSIDE.MP4"))
addSymlink(t, root, "PRIVATE/M4ROOT/CLIP/C0004.MP4", "absent")
addSymlink(t, root, "PRIVATE/M4ROOT/LINK", outside)
store := mustParseStore(t, os.DirFS(root))
assertPaths(t, "Videos", videoPaths(store), "PRIVATE/M4ROOT/CLIP/C0001.MP4")
if got := store.TotalVideoSize(); got != 3 {
t.Errorf("TotalVideoSize() = %d, want 3", got)
}
}
func TestParseStoreTreatsSymlinksAsMissingDirectories(t *testing.T) {
t.Parallel()
outside := newCardOnDisk(t,
"100ABCDE/ABCD0001.JPG",
"M4ROOT/C0001.MP4",
"C0002.MP4",
)
cases := []struct{ link, target string }{
{dcimDirectory, outside},
{dcimDirectory, dcimDirectory},
{privateDirectory, outside},
{privateDirectory, privateDirectory},
{m4RootDirectory, outside},
}
for _, testCase := range cases {
root := newMountPoint(t, filepath.Dir(testCase.link))
addSymlink(t, root, testCase.link, testCase.target)
store := mustParseStore(t, os.DirFS(root))
if !reflect.DeepEqual(store, &Store{RootDirectory: testRoot}) {
t.Errorf("symlink at %s to %s read as %+v, want an empty store",
testCase.link, testCase.target, store)
}
}
}
// failingFS is a card on which one call fails for one path, the way a failing
// card does. The call is Open, Lstat, ReadDir or Info.
type failingFS struct {
fstest.MapFS
call string
name string
}
func (f failingFS) Open(name string) (fs.File, error) {
err := f.failure("Open", name)
if err != nil {
return nil, err
}
return f.MapFS.Open(name)
}
func (f failingFS) Lstat(name string) (fs.FileInfo, error) {
err := f.failure("Lstat", name)
if err != nil {
return nil, err
}
return f.MapFS.Lstat(name)
}
func (f failingFS) ReadDir(name string) ([]fs.DirEntry, error) {
err := f.failure("ReadDir", name)
if err != nil {
return nil, err
}
entries, err := f.MapFS.ReadDir(name)
if err != nil {
return nil, err
}
for i, entry := range entries {
err = f.failure("Info", path.Join(name, entry.Name()))
if err != nil {
entries[i] = failingEntry{DirEntry: entry, err: err}
}
}
return entries, nil
}
func (f failingFS) failure(call, name string) error {
if call == f.call && name == f.name {
return &fs.PathError{Op: call, Path: name, Err: fs.ErrPermission}
}
return nil
}
// failingEntry is a directory entry whose Info always returns err.
type failingEntry struct {
fs.DirEntry
err error
}
func (e failingEntry) Info() (fs.FileInfo, error) {
return nil, e.err
}
func TestParseStoreReturnsReadErrorsNamingTheCard(t *testing.T) {
t.Parallel()
card := newCard(
"DCIM/100ABCDE/ABCD0001.JPG",
"DCIM/100ABCDE/SUB/README.TXT",
"DCIM/MISC/SUB/README.TXT",
"PRIVATE/M4ROOT/CLIP/C0001.MP4",
)
cases := map[string][]string{
"Lstat": {dcimDirectory, privateDirectory, m4RootDirectory},
"ReadDir": {
dcimDirectory,
"DCIM/100ABCDE",
"DCIM/100ABCDE/SUB",
"DCIM/MISC",
"DCIM/MISC/SUB",
m4RootDirectory,
"PRIVATE/M4ROOT/CLIP",
},
"Info": {"DCIM/100ABCDE/ABCD0001.JPG", "PRIVATE/M4ROOT/CLIP/C0001.MP4"},
}
for call, names := range cases {
for _, name := range names {
fsys := failingFS{MapFS: card, call: call, name: name}
_, err := parseStore(t.Context(), fsys, testRoot)
if !errors.Is(err, fs.ErrPermission) {
t.Errorf("%s of %s: error = %v, want %v",
call, name, err, fs.ErrPermission)
continue
}
want := "reading the card at " + testRoot + ": " +
call + " " + name + ": permission denied"
if err.Error() != want {
t.Errorf("error = %q, want %q", err.Error(), want)
}
}
}
}
// cancelingFS is a card that cancels the read when the read reaches the named
// directory. It records every directory read in read.
type cancelingFS struct {
fstest.MapFS
name string
cancel context.CancelFunc
read *[]string
}
func (f cancelingFS) ReadDir(name string) ([]fs.DirEntry, error) {
*f.read = append(*f.read, name)
if name == f.name {
f.cancel()
}
return f.MapFS.ReadDir(name)
}
func TestParseStoreStopsBeforeTheNextDirectoryWhenCancelled(t *testing.T) {
t.Parallel()
card := newCard(
"DCIM/100ABCDE/SUB/README.TXT",
"DCIM/OTHER/SUB/README.TXT",
"PRIVATE/M4ROOT/CLIP/SUB/C0001.MP4",
)
// Each of these directories holds another directory, which must not be read.
names := []string{
dcimDirectory,
"DCIM/100ABCDE",
"DCIM/OTHER",
m4RootDirectory,
"PRIVATE/M4ROOT/CLIP",
}
for _, name := range names {
ctx, cancel := context.WithCancel(t.Context())
var read []string
fsys := cancelingFS{MapFS: card, name: name, cancel: cancel, read: &read}
_, err := parseStore(ctx, fsys, testRoot)
if !errors.Is(err, context.Canceled) {
t.Errorf("cancelled at %s: error = %v, want %v",
name, err, context.Canceled)
}
if read[len(read)-1] != name {
t.Errorf("cancelled at %s: read %q", name, read)
}
}
}
func TestParseStoreReadsNoDirectoryWhenAlreadyCancelled(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(t.Context())
cancel()
cards := []fstest.MapFS{
newCard("DCIM/README.TXT"),
newCard("PRIVATE/M4ROOT/C0001.MP4"),
}
for _, card := range cards {
var read []string
// The context is already cancelled, so this needs no name to cancel at.
fsys := cancelingFS{MapFS: card, read: &read}
_, err := parseStore(ctx, fsys, testRoot)
if !errors.Is(err, context.Canceled) {
t.Errorf("error = %v, want %v", err, context.Canceled)
}
assertPaths(t, "directories read", read)
}
}
+77 -4
View File
@@ -2,14 +2,26 @@
# script/bootstrap: install all dependencies needed to build and develop # script/bootstrap: install all dependencies needed to build and develop
# this repo. Idempotent: every install is guarded by a check so already # this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew, # installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes nothing is present. Node is # or apk (detected in that order); assumes nothing is present. Go is
# used directly if installed; otherwise it is installed at a pinned # installed at the version the Dockerfile's Go image carries, from the
# version via nvm (installing nvm itself first, from a hash-verified # official release archive, into ~/.local/go, and then the Go modules.
# release archive, never curl | sh). # Node is used directly if installed; otherwise it is installed at a
# pinned version via nvm (installing nvm itself first, from a
# hash-verified release archive, never curl | sh).
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the Go in the Dockerfile's golang image; the sha256 of each
# archive is in install_go.
GO_VERSION="1.27.3"
# This script cannot change its caller's PATH, so script/fmt,
# script/fmt-check and script/build put this directory first on their
# own PATH, as is done here.
GO_DIR="$HOME/.local/go"
PATH="$GO_DIR/bin:$PATH"
# Pinned versions, 2026-07-06 # Pinned versions, 2026-07-06
NODE_VERSION="22.17.0" NODE_VERSION="22.17.0"
NVM_VERSION="0.40.3" NVM_VERSION="0.40.3"
@@ -39,6 +51,9 @@ detect_pkgmgr() {
if [ "$(id -u)" != "0" ]; then if [ "$(id -u)" != "0" ]; then
SUDO="sudo" SUDO="sudo"
fi fi
# This runs once, before the first install: a fresh host or
# runner image has no package lists yet.
$SUDO apt-get update
fi fi
} }
@@ -72,6 +87,52 @@ verify_sha256() {
fi fi
} }
# True when the go first on PATH reports exactly GO_VERSION. No go, a go
# that fails, or any other output is a mismatch.
go_version_matches() {
out="$(go version 2>/dev/null)" || return 1
case "$out" in
"go version go$GO_VERSION "*) return 0 ;;
*) return 1 ;;
esac
}
install_go() {
# sha256 of the go1.27.3 archives at https://go.dev/dl/, 2026-11-02
case "$(uname -s)-$(uname -m)" in
Linux-x86_64)
platform="linux-amd64"
sha256="65efbc80fdc1fa2cd2631cad5b481777a68e8725f34e89f50fc6ac714e379b40"
;;
Linux-aarch64)
platform="linux-arm64"
sha256="a06e99309dbcbdca1a8ccd2282c3ee146e55930886e26b4f1193269587165f49"
;;
Darwin-x86_64)
platform="darwin-amd64"
sha256="b6f8e931e4ab497fcde6da8ae4499a58f9d258e55c8a80824a516eae5950a750"
;;
Darwin-arm64)
platform="darwin-arm64"
sha256="dd3908cf20959d798c1e06ebe5b72213f0505ce01373b40ce3110975a6cd99c2"
;;
*)
echo "bootstrap: no Go archive pinned for $(uname -s) $(uname -m)" >&2
exit 1
;;
esac
if missing curl; then pkg_install curl curl curl curl; fi
tmp="$(mktemp -d)"
curl -fsSL -o "$tmp/go.tar.gz" \
"https://go.dev/dl/go${GO_VERSION}.${platform}.tar.gz"
verify_sha256 "$tmp/go.tar.gz" "$sha256"
# An archive unpacked over an older Go leaves a broken tree.
rm -rf "$GO_DIR"
mkdir -p "$GO_DIR"
tar -xzf "$tmp/go.tar.gz" -C "$GO_DIR" --strip-components=1
rm -rf "$tmp"
}
# nvm is a bash script; run a command in a bash with nvm loaded # nvm is a bash script; run a command in a bash with nvm loaded
nvm_sh() { nvm_sh() {
bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*" bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*"
@@ -126,6 +187,18 @@ main() {
if missing make; then pkg_install gnumake make make make; fi if missing make; then pkg_install gnumake make make make; fi
if missing git; then pkg_install git git git git; fi if missing git; then pkg_install git git git git; fi
if ! go_version_matches; then
install_go
hash -r
if ! go_version_matches; then
echo "bootstrap: $(command -v go) is not go$GO_VERSION after installing it" >&2
exit 1
fi
fi
go version
go mod download
ensure_node ensure_node
ensure_yarn ensure_yarn
install_js_deps install_js_deps
+6 -2
View File
@@ -1,11 +1,15 @@
#!/bin/sh #!/bin/sh
# script/build: compile every package. dcf is a library, so this checks # script/build: compile every package. dcf is a library, so this checks
# that it compiles and writes no binary. It needs Go on the host, and the # that it compiles and writes no binary. It needs the Go that
# final stage of the Dockerfile runs it too. # script/bootstrap installs, and the final stage of the Dockerfile runs
# it too.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Where script/bootstrap installs Go; it cannot put it on our PATH.
PATH="$HOME/.local/go/bin:$PATH"
main() { main() {
cd "$ROOT" cd "$ROOT"
CGO_ENABLED=0 go build ./... CGO_ENABLED=0 go build ./...
+7 -1
View File
@@ -1,9 +1,13 @@
#!/bin/sh #!/bin/sh
# script/fmt: format all files (writes). # script/fmt: format all files (writes): the Go code with go fmt, then
# the Markdown files with prettier.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Where script/bootstrap installs Go; it cannot put it on our PATH.
PATH="$HOME/.local/go/bin:$PATH"
# Must match the pin in script/bootstrap. # Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0" NODE_VERSION="22.17.0"
@@ -25,6 +29,8 @@ run_yarn() {
main() { main() {
cd "$ROOT" cd "$ROOT"
# Before prettier, because run_yarn replaces this shell with yarn.
go fmt ./...
run_yarn run prettier --write '**/*.md' --tab-width 4 --prose-wrap always run_yarn run prettier --write '**/*.md' --tab-width 4 --prose-wrap always
} }
+13 -1
View File
@@ -1,9 +1,13 @@
#!/bin/sh #!/bin/sh
# script/fmt-check: check formatting (read-only). # script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Where script/bootstrap installs Go; it cannot put it on our PATH.
PATH="$HOME/.local/go/bin:$PATH"
# Must match the pin in script/bootstrap. # Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0" NODE_VERSION="22.17.0"
@@ -25,6 +29,14 @@ run_yarn() {
main() { main() {
cd "$ROOT" cd "$ROOT"
# Before prettier, because run_yarn replaces this shell with yarn. On
# its own line so that set -e stops here if gofmt fails or is missing.
unformatted="$(gofmt -l .)"
if [ -n "$unformatted" ]; then
echo "fmt-check: run make fmt; gofmt would change:" >&2
echo "$unformatted" >&2
exit 1
fi
run_yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always run_yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always
} }