dcf/README.md

dcf

Description

dcf is a WTFPL-licensed Go library by @sneak that reads DCF memory cards from digital cameras and copies the files off them.

DCF is the "Design rule for Camera File system". It is also JEITA specification number CP-3461. JEITA is the Japan Electronics and Information Technology Industries Association. Wikipedia describes it at Design rule for Camera File system.

The DCF specification is why your digital camera puts images and videos in DCIM and PRIVATE/M4ROOT directories on the memory card.

dcf is incomplete and under development. It does not work yet.

Getting Started

Add the library to your Go module:

go get sneak.berlin/go/dcf

Then list the DCF stores on the mounted filesystems:

package main

import (
	"context"
	"fmt"
	"log"

	"sneak.berlin/go/dcf/pkg/dcf"
)

func main() {
	stores, err := dcf.GetStores(context.Background())
	if err != nil {
		log.Fatal(err)
	}

	for _, store := range stores {
		fmt.Println(store.String())
	}
}

Call dcf.GetStore instead when you expect exactly one DCF store. It returns dcf.ErrNoStore when none is mounted, and dcf.ErrMultipleStores when more than one is.

To copy the files off a card, call Import with a destination directory:

package main

import (
	"context"
	"fmt"
	"log"

	"sneak.berlin/go/dcf/pkg/dcf"
)

func main() {
	ctx := context.Background()

	store, err := dcf.GetStore(ctx)
	if err != nil {
		log.Fatal(err)
	}

	result, err := store.Import(ctx, "/path/to/photos")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("copied:", result.Copied)
	fmt.Println("skipped:", result.Skipped)
	fmt.Println("conflicts:", result.Conflicts)
}

To work on the library itself, you need docker and git:

git clone git@git.eeqj.de:sneak/dcf.git
cd dcf
script/setup
make check

Entrypoints

This repo adheres to the Scripts to Rule Them All standard. Each entrypoint is a POSIX sh script in script/. Use these instead of running the underlying tools directly.

• script/bootstrap (make bootstrap) installs the tools the host needs. It does not install docker or the linter. It installs:
  • make and git
  • Go into ~/.local/go, at the version in the Dockerfile, unless the go on the PATH is already that version
  • the Go modules
  • node and yarn
  • prettier
• script/setup (make setup) runs script/bootstrap and then installs the git pre-commit hook.
• script/test (make test) runs the tests by building the test stage of the Dockerfile.
• script/lint (make lint) checks the Go code by building the lint stage of the Dockerfile.
• script/fmt (make fmt) rewrites the Go code with go fmt and the Markdown files with prettier.
• script/fmt-check (make fmt-check) checks the same formatting and changes nothing.
• script/check (make check) runs the tests, then the linter, then the formatting check.
• script/build (make build) compiles every package. It needs the Go that script/bootstrap installs.
• script/docker (make docker) builds the development image, tagged dcf.
• script/cibuild runs the whole CI build. The Gitea workflow runs it on every push.
• script/precommit is what the git pre-commit hook runs. It runs script/check.
• script/install-precommit (make hooks) installs the git pre-commit hook.
• script/projectname prints the project name for the other scripts.

The development image cannot be built unless the lint and test stages pass. Run docker run --rm -it dcf for a shell inside it, with Go and make.

Rationale

I wanted to copy images off my memory cards for processing. I also like overengineering and reusable code.

Design

All the code is in the pkg/dcf package. The repo has no commands.

• dcf.go holds the public API for reading a card. GetStores finds the DCF stores and reads each one. GetStore returns the one DCF store when exactly one is mounted.
• helpers.go finds the mount points that hold a DCF store.
• parse.go reads one DCF store through an fs.FS rooted at its mount point.
• filegroup.go sorts the files of one DCF directory into file groups.
• import.go holds Store.Import, which copies files off a card.

The main types are:

• Store is one DCF store on one mounted filesystem.
• Directory is one DCF directory. It has a directory number, a name, the objects inside it and their file groups.
• Object is one file in a store. A file in a DCF directory also has a directory number and a file number. Object.ModTime is its modification time. Object.Hash reads the file through the fs.FS of its store. An Object you build yourself has none, so its Hash returns an error.
• FileGroup is one file group in a DCF directory.
• Image and Video embed Object. The file extension decides which one a file becomes. A file with any other extension, such as a THM thumbnail, is neither.
• ImportResult lists what Store.Import did with each file.

A mounted filesystem is a DCF store when its root holds a DCIM or PRIVATE/M4ROOT directory. It needs only one of them. dcf never mounts a filesystem. Media anywhere else on the filesystem is not part of the store.

dcf never follows a symlink.

GetStores and GetStore take a context.Context. When it is cancelled, the read stops before the next directory. The returned error matches context.Canceled.

dcf applies the naming rules of the DCF specification, CIPA DC-009-2010, under DCIM:

• A DCF directory sits directly under DCIM. Its name is a directory number from 100 to 999, then five characters, such as 100MSDCF.
• A DCF file sits directly in a DCF directory. Its name is four characters, then a file number from 0001 to 9999, then a dot and a three-character extension, such as DSC00001.ARW.
• The characters allowed are A to Z, 0 to 9 and _. Lower-case letters count as upper-case.
• Directories that share a directory number are not DCF directories.
• JPG files that share a file number in one directory are not DCF files.

Store.NonConforming lists the path of everything else under DCIM. It includes everything inside a directory that breaks the rules. It also includes every symlink, whatever its name. These entries have no DCF numbers, and they are not counted as images or videos.

The naming rules do not apply under PRIVATE/M4ROOT. The images and videos there are in Store.Images and Store.Videos, and their DCF numbers are 0. dcf skips any symlink there.

A file group is made of the DCF files in one DCF directory that share a file number. The specification calls it a DCF object. A RAW file and its JPEG form a file group, and so do a video and its THM thumbnail.

• The first four characters and the extension can differ within a group.
• A DCF file that shares its number with no other file is a group of one.
• Files in different DCF directories are never in the same group.
• Only DCF files are grouped. Files in Store.NonConforming and under PRIVATE/M4ROOT are in no group.

Directory.FileGroups lists the groups in file number order. Each FileGroup has:

• DirectoryNumber and FileNumber
• Members, which is every file in the group
• Primary, which is the member that stands for the whole group
• MembersCount, which returns the number of members
• TotalSize, which returns the total size of the members in bytes

The primary member is the first of these that the group holds:

1. a JPG file
2. any other file in Store.Images
3. a file in Store.Videos
4. any other file except a THM
5. a THM

Store.Images and Store.Videos list each file on its own, whatever its group.

Store.Import copies files off a card to a destination directory. It verifies each copy against the SHA256 hash of its source. It only reads the card, so it never deletes anything there.

It copies:

• every member of every file group
• the images and videos under PRIVATE/M4ROOT

It does not copy the files in Store.NonConforming.

Each file goes to YYYY-MM-DD/<file name> under the destination directory. The date is the day of the file's modification time, in the computer's time zone. Every member of a file group goes to the day of its primary member, so a group never splits across two folders. Each copy keeps the modification time of its source. Only the user who ran the import can read the copies.

ImportResult lists each destination path in one of three lists:

• Copied holds the copies this import wrote and verified.
• Skipped holds the paths that already held a file with the same SHA256 hash.
• Conflicts holds the paths that already held anything else. dcf leaves that alone and does not copy the file.

A conflict is not an error, so a caller must check Conflicts. These are the usual causes:

• Two different files have the same name and the same day. Two cards can reuse a file number, and so can two DCF directories on one card. The first file is copied, and the second is a conflict.
• A partial file remains because the process died during a copy. Delete it and import again.

A copy that fails or does not match the hash of its source is removed. One file that fails does not stop the others. The returned error joins the failures, each naming its file. The result still lists what was done.

When the context is cancelled, the import stops at once, even inside a file, and removes the partial copy. The returned error matches context.Canceled.

TODO

No open items.

License

WTFPL. See LICENSE.

Author

@sneak <sneak@sneak.berlin>