dcf/README.md

dcf

Description

dcf is a WTFPL-licensed Go library by @sneak that reads DCF memory cards from digital cameras.

DCF is the "Design rule for Camera File system". It is also JEITA specification number CP-3461. JEITA is the Japan Electronics and Information Technology Industries Association. Wikipedia describes it at Design rule for Camera File system.

The DCF specification is why your digital camera puts images and videos in DCIM and PRIVATE/M4ROOT directories on the memory card.

dcf is incomplete and under development. It does not work yet.

Getting Started

Add the library to your Go module:

go get sneak.berlin/go/dcf

Then list the DCF stores on the mounted filesystems:

package main

import (
	"context"
	"fmt"
	"log"

	"sneak.berlin/go/dcf/pkg/dcf"
)

func main() {
	stores, err := dcf.GetStores(context.Background())
	if err != nil {
		log.Fatal(err)
	}

	for _, store := range stores {
		fmt.Println(store.String())
	}
}

Call dcf.GetStore instead when you expect exactly one DCF store. It returns dcf.ErrNoStore when none is mounted, and dcf.ErrMultipleStores when more than one is.

To work on the library itself, you need docker and git:

git clone git@git.eeqj.de:sneak/dcf.git
cd dcf
script/setup
make check

Entrypoints

This repo adheres to the Scripts to Rule Them All standard. Each entrypoint is a POSIX sh script in script/. Use these instead of running the underlying tools directly.

• script/bootstrap (make bootstrap) installs the tools the host needs. It does not install docker or the linter. It installs:
  • make and git
  • Go into ~/.local/go, at the version in the Dockerfile, unless the go on the PATH is already that version
  • the Go modules
  • node and yarn
  • prettier
• script/setup (make setup) runs script/bootstrap and then installs the git pre-commit hook.
• script/test (make test) runs the tests by building the test stage of the Dockerfile.
• script/lint (make lint) checks the Go code by building the lint stage of the Dockerfile.
• script/fmt (make fmt) rewrites the Go code with go fmt and the Markdown files with prettier.
• script/fmt-check (make fmt-check) checks the same formatting and changes nothing.
• script/check (make check) runs the tests, then the linter, then the formatting check.
• script/build (make build) compiles every package. It needs the Go that script/bootstrap installs.
• script/docker (make docker) builds the development image, tagged dcf.
• script/cibuild runs the whole CI build. The Gitea workflow runs it on every push.
• script/precommit is what the git pre-commit hook runs. It runs script/check.
• script/install-precommit (make hooks) installs the git pre-commit hook.
• script/projectname prints the project name for the other scripts.

The development image cannot be built unless the lint and test stages pass. Run docker run --rm -it dcf for a shell inside it, with Go and make.

Rationale

I wanted to copy images off my memory cards for processing. I also like overengineering and reusable code.

Design

All the code is in the pkg/dcf package. The repo has no commands.

• dcf.go holds the public API. GetStores finds the DCF stores and reads each one. GetStore returns the one DCF store when exactly one is mounted.
• helpers.go finds the mount points that hold a DCF store.
• parse.go reads one DCF store through an fs.FS rooted at its mount point.
• filegroup.go sorts the files of one DCF directory into file groups.

The main types are:

• Store is one DCF store on one mounted filesystem.
• Directory is one DCF directory. It has a directory number, a name, the objects inside it and their file groups.
• Object is one file in a store. A file in a DCF directory also has a directory number and a file number. Object.Hash reads the file through the fs.FS of its store. An Object you build yourself has none, so its Hash returns an error.
• FileGroup is one file group in a DCF directory.
• Image and Video embed Object. The file extension decides which one a file becomes. A file with any other extension, such as a THM thumbnail, is neither.

A mounted filesystem is a DCF store when its root holds a DCIM or PRIVATE/M4ROOT directory. It needs only one of them. dcf never mounts a filesystem. Media anywhere else on the filesystem is not part of the store.

dcf never follows a symlink.

GetStores and GetStore take a context.Context. When it is cancelled, the read stops before the next directory. The returned error matches context.Canceled.

dcf applies the naming rules of the DCF specification, CIPA DC-009-2010, under DCIM:

• A DCF directory sits directly under DCIM. Its name is a directory number from 100 to 999, then five characters, such as 100MSDCF.
• A DCF file sits directly in a DCF directory. Its name is four characters, then a file number from 0001 to 9999, then a dot and a three-character extension, such as DSC00001.ARW.
• The characters allowed are A to Z, 0 to 9 and _. Lower-case letters count as upper-case.
• Directories that share a directory number are not DCF directories.
• JPG files that share a file number in one directory are not DCF files.

Store.NonConforming lists the path of everything else under DCIM. It includes everything inside a directory that breaks the rules. It also includes every symlink, whatever its name. These entries have no DCF numbers, and they are not counted as images or videos.

The naming rules do not apply under PRIVATE/M4ROOT. The images and videos there are in Store.Images and Store.Videos, and their DCF numbers are 0. dcf skips any symlink there.

A file group is made of the DCF files in one DCF directory that share a file number. The specification calls it a DCF object. A RAW file and its JPEG form a file group, and so do a video and its THM thumbnail.

• The first four characters and the extension can differ within a group.
• A DCF file that shares its number with no other file is a group of one.
• Files in different DCF directories are never in the same group.
• Only DCF files are grouped. Files in Store.NonConforming and under PRIVATE/M4ROOT are in no group.

Directory.FileGroups lists the groups in file number order. Each FileGroup has:

• DirectoryNumber and FileNumber
• Members, which is every file in the group
• Primary, which is the member that stands for the whole group
• MembersCount, which returns the number of members
• TotalSize, which returns the total size of the members in bytes

The primary member is the first of these that the group holds:

1. a JPG file
2. any other file in Store.Images
3. a file in Store.Videos
4. any other file except a THM
5. a THM

Store.Images and Store.Videos list each file on its own, whatever its group.

TODO

• Copy files off a card, and verify each copy against its SHA256 hash.

License

WTFPL. See LICENSE.

Author

@sneak <sneak@sneak.berlin>