dcf/README.md

# dcf

## Description

`dcf` is a WTFPL-licensed Go library by [@sneak](https://sneak.berlin) that
reads DCF memory cards from digital cameras and copies the files off them.

DCF is the "Design rule for Camera File system". It is also JEITA specification
number CP-3461. JEITA is the Japan Electronics and Information Technology
Industries Association. Wikipedia describes it at
[Design rule for Camera File system](https://en.wikipedia.org/wiki/Design_rule_for_Camera_File_system).

The DCF specification is why your digital camera puts images and videos in
`DCIM` and `PRIVATE/M4ROOT` directories on the memory card.

`dcf` is incomplete and under development. It does not work yet.

## Getting Started

Add the library to your Go module:

```sh
go get sneak.berlin/go/dcf
```

Then list the DCF stores on the mounted filesystems:

```go
package main

import (
	"context"
	"fmt"
	"log"

	"sneak.berlin/go/dcf/pkg/dcf"
)

func main() {
	stores, err := dcf.GetStores(context.Background())
	if err != nil {
		log.Fatal(err)
	}

	for _, store := range stores {
		fmt.Println(store.String())
	}
}
```

Call `dcf.GetStore` instead when you expect exactly one DCF store. It returns
`dcf.ErrNoStore` when none is mounted, and `dcf.ErrMultipleStores` when more
than one is.

To copy the files off a card, call `Import` with a destination directory:

```go
package main

import (
	"context"
	"fmt"
	"log"

	"sneak.berlin/go/dcf/pkg/dcf"
)

func main() {
	ctx := context.Background()

	store, err := dcf.GetStore(ctx)
	if err != nil {
		log.Fatal(err)
	}

	result, err := store.Import(ctx, "/path/to/photos")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("copied:", result.Copied)
	fmt.Println("skipped:", result.Skipped)
	fmt.Println("conflicts:", result.Conflicts)
}
```

To work on the library itself, you need docker and git:

```sh
git clone git@git.eeqj.de:sneak/dcf.git
cd dcf
script/setup
make check
```

## Entrypoints

This repo adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard. Each entrypoint is a POSIX sh script in `script/`. Use these instead
of running the underlying tools directly.

- `script/bootstrap` (`make bootstrap`) installs the tools the host needs. It
  does not install docker or the linter. It installs:
    - `make` and git
    - Go into `~/.local/go`, at the version in the `Dockerfile`, unless the `go`
      on the `PATH` is already that version
    - the Go modules
    - node and yarn
    - `prettier`
- `script/setup` (`make setup`) runs `script/bootstrap` and then installs the
  git pre-commit hook.
- `script/test` (`make test`) runs the tests by building the `test` stage of the
  `Dockerfile`.
- `script/lint` (`make lint`) checks the Go code by building the `lint` stage of
  the `Dockerfile`.
- `script/fmt` (`make fmt`) rewrites the Go code with `go fmt` and the Markdown
  files with `prettier`.
- `script/fmt-check` (`make fmt-check`) checks the same formatting and changes
  nothing.
- `script/check` (`make check`) runs the tests, then the linter, then the
  formatting check.
- `script/build` (`make build`) compiles every package. It needs the Go that
  `script/bootstrap` installs.
- `script/docker` (`make docker`) builds the development image, tagged `dcf`.
- `script/cibuild` runs the whole CI build. The Gitea workflow runs it on every
  push.
- `script/precommit` is what the git pre-commit hook runs. It runs
  `script/check`.
- `script/install-precommit` (`make hooks`) installs the git pre-commit hook.
- `script/projectname` prints the project name for the other scripts.

The development image cannot be built unless the `lint` and `test` stages pass.
Run `docker run --rm -it dcf` for a shell inside it, with Go and `make`.

## Rationale

I wanted to copy images off my memory cards for processing. I also like
overengineering and reusable code.

## Design

All the code is in the `pkg/dcf` package. The repo has no commands.

- `dcf.go` holds the public API for reading a card. `GetStores` finds the DCF
  stores and reads each one. `GetStore` returns the one DCF store when exactly
  one is mounted.
- `helpers.go` finds the mount points that hold a DCF store.
- `parse.go` reads one DCF store through an `fs.FS` rooted at its mount point.
- `filegroup.go` sorts the files of one DCF directory into file groups.
- `import.go` holds `Store.Import`, which copies files off a card.

The main types are:

- `Store` is one DCF store on one mounted filesystem.
- `Directory` is one DCF directory. It has a directory number, a name, the
  objects inside it and their file groups.
- `Object` is one file in a store. A file in a DCF directory also has a
  directory number and a file number. `Object.ModTime` is its modification time.
  `Object.Hash` reads the file through the `fs.FS` of its store. An `Object` you
  build yourself has none, so its `Hash` returns an error.
- `FileGroup` is one file group in a DCF directory.
- `Image` and `Video` embed `Object`. The file extension decides which one a
  file becomes. A file with any other extension, such as a `THM` thumbnail, is
  neither.
- `ImportResult` lists what `Store.Import` did with each file.

A mounted filesystem is a DCF store when its root holds a `DCIM` or
`PRIVATE/M4ROOT` directory. It needs only one of them. `dcf` never mounts a
filesystem. Media anywhere else on the filesystem is not part of the store.

`dcf` never follows a symlink.

`GetStores` and `GetStore` take a `context.Context`. When it is cancelled, the
read stops before the next directory. The returned error matches
`context.Canceled`.

`dcf` applies the naming rules of the DCF specification, CIPA DC-009-2010, under
`DCIM`:

- A DCF directory sits directly under `DCIM`. Its name is a directory number
  from 100 to 999, then five characters, such as `100MSDCF`.
- A DCF file sits directly in a DCF directory. Its name is four characters, then
  a file number from 0001 to 9999, then a dot and a three-character extension,
  such as `DSC00001.ARW`.
- The characters allowed are `A` to `Z`, `0` to `9` and `_`. Lower-case letters
  count as upper-case.
- Directories that share a directory number are not DCF directories.
- `JPG` files that share a file number in one directory are not DCF files.

`Store.NonConforming` lists the path of everything else under `DCIM`. It
includes everything inside a directory that breaks the rules. It also includes
every symlink, whatever its name. These entries have no DCF numbers, and they
are not counted as images or videos.

The naming rules do not apply under `PRIVATE/M4ROOT`. The images and videos
there are in `Store.Images` and `Store.Videos`, and their DCF numbers are 0.
`dcf` skips any symlink there.

A file group is made of the DCF files in one DCF directory that share a file
number. The specification calls it a DCF object. A RAW file and its JPEG form a
file group, and so do a video and its `THM` thumbnail.

- The first four characters and the extension can differ within a group.
- A DCF file that shares its number with no other file is a group of one.
- Files in different DCF directories are never in the same group.
- Only DCF files are grouped. Files in `Store.NonConforming` and under
  `PRIVATE/M4ROOT` are in no group.

`Directory.FileGroups` lists the groups in file number order. Each `FileGroup`
has:

- `DirectoryNumber` and `FileNumber`
- `Members`, which is every file in the group
- `Primary`, which is the member that stands for the whole group
- `MembersCount`, which returns the number of members
- `TotalSize`, which returns the total size of the members in bytes

The primary member is the first of these that the group holds:

1. a `JPG` file
2. any other file in `Store.Images`
3. a file in `Store.Videos`
4. any other file except a `THM`
5. a `THM`

`Store.Images` and `Store.Videos` list each file on its own, whatever its group.

`Store.Import` copies files off a card to a destination directory. It verifies
each copy against the SHA256 hash of its source. It only reads the card, so it
never deletes anything there.

It copies:

- every member of every file group
- the images and videos under `PRIVATE/M4ROOT`

It does not copy the files in `Store.NonConforming`.

Each file goes to `YYYY-MM-DD/<file name>` under the destination directory. The
date is the day of the file's modification time, in the computer's time zone.
Every member of a file group goes to the day of its primary member, so a group
never splits across two folders. Each copy keeps the modification time of its
source. Only the user who ran the import can read the copies.

`ImportResult` lists each destination path in one of three lists:

- `Copied` holds the copies this import wrote and verified.
- `Skipped` holds the paths that already held a file with the same SHA256 hash.
- `Conflicts` holds the paths that already held anything else. `dcf` leaves that
  alone and does not copy the file.

A conflict is not an error, so a caller must check `Conflicts`. These are the
usual causes:

- Two different files have the same name and the same day. Two cards can reuse a
  file number, and so can two DCF directories on one card. The first file is
  copied, and the second is a conflict.
- A partial file remains because the process died during a copy. Delete it and
  import again.

A copy that fails or does not match the hash of its source is removed. One file
that fails does not stop the others. The returned error joins the failures, each
naming its file. The result still lists what was done.

When the context is cancelled, the import stops at once, even inside a file, and
removes the partial copy. The returned error matches `context.Canceled`.

## TODO

No open items.

## License

WTFPL. See [LICENSE](LICENSE).

## Author

[@sneak](https://sneak.berlin) <[sneak@sneak.berlin](mailto:sneak@sneak.berlin)>