sneak/quak
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions 12 Packages Projects Releases Wiki Activity

Save path ./photos/YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.fileID.ext; download() from the cache first (closes #143)
check / check (push) Successful in 1m25s
Details

Browse Source 
Originals are saved at `{downloadDirectory}/YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.{fileID}{ext}`. The date is the photo's `takenAt` in local time, and `downloadDirectory` defaults to `./photos`, resolved when the library opens. The old `originals/` layout is gone.

`photo.download()` writes the original to `savePath`. It copies from the cache when the cache holds the original, and fetches otherwise. `lib.backup()` uses the same path and rule, and every album in `collections/` links to it. `isLocal` is true only when the original is at `savePath`.

For a file in several albums, one rule picks the copy everything uses: the most recently synced, with the lowest album ID breaking a tie.

Model: opus-5-5
This commit was merged in pull request #146.
This commit is contained in:
clawbot
2026-10-01 23:20:58 +02:00
parent 67d554fb46
commit 10e1a9ef39
16 changed files with 975 additions and 499 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/backup.ts
+112 -122
View File
@@ -2,29 +2,30 @@
//
// `lib.backup()` waits for a completed refresh of the library (a failed one
// fails the backup before any file is touched), then, for every file in scope,
// gets its original bytes onto disk under `downloadDirectory` and rebuilds the
// derived views (per-file sidecars, per-collection symlink trees,
// per-collection JSON) from the model. The on-disk layout is the historical
// one:
// puts its original at its save path under `downloadDirectory`, as
// `Photo.download()` does, and rebuilds the derived views (per-file sidecars,
// per-collection symlink trees, per-collection JSON) from the model. The
// on-disk layout:
//
// <downloadDirectory>/
// originals/<fileID>.<ext> the decrypted bytes
// originals/<fileID>.json per-file metadata sidecar
// collections/<name>/<title> symlink into ../../originals
// collections/<name>.json per-collection metadata
// failures.json durable ledger of unresolved failures
// YYYY/YYYY-MM/YYYY-MM-DD/
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path)
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar
// collections/<name>/<title> symlink to the original
// collections/<name>.json per-collection metadata
// failures.json durable ledger of unresolved failures
//
// A live photo's original is its image and its video, `<fileID>.<ext>` each
// with its own extension, and `originals/<fileID>.livephoto.json` naming them;
// its album folders link both.
// A live photo's original is its image and its video, each with its own
// extension, beside `YYYY-MM-DD.<fileID>.livephoto.json` naming them; its album
// folders link both.
//
// Crash-safety rests on two properties. Bytes are present-means-complete: an
// original appears under `originals/` only via the content layer's atomic
// original appears at its save path only via the content layer's atomic
// temp-then-rename, so a file that exists is whole and is never re-fetched — an
// interrupted run resumes by listing the directory. The derived views hold no
// unique state, so they are rebuilt every run; that repairs stale sidecars and
// missing or broken symlinks left by an earlier crash. A rebuild also removes
// the symlinks into originals/ that no longer belong to an album, and the
// interrupted run resumes by looking at the save paths. The derived views hold
// no unique state, so they are rebuilt every run; that repairs stale sidecars
// and missing or broken symlinks left by an earlier crash. A rebuild also
// removes the symlinks to originals that no longer belong to an album, and the
// directories of albums that no longer exist.
//
// Resilience (issue #8): no per-file condition aborts the run. A failed
@@ -48,24 +49,25 @@ import {
symlinkSync,
writeFileSync,
} from "node:fs";
import { copyFile, rename, rm } from "node:fs/promises";
import { basename, dirname, extname, join, relative } from "node:path";
import { dirname, extname, join, relative, resolve } from "node:path";
import { fsyncPath, removeLeftoverTempFiles } from "./download/index.js";
import { removeLeftoverTempFiles } from "./download/index.js";
import { sanitizeFileName, withExtension } from "./filename.js";
import {
nameInOriginals,
storedOriginal,
writeLivePhotoJSON,
copyAtomic,
placeOriginal,
savePath,
storedAtSavePath,
} from "./library/content.js";
import { representative } from "./library/records.js";
import type { Collection, EnteFile } from "./model/types.js";
export type ProgressCallback = (message: string) => void;
export interface BackupOptions {
// Where the backup tree lives. Required: with none, `backup()` throws
// before any network traffic. A library opened with a `downloadDirectory`
// supplies the default.
// Where the backup tree lives. `lib.backup()` defaults it to the library's
// download directory; `runBackup` with none throws before any network
// traffic.
downloadDirectory?: string;
// Fetch and store full-resolution originals. Default true.
includeOriginals?: boolean;
@@ -89,7 +91,7 @@ export interface BackupResult {
totalFiles: number;
// Originals fetched (or copied from the cache) this run.
downloaded: number;
// Originals already present and left untouched.
// Originals already at their save path and left untouched.
skipped: number;
// Files with an unresolved failure after this run (the ledger size); the
// CLI exits non-zero while this is above zero. A file can be both
@@ -107,8 +109,8 @@ export interface BackupLibrary {
listFiles(collectionID: number): EnteFile[];
// Get an original's bytes onto disk through the content cache/pools,
// returning where they landed: `destination` when they were fetched now,
// otherwise wherever they already were (the cache, or a prior backup). A
// live photo lands as its image and its video, fetched now beside
// otherwise wherever they already were (the cache, or the library's save
// path). A live photo lands as its image and its video, fetched now beside
// `destination`.
original(
fileID: number,
@@ -168,58 +170,6 @@ const classify = (err: unknown): FailureClass => {
const errorMessage = (err: unknown): string =>
err instanceof Error ? err.message : String(err);
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
// `dest` appears only once it is whole ("present means complete"). As in the
// download writer, the temp file is fsynced before the rename and the directory
// after it, so a power cut cannot leave a correctly named but short original.
// The temp name carries this process's ID so a later run can tell a leftover
// from a copy still in progress (see `removeLeftoverTempFiles`).
const copyAtomic = async (src: string, dest: string): Promise<void> => {
if (src === dest) return;
const tmp = join(
dirname(dest),
`.quak-backup-${basename(dest)}-${process.pid}-${Math.random()
.toString(36)
.slice(2)}.tmp`,
);
try {
await copyFile(src, tmp);
await fsyncPath(tmp);
// `rename` replaces the destination's directory entry: an existing
// symlink at `dest` is replaced, not followed, and the new file has
// the temp file's permissions (copied from `src`).
await rename(tmp, dest);
await fsyncPath(dirname(dest));
} finally {
await rm(tmp, { force: true });
}
};
// Put an original the library returned at `dest` in originals/, where a fresh
// fetch already wrote it. A live photo's image and video go beside `dest`: when
// they came from the cache they are copied, after removing whatever was at
// `dest` (an earlier version's ZIP of the two). Then the JSON file naming them
// is written, which is what makes the live photo count as stored.
const placeOriginal = async (
file: EnteFile,
dest: string,
got: { path: string; videoPath?: string },
): Promise<void> => {
if (got.videoPath === undefined) {
await copyAtomic(got.path, dest);
return;
}
const originalsDir = dirname(dest);
const path = join(originalsDir, basename(got.path));
const videoPath = join(originalsDir, basename(got.videoPath));
if (got.path !== path) {
await rm(dest, { force: true });
await copyAtomic(got.path, path);
await copyAtomic(got.videoPath, videoPath);
}
await writeLivePhotoJSON(originalsDir, file.id, { path, videoPath });
};
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
// non-symlink entry. Throws on failure (a directory in the way, no permission)
// so the caller records it and moves on rather than aborting the run.
@@ -291,36 +241,55 @@ const linksFor = (
}));
};
// Remove the symlinks in the album directory `dir` that point into
// `originalsDir` and are not named in `keep`. Nothing else in the directory
// is touched: anything else there was put there by the user.
// Every date folder (`YYYY/YYYY-MM/YYYY-MM-DD/`) under `root`, whether or not a
// file in this backup is saved there. A folder that cannot be read is skipped.
const dateFolders = (root: string): string[] => {
const subfolders = (dir: string, name: RegExp): string[] => {
try {
return readdirSync(dir, { withFileTypes: true })
.filter((e) => e.isDirectory() && name.test(e.name))
.map((e) => join(dir, e.name));
} catch {
return [];
}
};
return subfolders(root, /^\d{4}$/)
.flatMap((year) => subfolders(year, /^\d{4}-\d\d$/))
.flatMap((month) => subfolders(month, /^\d{4}-\d\d-\d\d$/));
};
// Whether the entry at `path` is a symlink a backup to `root` made: one to an
// original in a `YYYY/YYYY-MM/YYYY-MM-DD/` folder of `root`.
const linksToOriginal = (path: string, root: string): boolean => {
if (!lstatSync(path).isSymbolicLink()) return false;
const target = relative(root, resolve(dirname(path), readlinkSync(path)));
return /^\d{4}\/\d{4}-\d\d\/\d{4}-\d\d-\d\d\/[^/]+$/.test(target);
};
// Remove the symlinks in the album directory `dir` that point to an original
// in `root` and are not named in `keep`. Nothing else in the directory is
// touched: anything else there was put there by the user.
const removeStaleLinks = (
dir: string,
keep: Set<string>,
originalsDir: string,
root: string,
): void => {
const target = relative(dir, originalsDir);
for (const name of readdirSync(dir)) {
if (keep.has(name)) continue;
const path = join(dir, name);
if (
lstatSync(path).isSymbolicLink() &&
dirname(readlinkSync(path)) === target
) {
rmSync(path);
}
if (linksToOriginal(path, root)) rmSync(path);
}
};
// Remove the directories under `collectionsDir` that an earlier run wrote for
// an album that is gone or renamed: a directory not named in `current` with a
// `<name>.json` beside it holding an album ID, which is what a run writes. Its
// symlinks into originals/ are removed; if that leaves it empty, it and its
// JSON are deleted, otherwise both stay for what the user put there.
// symlinks to originals in `root` are removed; if that leaves it empty, it and
// its JSON are deleted, otherwise both stay for what the user put there.
const removeStaleAlbumDirs = (
collectionsDir: string,
current: Set<string>,
originalsDir: string,
root: string,
): void => {
for (const entry of readdirSync(collectionsDir, { withFileTypes: true })) {
if (!entry.isDirectory() || current.has(entry.name)) continue;
@@ -334,7 +303,7 @@ const removeStaleAlbumDirs = (
continue;
}
const dir = join(collectionsDir, entry.name);
removeStaleLinks(dir, new Set(), originalsDir);
removeStaleLinks(dir, new Set(), root);
if (readdirSync(dir).length > 0) continue;
rmdirSync(dir);
rmSync(jsonPath);
@@ -402,34 +371,48 @@ export const runBackup = async (
log("Refreshing library...");
await lib.refresh();
const originalsDir = join(downloadDirectory, "originals");
const collectionsDir = join(downloadDirectory, "collections");
const thumbnailsDir = join(downloadDirectory, "thumbnails");
mkdirSync(originalsDir, { recursive: true });
mkdirSync(collectionsDir, { recursive: true });
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
removeLeftoverTempFiles(originalsDir);
removeLeftoverTempFiles(thumbnailsDir);
for (const dir of dateFolders(downloadDirectory)) {
removeLeftoverTempFiles(dir);
}
const ledgerPath = join(downloadDirectory, "failures.json");
const ledger = loadLedger(ledgerPath);
const now = Date.now();
// Collections in scope, and the distinct files across them (a file shared
// by two albums is one original).
// by two albums is one original). Each file is the membership
// `representative` picks from all of its albums, in scope or not, so it is
// saved at the path `photo.savePath` names.
const allCollections = lib.listCollections();
const collections = allCollections.filter((c) =>
only ? only.has(c.name) : true,
);
const collectionName = new Map<number, string>();
for (const c of collections) collectionName.set(c.id, c.name);
for (const c of allCollections) collectionName.set(c.id, c.name);
const distinct = new Map<number, EnteFile>();
const memberships = new Map<number, EnteFile[]>();
const filesByCollection = new Map<number, EnteFile[]>();
for (const c of collections) {
for (const c of allCollections) {
const files = lib.listFiles(c.id);
filesByCollection.set(c.id, files);
for (const f of files) if (!distinct.has(f.id)) distinct.set(f.id, f);
for (const f of files) {
const arr = memberships.get(f.id);
if (arr) arr.push(f);
else memberships.set(f.id, [f]);
}
}
const distinct = new Map<number, EnteFile>();
for (const c of collections) {
for (const f of filesByCollection.get(c.id)!) {
if (!distinct.has(f.id)) {
distinct.set(f.id, representative(memberships.get(f.id)!));
}
}
}
const errors: BackupError[] = [];
@@ -465,25 +448,22 @@ export const runBackup = async (
failedThisRun.add(file.id);
};
// Phase 1: get the bytes. Fetch each pending original (and optional
// thumbnail) through the content cache/pools and place it under the backup
// tree; a present file is left as is.
// Phase 1: get the bytes. Put each pending original at its save path
// through the content cache/pools, as `Photo.download()` does, and fetch
// the optional thumbnails; a present file is left as is.
if (includeOriginals) {
for (const [fileID, file] of distinct) {
if (storedOriginal(originalsDir, file) !== undefined) {
if (storedAtSavePath(downloadDirectory, file) !== undefined) {
skipped++;
continue;
}
const dest = join(originalsDir, nameInOriginals(file));
try {
log(`Fetching original ${file.metadata.title} (${fileID})...`);
// A fetched original is written straight to `dest` (a live
// photo beside it); only one that was already cached elsewhere
// is copied.
await placeOriginal(
file,
dest,
await lib.original(fileID, dest),
// A fetched original is written straight to its save path (a
// live photo beside it); only one that was already cached
// elsewhere is copied.
await placeOriginal(downloadDirectory, file, (dest) =>
lib.original(fileID, dest),
);
downloaded++;
} catch (err) {
@@ -519,9 +499,10 @@ export const runBackup = async (
// Phase 2: rebuild the derived views from the model. Sidecars first, for
// every present original (this repairs stale ones).
if (includeOriginals) {
for (const [fileID, file] of distinct) {
if (storedOriginal(originalsDir, file) !== undefined) {
writeSidecar(join(originalsDir, `${fileID}.json`), file);
for (const file of distinct.values()) {
if (storedAtSavePath(downloadDirectory, file) !== undefined) {
const path = savePath(downloadDirectory, file);
writeSidecar(withExtension(path, ".json"), file);
}
}
}
@@ -543,7 +524,11 @@ export const runBackup = async (
allCollections.map((c, i) => [c.id, dirNames[i]!]),
);
try {
removeStaleAlbumDirs(collectionsDir, new Set(dirNames), originalsDir);
removeStaleAlbumDirs(
collectionsDir,
new Set(dirNames),
downloadDirectory,
);
} catch (err) {
log(`FAILED removing old album directories: ${errorMessage(err)}`);
}
@@ -553,13 +538,18 @@ export const runBackup = async (
const colDir = join(collectionsDir, colDirName);
mkdirSync(colDir, { recursive: true });
// Every album links the one original, saved from the file's entry in
// `distinct`.
const files = filesByCollection.get(c.id) ?? [];
const links = files.flatMap((f) =>
linksFor(f, storedOriginal(originalsDir, f)),
linksFor(
f,
storedAtSavePath(downloadDirectory, distinct.get(f.id)!),
),
);
const linkNames = uniqueNames(links, true);
try {
removeStaleLinks(colDir, new Set(linkNames), originalsDir);
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
} catch (err) {
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
}