Write each backed-up original once and skip the precache (closes #106) #124
@@ -510,22 +510,25 @@ symlink you put there stays, and a directory that still holds one after its
|
|||||||
symlinks are removed stays too, with its JSON.
|
symlinks are removed stays too, with its JSON.
|
||||||
|
|
||||||
Each file is downloaded exactly once regardless of how many collections it
|
Each file is downloaded exactly once regardless of how many collections it
|
||||||
appears in. On subsequent runs, existing originals are skipped. If a download
|
appears in, and written once: straight into `originals/`, with no copy left in
|
||||||
fails, the error is logged and the backup continues with the next file. The exit
|
the cache. An original the cache already held is copied from there instead. On
|
||||||
code is non-zero if any files failed.
|
subsequent runs, existing originals are skipped. If a download fails, the error
|
||||||
|
is logged and the backup continues with the next file. The exit code is non-zero
|
||||||
|
if any files failed. `quak backup` opens its library with the thumbnail and
|
||||||
|
originals precache off, so it fetches only what the backup stores.
|
||||||
|
|
||||||
Each original is copied to a temporary file named
|
Each original is written to a temporary file in the same directory, synced to
|
||||||
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp` in the same directory, synced
|
disk, and renamed into place, so an original is either complete or absent, even
|
||||||
to disk, and renamed into place, so an original is either complete or absent,
|
after a power cut. A downloaded original's temporary file is named
|
||||||
even after a power cut. A run that is killed can leave one of these temporary
|
`.quak-<pid>-<random>.tmp`, one copied from the cache
|
||||||
files behind; the next backup deletes those whose process is no longer running.
|
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp`. A run that is killed can leave
|
||||||
Downloads and the content cache use the same scheme with
|
one of these temporary files behind; the next backup deletes those whose process
|
||||||
`.quak-<pid>-<random>.tmp` names, and opening a library deletes those in the
|
is no longer running. The content cache uses the same scheme, and opening a
|
||||||
cache whose process is no longer running, so a download another process has in
|
library deletes the temporary files in the cache whose process is no longer
|
||||||
progress in the same cache is left alone. The rename replaces whatever was at
|
running, so a download another process has in progress in the same cache is left
|
||||||
the destination rather than writing through it: a symlink there is replaced, not
|
alone. The rename replaces whatever was at the destination rather than writing
|
||||||
followed, and the new file has the temporary file's permissions, not those of
|
through it: a symlink there is replaced, not followed, and the new file has the
|
||||||
the file it replaced.
|
temporary file's permissions, not those of the file it replaced.
|
||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
@@ -667,12 +670,15 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
|||||||
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
||||||
a query vector the caller produced elsewhere.
|
a query vector the caller produced elsewhere.
|
||||||
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every
|
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every
|
||||||
in-scope original (and, with `includeThumbnails`, thumbnails) through the
|
in-scope original not already in the backup (and, with `includeThumbnails`,
|
||||||
content cache, and rebuilds the on-disk backup tree with a durable failure
|
thumbnails) through the content cache, and rebuilds the on-disk backup tree
|
||||||
ledger. `BackupOptions`: `downloadDirectory` (falls back to the one `open()`
|
with a durable failure ledger. A fetched original is written straight into the
|
||||||
was given), `includeOriginals` (default `true`), `includeThumbnails` (default
|
backup's `originals/` and not into the cache, which then counts it as present;
|
||||||
`false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above for the
|
one the cache already held is copied from there. `BackupOptions`:
|
||||||
tree it writes.
|
`downloadDirectory` (falls back to the one `open()` was given),
|
||||||
|
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
|
||||||
|
`onlyAlbumNames`, and `onProgress`. See Backup layout above for the tree it
|
||||||
|
writes.
|
||||||
|
|
||||||
### Request pools
|
### Request pools
|
||||||
|
|
||||||
|
|||||||
@@ -18,6 +18,13 @@ Tag v1.0.0.
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-09-23: `quak backup` writes each original once and no longer fills the
|
||||||
|
cache (issue 106). An original fetched for a backup is written by the download
|
||||||
|
writer straight into the backup's `originals/`, and the content cache records
|
||||||
|
it there instead of keeping its own copy; one the cache already held is still
|
||||||
|
copied. `quak backup` opens its library with the thumbnail and originals
|
||||||
|
precache off.
|
||||||
|
|
||||||
- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
|
- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
|
||||||
`helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
|
`helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
|
||||||
awaits `lib.fresh()` before reading, so a file added since the cache was
|
awaits `lib.fresh()` before reading, so a file added since the cache was
|
||||||
|
|||||||
+8
-4
@@ -4,7 +4,8 @@
|
|||||||
// fails the backup before any file is touched), then, for every file in scope,
|
// fails the backup before any file is touched), then, for every file in scope,
|
||||||
// gets its original bytes onto disk under `downloadDirectory` and rebuilds the
|
// gets its original bytes onto disk under `downloadDirectory` and rebuilds the
|
||||||
// derived views (per-file sidecars, per-collection symlink trees,
|
// derived views (per-file sidecars, per-collection symlink trees,
|
||||||
// per-collection JSON) from the model. The on-disk layout is the historical one, unchanged:
|
// per-collection JSON) from the model. The on-disk layout is the historical
|
||||||
|
// one, unchanged:
|
||||||
//
|
//
|
||||||
// <downloadDirectory>/
|
// <downloadDirectory>/
|
||||||
// originals/<fileID>.<ext> the decrypted bytes
|
// originals/<fileID>.<ext> the decrypted bytes
|
||||||
@@ -96,8 +97,9 @@ export interface BackupLibrary {
|
|||||||
listCollections(): Collection[];
|
listCollections(): Collection[];
|
||||||
listFiles(collectionID: number): EnteFile[];
|
listFiles(collectionID: number): EnteFile[];
|
||||||
// Get an original's bytes onto disk through the content cache/pools,
|
// Get an original's bytes onto disk through the content cache/pools,
|
||||||
// returning where they landed (the cache, or a prior backup).
|
// returning where they landed: `destination` when they were fetched now,
|
||||||
original(fileID: number): Promise<{ path: string }>;
|
// otherwise wherever they already were (the cache, or a prior backup).
|
||||||
|
original(fileID: number, destination: string): Promise<{ path: string }>;
|
||||||
thumbnail(fileID: number): Promise<{ path: string }>;
|
thumbnail(fileID: number): Promise<{ path: string }>;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -424,7 +426,9 @@ export const runBackup = async (
|
|||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
log(`Fetching original ${file.metadata.title} (${fileID})...`);
|
log(`Fetching original ${file.metadata.title} (${fileID})...`);
|
||||||
const { path } = await lib.original(fileID);
|
// A fetched original is written straight to `dest`; only one
|
||||||
|
// that was already cached elsewhere is copied.
|
||||||
|
const { path } = await lib.original(fileID, dest);
|
||||||
await copyAtomic(path, dest);
|
await copyAtomic(path, dest);
|
||||||
downloaded++;
|
downloaded++;
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
|
|||||||
@@ -380,10 +380,14 @@ export const backupCommand = async (
|
|||||||
if (!client) return 1;
|
if (!client) return 1;
|
||||||
|
|
||||||
ctx.stderr.write("Starting backup...\n");
|
ctx.stderr.write("Starting backup...\n");
|
||||||
|
// The precache is off: the backup fetches what it needs, and must not
|
||||||
|
// also fill the cache with every thumbnail and the recent originals.
|
||||||
const lib = await Library.open({
|
const lib = await Library.open({
|
||||||
client,
|
client,
|
||||||
downloadDirectory: dir,
|
downloadDirectory: dir,
|
||||||
cacheDirectory: ctx.cacheDir,
|
cacheDirectory: ctx.cacheDir,
|
||||||
|
precacheThumbnails: false,
|
||||||
|
precacheOriginals: false,
|
||||||
});
|
});
|
||||||
try {
|
try {
|
||||||
const result = await lib.backup({
|
const result = await lib.backup({
|
||||||
|
|||||||
+21
-3
@@ -321,6 +321,23 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
|
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Get an original for a backup. One not present anywhere is written
|
||||||
|
// straight to `destination` and recorded there, so no second copy lands
|
||||||
|
// in the cache; one already present is returned where it is.
|
||||||
|
async backupOriginal(
|
||||||
|
fileID: number,
|
||||||
|
destination: string,
|
||||||
|
): Promise<ContentResult> {
|
||||||
|
const result = await this.acquire(
|
||||||
|
fileID,
|
||||||
|
"original",
|
||||||
|
"on-demand",
|
||||||
|
undefined,
|
||||||
|
{ destination },
|
||||||
|
);
|
||||||
|
return { path: result.path, bytes: result.bytes };
|
||||||
|
}
|
||||||
|
|
||||||
async ensure(args: EnsureOptions): Promise<EnsureResult[]> {
|
async ensure(args: EnsureOptions): Promise<EnsureResult[]> {
|
||||||
return this.ensureThumbnails(args);
|
return this.ensureThumbnails(args);
|
||||||
}
|
}
|
||||||
@@ -422,13 +439,14 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
|
|
||||||
// The core: return the cached path if present, else fetch through the pool,
|
// The core: return the cached path if present, else fetch through the pool,
|
||||||
// store, and return it. `cached` distinguishes a present hit (no network,
|
// store, and return it. `cached` distinguishes a present hit (no network,
|
||||||
// no download event) from a fresh fetch.
|
// no download event) from a fresh fetch. A fetched original is stored at
|
||||||
|
// `opts.destination` when given, instead of in `originalsDir`.
|
||||||
private async acquire(
|
private async acquire(
|
||||||
fileID: number,
|
fileID: number,
|
||||||
kind: Kind,
|
kind: Kind,
|
||||||
priority: Priority,
|
priority: Priority,
|
||||||
signal: AbortSignal | undefined,
|
signal: AbortSignal | undefined,
|
||||||
opts?: { onByte?: ProgressCallback },
|
opts?: { onByte?: ProgressCallback; destination?: string },
|
||||||
): Promise<{ path: string; bytes: number; cached: boolean }> {
|
): Promise<{ path: string; bytes: number; cached: boolean }> {
|
||||||
const file = this.getFile(fileID);
|
const file = this.getFile(fileID);
|
||||||
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
|
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
|
||||||
@@ -469,7 +487,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
kind === "original" ? this.originalsDir : this.thumbnailsDir;
|
kind === "original" ? this.originalsDir : this.thumbnailsDir;
|
||||||
const dest =
|
const dest =
|
||||||
kind === "original"
|
kind === "original"
|
||||||
? join(dir, originalName(file))
|
? (opts?.destination ?? join(dir, originalName(file)))
|
||||||
: join(dir, `${fileID}${THUMBNAIL_EXT}`);
|
: join(dir, `${fileID}${THUMBNAIL_EXT}`);
|
||||||
const pool =
|
const pool =
|
||||||
kind === "original" ? this.pools.content : this.pools.thumbnails;
|
kind === "original" ? this.pools.content : this.pools.thumbnails;
|
||||||
|
|||||||
@@ -571,7 +571,8 @@ export class Library {
|
|||||||
refresh: () => this.refreshNow(),
|
refresh: () => this.refreshNow(),
|
||||||
listCollections: () => this.store.listCollections(),
|
listCollections: () => this.store.listCollections(),
|
||||||
listFiles: (id) => this.store.listFiles(id),
|
listFiles: (id) => this.store.listFiles(id),
|
||||||
original: (fileID) => cache!.original(fileID),
|
original: (fileID, destination) =>
|
||||||
|
cache!.backupOriginal(fileID, destination),
|
||||||
thumbnail: (fileID) => cache!.thumbnail(fileID),
|
thumbnail: (fileID) => cache!.thumbnail(fileID),
|
||||||
},
|
},
|
||||||
{ ...opts, downloadDirectory },
|
{ ...opts, downloadDirectory },
|
||||||
|
|||||||
+29
-1
@@ -562,11 +562,39 @@ describe("lib.backup", () => {
|
|||||||
lib.close();
|
lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("fsyncs a copied original before the rename and its directory after", async () => {
|
it("fetches each original once and writes it only into the backup", async () => {
|
||||||
|
// A backup of a 500 GB account must write 500 GB, not a copy in the
|
||||||
|
// cache as well: an original fetched for the backup goes straight
|
||||||
|
// into its originals/, and the cache records it there.
|
||||||
|
const source = stubSource();
|
||||||
|
const lib = await openLibrary(source);
|
||||||
|
const outDir = join(root, "backup");
|
||||||
|
|
||||||
|
const result = await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|
||||||
|
expect(result.downloaded).toBe(3);
|
||||||
|
expect(source.originalCalls).toBe(3);
|
||||||
|
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
|
||||||
|
const stored = readdirSync(join(outDir, "originals")).filter(
|
||||||
|
(name) => !name.endsWith(".json"),
|
||||||
|
);
|
||||||
|
expect(stored.sort()).toEqual(["100.jpg", "101.jpg", "200.png"]);
|
||||||
|
// The cache counts the backup's copy as present: reading the
|
||||||
|
// original afterwards fetches nothing and answers with that copy.
|
||||||
|
const read = await lib.photos.byID({ fileID: 100 })!.original();
|
||||||
|
expect(read.path).toBe(join(outDir, "originals", "100.jpg"));
|
||||||
|
expect(source.originalCalls).toBe(3);
|
||||||
|
await lib.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("fsyncs an original copied from the cache before the rename and its directory after", async () => {
|
||||||
const lib = await openLibrary(stubSource());
|
const lib = await openLibrary(stubSource());
|
||||||
const outDir = join(root, "backup");
|
const outDir = join(root, "backup");
|
||||||
const originals = join(outDir, "originals");
|
const originals = join(outDir, "originals");
|
||||||
const dest = join(originals, "100.jpg");
|
const dest = join(originals, "100.jpg");
|
||||||
|
// Only an original already in the cache is copied into the backup;
|
||||||
|
// one fetched for the backup is written there by the download writer.
|
||||||
|
await lib.photos.byID({ fileID: 100 })!.original();
|
||||||
fsEvents.length = 0;
|
fsEvents.length = 0;
|
||||||
|
|
||||||
await lib.backup({ downloadDirectory: outDir });
|
await lib.backup({ downloadDirectory: outDir });
|
||||||
|
|||||||
@@ -13,6 +13,7 @@
|
|||||||
import {
|
import {
|
||||||
existsSync,
|
existsSync,
|
||||||
mkdtempSync,
|
mkdtempSync,
|
||||||
|
readdirSync,
|
||||||
readFileSync,
|
readFileSync,
|
||||||
rmSync,
|
rmSync,
|
||||||
statSync,
|
statSync,
|
||||||
@@ -427,6 +428,16 @@ describe("backup", () => {
|
|||||||
expect(stdout.text).toBe("");
|
expect(stdout.text).toBe("");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// The backup opens its library with the precache off: it fetches the
|
||||||
|
// originals it needs into the backup, and must not also fetch every
|
||||||
|
// thumbnail in the account, or keep originals, in the per-user cache.
|
||||||
|
it("leaves nothing in the cache's originals and thumbnails", async () => {
|
||||||
|
const dir = join(root, "backup");
|
||||||
|
expect(await backupCommand(context(), dir, {})).toBe(0);
|
||||||
|
expect(readdirSync(join(root, "cache", "thumbnails"))).toEqual([]);
|
||||||
|
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
it("exits 1 and lists the file when one download fails", async () => {
|
it("exits 1 and lists the file when one download fails", async () => {
|
||||||
const ctx = context(fakeClient({ failID: 101 }));
|
const ctx = context(fakeClient({ failID: 101 }));
|
||||||
expect(await backupCommand(ctx, join(root, "backup"), {})).toBe(1);
|
expect(await backupCommand(ctx, join(root, "backup"), {})).toBe(1);
|
||||||
|
|||||||
Reference in New Issue
Block a user