diff --git a/README.md b/README.md index 5a53ae7..54c4105 100644 --- a/README.md +++ b/README.md @@ -683,27 +683,41 @@ before any request with one line, `Invalid file ID: ` or `Invalid collection ID: `, and exit status 1. One bad `--file` ID refuses the whole command, so nothing is fixed. +Without `--out`, `get` writes the file to the working directory under its title, +and `get-thumb` writes the thumbnail there under `thumb_` followed by the title, +each with unsafe characters replaced. A name over 255 bytes, which most file +systems refuse, is cut short between two characters before its extension, so the +extension is kept whenever it fits. An extension too long to leave room for the +name's first character is cut short too, so the name never starts with a dot. + `get` and `get-thumb` resolve the file by ID directly, so `--collection` is accepted for backward compatibility but ignored. For a live photo, `get` writes its image and its video, each named after the title with its own extension, as -Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`). With -`--out PATH`, the image is written to `PATH` and the video beside it, with -`PATH`'s name and the video's extension; a `PATH` with the video's extension is -refused. `backup-metadata --exif` (alias `--all`) additionally fetches each -file's original through the cache and records, from it or a live photo's image, -its XMP metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is -read with [exifreader](https://github.com/mattiasw/ExifReader) from any image -format it reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's -`exif` field is exifreader's EXIF tag output: each tag by name, with its -`value`, `description` and `computed` value. An EXIF block exifreader finds but -reads no tag from is recorded, base64, as `exifRaw`, with the reason in -`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails` -and `helper fix-missing-thumbnails` take `--json` for machine-readable output. +Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`), and each name is +cut short as above once its own extension is in it. With `--out PATH`, the image +is written to `PATH` and the video beside it, with `PATH`'s name and the video's +extension; a `PATH` with the video's extension is refused. +`backup-metadata --exif` (alias `--all`) additionally fetches each file's +original through the cache and records, from it or a live photo's image, its XMP +metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is read with +[exifreader](https://github.com/mattiasw/ExifReader) from any image format it +reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's `exif` field +is exifreader's EXIF tag output: each tag by name, with its `value`, +`description` and `computed` value. An EXIF block exifreader finds but reads no +tag from is recorded, base64, as `exifRaw`, with the reason in `exifError`. +`collections`, `files`, `backup`, `helper list-missing-thumbnails` and +`helper fix-missing-thumbnails` take `--json` for machine-readable output. `backup --verify` also hashes the originals already in the backup and replaces any that do not match the content hash Ente records; one it cannot replace stays as it was and goes into `failures.json` (see "Backup layout"). +`backup-metadata ` writes `account.json` and, for each collection, a +directory `collections/-/` holding `_collection.json` and a +`.json` for each of its files. The collection's name has unsafe +characters replaced and is cut short between two characters so that the +directory's name fits in 255 bytes; the ID keeps the directories apart. + `backup-metadata` fetches ML data in requests of up to 200 files. When a request fails, the error is logged, each of its files is written with the reason in an `mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML @@ -756,9 +770,10 @@ Each original is saved at its save path, the same path `photo.savePath` gives and `photo.download()` writes (see Read surface below). The date is the photo's `takenAt` (the date set in Ente if it was edited, else its creation time) in the time zone of the machine running quak. The extension is the one in the file's -name as uploaded, case kept, or `.bin` when it has none or it holds anything but -letters and digits. When the date or the time zone changes, the next run saves -the original at its new path and leaves the old copy where it is. +name as uploaded, case kept, or `.bin` when it has none, it holds anything but +letters and digits, or it is more than 16 of them long. When the date or the +time zone changes, the next run saves the original at its new path and leaves +the old copy where it is. `account.json` holds the account's `email` and `userID`, as `backup-metadata` writes it. A collection's JSON holds its `id`, `name`, `type`, `ownerID`, @@ -851,11 +866,12 @@ gets its ID added as well: beside an album `x`, whose JSON is `x.json`, an album named `x.json` gets the directory `x.json (12)/`. A name too long for a file name, which most file systems limit to 255 bytes, gets its ID added too. A name with its ID added is cut short between two characters in the part before the ID, -just far enough that it fits. When a symlink's extension alone leaves no room -for the ID, the extension is cut short and the part before the ID keeps only the -bytes that cut leaves over. A collection's directory name also leaves room for -the `.json` its JSON adds. IDs never change, so a name stays the same from run -to run until such a clash appears or goes away. +just far enough that it fits. When a symlink's extension leaves no room for the +ID and the first character of the name, the extension is cut short, and the part +before the ID keeps that character and any bytes the cut leaves over. A +collection's directory name also leaves room for the `.json` its JSON adds. IDs +never change, so a name stays the same from run to run until such a clash +appears or goes away. A collection whose directory or JSON cannot be written, such as one whose directory name is taken by a file you put in `collections/`, does not stop the diff --git a/TODO.md b/TODO.md index 02a97ff..4dbe07f 100644 --- a/TODO.md +++ b/TODO.md @@ -17,15 +17,32 @@ pre-1.0 # Next Step -One bug is open: https://git.eeqj.de/sneak/quak/issues/205. The check of quak -against a live Ente account, https://git.eeqj.de/sneak/quak/issues/187, needs an -account only sneak has, so it waits on him. +No bug is open. Two changes sneak asked for are open: +https://git.eeqj.de/sneak/quak/issues/211 and +https://git.eeqj.de/sneak/quak/issues/212. The check of quak against a live Ente +account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak +has, so it waits on him. Tagging and releases are decided by sneak alone, and happen only when he declares one. # Completed Steps +- 2026-10-18: Names quak builds from an album name or a title fit in a file name + (https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in + `src/filename.ts` cuts a name short to a number of bytes, keeping its + extension when it fits and always its first character, and can put text right + before the extension; the backup's `withID` uses it for an ID. + `defaultFileName` there gives the name a file is saved under when the caller + gives no path: a prefix such as `thumb_` and the title with unsafe characters + replaced, cut short with it to 255 bytes, and, for a live photo's image or + video, with that part's own extension before the cut. `quak get`, + `quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without + `--out` or `outPath`. `backup-metadata` cuts an album's name short so that its + folder, `-`, fits in 255 bytes. `safeExtension` gives `.bin` for an + extension of more than 16 letters and digits, so a save path's name always + fits. + - 2026-10-18: `quak collections` no longer waits for the files of every collection (issue 210). On an empty cache it opened its library with a refresh that fetched and decrypted the files of each collection in turn, printing diff --git a/src/backup.ts b/src/backup.ts index 32a9412..d687131 100644 --- a/src/backup.ts +++ b/src/backup.ts @@ -82,6 +82,7 @@ import { describeError } from "./errors.js"; import { MAX_NAME_BYTES, sanitizeFileName, + shortenKeepingExtension, shortenToBytes, withExtension, } from "./filename.js"; @@ -289,9 +290,10 @@ const rebuildSymlink = (linkPath: string, target: string): void => { // case (two names that differ only in case are one entry on a case-insensitive // file system), or it is too long for a file name; then it gets ` ()`, // right before the extension when `beforeExtension` is set. To fit, the part -// before the ID is cut short as far as it must be. When the extension alone -// leaves no room for the ID, the extension is cut short and the part before -// the ID keeps only the bytes that cut leaves over. With `withJSON`, each +// before the ID is cut short as far as it must be. When the extension leaves +// no room for the ID and the first character of the name, the extension is +// cut short, and the part before the ID keeps that character and any bytes +// the cut leaves over (see `shortenKeepingExtension`). With `withJSON`, each // entry also has the JSON file `.json` beside it, as an album does: a // name then leaves room for `.json`, and one that is another entry's JSON name // gets its ID added too. @@ -305,11 +307,9 @@ const uniqueNames = ( const maxBytes = MAX_NAME_BYTES - (opts.withJSON ? ".json".length : 0); const withID = (id: number, name: string): string => { const tag = ` (${id})`; - const fullExt = opts.beforeExtension ? extname(name) : ""; - const stem = name.slice(0, name.length - fullExt.length); - const ext = shortenToBytes(fullExt, maxBytes - Buffer.byteLength(tag)); - const room = maxBytes - Buffer.byteLength(tag + ext); - return shortenToBytes(stem, room) + tag + ext; + return opts.beforeExtension + ? shortenKeepingExtension(name, maxBytes, tag) + : shortenToBytes(name, maxBytes - Buffer.byteLength(tag)) + tag; }; const names = entries.map((e) => e.name); const suffixed = new Set(); diff --git a/src/cli-commands.ts b/src/cli-commands.ts index 6e71734..c995432 100644 --- a/src/cli-commands.ts +++ b/src/cli-commands.ts @@ -332,8 +332,8 @@ export const getCommand = async ( const result = await photo.original(); // Default name is the file's own title, as the pre-library CLI used // (not the editedName-preferring projection title) (issue #52). - const outPath = opts.out ?? originalName(file); if (result.videoPath === undefined) { + const outPath = opts.out ?? originalName(file); copyFileSync(result.path, outPath); ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`); return 0; @@ -341,9 +341,13 @@ export const getCommand = async ( // A live photo is written as its image and its video, each named after // the title with its own extension, as Ente's clients name them. With // --out, the image goes there and the video beside it. - const imageOut = - opts.out ?? withExtension(outPath, extname(result.path)); - const videoOut = withExtension(outPath, extname(result.videoPath)); + const imageExt = extname(result.path); + const videoExt = extname(result.videoPath); + const imageOut = opts.out ?? originalName(file, imageExt); + const videoOut = + opts.out === undefined + ? originalName(file, videoExt) + : withExtension(opts.out, videoExt); if (imageOut.toLowerCase() === videoOut.toLowerCase()) { ctx.stderr.write( `File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`, diff --git a/src/cli-output.ts b/src/cli-output.ts index 2bf9b21..49cbd5d 100644 --- a/src/cli-output.ts +++ b/src/cli-output.ts @@ -9,7 +9,7 @@ // `metadata.title`, and issue #52 requires that output stay byte-identical, so // the commands shape their output from the raw `EnteFile` through here. -import { sanitizeFileName } from "./filename.js"; +import { defaultFileName } from "./filename.js"; import type { EnteFile, FileType, Microseconds } from "./model/types.js"; // One row of `quak files --json`. @@ -33,11 +33,13 @@ export const fileListRow = (file: EnteFile): FileListRow => ({ export const fileListLine = (file: EnteFile): string => `${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`; -// Default output path for `quak get` when `--out` is not given. The title comes -// from the server, so it is sanitized; `--out` is the user's and is used as is. -export const originalName = (file: EnteFile): string => - sanitizeFileName(file.metadata.title, `file-${file.id}`); +// Default output path for `quak get` when `--out` is not given, made from the +// title (see `defaultFileName`), with `ext` in place of its extension for a +// live photo's image or video; `--out` is the user's and is used as is. +export const originalName = (file: EnteFile, ext?: string): string => + defaultFileName(file.metadata.title, file.id, { ext }); -// Default output path for `quak get-thumb` when `--out` is not given. +// Default output path for `quak get-thumb` when `--out` is not given: `thumb_` +// and the title. export const thumbnailName = (file: EnteFile): string => - `thumb_${originalName(file)}`; + defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" }); diff --git a/src/download/index.ts b/src/download/index.ts index 59b510a..bfd6b7e 100644 --- a/src/download/index.ts +++ b/src/download/index.ts @@ -16,7 +16,7 @@ import { streamTagFinal, } from "../crypto/index.js"; import { TruncatedStreamError } from "../errors.js"; -import { safeExtension, sanitizeFileName, withExtension } from "../filename.js"; +import { defaultFileName, safeExtension, withExtension } from "../filename.js"; import { withRetry } from "../retry.js"; import type { ApiClient } from "../api/client.js"; import type { EnteFile } from "../model/types.js"; @@ -365,10 +365,14 @@ const openPart = async ( // name starts with `image` as the image and the one whose name starts with // `video` as the video, and refuses a ZIP holding a second of either. It is // written unpacked: each part is named `destination` with the extension -// replaced by its own entry's, and the two must differ ignoring case. When the -// file records a hash, `:` must match it, each over that -// part's own bytes. Only then are the image, then the video, renamed into -// place; on any failure neither is stored. +// replaced by its own entry's, and the two must differ ignoring case. When +// `defaultName` is set, `destination` is the name `defaultFileName` made from +// the title rather than a path the caller gave, and each part is named as +// `defaultFileName` names it with its own extension, so that the name is cut +// short to fit with that extension. When the file records a hash, +// `:` must match it, each over that part's own bytes. +// Only then are the image, then the video, renamed into place; on any failure +// neither is stored. // // The ZIP is chosen by its uploader and may expand enormously, so each part is // written as it decompresses and never held, and the ZIP is refused once the @@ -388,6 +392,7 @@ const decryptLivePhoto = async ( key: Uint8Array, onProgress: ProgressCallback | undefined, file: EnteFile, + defaultName: boolean, ): Promise => { const sliceSize = 4096; const dir = dirname(destination); @@ -486,8 +491,12 @@ const decryptLivePhoto = async ( `live photo's image and video have the same extension, ${video.ext}`, ); } - const path = withExtension(destination, image.ext); - const videoPath = withExtension(destination, video.ext); + const partPath = (ext: string): string => + defaultName + ? defaultFileName(file.metadata.title, file.id, { ext }) + : withExtension(destination, ext); + const path = partPath(image.ext); + const videoPath = partPath(video.ext); for (const part of parts) { await part.handle.sync(); await part.handle.close(); @@ -531,6 +540,9 @@ const decryptLivePhoto = async ( // and then removes its own temp file, and only the attempt that reaches // TAG_FINAL renames one into place, so a download that needed three tries still // performs exactly one rename over the destination. +// +// `defaultName` says `destination` is the name `defaultFileName` made, not a +// path the caller gave (see `decryptLivePhoto`). const fetchAndDecrypt = async ( api: ApiClient, openStream: () => Promise>, @@ -539,6 +551,7 @@ const fetchAndDecrypt = async ( destination: string, onProgress?: ProgressCallback, original?: EnteFile, + defaultName = false, ): Promise => withRetry(async () => { const stream = await openStream(); @@ -551,6 +564,7 @@ const fetchAndDecrypt = async ( key, onProgress, original, + defaultName, ); } const bytesWritten = await decryptToTemp( @@ -582,10 +596,11 @@ export const downloadFile = async ( outPath?: string, onProgress?: ProgressCallback, ): Promise => { - // `outPath` is the caller's and is used as is; the title is the server's - // and is sanitized so it can only name a file in the current directory. + // `outPath` is the caller's and is used as is. Without it, the file is + // written in the current directory under the name `defaultFileName` makes + // from the title, which is the server's. const resolvedPath = - outPath ?? sanitizeFileName(file.metadata.title, `file-${file.id}`); + outPath ?? defaultFileName(file.metadata.title, file.id); const header = fromBase64(file.file.decryptionHeader); return fetchAndDecrypt( api, @@ -595,6 +610,7 @@ export const downloadFile = async ( resolvedPath, onProgress, file, + outPath === undefined, ); }; @@ -604,9 +620,10 @@ export const downloadThumbnail = async ( outPath?: string, onProgress?: ProgressCallback, ): Promise => { + // As in `downloadFile`, with `thumb_` before the title. const resolvedPath = outPath ?? - `thumb_${sanitizeFileName(file.metadata.title, `file-${file.id}`)}`; + defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" }); const header = fromBase64(file.thumbnail.decryptionHeader); return fetchAndDecrypt( api, diff --git a/src/filename.ts b/src/filename.ts index e9781ca..95b7503 100644 --- a/src/filename.ts +++ b/src/filename.ts @@ -5,7 +5,9 @@ // used on disk goes through here, so it can only ever name one file inside the // directory the caller chose: never a path, never `..`, never hidden, never a // Windows device name. Such a name can still be too long for a file name, and -// `shortenToBytes` cuts one short. +// `shortenToBytes` or `shortenKeepingExtension` cuts one short. +// `defaultFileName` does both for the name a file is saved under when the +// caller gives no path. // // A path the user typed (`--out`, `outPath`) is not passed through here: the // caller is trusted, the server is not. @@ -30,11 +32,11 @@ export const sanitizeFileName = (name: string, fallback: string): string => { return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned; }; -// The extension of `title` (".jpg"), or ".bin" when it has none or it holds -// anything but letters and digits. +// The extension of `title` (".jpg"), or ".bin" when it has none, it holds +// anything but letters and digits, or it is more than 16 of them long. export const safeExtension = (title: string): string => { const ext = extname(title); - return /^\.[A-Za-z0-9]+$/.test(ext) ? ext : ".bin"; + return /^\.[A-Za-z0-9]{1,16}$/.test(ext) ? ext : ".bin"; }; // `name` with its extension, if it has one, replaced by `ext` (".mov"). @@ -57,3 +59,53 @@ export const shortenToBytes = (name: string, maxBytes: number): string => { } return name.slice(0, end); }; + +// `name` cut short to fit in `maxBytes` bytes with `insert` put right before +// its extension (`IMG_0001 (12).jpg`), each cut made as `shortenToBytes` +// makes it. The part before the extension is cut first, so the extension is +// kept whenever it fits, but that part always keeps its first character: +// when the extension leaves no room for `insert` and that character, the +// extension is cut short too, and the part before it keeps the bytes that cut +// leaves over. So a name that does not start with a dot never comes back +// starting with one, which would hide it. `insert` is never cut. With no +// `insert`, a name that fits comes back unchanged. +export const shortenKeepingExtension = ( + name: string, + maxBytes: number, + insert = "", +): string => { + const fullExt = extname(name); + const stem = name.slice(0, name.length - fullExt.length); + const first = Array.from(stem)[0] ?? ""; + const ext = shortenToBytes( + fullExt, + maxBytes - Buffer.byteLength(insert + first), + ); + const room = maxBytes - Buffer.byteLength(insert + ext); + return shortenToBytes(stem, room) + insert + ext; +}; + +// The name a file is saved under in the working directory when the caller +// gives no path (`quak get` without `--out`, `downloadFile` without +// `outPath`): `prefix` (`thumb_` for a thumbnail), then the file's title with +// unsafe characters replaced, or `file-` when the title is empty, cut +// short so that the whole fits in a file name, keeping the title's extension +// whenever it fits. `ext` (".mov"), when given, takes the place of the +// title's extension before the name is cut short; a live photo's image and +// video are each named with their own. The cut can leave a device name +// (`CONSOLE.` becomes `CON.`); that name gets +// a leading `_`, as `sanitizeFileName` gives one, and is cut a byte shorter to +// leave room for it. +export const defaultFileName = ( + title: string, + fileID: number, + opts: { prefix?: string; ext?: string } = {}, +): string => { + const { prefix = "", ext } = opts; + const safe = sanitizeFileName(title, `file-${fileID}`); + const name = ext === undefined ? safe : withExtension(safe, ext); + const room = MAX_NAME_BYTES - Buffer.byteLength(prefix); + const cut = prefix + shortenKeepingExtension(name, room); + if (!RESERVED_DEVICE_NAME.test(cut)) return cut; + return `_${prefix}${shortenKeepingExtension(name, room - 1)}`; +}; diff --git a/src/metadata-backup.ts b/src/metadata-backup.ts index 79fc4ed..9620efe 100644 --- a/src/metadata-backup.ts +++ b/src/metadata-backup.ts @@ -5,7 +5,11 @@ import type { Client } from "./client.js"; import { describeError } from "./errors.js"; import { readExifTags } from "./exif.js"; import type { Library, Photo } from "./library/index.js"; -import { sanitizeFileName } from "./filename.js"; +import { + MAX_NAME_BYTES, + sanitizeFileName, + shortenToBytes, +} from "./filename.js"; import { fetchMLDataBatch, MLDATA_BATCH_SIZE, @@ -127,7 +131,15 @@ export const runMetadataBackup = async ( const col = lib.getCollection(album.collectionID); if (!col) continue; - const dirName = `${col.id}-${sanitizeFileName(col.name, "unnamed")}`; + // The album's ID keeps the folders apart, so the name after it can be + // cut short to fit in a file name. + const prefix = `${col.id}-`; + const dirName = + prefix + + shortenToBytes( + sanitizeFileName(col.name, "unnamed"), + MAX_NAME_BYTES - prefix.length, + ); const colDir = join(outDir, "collections", dirName); mkdirSync(colDir, { recursive: true }); diff --git a/test/cli/backup.test.ts b/test/cli/backup.test.ts index 135ddfb..c6258dd 100644 --- a/test/cli/backup.test.ts +++ b/test/cli/backup.test.ts @@ -1795,12 +1795,11 @@ describe("backup album folders", () => { expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 }); }); - // The extensions of these titles leave no room for the ID, so each is cut - // short and the name before the ID keeps only the bytes that cut leaves - // over. File 1's extension is 303 bytes and is cut to 251, which leaves - // none. File 2's is 271 bytes and is cut to 250, since "€" takes three - // bytes, which leaves one, for the "a". Neither extension is only letters - // and digits, so both originals are saved with `.bin`, a name that fits. + // The extensions of these titles leave no room for the ID and the "a" + // before them, so each is cut short to 250 bytes, which leaves one, for + // the "a". File 1's extension is 303 bytes and file 2's 271. Neither + // extension is only letters and digits, so both originals are saved with + // `.bin`, a name that fits. it("cuts the extension of a title short when it leaves no room for the ID", async () => { const outDir = join(root, "backup"); const lib = libraryOf([ @@ -1815,7 +1814,7 @@ describe("backup album folders", () => { const result = await runBackup(lib, { downloadDirectory: outDir }); - const link = ` (1).${"L".repeat(250)}`; + const link = `a (1).${"L".repeat(249)}`; const link2 = `a (2).${"€".repeat(83)}`; expect(Buffer.byteLength(link)).toBe(255); expect(Buffer.byteLength(link2)).toBe(255); @@ -1830,9 +1829,9 @@ describe("backup album folders", () => { // Files 1 and 3 share a title, so each gets its ID before the // extension. File 2's title is too long, and its extension leaves no - // room for the ID. Were the ID put at the end of file 2's title cut - // short, its link would have file 1's name, and the link made second - // would replace the first. + // room for the ID and the "x" before it. Were the ID put at the end of + // file 2's title cut short, its link would have file 1's name, and the + // link made second would replace the first. it("keeps names unique when a title's extension leaves no room for the ID", async () => { const outDir = join(root, "backup"); const shared = `x.${"Q".repeat(245)} (2)`; @@ -1852,8 +1851,8 @@ describe("backup album folders", () => { expect(result.failed).toBe(0); expect(tree(outDir)).toEqual([ "Trip/", - `Trip/ (2).${"Q".repeat(250)} -> ${linkTo("2.bin")}`, `Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`, + `Trip/x (2).${"Q".repeat(249)} -> ${linkTo("2.bin")}`, `Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`, "Trip.json", ]); diff --git a/test/cli/commands.test.ts b/test/cli/commands.test.ts index 6da51e1..0c1686e 100644 --- a/test/cli/commands.test.ts +++ b/test/cli/commands.test.ts @@ -21,7 +21,7 @@ import { utimesSync, writeFileSync, } from "node:fs"; -import { join } from "node:path"; +import { basename, join } from "node:path"; import { tmpdir } from "node:os"; import { PassThrough } from "node:stream"; import * as jpegJs from "jpeg-js"; @@ -187,6 +187,9 @@ const FETCHED_BOTH_ALBUMS = "Fetched the files of 1 of 2 collections\n" + "Fetched the files of 2 of 2 collections\n"; +// The same for an account with one album. +const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n"; + // Collects everything written to it. class Output { text = ""; @@ -562,17 +565,109 @@ describe("get and get-thumb", () => { }); }); +// A file system refuses a file name over 255 bytes. The account here is one +// album, whose name is 300 bytes, holding one file, 700, with the title given. +describe("names from an album name or a title over 255 bytes", () => { + const longNames = (title: string): Client => + ({ + ...fakeClient(), + collectionsSince: async () => ({ + collections: [collection(7, "A".repeat(300))], + deleted: [], + cursor: 1, + }), + filesSince: async () => ({ + files: [file(700, 7, title)], + deleted: [], + cursor: 1, + }), + }) as unknown as Client; + const longTitle = `${"t".repeat(296)}.jpg`; + + // Run `command` in a new directory, where `get` and `get-thumb` without + // `--out` write, and return its exit code and the directory. + const inNewDirectory = async ( + command: () => Promise, + ): Promise<{ code: number; dir: string }> => { + const dir = mkdtempSync(join(root, "cwd-")); + const previous = process.cwd(); + process.chdir(dir); + try { + return { code: await command(), dir }; + } finally { + process.chdir(previous); + } + }; + + it("get cuts the title short to 255 bytes, keeping its extension, and exits 0", async () => { + const { code, dir } = await inNewDirectory(() => + getCommand(context(longNames(longTitle)), "700", {}), + ); + + const name = `${"t".repeat(251)}.jpg`; + expect(Buffer.byteLength(name)).toBe(255); + expect(code).toBe(0); + expect(readdirSync(dir)).toEqual([name]); + expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `7 bytes -> ${name}\n`); + }); + + it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => { + const { code, dir } = await inNewDirectory(() => + getThumbCommand(context(longNames(longTitle)), "700", {}), + ); + + const name = `thumb_${"t".repeat(245)}.jpg`; + expect(Buffer.byteLength(name)).toBe(255); + expect(code).toBe(0); + expect(readdirSync(dir)).toEqual([name]); + expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `3 bytes -> ${name}\n`); + }); + + it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => { + const dir = join(root, "dump"); + + const code = await backupMetadataCommand( + context(longNames("beach.jpg")), + dir, + {}, + ); + + const folder = `7-${"A".repeat(253)}`; + expect(Buffer.byteLength(folder)).toBe(255); + expect(code).toBe(0); + expect(readdirSync(join(dir, "collections"))).toEqual([folder]); + expect(readdirSync(join(dir, "collections", folder)).sort()).toEqual([ + "700.json", + "_collection.json", + ]); + }); + + // The title's extension is 300 letters, too long for the save path's + // name, so the original is saved with `.bin`. + it("backup saves a title with a 300-letter extension as .bin, and exits 0", async () => { + const dir = join(root, "backup"); + const title = `b.${"e".repeat(300)}`; + + const code = await backupCommand(context(longNames(title)), dir, {}); + + const path = savePath(dir, file(700, 7, title)); + expect(code).toBe(0); + expect(basename(path)).toMatch(/^\d{4}-\d{2}-\d{2}\.700\.bin$/); + expect(readFileSync(path)).toEqual(Buffer.alloc(7, 700 & 0xff)); + }); +}); + // A live photo, which Ente stores as one ZIP, is written as its image and its // video, which a photo viewer can open. The account here is Vacation holding // one live photo, 300, downloaded through the real download layer // (test/live-photo.ts). describe("a live photo", () => { - // What `get` prints on stderr while its refresh fetches Vacation's files. - const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n"; - - const livePhotoClient = async (image = IMAGE): Promise => { + const livePhotoClient = async ( + image = IMAGE, + title = "IMG_0300.HEIC", + ): Promise => { const { file: live, body } = await asLivePhoto( - file(300, 1, "IMG_0300.HEIC"), + file(300, 1, title), livePhotoZip({ "image.heic": image, "video.mov": VIDEO }), livePhotoHash(image, VIDEO), ); @@ -620,6 +715,44 @@ describe("a live photo", () => { ); }); + // Run `get` without `--out` in a new directory for a live photo titled + // `title`, and return its exit code and the names it wrote. + const getInNewDirectory = async ( + title: string, + ): Promise<{ code: number; names: string[] }> => { + const ctx = context(await livePhotoClient(IMAGE, title)); + const dir = mkdtempSync(join(root, "cwd-")); + const previous = process.cwd(); + process.chdir(dir); + try { + const code = await getCommand(ctx, "300", {}); + return { code, names: readdirSync(dir).sort() }; + } finally { + process.chdir(previous); + } + }; + const imageName = `${"t".repeat(250)}.heic`; + const videoName = `${"t".repeat(251)}.mov`; + + // The title fits, but ".heic" is one byte longer than ".jpg", so the + // image's name is cut short once its extension is in it. + it("get cuts its image's name short for a 255-byte title ending in .jpg, and exits 0", async () => { + const title = `${"t".repeat(251)}.jpg`; + + const result = await getInNewDirectory(title); + + expect(Buffer.byteLength(title)).toBe(255); + expect(Buffer.byteLength(imageName)).toBe(255); + expect(Buffer.byteLength(videoName)).toBe(255); + expect(result).toEqual({ code: 0, names: [imageName, videoName] }); + }); + + it("get cuts the names of its image and video short for a 300-byte title with no extension, and exits 0", async () => { + const result = await getInNewDirectory("t".repeat(300)); + + expect(result).toEqual({ code: 0, names: [imageName, videoName] }); + }); + it("get --out writes the image there and the video beside it", async () => { const out = join(root, "photo.jpg"); const video = join(root, "photo.mov"); diff --git a/test/download/download.test.ts b/test/download/download.test.ts index 7333b36..5dde3a2 100644 --- a/test/download/download.test.ts +++ b/test/download/download.test.ts @@ -658,6 +658,26 @@ describe("downloadFile", () => { expect(readdirSync(parent)).toEqual(["cwd"]); }); + // A file system refuses a file name over 255 bytes. + it("cuts a title too long for a file name short, keeping its extension", async () => { + const plaintext = new Uint8Array([1, 2, 3]); + const key = sodium.crypto_secretstream_xchacha20poly1305_keygen(); + const { header, ciphertext } = encryptFileBody(plaintext, key); + const thumbPush = + sodium.crypto_secretstream_xchacha20poly1305_init_push(key); + const file = buildMockEnteFile(key, header, thumbPush.header); + file.metadata.title = `${"t".repeat(296)}.jpg`; + const dir = mkdtempSync(join(testDir, "long-title-")); + + const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) }); + const result = await inDirectory(dir, () => downloadFile(api, file)); + + const name = `${"t".repeat(251)}.jpg`; + expect(Buffer.byteLength(name)).toBe(255); + expect(result.path).toBe(name); + expect(readdirSync(dir)).toEqual([name]); + }); + it("uses an explicit outPath verbatim, even one with ..", async () => { // The caller is trusted: its path is not sanitized. const { api, file } = fixtureFor( @@ -751,6 +771,28 @@ describe("downloadThumbnail", () => { expect(result.path).toBe("thumb__etc_passwd"); expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]); }); + + it("leaves room for thumb_ when it cuts a title too long for a file name short", async () => { + const plaintext = new Uint8Array([0xff, 0xd8, 0xff, 0xe0]); + const key = sodium.crypto_secretstream_xchacha20poly1305_keygen(); + const filePush = + sodium.crypto_secretstream_xchacha20poly1305_init_push(key); + const { header: thumbHeader, ciphertext: thumbCipher } = + encryptFileBody(plaintext, key); + const file = buildMockEnteFile(key, filePush.header, thumbHeader); + file.metadata.title = `${"t".repeat(296)}.jpg`; + const dir = mkdtempSync(join(testDir, "long-thumb-title-")); + + const api = new ApiClient({ fetch: mockFetchForBody(thumbCipher) }); + const result = await inDirectory(dir, () => + downloadThumbnail(api, file), + ); + + const name = `thumb_${"t".repeat(245)}.jpg`; + expect(Buffer.byteLength(name)).toBe(255); + expect(result.path).toBe(name); + expect(readdirSync(dir)).toEqual([name]); + }); }); // --------------------------------------------------------------------------- @@ -1881,6 +1923,53 @@ describe("downloadFile live photos", () => { ]); }); + // A file system refuses a file name over 255 bytes. Each name is cut + // short once its own extension is in it. + const imageName = `${"t".repeat(250)}.heic`; + const videoName = `${"t".repeat(251)}.mov`; + + // The title fits, but ".heic" is one byte longer than ".jpg". + it("cuts the image's name short for a 255-byte title ending in .jpg when no outPath is given", async () => { + const title = `${"t".repeat(251)}.jpg`; + const t = setup(livePhotoZip(), { ...livePhoto, title }); + + const result = await inDirectory(t.dir, () => + downloadFile(t.api, t.file), + ); + + expect(Buffer.byteLength(title)).toBe(255); + expect(Buffer.byteLength(imageName)).toBe(255); + expect(Buffer.byteLength(videoName)).toBe(255); + expect(result.path).toBe(imageName); + expect(result.videoPath).toBe(videoName); + expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]); + }); + + it("cuts the names of the image and the video short for a 300-byte title with no extension when no outPath is given", async () => { + const t = setup(livePhotoZip(), { + ...livePhoto, + title: "t".repeat(300), + }); + + const result = await inDirectory(t.dir, () => + downloadFile(t.api, t.file), + ); + + expect(result.path).toBe(imageName); + expect(result.videoPath).toBe(videoName); + expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]); + }); + + // The caller's path is used as is; only the extension is each part's own. + it("keeps an outPath too long for a file name as given", async () => { + const t = setup(livePhotoZip(), livePhoto); + const outPath = join(t.dir, `${"t".repeat(251)}.jpg`); + + await expect(downloadFile(t.api, t.file, outPath)).rejects.toThrow( + /ENAMETOOLONG/, + ); + }); + it("gives each part its own extension from the ZIP, letters and digits only", async () => { const zip = livePhotoZip({ "image.JPG": IMAGE, "video.m-4v": VIDEO }); const t = setup(zip, livePhoto); diff --git a/test/filename/filename.test.ts b/test/filename/filename.test.ts index 4cfce05..8abed29 100644 --- a/test/filename/filename.test.ts +++ b/test/filename/filename.test.ts @@ -15,8 +15,10 @@ import { describe, expect, it } from "vitest"; import { + defaultFileName, safeExtension, sanitizeFileName, + shortenKeepingExtension, shortenToBytes, } from "../../src/filename.js"; @@ -94,6 +96,12 @@ describe("safeExtension", () => { expect(safeExtension("x.jp g")).toBe(".bin"); expect(safeExtension("x.jpg\0")).toBe(".bin"); }); + + it("keeps an extension of 16 letters and digits, and uses .bin for a longer one", () => { + expect(safeExtension(`x.${"e".repeat(16)}`)).toBe(`.${"e".repeat(16)}`); + expect(safeExtension(`x.${"e".repeat(17)}`)).toBe(".bin"); + expect(safeExtension(`b.${"e".repeat(300)}`)).toBe(".bin"); + }); }); describe("shortenToBytes", () => { @@ -118,3 +126,103 @@ describe("shortenToBytes", () => { ); }); }); + +describe("shortenKeepingExtension", () => { + it("returns a name that fits unchanged", () => { + expect(shortenKeepingExtension("beach.jpg", 9)).toBe("beach.jpg"); + expect(shortenKeepingExtension("beach", 5)).toBe("beach"); + }); + + it("cuts the part before the extension short and keeps the extension", () => { + expect(shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255)).toBe( + `${"t".repeat(251)}.jpg`, + ); + expect(shortenKeepingExtension("é".repeat(150), 255)).toBe( + "é".repeat(127), + ); + }); + + // A name cut down to its extension would start with a dot, and so be + // hidden. + it("keeps the first character before the extension, cutting the extension short to leave room for it", () => { + expect(shortenKeepingExtension(`b.${"e".repeat(300)}`, 255)).toBe( + `b.${"e".repeat(253)}`, + ); + expect(shortenKeepingExtension(`ab.${"e".repeat(300)}`, 255)).toBe( + `a.${"e".repeat(253)}`, + ); + expect(shortenKeepingExtension(`é.${"e".repeat(300)}`, 255)).toBe( + `é.${"e".repeat(252)}`, + ); + // "€" takes three bytes, so the extension is cut to 253 and the + // part before it keeps the two bytes left over, for the "a". + expect(shortenKeepingExtension(`a.${"€".repeat(90)}`, 255)).toBe( + `a.${"€".repeat(84)}`, + ); + }); + + it("puts `insert` right before the extension and leaves room for it", () => { + expect(shortenKeepingExtension("beach.jpg", 255, " (7)")).toBe( + "beach (7).jpg", + ); + expect( + shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255, " (7)"), + ).toBe(`${"t".repeat(247)} (7).jpg`); + expect( + shortenKeepingExtension(`b.${"e".repeat(300)}`, 255, " (7)"), + ).toBe(`b (7).${"e".repeat(249)}`); + }); +}); + +describe("defaultFileName", () => { + it("is the title when it is safe and fits", () => { + expect(defaultFileName("IMG_0001.HEIC", 42)).toBe("IMG_0001.HEIC"); + }); + + it("replaces unsafe characters, and is file- for an empty title", () => { + expect(defaultFileName("../x.jpg", 42)).toBe("__x.jpg"); + expect(defaultFileName("", 42)).toBe("file-42"); + }); + + it("cuts a title over 255 bytes short, keeping its extension", () => { + expect(defaultFileName(`${"t".repeat(296)}.jpg`, 42)).toBe( + `${"t".repeat(251)}.jpg`, + ); + expect(defaultFileName(`b.${"e".repeat(300)}`, 42)).toBe( + `b.${"e".repeat(253)}`, + ); + }); + + it("puts `prefix` before the title and leaves room for it", () => { + expect(defaultFileName("beach.jpg", 42, { prefix: "thumb_" })).toBe( + "thumb_beach.jpg", + ); + expect( + defaultFileName(`${"t".repeat(296)}.jpg`, 42, { prefix: "thumb_" }), + ).toBe(`thumb_${"t".repeat(245)}.jpg`); + }); + + // A live photo's image and video are each named with their own + // extension, and the name is cut short once that extension is in it. + it("puts `ext` in place of the title's extension before cutting the name short", () => { + expect(defaultFileName("IMG_0001.HEIC", 42, { ext: ".mov" })).toBe( + "IMG_0001.mov", + ); + expect(defaultFileName("", 42, { ext: ".mov" })).toBe("file-42.mov"); + expect( + defaultFileName(`${"t".repeat(251)}.jpg`, 42, { ext: ".heic" }), + ).toBe(`${"t".repeat(250)}.heic`); + expect(defaultFileName("t".repeat(300), 42, { ext: ".heic" })).toBe( + `${"t".repeat(250)}.heic`, + ); + }); + + // Cutting `CONSOLE` or `NULL` short before a long extension leaves `CON` + // or `NUL`, so the name gets the `_` `sanitizeFileName` gives a device + // name, and the part before the extension gives up a byte for it. + it("never returns a Windows device name, even once cut short", () => { + const ext = `.${"e".repeat(251)}`; + expect(defaultFileName(`CONSOLE${ext}`, 42)).toBe(`_CO${ext}`); + expect(defaultFileName(`NULL${ext}`, 42)).toBe(`_NU${ext}`); + }); +});