Names quak builds from server data fit in 255 bytes (closes #205)
check / check (push) Waiting to run

The default names of `quak get`, `quak get-thumb`, `downloadFile` and `downloadThumbnail`, and the album folders of `backup-metadata`, could be over 255 bytes, which a file system refuses. `defaultFileName` and `shortenKeepingExtension` in `src/filename.ts` now cut such a name short, keeping its extension when it fits; a live photo's image and video get their own extension before the cut. A cut name never starts with a dot and is never a Windows device name. `--out` and `outPath` are kept as given. A `backup-metadata` folder keeps its `<id>-`. `safeExtension` gives `.bin` for an extension of more than 16 letters and digits, so a save path's name always fits.

Model: opus-5-5
This commit was merged in pull request #213.
This commit is contained in:
2026-10-18 20:01:34 +02:00
parent 982b878d69
commit 66e3ac77e8
12 changed files with 526 additions and 77 deletions
+37 -21
View File
@@ -683,27 +683,41 @@ before any request with one line, `Invalid file ID: <arg>` or
`Invalid collection ID: <arg>`, and exit status 1. One bad `--file` ID refuses `Invalid collection ID: <arg>`, and exit status 1. One bad `--file` ID refuses
the whole command, so nothing is fixed. the whole command, so nothing is fixed.
Without `--out`, `get` writes the file to the working directory under its title,
and `get-thumb` writes the thumbnail there under `thumb_` followed by the title,
each with unsafe characters replaced. A name over 255 bytes, which most file
systems refuse, is cut short between two characters before its extension, so the
extension is kept whenever it fits. An extension too long to leave room for the
name's first character is cut short too, so the name never starts with a dot.
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is `get` and `get-thumb` resolve the file by ID directly, so `--collection` is
accepted for backward compatibility but ignored. For a live photo, `get` writes accepted for backward compatibility but ignored. For a live photo, `get` writes
its image and its video, each named after the title with its own extension, as its image and its video, each named after the title with its own extension, as
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`). With Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`), and each name is
`--out PATH`, the image is written to `PATH` and the video beside it, with cut short as above once its own extension is in it. With `--out PATH`, the image
`PATH`'s name and the video's extension; a `PATH` with the video's extension is is written to `PATH` and the video beside it, with `PATH`'s name and the video's
refused. `backup-metadata --exif` (alias `--all`) additionally fetches each extension; a `PATH` with the video's extension is refused.
file's original through the cache and records, from it or a live photo's image, `backup-metadata --exif` (alias `--all`) additionally fetches each file's
its XMP metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is original through the cache and records, from it or a live photo's image, its XMP
read with [exifreader](https://github.com/mattiasw/ExifReader) from any image metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is read with
format it reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's [exifreader](https://github.com/mattiasw/ExifReader) from any image format it
`exif` field is exifreader's EXIF tag output: each tag by name, with its reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's `exif` field
`value`, `description` and `computed` value. An EXIF block exifreader finds but is exifreader's EXIF tag output: each tag by name, with its `value`,
reads no tag from is recorded, base64, as `exifRaw`, with the reason in `description` and `computed` value. An EXIF block exifreader finds but reads no
`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails` tag from is recorded, base64, as `exifRaw`, with the reason in `exifError`.
and `helper fix-missing-thumbnails` take `--json` for machine-readable output. `collections`, `files`, `backup`, `helper list-missing-thumbnails` and
`helper fix-missing-thumbnails` take `--json` for machine-readable output.
`backup --verify` also hashes the originals already in the backup and replaces `backup --verify` also hashes the originals already in the backup and replaces
any that do not match the content hash Ente records; one it cannot replace stays any that do not match the content hash Ente records; one it cannot replace stays
as it was and goes into `failures.json` (see "Backup layout"). as it was and goes into `failures.json` (see "Backup layout").
`backup-metadata <dir>` writes `account.json` and, for each collection, a
directory `collections/<id>-<name>/` holding `_collection.json` and a
`<fileID>.json` for each of its files. The collection's name has unsafe
characters replaced and is cut short between two characters so that the
directory's name fits in 255 bytes; the ID keeps the directories apart.
`backup-metadata` fetches ML data in requests of up to 200 files. When a request `backup-metadata` fetches ML data in requests of up to 200 files. When a request
fails, the error is logged, each of its files is written with the reason in an fails, the error is logged, each of its files is written with the reason in an
`mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML `mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML
@@ -756,9 +770,10 @@ Each original is saved at its save path, the same path `photo.savePath` gives
and `photo.download()` writes (see Read surface below). The date is the photo's and `photo.download()` writes (see Read surface below). The date is the photo's
`takenAt` (the date set in Ente if it was edited, else its creation time) in the `takenAt` (the date set in Ente if it was edited, else its creation time) in the
time zone of the machine running quak. The extension is the one in the file's time zone of the machine running quak. The extension is the one in the file's
name as uploaded, case kept, or `.bin` when it has none or it holds anything but name as uploaded, case kept, or `.bin` when it has none, it holds anything but
letters and digits. When the date or the time zone changes, the next run saves letters and digits, or it is more than 16 of them long. When the date or the
the original at its new path and leaves the old copy where it is. time zone changes, the next run saves the original at its new path and leaves
the old copy where it is.
`account.json` holds the account's `email` and `userID`, as `backup-metadata` `account.json` holds the account's `email` and `userID`, as `backup-metadata`
writes it. A collection's JSON holds its `id`, `name`, `type`, `ownerID`, writes it. A collection's JSON holds its `id`, `name`, `type`, `ownerID`,
@@ -851,11 +866,12 @@ gets its ID added as well: beside an album `x`, whose JSON is `x.json`, an album
named `x.json` gets the directory `x.json (12)/`. A name too long for a file named `x.json` gets the directory `x.json (12)/`. A name too long for a file
name, which most file systems limit to 255 bytes, gets its ID added too. A name name, which most file systems limit to 255 bytes, gets its ID added too. A name
with its ID added is cut short between two characters in the part before the ID, with its ID added is cut short between two characters in the part before the ID,
just far enough that it fits. When a symlink's extension alone leaves no room just far enough that it fits. When a symlink's extension leaves no room for the
for the ID, the extension is cut short and the part before the ID keeps only the ID and the first character of the name, the extension is cut short, and the part
bytes that cut leaves over. A collection's directory name also leaves room for before the ID keeps that character and any bytes the cut leaves over. A
the `.json` its JSON adds. IDs never change, so a name stays the same from run collection's directory name also leaves room for the `.json` its JSON adds. IDs
to run until such a clash appears or goes away. never change, so a name stays the same from run to run until such a clash
appears or goes away.
A collection whose directory or JSON cannot be written, such as one whose A collection whose directory or JSON cannot be written, such as one whose
directory name is taken by a file you put in `collections/`, does not stop the directory name is taken by a file you put in `collections/`, does not stop the
+20 -3
View File
@@ -17,15 +17,32 @@ pre-1.0
# Next Step # Next Step
One bug is open: https://git.eeqj.de/sneak/quak/issues/205. The check of quak No bug is open. Two changes sneak asked for are open:
against a live Ente account, https://git.eeqj.de/sneak/quak/issues/187, needs an https://git.eeqj.de/sneak/quak/issues/211 and
account only sneak has, so it waits on him. https://git.eeqj.de/sneak/quak/issues/212. The check of quak against a live Ente
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
has, so it waits on him.
Tagging and releases are decided by sneak alone, and happen only when he Tagging and releases are decided by sneak alone, and happen only when he
declares one. declares one.
# Completed Steps # Completed Steps
- 2026-10-18: Names quak builds from an album name or a title fit in a file name
(https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in
`src/filename.ts` cuts a name short to a number of bytes, keeping its
extension when it fits and always its first character, and can put text right
before the extension; the backup's `withID` uses it for an ID.
`defaultFileName` there gives the name a file is saved under when the caller
gives no path: a prefix such as `thumb_` and the title with unsafe characters
replaced, cut short with it to 255 bytes, and, for a live photo's image or
video, with that part's own extension before the cut. `quak get`,
`quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without
`--out` or `outPath`. `backup-metadata` cuts an album's name short so that its
folder, `<id>-<name>`, fits in 255 bytes. `safeExtension` gives `.bin` for an
extension of more than 16 letters and digits, so a save path's name always
fits.
- 2026-10-18: `quak collections` no longer waits for the files of every - 2026-10-18: `quak collections` no longer waits for the files of every
collection (issue 210). On an empty cache it opened its library with a refresh collection (issue 210). On an empty cache it opened its library with a refresh
that fetched and decrypted the files of each collection in turn, printing that fetched and decrypted the files of each collection in turn, printing
+8 -8
View File
@@ -82,6 +82,7 @@ import { describeError } from "./errors.js";
import { import {
MAX_NAME_BYTES, MAX_NAME_BYTES,
sanitizeFileName, sanitizeFileName,
shortenKeepingExtension,
shortenToBytes, shortenToBytes,
withExtension, withExtension,
} from "./filename.js"; } from "./filename.js";
@@ -289,9 +290,10 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
// case (two names that differ only in case are one entry on a case-insensitive // case (two names that differ only in case are one entry on a case-insensitive
// file system), or it is too long for a file name; then it gets ` (<id>)`, // file system), or it is too long for a file name; then it gets ` (<id>)`,
// right before the extension when `beforeExtension` is set. To fit, the part // right before the extension when `beforeExtension` is set. To fit, the part
// before the ID is cut short as far as it must be. When the extension alone // before the ID is cut short as far as it must be. When the extension leaves
// leaves no room for the ID, the extension is cut short and the part before // no room for the ID and the first character of the name, the extension is
// the ID keeps only the bytes that cut leaves over. With `withJSON`, each // cut short, and the part before the ID keeps that character and any bytes
// the cut leaves over (see `shortenKeepingExtension`). With `withJSON`, each
// entry also has the JSON file `<name>.json` beside it, as an album does: a // entry also has the JSON file `<name>.json` beside it, as an album does: a
// name then leaves room for `.json`, and one that is another entry's JSON name // name then leaves room for `.json`, and one that is another entry's JSON name
// gets its ID added too. // gets its ID added too.
@@ -305,11 +307,9 @@ const uniqueNames = (
const maxBytes = MAX_NAME_BYTES - (opts.withJSON ? ".json".length : 0); const maxBytes = MAX_NAME_BYTES - (opts.withJSON ? ".json".length : 0);
const withID = (id: number, name: string): string => { const withID = (id: number, name: string): string => {
const tag = ` (${id})`; const tag = ` (${id})`;
const fullExt = opts.beforeExtension ? extname(name) : ""; return opts.beforeExtension
const stem = name.slice(0, name.length - fullExt.length); ? shortenKeepingExtension(name, maxBytes, tag)
const ext = shortenToBytes(fullExt, maxBytes - Buffer.byteLength(tag)); : shortenToBytes(name, maxBytes - Buffer.byteLength(tag)) + tag;
const room = maxBytes - Buffer.byteLength(tag + ext);
return shortenToBytes(stem, room) + tag + ext;
}; };
const names = entries.map((e) => e.name); const names = entries.map((e) => e.name);
const suffixed = new Set<number>(); const suffixed = new Set<number>();
+8 -4
View File
@@ -332,8 +332,8 @@ export const getCommand = async (
const result = await photo.original(); const result = await photo.original();
// Default name is the file's own title, as the pre-library CLI used // Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52). // (not the editedName-preferring projection title) (issue #52).
const outPath = opts.out ?? originalName(file);
if (result.videoPath === undefined) { if (result.videoPath === undefined) {
const outPath = opts.out ?? originalName(file);
copyFileSync(result.path, outPath); copyFileSync(result.path, outPath);
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`); ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
return 0; return 0;
@@ -341,9 +341,13 @@ export const getCommand = async (
// A live photo is written as its image and its video, each named after // A live photo is written as its image and its video, each named after
// the title with its own extension, as Ente's clients name them. With // the title with its own extension, as Ente's clients name them. With
// --out, the image goes there and the video beside it. // --out, the image goes there and the video beside it.
const imageOut = const imageExt = extname(result.path);
opts.out ?? withExtension(outPath, extname(result.path)); const videoExt = extname(result.videoPath);
const videoOut = withExtension(outPath, extname(result.videoPath)); const imageOut = opts.out ?? originalName(file, imageExt);
const videoOut =
opts.out === undefined
? originalName(file, videoExt)
: withExtension(opts.out, videoExt);
if (imageOut.toLowerCase() === videoOut.toLowerCase()) { if (imageOut.toLowerCase() === videoOut.toLowerCase()) {
ctx.stderr.write( ctx.stderr.write(
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`, `File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`,
+9 -7
View File
@@ -9,7 +9,7 @@
// `metadata.title`, and issue #52 requires that output stay byte-identical, so // `metadata.title`, and issue #52 requires that output stay byte-identical, so
// the commands shape their output from the raw `EnteFile` through here. // the commands shape their output from the raw `EnteFile` through here.
import { sanitizeFileName } from "./filename.js"; import { defaultFileName } from "./filename.js";
import type { EnteFile, FileType, Microseconds } from "./model/types.js"; import type { EnteFile, FileType, Microseconds } from "./model/types.js";
// One row of `quak files --json`. // One row of `quak files --json`.
@@ -33,11 +33,13 @@ export const fileListRow = (file: EnteFile): FileListRow => ({
export const fileListLine = (file: EnteFile): string => export const fileListLine = (file: EnteFile): string =>
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`; `${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
// Default output path for `quak get` when `--out` is not given. The title comes // Default output path for `quak get` when `--out` is not given, made from the
// from the server, so it is sanitized; `--out` is the user's and is used as is. // title (see `defaultFileName`), with `ext` in place of its extension for a
export const originalName = (file: EnteFile): string => // live photo's image or video; `--out` is the user's and is used as is.
sanitizeFileName(file.metadata.title, `file-${file.id}`); export const originalName = (file: EnteFile, ext?: string): string =>
defaultFileName(file.metadata.title, file.id, { ext });
// Default output path for `quak get-thumb` when `--out` is not given. // Default output path for `quak get-thumb` when `--out` is not given: `thumb_`
// and the title.
export const thumbnailName = (file: EnteFile): string => export const thumbnailName = (file: EnteFile): string =>
`thumb_${originalName(file)}`; defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
+28 -11
View File
@@ -16,7 +16,7 @@ import {
streamTagFinal, streamTagFinal,
} from "../crypto/index.js"; } from "../crypto/index.js";
import { TruncatedStreamError } from "../errors.js"; import { TruncatedStreamError } from "../errors.js";
import { safeExtension, sanitizeFileName, withExtension } from "../filename.js"; import { defaultFileName, safeExtension, withExtension } from "../filename.js";
import { withRetry } from "../retry.js"; import { withRetry } from "../retry.js";
import type { ApiClient } from "../api/client.js"; import type { ApiClient } from "../api/client.js";
import type { EnteFile } from "../model/types.js"; import type { EnteFile } from "../model/types.js";
@@ -365,10 +365,14 @@ const openPart = async (
// name starts with `image` as the image and the one whose name starts with // name starts with `image` as the image and the one whose name starts with
// `video` as the video, and refuses a ZIP holding a second of either. It is // `video` as the video, and refuses a ZIP holding a second of either. It is
// written unpacked: each part is named `destination` with the extension // written unpacked: each part is named `destination` with the extension
// replaced by its own entry's, and the two must differ ignoring case. When the // replaced by its own entry's, and the two must differ ignoring case. When
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that // `defaultName` is set, `destination` is the name `defaultFileName` made from
// part's own bytes. Only then are the image, then the video, renamed into // the title rather than a path the caller gave, and each part is named as
// place; on any failure neither is stored. // `defaultFileName` names it with its own extension, so that the name is cut
// short to fit with that extension. When the file records a hash,
// `<imageHash>:<videoHash>` must match it, each over that part's own bytes.
// Only then are the image, then the video, renamed into place; on any failure
// neither is stored.
// //
// The ZIP is chosen by its uploader and may expand enormously, so each part is // The ZIP is chosen by its uploader and may expand enormously, so each part is
// written as it decompresses and never held, and the ZIP is refused once the // written as it decompresses and never held, and the ZIP is refused once the
@@ -388,6 +392,7 @@ const decryptLivePhoto = async (
key: Uint8Array, key: Uint8Array,
onProgress: ProgressCallback | undefined, onProgress: ProgressCallback | undefined,
file: EnteFile, file: EnteFile,
defaultName: boolean,
): Promise<DownloadResult> => { ): Promise<DownloadResult> => {
const sliceSize = 4096; const sliceSize = 4096;
const dir = dirname(destination); const dir = dirname(destination);
@@ -486,8 +491,12 @@ const decryptLivePhoto = async (
`live photo's image and video have the same extension, ${video.ext}`, `live photo's image and video have the same extension, ${video.ext}`,
); );
} }
const path = withExtension(destination, image.ext); const partPath = (ext: string): string =>
const videoPath = withExtension(destination, video.ext); defaultName
? defaultFileName(file.metadata.title, file.id, { ext })
: withExtension(destination, ext);
const path = partPath(image.ext);
const videoPath = partPath(video.ext);
for (const part of parts) { for (const part of parts) {
await part.handle.sync(); await part.handle.sync();
await part.handle.close(); await part.handle.close();
@@ -531,6 +540,9 @@ const decryptLivePhoto = async (
// and then removes its own temp file, and only the attempt that reaches // and then removes its own temp file, and only the attempt that reaches
// TAG_FINAL renames one into place, so a download that needed three tries still // TAG_FINAL renames one into place, so a download that needed three tries still
// performs exactly one rename over the destination. // performs exactly one rename over the destination.
//
// `defaultName` says `destination` is the name `defaultFileName` made, not a
// path the caller gave (see `decryptLivePhoto`).
const fetchAndDecrypt = async ( const fetchAndDecrypt = async (
api: ApiClient, api: ApiClient,
openStream: () => Promise<ReadableStream<Uint8Array>>, openStream: () => Promise<ReadableStream<Uint8Array>>,
@@ -539,6 +551,7 @@ const fetchAndDecrypt = async (
destination: string, destination: string,
onProgress?: ProgressCallback, onProgress?: ProgressCallback,
original?: EnteFile, original?: EnteFile,
defaultName = false,
): Promise<DownloadResult> => ): Promise<DownloadResult> =>
withRetry(async () => { withRetry(async () => {
const stream = await openStream(); const stream = await openStream();
@@ -551,6 +564,7 @@ const fetchAndDecrypt = async (
key, key,
onProgress, onProgress,
original, original,
defaultName,
); );
} }
const bytesWritten = await decryptToTemp( const bytesWritten = await decryptToTemp(
@@ -582,10 +596,11 @@ export const downloadFile = async (
outPath?: string, outPath?: string,
onProgress?: ProgressCallback, onProgress?: ProgressCallback,
): Promise<DownloadResult> => { ): Promise<DownloadResult> => {
// `outPath` is the caller's and is used as is; the title is the server's // `outPath` is the caller's and is used as is. Without it, the file is
// and is sanitized so it can only name a file in the current directory. // written in the current directory under the name `defaultFileName` makes
// from the title, which is the server's.
const resolvedPath = const resolvedPath =
outPath ?? sanitizeFileName(file.metadata.title, `file-${file.id}`); outPath ?? defaultFileName(file.metadata.title, file.id);
const header = fromBase64(file.file.decryptionHeader); const header = fromBase64(file.file.decryptionHeader);
return fetchAndDecrypt( return fetchAndDecrypt(
api, api,
@@ -595,6 +610,7 @@ export const downloadFile = async (
resolvedPath, resolvedPath,
onProgress, onProgress,
file, file,
outPath === undefined,
); );
}; };
@@ -604,9 +620,10 @@ export const downloadThumbnail = async (
outPath?: string, outPath?: string,
onProgress?: ProgressCallback, onProgress?: ProgressCallback,
): Promise<DownloadResult> => { ): Promise<DownloadResult> => {
// As in `downloadFile`, with `thumb_` before the title.
const resolvedPath = const resolvedPath =
outPath ?? outPath ??
`thumb_${sanitizeFileName(file.metadata.title, `file-${file.id}`)}`; defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
const header = fromBase64(file.thumbnail.decryptionHeader); const header = fromBase64(file.thumbnail.decryptionHeader);
return fetchAndDecrypt( return fetchAndDecrypt(
api, api,
+56 -4
View File
@@ -5,7 +5,9 @@
// used on disk goes through here, so it can only ever name one file inside the // used on disk goes through here, so it can only ever name one file inside the
// directory the caller chose: never a path, never `..`, never hidden, never a // directory the caller chose: never a path, never `..`, never hidden, never a
// Windows device name. Such a name can still be too long for a file name, and // Windows device name. Such a name can still be too long for a file name, and
// `shortenToBytes` cuts one short. // `shortenToBytes` or `shortenKeepingExtension` cuts one short.
// `defaultFileName` does both for the name a file is saved under when the
// caller gives no path.
// //
// A path the user typed (`--out`, `outPath`) is not passed through here: the // A path the user typed (`--out`, `outPath`) is not passed through here: the
// caller is trusted, the server is not. // caller is trusted, the server is not.
@@ -30,11 +32,11 @@ export const sanitizeFileName = (name: string, fallback: string): string => {
return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned; return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned;
}; };
// The extension of `title` (".jpg"), or ".bin" when it has none or it holds // The extension of `title` (".jpg"), or ".bin" when it has none, it holds
// anything but letters and digits. // anything but letters and digits, or it is more than 16 of them long.
export const safeExtension = (title: string): string => { export const safeExtension = (title: string): string => {
const ext = extname(title); const ext = extname(title);
return /^\.[A-Za-z0-9]+$/.test(ext) ? ext : ".bin"; return /^\.[A-Za-z0-9]{1,16}$/.test(ext) ? ext : ".bin";
}; };
// `name` with its extension, if it has one, replaced by `ext` (".mov"). // `name` with its extension, if it has one, replaced by `ext` (".mov").
@@ -57,3 +59,53 @@ export const shortenToBytes = (name: string, maxBytes: number): string => {
} }
return name.slice(0, end); return name.slice(0, end);
}; };
// `name` cut short to fit in `maxBytes` bytes with `insert` put right before
// its extension (`IMG_0001 (12).jpg`), each cut made as `shortenToBytes`
// makes it. The part before the extension is cut first, so the extension is
// kept whenever it fits, but that part always keeps its first character:
// when the extension leaves no room for `insert` and that character, the
// extension is cut short too, and the part before it keeps the bytes that cut
// leaves over. So a name that does not start with a dot never comes back
// starting with one, which would hide it. `insert` is never cut. With no
// `insert`, a name that fits comes back unchanged.
export const shortenKeepingExtension = (
name: string,
maxBytes: number,
insert = "",
): string => {
const fullExt = extname(name);
const stem = name.slice(0, name.length - fullExt.length);
const first = Array.from(stem)[0] ?? "";
const ext = shortenToBytes(
fullExt,
maxBytes - Buffer.byteLength(insert + first),
);
const room = maxBytes - Buffer.byteLength(insert + ext);
return shortenToBytes(stem, room) + insert + ext;
};
// The name a file is saved under in the working directory when the caller
// gives no path (`quak get` without `--out`, `downloadFile` without
// `outPath`): `prefix` (`thumb_` for a thumbnail), then the file's title with
// unsafe characters replaced, or `file-<id>` when the title is empty, cut
// short so that the whole fits in a file name, keeping the title's extension
// whenever it fits. `ext` (".mov"), when given, takes the place of the
// title's extension before the name is cut short; a live photo's image and
// video are each named with their own. The cut can leave a device name
// (`CONSOLE.<long extension>` becomes `CON.<long extension>`); that name gets
// a leading `_`, as `sanitizeFileName` gives one, and is cut a byte shorter to
// leave room for it.
export const defaultFileName = (
title: string,
fileID: number,
opts: { prefix?: string; ext?: string } = {},
): string => {
const { prefix = "", ext } = opts;
const safe = sanitizeFileName(title, `file-${fileID}`);
const name = ext === undefined ? safe : withExtension(safe, ext);
const room = MAX_NAME_BYTES - Buffer.byteLength(prefix);
const cut = prefix + shortenKeepingExtension(name, room);
if (!RESERVED_DEVICE_NAME.test(cut)) return cut;
return `_${prefix}${shortenKeepingExtension(name, room - 1)}`;
};
+14 -2
View File
@@ -5,7 +5,11 @@ import type { Client } from "./client.js";
import { describeError } from "./errors.js"; import { describeError } from "./errors.js";
import { readExifTags } from "./exif.js"; import { readExifTags } from "./exif.js";
import type { Library, Photo } from "./library/index.js"; import type { Library, Photo } from "./library/index.js";
import { sanitizeFileName } from "./filename.js"; import {
MAX_NAME_BYTES,
sanitizeFileName,
shortenToBytes,
} from "./filename.js";
import { import {
fetchMLDataBatch, fetchMLDataBatch,
MLDATA_BATCH_SIZE, MLDATA_BATCH_SIZE,
@@ -127,7 +131,15 @@ export const runMetadataBackup = async (
const col = lib.getCollection(album.collectionID); const col = lib.getCollection(album.collectionID);
if (!col) continue; if (!col) continue;
const dirName = `${col.id}-${sanitizeFileName(col.name, "unnamed")}`; // The album's ID keeps the folders apart, so the name after it can be
// cut short to fit in a file name.
const prefix = `${col.id}-`;
const dirName =
prefix +
shortenToBytes(
sanitizeFileName(col.name, "unnamed"),
MAX_NAME_BYTES - prefix.length,
);
const colDir = join(outDir, "collections", dirName); const colDir = join(outDir, "collections", dirName);
mkdirSync(colDir, { recursive: true }); mkdirSync(colDir, { recursive: true });
+10 -11
View File
@@ -1795,12 +1795,11 @@ describe("backup album folders", () => {
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 }); expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 });
}); });
// The extensions of these titles leave no room for the ID, so each is cut // The extensions of these titles leave no room for the ID and the "a"
// short and the name before the ID keeps only the bytes that cut leaves // before them, so each is cut short to 250 bytes, which leaves one, for
// over. File 1's extension is 303 bytes and is cut to 251, which leaves // the "a". File 1's extension is 303 bytes and file 2's 271. Neither
// none. File 2's is 271 bytes and is cut to 250, since "€" takes three // extension is only letters and digits, so both originals are saved with
// bytes, which leaves one, for the "a". Neither extension is only letters // `.bin`, a name that fits.
// and digits, so both originals are saved with `.bin`, a name that fits.
it("cuts the extension of a title short when it leaves no room for the ID", async () => { it("cuts the extension of a title short when it leaves no room for the ID", async () => {
const outDir = join(root, "backup"); const outDir = join(root, "backup");
const lib = libraryOf([ const lib = libraryOf([
@@ -1815,7 +1814,7 @@ describe("backup album folders", () => {
const result = await runBackup(lib, { downloadDirectory: outDir }); const result = await runBackup(lib, { downloadDirectory: outDir });
const link = ` (1).${"L".repeat(250)}`; const link = `a (1).${"L".repeat(249)}`;
const link2 = `a (2).${"€".repeat(83)}`; const link2 = `a (2).${"€".repeat(83)}`;
expect(Buffer.byteLength(link)).toBe(255); expect(Buffer.byteLength(link)).toBe(255);
expect(Buffer.byteLength(link2)).toBe(255); expect(Buffer.byteLength(link2)).toBe(255);
@@ -1830,9 +1829,9 @@ describe("backup album folders", () => {
// Files 1 and 3 share a title, so each gets its ID before the // Files 1 and 3 share a title, so each gets its ID before the
// extension. File 2's title is too long, and its extension leaves no // extension. File 2's title is too long, and its extension leaves no
// room for the ID. Were the ID put at the end of file 2's title cut // room for the ID and the "x" before it. Were the ID put at the end of
// short, its link would have file 1's name, and the link made second // file 2's title cut short, its link would have file 1's name, and the
// would replace the first. // link made second would replace the first.
it("keeps names unique when a title's extension leaves no room for the ID", async () => { it("keeps names unique when a title's extension leaves no room for the ID", async () => {
const outDir = join(root, "backup"); const outDir = join(root, "backup");
const shared = `x.${"Q".repeat(245)} (2)`; const shared = `x.${"Q".repeat(245)} (2)`;
@@ -1852,8 +1851,8 @@ describe("backup album folders", () => {
expect(result.failed).toBe(0); expect(result.failed).toBe(0);
expect(tree(outDir)).toEqual([ expect(tree(outDir)).toEqual([
"Trip/", "Trip/",
`Trip/ (2).${"Q".repeat(250)} -> ${linkTo("2.bin")}`,
`Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`, `Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`,
`Trip/x (2).${"Q".repeat(249)} -> ${linkTo("2.bin")}`,
`Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`, `Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`,
"Trip.json", "Trip.json",
]); ]);
+139 -6
View File
@@ -21,7 +21,7 @@ import {
utimesSync, utimesSync,
writeFileSync, writeFileSync,
} from "node:fs"; } from "node:fs";
import { join } from "node:path"; import { basename, join } from "node:path";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { PassThrough } from "node:stream"; import { PassThrough } from "node:stream";
import * as jpegJs from "jpeg-js"; import * as jpegJs from "jpeg-js";
@@ -187,6 +187,9 @@ const FETCHED_BOTH_ALBUMS =
"Fetched the files of 1 of 2 collections\n" + "Fetched the files of 1 of 2 collections\n" +
"Fetched the files of 2 of 2 collections\n"; "Fetched the files of 2 of 2 collections\n";
// The same for an account with one album.
const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n";
// Collects everything written to it. // Collects everything written to it.
class Output { class Output {
text = ""; text = "";
@@ -562,17 +565,109 @@ describe("get and get-thumb", () => {
}); });
}); });
// A file system refuses a file name over 255 bytes. The account here is one
// album, whose name is 300 bytes, holding one file, 700, with the title given.
describe("names from an album name or a title over 255 bytes", () => {
const longNames = (title: string): Client =>
({
...fakeClient(),
collectionsSince: async () => ({
collections: [collection(7, "A".repeat(300))],
deleted: [],
cursor: 1,
}),
filesSince: async () => ({
files: [file(700, 7, title)],
deleted: [],
cursor: 1,
}),
}) as unknown as Client;
const longTitle = `${"t".repeat(296)}.jpg`;
// Run `command` in a new directory, where `get` and `get-thumb` without
// `--out` write, and return its exit code and the directory.
const inNewDirectory = async (
command: () => Promise<number>,
): Promise<{ code: number; dir: string }> => {
const dir = mkdtempSync(join(root, "cwd-"));
const previous = process.cwd();
process.chdir(dir);
try {
return { code: await command(), dir };
} finally {
process.chdir(previous);
}
};
it("get cuts the title short to 255 bytes, keeping its extension, and exits 0", async () => {
const { code, dir } = await inNewDirectory(() =>
getCommand(context(longNames(longTitle)), "700", {}),
);
const name = `${"t".repeat(251)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(code).toBe(0);
expect(readdirSync(dir)).toEqual([name]);
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `7 bytes -> ${name}\n`);
});
it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => {
const { code, dir } = await inNewDirectory(() =>
getThumbCommand(context(longNames(longTitle)), "700", {}),
);
const name = `thumb_${"t".repeat(245)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(code).toBe(0);
expect(readdirSync(dir)).toEqual([name]);
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `3 bytes -> ${name}\n`);
});
it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => {
const dir = join(root, "dump");
const code = await backupMetadataCommand(
context(longNames("beach.jpg")),
dir,
{},
);
const folder = `7-${"A".repeat(253)}`;
expect(Buffer.byteLength(folder)).toBe(255);
expect(code).toBe(0);
expect(readdirSync(join(dir, "collections"))).toEqual([folder]);
expect(readdirSync(join(dir, "collections", folder)).sort()).toEqual([
"700.json",
"_collection.json",
]);
});
// The title's extension is 300 letters, too long for the save path's
// name, so the original is saved with `.bin`.
it("backup saves a title with a 300-letter extension as .bin, and exits 0", async () => {
const dir = join(root, "backup");
const title = `b.${"e".repeat(300)}`;
const code = await backupCommand(context(longNames(title)), dir, {});
const path = savePath(dir, file(700, 7, title));
expect(code).toBe(0);
expect(basename(path)).toMatch(/^\d{4}-\d{2}-\d{2}\.700\.bin$/);
expect(readFileSync(path)).toEqual(Buffer.alloc(7, 700 & 0xff));
});
});
// A live photo, which Ente stores as one ZIP, is written as its image and its // A live photo, which Ente stores as one ZIP, is written as its image and its
// video, which a photo viewer can open. The account here is Vacation holding // video, which a photo viewer can open. The account here is Vacation holding
// one live photo, 300, downloaded through the real download layer // one live photo, 300, downloaded through the real download layer
// (test/live-photo.ts). // (test/live-photo.ts).
describe("a live photo", () => { describe("a live photo", () => {
// What `get` prints on stderr while its refresh fetches Vacation's files. const livePhotoClient = async (
const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n"; image = IMAGE,
title = "IMG_0300.HEIC",
const livePhotoClient = async (image = IMAGE): Promise<Client> => { ): Promise<Client> => {
const { file: live, body } = await asLivePhoto( const { file: live, body } = await asLivePhoto(
file(300, 1, "IMG_0300.HEIC"), file(300, 1, title),
livePhotoZip({ "image.heic": image, "video.mov": VIDEO }), livePhotoZip({ "image.heic": image, "video.mov": VIDEO }),
livePhotoHash(image, VIDEO), livePhotoHash(image, VIDEO),
); );
@@ -620,6 +715,44 @@ describe("a live photo", () => {
); );
}); });
// Run `get` without `--out` in a new directory for a live photo titled
// `title`, and return its exit code and the names it wrote.
const getInNewDirectory = async (
title: string,
): Promise<{ code: number; names: string[] }> => {
const ctx = context(await livePhotoClient(IMAGE, title));
const dir = mkdtempSync(join(root, "cwd-"));
const previous = process.cwd();
process.chdir(dir);
try {
const code = await getCommand(ctx, "300", {});
return { code, names: readdirSync(dir).sort() };
} finally {
process.chdir(previous);
}
};
const imageName = `${"t".repeat(250)}.heic`;
const videoName = `${"t".repeat(251)}.mov`;
// The title fits, but ".heic" is one byte longer than ".jpg", so the
// image's name is cut short once its extension is in it.
it("get cuts its image's name short for a 255-byte title ending in .jpg, and exits 0", async () => {
const title = `${"t".repeat(251)}.jpg`;
const result = await getInNewDirectory(title);
expect(Buffer.byteLength(title)).toBe(255);
expect(Buffer.byteLength(imageName)).toBe(255);
expect(Buffer.byteLength(videoName)).toBe(255);
expect(result).toEqual({ code: 0, names: [imageName, videoName] });
});
it("get cuts the names of its image and video short for a 300-byte title with no extension, and exits 0", async () => {
const result = await getInNewDirectory("t".repeat(300));
expect(result).toEqual({ code: 0, names: [imageName, videoName] });
});
it("get --out writes the image there and the video beside it", async () => { it("get --out writes the image there and the video beside it", async () => {
const out = join(root, "photo.jpg"); const out = join(root, "photo.jpg");
const video = join(root, "photo.mov"); const video = join(root, "photo.mov");
+89
View File
@@ -658,6 +658,26 @@ describe("downloadFile", () => {
expect(readdirSync(parent)).toEqual(["cwd"]); expect(readdirSync(parent)).toEqual(["cwd"]);
}); });
// A file system refuses a file name over 255 bytes.
it("cuts a title too long for a file name short, keeping its extension", async () => {
const plaintext = new Uint8Array([1, 2, 3]);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key);
const thumbPush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const file = buildMockEnteFile(key, header, thumbPush.header);
file.metadata.title = `${"t".repeat(296)}.jpg`;
const dir = mkdtempSync(join(testDir, "long-title-"));
const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) });
const result = await inDirectory(dir, () => downloadFile(api, file));
const name = `${"t".repeat(251)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(result.path).toBe(name);
expect(readdirSync(dir)).toEqual([name]);
});
it("uses an explicit outPath verbatim, even one with ..", async () => { it("uses an explicit outPath verbatim, even one with ..", async () => {
// The caller is trusted: its path is not sanitized. // The caller is trusted: its path is not sanitized.
const { api, file } = fixtureFor( const { api, file } = fixtureFor(
@@ -751,6 +771,28 @@ describe("downloadThumbnail", () => {
expect(result.path).toBe("thumb__etc_passwd"); expect(result.path).toBe("thumb__etc_passwd");
expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]); expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]);
}); });
it("leaves room for thumb_ when it cuts a title too long for a file name short", async () => {
const plaintext = new Uint8Array([0xff, 0xd8, 0xff, 0xe0]);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const filePush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const { header: thumbHeader, ciphertext: thumbCipher } =
encryptFileBody(plaintext, key);
const file = buildMockEnteFile(key, filePush.header, thumbHeader);
file.metadata.title = `${"t".repeat(296)}.jpg`;
const dir = mkdtempSync(join(testDir, "long-thumb-title-"));
const api = new ApiClient({ fetch: mockFetchForBody(thumbCipher) });
const result = await inDirectory(dir, () =>
downloadThumbnail(api, file),
);
const name = `thumb_${"t".repeat(245)}.jpg`;
expect(Buffer.byteLength(name)).toBe(255);
expect(result.path).toBe(name);
expect(readdirSync(dir)).toEqual([name]);
});
}); });
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -1881,6 +1923,53 @@ describe("downloadFile live photos", () => {
]); ]);
}); });
// A file system refuses a file name over 255 bytes. Each name is cut
// short once its own extension is in it.
const imageName = `${"t".repeat(250)}.heic`;
const videoName = `${"t".repeat(251)}.mov`;
// The title fits, but ".heic" is one byte longer than ".jpg".
it("cuts the image's name short for a 255-byte title ending in .jpg when no outPath is given", async () => {
const title = `${"t".repeat(251)}.jpg`;
const t = setup(livePhotoZip(), { ...livePhoto, title });
const result = await inDirectory(t.dir, () =>
downloadFile(t.api, t.file),
);
expect(Buffer.byteLength(title)).toBe(255);
expect(Buffer.byteLength(imageName)).toBe(255);
expect(Buffer.byteLength(videoName)).toBe(255);
expect(result.path).toBe(imageName);
expect(result.videoPath).toBe(videoName);
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
});
it("cuts the names of the image and the video short for a 300-byte title with no extension when no outPath is given", async () => {
const t = setup(livePhotoZip(), {
...livePhoto,
title: "t".repeat(300),
});
const result = await inDirectory(t.dir, () =>
downloadFile(t.api, t.file),
);
expect(result.path).toBe(imageName);
expect(result.videoPath).toBe(videoName);
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
});
// The caller's path is used as is; only the extension is each part's own.
it("keeps an outPath too long for a file name as given", async () => {
const t = setup(livePhotoZip(), livePhoto);
const outPath = join(t.dir, `${"t".repeat(251)}.jpg`);
await expect(downloadFile(t.api, t.file, outPath)).rejects.toThrow(
/ENAMETOOLONG/,
);
});
it("gives each part its own extension from the ZIP, letters and digits only", async () => { it("gives each part its own extension from the ZIP, letters and digits only", async () => {
const zip = livePhotoZip({ "image.JPG": IMAGE, "video.m-4v": VIDEO }); const zip = livePhotoZip({ "image.JPG": IMAGE, "video.m-4v": VIDEO });
const t = setup(zip, livePhoto); const t = setup(zip, livePhoto);
+108
View File
@@ -15,8 +15,10 @@
import { describe, expect, it } from "vitest"; import { describe, expect, it } from "vitest";
import { import {
defaultFileName,
safeExtension, safeExtension,
sanitizeFileName, sanitizeFileName,
shortenKeepingExtension,
shortenToBytes, shortenToBytes,
} from "../../src/filename.js"; } from "../../src/filename.js";
@@ -94,6 +96,12 @@ describe("safeExtension", () => {
expect(safeExtension("x.jp g")).toBe(".bin"); expect(safeExtension("x.jp g")).toBe(".bin");
expect(safeExtension("x.jpg\0")).toBe(".bin"); expect(safeExtension("x.jpg\0")).toBe(".bin");
}); });
it("keeps an extension of 16 letters and digits, and uses .bin for a longer one", () => {
expect(safeExtension(`x.${"e".repeat(16)}`)).toBe(`.${"e".repeat(16)}`);
expect(safeExtension(`x.${"e".repeat(17)}`)).toBe(".bin");
expect(safeExtension(`b.${"e".repeat(300)}`)).toBe(".bin");
});
}); });
describe("shortenToBytes", () => { describe("shortenToBytes", () => {
@@ -118,3 +126,103 @@ describe("shortenToBytes", () => {
); );
}); });
}); });
describe("shortenKeepingExtension", () => {
it("returns a name that fits unchanged", () => {
expect(shortenKeepingExtension("beach.jpg", 9)).toBe("beach.jpg");
expect(shortenKeepingExtension("beach", 5)).toBe("beach");
});
it("cuts the part before the extension short and keeps the extension", () => {
expect(shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255)).toBe(
`${"t".repeat(251)}.jpg`,
);
expect(shortenKeepingExtension("é".repeat(150), 255)).toBe(
"é".repeat(127),
);
});
// A name cut down to its extension would start with a dot, and so be
// hidden.
it("keeps the first character before the extension, cutting the extension short to leave room for it", () => {
expect(shortenKeepingExtension(`b.${"e".repeat(300)}`, 255)).toBe(
`b.${"e".repeat(253)}`,
);
expect(shortenKeepingExtension(`ab.${"e".repeat(300)}`, 255)).toBe(
`a.${"e".repeat(253)}`,
);
expect(shortenKeepingExtension(`é.${"e".repeat(300)}`, 255)).toBe(
`é.${"e".repeat(252)}`,
);
// "€" takes three bytes, so the extension is cut to 253 and the
// part before it keeps the two bytes left over, for the "a".
expect(shortenKeepingExtension(`a.${"€".repeat(90)}`, 255)).toBe(
`a.${"€".repeat(84)}`,
);
});
it("puts `insert` right before the extension and leaves room for it", () => {
expect(shortenKeepingExtension("beach.jpg", 255, " (7)")).toBe(
"beach (7).jpg",
);
expect(
shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255, " (7)"),
).toBe(`${"t".repeat(247)} (7).jpg`);
expect(
shortenKeepingExtension(`b.${"e".repeat(300)}`, 255, " (7)"),
).toBe(`b (7).${"e".repeat(249)}`);
});
});
describe("defaultFileName", () => {
it("is the title when it is safe and fits", () => {
expect(defaultFileName("IMG_0001.HEIC", 42)).toBe("IMG_0001.HEIC");
});
it("replaces unsafe characters, and is file-<id> for an empty title", () => {
expect(defaultFileName("../x.jpg", 42)).toBe("__x.jpg");
expect(defaultFileName("", 42)).toBe("file-42");
});
it("cuts a title over 255 bytes short, keeping its extension", () => {
expect(defaultFileName(`${"t".repeat(296)}.jpg`, 42)).toBe(
`${"t".repeat(251)}.jpg`,
);
expect(defaultFileName(`b.${"e".repeat(300)}`, 42)).toBe(
`b.${"e".repeat(253)}`,
);
});
it("puts `prefix` before the title and leaves room for it", () => {
expect(defaultFileName("beach.jpg", 42, { prefix: "thumb_" })).toBe(
"thumb_beach.jpg",
);
expect(
defaultFileName(`${"t".repeat(296)}.jpg`, 42, { prefix: "thumb_" }),
).toBe(`thumb_${"t".repeat(245)}.jpg`);
});
// A live photo's image and video are each named with their own
// extension, and the name is cut short once that extension is in it.
it("puts `ext` in place of the title's extension before cutting the name short", () => {
expect(defaultFileName("IMG_0001.HEIC", 42, { ext: ".mov" })).toBe(
"IMG_0001.mov",
);
expect(defaultFileName("", 42, { ext: ".mov" })).toBe("file-42.mov");
expect(
defaultFileName(`${"t".repeat(251)}.jpg`, 42, { ext: ".heic" }),
).toBe(`${"t".repeat(250)}.heic`);
expect(defaultFileName("t".repeat(300), 42, { ext: ".heic" })).toBe(
`${"t".repeat(250)}.heic`,
);
});
// Cutting `CONSOLE` or `NULL` short before a long extension leaves `CON`
// or `NUL`, so the name gets the `_` `sanitizeFileName` gives a device
// name, and the part before the extension gives up a byte for it.
it("never returns a Windows device name, even once cut short", () => {
const ext = `.${"e".repeat(251)}`;
expect(defaultFileName(`CONSOLE${ext}`, 42)).toBe(`_CO${ext}`);
expect(defaultFileName(`NULL${ext}`, 42)).toBe(`_NU${ext}`);
});
});