Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3c8377cb2c |
+1
-4
@@ -3,10 +3,7 @@
|
|||||||
# and dropping it would change what the lint phase's prettier check sees.
|
# and dropping it would change what the lint phase's prettier check sees.
|
||||||
#
|
#
|
||||||
# .git is deliberately NOT excluded: the build derives the version it stamps
|
# .git is deliberately NOT excluded: the build derives the version it stamps
|
||||||
# from it (script/version). It is sent without its config, which holds the
|
# from it (script/version).
|
||||||
# clone's remote URL and any credential in it, and which the build stage, the
|
|
||||||
# final image, would otherwise carry. git describe does not need it.
|
|
||||||
.git/config
|
|
||||||
|
|
||||||
# OS
|
# OS
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
|||||||
@@ -195,15 +195,13 @@ The build fails if the checkout has `.git` and the version still comes out
|
|||||||
empty, `dev` or `unknown`: such a build could not be traced back to its commit.
|
empty, `dev` or `unknown`: such a build could not be traced back to its commit.
|
||||||
|
|
||||||
`.dockerignore` therefore does not leave out `.git`, so any `docker build .` of
|
`.dockerignore` therefore does not leave out `.git`, so any `docker build .` of
|
||||||
a clone stamps the commit it was built from; a shallow clone stamps a tag only
|
a clone stamps the commit it was built from; a shallow clone of one branch has
|
||||||
when the cloned commit itself carries one, and otherwise the short commit. It
|
no tags and stamps the short commit. `script/docker` (and so `make docker`) and
|
||||||
leaves out `.git/config`, which holds the clone's remote URL and any credential
|
`script/cibuild` pass the version they resolve on the host, with `--dirty`, as
|
||||||
in it, so the image carries `.git` without its config; `git describe` does not
|
the build arg, which takes precedence. The image's
|
||||||
need that file. `script/docker` (and so `make docker`) and `script/cibuild` pass
|
`org.opencontainers.image.version` label carries that build arg only, so a build
|
||||||
the version they resolve on the host, with `--dirty`, as the build arg, which
|
given none leaves it empty. `make build-bin` bundles the built `dist/`, so the
|
||||||
takes precedence. The image's `org.opencontainers.image.version` label carries
|
single binary reports the stamped version too.
|
||||||
that build arg only, so a build given none leaves it empty. `make build-bin`
|
|
||||||
bundles the built `dist/`, so the single binary reports the stamped version too.
|
|
||||||
|
|
||||||
## Rationale
|
## Rationale
|
||||||
|
|
||||||
@@ -909,7 +907,10 @@ current account's records name.
|
|||||||
|
|
||||||
A live photo's original is cached as at its save path: its image and its video,
|
A live photo's original is cached as at its save path: its image and its video,
|
||||||
each `originals/<fileID>.<ext>` with its own extension, and
|
each `originals/<fileID>.<ext>` with its own extension, and
|
||||||
`originals/<fileID>.livephoto.json` naming them; the two are evicted together.
|
`originals/<fileID>.livephoto.json` naming them; the two are evicted together. A
|
||||||
|
live photo that an earlier version cached as its ZIP is not served: the library
|
||||||
|
removes the ZIP when it opens the cache, and fetches the two files when the
|
||||||
|
photo is next read or precached.
|
||||||
|
|
||||||
A stored file appears only via an atomic temp-then-rename, so its presence means
|
A stored file appears only via an atomic temp-then-rename, so its presence means
|
||||||
it is complete. Every downloaded original (by `quak get`, the cache, or
|
it is complete. Every downloaded original (by `quak get`, the cache, or
|
||||||
|
|||||||
@@ -34,10 +34,6 @@ declares one.
|
|||||||
`make build-bin` bundles the built `dist/`, so the single binary reports the
|
`make build-bin` bundles the built `dist/`, so the single binary reports the
|
||||||
same version.
|
same version.
|
||||||
|
|
||||||
- 2026-10-01: The content cache no longer looks for a live photo that an earlier
|
|
||||||
version cached as one ZIP (issue 151). When the cache opens, a live photo's
|
|
||||||
file that no JSON file names is now always left alone.
|
|
||||||
|
|
||||||
- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
|
- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
|
||||||
every album downloads each photo to its save path, writes the photo's record
|
every album downloads each photo to its save path, writes the photo's record
|
||||||
and EXIF fields to a JSON file beside it, and writes the album's photos to
|
and EXIF fields to a JSON file beside it, and writes the album's photos to
|
||||||
|
|||||||
@@ -361,8 +361,8 @@ const openPart = async (
|
|||||||
// written unpacked: each part is named `destination` with the extension
|
// written unpacked: each part is named `destination` with the extension
|
||||||
// replaced by its own entry's, and the two must differ ignoring case. When the
|
// replaced by its own entry's, and the two must differ ignoring case. When the
|
||||||
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
|
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
|
||||||
// part's own bytes. Only then are the image, then the video, renamed into
|
// part's own bytes. Only then is whatever was at `destination` removed and the
|
||||||
// place; on any failure neither is stored.
|
// image, then the video, renamed into place; on any failure neither is stored.
|
||||||
//
|
//
|
||||||
// The ZIP is chosen by its uploader and may expand enormously, so each part is
|
// The ZIP is chosen by its uploader and may expand enormously, so each part is
|
||||||
// written as it decompresses and never held, and the ZIP is refused once the
|
// written as it decompresses and never held, and the ZIP is refused once the
|
||||||
@@ -486,6 +486,7 @@ const decryptLivePhoto = async (
|
|||||||
await part.handle.sync();
|
await part.handle.sync();
|
||||||
await part.handle.close();
|
await part.handle.close();
|
||||||
}
|
}
|
||||||
|
await rm(destination, { force: true });
|
||||||
await rename(image.tmpPath, path);
|
await rename(image.tmpPath, path);
|
||||||
try {
|
try {
|
||||||
await rename(video.tmpPath, videoPath);
|
await rename(video.tmpPath, videoPath);
|
||||||
@@ -569,7 +570,8 @@ const fetchAndDecrypt = async (
|
|||||||
}, api.getRetryOptions());
|
}, api.getRetryOptions());
|
||||||
|
|
||||||
// Write `file`'s original to `outPath`. A live photo is written as its image
|
// Write `file`'s original to `outPath`. A live photo is written as its image
|
||||||
// and its video beside `outPath` instead (see `decryptLivePhoto`).
|
// and its video beside `outPath` instead, and whatever was at `outPath` is
|
||||||
|
// removed (see `decryptLivePhoto`).
|
||||||
export const downloadFile = async (
|
export const downloadFile = async (
|
||||||
api: ApiClient,
|
api: ApiClient,
|
||||||
file: EnteFile,
|
file: EnteFile,
|
||||||
|
|||||||
+2
-4
@@ -1,7 +1,5 @@
|
|||||||
// A build reports the version script/build stamps into dist/package.json;
|
// package.json is the one place the version is written. tsc copies it to
|
||||||
// package.json's own version is reported only when running from source. tsc
|
// dist/package.json, so this path resolves from source and from dist/src/.
|
||||||
// copies package.json to dist/package.json, so this path resolves from source
|
|
||||||
// and from dist/src/.
|
|
||||||
import pkg from "../package.json" with { type: "json" };
|
import pkg from "../package.json" with { type: "json" };
|
||||||
|
|
||||||
export const VERSION: string = pkg.version;
|
export const VERSION: string = pkg.version;
|
||||||
|
|||||||
+38
-8
@@ -29,7 +29,14 @@
|
|||||||
// the cache does not count as saved there, but is copied there rather than
|
// the cache does not count as saved there, but is copied there rather than
|
||||||
// fetched again.
|
// fetched again.
|
||||||
|
|
||||||
import { existsSync, readFileSync, statSync } from "node:fs";
|
import {
|
||||||
|
closeSync,
|
||||||
|
existsSync,
|
||||||
|
openSync,
|
||||||
|
readFileSync,
|
||||||
|
readSync,
|
||||||
|
statSync,
|
||||||
|
} from "node:fs";
|
||||||
import {
|
import {
|
||||||
chmod,
|
chmod,
|
||||||
copyFile,
|
copyFile,
|
||||||
@@ -274,6 +281,25 @@ const fileSize = (path: string): number | undefined => {
|
|||||||
const hasContent = (path: string | undefined): boolean =>
|
const hasContent = (path: string | undefined): boolean =>
|
||||||
path !== undefined && (fileSize(path) ?? 0) > 0;
|
path !== undefined && (fileSize(path) ?? 0) > 0;
|
||||||
|
|
||||||
|
// Whether the file at `path` begins as a ZIP does, with `PK\x03\x04`. False
|
||||||
|
// when it cannot be read.
|
||||||
|
const isZip = (path: string): boolean => {
|
||||||
|
try {
|
||||||
|
const fd = openSync(path, "r");
|
||||||
|
try {
|
||||||
|
const head = Buffer.alloc(4);
|
||||||
|
return (
|
||||||
|
readSync(fd, head, 0, 4, 0) === 4 &&
|
||||||
|
head.toString("latin1") === "PK\x03\x04"
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
closeSync(fd);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
// A live photo's image and video are named with the extensions from inside its
|
// A live photo's image and video are named with the extensions from inside its
|
||||||
// ZIP, so their names alone do not say which is which. Wherever the cache or a
|
// ZIP, so their names alone do not say which is which. Wherever the cache or a
|
||||||
// save path stores one, a JSON file of this name beside them names both.
|
// save path stores one, a JSON file of this name beside them names both.
|
||||||
@@ -687,10 +713,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
await this.touch(cached.path);
|
await this.touch(cached.path);
|
||||||
return { ...cached, bytes: size, cached: true };
|
return { ...cached, bytes: size, cached: true };
|
||||||
}
|
}
|
||||||
// A recorded file that has since gone re-fetches below. So does a
|
// A recorded file that has since gone, or a live photo an earlier
|
||||||
// live photo recorded with no video: the cache opened before the
|
// version stored as one ZIP, re-fetches below.
|
||||||
// library's records said it is a live photo, while its image and
|
|
||||||
// video had no JSON file beside them yet.
|
|
||||||
known.delete(fileID);
|
known.delete(fileID);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -959,14 +983,20 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
if (id === undefined || !existsSync(path)) continue;
|
if (id === undefined || !existsSync(path)) continue;
|
||||||
// A live photo's image and video are one entry, as the JSON file
|
// A live photo's image and video are one entry, as the JSON file
|
||||||
// beside them names them. A live photo's file with no such JSON
|
// beside them names them. A live photo's file with no such JSON
|
||||||
// file is not its original and is left alone: another process may
|
// file is not its original. If it is a ZIP, it is the one an
|
||||||
// have just stored it and not yet written the JSON file.
|
// earlier version stored under the image's name, and is removed.
|
||||||
|
// Any other is left alone: another process may have just stored
|
||||||
|
// it and not yet written the JSON file.
|
||||||
const livePhoto = names.has(livePhotoJSONName(String(id)))
|
const livePhoto = names.has(livePhotoJSONName(String(id)))
|
||||||
? readLivePhotoJSON(dir, String(id))
|
? readLivePhotoJSON(dir, String(id))
|
||||||
: undefined;
|
: undefined;
|
||||||
if (livePhoto !== undefined) {
|
if (livePhoto !== undefined) {
|
||||||
into.set(id, livePhoto);
|
into.set(id, livePhoto);
|
||||||
} else if (!isLivePhoto(id)) {
|
} else if (isLivePhoto(id)) {
|
||||||
|
if (isZip(path)) {
|
||||||
|
await rm(path, { force: true }).catch(() => undefined);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
into.set(id, { path });
|
into.set(id, { path });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1883,6 +1883,15 @@ describe("downloadFile live photos", () => {
|
|||||||
expect(readdirSync(t.dir).sort()).toEqual(["f.JPG", "f.bin"]);
|
expect(readdirSync(t.dir).sort()).toEqual(["f.JPG", "f.bin"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("replaces what was at the destination, such as an earlier ZIP of the two", async () => {
|
||||||
|
const t = setup(livePhotoZip(), livePhoto);
|
||||||
|
writeFileSync(t.outPath, livePhotoZip());
|
||||||
|
|
||||||
|
await t.run();
|
||||||
|
|
||||||
|
expect(readdirSync(t.dir).sort()).toEqual(["f.heic", "f.mov"]);
|
||||||
|
});
|
||||||
|
|
||||||
it("renames the image and then the video into place, each from its own temp file", async () => {
|
it("renames the image and then the video into place, each from its own temp file", async () => {
|
||||||
const t = setup(livePhotoZip(), livePhoto);
|
const t = setup(livePhotoZip(), livePhoto);
|
||||||
|
|
||||||
|
|||||||
@@ -176,7 +176,7 @@ describe("Library content wiring", () => {
|
|||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("does not take a live photo's image or video with no JSON file as its original when it opens, and precaches both", async () => {
|
it("removes a live photo's ZIP an earlier version cached when it opens, and precaches its image and video", async () => {
|
||||||
const { file: live, body } = await asLivePhoto(file(1, 1));
|
const { file: live, body } = await asLivePhoto(file(1, 1));
|
||||||
class LiveClient extends MockClient {
|
class LiveClient extends MockClient {
|
||||||
override async filesSince(): Promise<FilesPage> {
|
override async filesSince(): Promise<FilesPage> {
|
||||||
@@ -197,8 +197,7 @@ describe("Library content wiring", () => {
|
|||||||
// A first run records the library, so the next one knows that file 1
|
// A first run records the library, so the next one knows that file 1
|
||||||
// is a live photo when it opens the cache.
|
// is a live photo when it opens the cache.
|
||||||
await (await open({})).close();
|
await (await open({})).close();
|
||||||
writeFileSync(join(originals, "1.heic"), "an image");
|
writeFileSync(join(originals, "1.jpg"), livePhotoZip());
|
||||||
writeFileSync(join(originals, "1.mov"), "a video");
|
|
||||||
|
|
||||||
let precached!: () => void;
|
let precached!: () => void;
|
||||||
const done = new Promise<void>((r) => (precached = r));
|
const done = new Promise<void>((r) => (precached = r));
|
||||||
@@ -209,9 +208,7 @@ describe("Library content wiring", () => {
|
|||||||
precached();
|
precached();
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
expect(
|
expect(existsSync(join(originals, "1.jpg"))).toBe(false);
|
||||||
lib.photos.byID({ fileID: 1 })!.record().originalPath,
|
|
||||||
).toBeUndefined();
|
|
||||||
await done;
|
await done;
|
||||||
|
|
||||||
expect(readdirSync(originals).sort()).toEqual([
|
expect(readdirSync(originals).sort()).toEqual([
|
||||||
@@ -219,17 +216,6 @@ describe("Library content wiring", () => {
|
|||||||
"1.livephoto.json",
|
"1.livephoto.json",
|
||||||
"1.mov",
|
"1.mov",
|
||||||
]);
|
]);
|
||||||
expect(readFileSync(join(originals, "1.heic"))).toEqual(
|
|
||||||
Buffer.from(IMAGE),
|
|
||||||
);
|
|
||||||
expect(readFileSync(join(originals, "1.mov"))).toEqual(
|
|
||||||
Buffer.from(VIDEO),
|
|
||||||
);
|
|
||||||
expect(
|
|
||||||
JSON.parse(
|
|
||||||
readFileSync(join(originals, "1.livephoto.json"), "utf-8"),
|
|
||||||
),
|
|
||||||
).toEqual({ image: "1.heic", video: "1.mov" });
|
|
||||||
expect(lib.photos.byID({ fileID: 1 })!.record().originalPath).toBe(
|
expect(lib.photos.byID({ fileID: 1 })!.record().originalPath).toBe(
|
||||||
join(originals, "1.heic"),
|
join(originals, "1.heic"),
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -566,6 +566,43 @@ describe("ContentCache live photos", () => {
|
|||||||
expect(events).toEqual(["skipped"]);
|
expect(events).toEqual(["skipped"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("replaces a live photo an earlier version stored as a ZIP under the image's name", async () => {
|
||||||
|
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
||||||
|
mkdirSync(originals(), { recursive: true });
|
||||||
|
writeFileSync(join(originals(), "5.HEIC"), livePhotoZip());
|
||||||
|
const cache = cacheOf([live], new Map([[5, body]]));
|
||||||
|
// Opened without being told that file 5 is a live photo, the cache
|
||||||
|
// records the ZIP, and does not serve it.
|
||||||
|
await cache.open();
|
||||||
|
|
||||||
|
const result = await cache.original(5);
|
||||||
|
|
||||||
|
expect(result.videoPath).toBe(join(originals(), "5.mov"));
|
||||||
|
expect(readdirSync(originals()).sort()).toEqual([
|
||||||
|
"5.heic",
|
||||||
|
"5.livephoto.json",
|
||||||
|
"5.mov",
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("removes a live photo's ZIP an earlier version stored when it opens, so the precache fetches the image and video", async () => {
|
||||||
|
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
||||||
|
mkdirSync(originals(), { recursive: true });
|
||||||
|
writeFileSync(join(originals(), "5.HEIC"), livePhotoZip());
|
||||||
|
const cache = cacheOf([live], new Map([[5, body]]));
|
||||||
|
|
||||||
|
await cache.open((fileID) => fileID === 5);
|
||||||
|
|
||||||
|
expect(readdirSync(originals())).toEqual([]);
|
||||||
|
expect(cache.pathsFor(5)).toEqual({});
|
||||||
|
const [fetched] = await cache.ensureOriginals({ fileIDs: [5] });
|
||||||
|
expect(fetched).toEqual({
|
||||||
|
fileID: 5,
|
||||||
|
path: join(originals(), "5.heic"),
|
||||||
|
});
|
||||||
|
expect(cache.pathsFor(5)).toEqual({ originalPath: fetched!.path });
|
||||||
|
});
|
||||||
|
|
||||||
it("leaves the image and video another process has just stored when it opens before their JSON file is written", async () => {
|
it("leaves the image and video another process has just stored when it opens before their JSON file is written", async () => {
|
||||||
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
||||||
const server = cdnSource(new Map([[5, body]]));
|
const server = cdnSource(new Map([[5, body]]));
|
||||||
@@ -596,36 +633,6 @@ describe("ContentCache live photos", () => {
|
|||||||
expect(second!.pathsFor(5)).toEqual({});
|
expect(second!.pathsFor(5)).toEqual({});
|
||||||
});
|
});
|
||||||
|
|
||||||
it("fetches a live photo's image and video again when the cache opened before knowing it is a live photo and no JSON file names them", async () => {
|
|
||||||
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
|
|
||||||
mkdirSync(originals(), { recursive: true });
|
|
||||||
writeFileSync(join(originals(), "5.heic"), "an image");
|
|
||||||
writeFileSync(join(originals(), "5.mov"), "a video");
|
|
||||||
const cache = cacheOf([live], new Map([[5, body]]));
|
|
||||||
// Opened without being told that file 5 is a live photo, the cache
|
|
||||||
// records one of the two files as its original, with no video.
|
|
||||||
await cache.open();
|
|
||||||
|
|
||||||
const events: string[] = [];
|
|
||||||
const result = await cache.original(5, {
|
|
||||||
onProgress: (e) => events.push(e.status),
|
|
||||||
});
|
|
||||||
|
|
||||||
expect(events.at(-1)).toBe("done");
|
|
||||||
expect(result).toEqual({
|
|
||||||
path: join(originals(), "5.heic"),
|
|
||||||
videoPath: join(originals(), "5.mov"),
|
|
||||||
bytes: IMAGE.length,
|
|
||||||
});
|
|
||||||
expect(readFileSync(result.path)).toEqual(Buffer.from(IMAGE));
|
|
||||||
expect(readFileSync(result.videoPath!)).toEqual(Buffer.from(VIDEO));
|
|
||||||
expect(
|
|
||||||
JSON.parse(
|
|
||||||
readFileSync(join(originals(), "5.livephoto.json"), "utf-8"),
|
|
||||||
),
|
|
||||||
).toEqual({ image: "5.heic", video: "5.mov" });
|
|
||||||
});
|
|
||||||
|
|
||||||
it.each(["missing", "empty"])(
|
it.each(["missing", "empty"])(
|
||||||
"fetches a live photo again when the video its JSON file names is %s",
|
"fetches a live photo again when the video its JSON file names is %s",
|
||||||
async (state) => {
|
async (state) => {
|
||||||
|
|||||||
@@ -54,12 +54,6 @@ describe(".dockerignore", () => {
|
|||||||
expect(dockerignore).not.toContain(".git/");
|
expect(dockerignore).not.toContain(".git/");
|
||||||
});
|
});
|
||||||
|
|
||||||
// The build stage is the final image, so a .git/config sent in would
|
|
||||||
// ship the clone's remote URL and any credential in it.
|
|
||||||
it("sends .git without its config", () => {
|
|
||||||
expect(dockerignore).toContain(".git/config");
|
|
||||||
});
|
|
||||||
|
|
||||||
// BuildKit lets a `Dockerfile.dockerignore` shadow the root one; such a
|
// BuildKit lets a `Dockerfile.dockerignore` shadow the root one; such a
|
||||||
// file would silently give the build a different, unreviewed context —
|
// file would silently give the build a different, unreviewed context —
|
||||||
// and eslint's flat config does not ignore dot-directories, so a stray
|
// and eslint's flat config does not ignore dot-directories, so a stray
|
||||||
|
|||||||
@@ -787,13 +787,12 @@ describe("fixMissingThumbnails", () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it("re-encodes smaller until the thumbnail fits the recorded size", async () => {
|
it("re-encodes smaller until the thumbnail fits the recorded size", async () => {
|
||||||
// A noisy 64x48 JPEG, which the default encoding (quality 50, not
|
// A noisy 400x300 JPEG, which the default encoding (quality 50, not
|
||||||
// resized because it is under 720 px) cannot compress below the size
|
// resized because it is under 720 px) cannot compress below the size
|
||||||
// recorded here: one byte less than that encoding's ciphertext. It is
|
// recorded here: one byte less than that encoding's ciphertext.
|
||||||
// small so that each encode is quick even on a busy host.
|
|
||||||
const fixMock = await buildThumbMock();
|
const fixMock = await buildThumbMock();
|
||||||
const w = 64;
|
const w = 400;
|
||||||
const h = 48;
|
const h = 300;
|
||||||
const noisy = new Uint8Array(
|
const noisy = new Uint8Array(
|
||||||
jpegJs.encode(
|
jpegJs.encode(
|
||||||
{
|
{
|
||||||
|
|||||||
Reference in New Issue
Block a user