Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
98f674c3a3 |
@@ -1102,8 +1102,8 @@ It also awaits it on a cache that holds only what `lib.refreshCollection()`
|
||||
stored. If that refresh fails, it opens onto what the cache holds and records
|
||||
the error in `lib.status()`. On any other cache it returns immediately and
|
||||
refreshes in the background, so an unreachable server does not block opening.
|
||||
With `refreshOnOpen: false` it starts no refresh and returns at once onto what
|
||||
the cache holds, empty or not; a refresh then starts only when `fresh()` or
|
||||
With `refreshOnOpen: false` it starts no refresh, and returns at once onto what
|
||||
the cache holds, empty or not. A refresh then starts only when `fresh()` or
|
||||
`lib.backup()` asks, or the background timer fires. The CLI's read commands open
|
||||
it so (see "CLI surface").
|
||||
|
||||
@@ -1132,8 +1132,8 @@ when the library opens; `lib.downloadDirectory` holds the result.
|
||||
`onProgress` receives a `RefreshEvent` with status `"started"` as a refresh, an
|
||||
ML data fetch or a precache sweep starts, and `"done"` or `"failed"` as it ends.
|
||||
In between, a refresh sends `"progress"` after each collection whose files it
|
||||
fetched, with `fetched`, how many collections it has fetched the files of so
|
||||
far, and `total`, how many it will. A refresh fetches the files of each
|
||||
fetched. That event's `fetched` is how many collections it has fetched the files
|
||||
of so far, and its `total` how many it will. A refresh fetches the files of each
|
||||
collection that is new or changed since the cache's copy, so on an empty cache
|
||||
it fetches those of every collection.
|
||||
|
||||
@@ -1166,27 +1166,31 @@ cache.
|
||||
|
||||
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
||||
synchronously from the copy held in RAM and never touch the network. A refresh
|
||||
changes that copy only once all its server requests have succeeded, and the
|
||||
background timer starts the next refresh `refreshIntervalSeconds` after the
|
||||
previous one ends. So a default read is immediate, but only as current as the
|
||||
last refresh whose requests all succeeded; right after opening an existing
|
||||
cache, it is the copy on disk.
|
||||
changes that copy only once all its server requests have succeeded.
|
||||
`lib.refreshCollection()` changes it the same way for one collection. So a
|
||||
default read is immediate, but only as current as the last refresh or
|
||||
`lib.refreshCollection()` whose requests all succeeded. Right after opening an
|
||||
existing cache, it is the copy on disk.
|
||||
|
||||
The background timer takes a turn `refreshIntervalSeconds` after its last turn
|
||||
ends. Each turn starts a refresh, unless a refresh or `lib.refreshCollection()`
|
||||
is already running.
|
||||
|
||||
`await lib.fresh()` waits for a refresh to complete and persist, and returns the
|
||||
same `{ albums, photos, timeline }` namespaces, which then reflect a completed
|
||||
server round-trip. When a refresh is already running, background or not,
|
||||
`fresh()` waits for that one, so its answer can come from requests made before
|
||||
the call; only when none is running does it start one. A refresh that fails
|
||||
rejects the caller (default reads stay silent and keep serving the last good
|
||||
copy). The CLI's read commands use fresh reads (issue
|
||||
https://git.eeqj.de/sneak/quak/issues/75).
|
||||
server round-trip. It first waits for a running `lib.refreshCollection()` to
|
||||
end. When a refresh is already running, background or not, `fresh()` waits for
|
||||
that one. Its answer can then come from requests made before the call. Only when
|
||||
none is running does it start one. A refresh that fails rejects the caller
|
||||
(default reads stay silent and keep serving the last good copy). The CLI's read
|
||||
commands use fresh reads (issue https://git.eeqj.de/sneak/quak/issues/75).
|
||||
|
||||
`await lib.freshCollections()` is the fresh read of the collection list alone:
|
||||
one collections request, answered with every live collection that decrypts, as
|
||||
the `Collection` objects `listCollections()` returns, in the server's order. It
|
||||
fetches no collection's files and stores nothing, not even the list: a refresh
|
||||
fetches no collection's files and stores nothing, not even the list. A refresh
|
||||
fetches a collection's files only when the collection is new or has changed
|
||||
since the stored copy, so a collection stored without its files would never get
|
||||
since the stored copy. A collection stored without its files would never get
|
||||
them. `quak collections` uses it (issue
|
||||
https://git.eeqj.de/sneak/quak/issues/210).
|
||||
|
||||
@@ -1200,19 +1204,20 @@ not decrypt. Otherwise it fetches the collection's files changed since the
|
||||
cache's copy, or all of them when there is none, then stores and saves them. The
|
||||
refresh cursor stays where it was, so the next refresh still fetches the files
|
||||
of every other collection that changed. It never runs at the same time as a
|
||||
refresh. It waits for a running refresh to end, and a `fresh()` or
|
||||
`lib.backup()` called meanwhile waits for it. `quak files` with a collection ID
|
||||
refresh. It first waits for a running refresh to end. A `fresh()` or
|
||||
`lib.backup()` called after that waits for it. `quak files` with a collection ID
|
||||
uses it.
|
||||
|
||||
An album or file in the server's answer that quak cannot decrypt, or whose
|
||||
metadata is not JSON, does not fail the refresh. The refresh leaves it out, an
|
||||
album with all its files, and drops any older copy the cache held, so reads
|
||||
answer from what did decrypt. The library keeps a list of what it left out, with
|
||||
the reason, in `metadata.json`: the next refresh asks the server only for what
|
||||
the reason, in `metadata.json`. The next refresh asks the server only for what
|
||||
changed, so it would not see the record again. A record stays on that list until
|
||||
a later refresh gets a copy that decrypts or learns that the server deleted it.
|
||||
`lib.backup()` reports each such album, and each such file in its scope that has
|
||||
no copy that decrypted (see "Backup layout").
|
||||
a later refresh or `lib.refreshCollection()` stores a copy that decrypts, or
|
||||
learns that the server deleted it. `lib.backup()` reports each such album, and
|
||||
each such file in its scope that has no copy that decrypted (see "Backup
|
||||
layout").
|
||||
|
||||
### Read surface
|
||||
|
||||
@@ -1306,10 +1311,11 @@ The GUI-facing records hold no key material and no binary, so they survive
|
||||
- `LibrarySnapshot`: `{ albums, photos, takenAt }`.
|
||||
|
||||
`lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first,
|
||||
photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`,
|
||||
`refreshedAt`) whenever a refresh alters the projection, and returns
|
||||
`{ unsubscribe }`; a refresh that changes nothing delivers nothing.
|
||||
photos newest first). `lib.subscribe({ onChange })` returns `{ unsubscribe }`.
|
||||
It delivers a `LibraryChange` (`albumsChanged`, `photosChanged`,
|
||||
`fileIDsRemoved`, `albumIDsRemoved`, `refreshedAt`) whenever a refresh or
|
||||
`lib.refreshCollection()` alters the projection. One that changes nothing
|
||||
delivers nothing.
|
||||
|
||||
### Thumbnails, ML search, and backup
|
||||
|
||||
@@ -1328,26 +1334,27 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
- `await lib.backup(opts?)` → `BackupResult`. It takes the lock in the download
|
||||
directory, and fails at once with an error whose `code` is `ELOCKED` while
|
||||
another backup of it runs. A caller can instead take the lock itself, before
|
||||
it opens its library, as `quak backup` does: `await lockBackupDirectory(dir)`
|
||||
takes it, failing the same way, and returns a `BackupLock`, whose `release()`
|
||||
removes the lock unless it was lost and whose `lost` holds why it was lost.
|
||||
The caller passes it to the backup as `lock`, and releases it once the backup
|
||||
ends. It waits for a refresh as `fresh()` does, puts every in-scope original
|
||||
not already at its save path there as `photo.download()` does (and, with
|
||||
`includeThumbnails`, fetches thumbnails) through the content cache, waits for
|
||||
an ML data fetch, and rebuilds the on-disk backup tree with a durable failure
|
||||
ledger. Each file's JSON holds its ML data and keywords. An image's JSON also
|
||||
holds the EXIF and XMP of its original, and its dimensions if it is a JPEG. A
|
||||
live photo's JSON holds these for its image. A video's JSON holds no EXIF, XMP
|
||||
or dimensions. A fetched original is written straight to its save path and not
|
||||
into the cache, which then counts it as present; one the cache already held is
|
||||
copied from there. `BackupOptions`: `downloadDirectory` (falls back to the
|
||||
library's), `includeOriginals` (default `true`), `includeThumbnails` (default
|
||||
`false`), `onlyAlbumNames`, `verify` (default `false`), `onProgress`, and
|
||||
`lock`. See Backup layout above for the tree it writes, and for the albums
|
||||
`BackupResult` lists in `undecryptableAlbums` and `failedAlbums`. Whether the
|
||||
backup took the lock or its caller did, once the lock is lost the backup
|
||||
rejects with `lost the lock on <dir>: <reason>` before it starts another file.
|
||||
it opens its library, as `quak backup` does. `await lockBackupDirectory(dir)`
|
||||
takes it, failing the same way, and returns a `BackupLock`. Its `release()`
|
||||
removes the lock unless it was lost, and its `lost` holds why it was lost. The
|
||||
caller passes it to the backup as `lock`, and releases it once the backup
|
||||
ends. The backup waits for a refresh as `fresh()` does. It then puts every
|
||||
in-scope original not already at its save path there, as `photo.download()`
|
||||
does, through the content cache. With `includeThumbnails` it also fetches
|
||||
thumbnails. It waits for an ML data fetch, and rebuilds the on-disk backup
|
||||
tree with a durable failure ledger. Each file's JSON holds its ML data and
|
||||
keywords. An image's JSON also holds the EXIF and XMP of its original, and its
|
||||
dimensions if it is a JPEG. A live photo's JSON holds these for its image. A
|
||||
video's JSON holds no EXIF, XMP or dimensions. A fetched original is written
|
||||
straight to its save path and not into the cache, which then counts it as
|
||||
present; one the cache already held is copied from there. `BackupOptions`:
|
||||
`downloadDirectory` (falls back to the library's), `includeOriginals` (default
|
||||
`true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
|
||||
(default `false`), `onProgress`, and `lock`. See Backup layout above for the
|
||||
tree it writes, and for the albums `BackupResult` lists in
|
||||
`undecryptableAlbums` and `failedAlbums`. Whether the backup took the lock or
|
||||
its caller did, once the lock is lost the backup rejects with
|
||||
`lost the lock on <dir>: <reason>` before it starts another file.
|
||||
|
||||
### Request pools
|
||||
|
||||
|
||||
+137
-120
@@ -1,32 +1,33 @@
|
||||
// The library surface over the local cache.
|
||||
//
|
||||
// `Library.open()` loads the on-disk metadata store (issue #41), then starts
|
||||
// the refresh loop. When no refresh has stored a collection yet (the stored
|
||||
// cursor is 0), `open()` awaits the first refresh. The cache is then empty, or
|
||||
// holds only what `refreshCollection()` stored, and the library must not serve
|
||||
// it as the whole account. Otherwise that first refresh runs in the
|
||||
// the refresh loop. When the stored cursor is 0, `open()` awaits the first
|
||||
// refresh. No refresh has then stored any collection: the cache is empty, or
|
||||
// holds only what `refreshCollection()` stored. The library must not serve
|
||||
// that as the whole account. Otherwise that first refresh runs in the
|
||||
// background. `open()` returns as soon as the cached data is ready to serve,
|
||||
// so a slow or unreachable server does not stall opening. A background timer
|
||||
// then refreshes every `refreshIntervalSeconds`. Every default read is
|
||||
// answered from RAM — no default read touches the network. There is
|
||||
// deliberately no `sync()`, no `refresh()`, no `serverReachable` flag, and no
|
||||
// "before each read" mode (design #36).
|
||||
// then takes a turn `refreshIntervalSeconds` after its last turn ends. Each
|
||||
// turn starts a refresh, unless a refresh or `refreshCollection()` is already
|
||||
// running. Every default read is answered from RAM — no default read touches
|
||||
// the network. There is deliberately no `sync()`, no `refresh()`, no
|
||||
// `serverReachable` flag, and no "before each read" mode (design #36).
|
||||
//
|
||||
// `fresh()` is the one exception (issue #75, an owner amendment to #36): it
|
||||
// forces a refresh, awaits it, and only then hands back the read namespaces, so
|
||||
// a caller that needs server-current data can ask for it. Concurrent `fresh()`
|
||||
// calls coalesce onto one in-flight refresh, and a refresh that fails rejects
|
||||
// the caller (the default reads stay silent and serve the last good copy). The
|
||||
// default methods and the background loop are unchanged.
|
||||
// `fresh()` is an exception (issue #75, an owner amendment to #36). It awaits
|
||||
// a refresh, and only then hands back the read namespaces. A caller that needs
|
||||
// server-current data uses it. Concurrent `fresh()` calls coalesce onto one
|
||||
// in-flight refresh. A refresh that fails rejects the caller. The default
|
||||
// reads stay silent and serve the last good copy. The default methods and the
|
||||
// background loop are unchanged.
|
||||
//
|
||||
// `freshCollections()` is the fresh read of the collection list alone (issue
|
||||
// #210): one collections request, no collection's files, and nothing stored,
|
||||
// for a caller such as `quak collections` that needs no more.
|
||||
// `refreshCollection()` brings one collection from that list up to date in the
|
||||
// store (issue #217). It fetches that collection's files alone, as a refresh
|
||||
// would. `quak files COLLECTION` uses it. It never runs at the same time as a
|
||||
// refresh. A caller that asks for each refresh it needs can also open the
|
||||
// library with `refreshOnOpen: false`, so that `open()` starts none.
|
||||
// #210). It sends one collections request and stores nothing.
|
||||
// `quak collections` uses it. `refreshCollection()` brings one collection
|
||||
// from that list up to date in the store (issue #217). It fetches that
|
||||
// collection's files alone, as a refresh would. `quak files COLLECTION` uses
|
||||
// it. It never runs at the same time as a refresh. A caller that asks for each
|
||||
// refresh it needs can also open the library with `refreshOnOpen: false`, so
|
||||
// that `open()` starts none.
|
||||
//
|
||||
// A refresh reports through `onProgress` each collection whose files it has
|
||||
// fetched, so a large account, whose first refresh fetches the files of every
|
||||
@@ -35,10 +36,12 @@
|
||||
// A refresh stages all of its network work first and only mutates the store
|
||||
// once every fetch has succeeded. A refresh that fails partway therefore never
|
||||
// becomes visible to reads: the last good snapshot stays in place, and the
|
||||
// failure surfaces through `onProgress` and `status()` instead. A refresh that
|
||||
// mutates RAM but then fails to persist keeps `status().lastError` set and the
|
||||
// store marked unsaved until a later save actually lands, so a stuck disk is
|
||||
// never masked by a subsequent empty refresh.
|
||||
// failure surfaces through `onProgress` and `status()` instead. A save can
|
||||
// still fail after the store has changed in RAM. The store then stays marked
|
||||
// unsaved until a later save lands. Until then, every `commit()` retries the
|
||||
// save. A refresh whose save fails sets `status().lastError`, which stays set
|
||||
// until a later refresh succeeds. `refreshCollection()` never sets or clears
|
||||
// it. So a stuck disk is never hidden by a later refresh that changes nothing.
|
||||
|
||||
import { rm } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
@@ -163,21 +166,23 @@ export interface LibraryClient {
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}>;
|
||||
// The byte source for the on-disk content cache. Optional so a mock client
|
||||
// that only serves metadata still satisfies the interface; when absent (and
|
||||
// no explicit `contentSource` is passed to `open`) the content cache is
|
||||
// disabled and `Photo.original`/`thumbnail` and `thumbnails.ensure` throw.
|
||||
// The byte source for the on-disk content cache. Optional, so a mock
|
||||
// client that only serves metadata still satisfies the interface. Without
|
||||
// it, and with no `contentSource` passed to `open`, the content cache is
|
||||
// disabled. `Photo.original`/`thumbnail` and `thumbnails.ensure` then
|
||||
// throw.
|
||||
contentSource?(): ContentSource;
|
||||
}
|
||||
|
||||
// A progress event for one unit of background work. A metadata "refresh" or an
|
||||
// ML "fetchMLData" pass each fire "started" before their network work and then
|
||||
// exactly one of "done" or "failed"; "failed" carries the error message and
|
||||
// an ML "done" reports how many payloads it stored and, in `mlDataErrors`,
|
||||
// each file whose ML data did not decrypt, gunzip or parse, with the reason.
|
||||
// In between, a refresh fires "progress" after each collection whose files it
|
||||
// fetched: `fetched` is how many collections it has fetched the files of so
|
||||
// far, `total` how many it will. The precache fills
|
||||
// A progress event for one unit of background work. A metadata "refresh" and
|
||||
// an ML "fetchMLData" pass each fire "started" before their network work. Each
|
||||
// then fires exactly one of "done" or "failed". A "failed" event carries the
|
||||
// error message. An ML "done" reports in `fetched` how many payloads it
|
||||
// stored. Its `mlDataErrors` lists each file whose ML data did not decrypt,
|
||||
// gunzip or parse, with the reason. In between, a refresh fires "progress"
|
||||
// after each collection whose files it fetched. Its `fetched` is how many
|
||||
// collections it has fetched the files of so far, and `total` how many it
|
||||
// will. The precache fills
|
||||
// ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
|
||||
// each sweep that has work, "done" reporting the count newly cached.
|
||||
export interface RefreshEvent {
|
||||
@@ -205,9 +210,10 @@ export interface LibraryOptions {
|
||||
// directory at open. The content cache treats an original already stored
|
||||
// at its save path as present.
|
||||
downloadDirectory?: string;
|
||||
// How long after one refresh ends the background timer starts the next.
|
||||
// `Infinity` starts none: the library then refreshes only as it opens and
|
||||
// when `fresh()` or `backup()` asks.
|
||||
// How long after its last turn ends the background timer takes the next.
|
||||
// A turn starts a refresh unless a refresh or `refreshCollection()` is
|
||||
// running. With `Infinity` the timer takes no turn: the library then
|
||||
// refreshes only as it opens and when `fresh()` or `backup()` asks.
|
||||
refreshIntervalSeconds?: number;
|
||||
// Whether `open()` starts a refresh. Defaults to true. With false, `open()`
|
||||
// returns at once onto what the cache holds, empty or not, and a refresh
|
||||
@@ -246,8 +252,9 @@ export interface LibraryStatus {
|
||||
// Wall-clock ms of the last refresh that succeeded, or undefined if none
|
||||
// has yet.
|
||||
lastRefreshAt?: number;
|
||||
// The message from the most recent refresh, set only while that refresh
|
||||
// failed; cleared by the next success.
|
||||
// The error of the most recent refresh, set only when it failed. The next
|
||||
// refresh that succeeds clears it. `refreshCollection()` never sets or
|
||||
// clears it.
|
||||
lastError?: string;
|
||||
// Wall-clock ms of the last ML fetch pass that succeeded, or undefined if
|
||||
// none has yet (or ML fetching is disabled).
|
||||
@@ -315,8 +322,9 @@ export class Library {
|
||||
// The background loop skips while it is set. A fresh read (issue #75)
|
||||
// coalesces onto a running refresh. It waits for a running
|
||||
// `refreshCollection()` to end, and then joins or starts a refresh. The
|
||||
// promise carries the real outcome (it rejects on failure); the background
|
||||
// loop ignores that, a fresh read propagates it.
|
||||
// promise carries the real outcome: it rejects on failure. The background
|
||||
// loop ignores a failure. A fresh read passes on the failure of the
|
||||
// refresh it joins or starts, and `refreshCollection()` passes on its own.
|
||||
private cycle?: Promise<void>;
|
||||
// Whether `cycle` is a refresh cycle rather than a `refreshCollection()`.
|
||||
private cycleIsRefresh = false;
|
||||
@@ -332,19 +340,20 @@ export class Library {
|
||||
private lastMLError?: string;
|
||||
// For each file whose ML data the last ML fetch pass that succeeded got
|
||||
// but could not decrypt, gunzip or parse, why not, by fileID. A pass
|
||||
// gathers its own and puts them here when it ends without failing, so
|
||||
// `status()` and `backup()` never see the list of a pass that is still
|
||||
// running or that failed. Kept in RAM only, never in the cache.
|
||||
// gathers its own and puts them here only when it ends without failing.
|
||||
// So `status()` and `backup()` never see the list of a pass still running
|
||||
// or one that failed. Kept in RAM only, never in the cache.
|
||||
private mlDataErrors = new Map<number, unknown>();
|
||||
// The plain-record projection as of the last refresh, and the GUI change
|
||||
// subscribers. A refresh that alters the projection notifies each with the
|
||||
// delta; `lastRecords` is kept current every refresh so a subscriber that
|
||||
// joins later diffs against the state its own `snapshot()` already returned.
|
||||
// The plain-record projection as of the last store change, and the GUI
|
||||
// change subscribers. A refresh or `refreshCollection()` that alters the
|
||||
// projection notifies each subscriber with the delta. `lastRecords`
|
||||
// advances on each `commit()` that changes the store. A subscriber that
|
||||
// joins later then diffs against the state its own `snapshot()` returned.
|
||||
private readonly subscribers = new Set<(change: LibraryChange) => void>();
|
||||
private lastRecords: DerivedRecords;
|
||||
// RAM holds changes disk has not yet accepted (an earlier save failed).
|
||||
// Cleared only when a save actually succeeds; keeps the store trying to
|
||||
// persist and the failure visible in `status()` until then.
|
||||
// Cleared only when a save succeeds. Until then, every `commit()` retries
|
||||
// the save.
|
||||
private unsaved = false;
|
||||
|
||||
private constructor(args: {
|
||||
@@ -374,7 +383,7 @@ export class Library {
|
||||
this.lastRecords = this.deriveNow();
|
||||
|
||||
// The read namespaces derive fresh from the store on each call, so they
|
||||
// always reflect the latest refresh.
|
||||
// always reflect what it holds now.
|
||||
const derive = (): DerivedRecords => this.deriveNow();
|
||||
const root = this.downloadDirectory;
|
||||
const saves: SavePathLookup = {
|
||||
@@ -426,8 +435,8 @@ export class Library {
|
||||
let store = await MetadataStore.load(metadataPath);
|
||||
// A cache directory given explicitly can hold another account's cache.
|
||||
// Its records and cursor are not this account's, so delete it and the
|
||||
// ML data beside it and start empty. A user ID of 0 means the cache
|
||||
// was never refreshed and so holds nothing to discard.
|
||||
// ML data beside it and start empty. A user ID of 0 means nothing was
|
||||
// ever stored there, so there is nothing to discard.
|
||||
if (store.userID !== 0 && store.userID !== userID) {
|
||||
await rm(metadataPath, { force: true });
|
||||
await rm(join(cacheDirectory, "mldata"), {
|
||||
@@ -450,10 +459,11 @@ export class Library {
|
||||
? await MLDataStore.open(join(cacheDirectory, "mldata"))
|
||||
: undefined;
|
||||
|
||||
// Build the content cache from an explicit source or the client's own,
|
||||
// and take its record of what is already cached (and reap orphan temp
|
||||
// files) before the first projection, so cached paths are present from
|
||||
// the start and the first refresh raises no spurious path-change diff.
|
||||
// Build the content cache from an explicit source or the client's own.
|
||||
// Take its record of what is already cached, and reap orphan temp
|
||||
// files, before the first projection. Cached paths are then present
|
||||
// from the start, and the first `commit()` raises no spurious
|
||||
// path-change diff.
|
||||
const source = opts.contentSource ?? opts.client.contentSource?.();
|
||||
let cache: ContentCache | undefined;
|
||||
let precache: Precache | undefined;
|
||||
@@ -507,10 +517,10 @@ export class Library {
|
||||
precache,
|
||||
});
|
||||
|
||||
// Start filling from whatever the loaded store already holds; each
|
||||
// refresh below re-kicks with the new files (and retries any that
|
||||
// failed). An empty store starts empty here and fills after its first
|
||||
// refresh.
|
||||
// Start filling from whatever the loaded store already holds. Each
|
||||
// later `commit()` re-kicks the fills with its new files, and retries
|
||||
// any that failed. An empty store starts empty here, and fills once a
|
||||
// `commit()` stores files.
|
||||
precache?.start();
|
||||
|
||||
if (opts.refreshOnOpen === false) {
|
||||
@@ -564,9 +574,9 @@ export class Library {
|
||||
return snapshotFrom(this.deriveNow(), Date.now());
|
||||
}
|
||||
|
||||
// Deliver a `LibraryChange` whenever a refresh alters the projection. A
|
||||
// refresh that changes nothing delivers nothing. The returned handle's
|
||||
// `unsubscribe` stops delivery.
|
||||
// Deliver a `LibraryChange` whenever a refresh or `refreshCollection()`
|
||||
// alters the projection. One that changes nothing delivers nothing. The
|
||||
// returned handle's `unsubscribe` stops delivery.
|
||||
subscribe(args: { onChange: (change: LibraryChange) => void }): {
|
||||
unsubscribe: () => void;
|
||||
} {
|
||||
@@ -609,12 +619,13 @@ export class Library {
|
||||
};
|
||||
}
|
||||
|
||||
// Fresh reads (issue #75, owner amendment to design #36). Force a refresh,
|
||||
// wait for it to complete and persist, then hand back the same
|
||||
// `albums`/`photos`/`timeline` namespaces — now guaranteed to reflect a
|
||||
// completed server round-trip. Concurrent calls coalesce onto one refresh;
|
||||
// a refresh that fails rejects here, where the default namespaces would
|
||||
// instead stay silent and serve the last good copy.
|
||||
// Fresh reads (issue #75, owner amendment to design #36). Wait for a
|
||||
// refresh to complete and persist, then hand back the same
|
||||
// `albums`/`photos`/`timeline` namespaces. They then reflect a completed
|
||||
// server round-trip. A running `refreshCollection()` is waited for first.
|
||||
// Concurrent calls coalesce onto one refresh. A refresh that fails rejects
|
||||
// here, where the default namespaces would stay silent and serve the last
|
||||
// good copy.
|
||||
async fresh(): Promise<FreshReads> {
|
||||
await this.refreshNow();
|
||||
return {
|
||||
@@ -627,10 +638,10 @@ export class Library {
|
||||
// The fresh read of the collection list alone (issue #210): every live
|
||||
// collection that decrypts, in the server's order, from one collections
|
||||
// request. Unlike a refresh, it fetches no collection's files, which on a
|
||||
// large account take long. Nothing is stored: a refresh fetches a
|
||||
// large account take long. It stores nothing. A refresh fetches a
|
||||
// collection's files only when the collection is new or has changed since
|
||||
// the stored copy, so a collection stored here without its files would
|
||||
// never have them fetched.
|
||||
// the stored copy. A collection stored here without its files would never
|
||||
// have them fetched.
|
||||
async freshCollections(): Promise<Collection[]> {
|
||||
if (this.closed) throw new Error("the library is closed");
|
||||
const { collections } = await this.client.collectionsSince({
|
||||
@@ -644,8 +655,8 @@ export class Library {
|
||||
// refresh does for a changed one. It fetches no other collection's files.
|
||||
// It runs in the same slot as a refresh, so the two never run at the same
|
||||
// time. It first waits for a running refresh or `refreshCollection()` to
|
||||
// end. A fresh read or `backup()` that starts meanwhile waits for it, and
|
||||
// the background loop skips its turn.
|
||||
// end. While its own work runs, a fresh read or `backup()` waits for it,
|
||||
// and the background loop skips its turn.
|
||||
async refreshCollection(collection: Collection): Promise<void> {
|
||||
while (this.cycle) await this.cycle.catch(() => {});
|
||||
if (this.closed) throw new Error("the library is closed");
|
||||
@@ -656,17 +667,18 @@ export class Library {
|
||||
|
||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||
// library's, each original at its save path, with a durable failure
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||
// caller passes the lock it took as `opts.lock`, failing at once while
|
||||
// another backup of it runs, and before it starts another file once the
|
||||
// lock is lost (see `lockBackupDirectory`). Waits for a
|
||||
// completed refresh, as `fresh()` does, joining one already running, and
|
||||
// rejects before touching any file but the lock when it fails. Then puts
|
||||
// pending originals at their save paths as `Photo.download()` does (and
|
||||
// optional thumbnails) through the content cache and pools, waits for an
|
||||
// ML data fetch, and rebuilds the derived symlink/JSON views from the
|
||||
// model, each file's JSON with its ML data, beside an `account.json` with
|
||||
// the account's email and user ID.
|
||||
// ledger (issue #51). It first takes the lock in that directory, unless
|
||||
// the caller passes the lock it took as `opts.lock`. Taking it fails at
|
||||
// once while another backup of that directory runs. Once the lock is lost,
|
||||
// the backup fails before it starts another file (see
|
||||
// `lockBackupDirectory`). It waits for a completed refresh as `fresh()`
|
||||
// does, joining one already running. When that refresh fails, it rejects
|
||||
// before touching any file but the lock. It then puts pending originals
|
||||
// at their save paths as `Photo.download()` does, through the content
|
||||
// cache and pools. With `includeThumbnails` it also fetches thumbnails. It
|
||||
// waits for an ML data fetch. It rebuilds the derived symlink/JSON views
|
||||
// from the model, each file's JSON with its ML data. An `account.json`
|
||||
// beside them holds the account's email and user ID.
|
||||
// Throws before any network work when no content cache backs the
|
||||
// originals it must fetch.
|
||||
backup(opts?: BackupOptions): Promise<BackupResult> {
|
||||
@@ -710,7 +722,8 @@ export class Library {
|
||||
// - the ML fetch pass;
|
||||
// - the precache fetches already running.
|
||||
// A caller can then remove the cache directory. A refresh failure is
|
||||
// reported through `status()`, not thrown here.
|
||||
// reported through `status()`, and a `refreshCollection()` failure to its
|
||||
// own caller. Neither is thrown here.
|
||||
async close(): Promise<void> {
|
||||
this.closed = true;
|
||||
const precacheClosed = this.precache?.close();
|
||||
@@ -736,17 +749,18 @@ export class Library {
|
||||
// The background loop's refresh: run a cycle unless a refresh or
|
||||
// `refreshCollection()` is in flight (or the library is closed), and never
|
||||
// let a failure escape. The background path reports errors through
|
||||
// `status()`/`onProgress`; it does not throw. Resolves once the cycle it
|
||||
// started (or skipped past) settles.
|
||||
// `status()`/`onProgress`; it does not throw. It resolves once the cycle
|
||||
// it started settles, or at once when it starts none.
|
||||
private runRefresh(): Promise<void> {
|
||||
if (this.closed || this.cycle) return Promise.resolve();
|
||||
return this.startCycle(true, () => this.refreshCycle()).catch(() => {});
|
||||
}
|
||||
|
||||
// A fresh read's refresh (issue #75): force a cycle and await it, rejecting
|
||||
// if it fails. Concurrent fresh reads coalesce onto the one in-flight cycle
|
||||
// — the background loop's included — so they never fan out into redundant
|
||||
// server round-trips. A running `refreshCollection()` is waited for first.
|
||||
// A fresh read's refresh (issue #75): join the running refresh or start
|
||||
// one, and await it, rejecting if it fails. Concurrent fresh reads
|
||||
// coalesce onto the one in-flight cycle — the background loop's included —
|
||||
// so they never fan out into redundant server round-trips. A running
|
||||
// `refreshCollection()` is waited for first.
|
||||
private async refreshNow(): Promise<void> {
|
||||
while (this.cycle && !this.cycleIsRefresh) {
|
||||
await this.cycle.catch(() => {});
|
||||
@@ -789,9 +803,9 @@ export class Library {
|
||||
this.lastError = undefined;
|
||||
this.emit({ operation: "refresh", status: "done" });
|
||||
// Backfill ML data for the files this refresh knows about. It runs
|
||||
// outside the refresh's success/failure so a fetch or disk problem
|
||||
// there never marks the metadata refresh failed, and it is not
|
||||
// awaited so it never stalls the refresh interval. Its failure is
|
||||
// outside the refresh's success or failure, so a fetch or disk
|
||||
// problem there never marks the metadata refresh failed. It is not
|
||||
// awaited, so it never stalls the refresh interval. Its failure is
|
||||
// reported through `status()` and `onProgress`.
|
||||
void this.fetchMLDataNow().catch(() => {});
|
||||
} catch (err) {
|
||||
@@ -849,8 +863,8 @@ export class Library {
|
||||
// No copy is stored, yet the stored cursor has reached this
|
||||
// `updationTime`. A refresh has then seen this version or a later one,
|
||||
// and dropped the collection: it was deleted, or did not decrypt.
|
||||
// Storing it would bring it back for good, since no later refresh sees
|
||||
// it again.
|
||||
// Storing it would bring it back, and no later refresh would see it
|
||||
// again until the server's copy changes.
|
||||
if (
|
||||
!known &&
|
||||
collection.updationTime <= this.store.collectionsSinceTime
|
||||
@@ -875,15 +889,16 @@ export class Library {
|
||||
}
|
||||
|
||||
// Commit a collections page and the file pages fetched for its collections
|
||||
// to the store, then save. A collection or file that does not decrypt is
|
||||
// left out, as if deleted. The reason is kept in the store. No later
|
||||
// refresh sees it again until the server's copy changes.
|
||||
// to the store. A collection or file that does not decrypt is left out, as
|
||||
// if deleted. The reason is kept in the store. No later refresh sees it
|
||||
// again until the server's copy changes. Last, save the store when it
|
||||
// holds changes not yet saved.
|
||||
private async commit(
|
||||
page: CollectionsPage,
|
||||
filePages: { collectionID: number; page: FilesPage }[],
|
||||
): Promise<void> {
|
||||
// Network work done; commit to the store and persist only if something
|
||||
// actually changed.
|
||||
// The network work is done, so change the store now. `changed` says
|
||||
// whether this commit changed anything.
|
||||
let changed = false;
|
||||
|
||||
if (this.store.userID !== this.userID) {
|
||||
@@ -946,8 +961,8 @@ export class Library {
|
||||
// Reproject and notify subscribers of the delta. This tracks RAM (what
|
||||
// reads see), so it fires whether or not the save below succeeds. A
|
||||
// failed save rejects the caller, and a refresh also records it in
|
||||
// `status().lastError`. `lastRecords` advances every changed refresh so
|
||||
// the next diff is against current state.
|
||||
// `status().lastError`. `lastRecords` advances on every commit that
|
||||
// changes something, so the next diff is against current state.
|
||||
if (changed) {
|
||||
const next = this.deriveNow();
|
||||
if (this.subscribers.size > 0) {
|
||||
@@ -958,8 +973,9 @@ export class Library {
|
||||
// Recompute the fill orders and pinned set against the new library.
|
||||
this.precache?.update(next);
|
||||
}
|
||||
// Re-kick the fills every cycle: a finished sweep starts afresh to pick
|
||||
// up new files and retry any that failed, and a running one is left be.
|
||||
// Re-kick the fills on every commit. A finished sweep starts afresh to
|
||||
// pick up new files and retry any that failed. A running one is left
|
||||
// be.
|
||||
this.precache?.start();
|
||||
|
||||
// Persist whenever RAM holds changes disk has not accepted — including
|
||||
@@ -982,13 +998,14 @@ export class Library {
|
||||
return this.mlFetch;
|
||||
}
|
||||
|
||||
// One ML fetch pass: fetch, decrypt and store the ML data for every file
|
||||
// the store knows about that is not cached (or whose `updationTime` has
|
||||
// advanced), through the metadata pool, and update the CLIP index. For a
|
||||
// file whose ML data does not decrypt, gunzip or parse, it gathers the
|
||||
// reason instead, and puts the reasons it gathered in `mlDataErrors` when
|
||||
// it ends without failing. A failure leaves `mlDataErrors` as it was and
|
||||
// is reported through `status()` and `onProgress`, then thrown.
|
||||
// One ML fetch pass. It fetches, decrypts and stores the ML data of every
|
||||
// file the store knows about that is not cached, or whose `updationTime`
|
||||
// has advanced. It fetches through the metadata pool, and updates the CLIP
|
||||
// index. For a file whose ML data does not decrypt, gunzip or parse, it
|
||||
// gathers the reason instead. It puts the reasons it gathered in
|
||||
// `mlDataErrors` when it ends without failing. A failure leaves
|
||||
// `mlDataErrors` as it was and is reported through `status()` and
|
||||
// `onProgress`, then thrown.
|
||||
private async runMLFetch(): Promise<void> {
|
||||
const mldata = this.mlStore;
|
||||
// Bind so the call keeps the client as its receiver when invoked
|
||||
@@ -1056,9 +1073,9 @@ export class Library {
|
||||
}));
|
||||
}
|
||||
|
||||
// The distinct files the store holds, one entry per fileID (a file in
|
||||
// several collections shares its ML data), each carrying the key and the
|
||||
// newest `updationTime` seen across its memberships.
|
||||
// The distinct files the store holds, one entry per fileID, since a file
|
||||
// in several collections shares its ML data. Each entry carries the key
|
||||
// and the newest `updationTime` seen across the file's memberships.
|
||||
private uniqueFiles(): {
|
||||
id: number;
|
||||
key: Uint8Array;
|
||||
|
||||
Reference in New Issue
Block a user