Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
98f674c3a3 |
No files matched your search
@@ -1102,8 +1102,8 @@ It also awaits it on a cache that holds only what `lib.refreshCollection()`
|
|||||||
stored. If that refresh fails, it opens onto what the cache holds and records
|
stored. If that refresh fails, it opens onto what the cache holds and records
|
||||||
the error in `lib.status()`. On any other cache it returns immediately and
|
the error in `lib.status()`. On any other cache it returns immediately and
|
||||||
refreshes in the background, so an unreachable server does not block opening.
|
refreshes in the background, so an unreachable server does not block opening.
|
||||||
With `refreshOnOpen: false` it starts no refresh and returns at once onto what
|
With `refreshOnOpen: false` it starts no refresh, and returns at once onto what
|
||||||
the cache holds, empty or not; a refresh then starts only when `fresh()` or
|
the cache holds, empty or not. A refresh then starts only when `fresh()` or
|
||||||
`lib.backup()` asks, or the background timer fires. The CLI's read commands open
|
`lib.backup()` asks, or the background timer fires. The CLI's read commands open
|
||||||
it so (see "CLI surface").
|
it so (see "CLI surface").
|
||||||
|
|
||||||
@@ -1132,8 +1132,8 @@ when the library opens; `lib.downloadDirectory` holds the result.
|
|||||||
`onProgress` receives a `RefreshEvent` with status `"started"` as a refresh, an
|
`onProgress` receives a `RefreshEvent` with status `"started"` as a refresh, an
|
||||||
ML data fetch or a precache sweep starts, and `"done"` or `"failed"` as it ends.
|
ML data fetch or a precache sweep starts, and `"done"` or `"failed"` as it ends.
|
||||||
In between, a refresh sends `"progress"` after each collection whose files it
|
In between, a refresh sends `"progress"` after each collection whose files it
|
||||||
fetched, with `fetched`, how many collections it has fetched the files of so
|
fetched. That event's `fetched` is how many collections it has fetched the files
|
||||||
far, and `total`, how many it will. A refresh fetches the files of each
|
of so far, and its `total` how many it will. A refresh fetches the files of each
|
||||||
collection that is new or changed since the cache's copy, so on an empty cache
|
collection that is new or changed since the cache's copy, so on an empty cache
|
||||||
it fetches those of every collection.
|
it fetches those of every collection.
|
||||||
|
|
||||||
@@ -1166,27 +1166,31 @@ cache.
|
|||||||
|
|
||||||
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
||||||
synchronously from the copy held in RAM and never touch the network. A refresh
|
synchronously from the copy held in RAM and never touch the network. A refresh
|
||||||
changes that copy only once all its server requests have succeeded, and the
|
changes that copy only once all its server requests have succeeded.
|
||||||
background timer starts the next refresh `refreshIntervalSeconds` after the
|
`lib.refreshCollection()` changes it the same way for one collection. So a
|
||||||
previous one ends. So a default read is immediate, but only as current as the
|
default read is immediate, but only as current as the last refresh or
|
||||||
last refresh whose requests all succeeded; right after opening an existing
|
`lib.refreshCollection()` whose requests all succeeded. Right after opening an
|
||||||
cache, it is the copy on disk.
|
existing cache, it is the copy on disk.
|
||||||
|
|
||||||
|
The background timer takes a turn `refreshIntervalSeconds` after its last turn
|
||||||
|
ends. Each turn starts a refresh, unless a refresh or `lib.refreshCollection()`
|
||||||
|
is already running.
|
||||||
|
|
||||||
`await lib.fresh()` waits for a refresh to complete and persist, and returns the
|
`await lib.fresh()` waits for a refresh to complete and persist, and returns the
|
||||||
same `{ albums, photos, timeline }` namespaces, which then reflect a completed
|
same `{ albums, photos, timeline }` namespaces, which then reflect a completed
|
||||||
server round-trip. When a refresh is already running, background or not,
|
server round-trip. It first waits for a running `lib.refreshCollection()` to
|
||||||
`fresh()` waits for that one, so its answer can come from requests made before
|
end. When a refresh is already running, background or not, `fresh()` waits for
|
||||||
the call; only when none is running does it start one. A refresh that fails
|
that one. Its answer can then come from requests made before the call. Only when
|
||||||
rejects the caller (default reads stay silent and keep serving the last good
|
none is running does it start one. A refresh that fails rejects the caller
|
||||||
copy). The CLI's read commands use fresh reads (issue
|
(default reads stay silent and keep serving the last good copy). The CLI's read
|
||||||
https://git.eeqj.de/sneak/quak/issues/75).
|
commands use fresh reads (issue https://git.eeqj.de/sneak/quak/issues/75).
|
||||||
|
|
||||||
`await lib.freshCollections()` is the fresh read of the collection list alone:
|
`await lib.freshCollections()` is the fresh read of the collection list alone:
|
||||||
one collections request, answered with every live collection that decrypts, as
|
one collections request, answered with every live collection that decrypts, as
|
||||||
the `Collection` objects `listCollections()` returns, in the server's order. It
|
the `Collection` objects `listCollections()` returns, in the server's order. It
|
||||||
fetches no collection's files and stores nothing, not even the list: a refresh
|
fetches no collection's files and stores nothing, not even the list. A refresh
|
||||||
fetches a collection's files only when the collection is new or has changed
|
fetches a collection's files only when the collection is new or has changed
|
||||||
since the stored copy, so a collection stored without its files would never get
|
since the stored copy. A collection stored without its files would never get
|
||||||
them. `quak collections` uses it (issue
|
them. `quak collections` uses it (issue
|
||||||
https://git.eeqj.de/sneak/quak/issues/210).
|
https://git.eeqj.de/sneak/quak/issues/210).
|
||||||
|
|
||||||
@@ -1200,19 +1204,20 @@ not decrypt. Otherwise it fetches the collection's files changed since the
|
|||||||
cache's copy, or all of them when there is none, then stores and saves them. The
|
cache's copy, or all of them when there is none, then stores and saves them. The
|
||||||
refresh cursor stays where it was, so the next refresh still fetches the files
|
refresh cursor stays where it was, so the next refresh still fetches the files
|
||||||
of every other collection that changed. It never runs at the same time as a
|
of every other collection that changed. It never runs at the same time as a
|
||||||
refresh. It waits for a running refresh to end, and a `fresh()` or
|
refresh. It first waits for a running refresh to end. A `fresh()` or
|
||||||
`lib.backup()` called meanwhile waits for it. `quak files` with a collection ID
|
`lib.backup()` called after that waits for it. `quak files` with a collection ID
|
||||||
uses it.
|
uses it.
|
||||||
|
|
||||||
An album or file in the server's answer that quak cannot decrypt, or whose
|
An album or file in the server's answer that quak cannot decrypt, or whose
|
||||||
metadata is not JSON, does not fail the refresh. The refresh leaves it out, an
|
metadata is not JSON, does not fail the refresh. The refresh leaves it out, an
|
||||||
album with all its files, and drops any older copy the cache held, so reads
|
album with all its files, and drops any older copy the cache held, so reads
|
||||||
answer from what did decrypt. The library keeps a list of what it left out, with
|
answer from what did decrypt. The library keeps a list of what it left out, with
|
||||||
the reason, in `metadata.json`: the next refresh asks the server only for what
|
the reason, in `metadata.json`. The next refresh asks the server only for what
|
||||||
changed, so it would not see the record again. A record stays on that list until
|
changed, so it would not see the record again. A record stays on that list until
|
||||||
a later refresh gets a copy that decrypts or learns that the server deleted it.
|
a later refresh or `lib.refreshCollection()` stores a copy that decrypts, or
|
||||||
`lib.backup()` reports each such album, and each such file in its scope that has
|
learns that the server deleted it. `lib.backup()` reports each such album, and
|
||||||
no copy that decrypted (see "Backup layout").
|
each such file in its scope that has no copy that decrypted (see "Backup
|
||||||
|
layout").
|
||||||
|
|
||||||
### Read surface
|
### Read surface
|
||||||
|
|
||||||
@@ -1306,10 +1311,11 @@ The GUI-facing records hold no key material and no binary, so they survive
|
|||||||
- `LibrarySnapshot`: `{ albums, photos, takenAt }`.
|
- `LibrarySnapshot`: `{ albums, photos, takenAt }`.
|
||||||
|
|
||||||
`lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first,
|
`lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first,
|
||||||
photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
photos newest first). `lib.subscribe({ onChange })` returns `{ unsubscribe }`.
|
||||||
(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`,
|
It delivers a `LibraryChange` (`albumsChanged`, `photosChanged`,
|
||||||
`refreshedAt`) whenever a refresh alters the projection, and returns
|
`fileIDsRemoved`, `albumIDsRemoved`, `refreshedAt`) whenever a refresh or
|
||||||
`{ unsubscribe }`; a refresh that changes nothing delivers nothing.
|
`lib.refreshCollection()` alters the projection. One that changes nothing
|
||||||
|
delivers nothing.
|
||||||
|
|
||||||
### Thumbnails, ML search, and backup
|
### Thumbnails, ML search, and backup
|
||||||
|
|
||||||
@@ -1328,26 +1334,27 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
|||||||
- `await lib.backup(opts?)` → `BackupResult`. It takes the lock in the download
|
- `await lib.backup(opts?)` → `BackupResult`. It takes the lock in the download
|
||||||
directory, and fails at once with an error whose `code` is `ELOCKED` while
|
directory, and fails at once with an error whose `code` is `ELOCKED` while
|
||||||
another backup of it runs. A caller can instead take the lock itself, before
|
another backup of it runs. A caller can instead take the lock itself, before
|
||||||
it opens its library, as `quak backup` does: `await lockBackupDirectory(dir)`
|
it opens its library, as `quak backup` does. `await lockBackupDirectory(dir)`
|
||||||
takes it, failing the same way, and returns a `BackupLock`, whose `release()`
|
takes it, failing the same way, and returns a `BackupLock`. Its `release()`
|
||||||
removes the lock unless it was lost and whose `lost` holds why it was lost.
|
removes the lock unless it was lost, and its `lost` holds why it was lost. The
|
||||||
The caller passes it to the backup as `lock`, and releases it once the backup
|
caller passes it to the backup as `lock`, and releases it once the backup
|
||||||
ends. It waits for a refresh as `fresh()` does, puts every in-scope original
|
ends. The backup waits for a refresh as `fresh()` does. It then puts every
|
||||||
not already at its save path there as `photo.download()` does (and, with
|
in-scope original not already at its save path there, as `photo.download()`
|
||||||
`includeThumbnails`, fetches thumbnails) through the content cache, waits for
|
does, through the content cache. With `includeThumbnails` it also fetches
|
||||||
an ML data fetch, and rebuilds the on-disk backup tree with a durable failure
|
thumbnails. It waits for an ML data fetch, and rebuilds the on-disk backup
|
||||||
ledger. Each file's JSON holds its ML data and keywords. An image's JSON also
|
tree with a durable failure ledger. Each file's JSON holds its ML data and
|
||||||
holds the EXIF and XMP of its original, and its dimensions if it is a JPEG. A
|
keywords. An image's JSON also holds the EXIF and XMP of its original, and its
|
||||||
live photo's JSON holds these for its image. A video's JSON holds no EXIF, XMP
|
dimensions if it is a JPEG. A live photo's JSON holds these for its image. A
|
||||||
or dimensions. A fetched original is written straight to its save path and not
|
video's JSON holds no EXIF, XMP or dimensions. A fetched original is written
|
||||||
into the cache, which then counts it as present; one the cache already held is
|
straight to its save path and not into the cache, which then counts it as
|
||||||
copied from there. `BackupOptions`: `downloadDirectory` (falls back to the
|
present; one the cache already held is copied from there. `BackupOptions`:
|
||||||
library's), `includeOriginals` (default `true`), `includeThumbnails` (default
|
`downloadDirectory` (falls back to the library's), `includeOriginals` (default
|
||||||
`false`), `onlyAlbumNames`, `verify` (default `false`), `onProgress`, and
|
`true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
|
||||||
`lock`. See Backup layout above for the tree it writes, and for the albums
|
(default `false`), `onProgress`, and `lock`. See Backup layout above for the
|
||||||
`BackupResult` lists in `undecryptableAlbums` and `failedAlbums`. Whether the
|
tree it writes, and for the albums `BackupResult` lists in
|
||||||
backup took the lock or its caller did, once the lock is lost the backup
|
`undecryptableAlbums` and `failedAlbums`. Whether the backup took the lock or
|
||||||
rejects with `lost the lock on <dir>: <reason>` before it starts another file.
|
its caller did, once the lock is lost the backup rejects with
|
||||||
|
`lost the lock on <dir>: <reason>` before it starts another file.
|
||||||
|
|
||||||
### Request pools
|
### Request pools
|
||||||
|
|
||||||
|
|||||||
+137
-120
@@ -1,32 +1,33 @@
|
|||||||
// The library surface over the local cache.
|
// The library surface over the local cache.
|
||||||
//
|
//
|
||||||
// `Library.open()` loads the on-disk metadata store (issue #41), then starts
|
// `Library.open()` loads the on-disk metadata store (issue #41), then starts
|
||||||
// the refresh loop. When no refresh has stored a collection yet (the stored
|
// the refresh loop. When the stored cursor is 0, `open()` awaits the first
|
||||||
// cursor is 0), `open()` awaits the first refresh. The cache is then empty, or
|
// refresh. No refresh has then stored any collection: the cache is empty, or
|
||||||
// holds only what `refreshCollection()` stored, and the library must not serve
|
// holds only what `refreshCollection()` stored. The library must not serve
|
||||||
// it as the whole account. Otherwise that first refresh runs in the
|
// that as the whole account. Otherwise that first refresh runs in the
|
||||||
// background. `open()` returns as soon as the cached data is ready to serve,
|
// background. `open()` returns as soon as the cached data is ready to serve,
|
||||||
// so a slow or unreachable server does not stall opening. A background timer
|
// so a slow or unreachable server does not stall opening. A background timer
|
||||||
// then refreshes every `refreshIntervalSeconds`. Every default read is
|
// then takes a turn `refreshIntervalSeconds` after its last turn ends. Each
|
||||||
// answered from RAM — no default read touches the network. There is
|
// turn starts a refresh, unless a refresh or `refreshCollection()` is already
|
||||||
// deliberately no `sync()`, no `refresh()`, no `serverReachable` flag, and no
|
// running. Every default read is answered from RAM — no default read touches
|
||||||
// "before each read" mode (design #36).
|
// the network. There is deliberately no `sync()`, no `refresh()`, no
|
||||||
|
// `serverReachable` flag, and no "before each read" mode (design #36).
|
||||||
//
|
//
|
||||||
// `fresh()` is the one exception (issue #75, an owner amendment to #36): it
|
// `fresh()` is an exception (issue #75, an owner amendment to #36). It awaits
|
||||||
// forces a refresh, awaits it, and only then hands back the read namespaces, so
|
// a refresh, and only then hands back the read namespaces. A caller that needs
|
||||||
// a caller that needs server-current data can ask for it. Concurrent `fresh()`
|
// server-current data uses it. Concurrent `fresh()` calls coalesce onto one
|
||||||
// calls coalesce onto one in-flight refresh, and a refresh that fails rejects
|
// in-flight refresh. A refresh that fails rejects the caller. The default
|
||||||
// the caller (the default reads stay silent and serve the last good copy). The
|
// reads stay silent and serve the last good copy. The default methods and the
|
||||||
// default methods and the background loop are unchanged.
|
// background loop are unchanged.
|
||||||
//
|
//
|
||||||
// `freshCollections()` is the fresh read of the collection list alone (issue
|
// `freshCollections()` is the fresh read of the collection list alone (issue
|
||||||
// #210): one collections request, no collection's files, and nothing stored,
|
// #210). It sends one collections request and stores nothing.
|
||||||
// for a caller such as `quak collections` that needs no more.
|
// `quak collections` uses it. `refreshCollection()` brings one collection
|
||||||
// `refreshCollection()` brings one collection from that list up to date in the
|
// from that list up to date in the store (issue #217). It fetches that
|
||||||
// store (issue #217). It fetches that collection's files alone, as a refresh
|
// collection's files alone, as a refresh would. `quak files COLLECTION` uses
|
||||||
// would. `quak files COLLECTION` uses it. It never runs at the same time as a
|
// it. It never runs at the same time as a refresh. A caller that asks for each
|
||||||
// refresh. A caller that asks for each refresh it needs can also open the
|
// refresh it needs can also open the library with `refreshOnOpen: false`, so
|
||||||
// library with `refreshOnOpen: false`, so that `open()` starts none.
|
// that `open()` starts none.
|
||||||
//
|
//
|
||||||
// A refresh reports through `onProgress` each collection whose files it has
|
// A refresh reports through `onProgress` each collection whose files it has
|
||||||
// fetched, so a large account, whose first refresh fetches the files of every
|
// fetched, so a large account, whose first refresh fetches the files of every
|
||||||
@@ -35,10 +36,12 @@
|
|||||||
// A refresh stages all of its network work first and only mutates the store
|
// A refresh stages all of its network work first and only mutates the store
|
||||||
// once every fetch has succeeded. A refresh that fails partway therefore never
|
// once every fetch has succeeded. A refresh that fails partway therefore never
|
||||||
// becomes visible to reads: the last good snapshot stays in place, and the
|
// becomes visible to reads: the last good snapshot stays in place, and the
|
||||||
// failure surfaces through `onProgress` and `status()` instead. A refresh that
|
// failure surfaces through `onProgress` and `status()` instead. A save can
|
||||||
// mutates RAM but then fails to persist keeps `status().lastError` set and the
|
// still fail after the store has changed in RAM. The store then stays marked
|
||||||
// store marked unsaved until a later save actually lands, so a stuck disk is
|
// unsaved until a later save lands. Until then, every `commit()` retries the
|
||||||
// never masked by a subsequent empty refresh.
|
// save. A refresh whose save fails sets `status().lastError`, which stays set
|
||||||
|
// until a later refresh succeeds. `refreshCollection()` never sets or clears
|
||||||
|
// it. So a stuck disk is never hidden by a later refresh that changes nothing.
|
||||||
|
|
||||||
import { rm } from "node:fs/promises";
|
import { rm } from "node:fs/promises";
|
||||||
import { join, resolve } from "node:path";
|
import { join, resolve } from "node:path";
|
||||||
@@ -163,21 +166,23 @@ export interface LibraryClient {
|
|||||||
payloads: Map<number, MLData>;
|
payloads: Map<number, MLData>;
|
||||||
errors: Map<number, unknown>;
|
errors: Map<number, unknown>;
|
||||||
}>;
|
}>;
|
||||||
// The byte source for the on-disk content cache. Optional so a mock client
|
// The byte source for the on-disk content cache. Optional, so a mock
|
||||||
// that only serves metadata still satisfies the interface; when absent (and
|
// client that only serves metadata still satisfies the interface. Without
|
||||||
// no explicit `contentSource` is passed to `open`) the content cache is
|
// it, and with no `contentSource` passed to `open`, the content cache is
|
||||||
// disabled and `Photo.original`/`thumbnail` and `thumbnails.ensure` throw.
|
// disabled. `Photo.original`/`thumbnail` and `thumbnails.ensure` then
|
||||||
|
// throw.
|
||||||
contentSource?(): ContentSource;
|
contentSource?(): ContentSource;
|
||||||
}
|
}
|
||||||
|
|
||||||
// A progress event for one unit of background work. A metadata "refresh" or an
|
// A progress event for one unit of background work. A metadata "refresh" and
|
||||||
// ML "fetchMLData" pass each fire "started" before their network work and then
|
// an ML "fetchMLData" pass each fire "started" before their network work. Each
|
||||||
// exactly one of "done" or "failed"; "failed" carries the error message and
|
// then fires exactly one of "done" or "failed". A "failed" event carries the
|
||||||
// an ML "done" reports how many payloads it stored and, in `mlDataErrors`,
|
// error message. An ML "done" reports in `fetched` how many payloads it
|
||||||
// each file whose ML data did not decrypt, gunzip or parse, with the reason.
|
// stored. Its `mlDataErrors` lists each file whose ML data did not decrypt,
|
||||||
// In between, a refresh fires "progress" after each collection whose files it
|
// gunzip or parse, with the reason. In between, a refresh fires "progress"
|
||||||
// fetched: `fetched` is how many collections it has fetched the files of so
|
// after each collection whose files it fetched. Its `fetched` is how many
|
||||||
// far, `total` how many it will. The precache fills
|
// collections it has fetched the files of so far, and `total` how many it
|
||||||
|
// will. The precache fills
|
||||||
// ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
|
// ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
|
||||||
// each sweep that has work, "done" reporting the count newly cached.
|
// each sweep that has work, "done" reporting the count newly cached.
|
||||||
export interface RefreshEvent {
|
export interface RefreshEvent {
|
||||||
@@ -205,9 +210,10 @@ export interface LibraryOptions {
|
|||||||
// directory at open. The content cache treats an original already stored
|
// directory at open. The content cache treats an original already stored
|
||||||
// at its save path as present.
|
// at its save path as present.
|
||||||
downloadDirectory?: string;
|
downloadDirectory?: string;
|
||||||
// How long after one refresh ends the background timer starts the next.
|
// How long after its last turn ends the background timer takes the next.
|
||||||
// `Infinity` starts none: the library then refreshes only as it opens and
|
// A turn starts a refresh unless a refresh or `refreshCollection()` is
|
||||||
// when `fresh()` or `backup()` asks.
|
// running. With `Infinity` the timer takes no turn: the library then
|
||||||
|
// refreshes only as it opens and when `fresh()` or `backup()` asks.
|
||||||
refreshIntervalSeconds?: number;
|
refreshIntervalSeconds?: number;
|
||||||
// Whether `open()` starts a refresh. Defaults to true. With false, `open()`
|
// Whether `open()` starts a refresh. Defaults to true. With false, `open()`
|
||||||
// returns at once onto what the cache holds, empty or not, and a refresh
|
// returns at once onto what the cache holds, empty or not, and a refresh
|
||||||
@@ -246,8 +252,9 @@ export interface LibraryStatus {
|
|||||||
// Wall-clock ms of the last refresh that succeeded, or undefined if none
|
// Wall-clock ms of the last refresh that succeeded, or undefined if none
|
||||||
// has yet.
|
// has yet.
|
||||||
lastRefreshAt?: number;
|
lastRefreshAt?: number;
|
||||||
// The message from the most recent refresh, set only while that refresh
|
// The error of the most recent refresh, set only when it failed. The next
|
||||||
// failed; cleared by the next success.
|
// refresh that succeeds clears it. `refreshCollection()` never sets or
|
||||||
|
// clears it.
|
||||||
lastError?: string;
|
lastError?: string;
|
||||||
// Wall-clock ms of the last ML fetch pass that succeeded, or undefined if
|
// Wall-clock ms of the last ML fetch pass that succeeded, or undefined if
|
||||||
// none has yet (or ML fetching is disabled).
|
// none has yet (or ML fetching is disabled).
|
||||||
@@ -315,8 +322,9 @@ export class Library {
|
|||||||
// The background loop skips while it is set. A fresh read (issue #75)
|
// The background loop skips while it is set. A fresh read (issue #75)
|
||||||
// coalesces onto a running refresh. It waits for a running
|
// coalesces onto a running refresh. It waits for a running
|
||||||
// `refreshCollection()` to end, and then joins or starts a refresh. The
|
// `refreshCollection()` to end, and then joins or starts a refresh. The
|
||||||
// promise carries the real outcome (it rejects on failure); the background
|
// promise carries the real outcome: it rejects on failure. The background
|
||||||
// loop ignores that, a fresh read propagates it.
|
// loop ignores a failure. A fresh read passes on the failure of the
|
||||||
|
// refresh it joins or starts, and `refreshCollection()` passes on its own.
|
||||||
private cycle?: Promise<void>;
|
private cycle?: Promise<void>;
|
||||||
// Whether `cycle` is a refresh cycle rather than a `refreshCollection()`.
|
// Whether `cycle` is a refresh cycle rather than a `refreshCollection()`.
|
||||||
private cycleIsRefresh = false;
|
private cycleIsRefresh = false;
|
||||||
@@ -332,19 +340,20 @@ export class Library {
|
|||||||
private lastMLError?: string;
|
private lastMLError?: string;
|
||||||
// For each file whose ML data the last ML fetch pass that succeeded got
|
// For each file whose ML data the last ML fetch pass that succeeded got
|
||||||
// but could not decrypt, gunzip or parse, why not, by fileID. A pass
|
// but could not decrypt, gunzip or parse, why not, by fileID. A pass
|
||||||
// gathers its own and puts them here when it ends without failing, so
|
// gathers its own and puts them here only when it ends without failing.
|
||||||
// `status()` and `backup()` never see the list of a pass that is still
|
// So `status()` and `backup()` never see the list of a pass still running
|
||||||
// running or that failed. Kept in RAM only, never in the cache.
|
// or one that failed. Kept in RAM only, never in the cache.
|
||||||
private mlDataErrors = new Map<number, unknown>();
|
private mlDataErrors = new Map<number, unknown>();
|
||||||
// The plain-record projection as of the last refresh, and the GUI change
|
// The plain-record projection as of the last store change, and the GUI
|
||||||
// subscribers. A refresh that alters the projection notifies each with the
|
// change subscribers. A refresh or `refreshCollection()` that alters the
|
||||||
// delta; `lastRecords` is kept current every refresh so a subscriber that
|
// projection notifies each subscriber with the delta. `lastRecords`
|
||||||
// joins later diffs against the state its own `snapshot()` already returned.
|
// advances on each `commit()` that changes the store. A subscriber that
|
||||||
|
// joins later then diffs against the state its own `snapshot()` returned.
|
||||||
private readonly subscribers = new Set<(change: LibraryChange) => void>();
|
private readonly subscribers = new Set<(change: LibraryChange) => void>();
|
||||||
private lastRecords: DerivedRecords;
|
private lastRecords: DerivedRecords;
|
||||||
// RAM holds changes disk has not yet accepted (an earlier save failed).
|
// RAM holds changes disk has not yet accepted (an earlier save failed).
|
||||||
// Cleared only when a save actually succeeds; keeps the store trying to
|
// Cleared only when a save succeeds. Until then, every `commit()` retries
|
||||||
// persist and the failure visible in `status()` until then.
|
// the save.
|
||||||
private unsaved = false;
|
private unsaved = false;
|
||||||
|
|
||||||
private constructor(args: {
|
private constructor(args: {
|
||||||
@@ -374,7 +383,7 @@ export class Library {
|
|||||||
this.lastRecords = this.deriveNow();
|
this.lastRecords = this.deriveNow();
|
||||||
|
|
||||||
// The read namespaces derive fresh from the store on each call, so they
|
// The read namespaces derive fresh from the store on each call, so they
|
||||||
// always reflect the latest refresh.
|
// always reflect what it holds now.
|
||||||
const derive = (): DerivedRecords => this.deriveNow();
|
const derive = (): DerivedRecords => this.deriveNow();
|
||||||
const root = this.downloadDirectory;
|
const root = this.downloadDirectory;
|
||||||
const saves: SavePathLookup = {
|
const saves: SavePathLookup = {
|
||||||
@@ -426,8 +435,8 @@ export class Library {
|
|||||||
let store = await MetadataStore.load(metadataPath);
|
let store = await MetadataStore.load(metadataPath);
|
||||||
// A cache directory given explicitly can hold another account's cache.
|
// A cache directory given explicitly can hold another account's cache.
|
||||||
// Its records and cursor are not this account's, so delete it and the
|
// Its records and cursor are not this account's, so delete it and the
|
||||||
// ML data beside it and start empty. A user ID of 0 means the cache
|
// ML data beside it and start empty. A user ID of 0 means nothing was
|
||||||
// was never refreshed and so holds nothing to discard.
|
// ever stored there, so there is nothing to discard.
|
||||||
if (store.userID !== 0 && store.userID !== userID) {
|
if (store.userID !== 0 && store.userID !== userID) {
|
||||||
await rm(metadataPath, { force: true });
|
await rm(metadataPath, { force: true });
|
||||||
await rm(join(cacheDirectory, "mldata"), {
|
await rm(join(cacheDirectory, "mldata"), {
|
||||||
@@ -450,10 +459,11 @@ export class Library {
|
|||||||
? await MLDataStore.open(join(cacheDirectory, "mldata"))
|
? await MLDataStore.open(join(cacheDirectory, "mldata"))
|
||||||
: undefined;
|
: undefined;
|
||||||
|
|
||||||
// Build the content cache from an explicit source or the client's own,
|
// Build the content cache from an explicit source or the client's own.
|
||||||
// and take its record of what is already cached (and reap orphan temp
|
// Take its record of what is already cached, and reap orphan temp
|
||||||
// files) before the first projection, so cached paths are present from
|
// files, before the first projection. Cached paths are then present
|
||||||
// the start and the first refresh raises no spurious path-change diff.
|
// from the start, and the first `commit()` raises no spurious
|
||||||
|
// path-change diff.
|
||||||
const source = opts.contentSource ?? opts.client.contentSource?.();
|
const source = opts.contentSource ?? opts.client.contentSource?.();
|
||||||
let cache: ContentCache | undefined;
|
let cache: ContentCache | undefined;
|
||||||
let precache: Precache | undefined;
|
let precache: Precache | undefined;
|
||||||
@@ -507,10 +517,10 @@ export class Library {
|
|||||||
precache,
|
precache,
|
||||||
});
|
});
|
||||||
|
|
||||||
// Start filling from whatever the loaded store already holds; each
|
// Start filling from whatever the loaded store already holds. Each
|
||||||
// refresh below re-kicks with the new files (and retries any that
|
// later `commit()` re-kicks the fills with its new files, and retries
|
||||||
// failed). An empty store starts empty here and fills after its first
|
// any that failed. An empty store starts empty here, and fills once a
|
||||||
// refresh.
|
// `commit()` stores files.
|
||||||
precache?.start();
|
precache?.start();
|
||||||
|
|
||||||
if (opts.refreshOnOpen === false) {
|
if (opts.refreshOnOpen === false) {
|
||||||
@@ -564,9 +574,9 @@ export class Library {
|
|||||||
return snapshotFrom(this.deriveNow(), Date.now());
|
return snapshotFrom(this.deriveNow(), Date.now());
|
||||||
}
|
}
|
||||||
|
|
||||||
// Deliver a `LibraryChange` whenever a refresh alters the projection. A
|
// Deliver a `LibraryChange` whenever a refresh or `refreshCollection()`
|
||||||
// refresh that changes nothing delivers nothing. The returned handle's
|
// alters the projection. One that changes nothing delivers nothing. The
|
||||||
// `unsubscribe` stops delivery.
|
// returned handle's `unsubscribe` stops delivery.
|
||||||
subscribe(args: { onChange: (change: LibraryChange) => void }): {
|
subscribe(args: { onChange: (change: LibraryChange) => void }): {
|
||||||
unsubscribe: () => void;
|
unsubscribe: () => void;
|
||||||
} {
|
} {
|
||||||
@@ -609,12 +619,13 @@ export class Library {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fresh reads (issue #75, owner amendment to design #36). Force a refresh,
|
// Fresh reads (issue #75, owner amendment to design #36). Wait for a
|
||||||
// wait for it to complete and persist, then hand back the same
|
// refresh to complete and persist, then hand back the same
|
||||||
// `albums`/`photos`/`timeline` namespaces — now guaranteed to reflect a
|
// `albums`/`photos`/`timeline` namespaces. They then reflect a completed
|
||||||
// completed server round-trip. Concurrent calls coalesce onto one refresh;
|
// server round-trip. A running `refreshCollection()` is waited for first.
|
||||||
// a refresh that fails rejects here, where the default namespaces would
|
// Concurrent calls coalesce onto one refresh. A refresh that fails rejects
|
||||||
// instead stay silent and serve the last good copy.
|
// here, where the default namespaces would stay silent and serve the last
|
||||||
|
// good copy.
|
||||||
async fresh(): Promise<FreshReads> {
|
async fresh(): Promise<FreshReads> {
|
||||||
await this.refreshNow();
|
await this.refreshNow();
|
||||||
return {
|
return {
|
||||||
@@ -627,10 +638,10 @@ export class Library {
|
|||||||
// The fresh read of the collection list alone (issue #210): every live
|
// The fresh read of the collection list alone (issue #210): every live
|
||||||
// collection that decrypts, in the server's order, from one collections
|
// collection that decrypts, in the server's order, from one collections
|
||||||
// request. Unlike a refresh, it fetches no collection's files, which on a
|
// request. Unlike a refresh, it fetches no collection's files, which on a
|
||||||
// large account take long. Nothing is stored: a refresh fetches a
|
// large account take long. It stores nothing. A refresh fetches a
|
||||||
// collection's files only when the collection is new or has changed since
|
// collection's files only when the collection is new or has changed since
|
||||||
// the stored copy, so a collection stored here without its files would
|
// the stored copy. A collection stored here without its files would never
|
||||||
// never have them fetched.
|
// have them fetched.
|
||||||
async freshCollections(): Promise<Collection[]> {
|
async freshCollections(): Promise<Collection[]> {
|
||||||
if (this.closed) throw new Error("the library is closed");
|
if (this.closed) throw new Error("the library is closed");
|
||||||
const { collections } = await this.client.collectionsSince({
|
const { collections } = await this.client.collectionsSince({
|
||||||
@@ -644,8 +655,8 @@ export class Library {
|
|||||||
// refresh does for a changed one. It fetches no other collection's files.
|
// refresh does for a changed one. It fetches no other collection's files.
|
||||||
// It runs in the same slot as a refresh, so the two never run at the same
|
// It runs in the same slot as a refresh, so the two never run at the same
|
||||||
// time. It first waits for a running refresh or `refreshCollection()` to
|
// time. It first waits for a running refresh or `refreshCollection()` to
|
||||||
// end. A fresh read or `backup()` that starts meanwhile waits for it, and
|
// end. While its own work runs, a fresh read or `backup()` waits for it,
|
||||||
// the background loop skips its turn.
|
// and the background loop skips its turn.
|
||||||
async refreshCollection(collection: Collection): Promise<void> {
|
async refreshCollection(collection: Collection): Promise<void> {
|
||||||
while (this.cycle) await this.cycle.catch(() => {});
|
while (this.cycle) await this.cycle.catch(() => {});
|
||||||
if (this.closed) throw new Error("the library is closed");
|
if (this.closed) throw new Error("the library is closed");
|
||||||
@@ -656,17 +667,18 @@ export class Library {
|
|||||||
|
|
||||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||||
// library's, each original at its save path, with a durable failure
|
// library's, each original at its save path, with a durable failure
|
||||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
// ledger (issue #51). It first takes the lock in that directory, unless
|
||||||
// caller passes the lock it took as `opts.lock`, failing at once while
|
// the caller passes the lock it took as `opts.lock`. Taking it fails at
|
||||||
// another backup of it runs, and before it starts another file once the
|
// once while another backup of that directory runs. Once the lock is lost,
|
||||||
// lock is lost (see `lockBackupDirectory`). Waits for a
|
// the backup fails before it starts another file (see
|
||||||
// completed refresh, as `fresh()` does, joining one already running, and
|
// `lockBackupDirectory`). It waits for a completed refresh as `fresh()`
|
||||||
// rejects before touching any file but the lock when it fails. Then puts
|
// does, joining one already running. When that refresh fails, it rejects
|
||||||
// pending originals at their save paths as `Photo.download()` does (and
|
// before touching any file but the lock. It then puts pending originals
|
||||||
// optional thumbnails) through the content cache and pools, waits for an
|
// at their save paths as `Photo.download()` does, through the content
|
||||||
// ML data fetch, and rebuilds the derived symlink/JSON views from the
|
// cache and pools. With `includeThumbnails` it also fetches thumbnails. It
|
||||||
// model, each file's JSON with its ML data, beside an `account.json` with
|
// waits for an ML data fetch. It rebuilds the derived symlink/JSON views
|
||||||
// the account's email and user ID.
|
// from the model, each file's JSON with its ML data. An `account.json`
|
||||||
|
// beside them holds the account's email and user ID.
|
||||||
// Throws before any network work when no content cache backs the
|
// Throws before any network work when no content cache backs the
|
||||||
// originals it must fetch.
|
// originals it must fetch.
|
||||||
backup(opts?: BackupOptions): Promise<BackupResult> {
|
backup(opts?: BackupOptions): Promise<BackupResult> {
|
||||||
@@ -710,7 +722,8 @@ export class Library {
|
|||||||
// - the ML fetch pass;
|
// - the ML fetch pass;
|
||||||
// - the precache fetches already running.
|
// - the precache fetches already running.
|
||||||
// A caller can then remove the cache directory. A refresh failure is
|
// A caller can then remove the cache directory. A refresh failure is
|
||||||
// reported through `status()`, not thrown here.
|
// reported through `status()`, and a `refreshCollection()` failure to its
|
||||||
|
// own caller. Neither is thrown here.
|
||||||
async close(): Promise<void> {
|
async close(): Promise<void> {
|
||||||
this.closed = true;
|
this.closed = true;
|
||||||
const precacheClosed = this.precache?.close();
|
const precacheClosed = this.precache?.close();
|
||||||
@@ -736,17 +749,18 @@ export class Library {
|
|||||||
// The background loop's refresh: run a cycle unless a refresh or
|
// The background loop's refresh: run a cycle unless a refresh or
|
||||||
// `refreshCollection()` is in flight (or the library is closed), and never
|
// `refreshCollection()` is in flight (or the library is closed), and never
|
||||||
// let a failure escape. The background path reports errors through
|
// let a failure escape. The background path reports errors through
|
||||||
// `status()`/`onProgress`; it does not throw. Resolves once the cycle it
|
// `status()`/`onProgress`; it does not throw. It resolves once the cycle
|
||||||
// started (or skipped past) settles.
|
// it started settles, or at once when it starts none.
|
||||||
private runRefresh(): Promise<void> {
|
private runRefresh(): Promise<void> {
|
||||||
if (this.closed || this.cycle) return Promise.resolve();
|
if (this.closed || this.cycle) return Promise.resolve();
|
||||||
return this.startCycle(true, () => this.refreshCycle()).catch(() => {});
|
return this.startCycle(true, () => this.refreshCycle()).catch(() => {});
|
||||||
}
|
}
|
||||||
|
|
||||||
// A fresh read's refresh (issue #75): force a cycle and await it, rejecting
|
// A fresh read's refresh (issue #75): join the running refresh or start
|
||||||
// if it fails. Concurrent fresh reads coalesce onto the one in-flight cycle
|
// one, and await it, rejecting if it fails. Concurrent fresh reads
|
||||||
// — the background loop's included — so they never fan out into redundant
|
// coalesce onto the one in-flight cycle — the background loop's included —
|
||||||
// server round-trips. A running `refreshCollection()` is waited for first.
|
// so they never fan out into redundant server round-trips. A running
|
||||||
|
// `refreshCollection()` is waited for first.
|
||||||
private async refreshNow(): Promise<void> {
|
private async refreshNow(): Promise<void> {
|
||||||
while (this.cycle && !this.cycleIsRefresh) {
|
while (this.cycle && !this.cycleIsRefresh) {
|
||||||
await this.cycle.catch(() => {});
|
await this.cycle.catch(() => {});
|
||||||
@@ -789,9 +803,9 @@ export class Library {
|
|||||||
this.lastError = undefined;
|
this.lastError = undefined;
|
||||||
this.emit({ operation: "refresh", status: "done" });
|
this.emit({ operation: "refresh", status: "done" });
|
||||||
// Backfill ML data for the files this refresh knows about. It runs
|
// Backfill ML data for the files this refresh knows about. It runs
|
||||||
// outside the refresh's success/failure so a fetch or disk problem
|
// outside the refresh's success or failure, so a fetch or disk
|
||||||
// there never marks the metadata refresh failed, and it is not
|
// problem there never marks the metadata refresh failed. It is not
|
||||||
// awaited so it never stalls the refresh interval. Its failure is
|
// awaited, so it never stalls the refresh interval. Its failure is
|
||||||
// reported through `status()` and `onProgress`.
|
// reported through `status()` and `onProgress`.
|
||||||
void this.fetchMLDataNow().catch(() => {});
|
void this.fetchMLDataNow().catch(() => {});
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
@@ -849,8 +863,8 @@ export class Library {
|
|||||||
// No copy is stored, yet the stored cursor has reached this
|
// No copy is stored, yet the stored cursor has reached this
|
||||||
// `updationTime`. A refresh has then seen this version or a later one,
|
// `updationTime`. A refresh has then seen this version or a later one,
|
||||||
// and dropped the collection: it was deleted, or did not decrypt.
|
// and dropped the collection: it was deleted, or did not decrypt.
|
||||||
// Storing it would bring it back for good, since no later refresh sees
|
// Storing it would bring it back, and no later refresh would see it
|
||||||
// it again.
|
// again until the server's copy changes.
|
||||||
if (
|
if (
|
||||||
!known &&
|
!known &&
|
||||||
collection.updationTime <= this.store.collectionsSinceTime
|
collection.updationTime <= this.store.collectionsSinceTime
|
||||||
@@ -875,15 +889,16 @@ export class Library {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Commit a collections page and the file pages fetched for its collections
|
// Commit a collections page and the file pages fetched for its collections
|
||||||
// to the store, then save. A collection or file that does not decrypt is
|
// to the store. A collection or file that does not decrypt is left out, as
|
||||||
// left out, as if deleted. The reason is kept in the store. No later
|
// if deleted. The reason is kept in the store. No later refresh sees it
|
||||||
// refresh sees it again until the server's copy changes.
|
// again until the server's copy changes. Last, save the store when it
|
||||||
|
// holds changes not yet saved.
|
||||||
private async commit(
|
private async commit(
|
||||||
page: CollectionsPage,
|
page: CollectionsPage,
|
||||||
filePages: { collectionID: number; page: FilesPage }[],
|
filePages: { collectionID: number; page: FilesPage }[],
|
||||||
): Promise<void> {
|
): Promise<void> {
|
||||||
// Network work done; commit to the store and persist only if something
|
// The network work is done, so change the store now. `changed` says
|
||||||
// actually changed.
|
// whether this commit changed anything.
|
||||||
let changed = false;
|
let changed = false;
|
||||||
|
|
||||||
if (this.store.userID !== this.userID) {
|
if (this.store.userID !== this.userID) {
|
||||||
@@ -946,8 +961,8 @@ export class Library {
|
|||||||
// Reproject and notify subscribers of the delta. This tracks RAM (what
|
// Reproject and notify subscribers of the delta. This tracks RAM (what
|
||||||
// reads see), so it fires whether or not the save below succeeds. A
|
// reads see), so it fires whether or not the save below succeeds. A
|
||||||
// failed save rejects the caller, and a refresh also records it in
|
// failed save rejects the caller, and a refresh also records it in
|
||||||
// `status().lastError`. `lastRecords` advances every changed refresh so
|
// `status().lastError`. `lastRecords` advances on every commit that
|
||||||
// the next diff is against current state.
|
// changes something, so the next diff is against current state.
|
||||||
if (changed) {
|
if (changed) {
|
||||||
const next = this.deriveNow();
|
const next = this.deriveNow();
|
||||||
if (this.subscribers.size > 0) {
|
if (this.subscribers.size > 0) {
|
||||||
@@ -958,8 +973,9 @@ export class Library {
|
|||||||
// Recompute the fill orders and pinned set against the new library.
|
// Recompute the fill orders and pinned set against the new library.
|
||||||
this.precache?.update(next);
|
this.precache?.update(next);
|
||||||
}
|
}
|
||||||
// Re-kick the fills every cycle: a finished sweep starts afresh to pick
|
// Re-kick the fills on every commit. A finished sweep starts afresh to
|
||||||
// up new files and retry any that failed, and a running one is left be.
|
// pick up new files and retry any that failed. A running one is left
|
||||||
|
// be.
|
||||||
this.precache?.start();
|
this.precache?.start();
|
||||||
|
|
||||||
// Persist whenever RAM holds changes disk has not accepted — including
|
// Persist whenever RAM holds changes disk has not accepted — including
|
||||||
@@ -982,13 +998,14 @@ export class Library {
|
|||||||
return this.mlFetch;
|
return this.mlFetch;
|
||||||
}
|
}
|
||||||
|
|
||||||
// One ML fetch pass: fetch, decrypt and store the ML data for every file
|
// One ML fetch pass. It fetches, decrypts and stores the ML data of every
|
||||||
// the store knows about that is not cached (or whose `updationTime` has
|
// file the store knows about that is not cached, or whose `updationTime`
|
||||||
// advanced), through the metadata pool, and update the CLIP index. For a
|
// has advanced. It fetches through the metadata pool, and updates the CLIP
|
||||||
// file whose ML data does not decrypt, gunzip or parse, it gathers the
|
// index. For a file whose ML data does not decrypt, gunzip or parse, it
|
||||||
// reason instead, and puts the reasons it gathered in `mlDataErrors` when
|
// gathers the reason instead. It puts the reasons it gathered in
|
||||||
// it ends without failing. A failure leaves `mlDataErrors` as it was and
|
// `mlDataErrors` when it ends without failing. A failure leaves
|
||||||
// is reported through `status()` and `onProgress`, then thrown.
|
// `mlDataErrors` as it was and is reported through `status()` and
|
||||||
|
// `onProgress`, then thrown.
|
||||||
private async runMLFetch(): Promise<void> {
|
private async runMLFetch(): Promise<void> {
|
||||||
const mldata = this.mlStore;
|
const mldata = this.mlStore;
|
||||||
// Bind so the call keeps the client as its receiver when invoked
|
// Bind so the call keeps the client as its receiver when invoked
|
||||||
@@ -1056,9 +1073,9 @@ export class Library {
|
|||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
// The distinct files the store holds, one entry per fileID (a file in
|
// The distinct files the store holds, one entry per fileID, since a file
|
||||||
// several collections shares its ML data), each carrying the key and the
|
// in several collections shares its ML data. Each entry carries the key
|
||||||
// newest `updationTime` seen across its memberships.
|
// and the newest `updationTime` seen across the file's memberships.
|
||||||
private uniqueFiles(): {
|
private uniqueFiles(): {
|
||||||
id: number;
|
id: number;
|
||||||
key: Uint8Array;
|
key: Uint8Array;
|
||||||
|
|||||||
Reference in new issue
Block a user