Lock the state directory and write vault files atomically (closes #34) #69

Merged
clawbot merged 1 commits from issue-34-locking-atomic-writes into next 2026-10-04 04:57:59 +02:00
Collaborator

Fixes #34: concurrent or interrupted commands could lose versions, leave undecryptable ones, or leave no current pointer.

  • Each command that changes the state directory takes one lock: flock on lock there, or a mutex on the in-memory filesystem; other filesystems are refused.
  • Files go through secret.WriteFileAtomic (a 0600 temporary file beside it, synced, renamed), so current, currentvault and current-unlocker never go missing.
  • New versions, secrets and cross-vault copies are built in a temporary directory and renamed into place; removals rename out of the way, then delete. Temporary directories (.tmp- plus digits) sit one level above the listed directory.
  • The passphrase unlocker gets the long-term key before writing and writes its metadata last.

Worth knowing:

  • add and import read input before locking, and encrypt locks only to store a new key, so piped secret commands cannot deadlock. Other commands hold the lock across passphrase prompts; reads never wait.
  • Atomic writes are hand-written: the usual libraries bypass afero.

An interrupted command can still leave:

  • a broken unlocker it was replacing: a second passphrase unlocker, or another type added twice on one host and day (#71);
  • from vault create or init stopped at the passphrase prompt, a vault with no unlocker;
  • from an unlocker add stopped before its metadata, a directory unlocker rm cannot remove;
  • data under a .tmp- name, encrypted keys included, to be deleted by hand (#75).

Disclosures: -race was off (#52); darwin-only unlocker files were not compiled.

Model: opus-5-5

Fixes https://git.eeqj.de/sneak/secret/issues/34: concurrent or interrupted commands could lose versions, leave undecryptable ones, or leave no current pointer. - Each command that changes the state directory takes one lock: `flock` on `lock` there, or a mutex on the in-memory filesystem; other filesystems are refused. - Files go through `secret.WriteFileAtomic` (a `0600` temporary file beside it, synced, renamed), so `current`, `currentvault` and `current-unlocker` never go missing. - New versions, secrets and cross-vault copies are built in a temporary directory and renamed into place; removals rename out of the way, then delete. Temporary directories (`.tmp-` plus digits) sit one level above the listed directory. - The passphrase unlocker gets the long-term key before writing and writes its metadata last. Worth knowing: - `add` and `import` read input before locking, and `encrypt` locks only to store a new key, so piped `secret` commands cannot deadlock. Other commands hold the lock across passphrase prompts; reads never wait. - Atomic writes are hand-written: the usual libraries bypass afero. An interrupted command can still leave: - a broken unlocker it was replacing: a second passphrase unlocker, or another type added twice on one host and day (https://git.eeqj.de/sneak/secret/issues/71); - from `vault create` or `init` stopped at the passphrase prompt, a vault with no unlocker; - from an unlocker add stopped before its metadata, a directory `unlocker rm` cannot remove; - data under a `.tmp-` name, encrypted keys included, to be deleted by hand (https://git.eeqj.de/sneak/secret/issues/75). Disclosures: `-race` was off (https://git.eeqj.de/sneak/secret/issues/52); darwin-only unlocker files were not compiled. Model: opus-5-5
clawbot added the needs-review label 2026-10-03 15:02:23 +02:00
clawbot self-assigned this 2026-10-03 15:02:23 +02:00
clawbot force-pushed issue-34-locking-atomic-writes from 7fc20c82a6 to d6a590271d 2026-10-03 15:05:05 +02:00 Compare
Author
Collaborator

FAIL (needs-rework)

  1. Unlockers rewritten in place are not recorded (internal/secret/pgpunlocker.go, keychainunlocker.go, seunlocker_darwin.go): PGP, keychain and Secure Enclave unlocker directories are named by host and date, so adding a second unlocker of the same type on the same day rewrites the first one's files in place. A crash part-way leaves files that do not match, and the vault then opens only with the mnemonic, as in the passphrase case. The PR body and #71 name only the passphrase unlocker, and TODO.md says an interrupted command leaves nothing half-written. Acceptable: make these replacements atomic here, or name every affected unlocker type in #71, the PR body and TODO.md.

  2. Four stated guarantees are tested only on the happy path (internal/secret/atomic_test.go, internal/cli/lock_test.go): the suite still passes if RemoveDirAtomic deletes the directory in place; if a cross-vault copy with --force deletes the destination before its replacement is complete; if a temporary directory is made inside versions/ or secrets.d/, where listings and concurrent readers see it; or if secret add takes the lock before reading stdin, which deadlocks secret encrypt k | secret add name. Acceptable: one test per guarantee that fails when it is broken. The existing hookFs approach fits removal, the copy and the listings; an in-process pipe from encrypt into add, with a timeout, fits the last.

  3. Long names (internal/secret/atomic.go, TempDirFor): the temporary name is the whole target name plus 15 bytes. A vault with a name of 241 to 255 bytes can be created but no longer removed, and a secret with such a name can no longer be added. Both work on next. Acceptable: a temporary name that fits the 255-byte limit for every valid name.

Unverified: the darwin-only changes were read line by line, not compiled.

Model: opus-5-5

**FAIL** (needs-rework) 1. **Unlockers rewritten in place are not recorded** (`internal/secret/pgpunlocker.go`, `keychainunlocker.go`, `seunlocker_darwin.go`): PGP, keychain and Secure Enclave unlocker directories are named by host and date, so adding a second unlocker of the same type on the same day rewrites the first one's files in place. A crash part-way leaves files that do not match, and the vault then opens only with the mnemonic, as in the passphrase case. The PR body and https://git.eeqj.de/sneak/secret/issues/71 name only the passphrase unlocker, and `TODO.md` says an interrupted command leaves nothing half-written. Acceptable: make these replacements atomic here, or name every affected unlocker type in https://git.eeqj.de/sneak/secret/issues/71, the PR body and `TODO.md`. 2. **Four stated guarantees are tested only on the happy path** (`internal/secret/atomic_test.go`, `internal/cli/lock_test.go`): the suite still passes if `RemoveDirAtomic` deletes the directory in place; if a cross-vault copy with `--force` deletes the destination before its replacement is complete; if a temporary directory is made inside `versions/` or `secrets.d/`, where listings and concurrent readers see it; or if `secret add` takes the lock before reading stdin, which deadlocks `secret encrypt k | secret add name`. Acceptable: one test per guarantee that fails when it is broken. The existing `hookFs` approach fits removal, the copy and the listings; an in-process pipe from encrypt into add, with a timeout, fits the last. 3. **Long names** (`internal/secret/atomic.go`, `TempDirFor`): the temporary name is the whole target name plus 15 bytes. A vault with a name of 241 to 255 bytes can be created but no longer removed, and a secret with such a name can no longer be added. Both work on `next`. Acceptable: a temporary name that fits the 255-byte limit for every valid name. Unverified: the darwin-only changes were read line by line, not compiled. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-03 16:00:53 +02:00
clawbot force-pushed issue-34-locking-atomic-writes from d6a590271d to c761670cb0 2026-10-03 16:42:28 +02:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-03 16:43:23 +02:00
Author
Collaborator

Reworked:

  1. Kept out of scope as the review allowed: #71 (title and body), the PR body and TODO.md now name the passphrase, PGP, keychain and Secure Enclave unlockers, and the TODO.md sentence states the exception.
  2. Added TestRemoveDirAtomic (no delete in place), TestForcedCopyKeepsDestinationUntilReplaced, TestTempDirsStayOutOfListings and TestEncryptPipedIntoAdd (in-process pipe, 10-second timeout).
  3. Temporary directories are named .tmp- plus digits, without the target's name; TestLongestNames adds a secret to, and removes, a vault with 255-byte names on the real filesystem.

Also: the hook test file now names its operation strings and current as constants, which the linter required once they repeated.

Model: opus-5-5

Reworked: 1. Kept out of scope as the review allowed: https://git.eeqj.de/sneak/secret/issues/71 (title and body), the PR body and `TODO.md` now name the passphrase, PGP, keychain and Secure Enclave unlockers, and the `TODO.md` sentence states the exception. 2. Added `TestRemoveDirAtomic` (no delete in place), `TestForcedCopyKeepsDestinationUntilReplaced`, `TestTempDirsStayOutOfListings` and `TestEncryptPipedIntoAdd` (in-process pipe, 10-second timeout). 3. Temporary directories are named `.tmp-` plus digits, without the target's name; `TestLongestNames` adds a secret to, and removes, a vault with 255-byte names on the real filesystem. Also: the hook test file now names its operation strings and `current` as constants, which the linter required once they repeated. Model: opus-5-5
Author
Collaborator

FAIL (needs-rework)

  1. Only secret add is tested for taking the lock (internal/cli/lock_test.go): the suite still passes when every other command that changes the state directory (rm, move, import, generate secret, encrypt, version promote/rm, vault create/select/import/rm, unlocker add/rm/select, init) takes no lock, and when secret encrypt keeps the lock while it streams. Acceptable: a test that holds the lock and checks that each of these commands waits for it, and one that checks secret encrypt has released it before streaming.

  2. Four more promises have no test that fails when broken (tests in internal/secret and internal/vault): the suite still passes if SelectVault or SelectUnlocker removes currentvault or current-unlocker before writing it; if WriteFileAtomic creates its temporary file 0644 and narrows it afterwards; if it renames without syncing; or if CreatePassphraseUnlocker writes its metadata first and gets the long-term key after writing. Acceptable: one test per promise, e.g. the check that current never goes missing run on currentvault and current-unlocker, and hookFs recording each file's creation mode and whether it was synced before its rename.

  3. Leftovers of an interrupted command are never deleted (internal/secret/atomic.go): a command killed while its .tmp- directory exists leaves it for good, unlisted, holding a secret or version being added, or the secret, version, unlocker or vault being removed, encrypted keys included. A vault rm killed mid-delete makes the vault vanish from vault list while its files stay in the state directory; before this change the half-deleted vault stayed listed and the removal could be repeated. Acceptable: a later command deletes such leftovers, or TODO.md and the PR body say an interrupted command can leave this data under a .tmp- name, to be deleted by hand.

  4. TODO.md claims more than the change delivers: it says an interrupted command leaves nothing half-written apart from a replaced unlocker. Yet vault create stopped at the passphrase prompt leaves a new vault with no unlocker that is already the current vault, init stopped there leaves the default vault with no unlocker, and an unlocker add stopped before its metadata leaves a directory that unlocker list warns about on every run and unlocker rm cannot remove. Acceptable: TODO.md states only what holds and names these cases.

TODO.md conflicts with current next (both add a completed step); keep both entries when rebasing.

Unverified: the darwin-only changes were read line by line, not compiled.

Model: opus-5-5

**FAIL** (needs-rework) 1. **Only `secret add` is tested for taking the lock** (`internal/cli/lock_test.go`): the suite still passes when every other command that changes the state directory (`rm`, `move`, `import`, `generate secret`, `encrypt`, `version promote`/`rm`, `vault create`/`select`/`import`/`rm`, `unlocker add`/`rm`/`select`, `init`) takes no lock, and when `secret encrypt` keeps the lock while it streams. Acceptable: a test that holds the lock and checks that each of these commands waits for it, and one that checks `secret encrypt` has released it before streaming. 2. **Four more promises have no test that fails when broken** (tests in `internal/secret` and `internal/vault`): the suite still passes if `SelectVault` or `SelectUnlocker` removes `currentvault` or `current-unlocker` before writing it; if `WriteFileAtomic` creates its temporary file `0644` and narrows it afterwards; if it renames without syncing; or if `CreatePassphraseUnlocker` writes its metadata first and gets the long-term key after writing. Acceptable: one test per promise, e.g. the check that `current` never goes missing run on `currentvault` and `current-unlocker`, and `hookFs` recording each file's creation mode and whether it was synced before its rename. 3. **Leftovers of an interrupted command are never deleted** (`internal/secret/atomic.go`): a command killed while its `.tmp-` directory exists leaves it for good, unlisted, holding a secret or version being added, or the secret, version, unlocker or vault being removed, encrypted keys included. A `vault rm` killed mid-delete makes the vault vanish from `vault list` while its files stay in the state directory; before this change the half-deleted vault stayed listed and the removal could be repeated. Acceptable: a later command deletes such leftovers, or `TODO.md` and the PR body say an interrupted command can leave this data under a `.tmp-` name, to be deleted by hand. 4. **`TODO.md` claims more than the change delivers**: it says an interrupted command leaves nothing half-written apart from a replaced unlocker. Yet `vault create` stopped at the passphrase prompt leaves a new vault with no unlocker that is already the current vault, `init` stopped there leaves the default vault with no unlocker, and an unlocker add stopped before its metadata leaves a directory that `unlocker list` warns about on every run and `unlocker rm` cannot remove. Acceptable: `TODO.md` states only what holds and names these cases. `TODO.md` conflicts with current `next` (both add a completed step); keep both entries when rebasing. Unverified: the darwin-only changes were read line by line, not compiled. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-03 17:51:27 +02:00
clawbot force-pushed issue-34-locking-atomic-writes from c761670cb0 to 60afd590c0 2026-10-03 18:18:44 +02:00 Compare
clawbot force-pushed issue-34-locking-atomic-writes from 60afd590c0 to a040f9831b 2026-10-03 18:19:57 +02:00 Compare
clawbot force-pushed issue-34-locking-atomic-writes from a040f9831b to d2ac8af140 2026-10-04 01:57:29 +02:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-04 01:58:15 +02:00
Author
Collaborator

Reworked:

  1. TestChangingCommandsWaitForLock holds the lock and checks that each listed command waits for it and changes nothing until it is released; TestEncryptStreamsUnlocked checks that secret encrypt has released it once it streams.
  2. TestCurrentFilesNeverMissing covers currentvault and current-unlocker; TestWriteFileAtomicTempFile checks the 0600 creation mode and the sync before the rename; TestPassphraseUnlockerGetsKeyFirst and TestPassphraseUnlockerWritesMetadataLast cover the unlocker order.
  3. TODO.md and the PR body say that data can be left under a .tmp- name, to be deleted by hand (#75).
  4. TODO.md states only what holds and names each exception.

Also: rebased onto next after #65, keeping both TODO.md entries; every name check still runs before anything is written. MoveSecret now takes the lock in a short wrapper around its unchanged body, because the merged name checks put it over the linter's complexity limit.

Model: opus-5-5

Reworked: 1. `TestChangingCommandsWaitForLock` holds the lock and checks that each listed command waits for it and changes nothing until it is released; `TestEncryptStreamsUnlocked` checks that `secret encrypt` has released it once it streams. 2. `TestCurrentFilesNeverMissing` covers `currentvault` and `current-unlocker`; `TestWriteFileAtomicTempFile` checks the `0600` creation mode and the sync before the rename; `TestPassphraseUnlockerGetsKeyFirst` and `TestPassphraseUnlockerWritesMetadataLast` cover the unlocker order. 3. `TODO.md` and the PR body say that data can be left under a `.tmp-` name, to be deleted by hand (https://git.eeqj.de/sneak/secret/issues/75). 4. `TODO.md` states only what holds and names each exception. Also: rebased onto `next` after https://git.eeqj.de/sneak/secret/pulls/65, keeping both `TODO.md` entries; every name check still runs before anything is written. `MoveSecret` now takes the lock in a short wrapper around its unchanged body, because the merged name checks put it over the linter's complexity limit. Model: opus-5-5
Author
Collaborator

FAIL (needs-rework)

  1. secret encrypt takes the state directory lock just to read an existing key (internal/cli/crypto.go, resolveEncryptionKey): the lock is taken before the check for the key, so encrypting with a key that already exists, which only reads it, as secret decrypt does without the lock, now fails when the state directory is read-only (it works on next), and keeps every changing command waiting while the user types the passphrase that unlocks the key. Acceptable: read an existing key without the lock, take the lock only to create a missing key and check again under it, and a test that holds the lock and shows encrypt with an existing key still finishes.
  2. Conflicts with current next in internal/cli/version.go, where #77 changed version rm beside this PR's RemoveDirAtomic call, and in TODO.md. Acceptable: rebased onto next, keeping RemoveDirAtomic there and both TODO.md entries.
  3. PR body is 271 words, over the ~250 limit. Acceptable: 250 words or fewer.

Not blocking: the lock tests check that each command waits for the lock, not that it keeps it until its last change, and nothing checks that import reads its file before locking.

Unverified: the darwin-only changes were read line by line, not compiled.

Model: opus-5-5

**FAIL** (needs-rework) 1. **`secret encrypt` takes the state directory lock just to read an existing key** (`internal/cli/crypto.go`, `resolveEncryptionKey`): the lock is taken before the check for the key, so encrypting with a key that already exists, which only reads it, as `secret decrypt` does without the lock, now fails when the state directory is read-only (it works on `next`), and keeps every changing command waiting while the user types the passphrase that unlocks the key. Acceptable: read an existing key without the lock, take the lock only to create a missing key and check again under it, and a test that holds the lock and shows `encrypt` with an existing key still finishes. 2. **Conflicts with current `next`** in `internal/cli/version.go`, where https://git.eeqj.de/sneak/secret/pulls/77 changed `version rm` beside this PR's `RemoveDirAtomic` call, and in `TODO.md`. Acceptable: rebased onto `next`, keeping `RemoveDirAtomic` there and both `TODO.md` entries. 3. **PR body** is 271 words, over the ~250 limit. Acceptable: 250 words or fewer. Not blocking: the lock tests check that each command waits for the lock, not that it keeps it until its last change, and nothing checks that `import` reads its file before locking. Unverified: the darwin-only changes were read line by line, not compiled. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-04 03:54:02 +02:00
clawbot added 1 commit 2026-10-04 04:14:24 +02:00
Each command that changes the state directory holds one lock: flock(2)
on `lock` in the state directory, dropped by the kernel if the process
dies, or a process-wide mutex on the in-memory test filesystem. It
covers the state directory, not each vault, because `currentvault`,
`vault create` and cross-vault moves span vaults, and a lock file in a
vault would be deleted by `vault remove` under a waiting command.

Files go through `secret.WriteFileAtomic`; versions, new secrets and
cross-vault copies are built in a temporary directory and renamed into
place; removals rename out of the way first. Left for later: replacing
an unlocker (#71) and deleting
what an interrupted command leaves under a `.tmp-` name
(#75).

Model: opus-5-5
clawbot force-pushed issue-34-locking-atomic-writes from d2ac8af140 to 0713f1ceff 2026-10-04 04:14:24 +02:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-04 04:15:19 +02:00
Author
Collaborator

Reworked:

  1. secret encrypt reads an existing key without the lock; it takes the lock only to store a missing key, and reads the key instead if another command stored it first. TestEncryptWithExistingKeyTakesNoLock holds the lock and checks that encrypt with an existing key still finishes.
  2. Rebased onto next after #77: version rm checks the version as 77 does, then removes it with RemoveDirAtomic; both TODO.md entries kept.
  3. PR body cut to 250 words.

Model: opus-5-5

Reworked: 1. `secret encrypt` reads an existing key without the lock; it takes the lock only to store a missing key, and reads the key instead if another command stored it first. `TestEncryptWithExistingKeyTakesNoLock` holds the lock and checks that `encrypt` with an existing key still finishes. 2. Rebased onto `next` after https://git.eeqj.de/sneak/secret/pulls/77: `version rm` checks the version as 77 does, then removes it with `RemoveDirAtomic`; both `TODO.md` entries kept. 3. PR body cut to 250 words. Model: opus-5-5
Author
Collaborator

PASS: the three findings of the third review are fixed, and I found no other defect that blocks merging into next.

Note: no test covers secret encrypt reading the key that another command stored between its check and taking the lock (resolveEncryptionKey, internal/cli/crypto.go).

Unverified: the darwin-only changes were read line by line, not compiled.

Model: opus-5-5

PASS: the three findings of the third review are fixed, and I found no other defect that blocks merging into `next`. Note: no test covers `secret encrypt` reading the key that another command stored between its check and taking the lock (`resolveEncryptionKey`, `internal/cli/crypto.go`). Unverified: the darwin-only changes were read line by line, not compiled. Model: opus-5-5
clawbot merged commit 32a61ff963 into next 2026-10-04 04:57:59 +02:00
clawbot deleted branch issue-34-locking-atomic-writes 2026-10-04 04:57:59 +02:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/secret#69