sneak/secret
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: comprehensive README.md and TODO.md for 1.0 release - Updated README.md with detailed documentation of all commands, architecture, and storage system - Added comprehensive TODO.md with critical, important, and trivial items for 1.0 release - Documented three-layer key hierarchy and vault system - Included examples, security considerations, and cross-platform notes - Identified key bugs including missing cobra usage printing after errors - Categorized 50+ items by priority with timeline estimates

Browse Source
This commit is contained in:
Jeffrey Paul
2025-05-29 05:58:21 -07:00
parent 2443256338
commit c526b68f58
2 changed files with 519 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files

439
README.md
View File

@@ -1,136 +1,353 @@
# architecture
# Secret - Hierarchical Secret Manager
* `secret vault` allows you to change 'vaults'. vaults are just a universe
of secrets (and a single associated long-term key). you can have multiple
vaults, each with its own long-term key and secrets. this is useful for
separating work and personal secrets, or for separating different projects
with different long-term keys.
Secret is a modern, secure command-line secret manager that implements a hierarchical key architecture for storing and managing sensitive data. It supports multiple vaults, various unlock mechanisms, and provides secure storage using the Age encryption library.
* `secret vault list`
* `secret vault create <name>` creates a new profile
* `secret vault select <name>` selects (switches to) a profile
## Core Architecture
the first and initial vault is titled `default`.
### Three-Layer Key Hierarchy
* `secret init` initializes a new vault and imports a user-provided BIP39
mnemonic phrase. The user must provide their own mnemonic phrase. The
long-term keypair is derived from this mnemonic. The long-term keypair is
used to encrypt and decrypt secrets. The long-term keypair is stored in the
vault. The private key for the vault is encrypted to a short-term keypair.
The short-term keypair private key is encrypted to a passphrase.
Secret implements a sophisticated three-layer key architecture:
Use `secret generate mnemonic` to create a new BIP39 mnemonic phrase if you
need one.
1. **Long-term Keys**: Derived from BIP39 mnemonic phrases, these provide the foundation for all encryption
2. **Unlock Keys**: Short-term keys that encrypt the long-term keys, supporting multiple authentication methods
3. **Secret-specific Keys**: Per-secret keys that encrypt individual secret values
if there is already a vault, `secret init` exits with an error.
### Vault System
* `secret import [vaultname]` will derive a long-term key pair from a bip32 seed
phrase and import it into the named vault. if no vault name is specified,
`default` is used. if the named vault already exists, it exits with
an error.
Vaults provide logical separation of secrets, each with its own long-term key and unlock key set. This allows for complete isolation between different contexts (work, personal, projects).
first:
## Installation
* the long term key pair will be derived in memory
* a random short term (unlock) key pair will be derived in memory
Build from source:
```bash
git clone <repository>
cd secret
make build
```
then:
## Quick Start
* the long term key pair public key will be written to disk
* the short term key pair public key will be written to disk
* the long term key pair private key will be encrypted to the short term
public key and written to disk
* the short term key pair private key will be encrypted to a passphrase
and written to disk
1. **Initialize the secret manager**:
```bash
secret init
```
This creates the default vault and prompts for a BIP39 mnemonic phrase.
* `secret enroll sep` creates a short-term keypair inside the secure enclave
of a macOS device. it will then use one of your existing short-term
keypairs to decrypt the long-term keypair, re-encrypt it to the secure
enclave short-term keypair, and write it to disk. it requires an existing
vault, and errors otherwise.
2. **Generate a mnemonic** (if needed):
```bash
secret generate mnemonic
```
* short-term keypairs are called 'unlock keys'.
3. **Add a secret**:
```bash
echo "my-password" | secret add myservice/password
```
* `secret add <secret>` adds a secret to the vault. this will generate a
keypair (secret-specific key) and encrypt the private portion of the
secret-specific key (called an 'unlock key') to the long-term keypair and
write it to disk. if the secret already exists it will not overwrite, but
will exit with an error, unless `--force`/`-f` is used. the secret
identifier is [a-z0-9\.\-\_\/]+ and is used as a storage directory name.
slashes are converted to % signs.
4. **Retrieve a secret**:
```bash
secret get myservice/password
```
in a future version, overwriting a secret will cause the current secret to
get moved to a timestamped history archive.
## Commands Reference
* `secret get <secret>` retrieves a secret from the vault. this will use an
unlock keypair to decrypt the long-term keypair in memory, then use the
long-term keypair to decrypt the secret-specific keypair, which is then
used to decrypt the secret. the secret is then returned in plaintext on
stdout.
### Initialization
* `secret keys list` lists the short-term keypairs in the current vault.
this will show the public keys of the short-term keypairs and their
creation dates, as well as any flags (such as `hsm`). their identifiers
are a metahash of the public key data using the sha256 algorithm.
#### `secret init`
Initializes the secret manager with a default vault. Prompts for a BIP39 mnemonic phrase and creates the initial directory structure.
* `secret keys rm <keyid>` removes a short-term keypair from the vault. this will
remove the short-term keypair from the vault, and remove the long-term
keypair from the short-term keypair.
**Environment Variables:**
- `SB_SECRET_MNEMONIC`: Pre-set mnemonic phrase
- `SB_UNLOCK_PASSPHRASE`: Pre-set unlock passphrase
* `secret keys add pgp <pgp keyid>` adds a new short-term keypair to the vault.
this will generate a new short-term keypair and encrypt it to a given gpg
key, to allow unlocking a vault with an existing gpg key, for people who
use yubikeys or other gpg keys with an agent. the new short-term keypair
is randomly generated, the public key stored, and the private key encrypted
to the gpg key and stored.
### Vault Management
* `secret key select <keyid>` selects a short-term keypair to use for
`secret get` operations.
#### `secret vault list [--json]`
Lists all available vaults.
# file layout
#### `secret vault create <name>`
Creates a new vault with the specified name.
$BASE = ~/.config/berlin.sneak.pkg.secret (on linux per XDG)
$BASE = ~/Library/Application Support/berlin.sneak.pkg.secret (on macOS)
#### `secret vault select <name>`
Switches to the specified vault for subsequent operations.
$BASE/configuration.json
$BASE/currentvault -> $BASE/vaults.d/default (symlink)
$BASE/vaults.d/default/
$BASE/vaults.d/default/vault-metadata.json
$BASE/vaults.d/default/pub.age
$BASE/vaults.d/default/current-unlock-key -> $BASE/vaults.d/default/unlock.d/passphrase (symlink)
$BASE/vaults.d/default/unlock.d/passphrase/unlock-metadata.json
$BASE/vaults.d/default/unlock.d/passphrase/pub.age
$BASE/vaults.d/default/unlock.d/passphrase/priv.age
$BASE/vaults.d/default/unlock.d/passphrase/longterm.age # long-term keypair, encrypted to this short-term keypair
$BASE/vaults.d/default/unlock.d/sep/unlock-metadata.json
$BASE/vaults.d/default/unlock.d/sep/pub.age
$BASE/vaults.d/default/unlock.d/sep/priv.age
$BASE/vaults.d/default/unlock.d/sep/longterm.age # long-term keypair, encrypted to this short-term keypair
$BASE/vaults.d/default/unlock.d/pgp/unlock-metadata.json
$BASE/vaults.d/default/unlock.d/pgp/pub.age
$BASE/vaults.d/default/unlock.d/pgp/priv.asc
$BASE/vaults.d/default/unlock.d/pgp/longterm.age # long-term keypair, encrypted to this short-term keypair
$BASE/vaults.d/default/secrets.d/my-tinder-password/value.age
$BASE/vaults.d/default/secrets.d/my-tinder-password/pub.age
$BASE/vaults.d/default/secrets.d/my-tinder-password/priv.age # secret-specific key, encrypted to long-term key
$BASE/vaults.d/default/secrets.d/my-tinder-password/secret-metadata.json
$BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/value.age
$BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/pub.age
$BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/priv.age
$BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/secret-metadata.json
### Secret Management
# example configuration.json
#### `secret add <secret-name> [--force]`
Adds a secret to the current vault. Reads the secret value from stdin.
- `--force, -f`: Overwrite existing secret
`json
{
"$id": "https://berlin.sneak.pkg.secret/configuration.json",
"$schema": "https://berlin.sneak.pkg.secret/configuration.schema.json",
"version": 1,
"configuration": {
"createdAt": "2025-01-01T00:00:00Z",
"requireAuth": false,
"pubKey": "<public key of the long-term keypair>",
"pubKeyFingerprint": "<fingerprint of the long-term keypair>"
}
}
`
**Secret Name Format:** `[a-z0-9\.\-\_\/]+`
- Forward slashes (`/`) are converted to percent signs (`%`) for storage
- Examples: `database/password`, `api.key`, `ssh_private_key`
#### `secret get <secret-name>`
Retrieves and outputs a secret value to stdout.
#### `secret list [filter] [--json]` / `secret ls`
Lists all secrets in the current vault. Optional filter for substring matching.
### Key Generation
#### `secret generate mnemonic`
Generates a cryptographically secure BIP39 mnemonic phrase.
#### `secret generate secret <name> [--length=16] [--type=base58] [--force]`
Generates and stores a random secret.
- `--length, -l`: Length of generated secret (default: 16)
- `--type, -t`: Type of secret (`base58`, `alnum`)
- `--force, -f`: Overwrite existing secret
### Unlock Key Management
#### `secret keys list [--json]`
Lists all unlock keys in the current vault with their metadata.
#### `secret keys add <type> [options]`
Creates a new unlock key of the specified type:
**Types:**
- `passphrase`: Password-protected unlock key
- `macos-sep`: macOS Secure Enclave unlock key (Touch ID/Face ID)
- `pgp`: GPG/PGP key unlock key
**Options:**
- `--keyid <id>`: GPG key ID (required for PGP type)
#### `secret keys rm <key-id>`
Removes an unlock key from the current vault.
#### `secret key select <key-id>`
Selects an unlock key as the current default for operations.
### Import Operations
#### `secret import [vault-name]`
Imports a mnemonic phrase into the specified vault (defaults to "default").
#### `secret enroll`
Enrolls a macOS Secure Enclave unlock key for biometric authentication.
### Encryption Operations
#### `secret encrypt <secret-name> [--input=file] [--output=file]`
Encrypts data using an Age key stored as a secret. If the secret doesn't exist, generates a new Age key.
#### `secret decrypt <secret-name> [--input=file] [--output=file]`
Decrypts data using an Age key stored as a secret.
## Storage Architecture
### Directory Structure
```
$BASE/ # ~/.config/berlin.sneak.pkg.secret (Linux) or ~/Library/Application Support/berlin.sneak.pkg.secret (macOS)
├── configuration.json # Global configuration
├── currentvault -> vaults.d/default # Symlink to current vault
└── vaults.d/
├── default/ # Default vault
│ ├── vault-metadata.json # Vault metadata
│ ├── pub.age # Long-term public key
│ ├── current-unlock-key -> unlock.d/passphrase # Current unlock key symlink
│ ├── unlock.d/ # Unlock keys directory
│ │ ├── passphrase/ # Passphrase unlock key
│ │ │ ├── unlock-metadata.json # Unlock key metadata
│ │ │ ├── pub.age # Unlock key public key
│ │ │ ├── priv.age # Unlock key private key (encrypted)
│ │ │ └── longterm.age # Long-term private key (encrypted to this unlock key)
│ │ ├── sep/ # Secure Enclave unlock key
│ │ │ ├── unlock-metadata.json
│ │ │ ├── pub.age
│ │ │ ├── priv.age
│ │ │ └── longterm.age
│ │ └── pgp/ # PGP unlock key
│ │ ├── unlock-metadata.json
│ │ ├── pub.age
│ │ ├── priv.asc # PGP-encrypted private key
│ │ └── longterm.age
│ └── secrets.d/ # Secrets directory
│ ├── my-service%password/ # Secret directory (slashes encoded as %)
│ │ ├── value.age # Encrypted secret value
│ │ ├── pub.age # Secret-specific public key
│ │ ├── priv.age # Secret-specific private key (encrypted to long-term key)
│ │ └── secret-metadata.json # Secret metadata
│ └── api%keys%production/
│ ├── value.age
│ ├── pub.age
│ ├── priv.age
│ └── secret-metadata.json
└── work/ # Additional vault
└── ... (same structure as default)
```
### Key Management and Encryption Flow
#### Long-term Keys
- **Source**: Derived from BIP39 mnemonic phrases using hierarchical deterministic (HD) key derivation
- **Purpose**: Master keys for each vault, used to encrypt secret-specific keys
- **Storage**: Public key stored as `pub.age`, private key encrypted by unlock keys
#### Unlock Keys
Unlock keys provide different authentication methods to access the long-term keys:
1. **Passphrase Unlock Keys**:
- Private key encrypted using a user-provided passphrase
- Stored as encrypted Age identity in `priv.age`
2. **macOS Secure Enclave Keys**:
- Private key stored in the Secure Enclave
- Requires biometric authentication (Touch ID/Face ID)
- Provides hardware-backed security
3. **PGP Unlock Keys**:
- Private key encrypted using an existing GPG key
- Compatible with hardware tokens (YubiKey, etc.)
- Stored as PGP-encrypted data in `priv.asc`
#### Secret-specific Keys
- Each secret has its own encryption key pair
- Private key encrypted to the vault's long-term key
- Provides forward secrecy and granular access control
### Environment Variables
- `SB_SECRET_STATE_DIR`: Custom state directory location
- `SB_SECRET_MNEMONIC`: Pre-set mnemonic phrase (avoids interactive prompt)
- `SB_UNLOCK_PASSPHRASE`: Pre-set unlock passphrase (avoids interactive prompt)
- `SB_GPG_KEY_ID`: GPG key ID for PGP unlock keys
## Security Features
### Encryption
- Uses the [Age encryption library](https://age-encryption.org/) with X25519 keys
- All private keys are encrypted at rest
- No plaintext secrets stored on disk
### Access Control
- Multiple authentication methods supported
- Hierarchical key architecture provides defense in depth
- Vault isolation prevents cross-contamination
### Forward Secrecy
- Per-secret encryption keys limit exposure if compromised
- Long-term keys protected by multiple unlock key layers
### Hardware Integration
- macOS Secure Enclave support for biometric authentication
- Hardware token support via PGP/GPG integration
## Examples
### Basic Workflow
```bash
# Initialize with a new mnemonic
secret generate mnemonic # Copy the output
secret init # Paste the mnemonic when prompted
# Add some secrets
echo "supersecret123" | secret add database/prod/password
echo "api-key-xyz" | secret add services/api/key
echo "ssh-private-key-content" | secret add ssh/servers/web01
# List and retrieve secrets
secret list
secret get database/prod/password
secret get services/api/key
```
### Multi-vault Setup
```bash
# Create separate vaults for different contexts
secret vault create work
secret vault create personal
# Work with work vault
secret vault select work
echo "work-db-pass" | secret add database/password
secret keys add macos-sep # Add Touch ID authentication
# Switch to personal vault
secret vault select personal
echo "personal-email-pass" | secret add email/password
# List all vaults
secret vault list
```
### Advanced Authentication
```bash
# Add multiple unlock methods
secret keys add passphrase # Password-based
secret keys add macos-sep # Touch ID (macOS only)
secret keys add pgp --keyid ABCD1234 # GPG key
# List unlock keys
secret keys list
# Select a specific unlock key
secret key select <key-id>
```
### Encryption/Decryption with Age Keys
```bash
# Generate an Age key and store it as a secret
secret generate secret encryption/mykey
# Encrypt a file using the stored key
secret encrypt encryption/mykey --input document.txt --output document.txt.age
# Decrypt the file
secret decrypt encryption/mykey --input document.txt.age --output document.txt
```
## Technical Details
### Cryptographic Primitives
- **Key Derivation**: BIP32/BIP39 hierarchical deterministic key derivation
- **Encryption**: Age (X25519 + ChaCha20-Poly1305)
- **Key Exchange**: X25519 elliptic curve Diffie-Hellman
- **Authentication**: Poly1305 MAC
### File Formats
- **Age Files**: Standard Age encryption format (.age extension)
- **Metadata**: JSON format with timestamps and type information
- **Configuration**: JSON configuration files
### Cross-Platform Support
- **macOS**: Full support including Secure Enclave integration
- **Linux**: Full support (excluding Secure Enclave features)
- **Windows**: Basic support (filesystem operations only)
## Security Considerations
### Threat Model
- Protects against unauthorized access to secret values
- Provides defense against compromise of individual components
- Supports hardware-backed authentication where available
### Best Practices
1. Use strong, unique passphrases for unlock keys
2. Enable hardware authentication (Secure Enclave, hardware tokens) when available
3. Regularly audit unlock keys and remove unused ones
4. Keep mnemonic phrases securely backed up offline
5. Use separate vaults for different security contexts
### Limitations
- Requires access to unlock keys for secret retrieval
- Mnemonic phrases must be securely stored and backed up
- Hardware features limited to supported platforms
## Development
### Building
```bash
make build # Build binary
make test # Run tests
make lint # Run linter
```
### Testing
The project includes comprehensive tests:
```bash
./test_secret_manager.sh # Full integration test suite
go test ./... # Unit tests
```