diff --git a/README.md b/README.md index 2125cba..7a08369 100644 --- a/README.md +++ b/README.md @@ -1,136 +1,353 @@ -# architecture +# Secret - Hierarchical Secret Manager -* `secret vault` allows you to change 'vaults'. vaults are just a universe - of secrets (and a single associated long-term key). you can have multiple - vaults, each with its own long-term key and secrets. this is useful for - separating work and personal secrets, or for separating different projects - with different long-term keys. +Secret is a modern, secure command-line secret manager that implements a hierarchical key architecture for storing and managing sensitive data. It supports multiple vaults, various unlock mechanisms, and provides secure storage using the Age encryption library. - * `secret vault list` - * `secret vault create ` creates a new profile - * `secret vault select ` selects (switches to) a profile +## Core Architecture - the first and initial vault is titled `default`. +### Three-Layer Key Hierarchy -* `secret init` initializes a new vault and imports a user-provided BIP39 - mnemonic phrase. The user must provide their own mnemonic phrase. The - long-term keypair is derived from this mnemonic. The long-term keypair is - used to encrypt and decrypt secrets. The long-term keypair is stored in the - vault. The private key for the vault is encrypted to a short-term keypair. - The short-term keypair private key is encrypted to a passphrase. +Secret implements a sophisticated three-layer key architecture: - Use `secret generate mnemonic` to create a new BIP39 mnemonic phrase if you - need one. +1. **Long-term Keys**: Derived from BIP39 mnemonic phrases, these provide the foundation for all encryption +2. **Unlock Keys**: Short-term keys that encrypt the long-term keys, supporting multiple authentication methods +3. **Secret-specific Keys**: Per-secret keys that encrypt individual secret values - if there is already a vault, `secret init` exits with an error. +### Vault System -* `secret import [vaultname]` will derive a long-term key pair from a bip32 seed - phrase and import it into the named vault. if no vault name is specified, - `default` is used. if the named vault already exists, it exits with - an error. +Vaults provide logical separation of secrets, each with its own long-term key and unlock key set. This allows for complete isolation between different contexts (work, personal, projects). - first: +## Installation - * the long term key pair will be derived in memory - * a random short term (unlock) key pair will be derived in memory +Build from source: +```bash +git clone +cd secret +make build +``` - then: +## Quick Start - * the long term key pair public key will be written to disk - * the short term key pair public key will be written to disk - * the long term key pair private key will be encrypted to the short term - public key and written to disk - * the short term key pair private key will be encrypted to a passphrase - and written to disk +1. **Initialize the secret manager**: + ```bash + secret init + ``` + This creates the default vault and prompts for a BIP39 mnemonic phrase. -* `secret enroll sep` creates a short-term keypair inside the secure enclave - of a macOS device. it will then use one of your existing short-term - keypairs to decrypt the long-term keypair, re-encrypt it to the secure - enclave short-term keypair, and write it to disk. it requires an existing - vault, and errors otherwise. +2. **Generate a mnemonic** (if needed): + ```bash + secret generate mnemonic + ``` -* short-term keypairs are called 'unlock keys'. +3. **Add a secret**: + ```bash + echo "my-password" | secret add myservice/password + ``` -* `secret add ` adds a secret to the vault. this will generate a - keypair (secret-specific key) and encrypt the private portion of the - secret-specific key (called an 'unlock key') to the long-term keypair and - write it to disk. if the secret already exists it will not overwrite, but - will exit with an error, unless `--force`/`-f` is used. the secret - identifier is [a-z0-9\.\-\_\/]+ and is used as a storage directory name. - slashes are converted to % signs. +4. **Retrieve a secret**: + ```bash + secret get myservice/password + ``` - in a future version, overwriting a secret will cause the current secret to - get moved to a timestamped history archive. +## Commands Reference -* `secret get ` retrieves a secret from the vault. this will use an - unlock keypair to decrypt the long-term keypair in memory, then use the - long-term keypair to decrypt the secret-specific keypair, which is then - used to decrypt the secret. the secret is then returned in plaintext on - stdout. +### Initialization -* `secret keys list` lists the short-term keypairs in the current vault. - this will show the public keys of the short-term keypairs and their - creation dates, as well as any flags (such as `hsm`). their identifiers - are a metahash of the public key data using the sha256 algorithm. +#### `secret init` +Initializes the secret manager with a default vault. Prompts for a BIP39 mnemonic phrase and creates the initial directory structure. -* `secret keys rm ` removes a short-term keypair from the vault. this will - remove the short-term keypair from the vault, and remove the long-term - keypair from the short-term keypair. +**Environment Variables:** +- `SB_SECRET_MNEMONIC`: Pre-set mnemonic phrase +- `SB_UNLOCK_PASSPHRASE`: Pre-set unlock passphrase -* `secret keys add pgp ` adds a new short-term keypair to the vault. - this will generate a new short-term keypair and encrypt it to a given gpg - key, to allow unlocking a vault with an existing gpg key, for people who - use yubikeys or other gpg keys with an agent. the new short-term keypair - is randomly generated, the public key stored, and the private key encrypted - to the gpg key and stored. +### Vault Management -* `secret key select ` selects a short-term keypair to use for - `secret get` operations. +#### `secret vault list [--json]` +Lists all available vaults. -# file layout +#### `secret vault create ` +Creates a new vault with the specified name. -$BASE = ~/.config/berlin.sneak.pkg.secret (on linux per XDG) -$BASE = ~/Library/Application Support/berlin.sneak.pkg.secret (on macOS) +#### `secret vault select ` +Switches to the specified vault for subsequent operations. - $BASE/configuration.json - $BASE/currentvault -> $BASE/vaults.d/default (symlink) - $BASE/vaults.d/default/ - $BASE/vaults.d/default/vault-metadata.json - $BASE/vaults.d/default/pub.age - $BASE/vaults.d/default/current-unlock-key -> $BASE/vaults.d/default/unlock.d/passphrase (symlink) - $BASE/vaults.d/default/unlock.d/passphrase/unlock-metadata.json - $BASE/vaults.d/default/unlock.d/passphrase/pub.age - $BASE/vaults.d/default/unlock.d/passphrase/priv.age - $BASE/vaults.d/default/unlock.d/passphrase/longterm.age # long-term keypair, encrypted to this short-term keypair - $BASE/vaults.d/default/unlock.d/sep/unlock-metadata.json - $BASE/vaults.d/default/unlock.d/sep/pub.age - $BASE/vaults.d/default/unlock.d/sep/priv.age - $BASE/vaults.d/default/unlock.d/sep/longterm.age # long-term keypair, encrypted to this short-term keypair - $BASE/vaults.d/default/unlock.d/pgp/unlock-metadata.json - $BASE/vaults.d/default/unlock.d/pgp/pub.age - $BASE/vaults.d/default/unlock.d/pgp/priv.asc - $BASE/vaults.d/default/unlock.d/pgp/longterm.age # long-term keypair, encrypted to this short-term keypair - $BASE/vaults.d/default/secrets.d/my-tinder-password/value.age - $BASE/vaults.d/default/secrets.d/my-tinder-password/pub.age - $BASE/vaults.d/default/secrets.d/my-tinder-password/priv.age # secret-specific key, encrypted to long-term key - $BASE/vaults.d/default/secrets.d/my-tinder-password/secret-metadata.json - $BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/value.age - $BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/pub.age - $BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/priv.age - $BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/secret-metadata.json +### Secret Management -# example configuration.json +#### `secret add [--force]` +Adds a secret to the current vault. Reads the secret value from stdin. +- `--force, -f`: Overwrite existing secret -`json -{ - "$id": "https://berlin.sneak.pkg.secret/configuration.json", - "$schema": "https://berlin.sneak.pkg.secret/configuration.schema.json", - "version": 1, - "configuration": { - "createdAt": "2025-01-01T00:00:00Z", - "requireAuth": false, - "pubKey": "", - "pubKeyFingerprint": "" - } -} -` +**Secret Name Format:** `[a-z0-9\.\-\_\/]+` +- Forward slashes (`/`) are converted to percent signs (`%`) for storage +- Examples: `database/password`, `api.key`, `ssh_private_key` + +#### `secret get ` +Retrieves and outputs a secret value to stdout. + +#### `secret list [filter] [--json]` / `secret ls` +Lists all secrets in the current vault. Optional filter for substring matching. + +### Key Generation + +#### `secret generate mnemonic` +Generates a cryptographically secure BIP39 mnemonic phrase. + +#### `secret generate secret [--length=16] [--type=base58] [--force]` +Generates and stores a random secret. +- `--length, -l`: Length of generated secret (default: 16) +- `--type, -t`: Type of secret (`base58`, `alnum`) +- `--force, -f`: Overwrite existing secret + +### Unlock Key Management + +#### `secret keys list [--json]` +Lists all unlock keys in the current vault with their metadata. + +#### `secret keys add [options]` +Creates a new unlock key of the specified type: + +**Types:** +- `passphrase`: Password-protected unlock key +- `macos-sep`: macOS Secure Enclave unlock key (Touch ID/Face ID) +- `pgp`: GPG/PGP key unlock key + +**Options:** +- `--keyid `: GPG key ID (required for PGP type) + +#### `secret keys rm ` +Removes an unlock key from the current vault. + +#### `secret key select ` +Selects an unlock key as the current default for operations. + +### Import Operations + +#### `secret import [vault-name]` +Imports a mnemonic phrase into the specified vault (defaults to "default"). + +#### `secret enroll` +Enrolls a macOS Secure Enclave unlock key for biometric authentication. + +### Encryption Operations + +#### `secret encrypt [--input=file] [--output=file]` +Encrypts data using an Age key stored as a secret. If the secret doesn't exist, generates a new Age key. + +#### `secret decrypt [--input=file] [--output=file]` +Decrypts data using an Age key stored as a secret. + +## Storage Architecture + +### Directory Structure + +``` +$BASE/ # ~/.config/berlin.sneak.pkg.secret (Linux) or ~/Library/Application Support/berlin.sneak.pkg.secret (macOS) +├── configuration.json # Global configuration +├── currentvault -> vaults.d/default # Symlink to current vault +└── vaults.d/ + ├── default/ # Default vault + │ ├── vault-metadata.json # Vault metadata + │ ├── pub.age # Long-term public key + │ ├── current-unlock-key -> unlock.d/passphrase # Current unlock key symlink + │ ├── unlock.d/ # Unlock keys directory + │ │ ├── passphrase/ # Passphrase unlock key + │ │ │ ├── unlock-metadata.json # Unlock key metadata + │ │ │ ├── pub.age # Unlock key public key + │ │ │ ├── priv.age # Unlock key private key (encrypted) + │ │ │ └── longterm.age # Long-term private key (encrypted to this unlock key) + │ │ ├── sep/ # Secure Enclave unlock key + │ │ │ ├── unlock-metadata.json + │ │ │ ├── pub.age + │ │ │ ├── priv.age + │ │ │ └── longterm.age + │ │ └── pgp/ # PGP unlock key + │ │ ├── unlock-metadata.json + │ │ ├── pub.age + │ │ ├── priv.asc # PGP-encrypted private key + │ │ └── longterm.age + │ └── secrets.d/ # Secrets directory + │ ├── my-service%password/ # Secret directory (slashes encoded as %) + │ │ ├── value.age # Encrypted secret value + │ │ ├── pub.age # Secret-specific public key + │ │ ├── priv.age # Secret-specific private key (encrypted to long-term key) + │ │ └── secret-metadata.json # Secret metadata + │ └── api%keys%production/ + │ ├── value.age + │ ├── pub.age + │ ├── priv.age + │ └── secret-metadata.json + └── work/ # Additional vault + └── ... (same structure as default) +``` + +### Key Management and Encryption Flow + +#### Long-term Keys +- **Source**: Derived from BIP39 mnemonic phrases using hierarchical deterministic (HD) key derivation +- **Purpose**: Master keys for each vault, used to encrypt secret-specific keys +- **Storage**: Public key stored as `pub.age`, private key encrypted by unlock keys + +#### Unlock Keys +Unlock keys provide different authentication methods to access the long-term keys: + +1. **Passphrase Unlock Keys**: + - Private key encrypted using a user-provided passphrase + - Stored as encrypted Age identity in `priv.age` + +2. **macOS Secure Enclave Keys**: + - Private key stored in the Secure Enclave + - Requires biometric authentication (Touch ID/Face ID) + - Provides hardware-backed security + +3. **PGP Unlock Keys**: + - Private key encrypted using an existing GPG key + - Compatible with hardware tokens (YubiKey, etc.) + - Stored as PGP-encrypted data in `priv.asc` + +#### Secret-specific Keys +- Each secret has its own encryption key pair +- Private key encrypted to the vault's long-term key +- Provides forward secrecy and granular access control + +### Environment Variables + +- `SB_SECRET_STATE_DIR`: Custom state directory location +- `SB_SECRET_MNEMONIC`: Pre-set mnemonic phrase (avoids interactive prompt) +- `SB_UNLOCK_PASSPHRASE`: Pre-set unlock passphrase (avoids interactive prompt) +- `SB_GPG_KEY_ID`: GPG key ID for PGP unlock keys + +## Security Features + +### Encryption +- Uses the [Age encryption library](https://age-encryption.org/) with X25519 keys +- All private keys are encrypted at rest +- No plaintext secrets stored on disk + +### Access Control +- Multiple authentication methods supported +- Hierarchical key architecture provides defense in depth +- Vault isolation prevents cross-contamination + +### Forward Secrecy +- Per-secret encryption keys limit exposure if compromised +- Long-term keys protected by multiple unlock key layers + +### Hardware Integration +- macOS Secure Enclave support for biometric authentication +- Hardware token support via PGP/GPG integration + +## Examples + +### Basic Workflow +```bash +# Initialize with a new mnemonic +secret generate mnemonic # Copy the output +secret init # Paste the mnemonic when prompted + +# Add some secrets +echo "supersecret123" | secret add database/prod/password +echo "api-key-xyz" | secret add services/api/key +echo "ssh-private-key-content" | secret add ssh/servers/web01 + +# List and retrieve secrets +secret list +secret get database/prod/password +secret get services/api/key +``` + +### Multi-vault Setup +```bash +# Create separate vaults for different contexts +secret vault create work +secret vault create personal + +# Work with work vault +secret vault select work +echo "work-db-pass" | secret add database/password +secret keys add macos-sep # Add Touch ID authentication + +# Switch to personal vault +secret vault select personal +echo "personal-email-pass" | secret add email/password + +# List all vaults +secret vault list +``` + +### Advanced Authentication +```bash +# Add multiple unlock methods +secret keys add passphrase # Password-based +secret keys add macos-sep # Touch ID (macOS only) +secret keys add pgp --keyid ABCD1234 # GPG key + +# List unlock keys +secret keys list + +# Select a specific unlock key +secret key select +``` + +### Encryption/Decryption with Age Keys +```bash +# Generate an Age key and store it as a secret +secret generate secret encryption/mykey + +# Encrypt a file using the stored key +secret encrypt encryption/mykey --input document.txt --output document.txt.age + +# Decrypt the file +secret decrypt encryption/mykey --input document.txt.age --output document.txt +``` + +## Technical Details + +### Cryptographic Primitives +- **Key Derivation**: BIP32/BIP39 hierarchical deterministic key derivation +- **Encryption**: Age (X25519 + ChaCha20-Poly1305) +- **Key Exchange**: X25519 elliptic curve Diffie-Hellman +- **Authentication**: Poly1305 MAC + +### File Formats +- **Age Files**: Standard Age encryption format (.age extension) +- **Metadata**: JSON format with timestamps and type information +- **Configuration**: JSON configuration files + +### Cross-Platform Support +- **macOS**: Full support including Secure Enclave integration +- **Linux**: Full support (excluding Secure Enclave features) +- **Windows**: Basic support (filesystem operations only) + +## Security Considerations + +### Threat Model +- Protects against unauthorized access to secret values +- Provides defense against compromise of individual components +- Supports hardware-backed authentication where available + +### Best Practices +1. Use strong, unique passphrases for unlock keys +2. Enable hardware authentication (Secure Enclave, hardware tokens) when available +3. Regularly audit unlock keys and remove unused ones +4. Keep mnemonic phrases securely backed up offline +5. Use separate vaults for different security contexts + +### Limitations +- Requires access to unlock keys for secret retrieval +- Mnemonic phrases must be securely stored and backed up +- Hardware features limited to supported platforms + +## Development + +### Building +```bash +make build # Build binary +make test # Run tests +make lint # Run linter +``` + +### Testing +The project includes comprehensive tests: +```bash +./test_secret_manager.sh # Full integration test suite +go test ./... # Unit tests +``` diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..c52a83d --- /dev/null +++ b/TODO.md @@ -0,0 +1,191 @@ +# TODO for 1.0 Release + +This document outlines the bugs, issues, and improvements that need to be addressed before the 1.0 release of the secret manager. + +## Critical (Blockers for Release) + +### Error Handling and User Experience + +- [ ] **Missing Cobra usage printing after errors**: Commands should print usage information when they fail due to incorrect arguments or usage. Need to configure `SilenceUsage: false` and `SilenceErrors: false` on cobra commands. + +- [ ] **Inconsistent error messages**: Error messages need standardization and should be user-friendly. Many errors currently expose internal implementation details. + +- [ ] **Missing validation for vault names**: Vault names should be validated against a safe character set to prevent filesystem issues. + +- [ ] **No graceful handling of corrupted state**: If key files are corrupted or missing, the tool should provide clear error messages and recovery suggestions. + +### Core Functionality Bugs + +- [ ] **Directory structure inconsistency**: The README and test script reference different directory structures: + - Current code uses `unlock.d/` but documentation shows `unlock-keys.d/` + - Secret files use inconsistent naming (`secret.age` vs `value.age`) + +- [ ] **Symlink handling on non-Unix systems**: The symlink resolution in `resolveVaultSymlink()` may fail on Windows or in certain environments. + +- [ ] **Missing current unlock key initialization**: When creating vaults, no default unlock key is selected, which can cause operations to fail. + +- [ ] **Race conditions in file operations**: Multiple concurrent operations could corrupt the vault state due to lack of file locking. + +### Security Issues + +- [ ] **Insecure temporary file handling**: Temporary files containing sensitive data may not be properly cleaned up or secured. + +- [ ] **Missing secure memory clearing**: Sensitive data in memory (passphrases, keys) should be cleared after use. + +- [ ] **Weak default permissions**: Some files may be created with overly permissive default permissions. + +## Important (Should be fixed before release) + +### User Interface Improvements + +- [ ] **Add confirmation prompts for destructive operations**: Operations like `keys rm` and vault deletion should require confirmation. + +- [ ] **Improve progress indicators**: Long operations (key generation, encryption) should show progress. + +- [ ] **Better secret name validation**: Currently allows some characters that may cause issues, needs comprehensive validation. + +- [ ] **Add `--help` examples**: Command help should include practical examples for each operation. + +### Command Implementation Gaps + +- [ ] **`secret keys rm` not fully implemented**: Based on test output, this command may not be working correctly. + +- [ ] **`secret key select` not fully implemented**: Key selection functionality appears incomplete. + +- [ ] **Missing vault deletion command**: No way to delete vaults that are no longer needed. + +- [ ] **No secret deletion command**: Missing `secret rm ` functionality. + +- [ ] **Missing secret history/versioning**: No way to see previous versions of secrets or restore old values. + +### Configuration and Environment + +- [ ] **Global configuration not fully implemented**: The `configuration.json` file structure exists but isn't used consistently. + +- [ ] **Missing environment variable validation**: Environment variables should be validated for format and security. + +- [ ] **No configuration file validation**: JSON configuration files should be validated against schemas. + +### PGP Integration Issues + +- [ ] **Incomplete PGP unlock key implementation**: The `--keyid` parameter processing may not be fully working. + +- [ ] **Missing GPG agent integration**: Should detect and use existing GPG agent when available. + +- [ ] **No validation of GPG key existence**: Should verify the specified GPG key exists before creating PGP unlock keys. + +### Cross-Platform Issues + +- [ ] **macOS Secure Enclave error handling**: Better error messages when biometric authentication fails or isn't available. + +- [ ] **Windows path handling**: File paths may not work correctly on Windows systems. + +- [ ] **XDG compliance on Linux**: Should respect `XDG_CONFIG_HOME` and other XDG environment variables. + +## Trivial (Nice to have) + +### Code Quality + +- [ ] **Add more comprehensive unit tests**: Current test coverage could be improved, especially for error conditions. + +- [ ] **Reduce code duplication**: Several functions have similar patterns that could be refactored. + +- [ ] **Improve function documentation**: Many functions lack proper Go documentation comments. + +- [ ] **Add static analysis**: Integrate tools like `staticcheck`, `golangci-lint` with more linters. + +### Performance Optimizations + +- [ ] **Cache unlock key operations**: Avoid re-reading unlock key metadata on every operation. + +- [ ] **Optimize file I/O**: Batch file operations where possible to reduce syscalls. + +- [ ] **Add connection pooling for HSM operations**: For hardware security module operations. + +### User Experience Enhancements + +- [ ] **Add shell completion**: Bash/Zsh completion for commands and secret names. + +- [ ] **Colored output**: Use colors to improve readability of lists and error messages. + +- [ ] **Add `--quiet` flag**: Option to suppress non-essential output. + +- [ ] **Smart secret name suggestions**: When a secret name is not found, suggest similar names. + +### Additional Features + +- [ ] **Secret templates**: Predefined templates for common secret types (database URLs, API keys, etc.). + +- [ ] **Bulk operations**: Import/export multiple secrets at once. + +- [ ] **Secret sharing**: Secure sharing of secrets between vaults or users. + +- [ ] **Audit logging**: Log all secret access and modifications. + +- [ ] **Integration tests for hardware features**: Automated testing of Secure Enclave and GPG functionality. + +### Documentation + +- [ ] **Man pages**: Generate and install proper Unix man pages. + +- [ ] **API documentation**: Document the internal API for potential library use. + +- [ ] **Migration guide**: Document how to migrate from other secret managers. + +- [ ] **Security audit documentation**: Document security assumptions and threat model. + +## Architecture Improvements + +### Code Structure + +- [ ] **Consistent interface implementation**: Ensure all unlock key types properly implement the UnlockKey interface. + +- [ ] **Better separation of concerns**: Some functions in CLI do too much and should be split. + +- [ ] **Improved error types**: Create specific error types instead of using generic `fmt.Errorf`. + +### Testing Infrastructure + +- [ ] **Mock filesystem consistency**: Ensure mock filesystem behavior matches real filesystem in all cases. + +- [ ] **Integration test isolation**: Tests should not affect each other or the host system. + +- [ ] **Performance benchmarks**: Add benchmarks for crypto operations and file I/O. + +## Technical Debt + +- [ ] **Remove unused code**: Clean up any dead code or unused imports. + +- [ ] **Standardize JSON schemas**: Create proper JSON schemas for all configuration files. + +- [ ] **Improve error propagation**: Many functions swallow important context in error messages. + +- [ ] **Consistent naming conventions**: Some variables and functions use inconsistent naming. + +## Development Workflow + +- [ ] **Add pre-commit hooks**: Ensure code quality and formatting before commits. + +- [ ] **Continuous integration**: Set up CI/CD pipeline with automated testing. + +- [ ] **Release automation**: Automate the build and release process. + +- [ ] **Dependency management**: Regular updates and security scanning of dependencies. + +--- + +## Priority Assessment + +**Critical items** block the 1.0 release and must be fixed for basic functionality and security. + +**Important items** should be addressed for a polished user experience but don't block the release. + +**Trivial items** are enhancements that can be addressed in future releases. + +## Estimated Timeline + +- Critical: 2-3 weeks +- Important: 3-4 weeks +- Trivial: Ongoing post-1.0 + +Total estimated time to 1.0: 5-7 weeks with focused development effort. \ No newline at end of file