sneak/secret
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: comprehensive README.md and TODO.md for 1.0 release - Updated README.md with detailed documentation of all commands, architecture, and storage system - Added comprehensive TODO.md with critical, important, and trivial items for 1.0 release - Documented three-layer key hierarchy and vault system - Included examples, security considerations, and cross-platform notes - Identified key bugs including missing cobra usage printing after errors - Categorized 50+ items by priority with timeline estimates

Browse Source
This commit is contained in:
Jeffrey Paul 2025-05-29 05:58:21 -07:00
parent 2443256338
commit c526b68f58
2 changed files with 519 additions and 111 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

439
README.md
View File

@ -1,136 +1,353 @@
# architecture
# Secret - Hierarchical Secret Manager
* `secret vault` allows you to change 'vaults'. vaults are just a universe
of secrets (and a single associated long-term key). you can have multiple
vaults, each with its own long-term key and secrets. this is useful for
separating work and personal secrets, or for separating different projects
with different long-term keys.
Secret is a modern, secure command-line secret manager that implements a hierarchical key architecture for storing and managing sensitive data. It supports multiple vaults, various unlock mechanisms, and provides secure storage using the Age encryption library.
* `secret vault list`
* `secret vault create <name>` creates a new profile
* `secret vault select <name>` selects (switches to) a profile
## Core Architecture
the first and initial vault is titled `default`.
### Three-Layer Key Hierarchy
* `secret init` initializes a new vault and imports a user-provided BIP39
mnemonic phrase. The user must provide their own mnemonic phrase. The
long-term keypair is derived from this mnemonic. The long-term keypair is
used to encrypt and decrypt secrets. The long-term keypair is stored in the
vault. The private key for the vault is encrypted to a short-term keypair.
The short-term keypair private key is encrypted to a passphrase.
Secret implements a sophisticated three-layer key architecture:
Use `secret generate mnemonic` to create a new BIP39 mnemonic phrase if you
need one.
1. **Long-term Keys**: Derived from BIP39 mnemonic phrases, these provide the foundation for all encryption
2. **Unlock Keys**: Short-term keys that encrypt the long-term keys, supporting multiple authentication methods
3. **Secret-specific Keys**: Per-secret keys that encrypt individual secret values
if there is already a vault, `secret init` exits with an error.
### Vault System
* `secret import [vaultname]` will derive a long-term key pair from a bip32 seed
phrase and import it into the named vault. if no vault name is specified,
`default` is used. if the named vault already exists, it exits with
an error.
Vaults provide logical separation of secrets, each with its own long-term key and unlock key set. This allows for complete isolation between different contexts (work, personal, projects).
first:
## Installation
* the long term key pair will be derived in memory
* a random short term (unlock) key pair will be derived in memory
Build from source:
```bash
git clone <repository>
cd secret
make build
```
then:
## Quick Start
* the long term key pair public key will be written to disk
* the short term key pair public key will be written to disk
* the long term key pair private key will be encrypted to the short term
public key and written to disk
* the short term key pair private key will be encrypted to a passphrase
and written to disk
1. **Initialize the secret manager**:
```bash
secret init
```
This creates the default vault and prompts for a BIP39 mnemonic phrase.
* `secret enroll sep` creates a short-term keypair inside the secure enclave
of a macOS device. it will then use one of your existing short-term
keypairs to decrypt the long-term keypair, re-encrypt it to the secure
enclave short-term keypair, and write it to disk. it requires an existing
vault, and errors otherwise.
2. **Generate a mnemonic** (if needed):
```bash
secret generate mnemonic
```
* short-term keypairs are called 'unlock keys'.
3. **Add a secret**:
```bash
echo "my-password" | secret add myservice/password
```
* `secret add <secret>` adds a secret to the vault. this will generate a
keypair (secret-specific key) and encrypt the private portion of the
secret-specific key (called an 'unlock key') to the long-term keypair and
write it to disk. if the secret already exists it will not overwrite, but
will exit with an error, unless `--force`/`-f` is used. the secret
identifier is [a-z0-9\.\-\_\/]+ and is used as a storage directory name.
slashes are converted to % signs.
4. **Retrieve a secret**:
```bash
secret get myservice/password
```
in a future version, overwriting a secret will cause the current secret to
get moved to a timestamped history archive.
## Commands Reference
* `secret get <secret>` retrieves a secret from the vault. this will use an
unlock keypair to decrypt the long-term keypair in memory, then use the
long-term keypair to decrypt the secret-specific keypair, which is then
used to decrypt the secret. the secret is then returned in plaintext on
stdout.
### Initialization
* `secret keys list` lists the short-term keypairs in the current vault.
this will show the public keys of the short-term keypairs and their
creation dates, as well as any flags (such as `hsm`). their identifiers
are a metahash of the public key data using the sha256 algorithm.
#### `secret init`
Initializes the secret manager with a default vault. Prompts for a BIP39 mnemonic phrase and creates the initial directory structure.
* `secret keys rm <keyid>` removes a short-term keypair from the vault. this will
remove the short-term keypair from the vault, and remove the long-term
keypair from the short-term keypair.
**Environment Variables:**
- `SB_SECRET_MNEMONIC`: Pre-set mnemonic phrase
- `SB_UNLOCK_PASSPHRASE`: Pre-set unlock passphrase
* `secret keys add pgp <pgp keyid>` adds a new short-term keypair to the vault.
this will generate a new short-term keypair and encrypt it to a given gpg
key, to allow unlocking a vault with an existing gpg key, for people who
use yubikeys or other gpg keys with an agent. the new short-term keypair
is randomly generated, the public key stored, and the private key encrypted
to the gpg key and stored.
### Vault Management
* `secret key select <keyid>` selects a short-term keypair to use for
`secret get` operations.
#### `secret vault list [--json]`
Lists all available vaults.
# file layout
#### `secret vault create <name>`
Creates a new vault with the specified name.
$BASE = ~/.config/berlin.sneak.pkg.secret (on linux per XDG)
$BASE = ~/Library/Application Support/berlin.sneak.pkg.secret (on macOS)
#### `secret vault select <name>`
Switches to the specified vault for subsequent operations.
$BASE/configuration.json
$BASE/currentvault -> $BASE/vaults.d/default (symlink)
$BASE/vaults.d/default/
$BASE/vaults.d/default/vault-metadata.json
$BASE/vaults.d/default/pub.age
$BASE/vaults.d/default/current-unlock-key -> $BASE/vaults.d/default/unlock.d/passphrase (symlink)
$BASE/vaults.d/default/unlock.d/passphrase/unlock-metadata.json
$BASE/vaults.d/default/unlock.d/passphrase/pub.age
$BASE/vaults.d/default/unlock.d/passphrase/priv.age
$BASE/vaults.d/default/unlock.d/passphrase/longterm.age # long-term keypair, encrypted to this short-term keypair
$BASE/vaults.d/default/unlock.d/sep/unlock-metadata.json
$BASE/vaults.d/default/unlock.d/sep/pub.age
$BASE/vaults.d/default/unlock.d/sep/priv.age
$BASE/vaults.d/default/unlock.d/sep/longterm.age # long-term keypair, encrypted to this short-term keypair
$BASE/vaults.d/default/unlock.d/pgp/unlock-metadata.json
$BASE/vaults.d/default/unlock.d/pgp/pub.age
$BASE/vaults.d/default/unlock.d/pgp/priv.asc
$BASE/vaults.d/default/unlock.d/pgp/longterm.age # long-term keypair, encrypted to this short-term keypair
$BASE/vaults.d/default/secrets.d/my-tinder-password/value.age
$BASE/vaults.d/default/secrets.d/my-tinder-password/pub.age
$BASE/vaults.d/default/secrets.d/my-tinder-password/priv.age # secret-specific key, encrypted to long-term key
$BASE/vaults.d/default/secrets.d/my-tinder-password/secret-metadata.json
$BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/value.age
$BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/pub.age
$BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/priv.age
$BASE/vaults.d/default/secrets.d/mail%berlin.sneak.secrets.imaplogin/secret-metadata.json
### Secret Management
# example configuration.json
#### `secret add <secret-name> [--force]`
Adds a secret to the current vault. Reads the secret value from stdin.
- `--force, -f`: Overwrite existing secret
`json
{
"$id": "https://berlin.sneak.pkg.secret/configuration.json",
"$schema": "https://berlin.sneak.pkg.secret/configuration.schema.json",
"version": 1,
"configuration": {
"createdAt": "2025-01-01T00:00:00Z",
"requireAuth": false,
"pubKey": "<public key of the long-term keypair>",
"pubKeyFingerprint": "<fingerprint of the long-term keypair>"
}
}
`
**Secret Name Format:** `[a-z0-9\.\-\_\/]+`
- Forward slashes (`/`) are converted to percent signs (`%`) for storage
- Examples: `database/password`, `api.key`, `ssh_private_key`
#### `secret get <secret-name>`
Retrieves and outputs a secret value to stdout.
#### `secret list [filter] [--json]` / `secret ls`
Lists all secrets in the current vault. Optional filter for substring matching.
### Key Generation
#### `secret generate mnemonic`
Generates a cryptographically secure BIP39 mnemonic phrase.
#### `secret generate secret <name> [--length=16] [--type=base58] [--force]`
Generates and stores a random secret.
- `--length, -l`: Length of generated secret (default: 16)
- `--type, -t`: Type of secret (`base58`, `alnum`)
- `--force, -f`: Overwrite existing secret
### Unlock Key Management
#### `secret keys list [--json]`
Lists all unlock keys in the current vault with their metadata.
#### `secret keys add <type> [options]`
Creates a new unlock key of the specified type:
**Types:**
- `passphrase`: Password-protected unlock key
- `macos-sep`: macOS Secure Enclave unlock key (Touch ID/Face ID)
- `pgp`: GPG/PGP key unlock key
**Options:**
- `--keyid <id>`: GPG key ID (required for PGP type)
#### `secret keys rm <key-id>`
Removes an unlock key from the current vault.
#### `secret key select <key-id>`
Selects an unlock key as the current default for operations.
### Import Operations
#### `secret import [vault-name]`
Imports a mnemonic phrase into the specified vault (defaults to "default").
#### `secret enroll`
Enrolls a macOS Secure Enclave unlock key for biometric authentication.
### Encryption Operations
#### `secret encrypt <secret-name> [--input=file] [--output=file]`
Encrypts data using an Age key stored as a secret. If the secret doesn't exist, generates a new Age key.
#### `secret decrypt <secret-name> [--input=file] [--output=file]`
Decrypts data using an Age key stored as a secret.
## Storage Architecture
### Directory Structure
```
$BASE/ # ~/.config/berlin.sneak.pkg.secret (Linux) or ~/Library/Application Support/berlin.sneak.pkg.secret (macOS)
├── configuration.json # Global configuration
├── currentvault -> vaults.d/default # Symlink to current vault
└── vaults.d/
├── default/ # Default vault
│ ├── vault-metadata.json # Vault metadata
│ ├── pub.age # Long-term public key
│ ├── current-unlock-key -> unlock.d/passphrase # Current unlock key symlink
│ ├── unlock.d/ # Unlock keys directory
│ │ ├── passphrase/ # Passphrase unlock key
│ │ │ ├── unlock-metadata.json # Unlock key metadata
│ │ │ ├── pub.age # Unlock key public key
│ │ │ ├── priv.age # Unlock key private key (encrypted)
│ │ │ └── longterm.age # Long-term private key (encrypted to this unlock key)
│ │ ├── sep/ # Secure Enclave unlock key
│ │ │ ├── unlock-metadata.json
│ │ │ ├── pub.age
│ │ │ ├── priv.age
│ │ │ └── longterm.age
│ │ └── pgp/ # PGP unlock key
│ │ ├── unlock-metadata.json
│ │ ├── pub.age
│ │ ├── priv.asc # PGP-encrypted private key
│ │ └── longterm.age
│ └── secrets.d/ # Secrets directory
│ ├── my-service%password/ # Secret directory (slashes encoded as %)
│ │ ├── value.age # Encrypted secret value
│ │ ├── pub.age # Secret-specific public key
│ │ ├── priv.age # Secret-specific private key (encrypted to long-term key)
│ │ └── secret-metadata.json # Secret metadata
│ └── api%keys%production/
│ ├── value.age
│ ├── pub.age
│ ├── priv.age
│ └── secret-metadata.json
└── work/ # Additional vault
└── ... (same structure as default)
```
### Key Management and Encryption Flow
#### Long-term Keys
- **Source**: Derived from BIP39 mnemonic phrases using hierarchical deterministic (HD) key derivation
- **Purpose**: Master keys for each vault, used to encrypt secret-specific keys
- **Storage**: Public key stored as `pub.age`, private key encrypted by unlock keys
#### Unlock Keys
Unlock keys provide different authentication methods to access the long-term keys:
1. **Passphrase Unlock Keys**:
- Private key encrypted using a user-provided passphrase
- Stored as encrypted Age identity in `priv.age`
2. **macOS Secure Enclave Keys**:
- Private key stored in the Secure Enclave
- Requires biometric authentication (Touch ID/Face ID)
- Provides hardware-backed security
3. **PGP Unlock Keys**:
- Private key encrypted using an existing GPG key
- Compatible with hardware tokens (YubiKey, etc.)
- Stored as PGP-encrypted data in `priv.asc`
#### Secret-specific Keys
- Each secret has its own encryption key pair
- Private key encrypted to the vault's long-term key
- Provides forward secrecy and granular access control
### Environment Variables
- `SB_SECRET_STATE_DIR`: Custom state directory location
- `SB_SECRET_MNEMONIC`: Pre-set mnemonic phrase (avoids interactive prompt)
- `SB_UNLOCK_PASSPHRASE`: Pre-set unlock passphrase (avoids interactive prompt)
- `SB_GPG_KEY_ID`: GPG key ID for PGP unlock keys
## Security Features
### Encryption
- Uses the [Age encryption library](https://age-encryption.org/) with X25519 keys
- All private keys are encrypted at rest
- No plaintext secrets stored on disk
### Access Control
- Multiple authentication methods supported
- Hierarchical key architecture provides defense in depth
- Vault isolation prevents cross-contamination
### Forward Secrecy
- Per-secret encryption keys limit exposure if compromised
- Long-term keys protected by multiple unlock key layers
### Hardware Integration
- macOS Secure Enclave support for biometric authentication
- Hardware token support via PGP/GPG integration
## Examples
### Basic Workflow
```bash
# Initialize with a new mnemonic
secret generate mnemonic # Copy the output
secret init # Paste the mnemonic when prompted
# Add some secrets
echo "supersecret123" | secret add database/prod/password
echo "api-key-xyz" | secret add services/api/key
echo "ssh-private-key-content" | secret add ssh/servers/web01
# List and retrieve secrets
secret list
secret get database/prod/password
secret get services/api/key
```
### Multi-vault Setup
```bash
# Create separate vaults for different contexts
secret vault create work
secret vault create personal
# Work with work vault
secret vault select work
echo "work-db-pass" | secret add database/password
secret keys add macos-sep # Add Touch ID authentication
# Switch to personal vault
secret vault select personal
echo "personal-email-pass" | secret add email/password
# List all vaults
secret vault list
```
### Advanced Authentication
```bash
# Add multiple unlock methods
secret keys add passphrase # Password-based
secret keys add macos-sep # Touch ID (macOS only)
secret keys add pgp --keyid ABCD1234 # GPG key
# List unlock keys
secret keys list
# Select a specific unlock key
secret key select <key-id>
```
### Encryption/Decryption with Age Keys
```bash
# Generate an Age key and store it as a secret
secret generate secret encryption/mykey
# Encrypt a file using the stored key
secret encrypt encryption/mykey --input document.txt --output document.txt.age
# Decrypt the file
secret decrypt encryption/mykey --input document.txt.age --output document.txt
```
## Technical Details
### Cryptographic Primitives
- **Key Derivation**: BIP32/BIP39 hierarchical deterministic key derivation
- **Encryption**: Age (X25519 + ChaCha20-Poly1305)
- **Key Exchange**: X25519 elliptic curve Diffie-Hellman
- **Authentication**: Poly1305 MAC
### File Formats
- **Age Files**: Standard Age encryption format (.age extension)
- **Metadata**: JSON format with timestamps and type information
- **Configuration**: JSON configuration files
### Cross-Platform Support
- **macOS**: Full support including Secure Enclave integration
- **Linux**: Full support (excluding Secure Enclave features)
- **Windows**: Basic support (filesystem operations only)
## Security Considerations
### Threat Model
- Protects against unauthorized access to secret values
- Provides defense against compromise of individual components
- Supports hardware-backed authentication where available
### Best Practices
1. Use strong, unique passphrases for unlock keys
2. Enable hardware authentication (Secure Enclave, hardware tokens) when available
3. Regularly audit unlock keys and remove unused ones
4. Keep mnemonic phrases securely backed up offline
5. Use separate vaults for different security contexts
### Limitations
- Requires access to unlock keys for secret retrieval
- Mnemonic phrases must be securely stored and backed up
- Hardware features limited to supported platforms
## Development
### Building
```bash
make build # Build binary
make test # Run tests
make lint # Run linter
```
### Testing
The project includes comprehensive tests:
```bash
./test_secret_manager.sh # Full integration test suite
go test ./... # Unit tests
```

191
TODO.md Normal file
View File

@ -0,0 +1,191 @@
# TODO for 1.0 Release
This document outlines the bugs, issues, and improvements that need to be addressed before the 1.0 release of the secret manager.
## Critical (Blockers for Release)
### Error Handling and User Experience
- [ ] **Missing Cobra usage printing after errors**: Commands should print usage information when they fail due to incorrect arguments or usage. Need to configure `SilenceUsage: false` and `SilenceErrors: false` on cobra commands.
- [ ] **Inconsistent error messages**: Error messages need standardization and should be user-friendly. Many errors currently expose internal implementation details.
- [ ] **Missing validation for vault names**: Vault names should be validated against a safe character set to prevent filesystem issues.
- [ ] **No graceful handling of corrupted state**: If key files are corrupted or missing, the tool should provide clear error messages and recovery suggestions.
### Core Functionality Bugs
- [ ] **Directory structure inconsistency**: The README and test script reference different directory structures:
- Current code uses `unlock.d/` but documentation shows `unlock-keys.d/`
- Secret files use inconsistent naming (`secret.age` vs `value.age`)
- [ ] **Symlink handling on non-Unix systems**: The symlink resolution in `resolveVaultSymlink()` may fail on Windows or in certain environments.
- [ ] **Missing current unlock key initialization**: When creating vaults, no default unlock key is selected, which can cause operations to fail.
- [ ] **Race conditions in file operations**: Multiple concurrent operations could corrupt the vault state due to lack of file locking.
### Security Issues
- [ ] **Insecure temporary file handling**: Temporary files containing sensitive data may not be properly cleaned up or secured.
- [ ] **Missing secure memory clearing**: Sensitive data in memory (passphrases, keys) should be cleared after use.
- [ ] **Weak default permissions**: Some files may be created with overly permissive default permissions.
## Important (Should be fixed before release)
### User Interface Improvements
- [ ] **Add confirmation prompts for destructive operations**: Operations like `keys rm` and vault deletion should require confirmation.
- [ ] **Improve progress indicators**: Long operations (key generation, encryption) should show progress.
- [ ] **Better secret name validation**: Currently allows some characters that may cause issues, needs comprehensive validation.
- [ ] **Add `--help` examples**: Command help should include practical examples for each operation.
### Command Implementation Gaps
- [ ] **`secret keys rm` not fully implemented**: Based on test output, this command may not be working correctly.
- [ ] **`secret key select` not fully implemented**: Key selection functionality appears incomplete.
- [ ] **Missing vault deletion command**: No way to delete vaults that are no longer needed.
- [ ] **No secret deletion command**: Missing `secret rm <secret-name>` functionality.
- [ ] **Missing secret history/versioning**: No way to see previous versions of secrets or restore old values.
### Configuration and Environment
- [ ] **Global configuration not fully implemented**: The `configuration.json` file structure exists but isn't used consistently.
- [ ] **Missing environment variable validation**: Environment variables should be validated for format and security.
- [ ] **No configuration file validation**: JSON configuration files should be validated against schemas.
### PGP Integration Issues
- [ ] **Incomplete PGP unlock key implementation**: The `--keyid` parameter processing may not be fully working.
- [ ] **Missing GPG agent integration**: Should detect and use existing GPG agent when available.
- [ ] **No validation of GPG key existence**: Should verify the specified GPG key exists before creating PGP unlock keys.
### Cross-Platform Issues
- [ ] **macOS Secure Enclave error handling**: Better error messages when biometric authentication fails or isn't available.
- [ ] **Windows path handling**: File paths may not work correctly on Windows systems.
- [ ] **XDG compliance on Linux**: Should respect `XDG_CONFIG_HOME` and other XDG environment variables.
## Trivial (Nice to have)
### Code Quality
- [ ] **Add more comprehensive unit tests**: Current test coverage could be improved, especially for error conditions.
- [ ] **Reduce code duplication**: Several functions have similar patterns that could be refactored.
- [ ] **Improve function documentation**: Many functions lack proper Go documentation comments.
- [ ] **Add static analysis**: Integrate tools like `staticcheck`, `golangci-lint` with more linters.
### Performance Optimizations
- [ ] **Cache unlock key operations**: Avoid re-reading unlock key metadata on every operation.
- [ ] **Optimize file I/O**: Batch file operations where possible to reduce syscalls.
- [ ] **Add connection pooling for HSM operations**: For hardware security module operations.
### User Experience Enhancements
- [ ] **Add shell completion**: Bash/Zsh completion for commands and secret names.
- [ ] **Colored output**: Use colors to improve readability of lists and error messages.
- [ ] **Add `--quiet` flag**: Option to suppress non-essential output.
- [ ] **Smart secret name suggestions**: When a secret name is not found, suggest similar names.
### Additional Features
- [ ] **Secret templates**: Predefined templates for common secret types (database URLs, API keys, etc.).
- [ ] **Bulk operations**: Import/export multiple secrets at once.
- [ ] **Secret sharing**: Secure sharing of secrets between vaults or users.
- [ ] **Audit logging**: Log all secret access and modifications.
- [ ] **Integration tests for hardware features**: Automated testing of Secure Enclave and GPG functionality.
### Documentation
- [ ] **Man pages**: Generate and install proper Unix man pages.
- [ ] **API documentation**: Document the internal API for potential library use.
- [ ] **Migration guide**: Document how to migrate from other secret managers.
- [ ] **Security audit documentation**: Document security assumptions and threat model.
## Architecture Improvements
### Code Structure
- [ ] **Consistent interface implementation**: Ensure all unlock key types properly implement the UnlockKey interface.
- [ ] **Better separation of concerns**: Some functions in CLI do too much and should be split.
- [ ] **Improved error types**: Create specific error types instead of using generic `fmt.Errorf`.
### Testing Infrastructure
- [ ] **Mock filesystem consistency**: Ensure mock filesystem behavior matches real filesystem in all cases.
- [ ] **Integration test isolation**: Tests should not affect each other or the host system.
- [ ] **Performance benchmarks**: Add benchmarks for crypto operations and file I/O.
## Technical Debt
- [ ] **Remove unused code**: Clean up any dead code or unused imports.
- [ ] **Standardize JSON schemas**: Create proper JSON schemas for all configuration files.
- [ ] **Improve error propagation**: Many functions swallow important context in error messages.
- [ ] **Consistent naming conventions**: Some variables and functions use inconsistent naming.
## Development Workflow
- [ ] **Add pre-commit hooks**: Ensure code quality and formatting before commits.
- [ ] **Continuous integration**: Set up CI/CD pipeline with automated testing.
- [ ] **Release automation**: Automate the build and release process.
- [ ] **Dependency management**: Regular updates and security scanning of dependencies.
---
## Priority Assessment
**Critical items** block the 1.0 release and must be fixed for basic functionality and security.
**Important items** should be addressed for a polished user experience but don't block the release.
**Trivial items** are enhancements that can be addressed in future releases.
## Estimated Timeline
- Critical: 2-3 weeks
- Important: 3-4 weeks
- Trivial: Ongoing post-1.0
Total estimated time to 1.0: 5-7 weeks with focused development effort.