vaultik/TODO.md

Implementation TODO

Local Index Database

1. Implement SQLite schema creation
2. Create Index type with all database operations
3. Add transaction support and proper locking
4. Implement file tracking (save, lookup, delete)
5. Implement chunk tracking and deduplication
6. Implement blob tracking and chunk-to-blob mapping
7. Write tests for all index operations

Chunking and Hashing

1. Implement Rabin fingerprint chunker
2. Create streaming chunk processor
3. Implement SHA256 hashing for chunks
4. Add configurable chunk size parameters
5. Write tests for chunking consistency

Compression and Encryption

1. Implement zstd compression wrapper
2. Integrate age encryption library
3. Create Encryptor type for public key encryption
4. Create Decryptor type for private key decryption
5. Implement streaming encrypt/decrypt pipelines
6. Write tests for compression and encryption

Blob Packing

1. Implement BlobWriter with size limits
2. Add chunk accumulation and flushing
3. Create blob hash calculation
4. Implement proper error handling and rollback
5. Write tests for blob packing scenarios

S3 Operations

1. Integrate MinIO client library
2. Implement S3Client wrapper type
3. Add multipart upload support for large blobs
4. Implement retry logic with exponential backoff
5. Add connection pooling and timeout handling
6. Write tests using MinIO container

Backup Command - Basic

1. Implement directory walking with exclusion patterns
2. Add file change detection using index
3. Integrate chunking pipeline for changed files
4. Implement blob upload coordination
5. Add progress reporting to stderr
6. Write integration tests for backup

Snapshot Metadata

1. Implement snapshot metadata extraction from index
2. Create SQLite snapshot database builder
3. Add metadata compression and encryption
4. Implement metadata chunking for large snapshots
5. Add hash calculation and verification
6. Implement metadata upload to S3
7. Write tests for metadata operations

Restore Command

1. Implement snapshot listing and selection
2. Add metadata download and reconstruction
3. Implement hash verification for metadata
4. Create file restoration logic with chunk retrieval
5. Add blob caching for efficiency
6. Implement proper file permissions and mtime restoration
7. Write integration tests for restore

Prune Command

1. Implement latest snapshot detection
2. Add referenced blob extraction from metadata
3. Create S3 blob listing and comparison
4. Implement safe deletion of unreferenced blobs
5. Add dry-run mode for safety
6. Write tests for prune scenarios

Verify Command

1. Implement metadata integrity checking
2. Add blob existence verification
3. Implement quick mode (S3 hash checking)
4. Implement deep mode (download and verify chunks)
5. Add detailed error reporting
6. Write tests for verification

Fetch Command

1. Implement single-file metadata query
2. Add minimal blob downloading for file
3. Create streaming file reconstruction
4. Add support for output redirection
5. Write tests for fetch command

Daemon Mode

1. Implement inotify watcher for Linux
2. Add dirty path tracking in index
3. Create periodic full scan scheduler
4. Implement backup interval enforcement
5. Add proper signal handling and shutdown
6. Write tests for daemon behavior

Cron Mode

1. Implement silent operation mode
2. Add proper exit codes for cron
3. Implement lock file to prevent concurrent runs
4. Add error summary reporting
5. Write tests for cron mode

Finalization

1. Add comprehensive logging throughout
2. Implement proper error wrapping and context
3. Add performance metrics collection
4. Create end-to-end integration tests
5. Write documentation and examples
6. Set up CI/CD pipeline