diff --git a/DESIGN.md b/DESIGN.md index e3e630b..1e4afe3 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -354,124 +354,3 @@ func EncryptAndUploadMetadata(path string, cfg *Config, snapshotID string) error func RunPrune(bucket, prefix, privateKey string) error ``` ---- - -## Implementation TODO - -### Core Infrastructure -1. Set up Go module and project structure -1. Create Makefile with test, fmt, and lint targets -1. Set up cobra CLI skeleton with all commands -1. Implement config loading and validation from YAML -1. Create data structures for FileInfo, ChunkInfo, BlobInfo, etc. - -### Local Index Database -1. Implement SQLite schema creation and migrations -1. Create Index type with all database operations -1. Add transaction support and proper locking -1. Implement file tracking (save, lookup, delete) -1. Implement chunk tracking and deduplication -1. Implement blob tracking and chunk-to-blob mapping -1. Write tests for all index operations - -### Chunking and Hashing -1. Implement Rabin fingerprint chunker -1. Create streaming chunk processor -1. Implement SHA256 hashing for chunks -1. Add configurable chunk size parameters -1. Write tests for chunking consistency - -### Compression and Encryption -1. Implement zstd compression wrapper -1. Integrate age encryption library -1. Create Encryptor type for public key encryption -1. Create Decryptor type for private key decryption -1. Implement streaming encrypt/decrypt pipelines -1. Write tests for compression and encryption - -### Blob Packing -1. Implement BlobWriter with size limits -1. Add chunk accumulation and flushing -1. Create blob hash calculation -1. Implement proper error handling and rollback -1. Write tests for blob packing scenarios - -### S3 Operations -1. Integrate MinIO client library -1. Implement S3Client wrapper type -1. Add multipart upload support for large blobs -1. Implement retry logic with exponential backoff -1. Add connection pooling and timeout handling -1. Write tests using MinIO container - -### Backup Command - Basic -1. Implement directory walking with exclusion patterns -1. Add file change detection using index -1. Integrate chunking pipeline for changed files -1. Implement blob upload coordination -1. Add progress reporting to stderr -1. Write integration tests for backup - -### Snapshot Metadata -1. Implement snapshot metadata extraction from index -1. Create SQLite snapshot database builder -1. Add metadata compression and encryption -1. Implement metadata chunking for large snapshots -1. Add hash calculation and verification -1. Implement metadata upload to S3 -1. Write tests for metadata operations - -### Restore Command -1. Implement snapshot listing and selection -1. Add metadata download and reconstruction -1. Implement hash verification for metadata -1. Create file restoration logic with chunk retrieval -1. Add blob caching for efficiency -1. Implement proper file permissions and mtime restoration -1. Write integration tests for restore - -### Prune Command -1. Implement latest snapshot detection -1. Add referenced blob extraction from metadata -1. Create S3 blob listing and comparison -1. Implement safe deletion of unreferenced blobs -1. Add dry-run mode for safety -1. Write tests for prune scenarios - -### Verify Command -1. Implement metadata integrity checking -1. Add blob existence verification -1. Implement quick mode (S3 hash checking) -1. Implement deep mode (download and verify chunks) -1. Add detailed error reporting -1. Write tests for verification - -### Fetch Command -1. Implement single-file metadata query -1. Add minimal blob downloading for file -1. Create streaming file reconstruction -1. Add support for output redirection -1. Write tests for fetch command - -### Daemon Mode -1. Implement inotify watcher for Linux -1. Add dirty path tracking in index -1. Create periodic full scan scheduler -1. Implement backup interval enforcement -1. Add proper signal handling and shutdown -1. Write tests for daemon behavior - -### Cron Mode -1. Implement silent operation mode -1. Add proper exit codes for cron -1. Implement lock file to prevent concurrent runs -1. Add error summary reporting -1. Write tests for cron mode - -### Finalization -1. Add comprehensive logging throughout -1. Implement proper error wrapping and context -1. Add performance metrics collection -1. Create end-to-end integration tests -1. Write documentation and examples -1. Set up CI/CD pipeline diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..57ffa57 --- /dev/null +++ b/TODO.md @@ -0,0 +1,112 @@ +# Implementation TODO + +## Local Index Database +1. Implement SQLite schema creation +1. Create Index type with all database operations +1. Add transaction support and proper locking +1. Implement file tracking (save, lookup, delete) +1. Implement chunk tracking and deduplication +1. Implement blob tracking and chunk-to-blob mapping +1. Write tests for all index operations + +## Chunking and Hashing +1. Implement Rabin fingerprint chunker +1. Create streaming chunk processor +1. Implement SHA256 hashing for chunks +1. Add configurable chunk size parameters +1. Write tests for chunking consistency + +## Compression and Encryption +1. Implement zstd compression wrapper +1. Integrate age encryption library +1. Create Encryptor type for public key encryption +1. Create Decryptor type for private key decryption +1. Implement streaming encrypt/decrypt pipelines +1. Write tests for compression and encryption + +## Blob Packing +1. Implement BlobWriter with size limits +1. Add chunk accumulation and flushing +1. Create blob hash calculation +1. Implement proper error handling and rollback +1. Write tests for blob packing scenarios + +## S3 Operations +1. Integrate MinIO client library +1. Implement S3Client wrapper type +1. Add multipart upload support for large blobs +1. Implement retry logic with exponential backoff +1. Add connection pooling and timeout handling +1. Write tests using MinIO container + +## Backup Command - Basic +1. Implement directory walking with exclusion patterns +1. Add file change detection using index +1. Integrate chunking pipeline for changed files +1. Implement blob upload coordination +1. Add progress reporting to stderr +1. Write integration tests for backup + +## Snapshot Metadata +1. Implement snapshot metadata extraction from index +1. Create SQLite snapshot database builder +1. Add metadata compression and encryption +1. Implement metadata chunking for large snapshots +1. Add hash calculation and verification +1. Implement metadata upload to S3 +1. Write tests for metadata operations + +## Restore Command +1. Implement snapshot listing and selection +1. Add metadata download and reconstruction +1. Implement hash verification for metadata +1. Create file restoration logic with chunk retrieval +1. Add blob caching for efficiency +1. Implement proper file permissions and mtime restoration +1. Write integration tests for restore + +## Prune Command +1. Implement latest snapshot detection +1. Add referenced blob extraction from metadata +1. Create S3 blob listing and comparison +1. Implement safe deletion of unreferenced blobs +1. Add dry-run mode for safety +1. Write tests for prune scenarios + +## Verify Command +1. Implement metadata integrity checking +1. Add blob existence verification +1. Implement quick mode (S3 hash checking) +1. Implement deep mode (download and verify chunks) +1. Add detailed error reporting +1. Write tests for verification + +## Fetch Command +1. Implement single-file metadata query +1. Add minimal blob downloading for file +1. Create streaming file reconstruction +1. Add support for output redirection +1. Write tests for fetch command + +## Daemon Mode +1. Implement inotify watcher for Linux +1. Add dirty path tracking in index +1. Create periodic full scan scheduler +1. Implement backup interval enforcement +1. Add proper signal handling and shutdown +1. Write tests for daemon behavior + +## Cron Mode +1. Implement silent operation mode +1. Add proper exit codes for cron +1. Implement lock file to prevent concurrent runs +1. Add error summary reporting +1. Write tests for cron mode + +## Finalization +1. Add comprehensive logging throughout +1. Implement proper error wrapping and context +1. Add performance metrics collection +1. Create end-to-end integration tests +1. Write documentation and examples +1. Set up CI/CD pipeline \ No newline at end of file