Move TODO list to separate file and update verify command
- Extract implementation TODO from DESIGN.md to TODO.md - Remove completed Phase 1 tasks - Add --quick option to verify command for S3 hash checking - Update documentation to reflect deep verification as default
This commit is contained in:
parent
bcbc186286
commit
9de439a0a4
121
DESIGN.md
121
DESIGN.md
@ -354,124 +354,3 @@ func EncryptAndUploadMetadata(path string, cfg *Config, snapshotID string) error
|
|||||||
func RunPrune(bucket, prefix, privateKey string) error
|
func RunPrune(bucket, prefix, privateKey string) error
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Implementation TODO
|
|
||||||
|
|
||||||
### Core Infrastructure
|
|
||||||
1. Set up Go module and project structure
|
|
||||||
1. Create Makefile with test, fmt, and lint targets
|
|
||||||
1. Set up cobra CLI skeleton with all commands
|
|
||||||
1. Implement config loading and validation from YAML
|
|
||||||
1. Create data structures for FileInfo, ChunkInfo, BlobInfo, etc.
|
|
||||||
|
|
||||||
### Local Index Database
|
|
||||||
1. Implement SQLite schema creation and migrations
|
|
||||||
1. Create Index type with all database operations
|
|
||||||
1. Add transaction support and proper locking
|
|
||||||
1. Implement file tracking (save, lookup, delete)
|
|
||||||
1. Implement chunk tracking and deduplication
|
|
||||||
1. Implement blob tracking and chunk-to-blob mapping
|
|
||||||
1. Write tests for all index operations
|
|
||||||
|
|
||||||
### Chunking and Hashing
|
|
||||||
1. Implement Rabin fingerprint chunker
|
|
||||||
1. Create streaming chunk processor
|
|
||||||
1. Implement SHA256 hashing for chunks
|
|
||||||
1. Add configurable chunk size parameters
|
|
||||||
1. Write tests for chunking consistency
|
|
||||||
|
|
||||||
### Compression and Encryption
|
|
||||||
1. Implement zstd compression wrapper
|
|
||||||
1. Integrate age encryption library
|
|
||||||
1. Create Encryptor type for public key encryption
|
|
||||||
1. Create Decryptor type for private key decryption
|
|
||||||
1. Implement streaming encrypt/decrypt pipelines
|
|
||||||
1. Write tests for compression and encryption
|
|
||||||
|
|
||||||
### Blob Packing
|
|
||||||
1. Implement BlobWriter with size limits
|
|
||||||
1. Add chunk accumulation and flushing
|
|
||||||
1. Create blob hash calculation
|
|
||||||
1. Implement proper error handling and rollback
|
|
||||||
1. Write tests for blob packing scenarios
|
|
||||||
|
|
||||||
### S3 Operations
|
|
||||||
1. Integrate MinIO client library
|
|
||||||
1. Implement S3Client wrapper type
|
|
||||||
1. Add multipart upload support for large blobs
|
|
||||||
1. Implement retry logic with exponential backoff
|
|
||||||
1. Add connection pooling and timeout handling
|
|
||||||
1. Write tests using MinIO container
|
|
||||||
|
|
||||||
### Backup Command - Basic
|
|
||||||
1. Implement directory walking with exclusion patterns
|
|
||||||
1. Add file change detection using index
|
|
||||||
1. Integrate chunking pipeline for changed files
|
|
||||||
1. Implement blob upload coordination
|
|
||||||
1. Add progress reporting to stderr
|
|
||||||
1. Write integration tests for backup
|
|
||||||
|
|
||||||
### Snapshot Metadata
|
|
||||||
1. Implement snapshot metadata extraction from index
|
|
||||||
1. Create SQLite snapshot database builder
|
|
||||||
1. Add metadata compression and encryption
|
|
||||||
1. Implement metadata chunking for large snapshots
|
|
||||||
1. Add hash calculation and verification
|
|
||||||
1. Implement metadata upload to S3
|
|
||||||
1. Write tests for metadata operations
|
|
||||||
|
|
||||||
### Restore Command
|
|
||||||
1. Implement snapshot listing and selection
|
|
||||||
1. Add metadata download and reconstruction
|
|
||||||
1. Implement hash verification for metadata
|
|
||||||
1. Create file restoration logic with chunk retrieval
|
|
||||||
1. Add blob caching for efficiency
|
|
||||||
1. Implement proper file permissions and mtime restoration
|
|
||||||
1. Write integration tests for restore
|
|
||||||
|
|
||||||
### Prune Command
|
|
||||||
1. Implement latest snapshot detection
|
|
||||||
1. Add referenced blob extraction from metadata
|
|
||||||
1. Create S3 blob listing and comparison
|
|
||||||
1. Implement safe deletion of unreferenced blobs
|
|
||||||
1. Add dry-run mode for safety
|
|
||||||
1. Write tests for prune scenarios
|
|
||||||
|
|
||||||
### Verify Command
|
|
||||||
1. Implement metadata integrity checking
|
|
||||||
1. Add blob existence verification
|
|
||||||
1. Implement quick mode (S3 hash checking)
|
|
||||||
1. Implement deep mode (download and verify chunks)
|
|
||||||
1. Add detailed error reporting
|
|
||||||
1. Write tests for verification
|
|
||||||
|
|
||||||
### Fetch Command
|
|
||||||
1. Implement single-file metadata query
|
|
||||||
1. Add minimal blob downloading for file
|
|
||||||
1. Create streaming file reconstruction
|
|
||||||
1. Add support for output redirection
|
|
||||||
1. Write tests for fetch command
|
|
||||||
|
|
||||||
### Daemon Mode
|
|
||||||
1. Implement inotify watcher for Linux
|
|
||||||
1. Add dirty path tracking in index
|
|
||||||
1. Create periodic full scan scheduler
|
|
||||||
1. Implement backup interval enforcement
|
|
||||||
1. Add proper signal handling and shutdown
|
|
||||||
1. Write tests for daemon behavior
|
|
||||||
|
|
||||||
### Cron Mode
|
|
||||||
1. Implement silent operation mode
|
|
||||||
1. Add proper exit codes for cron
|
|
||||||
1. Implement lock file to prevent concurrent runs
|
|
||||||
1. Add error summary reporting
|
|
||||||
1. Write tests for cron mode
|
|
||||||
|
|
||||||
### Finalization
|
|
||||||
1. Add comprehensive logging throughout
|
|
||||||
1. Implement proper error wrapping and context
|
|
||||||
1. Add performance metrics collection
|
|
||||||
1. Create end-to-end integration tests
|
|
||||||
1. Write documentation and examples
|
|
||||||
1. Set up CI/CD pipeline
|
|
||||||
|
112
TODO.md
Normal file
112
TODO.md
Normal file
@ -0,0 +1,112 @@
|
|||||||
|
# Implementation TODO
|
||||||
|
|
||||||
|
## Local Index Database
|
||||||
|
1. Implement SQLite schema creation
|
||||||
|
1. Create Index type with all database operations
|
||||||
|
1. Add transaction support and proper locking
|
||||||
|
1. Implement file tracking (save, lookup, delete)
|
||||||
|
1. Implement chunk tracking and deduplication
|
||||||
|
1. Implement blob tracking and chunk-to-blob mapping
|
||||||
|
1. Write tests for all index operations
|
||||||
|
|
||||||
|
## Chunking and Hashing
|
||||||
|
1. Implement Rabin fingerprint chunker
|
||||||
|
1. Create streaming chunk processor
|
||||||
|
1. Implement SHA256 hashing for chunks
|
||||||
|
1. Add configurable chunk size parameters
|
||||||
|
1. Write tests for chunking consistency
|
||||||
|
|
||||||
|
## Compression and Encryption
|
||||||
|
1. Implement zstd compression wrapper
|
||||||
|
1. Integrate age encryption library
|
||||||
|
1. Create Encryptor type for public key encryption
|
||||||
|
1. Create Decryptor type for private key decryption
|
||||||
|
1. Implement streaming encrypt/decrypt pipelines
|
||||||
|
1. Write tests for compression and encryption
|
||||||
|
|
||||||
|
## Blob Packing
|
||||||
|
1. Implement BlobWriter with size limits
|
||||||
|
1. Add chunk accumulation and flushing
|
||||||
|
1. Create blob hash calculation
|
||||||
|
1. Implement proper error handling and rollback
|
||||||
|
1. Write tests for blob packing scenarios
|
||||||
|
|
||||||
|
## S3 Operations
|
||||||
|
1. Integrate MinIO client library
|
||||||
|
1. Implement S3Client wrapper type
|
||||||
|
1. Add multipart upload support for large blobs
|
||||||
|
1. Implement retry logic with exponential backoff
|
||||||
|
1. Add connection pooling and timeout handling
|
||||||
|
1. Write tests using MinIO container
|
||||||
|
|
||||||
|
## Backup Command - Basic
|
||||||
|
1. Implement directory walking with exclusion patterns
|
||||||
|
1. Add file change detection using index
|
||||||
|
1. Integrate chunking pipeline for changed files
|
||||||
|
1. Implement blob upload coordination
|
||||||
|
1. Add progress reporting to stderr
|
||||||
|
1. Write integration tests for backup
|
||||||
|
|
||||||
|
## Snapshot Metadata
|
||||||
|
1. Implement snapshot metadata extraction from index
|
||||||
|
1. Create SQLite snapshot database builder
|
||||||
|
1. Add metadata compression and encryption
|
||||||
|
1. Implement metadata chunking for large snapshots
|
||||||
|
1. Add hash calculation and verification
|
||||||
|
1. Implement metadata upload to S3
|
||||||
|
1. Write tests for metadata operations
|
||||||
|
|
||||||
|
## Restore Command
|
||||||
|
1. Implement snapshot listing and selection
|
||||||
|
1. Add metadata download and reconstruction
|
||||||
|
1. Implement hash verification for metadata
|
||||||
|
1. Create file restoration logic with chunk retrieval
|
||||||
|
1. Add blob caching for efficiency
|
||||||
|
1. Implement proper file permissions and mtime restoration
|
||||||
|
1. Write integration tests for restore
|
||||||
|
|
||||||
|
## Prune Command
|
||||||
|
1. Implement latest snapshot detection
|
||||||
|
1. Add referenced blob extraction from metadata
|
||||||
|
1. Create S3 blob listing and comparison
|
||||||
|
1. Implement safe deletion of unreferenced blobs
|
||||||
|
1. Add dry-run mode for safety
|
||||||
|
1. Write tests for prune scenarios
|
||||||
|
|
||||||
|
## Verify Command
|
||||||
|
1. Implement metadata integrity checking
|
||||||
|
1. Add blob existence verification
|
||||||
|
1. Implement quick mode (S3 hash checking)
|
||||||
|
1. Implement deep mode (download and verify chunks)
|
||||||
|
1. Add detailed error reporting
|
||||||
|
1. Write tests for verification
|
||||||
|
|
||||||
|
## Fetch Command
|
||||||
|
1. Implement single-file metadata query
|
||||||
|
1. Add minimal blob downloading for file
|
||||||
|
1. Create streaming file reconstruction
|
||||||
|
1. Add support for output redirection
|
||||||
|
1. Write tests for fetch command
|
||||||
|
|
||||||
|
## Daemon Mode
|
||||||
|
1. Implement inotify watcher for Linux
|
||||||
|
1. Add dirty path tracking in index
|
||||||
|
1. Create periodic full scan scheduler
|
||||||
|
1. Implement backup interval enforcement
|
||||||
|
1. Add proper signal handling and shutdown
|
||||||
|
1. Write tests for daemon behavior
|
||||||
|
|
||||||
|
## Cron Mode
|
||||||
|
1. Implement silent operation mode
|
||||||
|
1. Add proper exit codes for cron
|
||||||
|
1. Implement lock file to prevent concurrent runs
|
||||||
|
1. Add error summary reporting
|
||||||
|
1. Write tests for cron mode
|
||||||
|
|
||||||
|
## Finalization
|
||||||
|
1. Add comprehensive logging throughout
|
||||||
|
1. Implement proper error wrapping and context
|
||||||
|
1. Add performance metrics collection
|
||||||
|
1. Create end-to-end integration tests
|
||||||
|
1. Write documentation and examples
|
||||||
|
1. Set up CI/CD pipeline
|
Loading…
Reference in New Issue
Block a user