Add comprehensive godoc comments to all exported types, functions,
and constants throughout the codebase. Create README.md documenting
the project architecture, execution flow, database schema, and
component relationships.
Expand the documentation comment for CLIEntry to provide more context
about what the function does, including its use of the fx dependency
injection framework, signal handling, and blocking behavior.
- Created JSONValidationMiddleware that validates all JSON responses
- Ensures that even on timeout or internal errors, a valid JSON error response is returned
- Applied to all API endpoints including /status.json
- Prevents client-side JSON parse errors when server encounters issues
- Added GetASPeers method to database to fetch all peering relationships
- Updated AS detail handler to fetch and pass peers to template
- Added peers section to AS detail page showing all peer ASNs with their info
- Added peer count to the info cards at the top of the page
- Shows handle, description, and first/last seen dates for each peer
- Remove UUID primary keys from ASNs table, use ASN number as primary key
- Update announcements table to reference ASN numbers directly
- Rename asns.number column to asns.asn for consistency
- Add prefix tracking to PrefixHandler to populate prefixes_v4/v6 tables
- Add UpdatePrefixesBatch method for efficient batch updates
- Update all database methods and models to use new schema
- Fix all references in code to use ASN field instead of Number
- Update test mocks to match new interfaces
The prefixes_v4 and prefixes_v6 tables were never being populated
because GetOrCreatePrefix was not being called anywhere. Since we
already track all prefixes in live_routes_v4 and live_routes_v6,
update stats queries to count distinct prefixes from those tables.
- Create separate tables for IPv4 and IPv6 prefixes in schema.sql
- Update indexes for new prefix tables
- Update getOrCreatePrefix to use appropriate table based on IP version
- Update GetStatsContext to count prefixes from both tables
- Remove ip_version column since it's implicit in the table name
- Update status page JavaScript to reset all fields to '-' on error
- Fix status page to not show 'Connected' when API returns error
- Update remaining database methods to use new live_routes_v4/v6 tables
- Fix GetStatsContext to count routes from both IPv4 and IPv6 tables
- Fix UpsertLiveRoute to insert into correct table based on IP version
- Fix DeleteLiveRoute to determine table from prefix IP version
- Create live_routes_v4 and live_routes_v6 tables
- Update all database methods to use appropriate table
- Add IP version detection in database queries
- Remove filtering by ip_version column for better performance
- Fix route count queries that were timing out
- Update PrefixHandler to include IP version in deletions
- Remove ad-hoc index creation from database.go Initialize method
- Add clear comments to both database.go and schema.sql warning that
ALL schema changes must be made in schema.sql only
- We do not support migrations, schema changes outside schema.sql are forbidden
- Increase PrefixHandler batch size from 20k to 25k (25% increase)
- Add missing index on origin_asn for live_routes table
- This index significantly speeds up AS detail page queries
- Add code to create missing indexes on existing databases
- Use separate variables for IPv4 and IPv6 query results
- Add nil checks before closing rows to prevent panic
- Prevents crash when database queries timeout or fail
- Add context-aware versions of all read operations in the database
- Update handlers to use context from HTTP requests
- Allows database queries to be cancelled when HTTP requests timeout
- Prevents database operations from continuing after client disconnects
- Change GetRandomPrefixesByLength to return unique prefixes instead of all routes
- Use CTE to first select random unique prefixes, then join to get their latest route info
- This ensures each prefix appears only once in the list
- Add compound index on (ip_version, mask_length, prefix) to speed up prefix distribution queries
- This index will significantly improve performance of COUNT(DISTINCT prefix) operations
- Note: Existing databases will need to manually create this index or recreate the database
- Fix GetPrefixDistribution to count unique prefixes using COUNT(DISTINCT prefix) instead of COUNT(*)
- Add /prefixlength/<length> route showing random sample of 500 prefixes
- Make prefix counts on status page clickable links to prefix length pages
- Add GetRandomPrefixesByLength database method
- Create prefix_length.html template with sortable table
- Show prefix age and origin AS with descriptions
- Reduce cache size from 8GB to 512MB (still plenty for 2.4GB DB)
- Reduce mmap_size from 10GB to 256MB (reasonable default)
- Use default page size (4KB) instead of 8KB
- Use default WAL checkpoint interval (1000 pages)
- Remove redundant pragmas (threads, cache_spill, read_uncommitted)
- Clean up connection string to only use _txlock parameter
- Keep synchronous=OFF for performance (since we have mutex protection)
- Add internal mutex to Database struct with lock/unlock wrappers
- Add debug logging for lock acquisition and release with timing
- Wrap all write operations with database mutex
- Use _txlock=immediate in SQLite connection string
This works around apparent issues with SQLite's internal locking
not properly respecting busy_timeout in production environment.
- Add _txlock=immediate to SQLite connection string
- This prevents deadlocks by acquiring write locks at transaction start
- Multiple concurrent writers now queue properly instead of failing instantly
- Resolves 'database table is locked' errors in production
- Fix Ctrl-C shutdown by using fx.Shutdowner instead of just canceling context
- Pass context from fx lifecycle to rw.Run() for proper cancellation
- Adjust WAL settings: checkpoint at 50MB, max size 100MB
- Reduce busy timeout from 30s to 2s to fail fast on lock contention
This should fix the issue where Ctrl-C doesn't cause shutdown and improve
database responsiveness under heavy load.
- Add VACUUM on startup to defragment database
- Increase cache size from 256MB to 2GB for better performance
- Increase mmap_size from 256MB to 512MB
- Add PRAGMA analysis_limit=0 to disable automatic ANALYZE
- Remove PRAGMA optimize which could trigger slow ANALYZE
These changes should dramatically improve query performance and prevent
the 5+ second query times seen in production.
- Add debug logging for goroutines and memory usage (enabled via DEBUG=routewatch)
- Increase SQLite connection pool from 1 to 10 connections for better concurrency
- Optimize SQLite pragmas for balanced performance and safety
- Add proper shutdown handling for peering handler
- Define constants to avoid magic numbers in code
Co-Authored-By: Claude <noreply@anthropic.com>
- Fix GetASDetails to properly handle timestamp from MAX(last_updated)
- Parse timestamp string from SQLite aggregate function result
- Add natural sorting of prefixes by IP address in AS detail view
- Sort IPv4 and IPv6 prefixes separately by network address
- Remove SQL ORDER BY since we're sorting in Go
- This fixes the issue where AS detail pages showed no prefixes
- Update GetASDetails query to GROUP BY prefix instead of using DISTINCT
- Use MAX(last_updated) to get the most recent update time for each prefix
- This prevents duplicate prefixes from appearing when announced by multiple peers
- Maintains the same prefix count and ordering
- Move all handler functions to handlers.go
- Move setupRoutes to routes.go
- Clean up server.go to only contain core server logic
- Add missing GetASDetails and GetPrefixDetails to mockStore for tests
- Fix linter errors (magic numbers, unused parameters, blank lines)
- Add last_updated timestamp and age fields to ASInfo
- Include route's last_updated time from live_routes table
- Calculate and display age as human-readable duration
- Update both IPv4 and IPv6 queries to fetch timestamp
- Fix error handling to return 400 for invalid IPs
- Add handleIPLookup handler that uses GetASInfoForIP
- Create writeJSONError and writeJSONSuccess helper functions
- Refactor all JSON error responses to use the helpers
- Add GetASInfoForIP to Store interface
- Add mock implementation for tests
- Fix all linter warnings
- Add v4_ip_start and v4_ip_end columns to live_routes table
- Calculate IPv4 CIDR ranges as 32-bit integers for fast lookups
- Update PrefixHandler to populate IPv4 range fields
- Add GetASInfoForIP method with optimized IPv4 queries
- Add comprehensive tests for IP conversion functions
- Remove routingtable package entirely as database handles all routing data
- Remove snapshotter package as database contains all information
- Rename 'Connection Status' box to 'RouteWatch' and add Go version, goroutines, memory usage
- Move IPv4/IPv6 prefix counts from Database Statistics to Routing Table box
- Add Peers count to Database Statistics box
- Add go-humanize dependency for memory formatting
- Update server to include new metrics in API responses
- Remove RoutingTableHandler as PrefixHandler maintains live_routes table
- Update server to get route counts from database instead of in-memory routing table
- Add GetLiveRouteCounts method to database for IPv4/IPv6 route counts
- Use metrics tracker in PrefixHandler for route update rates
- Remove snapshotter entirely as database contains all information
- Update tests to work without routing table
- Increase PrefixHandler queue size to 500k and batch size to 25k
- Set SQLite PRAGMA synchronous=OFF for faster writes (trades durability)
- Increase SQLite cache to 1GB and mmap to 512MB
- Increase WAL checkpoint interval to 10000 pages
- Set page size to 8KB for better performance
- Increase busy timeout to 30 seconds
- Keep single connection to avoid SQLite locking issues
- Rename asn_peerings table to peerings
- Change columns from from_asn_id/to_asn_id to as_a/as_b (integers)
- Remove foreign key constraints to asns table
- Update RecordPeering to use AS numbers directly
- Add validation in RecordPeering to ensure:
- Both AS numbers are > 0
- AS numbers are different
- as_a is always lower than as_b (normalized)
- Update PeeringHandler to no longer need ASN cache
- Simplify the code by removing unnecessary ASN lookups
- Added new live_routes table with mask_length column for tracking CIDR prefix lengths
- Updated PrefixHandler to maintain live routing table with additions and deletions
- Added route expiration functionality (5 minute timeout) to in-memory routing table
- Added prefix distribution stats showing count of prefixes by mask length
- Added IPv4/IPv6 prefix distribution cards to status page
- Updated database interface with UpsertLiveRoute, DeleteLiveRoute, and GetPrefixDistribution
- Set all handler queue depths to 50000 for consistency
- Doubled DBHandler batch size to 32000 for better throughput
- Fixed withdrawal handling to delete routes when origin ASN is available
- Create internal/logger package with Logger wrapper around slog
- Logger automatically adds source file, line number, and function name to all log entries
- Use golang.org/x/term to properly detect if stdout is a terminal
- Replace all slog.Logger usage with logger.Logger throughout the codebase
- Remove verbose logging from database GetStats() method
- Update all constructors and dependencies to use the new logger
- Remove old database Config struct and related functions
- Update database.New() to accept config.Config parameter
- Update routingtable.New() to accept config.Config parameter
- Update snapshotter.New() to accept config.Config parameter
- Simplify fx module providers in app.go
- Fix truthiness check for environment variables
- Handle empty state directory gracefully in routing table and snapshotter
- Update all tests to use empty state directory for testing
- Add database file size tracking to Stats struct and GetStats()
- Move routing table metrics to separate 'Routing Table' status box
- Add IPv4/IPv6 updates per second to routing table metrics
- Database box now shows: ASNs, prefixes, peerings, and database size
- Routing table box shows: live routes, IPv4/IPv6 counts, and update rates
- Add handle and description columns to asns table
- Look up ASN info using asinfo package when creating new ASNs
- Remove noisy debug logging for individual route updates
- Add IPv4/IPv6 route counters and update rate tracking
- Log routing table statistics every 15 seconds with IPv4/IPv6 breakdown
- Track updates per second for both IPv4 and IPv6 routes separately
- Remove live_routes table from SQL schema and all related indexes
- Create new internal/routingtable package with thread-safe RoutingTable
- Implement RouteKey-based indexing with secondary indexes for efficient lookups
- Add RoutingTableHandler to manage in-memory routes separately from database
- Update DatabaseHandler to only handle persistent database operations
- Wire up RoutingTable through fx dependency injection
- Update server to get live route count from routing table instead of database
- Remove LiveRoutes field from database.Stats struct
- Update tests to work with new architecture
- Replace modernc.org/sqlite with github.com/mattn/go-sqlite3
- Update connection string for go-sqlite3 syntax
- Keep all performance optimizations and pragmas
The CGO driver may provide better performance for write-heavy
workloads compared to the pure Go implementation.
- Set concurrent handlers back to 100 (from 200)
- Set slow query threshold to 50ms (from 10ms)
These values provide a good balance between throughput and
system resource usage.
Queries in the 50-70ms range are acceptable for now given SQLite's
write serialization constraints. Setting threshold to 100ms to focus
on truly problematic queries.
- Use SQLite UPSERT for UpdateLiveRoute to eliminate SELECT+UPDATE/INSERT pattern
- Add connection string optimizations (synchronous=NORMAL, cache_size)
- Add WAL checkpoint configuration for better write performance
- Add index on live_routes(id) for UPDATE operations
- Set WAL autocheckpoint to 1000 pages
These changes should reduce write amplification and improve overall
throughput by:
1. Reducing from 2 queries to 1 for route updates
2. Better WAL checkpoint management
3. More efficient UPDATE operations with dedicated index
The 10ms threshold was too noisy - queries in the 10-20ms range are
actually performing well after the index optimizations. Setting the
threshold to 50ms will help identify truly problematic queries.
- Add covering index for SELECT id queries (avoids table lookups)
- Add stats index for COUNT queries on live_routes
- Add index on asns.number for ASN lookups
- Add index on bgp_peers.peer_ip for peer lookups
These indexes should further reduce query times, especially:
- The covering index includes the id column, eliminating the need
to access the table after index lookup
- The stats index helps with COUNT(*) queries
- The asns.number index speeds up ASN lookups in transactions
- Move schema from database.go to schema.sql
- Use go:embed to include schema at compile time
- Add 'out' debug file to .gitignore
This makes the schema more maintainable and easier to review.
