Ensure that numeric conversions are lossless by verifying the converted value matches the original string when converted back. This prevents: - Loss of leading zeros (0123 stays as string) - Loss of trailing zeros in floats (1.50 stays as string) - Loss of plus signs (+123 stays as string) - Changes in notation (1e10 stays as string) Only convert to numeric types when the string representation would be preserved exactly. This maintains data integrity while still providing convenient type conversions for clean numeric values. Added comprehensive tests to verify the safety check behavior. |
||
---|---|---|
cmd/smartconfig | ||
test | ||
.gitignore | ||
AGENTS.md | ||
cli_missing_env_test.go | ||
cli_test.go | ||
DESIGN.md | ||
go.mod | ||
go.sum | ||
interpolate.go | ||
LICENSE | ||
Makefile | ||
README.md | ||
resolver_awssm.go | ||
resolver_azure.go | ||
resolver_consul.go | ||
resolver_env.go | ||
resolver_etcd.go | ||
resolver_exec.go | ||
resolver_file.go | ||
resolver_gcpsm.go | ||
resolver_json.go | ||
resolver_k8s.go | ||
resolver_vault.go | ||
resolver_yaml.go | ||
smartconfig_test.go | ||
smartconfig.go | ||
typed_interpolation_test.go | ||
yaml_syntax_test.go |
smartconfig
A Go library for YAML configuration files with powerful variable interpolation from multiple sources including environment variables, files, cloud secret managers, and more.
Table of Contents
- Key Features
- Installation
- Quick Start
- Type-Preserving Interpolation
- Supported Resolvers
- API Reference
- Advanced Features
- CLI Tool
- Security Considerations
- Comparison with Other Libraries
- Requirements
- Contributing
- License
Key Features
- Type-Preserving Interpolation: Standalone interpolations preserve their types (numbers, booleans, strings)
- Multiple Data Sources: Environment variables, files, cloud secrets (AWS, GCP, Azure), Vault, Consul, K8S, and more
- Nested Interpolation: Support for complex variable references like
${ENV:PREFIX_${ENV:SUFFIX}}
- JSON5 Support: JSON resolver supports comments and trailing commas
- Environment Injection: Automatically export config values as environment variables
- Extensible: Add custom resolvers for your own data sources
- CLI Tool: Command-line tool for processing YAML files with interpolation
Installation
go get git.eeqj.de/sneak/smartconfig
Quick Start
Basic Example
# config.yaml
# Showcase the power of multiple resolver types
app_name: ${ENV:APP_NAME}
version: ${FILE:/app/VERSION} # Read from file
port: ${ENV:PORT} # Number if PORT="8080"
debug_mode: ${ENV:DEBUG_MODE} # Boolean if "true"/"false"
# External service configuration from JSON config file
services: ${JSON:/etc/config/services.json:production} # Load entire object
# API credentials from cloud secret managers
api_keys:
stripe: ${GCPSM:projects/myproject/secrets/stripe-api-key}
sendgrid: ${AWSSM:prod/sendgrid-api-key}
twilio: ${VAULT:secret/data/api:twilio_key}
datadog: ${ENV:DD_API_KEY} # Fallback to env for local dev
# Database configuration with mixed sources
database:
host: ${CONSUL:service/postgres/address}
port: ${CONSUL:service/postgres/port}
name: ${ENV:DB_NAME}
username: ${ENV:DB_USER}
password: ${GCPSM:projects/myproject/secrets/db-password}
# SSL configuration from multiple sources
ssl:
enabled: ${ENV:DB_SSL_ENABLED}
ca_cert: ${FILE:/etc/ssl/db-ca.crt}
client_cert: ${K8SS:default/db-certs:client.crt}
# Feature flags from JSON file
features: ${JSON:/etc/config/features.json:${ENV:ENVIRONMENT}}
# Server configuration
server:
listen: "0.0.0.0:${ENV:PORT}"
workers: ${EXEC:nproc} # Dynamic based on CPU cores
hostname: ${EXEC:hostname -f}
env:
# Export these as environment variables for child processes
DATABASE_URL: "postgres://${ENV:DB_USER}:${GCPSM:projects/myproject/secrets/db-password}@${CONSUL:service/postgres/address}:5432/${ENV:DB_NAME}"
NEW_RELIC_LICENSE: ${AWSSM:monitoring/newrelic-license}
Go Code
package main
import (
"fmt"
"log"
"os"
"git.eeqj.de/sneak/smartconfig"
)
func main() {
// Load from /etc/myapp/config.yml
config, err := smartconfig.NewFromAppName("myapp")
if err != nil {
log.Fatal(err)
}
// Access typed values
appName, _ := config.GetString("app_name")
port, _ := config.GetInt("port")
debugMode, _ := config.GetBool("debug_mode")
fmt.Printf("Starting %s on port %d (debug: %v)\n", appName, port, debugMode)
// Access nested values
data := config.Data()
// API keys configuration
if apiKeys, ok := data["api_keys"].(map[string]interface{}); ok {
if stripe, ok := apiKeys["stripe"].(string); ok {
fmt.Printf("Stripe API key loaded: %s...\n", stripe[:8])
}
}
// Database configuration
if db, ok := data["database"].(map[string]interface{}); ok {
host, _ := db["host"].(string)
port, _ := db["port"].(int)
fmt.Printf("Database: %s:%d\n", host, port)
}
// Check loaded services from JSON file
if services, ok := data["services"].(map[string]interface{}); ok {
fmt.Printf("Loaded %d services from JSON config\n", len(services))
}
// Environment variables are now available
fmt.Printf("DATABASE_URL env var: %s\n", os.Getenv("DATABASE_URL"))
}
Example JSON files referenced above:
// /etc/config/services.json
{
"production": {
"api": {
"endpoint": "https://api.example.com",
"timeout": 30,
"retries": 3
},
"cache": {
"provider": "redis",
"ttl": 3600
}
},
"staging": {
"api": {
"endpoint": "https://api-staging.example.com",
"timeout": 60,
"retries": 5
}
}
}
// /etc/config/features.json
{
"production": {
"new_ui": false,
"rate_limiting": true,
"analytics": true
},
"staging": {
"new_ui": true,
"rate_limiting": true,
"analytics": false
}
}
Type-Preserving Interpolation
smartconfig parses YAML first, then performs interpolation. This allows proper type preservation:
How It Works
# Standalone interpolations preserve types
port: ${ENV:PORT} # If PORT="8080", becomes integer 8080
enabled: ${ENV:ENABLED} # If ENABLED="true", becomes boolean true
timeout: ${ENV:TIMEOUT} # If TIMEOUT="30.5", becomes float 30.5
# Mixed content always returns strings
message: "Hello ${ENV:USER}!" # Always a string
port_label: "Port: ${ENV:PORT}" # Always a string
debug_flag: "debug-${ENV:DEBUG}" # Always a string
# Force string by adding any prefix/suffix
port_string: "${ENV:PORT}-suffix" # Forces string output
bool_string: "prefix-${ENV:ENABLED}" # Forces string output
Type Conversion Rules
For standalone interpolations, smartconfig automatically converts:
"true"
→true
(boolean)"false"
→false
(boolean)- Numeric strings → numbers (int or float64) with safety checks
- Everything else → string
Safety Check: Numbers are only converted if the conversion is lossless. This means:
"123"
→123
(converts to int)"0123"
→"0123"
(stays string - leading zeros)"123.45"
→123.45
(converts to float)"1.50"
→"1.50"
(stays string - trailing zeros would be lost)"+123"
→"+123"
(stays string - plus sign would be lost)"1e10"
→"1e10"
(stays string - notation would change)
Supported Resolvers
Local Resolvers
ENV - Environment Variables
# Basic usage
api_key: ${ENV:API_KEY}
# With nested interpolation
database: ${ENV:DB_${ENV:ENVIRONMENT}}
FILE - Read File Contents
# Read entire file (trimmed)
ssl_cert: ${FILE:/etc/ssl/cert.pem}
machine_id: ${FILE:/etc/machine-id}
# Read system files
cpu_temp: ${FILE:/sys/class/thermal/thermal_zone0/temp}
EXEC - Execute Shell Commands
# Simple commands
hostname: ${EXEC:hostname -f}
timestamp: ${EXEC:date +%s}
git_hash: ${EXEC:git rev-parse HEAD}
# Complex commands with pipes
users_count: ${EXEC:who | wc -l}
docker_running: ${EXEC:docker ps -q | wc -l}
JSON - Read from JSON Files (with JSON5 support)
# Supports gjson path syntax
api_endpoint: ${JSON:/etc/config.json:services.api.endpoint}
first_server: ${JSON:/etc/servers.json:servers.0.host}
all_features: ${JSON:/etc/features.json:features}
# JSON5 features: comments and trailing commas supported
config_value: ${JSON:/etc/app.json5:debug.level}
YAML - Read from YAML Files
# Read specific paths from YAML files
db_config: ${YAML:/etc/database.yml:production.primary}
replica_host: ${YAML:/etc/database.yml:production.replica.host}
Cloud Secret Managers
AWS Secrets Manager
# Requires AWS credentials via:
# - Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
# - IAM instance role (on EC2)
# - AWS config files (~/.aws/credentials)
database:
password: ${AWSSM:prod/db/password}
api_key: ${AWSSM:external-api-key}
# With versioning
secret: ${AWSSM:mysecret:AWSCURRENT}
Google Cloud Secret Manager
# Requires GCP credentials via:
# - Environment variable: GOOGLE_APPLICATION_CREDENTIALS
# - Default service account (on GCE/GKE)
# - gcloud auth application-default login
secrets:
api_key: ${GCPSM:projects/my-project/secrets/api-key}
db_pass: ${GCPSM:projects/my-project/secrets/db-password/versions/latest}
Azure Key Vault
# Requires Azure credentials via:
# - Environment variables: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID
# - Managed Service Identity (on Azure VMs)
# - Azure CLI authentication (az login)
credentials:
cert: ${AZURESM:https://myvault.vault.azure.net:mycert}
key: ${AZURESM:https://myvault.vault.azure.net:mykey}
Infrastructure Tools
HashiCorp Vault
# Requires VAULT_ADDR and VAULT_TOKEN configured
database:
password: ${VAULT:secret/data/myapp:db_password}
api_key: ${VAULT:secret/data/external:api_key}
# KV v2 paths
secret: ${VAULT:secret/data/myapp:password}
HashiCorp Consul
# Requires CONSUL_HTTP_ADDR configured
config:
feature_flags: ${CONSUL:myapp/features}
db_host: ${CONSUL:service/database/host}
Kubernetes Secrets
# Requires in-cluster or kubeconfig access
database:
password: ${K8SS:default/db-secret:password}
cert: ${K8SS:kube-system/tls-secret:tls.crt}
etcd
# Requires ETCD_ENDPOINTS configured
config:
cluster_size: ${ETCD:/cluster/size}
node_role: ${ETCD:/node/role}
API Reference
Loading Configuration
// Load from /etc/{appname}/config.yml
config, err := smartconfig.NewFromAppName("myapp")
// Load from specific file
config, err := smartconfig.NewFromConfigPath("/path/to/config.yaml")
// Load from io.Reader
reader := strings.NewReader(yamlContent)
config, err := smartconfig.NewFromReader(reader)
Accessing Values
Important: The Get()
method only supports top-level keys. For nested values, you need to navigate the structure manually.
// Get top-level value only
value, exists := config.Get("server") // Returns the entire server map
if !exists {
log.Fatal("server configuration not found")
}
// For nested values, cast and navigate
if serverMap, ok := value.(map[string]interface{}); ok {
if port, ok := serverMap["port"].(int); ok {
fmt.Printf("Port: %d\n", port)
}
}
// Or use typed getters for top-level values
port, err := config.GetInt("port") // Works for top-level
host, err := config.GetString("host") // Works for top-level
Typed Getters
All typed getters work with top-level keys only:
// String values
name, err := config.GetString("app_name")
// Integer values (works with both int and string values)
port, err := config.GetInt("port")
// Unsigned integers
maxConn, err := config.GetUint("max_connections")
// Float values
timeout, err := config.GetFloat("timeout_seconds")
// Boolean values (works with bool and string "true"/"false")
debug, err := config.GetBool("debug_mode")
// Byte sizes with human-readable formats ("10GB", "512MiB", etc.)
maxSize, err := config.GetBytes("max_file_size")
// Get entire config as map
data := config.Data()
Working with Nested Values
Since the API doesn't support dot notation, here's how to work with nested values:
// Given this YAML:
// server:
// host: localhost
// port: 8080
// ssl:
// enabled: true
// cert: /etc/ssl/cert.pem
data := config.Data()
// Navigate manually
if server, ok := data["server"].(map[string]interface{}); ok {
if ssl, ok := server["ssl"].(map[string]interface{}); ok {
if enabled, ok := ssl["enabled"].(bool); ok {
fmt.Printf("SSL enabled: %v\n", enabled)
}
}
}
// Or unmarshal into a struct
type Config struct {
Server struct {
Host string
Port int
SSL struct {
Enabled bool
Cert string
}
}
}
var cfg Config
data := config.Data()
// Use a YAML/JSON marshaling library to convert data to your struct
Complete Example
Here's a comprehensive example showing all features:
# production.yaml - Complete example with all features
# Top-level values for easy API access
app_name: ${ENV:APP_NAME}
environment: ${ENV:DEPLOY_ENV}
version: ${FILE:/app/VERSION}
build_info: ${EXEC:git describe --always --dirty}
# Type preservation examples
server_port: ${ENV:PORT} # Integer: 8080
debug_enabled: ${ENV:DEBUG} # Boolean: true/false
request_timeout: ${ENV:TIMEOUT_SECONDS} # Float: 30.5
max_upload_size: ${ENV:MAX_UPLOAD_SIZE} # Bytes: "100MB"
# Nested configuration
server:
listen_address: "0.0.0.0:${ENV:PORT}" # String concatenation
workers: ${ENV:WORKER_COUNT}
tls:
enabled: ${ENV:TLS_ENABLED}
cert: ${FILE:/etc/ssl/certs/server.crt}
key: ${VAULT:secret/data/ssl:private_key}
# Database configuration with nested interpolation
database:
primary:
host: ${ENV:DB_HOST_${ENV:DEPLOY_ENV}} # e.g., DB_HOST_prod
port: ${ENV:DB_PORT}
name: ${ENV:DB_NAME}
user: ${ENV:DB_USER}
password: ${AWSSM:${ENV:APP_NAME}/${ENV:DEPLOY_ENV}/db_password}
replica:
host: ${CONSUL:service/db-replica-${ENV:DEPLOY_ENV}/address}
port: ${CONSUL:service/db-replica-${ENV:DEPLOY_ENV}/port}
# External services
services:
redis:
url: ${ETCD:/config/${ENV:APP_NAME}/redis_url}
password: ${K8SS:default/redis-secret:password}
elasticsearch:
hosts: ${JSON:/etc/config/services.json:elasticsearch.hosts}
api_key: ${GCPSM:projects/${ENV:GCP_PROJECT}/secrets/es_api_key}
# Feature flags from various sources
features:
new_ui: ${CONSUL:features/${ENV:APP_NAME}/new_ui}
rate_limiting: ${ENV:FEATURE_RATE_LIMITING}
analytics: ${YAML:/etc/features.yaml:features.analytics.enabled}
# Cloud storage configuration
storage:
provider: ${ENV:STORAGE_PROVIDER}
config:
bucket: ${ENV:STORAGE_BUCKET}
region: ${ENV:AWS_REGION}
access_key: ${AWSSM:storage_access_key}
secret_key: ${AWSSM:storage_secret_key}
# Monitoring and logging
monitoring:
datadog:
api_key: ${VAULT:secret/data/monitoring:datadog_api_key}
app_key: ${VAULT:secret/data/monitoring:datadog_app_key}
sentry:
dsn: ${ENV:SENTRY_DSN}
environment: ${ENV:DEPLOY_ENV}
release: ${EXEC:git rev-parse HEAD}
# Environment variables to inject
env:
# These will be set as environment variables in the process
DATABASE_URL: "postgres://${ENV:DB_USER}:${AWSSM:${ENV:APP_NAME}/${ENV:DEPLOY_ENV}/db_password}@${ENV:DB_HOST_${ENV:DEPLOY_ENV}}:${ENV:DB_PORT}/${ENV:DB_NAME}"
REDIS_URL: ${ETCD:/config/${ENV:APP_NAME}/redis_url}
NEW_RELIC_LICENSE_KEY: ${AZURESM:https://myvault.vault.azure.net:newrelic-license}
DD_TRACE_ENABLED: ${ENV:DD_TRACE_ENABLED}
// main.go - Using the complete configuration
package main
import (
"fmt"
"log"
"os"
"git.eeqj.de/sneak/smartconfig"
)
func main() {
// Load configuration
config, err := smartconfig.NewFromConfigPath("production.yaml")
if err != nil {
log.Fatalf("Failed to load config: %v", err)
}
// Access top-level typed values
appName, _ := config.GetString("app_name")
serverPort, _ := config.GetInt("server_port")
debugEnabled, _ := config.GetBool("debug_enabled")
maxUploadSize, _ := config.GetBytes("max_upload_size")
fmt.Printf("Starting %s on port %d (debug: %v)\n", appName, serverPort, debugEnabled)
fmt.Printf("Max upload size: %d bytes\n", maxUploadSize)
// Access nested configuration
data := config.Data()
// Database configuration
if db, ok := data["database"].(map[string]interface{}); ok {
if primary, ok := db["primary"].(map[string]interface{}); ok {
dbHost, _ := primary["host"].(string)
dbPort, _ := primary["port"].(int)
fmt.Printf("Database: %s:%d\n", dbHost, dbPort)
}
}
// Check injected environment variables
fmt.Printf("DATABASE_URL: %s\n", os.Getenv("DATABASE_URL"))
fmt.Printf("REDIS_URL: %s\n", os.Getenv("REDIS_URL"))
// Feature flags
if features, ok := data["features"].(map[string]interface{}); ok {
if newUI, ok := features["new_ui"].(bool); ok && newUI {
fmt.Println("New UI is enabled!")
}
}
}
Advanced Features
Nested Interpolation
Interpolations can be nested up to 3 levels deep:
# Dynamic environment selection
environment: prod
database:
host: ${ENV:DB_HOST_${ENV:ENVIRONMENT}} # Looks for DB_HOST_prod
# Multi-level nesting
config:
value: ${ENV:${ENV:PREFIX}_${ENV:SUFFIX}_KEY}
# With different resolvers
secret: ${VAULT:secret/${ENV:APP_NAME}/${ENV:ENVIRONMENT}:password}
Environment Variable Injection
Values under the env
key are automatically exported as environment variables:
# These become environment variables in your process
env:
DATABASE_URL: "postgres://${ENV:DB_USER}:${AWSSM:db-password}@${ENV:DB_HOST}/myapp"
REDIS_URL: ${CONSUL:service/redis/url}
API_KEY: ${VAULT:secret/data/external:api_key}
# Your application can now use os.Getenv("DATABASE_URL"), etc.
Custom Resolvers
Extend smartconfig with your own resolvers:
// Implement the Resolver interface
type RedisResolver struct {
client *redis.Client
}
func (r *RedisResolver) Resolve(key string) (string, error) {
return r.client.Get(key).Result()
}
// Register your resolver
config := smartconfig.New()
config.RegisterResolver("REDIS", &RedisResolver{client: redisClient})
// Use in YAML
// cache_ttl: ${REDIS:config:cache:ttl}
Error Handling
Always handle errors appropriately:
config, err := smartconfig.NewFromAppName("myapp")
if err != nil {
if os.IsNotExist(err) {
log.Fatal("Config file not found in /etc/myapp/config.yml")
}
log.Fatalf("Failed to load config: %v", err)
}
// Handle missing keys
port, err := config.GetInt("server_port")
if err != nil {
if strings.Contains(err.Error(), "not found") {
// Use default
port = 8080
} else {
log.Fatalf("Invalid port configuration: %v", err)
}
}
// Handle type conversion errors
timeout, err := config.GetFloat("timeout")
if err != nil {
if strings.Contains(err.Error(), "cannot convert") {
log.Fatalf("Timeout must be a number, got: %v", err)
}
}
CLI Tool
A command-line tool is provided for testing and preprocessing configuration files:
# Install the CLI tool
go install git.eeqj.de/sneak/smartconfig/cmd/smartconfig@latest
# Process a config file (outputs YAML)
smartconfig < config.yaml > processed.yaml
# Output as JSON
smartconfig --json < config.yaml > processed.json
# Test interpolation
export DB_PASSWORD="secret123"
echo 'password: ${ENV:DB_PASSWORD}' | smartconfig
# Output: password: secret123
# Type preservation in action
export PORT="8080"
export ENABLED="true"
export TIMEOUT="30.5"
cat <<'EOF' | smartconfig --json
port: ${ENV:PORT}
enabled: ${ENV:ENABLED}
timeout: ${ENV:TIMEOUT}
EOF
# Output: {"enabled":true,"port":8080,"timeout":30.5}
Security Considerations
Arbitrary Code Execution
The EXEC
resolver can execute any shell command. This is by design but has security implications:
- Only use trusted configuration files
- Validate configuration sources in production
- Consider disabling EXEC resolver in sensitive environments
- Use appropriate file permissions on configuration files
Secret Management Best Practices
- Never commit secrets to version control
- Use appropriate secret managers (Vault, AWS SM, etc.) for production
- Limit access to configuration files containing secret references
- Rotate secrets regularly
- Monitor secret access
File Access
The FILE
resolver can read any file accessible to the process:
- Run applications with minimal required permissions
- Use separate users for different applications
- Consider using secrets managers instead of direct file access
Comparison with Other Libraries
vs. Viper
- smartconfig: Focuses on interpolation from multiple sources with type preservation
- Viper: More features but heavier, includes watching, flags, etc.
vs. envconfig
- smartconfig: YAML-based with interpolation from many sources
- envconfig: Environment variables only, struct tags
vs. koanf
- smartconfig: Simpler API, focused on interpolation
- koanf: More providers and formats, more complex API
Requirements
- Go 1.18 or later
- Valid YAML syntax in configuration files
- Appropriate credentials for cloud providers (AWS, GCP, Azure)
- Network access for remote resolvers (Vault, Consul, etcd)
Contributing
Contributions are welcome! Please ensure:
- Tests pass:
make test
- Code is formatted:
make fmt
- No linting errors:
make lint
License
WTFPL
Author
sneak sneak@sneak.berlin
Inspired by Ryan Smith's configuration format ideas.