- Update Makefile to inject VERSION and GIT_COMMIT at build time - Change Version from const to var to allow ldflags injection - Add build target to create binary with proper version info |
||
---|---|---|
cmd/smartconfig | ||
test | ||
.gitignore | ||
AGENTS.md | ||
cli_missing_env_test.go | ||
cli_test.go | ||
DESIGN.md | ||
gjson_edge_cases_refactored_test.go | ||
gjson_edge_cases_test.go | ||
gjson_malformed_test.go | ||
gjson_path_test.go | ||
go.mod | ||
go.sum | ||
interpolate.go | ||
LICENSE | ||
Makefile | ||
readme_examples_test.go | ||
README.md | ||
resolver_awssm.go | ||
resolver_azure.go | ||
resolver_consul.go | ||
resolver_env.go | ||
resolver_etcd.go | ||
resolver_exec.go | ||
resolver_file.go | ||
resolver_gcpsm.go | ||
resolver_json.go | ||
resolver_k8s.go | ||
resolver_vault.go | ||
resolver_yaml_test.go | ||
resolver_yaml.go | ||
smartconfig_test.go | ||
smartconfig.go | ||
test_helpers_test.go | ||
TODO.md | ||
typed_interpolation_test.go | ||
version.go | ||
yaml_syntax_test.go |
smartconfig
A Go library for YAML configuration files with powerful variable interpolation from multiple sources including environment variables, files, cloud secret managers, and more.
Table of Contents
- Key Features
- Installation
- Quick Start
- Type-Preserving Interpolation
- Supported Resolvers
- API Reference
- Advanced Features
- CLI Tool
- Security Considerations
- Comparison with Other Libraries
- Requirements
- Contributing
- License
Key Features
- Type-Preserving Interpolation: Standalone interpolations preserve their types (numbers, booleans, strings)
- Multiple Data Sources: Environment variables, files, cloud secrets (AWS, GCP, Azure), Vault, Consul, K8S, and more
- Nested Interpolation: Support for complex variable references like
${ENV:PREFIX_${ENV:SUFFIX}}
- JSON5 Support: JSON resolver supports comments and trailing commas
- gjson Path Support: Access nested configuration values using gjson syntax like
server.ssl.enabled
ordatabase.replicas.0.host
- Environment Injection: Automatically export config values as environment variables
- Extensible: Add custom resolvers for your own data sources
- CLI Tool: Command-line tool for processing YAML files with interpolation
Installation
go get git.eeqj.de/sneak/smartconfig
Quick Start
Basic Example
# config.yaml
# Showcase the power of multiple resolver types
app_name: ${ENV:APP_NAME}
version: ${FILE:/app/VERSION} # Read from file
port: ${ENV:PORT} # Number if PORT="8080"
debug_mode: ${ENV:DEBUG_MODE} # Boolean if "true"/"false"
# External service configuration from JSON config file
services: ${JSON:/etc/config/services.json:production} # Load entire object
# API credentials from cloud secret managers
api_keys:
stripe: ${GCPSM:projects/myproject/secrets/stripe-api-key}
sendgrid: ${AWSSM:prod/sendgrid-api-key}
twilio: ${VAULT:secret/data/api:twilio_key}
datadog: ${ENV:DD_API_KEY} # Fallback to env for local dev
# Database configuration with mixed sources
database:
host: ${CONSUL:service/postgres/address}
port: ${CONSUL:service/postgres/port}
name: ${ENV:DB_NAME}
username: ${ENV:DB_USER}
password: ${GCPSM:projects/myproject/secrets/db-password}
# SSL configuration from multiple sources
ssl:
enabled: ${ENV:DB_SSL_ENABLED}
ca_cert: ${FILE:/etc/ssl/db-ca.crt}
client_cert: ${K8SS:default/db-certs:client.crt}
# Feature flags from JSON file
features: ${JSON:/etc/config/features.json:${ENV:ENVIRONMENT}}
# Server configuration
server:
listen: "0.0.0.0:${ENV:PORT}"
workers: ${EXEC:nproc} # Dynamic based on CPU cores
hostname: ${EXEC:hostname -f}
env:
# Export these as environment variables for child processes
DATABASE_URL: "postgres://${ENV:DB_USER}:${GCPSM:projects/myproject/secrets/db-password}@${CONSUL:service/postgres/address}:5432/${ENV:DB_NAME}"
NEW_RELIC_LICENSE: ${AWSSM:monitoring/newrelic-license}
Go Code
package main
import (
"fmt"
"log"
"os"
"git.eeqj.de/sneak/smartconfig"
)
func main() {
// Load from /etc/myapp/config.yml
config, err := smartconfig.NewFromAppName("myapp")
if err != nil {
log.Fatal(err)
}
// Access typed values
appName, _ := config.GetString("app_name")
port, _ := config.GetInt("port")
debugMode, _ := config.GetBool("debug_mode")
fmt.Printf("Starting %s on port %d (debug: %v)\n", appName, port, debugMode)
// Access nested values using gjson paths
stripeKey, _ := config.GetString("api_keys.stripe")
fmt.Printf("Stripe API key loaded: %s...\n", stripeKey[:8])
// Database configuration
dbHost, _ := config.GetString("database.host")
dbPort, _ := config.GetInt("database.port")
sslEnabled, _ := config.GetBool("database.ssl.enabled")
fmt.Printf("Database: %s:%d (SSL: %v)\n", dbHost, dbPort, sslEnabled)
// Check loaded services from JSON file
services, exists := config.Get("services")
if exists && services != nil {
if servicesMap, ok := services.(map[string]interface{}); ok {
fmt.Printf("Loaded %d services from JSON config\n", len(servicesMap))
}
}
// Environment variables are now available
fmt.Printf("DATABASE_URL env var: %s\n", os.Getenv("DATABASE_URL"))
}
Example JSON files referenced above:
// /etc/config/services.json
{
"production": {
"api": {
"endpoint": "https://api.example.com",
"timeout": 30,
"retries": 3
},
"cache": {
"provider": "redis",
"ttl": 3600
}
},
"staging": {
"api": {
"endpoint": "https://api-staging.example.com",
"timeout": 60,
"retries": 5
}
}
}
// /etc/config/features.json
{
"production": {
"new_ui": false,
"rate_limiting": true,
"analytics": true
},
"staging": {
"new_ui": true,
"rate_limiting": true,
"analytics": false
}
}
Type-Preserving Interpolation
smartconfig parses YAML first, then performs interpolation. This allows proper type preservation:
How It Works
# Standalone interpolations preserve types
port: ${ENV:PORT} # If PORT="8080", becomes integer 8080
enabled: ${ENV:ENABLED} # If ENABLED="true", becomes boolean true
timeout: ${ENV:TIMEOUT} # If TIMEOUT="30.5", becomes float 30.5
# Mixed content always returns strings
message: "Hello ${ENV:USER}!" # Always a string
port_label: "Port: ${ENV:PORT}" # Always a string
debug_flag: "debug-${ENV:DEBUG}" # Always a string
# Force string by adding any prefix/suffix
port_string: "${ENV:PORT}-suffix" # Forces string output
bool_string: "prefix-${ENV:ENABLED}" # Forces string output
Type Conversion Rules
For standalone interpolations, smartconfig automatically converts:
"true"
→true
(boolean)"false"
→false
(boolean)- Numeric strings → numbers (int or float64) with safety checks
- Everything else → string
Safety Check: Numbers are only converted if the conversion is lossless. This means:
"123"
→123
(converts to int)"0123"
→"0123"
(stays string - leading zeros)"123.45"
→123.45
(converts to float)"1.50"
→"1.50"
(stays string - trailing zeros would be lost)"+123"
→"+123"
(stays string - plus sign would be lost)"1e10"
→"1e10"
(stays string - notation would change)
Supported Resolvers
Local Resolvers
ENV - Environment Variables
# Basic usage
api_key: ${ENV:API_KEY}
# With nested interpolation
database: ${ENV:DB_${ENV:ENVIRONMENT}}
FILE - Read File Contents
# Read entire file (trimmed)
ssl_cert: ${FILE:/etc/ssl/cert.pem}
machine_id: ${FILE:/etc/machine-id}
# Read system files
cpu_temp: ${FILE:/sys/class/thermal/thermal_zone0/temp}
EXEC - Execute Shell Commands
# Simple commands
hostname: ${EXEC:hostname -f}
timestamp: ${EXEC:date +%s}
git_hash: ${EXEC:git rev-parse HEAD}
# Complex commands with pipes
users_count: ${EXEC:who | wc -l}
docker_running: ${EXEC:docker ps -q | wc -l}
JSON - Read from JSON Files (with JSON5 support)
# Supports gjson path syntax
api_endpoint: ${JSON:/etc/config.json:services.api.endpoint}
first_server: ${JSON:/etc/servers.json:servers.0.host}
all_features: ${JSON:/etc/features.json:features}
# JSON5 features: comments and trailing commas supported
config_value: ${JSON:/etc/app.json5:debug.level}
YAML - Read from YAML Files
# Read specific paths from YAML files
db_config: ${YAML:/etc/database.yml:production.primary}
replica_host: ${YAML:/etc/database.yml:production.replica.host}
Cloud Secret Managers
AWS Secrets Manager
# Requires AWS credentials via:
# - Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
# - IAM instance role (on EC2)
# - AWS config files (~/.aws/credentials)
database:
password: ${AWSSM:prod/db/password}
api_key: ${AWSSM:external-api-key}
# With versioning
secret: ${AWSSM:mysecret:AWSCURRENT}
Google Cloud Secret Manager
# Requires GCP credentials via:
# - Environment variable: GOOGLE_APPLICATION_CREDENTIALS
# - Default service account (on GCE/GKE)
# - gcloud auth application-default login
secrets:
api_key: ${GCPSM:projects/my-project/secrets/api-key}
db_pass: ${GCPSM:projects/my-project/secrets/db-password/versions/latest}
Azure Key Vault
# Requires Azure credentials via:
# - Environment variables: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID
# - Managed Service Identity (on Azure VMs)
# - Azure CLI authentication (az login)
credentials:
cert: ${AZURESM:https://myvault.vault.azure.net:mycert}
key: ${AZURESM:https://myvault.vault.azure.net:mykey}
Infrastructure Tools
HashiCorp Vault
# Requires VAULT_ADDR and VAULT_TOKEN configured
database:
password: ${VAULT:secret/data/myapp:db_password}
api_key: ${VAULT:secret/data/external:api_key}
# KV v2 paths
secret: ${VAULT:secret/data/myapp:password}
HashiCorp Consul
# Requires CONSUL_HTTP_ADDR configured
config:
feature_flags: ${CONSUL:myapp/features}
db_host: ${CONSUL:service/database/host}
Kubernetes Secrets
# Requires in-cluster or kubeconfig access
database:
password: ${K8SS:default/db-secret:password}
cert: ${K8SS:kube-system/tls-secret:tls.crt}
etcd
# Requires ETCD_ENDPOINTS configured
config:
cluster_size: ${ETCD:/cluster/size}
node_role: ${ETCD:/node/role}
API Reference
Loading Configuration
// Load from /etc/{appname}/config.yml
config, err := smartconfig.NewFromAppName("myapp")
// Load from specific file
config, err := smartconfig.NewFromConfigPath("/path/to/config.yaml")
// Load from io.Reader
reader := strings.NewReader(yamlContent)
config, err := smartconfig.NewFromReader(reader)
Accessing Values
All accessor methods now support gjson path syntax for accessing nested values:
// Get nested values using gjson paths
host, err := config.GetString("server.host") // "localhost"
port, err := config.GetInt("database.replicas.0.port") // 5433
enabled, err := config.GetBool("server.ssl.enabled") // true
// Get() method also supports gjson paths
value, exists := config.Get("database.primary.credentials")
if exists {
// value is map[string]interface{} with username and password
}
// Backward compatibility: top-level keys still work
appName, err := config.GetString("app_name") // Direct top-level access
Typed Getters
All typed getters support gjson path syntax for accessing nested values:
// String values
name, err := config.GetString("app_name") // Top-level
dbHost, err := config.GetString("database.primary.host") // Nested path
// Integer values (works with both int and string values)
port, err := config.GetInt("server.port") // Nested
replicaPort, err := config.GetInt("database.replicas.1.port") // Array element
// Unsigned integers
maxConn, err := config.GetUint("features.max_connections") // Nested
// Float values
timeout, err := config.GetFloat("features.timeout_seconds") // Nested
// Boolean values (works with bool and string "true"/"false")
debug, err := config.GetBool("features.debug_mode") // Nested
sslEnabled, err := config.GetBool("server.ssl.enabled") // Deep nested
// Byte sizes with human-readable formats ("10GB", "512MiB", etc.)
maxSize, err := config.GetBytes("features.max_file_size") // Nested
// Get entire config as map
data := config.Data()
Working with Nested Values
With gjson path support, accessing nested values is now straightforward:
// Given this YAML:
// server:
// host: localhost
// port: 8080
// ssl:
// enabled: true
// cert: /etc/ssl/cert.pem
// Direct access using gjson paths
sslEnabled, _ := config.GetBool("server.ssl.enabled") // true
sslCert, _ := config.GetString("server.ssl.cert") // "/etc/ssl/cert.pem"
serverPort, _ := config.GetInt("server.port") // 8080
// Array access
// database:
// replicas:
// - host: db1.example.com
// port: 5433
// - host: db2.example.com
// port: 5434
firstReplica, _ := config.GetString("database.replicas.0.host") // "db1.example.com"
secondPort, _ := config.GetInt("database.replicas.1.port") // 5434
// You can still unmarshal into a struct if preferred
type Config struct {
Server struct {
Host string
Port int
SSL struct {
Enabled bool
Cert string
}
}
}
var cfg Config
data := config.Data()
// Use a YAML/JSON marshaling library to convert data to your struct
Complete Example
Here's a comprehensive example showing all features:
# production.yaml - Complete example with all features
# Top-level values for easy API access
app_name: ${ENV:APP_NAME}
environment: ${ENV:DEPLOY_ENV}
version: ${FILE:/app/VERSION}
build_info: ${EXEC:git describe --always --dirty}
# Type preservation examples
server_port: ${ENV:PORT} # Integer: 8080
debug_enabled: ${ENV:DEBUG} # Boolean: true/false
request_timeout: ${ENV:TIMEOUT_SECONDS} # Float: 30.5
max_upload_size: ${ENV:MAX_UPLOAD_SIZE} # Bytes: "100MB"
# Nested configuration
server:
listen_address: "0.0.0.0:${ENV:PORT}" # String concatenation
workers: ${ENV:WORKER_COUNT}
tls:
enabled: ${ENV:TLS_ENABLED}
cert: ${FILE:/etc/ssl/certs/server.crt}
key: ${VAULT:secret/data/ssl:private_key}
# Database configuration with nested interpolation
database:
primary:
host: ${ENV:DB_HOST_${ENV:DEPLOY_ENV}} # e.g., DB_HOST_prod
port: ${ENV:DB_PORT}
name: ${ENV:DB_NAME}
user: ${ENV:DB_USER}
password: ${AWSSM:${ENV:APP_NAME}/${ENV:DEPLOY_ENV}/db_password}
replica:
host: ${CONSUL:service/db-replica-${ENV:DEPLOY_ENV}/address}
port: ${CONSUL:service/db-replica-${ENV:DEPLOY_ENV}/port}
# External services
services:
redis:
url: ${ETCD:/config/${ENV:APP_NAME}/redis_url}
password: ${K8SS:default/redis-secret:password}
elasticsearch:
hosts: ${JSON:/etc/config/services.json:elasticsearch.hosts}
api_key: ${GCPSM:projects/${ENV:GCP_PROJECT}/secrets/es_api_key}
# Feature flags from various sources
features:
new_ui: ${CONSUL:features/${ENV:APP_NAME}/new_ui}
rate_limiting: ${ENV:FEATURE_RATE_LIMITING}
analytics: ${YAML:/etc/features.yaml:features.analytics.enabled}
# Cloud storage configuration
storage:
provider: ${ENV:STORAGE_PROVIDER}
config:
bucket: ${ENV:STORAGE_BUCKET}
region: ${ENV:AWS_REGION}
access_key: ${AWSSM:storage_access_key}
secret_key: ${AWSSM:storage_secret_key}
# Monitoring and logging
monitoring:
datadog:
api_key: ${VAULT:secret/data/monitoring:datadog_api_key}
app_key: ${VAULT:secret/data/monitoring:datadog_app_key}
sentry:
dsn: ${ENV:SENTRY_DSN}
environment: ${ENV:DEPLOY_ENV}
release: ${EXEC:git rev-parse HEAD}
# Environment variables to inject
env:
# These will be set as environment variables in the process
DATABASE_URL: "postgres://${ENV:DB_USER}:${AWSSM:${ENV:APP_NAME}/${ENV:DEPLOY_ENV}/db_password}@${ENV:DB_HOST_${ENV:DEPLOY_ENV}}:${ENV:DB_PORT}/${ENV:DB_NAME}"
REDIS_URL: ${ETCD:/config/${ENV:APP_NAME}/redis_url}
NEW_RELIC_LICENSE_KEY: ${AZURESM:https://myvault.vault.azure.net:newrelic-license}
DD_TRACE_ENABLED: ${ENV:DD_TRACE_ENABLED}
// main.go - Using the complete configuration
package main
import (
"fmt"
"log"
"os"
"git.eeqj.de/sneak/smartconfig"
)
func main() {
// Load configuration
config, err := smartconfig.NewFromConfigPath("production.yaml")
if err != nil {
log.Fatalf("Failed to load config: %v", err)
}
// Access top-level typed values
appName, _ := config.GetString("app_name")
serverPort, _ := config.GetInt("server_port")
debugEnabled, _ := config.GetBool("debug_enabled")
maxUploadSize, _ := config.GetBytes("max_upload_size")
fmt.Printf("Starting %s on port %d (debug: %v)\n", appName, serverPort, debugEnabled)
fmt.Printf("Max upload size: %d bytes\n", maxUploadSize)
// Access nested configuration using gjson paths
dbHost, _ := config.GetString("database.primary.host")
dbPort, _ := config.GetInt("database.primary.port")
fmt.Printf("Database: %s:%d\n", dbHost, dbPort)
// Access array elements
replicaHost, _ := config.GetString("database.replica.host")
fmt.Printf("Replica: %s\n", replicaHost)
// Check injected environment variables
fmt.Printf("DATABASE_URL: %s\n", os.Getenv("DATABASE_URL"))
fmt.Printf("REDIS_URL: %s\n", os.Getenv("REDIS_URL"))
// Feature flags
if features, ok := data["features"].(map[string]interface{}); ok {
if newUI, ok := features["new_ui"].(bool); ok && newUI {
fmt.Println("New UI is enabled!")
}
}
}
Advanced Features
Nested Interpolation
Interpolations can be nested up to 3 levels deep:
# Dynamic environment selection
environment: prod
database:
host: ${ENV:DB_HOST_${ENV:ENVIRONMENT}} # Looks for DB_HOST_prod
# Multi-level nesting
config:
value: ${ENV:${ENV:PREFIX}_${ENV:SUFFIX}_KEY}
# With different resolvers
secret: ${VAULT:secret/${ENV:APP_NAME}/${ENV:ENVIRONMENT}:password}
Environment Variable Injection
Values under the env
key are automatically exported as environment variables:
# These become environment variables in your process
env:
DATABASE_URL: "postgres://${ENV:DB_USER}:${AWSSM:db-password}@${ENV:DB_HOST}/myapp"
REDIS_URL: ${CONSUL:service/redis/url}
API_KEY: ${VAULT:secret/data/external:api_key}
# Your application can now use os.Getenv("DATABASE_URL"), etc.
Custom Resolvers
Extend smartconfig with your own resolvers:
// Implement the Resolver interface
type RedisResolver struct {
client *redis.Client
}
func (r *RedisResolver) Resolve(key string) (string, error) {
return r.client.Get(key).Result()
}
// Register your resolver
config := smartconfig.New()
config.RegisterResolver("REDIS", &RedisResolver{client: redisClient})
// Use in YAML
// cache_ttl: ${REDIS:config:cache:ttl}
Error Handling
Always handle errors appropriately:
config, err := smartconfig.NewFromAppName("myapp")
if err != nil {
if os.IsNotExist(err) {
log.Fatal("Config file not found in /etc/myapp/config.yml")
}
log.Fatalf("Failed to load config: %v", err)
}
// Handle missing keys
port, err := config.GetInt("server_port")
if err != nil {
if strings.Contains(err.Error(), "not found") {
// Use default
port = 8080
} else {
log.Fatalf("Invalid port configuration: %v", err)
}
}
// Handle type conversion errors
timeout, err := config.GetFloat("timeout")
if err != nil {
if strings.Contains(err.Error(), "cannot convert") {
log.Fatalf("Timeout must be a number, got: %v", err)
}
}
CLI Tool
A command-line tool is provided for testing and preprocessing configuration files:
# Install the CLI tool
go install git.eeqj.de/sneak/smartconfig/cmd/smartconfig@latest
# Process a config file (outputs YAML)
smartconfig < config.yaml > processed.yaml
# Output as JSON
smartconfig --json < config.yaml > processed.json
# Test interpolation
export DB_PASSWORD="secret123"
echo 'password: ${ENV:DB_PASSWORD}' | smartconfig
# Output: password: secret123
# Type preservation in action
export PORT="8080"
export ENABLED="true"
export TIMEOUT="30.5"
cat <<'EOF' | smartconfig --json
port: ${ENV:PORT}
enabled: ${ENV:ENABLED}
timeout: ${ENV:TIMEOUT}
EOF
# Output: {"enabled":true,"port":8080,"timeout":30.5}
Security Considerations
Arbitrary Code Execution
The EXEC
resolver can execute any shell command. This is by design but has security implications:
- Only use trusted configuration files
- Validate configuration sources in production
- Consider disabling EXEC resolver in sensitive environments
- Use appropriate file permissions on configuration files
Secret Management Best Practices
- Never commit secrets to version control
- Use appropriate secret managers (Vault, AWS SM, etc.) for production
- Limit access to configuration files containing secret references
- Rotate secrets regularly
- Monitor secret access
File Access
The FILE
resolver can read any file accessible to the process:
- Run applications with minimal required permissions
- Use separate users for different applications
- Consider using secrets managers instead of direct file access
Comparison with Other Libraries
vs. Viper
- smartconfig: Focuses on interpolation from multiple sources with type preservation
- Viper: More features but heavier, includes watching, flags, etc.
vs. envconfig
- smartconfig: YAML-based with interpolation from many sources
- envconfig: Environment variables only, struct tags
vs. koanf
- smartconfig: Simpler API, focused on interpolation
- koanf: More providers and formats, more complex API
Requirements
- Go 1.24.4 or later
- Valid YAML syntax in configuration files
- Appropriate credentials for cloud providers (AWS, GCP, Azure)
- Network access for remote resolvers (Vault, Consul, etcd)
Contributing
Contributions are welcome! Please ensure:
- Tests pass:
make test
- Code is formatted:
make fmt
- No linting errors:
make lint
License
WTFPL
Author
sneak sneak@sneak.berlin
Inspired by Ryan Smith's configuration format ideas.