smartconfig/README.md

smartconfig

A Go library for YAML configuration files with powerful variable interpolation from multiple sources including environment variables, files, cloud secret managers, and more.

Table of Contents

• Key Features
• Installation
• Quick Start
• Type-Preserving Interpolation
• Supported Resolvers
• API Reference
• Advanced Features
• CLI Tool
• Security Considerations
• Comparison with Other Libraries
• Requirements
• Contributing
• License

Key Features

• Type-Preserving Interpolation: Standalone interpolations preserve their types (numbers, booleans, strings)
• Multiple Data Sources: Environment variables, files, cloud secrets (AWS, GCP, Azure), Vault, Consul, K8S, and more
• Nested Interpolation: Support for complex variable references like ${ENV:PREFIX_${ENV:SUFFIX}}
• JSON5 Support: JSON resolver supports comments and trailing commas
• Environment Injection: Automatically export config values as environment variables
• Extensible: Add custom resolvers for your own data sources
• CLI Tool: Command-line tool for processing YAML files with interpolation

Installation

go get git.eeqj.de/sneak/smartconfig

Quick Start

Basic Example

# config.yaml
# Example with top-level values for easy access
app_name: ${ENV:APP_NAME}
port: ${ENV:PORT}                      # Becomes number 8080 if PORT="8080"
debug_mode: ${ENV:DEBUG_MODE}          # Becomes boolean true if DEBUG_MODE="true"
cache_size_bytes: ${ENV:CACHE_SIZE}    # Human-readable: "100MB", "1.5GB"

# Nested configuration for organization
server:
  host: localhost
  port: ${ENV:SERVER_PORT}
  name: "server-${ENV:INSTANCE_ID}"    # Always a string due to mixed content

database:
  host: ${ENV:DB_HOST}
  port: ${ENV:DB_PORT}
  username: ${ENV:DB_USER}
  password: ${VAULT:secret/data/myapp:db_password}

features:
  cache_enabled: ${ENV:CACHE_ENABLED}  # Boolean if "true"/"false"
  max_connections: ${ENV:MAX_CONN}     # Number if numeric string
  debug: ${ENV:FEATURES_DEBUG}

env:
  # These will be exported as environment variables
  API_TOKEN: ${AWSSM:api-token}
  SLACK_WEBHOOK: ${FILE:/etc/secrets/slack_webhook.txt}

Go Code

package main

import (
    "fmt"
    "log"
    
    "git.eeqj.de/sneak/smartconfig"
)

func main() {
    // Load from /etc/myapp/config.yml
    config, err := smartconfig.NewFromAppName("myapp")
    if err != nil {
        log.Fatal(err)
    }
    
    // Access typed values (top-level keys only)
    port, err := config.GetInt("port")
    if err != nil {
        log.Fatal(err)
    }
    
    // For nested values, navigate manually
    data := config.Data()
    var debugMode bool
    if features, ok := data["features"].(map[string]interface{}); ok {
        if debug, ok := features["debug"].(bool); ok {
            debugMode = debug
        }
    }
    
    // Human-readable byte sizes (top-level)
    cacheSize, err := config.GetBytes("cache_size_bytes")
    if err != nil {
        log.Fatal(err)
    }
    
    fmt.Printf("Server running on port %d (debug: %v)\n", port, debugMode)
    fmt.Printf("Cache size: %d bytes\n", cacheSize)
}

Type-Preserving Interpolation

smartconfig parses YAML first, then performs interpolation. This allows proper type preservation:

How It Works

# Standalone interpolations preserve types
port: ${ENV:PORT}           # If PORT="8080", becomes integer 8080
enabled: ${ENV:ENABLED}     # If ENABLED="true", becomes boolean true
timeout: ${ENV:TIMEOUT}     # If TIMEOUT="30.5", becomes float 30.5

# Mixed content always returns strings
message: "Hello ${ENV:USER}!"              # Always a string
port_label: "Port: ${ENV:PORT}"            # Always a string
debug_flag: "debug-${ENV:DEBUG}"           # Always a string

# Force string by adding any prefix/suffix
port_string: "${ENV:PORT}-suffix"          # Forces string output
bool_string: "prefix-${ENV:ENABLED}"       # Forces string output

Type Conversion Rules

For standalone interpolations, smartconfig automatically converts:

• "true" → true (boolean)
• "false" → false (boolean)
• Numeric strings → numbers (int or float64)
• Everything else → string

Supported Resolvers

Local Resolvers

ENV - Environment Variables

# Basic usage
api_key: ${ENV:API_KEY}

# With nested interpolation
database: ${ENV:DB_${ENV:ENVIRONMENT}}

FILE - Read File Contents

# Read entire file (trimmed)
ssl_cert: ${FILE:/etc/ssl/cert.pem}
machine_id: ${FILE:/etc/machine-id}

# Read system files
cpu_temp: ${FILE:/sys/class/thermal/thermal_zone0/temp}

EXEC - Execute Shell Commands

# Simple commands
hostname: ${EXEC:hostname -f}
timestamp: ${EXEC:date +%s}
git_hash: ${EXEC:git rev-parse HEAD}

# Complex commands with pipes
users_count: ${EXEC:who | wc -l}
docker_running: ${EXEC:docker ps -q | wc -l}

JSON - Read from JSON Files (with JSON5 support)

# Supports gjson path syntax
api_endpoint: ${JSON:/etc/config.json:services.api.endpoint}
first_server: ${JSON:/etc/servers.json:servers.0.host}
all_features: ${JSON:/etc/features.json:features}

# JSON5 features: comments and trailing commas supported
config_value: ${JSON:/etc/app.json5:debug.level}

YAML - Read from YAML Files

# Read specific paths from YAML files
db_config: ${YAML:/etc/database.yml:production.primary}
replica_host: ${YAML:/etc/database.yml:production.replica.host}

Cloud Secret Managers

AWS Secrets Manager

# Requires AWS credentials via:
# - Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
# - IAM instance role (on EC2)
# - AWS config files (~/.aws/credentials)

database:
  password: ${AWSSM:prod/db/password}
  api_key: ${AWSSM:external-api-key}
  
# With versioning
secret: ${AWSSM:mysecret:AWSCURRENT}

Google Cloud Secret Manager

# Requires GCP credentials via:
# - Environment variable: GOOGLE_APPLICATION_CREDENTIALS
# - Default service account (on GCE/GKE)
# - gcloud auth application-default login

secrets:
  api_key: ${GCPSM:projects/my-project/secrets/api-key}
  db_pass: ${GCPSM:projects/my-project/secrets/db-password/versions/latest}

Azure Key Vault

# Requires Azure credentials via:
# - Environment variables: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID
# - Managed Service Identity (on Azure VMs)
# - Azure CLI authentication (az login)

credentials:
  cert: ${AZURESM:https://myvault.vault.azure.net:mycert}
  key: ${AZURESM:https://myvault.vault.azure.net:mykey}

Infrastructure Tools

HashiCorp Vault

# Requires VAULT_ADDR and VAULT_TOKEN configured
database:
  password: ${VAULT:secret/data/myapp:db_password}
  api_key: ${VAULT:secret/data/external:api_key}
  
# KV v2 paths
secret: ${VAULT:secret/data/myapp:password}

HashiCorp Consul

# Requires CONSUL_HTTP_ADDR configured
config:
  feature_flags: ${CONSUL:myapp/features}
  db_host: ${CONSUL:service/database/host}

Kubernetes Secrets

# Requires in-cluster or kubeconfig access
database:
  password: ${K8SS:default/db-secret:password}
  cert: ${K8SS:kube-system/tls-secret:tls.crt}

etcd

# Requires ETCD_ENDPOINTS configured
config:
  cluster_size: ${ETCD:/cluster/size}
  node_role: ${ETCD:/node/role}

API Reference

Loading Configuration

// Load from /etc/{appname}/config.yml
config, err := smartconfig.NewFromAppName("myapp")

// Load from specific file
config, err := smartconfig.NewFromConfigPath("/path/to/config.yaml")

// Load from io.Reader
reader := strings.NewReader(yamlContent)
config, err := smartconfig.NewFromReader(reader)

Accessing Values

Important: The Get() method only supports top-level keys. For nested values, you need to navigate the structure manually.

// Get top-level value only
value, exists := config.Get("server")  // Returns the entire server map
if !exists {
    log.Fatal("server configuration not found")
}

// For nested values, cast and navigate
if serverMap, ok := value.(map[string]interface{}); ok {
    if port, ok := serverMap["port"].(int); ok {
        fmt.Printf("Port: %d\n", port)
    }
}

// Or use typed getters for top-level values
port, err := config.GetInt("port")          // Works for top-level
host, err := config.GetString("host")       // Works for top-level

Typed Getters

All typed getters work with top-level keys only:

// String values
name, err := config.GetString("app_name")

// Integer values (works with both int and string values)
port, err := config.GetInt("port")

// Unsigned integers
maxConn, err := config.GetUint("max_connections")

// Float values
timeout, err := config.GetFloat("timeout_seconds")

// Boolean values (works with bool and string "true"/"false")
debug, err := config.GetBool("debug_mode")

// Byte sizes with human-readable formats ("10GB", "512MiB", etc.)
maxSize, err := config.GetBytes("max_file_size")

// Get entire config as map
data := config.Data()

Working with Nested Values

Since the API doesn't support dot notation, here's how to work with nested values:

// Given this YAML:
// server:
//   host: localhost
//   port: 8080
//   ssl:
//     enabled: true
//     cert: /etc/ssl/cert.pem

data := config.Data()

// Navigate manually
if server, ok := data["server"].(map[string]interface{}); ok {
    if ssl, ok := server["ssl"].(map[string]interface{}); ok {
        if enabled, ok := ssl["enabled"].(bool); ok {
            fmt.Printf("SSL enabled: %v\n", enabled)
        }
    }
}

// Or unmarshal into a struct
type Config struct {
    Server struct {
        Host string
        Port int
        SSL struct {
            Enabled bool
            Cert    string
        }
    }
}

var cfg Config
data := config.Data()
// Use a YAML/JSON marshaling library to convert data to your struct

Complete Example

Here's a comprehensive example showing all features:

# production.yaml - Complete example with all features

# Top-level values for easy API access
app_name: ${ENV:APP_NAME}
environment: ${ENV:DEPLOY_ENV}
version: ${FILE:/app/VERSION}
build_info: ${EXEC:git describe --always --dirty}

# Type preservation examples
server_port: ${ENV:PORT}                    # Integer: 8080
debug_enabled: ${ENV:DEBUG}                 # Boolean: true/false
request_timeout: ${ENV:TIMEOUT_SECONDS}     # Float: 30.5
max_upload_size: ${ENV:MAX_UPLOAD_SIZE}     # Bytes: "100MB"

# Nested configuration
server:
  listen_address: "0.0.0.0:${ENV:PORT}"    # String concatenation
  workers: ${ENV:WORKER_COUNT}
  tls:
    enabled: ${ENV:TLS_ENABLED}
    cert: ${FILE:/etc/ssl/certs/server.crt}
    key: ${VAULT:secret/data/ssl:private_key}

# Database configuration with nested interpolation
database:
  primary:
    host: ${ENV:DB_HOST_${ENV:DEPLOY_ENV}}  # e.g., DB_HOST_prod
    port: ${ENV:DB_PORT}
    name: ${ENV:DB_NAME}
    user: ${ENV:DB_USER}
    password: ${AWSSM:${ENV:APP_NAME}/${ENV:DEPLOY_ENV}/db_password}
  
  replica:
    host: ${CONSUL:service/db-replica-${ENV:DEPLOY_ENV}/address}
    port: ${CONSUL:service/db-replica-${ENV:DEPLOY_ENV}/port}

# External services
services:
  redis:
    url: ${ETCD:/config/${ENV:APP_NAME}/redis_url}
    password: ${K8SS:default/redis-secret:password}
  
  elasticsearch:
    hosts: ${JSON:/etc/config/services.json:elasticsearch.hosts}
    api_key: ${GCPSM:projects/${ENV:GCP_PROJECT}/secrets/es_api_key}

# Feature flags from various sources
features:
  new_ui: ${CONSUL:features/${ENV:APP_NAME}/new_ui}
  rate_limiting: ${ENV:FEATURE_RATE_LIMITING}
  analytics: ${YAML:/etc/features.yaml:features.analytics.enabled}

# Cloud storage configuration
storage:
  provider: ${ENV:STORAGE_PROVIDER}
  config:
    bucket: ${ENV:STORAGE_BUCKET}
    region: ${ENV:AWS_REGION}
    access_key: ${AWSSM:storage_access_key}
    secret_key: ${AWSSM:storage_secret_key}

# Monitoring and logging
monitoring:
  datadog:
    api_key: ${VAULT:secret/data/monitoring:datadog_api_key}
    app_key: ${VAULT:secret/data/monitoring:datadog_app_key}
    
  sentry:
    dsn: ${ENV:SENTRY_DSN}
    environment: ${ENV:DEPLOY_ENV}
    release: ${EXEC:git rev-parse HEAD}

# Environment variables to inject
env:
  # These will be set as environment variables in the process
  DATABASE_URL: "postgres://${ENV:DB_USER}:${AWSSM:${ENV:APP_NAME}/${ENV:DEPLOY_ENV}/db_password}@${ENV:DB_HOST_${ENV:DEPLOY_ENV}}:${ENV:DB_PORT}/${ENV:DB_NAME}"
  REDIS_URL: ${ETCD:/config/${ENV:APP_NAME}/redis_url}
  NEW_RELIC_LICENSE_KEY: ${AZURESM:https://myvault.vault.azure.net:newrelic-license}
  DD_TRACE_ENABLED: ${ENV:DD_TRACE_ENABLED}

// main.go - Using the complete configuration
package main

import (
    "fmt"
    "log"
    "os"
    
    "git.eeqj.de/sneak/smartconfig"
)

func main() {
    // Load configuration
    config, err := smartconfig.NewFromConfigPath("production.yaml")
    if err != nil {
        log.Fatalf("Failed to load config: %v", err)
    }
    
    // Access top-level typed values
    appName, _ := config.GetString("app_name")
    serverPort, _ := config.GetInt("server_port")
    debugEnabled, _ := config.GetBool("debug_enabled")
    maxUploadSize, _ := config.GetBytes("max_upload_size")
    
    fmt.Printf("Starting %s on port %d (debug: %v)\n", appName, serverPort, debugEnabled)
    fmt.Printf("Max upload size: %d bytes\n", maxUploadSize)
    
    // Access nested configuration
    data := config.Data()
    
    // Database configuration
    if db, ok := data["database"].(map[string]interface{}); ok {
        if primary, ok := db["primary"].(map[string]interface{}); ok {
            dbHost, _ := primary["host"].(string)
            dbPort, _ := primary["port"].(int)
            fmt.Printf("Database: %s:%d\n", dbHost, dbPort)
        }
    }
    
    // Check injected environment variables
    fmt.Printf("DATABASE_URL: %s\n", os.Getenv("DATABASE_URL"))
    fmt.Printf("REDIS_URL: %s\n", os.Getenv("REDIS_URL"))
    
    // Feature flags
    if features, ok := data["features"].(map[string]interface{}); ok {
        if newUI, ok := features["new_ui"].(bool); ok && newUI {
            fmt.Println("New UI is enabled!")
        }
    }
}

Advanced Features

Nested Interpolation

Interpolations can be nested up to 3 levels deep:

# Dynamic environment selection
environment: prod
database:
  host: ${ENV:DB_HOST_${ENV:ENVIRONMENT}}  # Looks for DB_HOST_prod
  
# Multi-level nesting
config:
  value: ${ENV:${ENV:PREFIX}_${ENV:SUFFIX}_KEY}
  
# With different resolvers
secret: ${VAULT:secret/${ENV:APP_NAME}/${ENV:ENVIRONMENT}:password}

Environment Variable Injection

Values under the env key are automatically exported as environment variables:

# These become environment variables in your process
env:
  DATABASE_URL: "postgres://${ENV:DB_USER}:${AWSSM:db-password}@${ENV:DB_HOST}/myapp"
  REDIS_URL: ${CONSUL:service/redis/url}
  API_KEY: ${VAULT:secret/data/external:api_key}
  
# Your application can now use os.Getenv("DATABASE_URL"), etc.

Custom Resolvers

Extend smartconfig with your own resolvers:

// Implement the Resolver interface
type RedisResolver struct {
    client *redis.Client
}

func (r *RedisResolver) Resolve(key string) (string, error) {
    return r.client.Get(key).Result()
}

// Register your resolver
config := smartconfig.New()
config.RegisterResolver("REDIS", &RedisResolver{client: redisClient})

// Use in YAML
// cache_ttl: ${REDIS:config:cache:ttl}

Error Handling

Always handle errors appropriately:

config, err := smartconfig.NewFromAppName("myapp")
if err != nil {
    if os.IsNotExist(err) {
        log.Fatal("Config file not found in /etc/myapp/config.yml")
    }
    log.Fatalf("Failed to load config: %v", err)
}

// Handle missing keys
port, err := config.GetInt("server_port")
if err != nil {
    if strings.Contains(err.Error(), "not found") {
        // Use default
        port = 8080
    } else {
        log.Fatalf("Invalid port configuration: %v", err)
    }
}

// Handle type conversion errors
timeout, err := config.GetFloat("timeout")
if err != nil {
    if strings.Contains(err.Error(), "cannot convert") {
        log.Fatalf("Timeout must be a number, got: %v", err)
    }
}

CLI Tool

A command-line tool is provided for testing and preprocessing configuration files:

# Install the CLI tool
go install git.eeqj.de/sneak/smartconfig/cmd/smartconfig@latest

# Process a config file (outputs YAML)
smartconfig < config.yaml > processed.yaml

# Output as JSON
smartconfig --json < config.yaml > processed.json

# Test interpolation
export DB_PASSWORD="secret123"
echo 'password: ${ENV:DB_PASSWORD}' | smartconfig
# Output: password: secret123

# Type preservation in action
export PORT="8080"
export ENABLED="true"
export TIMEOUT="30.5"
cat <<'EOF' | smartconfig --json
port: ${ENV:PORT}
enabled: ${ENV:ENABLED}
timeout: ${ENV:TIMEOUT}
EOF
# Output: {"enabled":true,"port":8080,"timeout":30.5}

Security Considerations

Arbitrary Code Execution

The EXEC resolver can execute any shell command. This is by design but has security implications:

• Only use trusted configuration files
• Validate configuration sources in production
• Consider disabling EXEC resolver in sensitive environments
• Use appropriate file permissions on configuration files

Secret Management Best Practices

• Never commit secrets to version control
• Use appropriate secret managers (Vault, AWS SM, etc.) for production
• Limit access to configuration files containing secret references
• Rotate secrets regularly
• Monitor secret access

File Access

The FILE resolver can read any file accessible to the process:

• Run applications with minimal required permissions
• Use separate users for different applications
• Consider using secrets managers instead of direct file access

Comparison with Other Libraries

vs. Viper

• smartconfig: Focuses on interpolation from multiple sources with type preservation
• Viper: More features but heavier, includes watching, flags, etc.

vs. envconfig

• smartconfig: YAML-based with interpolation from many sources
• envconfig: Environment variables only, struct tags

vs. koanf

• smartconfig: Simpler API, focused on interpolation
• koanf: More providers and formats, more complex API

Requirements

• Go 1.18 or later
• Valid YAML syntax in configuration files
• Appropriate credentials for cloud providers (AWS, GCP, Azure)
• Network access for remote resolvers (Vault, Consul, etcd)

Contributing

Contributions are welcome! Please ensure:

• Tests pass: make test
• Code is formatted: make fmt
• No linting errors: make lint

License

WTFPL

Author

sneak sneak@sneak.berlin

---

Inspired by Ryan Smith's configuration format ideas.