smartconfig/README.md

# smartconfig

A Go library for YAML configuration files with powerful variable interpolation from multiple sources including environment variables, files, cloud secret managers, and more.

## Table of Contents

- [Key Features](#key-features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Type-Preserving Interpolation](#type-preserving-interpolation)
- [Supported Resolvers](#supported-resolvers)
- [API Reference](#api-reference)
- [Advanced Features](#advanced-features)
- [CLI Tool](#cli-tool)
- [Security Considerations](#security-considerations)
- [Comparison with Other Libraries](#comparison-with-other-libraries)
- [Requirements](#requirements)
- [Contributing](#contributing)
- [License](#license)

## Key Features

- **Type-Preserving Interpolation**: Standalone interpolations preserve their types (numbers, booleans, strings)
- **Multiple Data Sources**: Environment variables, files, cloud secrets (AWS, GCP, Azure), Vault, Consul, K8S, and more
- **Nested Interpolation**: Support for complex variable references like `${ENV:PREFIX_${ENV:SUFFIX}}`
- **JSON5 Support**: JSON resolver supports comments and trailing commas
- **Environment Injection**: Automatically export config values as environment variables
- **Extensible**: Add custom resolvers for your own data sources
- **CLI Tool**: Command-line tool for processing YAML files with interpolation

## Installation

```bash
go get git.eeqj.de/sneak/smartconfig
```

## Quick Start

### Basic Example

```yaml
# config.yaml
# Showcase the power of multiple resolver types
app_name: ${ENV:APP_NAME}
version: ${FILE:/app/VERSION}                          # Read from file
port: ${ENV:PORT}                                      # Number if PORT="8080"
debug_mode: ${ENV:DEBUG_MODE}                          # Boolean if "true"/"false"

# External service configuration from JSON config file
services: ${JSON:/etc/config/services.json:production} # Load entire object

# API credentials from cloud secret managers
api_keys:
  stripe: ${GCPSM:projects/myproject/secrets/stripe-api-key}
  sendgrid: ${AWSSM:prod/sendgrid-api-key}
  twilio: ${VAULT:secret/data/api:twilio_key}
  datadog: ${ENV:DD_API_KEY}  # Fallback to env for local dev

# Database configuration with mixed sources
database:
  host: ${CONSUL:service/postgres/address}
  port: ${CONSUL:service/postgres/port}
  name: ${ENV:DB_NAME}
  username: ${ENV:DB_USER}
  password: ${GCPSM:projects/myproject/secrets/db-password}
  
  # SSL configuration from multiple sources
  ssl:
    enabled: ${ENV:DB_SSL_ENABLED}
    ca_cert: ${FILE:/etc/ssl/db-ca.crt}
    client_cert: ${K8SS:default/db-certs:client.crt}

# Feature flags from JSON file
features: ${JSON:/etc/config/features.json:${ENV:ENVIRONMENT}}

# Server configuration
server:
  listen: "0.0.0.0:${ENV:PORT}"
  workers: ${EXEC:nproc}                               # Dynamic based on CPU cores
  hostname: ${EXEC:hostname -f}

env:
  # Export these as environment variables for child processes
  DATABASE_URL: "postgres://${ENV:DB_USER}:${GCPSM:projects/myproject/secrets/db-password}@${CONSUL:service/postgres/address}:5432/${ENV:DB_NAME}"
  NEW_RELIC_LICENSE: ${AWSSM:monitoring/newrelic-license}
```

### Go Code

```go
package main

import (
    "fmt"
    "log"
    "os"
    
    "git.eeqj.de/sneak/smartconfig"
)

func main() {
    // Load from /etc/myapp/config.yml
    config, err := smartconfig.NewFromAppName("myapp")
    if err != nil {
        log.Fatal(err)
    }
    
    // Access typed values
    appName, _ := config.GetString("app_name")
    port, _ := config.GetInt("port")
    debugMode, _ := config.GetBool("debug_mode")
    
    fmt.Printf("Starting %s on port %d (debug: %v)\n", appName, port, debugMode)
    
    // Access nested values
    data := config.Data()
    
    // API keys configuration
    if apiKeys, ok := data["api_keys"].(map[string]interface{}); ok {
        if stripe, ok := apiKeys["stripe"].(string); ok {
            fmt.Printf("Stripe API key loaded: %s...\n", stripe[:8])
        }
    }
    
    // Database configuration
    if db, ok := data["database"].(map[string]interface{}); ok {
        host, _ := db["host"].(string)
        port, _ := db["port"].(int)
        fmt.Printf("Database: %s:%d\n", host, port)
    }
    
    // Check loaded services from JSON file
    if services, ok := data["services"].(map[string]interface{}); ok {
        fmt.Printf("Loaded %d services from JSON config\n", len(services))
    }
    
    // Environment variables are now available
    fmt.Printf("DATABASE_URL env var: %s\n", os.Getenv("DATABASE_URL"))
}
```

Example JSON files referenced above:

```json
// /etc/config/services.json
{
  "production": {
    "api": {
      "endpoint": "https://api.example.com",
      "timeout": 30,
      "retries": 3
    },
    "cache": {
      "provider": "redis",
      "ttl": 3600
    }
  },
  "staging": {
    "api": {
      "endpoint": "https://api-staging.example.com",
      "timeout": 60,
      "retries": 5
    }
  }
}

// /etc/config/features.json
{
  "production": {
    "new_ui": false,
    "rate_limiting": true,
    "analytics": true
  },
  "staging": {
    "new_ui": true,
    "rate_limiting": true,
    "analytics": false
  }
}
```

## Type-Preserving Interpolation

smartconfig parses YAML first, then performs interpolation. This allows proper type preservation:

### How It Works

```yaml
# Standalone interpolations preserve types
port: ${ENV:PORT}           # If PORT="8080", becomes integer 8080
enabled: ${ENV:ENABLED}     # If ENABLED="true", becomes boolean true
timeout: ${ENV:TIMEOUT}     # If TIMEOUT="30.5", becomes float 30.5

# Mixed content always returns strings
message: "Hello ${ENV:USER}!"              # Always a string
port_label: "Port: ${ENV:PORT}"            # Always a string
debug_flag: "debug-${ENV:DEBUG}"           # Always a string

# Force string by adding any prefix/suffix
port_string: "${ENV:PORT}-suffix"          # Forces string output
bool_string: "prefix-${ENV:ENABLED}"       # Forces string output
```

### Type Conversion Rules

For standalone interpolations, smartconfig automatically converts:
- `"true"` → `true` (boolean)
- `"false"` → `false` (boolean)
- Numeric strings → numbers (int or float64)
- Everything else → string

## Supported Resolvers

### Local Resolvers

#### ENV - Environment Variables
```yaml
# Basic usage
api_key: ${ENV:API_KEY}

# With nested interpolation
database: ${ENV:DB_${ENV:ENVIRONMENT}}
```

#### FILE - Read File Contents
```yaml
# Read entire file (trimmed)
ssl_cert: ${FILE:/etc/ssl/cert.pem}
machine_id: ${FILE:/etc/machine-id}

# Read system files
cpu_temp: ${FILE:/sys/class/thermal/thermal_zone0/temp}
```

#### EXEC - Execute Shell Commands
```yaml
# Simple commands
hostname: ${EXEC:hostname -f}
timestamp: ${EXEC:date +%s}
git_hash: ${EXEC:git rev-parse HEAD}

# Complex commands with pipes
users_count: ${EXEC:who | wc -l}
docker_running: ${EXEC:docker ps -q | wc -l}
```

#### JSON - Read from JSON Files (with JSON5 support)
```yaml
# Supports gjson path syntax
api_endpoint: ${JSON:/etc/config.json:services.api.endpoint}
first_server: ${JSON:/etc/servers.json:servers.0.host}
all_features: ${JSON:/etc/features.json:features}

# JSON5 features: comments and trailing commas supported
config_value: ${JSON:/etc/app.json5:debug.level}
```

#### YAML - Read from YAML Files
```yaml
# Read specific paths from YAML files
db_config: ${YAML:/etc/database.yml:production.primary}
replica_host: ${YAML:/etc/database.yml:production.replica.host}
```

### Cloud Secret Managers

#### AWS Secrets Manager
```yaml
# Requires AWS credentials via:
# - Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
# - IAM instance role (on EC2)
# - AWS config files (~/.aws/credentials)

database:
  password: ${AWSSM:prod/db/password}
  api_key: ${AWSSM:external-api-key}
  
# With versioning
secret: ${AWSSM:mysecret:AWSCURRENT}
```

#### Google Cloud Secret Manager
```yaml
# Requires GCP credentials via:
# - Environment variable: GOOGLE_APPLICATION_CREDENTIALS
# - Default service account (on GCE/GKE)
# - gcloud auth application-default login

secrets:
  api_key: ${GCPSM:projects/my-project/secrets/api-key}
  db_pass: ${GCPSM:projects/my-project/secrets/db-password/versions/latest}
```

#### Azure Key Vault
```yaml
# Requires Azure credentials via:
# - Environment variables: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID
# - Managed Service Identity (on Azure VMs)
# - Azure CLI authentication (az login)

credentials:
  cert: ${AZURESM:https://myvault.vault.azure.net:mycert}
  key: ${AZURESM:https://myvault.vault.azure.net:mykey}
```

### Infrastructure Tools

#### HashiCorp Vault
```yaml
# Requires VAULT_ADDR and VAULT_TOKEN configured
database:
  password: ${VAULT:secret/data/myapp:db_password}
  api_key: ${VAULT:secret/data/external:api_key}
  
# KV v2 paths
secret: ${VAULT:secret/data/myapp:password}
```

#### HashiCorp Consul
```yaml
# Requires CONSUL_HTTP_ADDR configured
config:
  feature_flags: ${CONSUL:myapp/features}
  db_host: ${CONSUL:service/database/host}
```

#### Kubernetes Secrets
```yaml
# Requires in-cluster or kubeconfig access
database:
  password: ${K8SS:default/db-secret:password}
  cert: ${K8SS:kube-system/tls-secret:tls.crt}
```

#### etcd
```yaml
# Requires ETCD_ENDPOINTS configured
config:
  cluster_size: ${ETCD:/cluster/size}
  node_role: ${ETCD:/node/role}
```

## API Reference

### Loading Configuration

```go
// Load from /etc/{appname}/config.yml
config, err := smartconfig.NewFromAppName("myapp")

// Load from specific file
config, err := smartconfig.NewFromConfigPath("/path/to/config.yaml")

// Load from io.Reader
reader := strings.NewReader(yamlContent)
config, err := smartconfig.NewFromReader(reader)
```

### Accessing Values

**Important**: The `Get()` method only supports top-level keys. For nested values, you need to navigate the structure manually.

```go
// Get top-level value only
value, exists := config.Get("server")  // Returns the entire server map
if !exists {
    log.Fatal("server configuration not found")
}

// For nested values, cast and navigate
if serverMap, ok := value.(map[string]interface{}); ok {
    if port, ok := serverMap["port"].(int); ok {
        fmt.Printf("Port: %d\n", port)
    }
}

// Or use typed getters for top-level values
port, err := config.GetInt("port")          // Works for top-level
host, err := config.GetString("host")       // Works for top-level
```

### Typed Getters

All typed getters work with top-level keys only:

```go
// String values
name, err := config.GetString("app_name")

// Integer values (works with both int and string values)
port, err := config.GetInt("port")

// Unsigned integers
maxConn, err := config.GetUint("max_connections")

// Float values
timeout, err := config.GetFloat("timeout_seconds")

// Boolean values (works with bool and string "true"/"false")
debug, err := config.GetBool("debug_mode")

// Byte sizes with human-readable formats ("10GB", "512MiB", etc.)
maxSize, err := config.GetBytes("max_file_size")

// Get entire config as map
data := config.Data()
```

### Working with Nested Values

Since the API doesn't support dot notation, here's how to work with nested values:

```go
// Given this YAML:
// server:
//   host: localhost
//   port: 8080
//   ssl:
//     enabled: true
//     cert: /etc/ssl/cert.pem

data := config.Data()

// Navigate manually
if server, ok := data["server"].(map[string]interface{}); ok {
    if ssl, ok := server["ssl"].(map[string]interface{}); ok {
        if enabled, ok := ssl["enabled"].(bool); ok {
            fmt.Printf("SSL enabled: %v\n", enabled)
        }
    }
}

// Or unmarshal into a struct
type Config struct {
    Server struct {
        Host string
        Port int
        SSL struct {
            Enabled bool
            Cert    string
        }
    }
}

var cfg Config
data := config.Data()
// Use a YAML/JSON marshaling library to convert data to your struct
```

## Complete Example

Here's a comprehensive example showing all features:

```yaml
# production.yaml - Complete example with all features

# Top-level values for easy API access
app_name: ${ENV:APP_NAME}
environment: ${ENV:DEPLOY_ENV}
version: ${FILE:/app/VERSION}
build_info: ${EXEC:git describe --always --dirty}

# Type preservation examples
server_port: ${ENV:PORT}                    # Integer: 8080
debug_enabled: ${ENV:DEBUG}                 # Boolean: true/false
request_timeout: ${ENV:TIMEOUT_SECONDS}     # Float: 30.5
max_upload_size: ${ENV:MAX_UPLOAD_SIZE}     # Bytes: "100MB"

# Nested configuration
server:
  listen_address: "0.0.0.0:${ENV:PORT}"    # String concatenation
  workers: ${ENV:WORKER_COUNT}
  tls:
    enabled: ${ENV:TLS_ENABLED}
    cert: ${FILE:/etc/ssl/certs/server.crt}
    key: ${VAULT:secret/data/ssl:private_key}

# Database configuration with nested interpolation
database:
  primary:
    host: ${ENV:DB_HOST_${ENV:DEPLOY_ENV}}  # e.g., DB_HOST_prod
    port: ${ENV:DB_PORT}
    name: ${ENV:DB_NAME}
    user: ${ENV:DB_USER}
    password: ${AWSSM:${ENV:APP_NAME}/${ENV:DEPLOY_ENV}/db_password}
  
  replica:
    host: ${CONSUL:service/db-replica-${ENV:DEPLOY_ENV}/address}
    port: ${CONSUL:service/db-replica-${ENV:DEPLOY_ENV}/port}

# External services
services:
  redis:
    url: ${ETCD:/config/${ENV:APP_NAME}/redis_url}
    password: ${K8SS:default/redis-secret:password}
  
  elasticsearch:
    hosts: ${JSON:/etc/config/services.json:elasticsearch.hosts}
    api_key: ${GCPSM:projects/${ENV:GCP_PROJECT}/secrets/es_api_key}

# Feature flags from various sources
features:
  new_ui: ${CONSUL:features/${ENV:APP_NAME}/new_ui}
  rate_limiting: ${ENV:FEATURE_RATE_LIMITING}
  analytics: ${YAML:/etc/features.yaml:features.analytics.enabled}

# Cloud storage configuration
storage:
  provider: ${ENV:STORAGE_PROVIDER}
  config:
    bucket: ${ENV:STORAGE_BUCKET}
    region: ${ENV:AWS_REGION}
    access_key: ${AWSSM:storage_access_key}
    secret_key: ${AWSSM:storage_secret_key}

# Monitoring and logging
monitoring:
  datadog:
    api_key: ${VAULT:secret/data/monitoring:datadog_api_key}
    app_key: ${VAULT:secret/data/monitoring:datadog_app_key}
    
  sentry:
    dsn: ${ENV:SENTRY_DSN}
    environment: ${ENV:DEPLOY_ENV}
    release: ${EXEC:git rev-parse HEAD}

# Environment variables to inject
env:
  # These will be set as environment variables in the process
  DATABASE_URL: "postgres://${ENV:DB_USER}:${AWSSM:${ENV:APP_NAME}/${ENV:DEPLOY_ENV}/db_password}@${ENV:DB_HOST_${ENV:DEPLOY_ENV}}:${ENV:DB_PORT}/${ENV:DB_NAME}"
  REDIS_URL: ${ETCD:/config/${ENV:APP_NAME}/redis_url}
  NEW_RELIC_LICENSE_KEY: ${AZURESM:https://myvault.vault.azure.net:newrelic-license}
  DD_TRACE_ENABLED: ${ENV:DD_TRACE_ENABLED}
```

```go
// main.go - Using the complete configuration
package main

import (
    "fmt"
    "log"
    "os"
    
    "git.eeqj.de/sneak/smartconfig"
)

func main() {
    // Load configuration
    config, err := smartconfig.NewFromConfigPath("production.yaml")
    if err != nil {
        log.Fatalf("Failed to load config: %v", err)
    }
    
    // Access top-level typed values
    appName, _ := config.GetString("app_name")
    serverPort, _ := config.GetInt("server_port")
    debugEnabled, _ := config.GetBool("debug_enabled")
    maxUploadSize, _ := config.GetBytes("max_upload_size")
    
    fmt.Printf("Starting %s on port %d (debug: %v)\n", appName, serverPort, debugEnabled)
    fmt.Printf("Max upload size: %d bytes\n", maxUploadSize)
    
    // Access nested configuration
    data := config.Data()
    
    // Database configuration
    if db, ok := data["database"].(map[string]interface{}); ok {
        if primary, ok := db["primary"].(map[string]interface{}); ok {
            dbHost, _ := primary["host"].(string)
            dbPort, _ := primary["port"].(int)
            fmt.Printf("Database: %s:%d\n", dbHost, dbPort)
        }
    }
    
    // Check injected environment variables
    fmt.Printf("DATABASE_URL: %s\n", os.Getenv("DATABASE_URL"))
    fmt.Printf("REDIS_URL: %s\n", os.Getenv("REDIS_URL"))
    
    // Feature flags
    if features, ok := data["features"].(map[string]interface{}); ok {
        if newUI, ok := features["new_ui"].(bool); ok && newUI {
            fmt.Println("New UI is enabled!")
        }
    }
}
```

## Advanced Features

### Nested Interpolation

Interpolations can be nested up to 3 levels deep:

```yaml
# Dynamic environment selection
environment: prod
database:
  host: ${ENV:DB_HOST_${ENV:ENVIRONMENT}}  # Looks for DB_HOST_prod
  
# Multi-level nesting
config:
  value: ${ENV:${ENV:PREFIX}_${ENV:SUFFIX}_KEY}
  
# With different resolvers
secret: ${VAULT:secret/${ENV:APP_NAME}/${ENV:ENVIRONMENT}:password}
```

### Environment Variable Injection

Values under the `env` key are automatically exported as environment variables:

```yaml
# These become environment variables in your process
env:
  DATABASE_URL: "postgres://${ENV:DB_USER}:${AWSSM:db-password}@${ENV:DB_HOST}/myapp"
  REDIS_URL: ${CONSUL:service/redis/url}
  API_KEY: ${VAULT:secret/data/external:api_key}
  
# Your application can now use os.Getenv("DATABASE_URL"), etc.
```

### Custom Resolvers

Extend smartconfig with your own resolvers:

```go
// Implement the Resolver interface
type RedisResolver struct {
    client *redis.Client
}

func (r *RedisResolver) Resolve(key string) (string, error) {
    return r.client.Get(key).Result()
}

// Register your resolver
config := smartconfig.New()
config.RegisterResolver("REDIS", &RedisResolver{client: redisClient})

// Use in YAML
// cache_ttl: ${REDIS:config:cache:ttl}
```

### Error Handling

Always handle errors appropriately:

```go
config, err := smartconfig.NewFromAppName("myapp")
if err != nil {
    if os.IsNotExist(err) {
        log.Fatal("Config file not found in /etc/myapp/config.yml")
    }
    log.Fatalf("Failed to load config: %v", err)
}

// Handle missing keys
port, err := config.GetInt("server_port")
if err != nil {
    if strings.Contains(err.Error(), "not found") {
        // Use default
        port = 8080
    } else {
        log.Fatalf("Invalid port configuration: %v", err)
    }
}

// Handle type conversion errors
timeout, err := config.GetFloat("timeout")
if err != nil {
    if strings.Contains(err.Error(), "cannot convert") {
        log.Fatalf("Timeout must be a number, got: %v", err)
    }
}
```

## CLI Tool

A command-line tool is provided for testing and preprocessing configuration files:

```bash
# Install the CLI tool
go install git.eeqj.de/sneak/smartconfig/cmd/smartconfig@latest

# Process a config file (outputs YAML)
smartconfig < config.yaml > processed.yaml

# Output as JSON
smartconfig --json < config.yaml > processed.json

# Test interpolation
export DB_PASSWORD="secret123"
echo 'password: ${ENV:DB_PASSWORD}' | smartconfig
# Output: password: secret123

# Type preservation in action
export PORT="8080"
export ENABLED="true"
export TIMEOUT="30.5"
cat <<'EOF' | smartconfig --json
port: ${ENV:PORT}
enabled: ${ENV:ENABLED}
timeout: ${ENV:TIMEOUT}
EOF
# Output: {"enabled":true,"port":8080,"timeout":30.5}
```

## Security Considerations

### Arbitrary Code Execution
The `EXEC` resolver can execute any shell command. This is by design but has security implications:
- Only use trusted configuration files
- Validate configuration sources in production
- Consider disabling EXEC resolver in sensitive environments
- Use appropriate file permissions on configuration files

### Secret Management Best Practices
- Never commit secrets to version control
- Use appropriate secret managers (Vault, AWS SM, etc.) for production
- Limit access to configuration files containing secret references
- Rotate secrets regularly
- Monitor secret access

### File Access
The `FILE` resolver can read any file accessible to the process:
- Run applications with minimal required permissions
- Use separate users for different applications
- Consider using secrets managers instead of direct file access

## Comparison with Other Libraries

### vs. Viper
- **smartconfig**: Focuses on interpolation from multiple sources with type preservation
- **Viper**: More features but heavier, includes watching, flags, etc.

### vs. envconfig
- **smartconfig**: YAML-based with interpolation from many sources
- **envconfig**: Environment variables only, struct tags

### vs. koanf
- **smartconfig**: Simpler API, focused on interpolation
- **koanf**: More providers and formats, more complex API

## Requirements

- Go 1.18 or later
- Valid YAML syntax in configuration files
- Appropriate credentials for cloud providers (AWS, GCP, Azure)
- Network access for remote resolvers (Vault, Consul, etcd)

## Contributing

Contributions are welcome! Please ensure:
- Tests pass: `make test`
- Code is formatted: `make fmt`
- No linting errors: `make lint`

## License

WTFPL

## Author

sneak <sneak@sneak.berlin>

---

*Inspired by Ryan Smith's configuration format ideas.*