diff --git a/README.md b/README.md index 635e550..67281bc 100644 --- a/README.md +++ b/README.md @@ -1,131 +1,32 @@ # smartconfig -smartconfig is a go library that reads YAML configuration files from the -filesystem and interpolates variables in them from several sources, then -parses the resulting YAML. +A Go library for YAML configuration files with powerful variable interpolation from multiple sources including environment variables, files, cloud secret managers, and more. -**Important:** smartconfig only supports valid YAML files as input. The file -must be valid YAML syntax before interpolation occurs. +## Table of Contents -This was inspired by Ryan Smith and one of his config file formats/parsers. -It struck me as very clever and I wanted to begin using it. +- [Key Features](#key-features) +- [Installation](#installation) +- [Quick Start](#quick-start) +- [Type-Preserving Interpolation](#type-preserving-interpolation) +- [Supported Resolvers](#supported-resolvers) +- [API Reference](#api-reference) +- [Advanced Features](#advanced-features) +- [CLI Tool](#cli-tool) +- [Security Considerations](#security-considerations) +- [Comparison with Other Libraries](#comparison-with-other-libraries) +- [Requirements](#requirements) +- [Contributing](#contributing) +- [License](#license) -# concept +## Key Features -Configuration files often need to store secrets, and these secrets come from -various sources such as environment variables, cloud provider secrets -managers, or other secret management systems. Instead of having to -implement all of these APIs in your application, tightly binding it to your -platform (and locking you in to that vendor), this allows your config file -to serve as pointers to the various sources of secrets, supporting pluggable -secret management sources. - -Yes, it supports shelling out to external commands to get values. This -equates being able to write your config file with arbitrary code execution, -which may not be the case in your environment, but is a baked-in assumption -here (and one I think will be fine/correct for 99% of users). - -It has only one magic property other than the interpolation: anything -specified under the top level key "env" will be interpolated and added, -to the environment of the process that reads the config file. This allows a -config file to serve as a bridge between fancy backend secret management -services and "traditional" configuration via env vars. - -## Type-Preserving Interpolation - -smartconfig uses a YAML-first approach: it parses the YAML file first, then -walks the structure to perform interpolations. This means: - -1. **Type preservation**: Interpolated values can return appropriate types (numbers, booleans, strings) -2. **User control**: The YAML syntax determines the output type -3. **Clean design**: The YAML parser handles all quoting and escaping - -### How It Works - -When an interpolation is the entire value (not embedded in other text), smartconfig -attempts to convert the resolved value to the appropriate type: - -```yaml -# Numeric output - if ENV:PORT is "8080", this becomes the number 8080 -port: ${ENV:PORT} - -# Boolean output - if ENV:ENABLED is "true", this becomes the boolean true -enabled: ${ENV:ENABLED} - -# String output - mixed content always returns strings -message: "Hello ${ENV:NAME}!" - -# Force string output - add any prefix/suffix -port_str: "port-${ENV:PORT}" -``` - -### Type Conversion Rules - -For standalone interpolations (`value: ${...}`), smartconfig converts: -- `"true"` → `true` (boolean) -- `"false"` → `false` (boolean) -- `"123"` → `123` (integer) -- `"12.5"` → `12.5` (float) -- Everything else → string - -### Important Notes - -- YAML parser removes quotes, so `"${ENV:VAR}"` and `${ENV:VAR}` are treated identically -- To force string output, embed the interpolation in other text: `"prefix-${ENV:VAR}"` -- Mixed content (text with embedded interpolations) always returns strings -- Nested interpolations are fully supported: `${ENV:PREFIX_${ENV:SUFFIX}}` - -# Usage - -```yaml -# config.yaml - -name: ${ENV:APPLICATION_NAME} -host: ${EXEC:hostname -s} -port: ${ENV:PORT} - -vhost: - tls_sni_host: ${JSON:/etc/config/hosts.json:tls_sni_host.0} - -machine: - id: ${FILE:/etc/machine-id} - temperature: ${FILE:/sys/class/thermal/thermal_zone0/temp} - num_cpus: ${EXEC:nproc} - -api: - root_user: "admin" - root_password: ${AWSSM:root_password} - -db: - host: ${GCPSM:${ENV:APPLICATION_NAME}_DB_HOST} - user: ${GCPSM:${ENV:APPLICATION_NAME}_DB_USER} - password: ${GCPSM:${ENV:APPLICATION_NAME}_DB_PASSWORD} - -external: - google_api_key: ${CONSUL:secret:google_api_key} - twilio_api_key: ${VAULT:secret:twilio_api_key} - -env: - ENCRYPTION_PUBLIC_KEY: ${EXEC:secret get myapp/encryption_public_key} - SLACK_WEBHOOK_URL: ${FILE:/etc/config/slack_webhook_url.txt} -``` - -# Supported Providers - -* ENV - environment variables -* EXEC - shell out to an external command -* AWSSM - AWS Secrets Manager -* GCPSM - Google Cloud Secret Manager -* VAULT - HashiCorp Vault -* CONSUL - HashiCorp Consul KV Store -* AZURESM - Azure Key Vault -* K8SS - Kubernetes Secrets -* FILE - read from a file -* JSON - read from a JSON file (supports JSON5 syntax including comments and trailing commas) -* YAML - read from a YAML file -* ETCD - etcd key-value store - -# API Documentation +- **Type-Preserving Interpolation**: Standalone interpolations preserve their types (numbers, booleans, strings) +- **Multiple Data Sources**: Environment variables, files, cloud secrets (AWS, GCP, Azure), Vault, Consul, K8S, and more +- **Nested Interpolation**: Support for complex variable references like `${ENV:PREFIX_${ENV:SUFFIX}}` +- **JSON5 Support**: JSON resolver supports comments and trailing commas +- **Environment Injection**: Automatically export config values as environment variables +- **Extensible**: Add custom resolvers for your own data sources +- **CLI Tool**: Command-line tool for processing YAML files with interpolation ## Installation @@ -133,7 +34,42 @@ env: go get git.eeqj.de/sneak/smartconfig ``` -## Basic Usage +## Quick Start + +### Basic Example + +```yaml +# config.yaml +# Example with top-level values for easy access +app_name: ${ENV:APP_NAME} +port: ${ENV:PORT} # Becomes number 8080 if PORT="8080" +debug_mode: ${ENV:DEBUG_MODE} # Becomes boolean true if DEBUG_MODE="true" +cache_size_bytes: ${ENV:CACHE_SIZE} # Human-readable: "100MB", "1.5GB" + +# Nested configuration for organization +server: + host: localhost + port: ${ENV:SERVER_PORT} + name: "server-${ENV:INSTANCE_ID}" # Always a string due to mixed content + +database: + host: ${ENV:DB_HOST} + port: ${ENV:DB_PORT} + username: ${ENV:DB_USER} + password: ${VAULT:secret/data/myapp:db_password} + +features: + cache_enabled: ${ENV:CACHE_ENABLED} # Boolean if "true"/"false" + max_connections: ${ENV:MAX_CONN} # Number if numeric string + debug: ${ENV:FEATURES_DEBUG} + +env: + # These will be exported as environment variables + API_TOKEN: ${AWSSM:api-token} + SLACK_WEBHOOK: ${FILE:/etc/secrets/slack_webhook.txt} +``` + +### Go Code ```go package main @@ -146,261 +82,623 @@ import ( ) func main() { - // Load configuration from /etc/myapp/config.yml + // Load from /etc/myapp/config.yml config, err := smartconfig.NewFromAppName("myapp") if err != nil { log.Fatal(err) } - // Or load from a specific path - config, err = smartconfig.NewFromConfigPath("/path/to/config.yaml") + // Access typed values (top-level keys only) + port, err := config.GetInt("port") if err != nil { log.Fatal(err) } - // Access configuration values - dbHost, _ := config.GetString("db_host") - fmt.Printf("Database host: %s\n", dbHost) - - // Get entire configuration as a map + // For nested values, navigate manually data := config.Data() - fmt.Printf("Full config: %+v\n", data) + var debugMode bool + if features, ok := data["features"].(map[string]interface{}); ok { + if debug, ok := features["debug"].(bool); ok { + debugMode = debug + } + } + + // Human-readable byte sizes (top-level) + cacheSize, err := config.GetBytes("cache_size_bytes") + if err != nil { + log.Fatal(err) + } + + fmt.Printf("Server running on port %d (debug: %v)\n", port, debugMode) + fmt.Printf("Cache size: %d bytes\n", cacheSize) } ``` -## Loading Configuration +## Type-Preserving Interpolation -### From App Name (looks in /etc/appname/config.yml) +smartconfig parses YAML first, then performs interpolation. This allows proper type preservation: + +### How It Works + +```yaml +# Standalone interpolations preserve types +port: ${ENV:PORT} # If PORT="8080", becomes integer 8080 +enabled: ${ENV:ENABLED} # If ENABLED="true", becomes boolean true +timeout: ${ENV:TIMEOUT} # If TIMEOUT="30.5", becomes float 30.5 + +# Mixed content always returns strings +message: "Hello ${ENV:USER}!" # Always a string +port_label: "Port: ${ENV:PORT}" # Always a string +debug_flag: "debug-${ENV:DEBUG}" # Always a string + +# Force string by adding any prefix/suffix +port_string: "${ENV:PORT}-suffix" # Forces string output +bool_string: "prefix-${ENV:ENABLED}" # Forces string output +``` + +### Type Conversion Rules + +For standalone interpolations, smartconfig automatically converts: +- `"true"` → `true` (boolean) +- `"false"` → `false` (boolean) +- Numeric strings → numbers (int or float64) +- Everything else → string + +## Supported Resolvers + +### Local Resolvers + +#### ENV - Environment Variables +```yaml +# Basic usage +api_key: ${ENV:API_KEY} + +# With nested interpolation +database: ${ENV:DB_${ENV:ENVIRONMENT}} +``` + +#### FILE - Read File Contents +```yaml +# Read entire file (trimmed) +ssl_cert: ${FILE:/etc/ssl/cert.pem} +machine_id: ${FILE:/etc/machine-id} + +# Read system files +cpu_temp: ${FILE:/sys/class/thermal/thermal_zone0/temp} +``` + +#### EXEC - Execute Shell Commands +```yaml +# Simple commands +hostname: ${EXEC:hostname -f} +timestamp: ${EXEC:date +%s} +git_hash: ${EXEC:git rev-parse HEAD} + +# Complex commands with pipes +users_count: ${EXEC:who | wc -l} +docker_running: ${EXEC:docker ps -q | wc -l} +``` + +#### JSON - Read from JSON Files (with JSON5 support) +```yaml +# Supports gjson path syntax +api_endpoint: ${JSON:/etc/config.json:services.api.endpoint} +first_server: ${JSON:/etc/servers.json:servers.0.host} +all_features: ${JSON:/etc/features.json:features} + +# JSON5 features: comments and trailing commas supported +config_value: ${JSON:/etc/app.json5:debug.level} +``` + +#### YAML - Read from YAML Files +```yaml +# Read specific paths from YAML files +db_config: ${YAML:/etc/database.yml:production.primary} +replica_host: ${YAML:/etc/database.yml:production.replica.host} +``` + +### Cloud Secret Managers + +#### AWS Secrets Manager +```yaml +# Requires AWS credentials via: +# - Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY +# - IAM instance role (on EC2) +# - AWS config files (~/.aws/credentials) + +database: + password: ${AWSSM:prod/db/password} + api_key: ${AWSSM:external-api-key} + +# With versioning +secret: ${AWSSM:mysecret:AWSCURRENT} +``` + +#### Google Cloud Secret Manager +```yaml +# Requires GCP credentials via: +# - Environment variable: GOOGLE_APPLICATION_CREDENTIALS +# - Default service account (on GCE/GKE) +# - gcloud auth application-default login + +secrets: + api_key: ${GCPSM:projects/my-project/secrets/api-key} + db_pass: ${GCPSM:projects/my-project/secrets/db-password/versions/latest} +``` + +#### Azure Key Vault +```yaml +# Requires Azure credentials via: +# - Environment variables: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID +# - Managed Service Identity (on Azure VMs) +# - Azure CLI authentication (az login) + +credentials: + cert: ${AZURESM:https://myvault.vault.azure.net:mycert} + key: ${AZURESM:https://myvault.vault.azure.net:mykey} +``` + +### Infrastructure Tools + +#### HashiCorp Vault +```yaml +# Requires VAULT_ADDR and VAULT_TOKEN configured +database: + password: ${VAULT:secret/data/myapp:db_password} + api_key: ${VAULT:secret/data/external:api_key} + +# KV v2 paths +secret: ${VAULT:secret/data/myapp:password} +``` + +#### HashiCorp Consul +```yaml +# Requires CONSUL_HTTP_ADDR configured +config: + feature_flags: ${CONSUL:myapp/features} + db_host: ${CONSUL:service/database/host} +``` + +#### Kubernetes Secrets +```yaml +# Requires in-cluster or kubeconfig access +database: + password: ${K8SS:default/db-secret:password} + cert: ${K8SS:kube-system/tls-secret:tls.crt} +``` + +#### etcd +```yaml +# Requires ETCD_ENDPOINTS configured +config: + cluster_size: ${ETCD:/cluster/size} + node_role: ${ETCD:/node/role} +``` + +## API Reference + +### Loading Configuration + +```go +// Load from /etc/{appname}/config.yml +config, err := smartconfig.NewFromAppName("myapp") + +// Load from specific file +config, err := smartconfig.NewFromConfigPath("/path/to/config.yaml") + +// Load from io.Reader +reader := strings.NewReader(yamlContent) +config, err := smartconfig.NewFromReader(reader) +``` + +### Accessing Values + +**Important**: The `Get()` method only supports top-level keys. For nested values, you need to navigate the structure manually. + +```go +// Get top-level value only +value, exists := config.Get("server") // Returns the entire server map +if !exists { + log.Fatal("server configuration not found") +} + +// For nested values, cast and navigate +if serverMap, ok := value.(map[string]interface{}); ok { + if port, ok := serverMap["port"].(int); ok { + fmt.Printf("Port: %d\n", port) + } +} + +// Or use typed getters for top-level values +port, err := config.GetInt("port") // Works for top-level +host, err := config.GetString("host") // Works for top-level +``` + +### Typed Getters + +All typed getters work with top-level keys only: + +```go +// String values +name, err := config.GetString("app_name") + +// Integer values (works with both int and string values) +port, err := config.GetInt("port") + +// Unsigned integers +maxConn, err := config.GetUint("max_connections") + +// Float values +timeout, err := config.GetFloat("timeout_seconds") + +// Boolean values (works with bool and string "true"/"false") +debug, err := config.GetBool("debug_mode") + +// Byte sizes with human-readable formats ("10GB", "512MiB", etc.) +maxSize, err := config.GetBytes("max_file_size") + +// Get entire config as map +data := config.Data() +``` + +### Working with Nested Values + +Since the API doesn't support dot notation, here's how to work with nested values: + +```go +// Given this YAML: +// server: +// host: localhost +// port: 8080 +// ssl: +// enabled: true +// cert: /etc/ssl/cert.pem + +data := config.Data() + +// Navigate manually +if server, ok := data["server"].(map[string]interface{}); ok { + if ssl, ok := server["ssl"].(map[string]interface{}); ok { + if enabled, ok := ssl["enabled"].(bool); ok { + fmt.Printf("SSL enabled: %v\n", enabled) + } + } +} + +// Or unmarshal into a struct +type Config struct { + Server struct { + Host string + Port int + SSL struct { + Enabled bool + Cert string + } + } +} + +var cfg Config +data := config.Data() +// Use a YAML/JSON marshaling library to convert data to your struct +``` + +## Complete Example + +Here's a comprehensive example showing all features: + +```yaml +# production.yaml - Complete example with all features + +# Top-level values for easy API access +app_name: ${ENV:APP_NAME} +environment: ${ENV:DEPLOY_ENV} +version: ${FILE:/app/VERSION} +build_info: ${EXEC:git describe --always --dirty} + +# Type preservation examples +server_port: ${ENV:PORT} # Integer: 8080 +debug_enabled: ${ENV:DEBUG} # Boolean: true/false +request_timeout: ${ENV:TIMEOUT_SECONDS} # Float: 30.5 +max_upload_size: ${ENV:MAX_UPLOAD_SIZE} # Bytes: "100MB" + +# Nested configuration +server: + listen_address: "0.0.0.0:${ENV:PORT}" # String concatenation + workers: ${ENV:WORKER_COUNT} + tls: + enabled: ${ENV:TLS_ENABLED} + cert: ${FILE:/etc/ssl/certs/server.crt} + key: ${VAULT:secret/data/ssl:private_key} + +# Database configuration with nested interpolation +database: + primary: + host: ${ENV:DB_HOST_${ENV:DEPLOY_ENV}} # e.g., DB_HOST_prod + port: ${ENV:DB_PORT} + name: ${ENV:DB_NAME} + user: ${ENV:DB_USER} + password: ${AWSSM:${ENV:APP_NAME}/${ENV:DEPLOY_ENV}/db_password} + + replica: + host: ${CONSUL:service/db-replica-${ENV:DEPLOY_ENV}/address} + port: ${CONSUL:service/db-replica-${ENV:DEPLOY_ENV}/port} + +# External services +services: + redis: + url: ${ETCD:/config/${ENV:APP_NAME}/redis_url} + password: ${K8SS:default/redis-secret:password} + + elasticsearch: + hosts: ${JSON:/etc/config/services.json:elasticsearch.hosts} + api_key: ${GCPSM:projects/${ENV:GCP_PROJECT}/secrets/es_api_key} + +# Feature flags from various sources +features: + new_ui: ${CONSUL:features/${ENV:APP_NAME}/new_ui} + rate_limiting: ${ENV:FEATURE_RATE_LIMITING} + analytics: ${YAML:/etc/features.yaml:features.analytics.enabled} + +# Cloud storage configuration +storage: + provider: ${ENV:STORAGE_PROVIDER} + config: + bucket: ${ENV:STORAGE_BUCKET} + region: ${ENV:AWS_REGION} + access_key: ${AWSSM:storage_access_key} + secret_key: ${AWSSM:storage_secret_key} + +# Monitoring and logging +monitoring: + datadog: + api_key: ${VAULT:secret/data/monitoring:datadog_api_key} + app_key: ${VAULT:secret/data/monitoring:datadog_app_key} + + sentry: + dsn: ${ENV:SENTRY_DSN} + environment: ${ENV:DEPLOY_ENV} + release: ${EXEC:git rev-parse HEAD} + +# Environment variables to inject +env: + # These will be set as environment variables in the process + DATABASE_URL: "postgres://${ENV:DB_USER}:${AWSSM:${ENV:APP_NAME}/${ENV:DEPLOY_ENV}/db_password}@${ENV:DB_HOST_${ENV:DEPLOY_ENV}}:${ENV:DB_PORT}/${ENV:DB_NAME}" + REDIS_URL: ${ETCD:/config/${ENV:APP_NAME}/redis_url} + NEW_RELIC_LICENSE_KEY: ${AZURESM:https://myvault.vault.azure.net:newrelic-license} + DD_TRACE_ENABLED: ${ENV:DD_TRACE_ENABLED} +``` + +```go +// main.go - Using the complete configuration +package main + +import ( + "fmt" + "log" + "os" + + "git.eeqj.de/sneak/smartconfig" +) + +func main() { + // Load configuration + config, err := smartconfig.NewFromConfigPath("production.yaml") + if err != nil { + log.Fatalf("Failed to load config: %v", err) + } + + // Access top-level typed values + appName, _ := config.GetString("app_name") + serverPort, _ := config.GetInt("server_port") + debugEnabled, _ := config.GetBool("debug_enabled") + maxUploadSize, _ := config.GetBytes("max_upload_size") + + fmt.Printf("Starting %s on port %d (debug: %v)\n", appName, serverPort, debugEnabled) + fmt.Printf("Max upload size: %d bytes\n", maxUploadSize) + + // Access nested configuration + data := config.Data() + + // Database configuration + if db, ok := data["database"].(map[string]interface{}); ok { + if primary, ok := db["primary"].(map[string]interface{}); ok { + dbHost, _ := primary["host"].(string) + dbPort, _ := primary["port"].(int) + fmt.Printf("Database: %s:%d\n", dbHost, dbPort) + } + } + + // Check injected environment variables + fmt.Printf("DATABASE_URL: %s\n", os.Getenv("DATABASE_URL")) + fmt.Printf("REDIS_URL: %s\n", os.Getenv("REDIS_URL")) + + // Feature flags + if features, ok := data["features"].(map[string]interface{}); ok { + if newUI, ok := features["new_ui"].(bool); ok && newUI { + fmt.Println("New UI is enabled!") + } + } +} +``` + +## Advanced Features + +### Nested Interpolation + +Interpolations can be nested up to 3 levels deep: + +```yaml +# Dynamic environment selection +environment: prod +database: + host: ${ENV:DB_HOST_${ENV:ENVIRONMENT}} # Looks for DB_HOST_prod + +# Multi-level nesting +config: + value: ${ENV:${ENV:PREFIX}_${ENV:SUFFIX}_KEY} + +# With different resolvers +secret: ${VAULT:secret/${ENV:APP_NAME}/${ENV:ENVIRONMENT}:password} +``` + +### Environment Variable Injection + +Values under the `env` key are automatically exported as environment variables: + +```yaml +# These become environment variables in your process +env: + DATABASE_URL: "postgres://${ENV:DB_USER}:${AWSSM:db-password}@${ENV:DB_HOST}/myapp" + REDIS_URL: ${CONSUL:service/redis/url} + API_KEY: ${VAULT:secret/data/external:api_key} + +# Your application can now use os.Getenv("DATABASE_URL"), etc. +``` + +### Custom Resolvers + +Extend smartconfig with your own resolvers: + +```go +// Implement the Resolver interface +type RedisResolver struct { + client *redis.Client +} + +func (r *RedisResolver) Resolve(key string) (string, error) { + return r.client.Get(key).Result() +} + +// Register your resolver +config := smartconfig.New() +config.RegisterResolver("REDIS", &RedisResolver{client: redisClient}) + +// Use in YAML +// cache_ttl: ${REDIS:config:cache:ttl} +``` + +### Error Handling + +Always handle errors appropriately: ```go config, err := smartconfig.NewFromAppName("myapp") if err != nil { - log.Fatal(err) + if os.IsNotExist(err) { + log.Fatal("Config file not found in /etc/myapp/config.yml") + } + log.Fatalf("Failed to load config: %v", err) } -``` -### From a File Path - -```go -config, err := smartconfig.NewFromConfigPath("/path/to/config.yaml") +// Handle missing keys +port, err := config.GetInt("server_port") if err != nil { - log.Fatal(err) + if strings.Contains(err.Error(), "not found") { + // Use default + port = 8080 + } else { + log.Fatalf("Invalid port configuration: %v", err) + } } -``` -### From an io.Reader - -```go -reader := strings.NewReader(yamlContent) -config, err := smartconfig.NewFromReader(reader) +// Handle type conversion errors +timeout, err := config.GetFloat("timeout") if err != nil { - log.Fatal(err) + if strings.Contains(err.Error(), "cannot convert") { + log.Fatalf("Timeout must be a number, got: %v", err) + } } ``` -## Accessing Configuration Values - -### Get Any Value - -```go -// Returns (value, exists) -value, exists := config.Get("database.host") -if exists { - fmt.Printf("Database host: %v\n", value) -} -``` - -### Get String Value - -```go -// Returns (string, error) -host, err := config.GetString("database.host") -if err != nil { - log.Printf("Error: %v", err) -} -``` - -### Get Integer Value - -```go -// Returns (int, error) -// Works with both native integers and string values -port, err := config.GetInt("server.port") -if err != nil { - log.Printf("Error: %v", err) -} -``` - -### Get Unsigned Integer Value - -```go -// Returns (uint, error) -maxConnections, err := config.GetUint("server.max_connections") -if err != nil { - log.Printf("Error: %v", err) -} -``` - -### Get Float Value - -```go -// Returns (float64, error) -timeout, err := config.GetFloat("server.timeout_seconds") -if err != nil { - log.Printf("Error: %v", err) -} -``` - -### Get Boolean Value - -```go -// Returns (bool, error) -debugMode, err := config.GetBool("debug_enabled") -if err != nil { - log.Printf("Error: %v", err) -} -``` - -### Get Byte Size Value (Human-Readable) - -```go -// Supports formats like "10G", "20KiB", "25TB" -// Returns (uint64, error) - size in bytes -maxFileSize, err := config.GetBytes("upload.max_file_size") -if err != nil { - log.Printf("Error: %v", err) -} - -// Example config values: -// max_file_size: "100MB" -> 100000000 bytes -// max_file_size: "10GiB" -> 10737418240 bytes -// max_file_size: "1.5TB" -> 1500000000000 bytes -``` - -### Get Entire Configuration - -```go -// Returns the full configuration as a map -data := config.Data() -``` - -## Custom Resolvers - -You can add custom resolvers to extend the interpolation capabilities: - -```go -// Define a custom resolver -type CustomResolver struct{} - -func (r *CustomResolver) Resolve(value string) (string, error) { - // Your custom resolution logic here - return "resolved-value", nil -} - -// Register the resolver -config := smartconfig.New() -config.RegisterResolver("CUSTOM", &CustomResolver{}) - -// Use in config: ${CUSTOM:some-value} -``` - -## Resolver Formats - -### Local Resolvers - -* `${ENV:VARIABLE_NAME}` - Environment variable -* `${EXEC:command}` - Execute shell command -* `${FILE:/path/to/file}` - Read file contents -* `${JSON:/path/to/file.json:json.path}` - Read JSON value (uses gjson syntax, e.g., `field.subfield` or `array.0`) -* `${YAML:/path/to/file.yaml:yaml.path}` - Read YAML value - -### Cloud Resolvers - -* `${AWSSM:secret-name}` - AWS Secrets Manager -* `${GCPSM:projects/PROJECT/secrets/NAME}` - GCP Secret Manager -* `${VAULT:path:key}` - HashiCorp Vault (e.g., `${VAULT:secret/data/myapp:password}`) -* `${CONSUL:key/path}` - Consul KV store -* `${AZURESM:https://vault.azure.net:secret}` - Azure Key Vault -* `${K8SS:namespace/secret:key}` - Kubernetes Secrets -* `${ETCD:/key/path}` - etcd (requires ETCD_ENDPOINTS env var) - -## Nested Interpolation - -Smartconfig supports nested interpolations up to 3 levels deep: - -```yaml -# Example: Using environment suffix for dynamic config selection -# First, set the ENV_SUFFIX environment variable: -# export ENV_SUFFIX=prod - -database: - # This will look for the environment variable DB_HOST_prod - host: ${ENV:DB_HOST_${ENV:ENV_SUFFIX}} - - # You can also use it with other resolvers - password: ${VAULT:secret/data/${ENV:ENV_SUFFIX}/db:password} - -# Another example with multiple environments -api: - # If ENV_SUFFIX=staging, this resolves to API_KEY_STAGING - key: ${ENV:API_KEY_${ENV:ENV_SUFFIX}} -``` - -Note: The `env_suffix` shown in the config is just a regular YAML key. To use it in interpolations, you need to reference it as `${ENV:ENV_SUFFIX}` where `ENV_SUFFIX` is an actual environment variable. - -## Environment Variable Injection - -Values under the `env` key are automatically set as environment variables: - -```yaml -env: - API_KEY: ${VAULT:secret/api:key} - DB_PASSWORD: ${AWSSM:db-password} - -# After loading, these are available as environment variables -``` - ## CLI Tool -A command-line tool is included that reads YAML from stdin, interpolates all variables, and writes the result to stdout: +A command-line tool is provided for testing and preprocessing configuration files: ```bash -# Build the CLI tool -go build ./cmd/smartconfig +# Install the CLI tool +go install git.eeqj.de/sneak/smartconfig/cmd/smartconfig@latest -# Use it to interpolate config files (default YAML output) -cat config.yaml | ./smartconfig > interpolated.yaml +# Process a config file (outputs YAML) +smartconfig < config.yaml > processed.yaml -# Output as formatted JSON instead of YAML -cat config.yaml | ./smartconfig --json > interpolated.json +# Output as JSON +smartconfig --json < config.yaml > processed.json -# Or use it in a pipeline +# Test interpolation export DB_PASSWORD="secret123" -echo 'password: ${ENV:DB_PASSWORD}' | ./smartconfig +echo 'password: ${ENV:DB_PASSWORD}' | smartconfig # Output: password: secret123 -# Type preservation example +# Type preservation in action export PORT="8080" export ENABLED="true" -echo -e 'port: ${ENV:PORT}\nenabled: ${ENV:ENABLED}' | ./smartconfig -# Output: -# port: 8080 # <-- numeric value -# enabled: true # <-- boolean value - -# JSON output preserves types -echo -e 'port: ${ENV:PORT}\nenabled: ${ENV:ENABLED}' | ./smartconfig --json -# Output: { -# "port": 8080, -# "enabled": true -# } +export TIMEOUT="30.5" +cat <<'EOF' | smartconfig --json +port: ${ENV:PORT} +enabled: ${ENV:ENABLED} +timeout: ${ENV:TIMEOUT} +EOF +# Output: {"enabled":true,"port":8080,"timeout":30.5} ``` -Note: If an environment variable or other resource is not found, the tool will exit with an error. +## Security Considerations -# License +### Arbitrary Code Execution +The `EXEC` resolver can execute any shell command. This is by design but has security implications: +- Only use trusted configuration files +- Validate configuration sources in production +- Consider disabling EXEC resolver in sensitive environments +- Use appropriate file permissions on configuration files + +### Secret Management Best Practices +- Never commit secrets to version control +- Use appropriate secret managers (Vault, AWS SM, etc.) for production +- Limit access to configuration files containing secret references +- Rotate secrets regularly +- Monitor secret access + +### File Access +The `FILE` resolver can read any file accessible to the process: +- Run applications with minimal required permissions +- Use separate users for different applications +- Consider using secrets managers instead of direct file access + +## Comparison with Other Libraries + +### vs. Viper +- **smartconfig**: Focuses on interpolation from multiple sources with type preservation +- **Viper**: More features but heavier, includes watching, flags, etc. + +### vs. envconfig +- **smartconfig**: YAML-based with interpolation from many sources +- **envconfig**: Environment variables only, struct tags + +### vs. koanf +- **smartconfig**: Simpler API, focused on interpolation +- **koanf**: More providers and formats, more complex API + +## Requirements + +- Go 1.18 or later +- Valid YAML syntax in configuration files +- Appropriate credentials for cloud providers (AWS, GCP, Azure) +- Network access for remote resolvers (Vault, Consul, etcd) + +## Contributing + +Contributions are welcome! Please ensure: +- Tests pass: `make test` +- Code is formatted: `make fmt` +- No linting errors: `make lint` + +## License WTFPL -# Author +## Author -sneak <sneak@sneak.berlin> +sneak + +--- + +*Inspired by Ryan Smith's configuration format ideas.* \ No newline at end of file