commit e15edaedd786254cf22c291a63a02f7cff33b119 Author: sneak <sneak@sneak.berlin> Date: Mon Jul 21 15:17:12 2025 +0200 Implement YAML-first interpolation with type preservation Previously, interpolations were performed during string manipulation before YAML parsing, which caused issues with quoting and escaping. This commit fundamentally changes the approach: - Parse YAML first, then walk the structure to interpolate values - Preserve types: standalone interpolations can return numbers/booleans - Mixed content (text with embedded interpolations) always returns strings - Users control types through YAML syntax, not our quoting logic - Properly handle nested interpolations without quote accumulation This gives users explicit control over output types while eliminating the complex and error-prone manual quoting logic. |
||
---|---|---|
cmd/smartconfig | ||
test | ||
.gitignore | ||
AGENTS.md | ||
cli_missing_env_test.go | ||
cli_test.go | ||
DESIGN.md | ||
go.mod | ||
go.sum | ||
interpolate.go | ||
LICENSE | ||
Makefile | ||
README.md | ||
resolver_awssm.go | ||
resolver_azure.go | ||
resolver_consul.go | ||
resolver_env.go | ||
resolver_etcd.go | ||
resolver_exec.go | ||
resolver_file.go | ||
resolver_gcpsm.go | ||
resolver_json.go | ||
resolver_k8s.go | ||
resolver_vault.go | ||
resolver_yaml.go | ||
smartconfig_test.go | ||
smartconfig.go | ||
typed_interpolation_test.go | ||
yaml_syntax_test.go |
smartconfig
smartconfig is a go library that reads YAML configuration files from the filesystem and interpolates variables in them from several sources, then parses the resulting YAML.
Important: smartconfig only supports valid YAML files as input. The file must be valid YAML syntax before interpolation occurs.
This was inspired by Ryan Smith and one of his config file formats/parsers. It struck me as very clever and I wanted to begin using it.
concept
Configuration files often need to store secrets, and these secrets come from various sources such as environment variables, cloud provider secrets managers, or other secret management systems. Instead of having to implement all of these APIs in your application, tightly binding it to your platform (and locking you in to that vendor), this allows your config file to serve as pointers to the various sources of secrets, supporting pluggable secret management sources.
Yes, it supports shelling out to external commands to get values. This equates being able to write your config file with arbitrary code execution, which may not be the case in your environment, but is a baked-in assumption here (and one I think will be fine/correct for 99% of users).
It has only one magic property other than the interpolation: anything specified under the top level key "env" will be interpolated and added, to the environment of the process that reads the config file. This allows a config file to serve as a bridge between fancy backend secret management services and "traditional" configuration via env vars.
Type-Preserving Interpolation
smartconfig uses a YAML-first approach: it parses the YAML file first, then walks the structure to perform interpolations. This means:
- Type preservation: Interpolated values can return appropriate types (numbers, booleans, strings)
- User control: The YAML syntax determines the output type
- Clean design: The YAML parser handles all quoting and escaping
How It Works
When an interpolation is the entire value (not embedded in other text), smartconfig attempts to convert the resolved value to the appropriate type:
# Numeric output - if ENV:PORT is "8080", this becomes the number 8080
port: ${ENV:PORT}
# Boolean output - if ENV:ENABLED is "true", this becomes the boolean true
enabled: ${ENV:ENABLED}
# String output - mixed content always returns strings
message: "Hello ${ENV:NAME}!"
# Force string output - add any prefix/suffix
port_str: "port-${ENV:PORT}"
Type Conversion Rules
For standalone interpolations (value: ${...}
), smartconfig converts:
"true"
→true
(boolean)"false"
→false
(boolean)"123"
→123
(integer)"12.5"
→12.5
(float)- Everything else → string
Important Notes
- YAML parser removes quotes, so
"${ENV:VAR}"
and${ENV:VAR}
are treated identically - To force string output, embed the interpolation in other text:
"prefix-${ENV:VAR}"
- Mixed content (text with embedded interpolations) always returns strings
- Nested interpolations are fully supported:
${ENV:PREFIX_${ENV:SUFFIX}}
Usage
# config.yaml
name: ${ENV:APPLICATION_NAME}
host: ${EXEC:hostname -s}
port: ${ENV:PORT}
vhost:
tls_sni_host: ${JSON:/etc/config/hosts.json:tls_sni_host.0}
machine:
id: ${FILE:/etc/machine-id}
temperature: ${FILE:/sys/class/thermal/thermal_zone0/temp}
num_cpus: ${EXEC:nproc}
api:
root_user: "admin"
root_password: ${AWSSM:root_password}
db:
host: ${GCPSM:${ENV:APPLICATION_NAME}_DB_HOST}
user: ${GCPSM:${ENV:APPLICATION_NAME}_DB_USER}
password: ${GCPSM:${ENV:APPLICATION_NAME}_DB_PASSWORD}
external:
google_api_key: ${CONSUL:secret:google_api_key}
twilio_api_key: ${VAULT:secret:twilio_api_key}
env:
ENCRYPTION_PUBLIC_KEY: ${EXEC:secret get myapp/encryption_public_key}
SLACK_WEBHOOK_URL: ${FILE:/etc/config/slack_webhook_url.txt}
Supported Providers
- ENV - environment variables
- EXEC - shell out to an external command
- AWSSM - AWS Secrets Manager
- GCPSM - Google Cloud Secret Manager
- VAULT - HashiCorp Vault
- CONSUL - HashiCorp Consul KV Store
- AZURESM - Azure Key Vault
- K8SS - Kubernetes Secrets
- FILE - read from a file
- JSON - read from a JSON file (supports JSON5 syntax including comments and trailing commas)
- YAML - read from a YAML file
- ETCD - etcd key-value store
API Documentation
Installation
go get git.eeqj.de/sneak/smartconfig
Basic Usage
package main
import (
"fmt"
"log"
"git.eeqj.de/sneak/smartconfig"
)
func main() {
// Load configuration from /etc/myapp/config.yml
config, err := smartconfig.NewFromAppName("myapp")
if err != nil {
log.Fatal(err)
}
// Or load from a specific path
config, err = smartconfig.NewFromConfigPath("/path/to/config.yaml")
if err != nil {
log.Fatal(err)
}
// Access configuration values
dbHost, _ := config.GetString("db_host")
fmt.Printf("Database host: %s\n", dbHost)
// Get entire configuration as a map
data := config.Data()
fmt.Printf("Full config: %+v\n", data)
}
Loading Configuration
From App Name (looks in /etc/appname/config.yml)
config, err := smartconfig.NewFromAppName("myapp")
if err != nil {
log.Fatal(err)
}
From a File Path
config, err := smartconfig.NewFromConfigPath("/path/to/config.yaml")
if err != nil {
log.Fatal(err)
}
From an io.Reader
reader := strings.NewReader(yamlContent)
config, err := smartconfig.NewFromReader(reader)
if err != nil {
log.Fatal(err)
}
Accessing Configuration Values
Get Any Value
// Returns (value, exists)
value, exists := config.Get("database.host")
if exists {
fmt.Printf("Database host: %v\n", value)
}
Get String Value
// Returns (string, error)
host, err := config.GetString("database.host")
if err != nil {
log.Printf("Error: %v", err)
}
Get Integer Value
// Returns (int, error)
// Works with both native integers and string values
port, err := config.GetInt("server.port")
if err != nil {
log.Printf("Error: %v", err)
}
Get Unsigned Integer Value
// Returns (uint, error)
maxConnections, err := config.GetUint("server.max_connections")
if err != nil {
log.Printf("Error: %v", err)
}
Get Float Value
// Returns (float64, error)
timeout, err := config.GetFloat("server.timeout_seconds")
if err != nil {
log.Printf("Error: %v", err)
}
Get Boolean Value
// Returns (bool, error)
debugMode, err := config.GetBool("debug_enabled")
if err != nil {
log.Printf("Error: %v", err)
}
Get Byte Size Value (Human-Readable)
// Supports formats like "10G", "20KiB", "25TB"
// Returns (uint64, error) - size in bytes
maxFileSize, err := config.GetBytes("upload.max_file_size")
if err != nil {
log.Printf("Error: %v", err)
}
// Example config values:
// max_file_size: "100MB" -> 100000000 bytes
// max_file_size: "10GiB" -> 10737418240 bytes
// max_file_size: "1.5TB" -> 1500000000000 bytes
Get Entire Configuration
// Returns the full configuration as a map
data := config.Data()
Custom Resolvers
You can add custom resolvers to extend the interpolation capabilities:
// Define a custom resolver
type CustomResolver struct{}
func (r *CustomResolver) Resolve(value string) (string, error) {
// Your custom resolution logic here
return "resolved-value", nil
}
// Register the resolver
config := smartconfig.New()
config.RegisterResolver("CUSTOM", &CustomResolver{})
// Use in config: ${CUSTOM:some-value}
Resolver Formats
Local Resolvers
${ENV:VARIABLE_NAME}
- Environment variable${EXEC:command}
- Execute shell command${FILE:/path/to/file}
- Read file contents${JSON:/path/to/file.json:json.path}
- Read JSON value (uses gjson syntax, e.g.,field.subfield
orarray.0
)${YAML:/path/to/file.yaml:yaml.path}
- Read YAML value
Cloud Resolvers
${AWSSM:secret-name}
- AWS Secrets Manager${GCPSM:projects/PROJECT/secrets/NAME}
- GCP Secret Manager${VAULT:path:key}
- HashiCorp Vault (e.g.,${VAULT:secret/data/myapp:password}
)${CONSUL:key/path}
- Consul KV store${AZURESM:https://vault.azure.net:secret}
- Azure Key Vault${K8SS:namespace/secret:key}
- Kubernetes Secrets${ETCD:/key/path}
- etcd (requires ETCD_ENDPOINTS env var)
Nested Interpolation
Smartconfig supports nested interpolations up to 3 levels deep:
# Example: Using environment suffix for dynamic config selection
# First, set the ENV_SUFFIX environment variable:
# export ENV_SUFFIX=prod
database:
# This will look for the environment variable DB_HOST_prod
host: ${ENV:DB_HOST_${ENV:ENV_SUFFIX}}
# You can also use it with other resolvers
password: ${VAULT:secret/data/${ENV:ENV_SUFFIX}/db:password}
# Another example with multiple environments
api:
# If ENV_SUFFIX=staging, this resolves to API_KEY_STAGING
key: ${ENV:API_KEY_${ENV:ENV_SUFFIX}}
Note: The env_suffix
shown in the config is just a regular YAML key. To use it in interpolations, you need to reference it as ${ENV:ENV_SUFFIX}
where ENV_SUFFIX
is an actual environment variable.
Environment Variable Injection
Values under the env
key are automatically set as environment variables:
env:
API_KEY: ${VAULT:secret/api:key}
DB_PASSWORD: ${AWSSM:db-password}
# After loading, these are available as environment variables
CLI Tool
A command-line tool is included that reads YAML from stdin, interpolates all variables, and writes the result to stdout:
# Build the CLI tool
go build ./cmd/smartconfig
# Use it to interpolate config files (default YAML output)
cat config.yaml | ./smartconfig > interpolated.yaml
# Output as formatted JSON instead of YAML
cat config.yaml | ./smartconfig --json > interpolated.json
# Or use it in a pipeline
export DB_PASSWORD="secret123"
echo 'password: ${ENV:DB_PASSWORD}' | ./smartconfig
# Output: password: secret123
# Type preservation example
export PORT="8080"
export ENABLED="true"
echo -e 'port: ${ENV:PORT}\nenabled: ${ENV:ENABLED}' | ./smartconfig
# Output:
# port: 8080 # <-- numeric value
# enabled: true # <-- boolean value
# JSON output preserves types
echo -e 'port: ${ENV:PORT}\nenabled: ${ENV:ENABLED}' | ./smartconfig --json
# Output: {
# "port": 8080,
# "enabled": true
# }
Note: If an environment variable or other resource is not found, the tool will exit with an error.
License
WTFPL
Author
sneak <sneak@sneak.berlin>