smartconfig/README.md

smartconfig

smartconfig is a go library that reads a configuration file from the filesystem and interpolates variables in it from several sources, then parses the config file as yaml.

This was inspired by Ryan Smith and one of his config file formats/parsers. It struck me as very clever and I wanted to begin using it.

concept

Configuration files often need to store secrets, and these secrets come from various sources such as environment variables, cloud provider secrets managers, or other secret management systems. Instead of having to implement all of these APIs in your application, tightly binding it to your platform (and locking you in to that vendor), this allows your config file to serve as pointers to the various sources of secrets, supporting pluggable secret management sources.

Yes, it supports shelling out to external commands to get values. This equates being able to write your config file with arbitrary code execution, which may not be the case in your environment, but is a baked-in assumption here (and one I think will be fine/correct for 99% of users).

It has only one magic property other than the interpolation: anything specified under the top level key "env" will be interpolated and added, to the environment of the process that reads the config file. This allows a config file to serve as a bridge between fancy backend secret management services and "traditional" configuration via env vars.

Usage

# config.yaml

name: ${ENV:APPLICATION_NAME}
host: ${EXEC:"hostname -s"}
port: ${ENV:PORT}

vhost:
    tls_sni_host: "${JSON:/etc/config/hosts.json:'.tls_sni_host[0]'}"

machine:
    id: "${FILE:/etc/machine-id}"
    temperature: "${FILE:/sys/class/thermal/thermal_zone0/temp}"
    num_cpus: "${EXEC:nproc}"

api:
    root_user: "admin"
    root_password: "${AWSSM:root_password}"

db:
    host: "${GCPSM:${ENV:APPLICATION_NAME}_DB_HOST}"
    user: "${GCPSM:${ENV:APPLICATION_NAME}_DB_USER}"
    password: "${GCPSM:${ENV:APPLICATION_NAME}_DB_PASSWORD}"

external:
    google_api_key: "${CONSUL:secret:google_api_key}"
    twilio_api_key: "${VAULT:secret:twilio_api_key}"

env:
    ENCRYPTION_PUBLIC_KEY: "${EXEC:secret get myapp/encryption_public_key}"
    SLACK_WEBHOOK_URL: "${FILE:/etc/config/slack_webhook_url.txt}"

Supported Providers

• ENV - environment variables
• EXEC - shell out to an external command
• AWSSM - AWS Secrets Manager
• GCPSM - Google Cloud Secret Manager
• VAULT - HashiCorp Vault
• CONSUL - HashiCorp Consul KV Store
• AZURESM - Azure Key Vault
• K8SS - Kubernetes Secrets
• FILE - read from a file
• JSON - read from a JSON file (supports json5)
• YAML - read from a YAML file
• ETCD - etcd key-value store

API Documentation

Installation

go get git.eeqj.de/sneak/smartconfig

Basic Usage

package main

import (
    "fmt"
    "log"
    
    "git.eeqj.de/sneak/smartconfig"
)

func main() {
    // Load configuration from /etc/myapp/config.yml
    config, err := smartconfig.NewFromAppName("myapp")
    if err != nil {
        log.Fatal(err)
    }
    
    // Or load from a specific path
    config, err = smartconfig.NewFromConfigPath("/path/to/config.yaml")
    if err != nil {
        log.Fatal(err)
    }
    
    // Access configuration values
    dbHost, _ := config.GetString("db_host")
    fmt.Printf("Database host: %s\n", dbHost)
    
    // Get entire configuration as a map
    data := config.Data()
    fmt.Printf("Full config: %+v\n", data)
}

Loading Configuration

From App Name (looks in /etc/appname/config.yml)

config, err := smartconfig.NewFromAppName("myapp")
if err != nil {
    log.Fatal(err)
}

From a File Path

config, err := smartconfig.NewFromConfigPath("/path/to/config.yaml")
if err != nil {
    log.Fatal(err)
}

From an io.Reader

reader := strings.NewReader(yamlContent)
config, err := smartconfig.NewFromReader(reader)
if err != nil {
    log.Fatal(err)
}

Accessing Configuration Values

Get Any Value

// Returns (value, exists)
value, exists := config.Get("database.host")
if exists {
    fmt.Printf("Database host: %v\n", value)
}

Get String Value

// Returns (string, error)
host, err := config.GetString("database.host")
if err != nil {
    log.Printf("Error: %v", err)
}

Get Entire Configuration

// Returns the full configuration as a map
data := config.Data()

Custom Resolvers

You can add custom resolvers to extend the interpolation capabilities:

// Define a custom resolver
type CustomResolver struct{}

func (r *CustomResolver) Resolve(value string) (string, error) {
    // Your custom resolution logic here
    return "resolved-value", nil
}

// Register the resolver
config := smartconfig.New()
config.RegisterResolver("CUSTOM", &CustomResolver{})

// Use in config: ${CUSTOM:some-value}

Resolver Formats

Local Resolvers

• ${ENV:VARIABLE_NAME} - Environment variable
• ${EXEC:command} - Execute shell command
• ${FILE:/path/to/file} - Read file contents
• ${JSON:/path/to/file.json:json.path} - Read JSON value
• ${YAML:/path/to/file.yaml:yaml.path} - Read YAML value

Cloud Resolvers

• ${AWSSM:secret-name} - AWS Secrets Manager
• ${GCPSM:projects/PROJECT/secrets/NAME} - GCP Secret Manager
• ${VAULT:path:key} - HashiCorp Vault (e.g., ${VAULT:secret/data/myapp:password})
• ${CONSUL:key/path} - Consul KV store
• ${AZURESM:https://vault.azure.net:secret} - Azure Key Vault
• ${K8SS:namespace/secret:key} - Kubernetes Secrets
• ${ETCD:/key/path} - etcd (requires ETCD_ENDPOINTS env var)

Nested Interpolation

Smartconfig supports nested interpolations up to 3 levels deep:

env_suffix: "prod"
database:
  host: "${ENV:DB_HOST_${ENV:ENV_SUFFIX}}"  # Resolves DB_HOST_prod

Environment Variable Injection

Values under the env key are automatically set as environment variables:

env:
  API_KEY: "${VAULT:secret/api:key}"
  DB_PASSWORD: "${AWSSM:db-password}"
  
# After loading, these are available as environment variables

CLI Tool

A command-line tool is included that reads YAML from stdin, interpolates all variables, and writes the result to stdout:

# Build the CLI tool
go build ./cmd/smartconfig

# Use it to interpolate config files
cat config.yaml | ./smartconfig > interpolated.yaml

# Or use it in a pipeline
export DB_PASSWORD="secret123"
echo 'password: ${ENV:DB_PASSWORD}' | ./smartconfig
# Output: password: secret123

Note: If an environment variable or other resource is not found, the tool will exit with an error.

License

WTFPL

Author

sneak <sneak@sneak.berlin>