smartconfig/README.md

# smartconfig

smartconfig is a go library that reads YAML configuration files from the
filesystem and interpolates variables in them from several sources, then
parses the resulting YAML.

**Important:** smartconfig only supports valid YAML files as input. The file
must be valid YAML syntax before interpolation occurs.

This was inspired by Ryan Smith and one of his config file formats/parsers.
It struck me as very clever and I wanted to begin using it.

# concept

Configuration files often need to store secrets, and these secrets come from
various sources such as environment variables, cloud provider secrets
managers, or other secret management systems.  Instead of having to
implement all of these APIs in your application, tightly binding it to your
platform (and locking you in to that vendor), this allows your config file
to serve as pointers to the various sources of secrets, supporting pluggable
secret management sources.

Yes, it supports shelling out to external commands to get values.  This
equates being able to write your config file with arbitrary code execution,
which may not be the case in your environment, but is a baked-in assumption
here (and one I think will be fine/correct for 99% of users).

It has only one magic property other than the interpolation: anything
specified under the top level key "env" will be interpolated and added,
to the environment of the process that reads the config file.  This allows a
config file to serve as a bridge between fancy backend secret management
services and "traditional" configuration via env vars.

## Important: Automatic Quoting

All interpolated values are automatically quoted and escaped for YAML safety.
This means you should NOT wrap interpolations in quotes:

```yaml
# Correct - no quotes needed
password: ${ENV:DB_PASSWORD}
enabled: ${ENV:FEATURE_ENABLED}

# Incorrect - will result in double-quoted values
password: "${ENV:DB_PASSWORD}"
```

**Note:** ALL interpolated values are output as quoted strings, regardless of
their content. This means:

1. YAML keywords like `no`, `yes`, `true`, `false`, `on`, `off`, etc. will 
   always be treated as strings, never as boolean values
2. Numbers will always be treated as strings, never as numeric values

```yaml
# If ENV:FEATURE_ENABLED is "true", this will be the string "true", not boolean true
enabled: ${ENV:FEATURE_ENABLED}

# If ENV:DEBUG_MODE is "no", this will be the string "no", not boolean false
debug: ${ENV:DEBUG_MODE}

# If ENV:PORT is "8080", this will be the string "8080", not the number 8080
port: ${ENV:PORT}

# If ENV:TIMEOUT is "30.5", this will be the string "30.5", not the float 30.5
timeout: ${ENV:TIMEOUT}
```

This design choice prevents ambiguity and parsing errors, ensuring consistent 
behavior across all interpolated values. Use the typed getter methods (GetInt, 
GetFloat, GetBool, etc.) to convert string values to their appropriate types
when accessing configuration values.

# Usage

```yaml
# config.yaml

name: ${ENV:APPLICATION_NAME}
host: ${EXEC:hostname -s}
port: ${ENV:PORT}

vhost:
    tls_sni_host: ${JSON:/etc/config/hosts.json:tls_sni_host.0}

machine:
    id: ${FILE:/etc/machine-id}
    temperature: ${FILE:/sys/class/thermal/thermal_zone0/temp}
    num_cpus: ${EXEC:nproc}

api:
    root_user: "admin"
    root_password: ${AWSSM:root_password}

db:
    host: ${GCPSM:${ENV:APPLICATION_NAME}_DB_HOST}
    user: ${GCPSM:${ENV:APPLICATION_NAME}_DB_USER}
    password: ${GCPSM:${ENV:APPLICATION_NAME}_DB_PASSWORD}

external:
    google_api_key: ${CONSUL:secret:google_api_key}
    twilio_api_key: ${VAULT:secret:twilio_api_key}

env:
    ENCRYPTION_PUBLIC_KEY: ${EXEC:secret get myapp/encryption_public_key}
    SLACK_WEBHOOK_URL: ${FILE:/etc/config/slack_webhook_url.txt}
```

# Supported Providers

* ENV - environment variables
* EXEC - shell out to an external command
* AWSSM - AWS Secrets Manager
* GCPSM - Google Cloud Secret Manager
* VAULT - HashiCorp Vault
* CONSUL - HashiCorp Consul KV Store
* AZURESM - Azure Key Vault
* K8SS - Kubernetes Secrets
* FILE - read from a file
* JSON - read from a JSON file (supports JSON5 syntax including comments and trailing commas)
* YAML - read from a YAML file
* ETCD - etcd key-value store

# API Documentation

## Installation

```bash
go get git.eeqj.de/sneak/smartconfig
```

## Basic Usage

```go
package main

import (
    "fmt"
    "log"
    
    "git.eeqj.de/sneak/smartconfig"
)

func main() {
    // Load configuration from /etc/myapp/config.yml
    config, err := smartconfig.NewFromAppName("myapp")
    if err != nil {
        log.Fatal(err)
    }
    
    // Or load from a specific path
    config, err = smartconfig.NewFromConfigPath("/path/to/config.yaml")
    if err != nil {
        log.Fatal(err)
    }
    
    // Access configuration values
    dbHost, _ := config.GetString("db_host")
    fmt.Printf("Database host: %s\n", dbHost)
    
    // Get entire configuration as a map
    data := config.Data()
    fmt.Printf("Full config: %+v\n", data)
}
```

## Loading Configuration

### From App Name (looks in /etc/appname/config.yml)

```go
config, err := smartconfig.NewFromAppName("myapp")
if err != nil {
    log.Fatal(err)
}
```

### From a File Path

```go
config, err := smartconfig.NewFromConfigPath("/path/to/config.yaml")
if err != nil {
    log.Fatal(err)
}
```

### From an io.Reader

```go
reader := strings.NewReader(yamlContent)
config, err := smartconfig.NewFromReader(reader)
if err != nil {
    log.Fatal(err)
}
```

## Accessing Configuration Values

### Get Any Value

```go
// Returns (value, exists)
value, exists := config.Get("database.host")
if exists {
    fmt.Printf("Database host: %v\n", value)
}
```

### Get String Value

```go
// Returns (string, error)
host, err := config.GetString("database.host")
if err != nil {
    log.Printf("Error: %v", err)
}
```

### Get Integer Value

```go
// Returns (int, error)
port, err := config.GetInt("server.port")
if err != nil {
    log.Printf("Error: %v", err)
}
```

### Get Unsigned Integer Value

```go
// Returns (uint, error)
maxConnections, err := config.GetUint("server.max_connections")
if err != nil {
    log.Printf("Error: %v", err)
}
```

### Get Float Value

```go
// Returns (float64, error)
timeout, err := config.GetFloat("server.timeout_seconds")
if err != nil {
    log.Printf("Error: %v", err)
}
```

### Get Boolean Value

```go
// Returns (bool, error)
debugMode, err := config.GetBool("debug_enabled")
if err != nil {
    log.Printf("Error: %v", err)
}
```

### Get Byte Size Value (Human-Readable)

```go
// Supports formats like "10G", "20KiB", "25TB"
// Returns (uint64, error) - size in bytes
maxFileSize, err := config.GetBytes("upload.max_file_size")
if err != nil {
    log.Printf("Error: %v", err)
}

// Example config values:
// max_file_size: "100MB"    -> 100000000 bytes
// max_file_size: "10GiB"    -> 10737418240 bytes
// max_file_size: "1.5TB"    -> 1500000000000 bytes
```

### Get Entire Configuration

```go
// Returns the full configuration as a map
data := config.Data()
```

## Custom Resolvers

You can add custom resolvers to extend the interpolation capabilities:

```go
// Define a custom resolver
type CustomResolver struct{}

func (r *CustomResolver) Resolve(value string) (string, error) {
    // Your custom resolution logic here
    return "resolved-value", nil
}

// Register the resolver
config := smartconfig.New()
config.RegisterResolver("CUSTOM", &CustomResolver{})

// Use in config: ${CUSTOM:some-value}
```

## Resolver Formats

### Local Resolvers

* `${ENV:VARIABLE_NAME}` - Environment variable
* `${EXEC:command}` - Execute shell command
* `${FILE:/path/to/file}` - Read file contents
* `${JSON:/path/to/file.json:json.path}` - Read JSON value (uses gjson syntax, e.g., `field.subfield` or `array.0`)
* `${YAML:/path/to/file.yaml:yaml.path}` - Read YAML value

### Cloud Resolvers

* `${AWSSM:secret-name}` - AWS Secrets Manager
* `${GCPSM:projects/PROJECT/secrets/NAME}` - GCP Secret Manager
* `${VAULT:path:key}` - HashiCorp Vault (e.g., `${VAULT:secret/data/myapp:password}`)
* `${CONSUL:key/path}` - Consul KV store
* `${AZURESM:https://vault.azure.net:secret}` - Azure Key Vault
* `${K8SS:namespace/secret:key}` - Kubernetes Secrets
* `${ETCD:/key/path}` - etcd (requires ETCD_ENDPOINTS env var)

## Nested Interpolation

Smartconfig supports nested interpolations up to 3 levels deep:

```yaml
# Example: Using environment suffix for dynamic config selection
# First, set the ENV_SUFFIX environment variable:
# export ENV_SUFFIX=prod

database:
  # This will look for the environment variable DB_HOST_prod
  host: ${ENV:DB_HOST_${ENV:ENV_SUFFIX}}
  
  # You can also use it with other resolvers
  password: ${VAULT:secret/data/${ENV:ENV_SUFFIX}/db:password}

# Another example with multiple environments
api:
  # If ENV_SUFFIX=staging, this resolves to API_KEY_STAGING
  key: ${ENV:API_KEY_${ENV:ENV_SUFFIX}}
```

Note: The `env_suffix` shown in the config is just a regular YAML key. To use it in interpolations, you need to reference it as `${ENV:ENV_SUFFIX}` where `ENV_SUFFIX` is an actual environment variable.

## Environment Variable Injection

Values under the `env` key are automatically set as environment variables:

```yaml
env:
  API_KEY: ${VAULT:secret/api:key}
  DB_PASSWORD: ${AWSSM:db-password}
  
# After loading, these are available as environment variables
```

## CLI Tool

A command-line tool is included that reads YAML from stdin, interpolates all variables, and writes the result to stdout:

```bash
# Build the CLI tool
go build ./cmd/smartconfig

# Use it to interpolate config files (default YAML output)
cat config.yaml | ./smartconfig > interpolated.yaml

# Output as formatted JSON instead of YAML
cat config.yaml | ./smartconfig --json > interpolated.json

# Or use it in a pipeline
export DB_PASSWORD="secret123"
echo 'password: ${ENV:DB_PASSWORD}' | ./smartconfig
# Output: password: secret123

# JSON output in a pipeline
echo 'password: ${ENV:DB_PASSWORD}' | ./smartconfig --json
# Output: {
#   "password": "secret123"
# }
```

Note: If an environment variable or other resource is not found, the tool will exit with an error.

# License

WTFPL

# Author

sneak <sneak@sneak.berlin>