271 lines
6.5 KiB
Markdown
271 lines
6.5 KiB
Markdown
# smartconfig
|
|
|
|
smartconfig is a go library that reads a configuration file from the
|
|
filesystem and interpolates variables in it from several sources, then
|
|
parses the config file as yaml.
|
|
|
|
This was inspired by Ryan Smith and one of his config file formats/parsers.
|
|
It struck me as very clever and I wanted to begin using it.
|
|
|
|
# concept
|
|
|
|
Configuration files often need to store secrets, and these secrets come from
|
|
various sources such as environment variables, cloud provider secrets
|
|
managers, or other secret management systems. Instead of having to
|
|
implement all of these APIs in your application, tightly binding it to your
|
|
platform (and locking you in to that vendor), this allows your config file
|
|
to serve as pointers to the various sources of secrets, supporting pluggable
|
|
secret management sources.
|
|
|
|
Yes, it supports shelling out to external commands to get values. This
|
|
equates being able to write your config file with arbitrary code execution,
|
|
which may not be the case in your environment, but is a baked-in assumption
|
|
here (and one I think will be fine/correct for 99% of users).
|
|
|
|
It has only one magic property other than the interpolation: anything
|
|
specified under the top level key "env" will be interpolated and added,
|
|
to the environment of the process that reads the config file. This allows a
|
|
config file to serve as a bridge between fancy backend secret management
|
|
services and "traditional" configuration via env vars.
|
|
|
|
# Usage
|
|
|
|
```yaml
|
|
# config.yaml
|
|
|
|
name: ${ENV:APPLICATION_NAME}
|
|
host: ${EXEC:"hostname -s"}
|
|
port: ${ENV:PORT}
|
|
|
|
vhost:
|
|
tls_sni_host: "${JSON:/etc/config/hosts.json:'.tls_sni_host[0]'}"
|
|
|
|
machine:
|
|
id: "${FILE:/etc/machine-id}"
|
|
temperature: "${FILE:/sys/class/thermal/thermal_zone0/temp}"
|
|
num_cpus: "${EXEC:nproc}"
|
|
|
|
api:
|
|
root_user: "admin"
|
|
root_password: "${AWSSM:root_password}"
|
|
|
|
db:
|
|
host: "${GCPSM:${ENV:APPLICATION_NAME}_DB_HOST}"
|
|
user: "${GCPSM:${ENV:APPLICATION_NAME}_DB_USER}"
|
|
password: "${GCPSM:${ENV:APPLICATION_NAME}_DB_PASSWORD}"
|
|
|
|
external:
|
|
google_api_key: "${CONSUL:secret:google_api_key}"
|
|
twilio_api_key: "${VAULT:secret:twilio_api_key}"
|
|
|
|
env:
|
|
ENCRYPTION_PUBLIC_KEY: "${EXEC:secret get myapp/encryption_public_key}"
|
|
SLACK_WEBHOOK_URL: "${FILE:/etc/config/slack_webhook_url.txt}"
|
|
```
|
|
|
|
# Supported Providers
|
|
|
|
* ENV - environment variables
|
|
* EXEC - shell out to an external command
|
|
* AWSSM - AWS Secrets Manager
|
|
* GCPSM - Google Cloud Secret Manager
|
|
* VAULT - HashiCorp Vault
|
|
* CONSUL - HashiCorp Consul KV Store
|
|
* AZURESM - Azure Key Vault
|
|
* K8SS - Kubernetes Secrets
|
|
* FILE - read from a file
|
|
* JSON - read from a JSON file (supports json5)
|
|
* YAML - read from a YAML file
|
|
* ETCD - etcd key-value store
|
|
|
|
# API Documentation
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
go get git.eeqj.de/sneak/smartconfig
|
|
```
|
|
|
|
## Basic Usage
|
|
|
|
```go
|
|
package main
|
|
|
|
import (
|
|
"fmt"
|
|
"log"
|
|
|
|
"git.eeqj.de/sneak/smartconfig"
|
|
)
|
|
|
|
func main() {
|
|
// Load configuration from /etc/myapp/config.yml
|
|
config, err := smartconfig.NewFromAppName("myapp")
|
|
if err != nil {
|
|
log.Fatal(err)
|
|
}
|
|
|
|
// Or load from a specific path
|
|
config, err = smartconfig.NewFromConfigPath("/path/to/config.yaml")
|
|
if err != nil {
|
|
log.Fatal(err)
|
|
}
|
|
|
|
// Access configuration values
|
|
dbHost, _ := config.GetString("db_host")
|
|
fmt.Printf("Database host: %s\n", dbHost)
|
|
|
|
// Get entire configuration as a map
|
|
data := config.Data()
|
|
fmt.Printf("Full config: %+v\n", data)
|
|
}
|
|
```
|
|
|
|
## Loading Configuration
|
|
|
|
### From App Name (looks in /etc/appname/config.yml)
|
|
|
|
```go
|
|
config, err := smartconfig.NewFromAppName("myapp")
|
|
if err != nil {
|
|
log.Fatal(err)
|
|
}
|
|
```
|
|
|
|
### From a File Path
|
|
|
|
```go
|
|
config, err := smartconfig.NewFromConfigPath("/path/to/config.yaml")
|
|
if err != nil {
|
|
log.Fatal(err)
|
|
}
|
|
```
|
|
|
|
### From an io.Reader
|
|
|
|
```go
|
|
reader := strings.NewReader(yamlContent)
|
|
config, err := smartconfig.NewFromReader(reader)
|
|
if err != nil {
|
|
log.Fatal(err)
|
|
}
|
|
```
|
|
|
|
## Accessing Configuration Values
|
|
|
|
### Get Any Value
|
|
|
|
```go
|
|
// Returns (value, exists)
|
|
value, exists := config.Get("database.host")
|
|
if exists {
|
|
fmt.Printf("Database host: %v\n", value)
|
|
}
|
|
```
|
|
|
|
### Get String Value
|
|
|
|
```go
|
|
// Returns (string, error)
|
|
host, err := config.GetString("database.host")
|
|
if err != nil {
|
|
log.Printf("Error: %v", err)
|
|
}
|
|
```
|
|
|
|
### Get Entire Configuration
|
|
|
|
```go
|
|
// Returns the full configuration as a map
|
|
data := config.Data()
|
|
```
|
|
|
|
## Custom Resolvers
|
|
|
|
You can add custom resolvers to extend the interpolation capabilities:
|
|
|
|
```go
|
|
// Define a custom resolver
|
|
type CustomResolver struct{}
|
|
|
|
func (r *CustomResolver) Resolve(value string) (string, error) {
|
|
// Your custom resolution logic here
|
|
return "resolved-value", nil
|
|
}
|
|
|
|
// Register the resolver
|
|
config := smartconfig.New()
|
|
config.RegisterResolver("CUSTOM", &CustomResolver{})
|
|
|
|
// Use in config: ${CUSTOM:some-value}
|
|
```
|
|
|
|
## Resolver Formats
|
|
|
|
### Local Resolvers
|
|
|
|
* `${ENV:VARIABLE_NAME}` - Environment variable
|
|
* `${EXEC:command}` - Execute shell command
|
|
* `${FILE:/path/to/file}` - Read file contents
|
|
* `${JSON:/path/to/file.json:json.path}` - Read JSON value
|
|
* `${YAML:/path/to/file.yaml:yaml.path}` - Read YAML value
|
|
|
|
### Cloud Resolvers
|
|
|
|
* `${AWSSM:secret-name}` - AWS Secrets Manager
|
|
* `${GCPSM:projects/PROJECT/secrets/NAME}` - GCP Secret Manager
|
|
* `${VAULT:path:key}` - HashiCorp Vault (e.g., `${VAULT:secret/data/myapp:password}`)
|
|
* `${CONSUL:key/path}` - Consul KV store
|
|
* `${AZURESM:https://vault.azure.net:secret}` - Azure Key Vault
|
|
* `${K8SS:namespace/secret:key}` - Kubernetes Secrets
|
|
* `${ETCD:/key/path}` - etcd (requires ETCD_ENDPOINTS env var)
|
|
|
|
## Nested Interpolation
|
|
|
|
Smartconfig supports nested interpolations up to 3 levels deep:
|
|
|
|
```yaml
|
|
env_suffix: "prod"
|
|
database:
|
|
host: "${ENV:DB_HOST_${ENV:ENV_SUFFIX}}" # Resolves DB_HOST_prod
|
|
```
|
|
|
|
## Environment Variable Injection
|
|
|
|
Values under the `env` key are automatically set as environment variables:
|
|
|
|
```yaml
|
|
env:
|
|
API_KEY: "${VAULT:secret/api:key}"
|
|
DB_PASSWORD: "${AWSSM:db-password}"
|
|
|
|
# After loading, these are available as environment variables
|
|
```
|
|
|
|
## CLI Tool
|
|
|
|
A command-line tool is included that reads YAML from stdin, interpolates all variables, and writes the result to stdout:
|
|
|
|
```bash
|
|
# Build the CLI tool
|
|
go build ./cmd/smartconfig
|
|
|
|
# Use it to interpolate config files
|
|
cat config.yaml | ./smartconfig > interpolated.yaml
|
|
|
|
# Or use it in a pipeline
|
|
export DB_PASSWORD="secret123"
|
|
echo 'password: ${ENV:DB_PASSWORD}' | ./smartconfig
|
|
# Output: password: secret123
|
|
```
|
|
|
|
Note: If an environment variable or other resource is not found, the tool will exit with an error.
|
|
|
|
# License
|
|
|
|
WTFPL
|
|
|
|
# Author
|
|
|
|
sneak <sneak@sneak.berlin>
|