watcher: follow a watched name's CNAME for port and TLS checks (closes #203)
check / check (push) Failing after 2m1s
check / check (push) Failing after 2m1s
When a watched name's nameservers answer with a CNAME and no address, the DNS check asks ResolveIPAddresses for the name, which looks it up again and follows the chain, and saves the addresses at its end in the hostname state as cnameAddresses. The port and TLS checks use them. Before, a CNAME into another zone got no port or TLS checks. A change in those addresses is notified as a CNAME address change. When following fails, or none of the name's nameservers answered, the addresses the last check saved are kept. The domain check now runs the hostname check for the apex instead of a copy of it. Model: opus-5-5
This commit is contained in:
@@ -121,12 +121,22 @@ notification endpoint set, changes show only on the dashboard; see
|
||||
failed on it, and answers differently is reported on the check where it
|
||||
answers. If a pair agrees again and later disagrees, the alert is sent
|
||||
again.
|
||||
- **CNAME address change**: For a name whose nameservers answer with a CNAME
|
||||
and no address, the addresses at the end of its CNAME chain differ from
|
||||
those of the previous check, including when there are none now. Nothing is
|
||||
sent when the previous check saved none, or when the previous addresses
|
||||
were kept because the chain could not be followed or none of the name's
|
||||
nameservers answered.
|
||||
|
||||
### TCP Port Monitoring
|
||||
|
||||
- For every configured domain and hostname, constructs a deduplicated list of
|
||||
all IPv4 and IPv6 addresses resolved via A, AAAA, and CNAME chain resolution
|
||||
across all authoritative nameservers.
|
||||
the IPv4 and IPv6 addresses in the A and AAAA records its authoritative
|
||||
nameservers returned. When they returned a CNAME and no address, the CNAME
|
||||
chain is followed and the addresses at its end are used, and a change in those
|
||||
is notified as a CNAME address change. When the chain cannot be followed, or
|
||||
none of the name's nameservers answered, the addresses the last check found at
|
||||
its end are used.
|
||||
- Checks TCP connectivity on ports **80** and **443** for each IP address.
|
||||
- Every **1 hour**, re-checks all ports.
|
||||
- Any change in port availability triggers a notification:
|
||||
@@ -173,6 +183,8 @@ includes:
|
||||
- **DNS NS changes**: Which domain, which nameservers were added/removed.
|
||||
- **NS address changes**: Which domain, which nameserver, its old and new
|
||||
addresses.
|
||||
- **CNAME address changes**: Which hostname, the old and new addresses at the
|
||||
end of its CNAME chain.
|
||||
- **NS query failures**: Which nameserver failed, error type (timeout, SERVFAIL,
|
||||
REFUSED, network error), which hostname/domain affected.
|
||||
- **NS recoveries**: Which nameserver recovered, which hostname/domain.
|
||||
@@ -389,8 +401,11 @@ This approach ensures:
|
||||
servers.
|
||||
- Visibility into the full delegation chain.
|
||||
|
||||
For hostname monitoring, the resolver follows CNAME chains (with a depth limit
|
||||
to prevent loops) before collecting terminal A/AAAA records.
|
||||
A watched name's records are stored as its nameservers return them, CNAME
|
||||
included. When they return a CNAME and no address, the CNAME chain is followed
|
||||
(with a depth limit to prevent loops) to the A and AAAA records at its end, and
|
||||
the port and TLS checks use those addresses. Nameservers' addresses are found
|
||||
the same way.
|
||||
|
||||
---
|
||||
|
||||
@@ -478,6 +493,12 @@ nameservers, has status `error`, empty `records`, and the reason in `error`.
|
||||
resolves to. A state file without it loads, and the next check fills it in
|
||||
without a notification.
|
||||
|
||||
`cnameAddresses` lists the sorted addresses at the end of a hostname's CNAME
|
||||
chain, found when its nameservers answered with a CNAME and no address, and kept
|
||||
from the previous check when the chain cannot be followed or none of the name's
|
||||
nameservers answered. It is left out otherwise. A state file without it loads,
|
||||
and the next check fills it in without a notification.
|
||||
|
||||
---
|
||||
|
||||
## Entrypoints
|
||||
@@ -617,7 +638,9 @@ docker run -d \
|
||||
completes.
|
||||
- Port and TLS checks always use freshly resolved IP addresses from the DNS
|
||||
phase that immediately precedes them — never stale IPs from a previous
|
||||
cycle.
|
||||
cycle, with one exception: when a name's CNAME chain cannot be followed,
|
||||
or none of the name's nameservers answered, the addresses the previous
|
||||
cycle found at the end of the chain are used.
|
||||
4. **On change detection**: Send notifications to all configured endpoints,
|
||||
update in-memory state, persist to disk.
|
||||
5. **Shutdown**: The watcher stops checking and saves the final state to disk,
|
||||
|
||||
@@ -19,6 +19,8 @@ trial run of the finished image: https://git.eeqj.de/sneak/dnswatcher/issues/149
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-01: a watched name whose nameservers answer with a CNAME and no
|
||||
address gets port and TLS checks at the end of its CNAME chain (closes #203).
|
||||
- 2026-10-01: a certificate within the expiry warning period is warned about on
|
||||
every TLS check, where some checks used to skip it at random (closes #204).
|
||||
- 2026-10-01: a domain's NS set is its delegation from the parent zone's
|
||||
|
||||
@@ -53,8 +53,12 @@ type NameserverRecordState struct {
|
||||
}
|
||||
|
||||
// HostnameState holds per-nameserver monitoring state for a hostname.
|
||||
// CNAMEAddresses holds the sorted addresses at the end of the name's
|
||||
// CNAME chain, found when its nameservers answered with a CNAME and no
|
||||
// address; it is empty otherwise.
|
||||
type HostnameState struct {
|
||||
RecordsByNameserver map[string]*NameserverRecordState `json:"recordsByNameserver"`
|
||||
CNAMEAddresses []string `json:"cnameAddresses,omitempty"`
|
||||
LastChecked time.Time `json:"lastChecked"`
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
package watcher_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/dnswatcher/internal/livednstest"
|
||||
"sneak.berlin/go/dnswatcher/internal/resolver"
|
||||
"sneak.berlin/go/dnswatcher/internal/state"
|
||||
"sneak.berlin/go/dnswatcher/internal/watcher"
|
||||
)
|
||||
|
||||
// cnameHost is a CNAME into another zone: its nameservers answer with
|
||||
// the CNAME and no address.
|
||||
const cnameHost = "www.python.org"
|
||||
|
||||
// TestCNAMEIntoAnotherZonePortAndTLSChecks checks cnameHost against
|
||||
// live DNS. Its port and TLS checks must use the addresses at the end
|
||||
// of its CNAME chain.
|
||||
func TestCNAMEIntoAnotherZonePortAndTLSChecks(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := defaultTestConfig(t)
|
||||
cfg.Hostnames = []string{cnameHost}
|
||||
|
||||
deps := runChecks(t, cfg, nil, nil)
|
||||
|
||||
snap := deps.state.GetSnapshot()
|
||||
hs := snap.Hostnames[cnameHost]
|
||||
|
||||
if len(hs.CNAMEAddresses) == 0 {
|
||||
t.Fatalf(
|
||||
"%s: no addresses saved from following its CNAME; if it "+
|
||||
"is no longer a CNAME into another zone, this test "+
|
||||
"needs another name",
|
||||
cnameHost,
|
||||
)
|
||||
}
|
||||
|
||||
for _, ip := range hs.CNAMEAddresses {
|
||||
ps, ok := snap.Ports[ip+":443"]
|
||||
if !ok || !slices.Contains(ps.Hostnames, cnameHost) {
|
||||
t.Errorf("no port state for %s at %s:443", cnameHost, ip)
|
||||
}
|
||||
|
||||
certKey := ip + ":443:" + cnameHost
|
||||
if _, ok := snap.Certificates[certKey]; !ok {
|
||||
t.Errorf("no certificate state %s", certKey)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCNAMEThatCannotBeFollowedKeepsPrevious gives a name under
|
||||
// .invalid, whose lookup fails, answers with a CNAME and no address.
|
||||
// The addresses the previous check saved from following its CNAME are
|
||||
// kept.
|
||||
func TestCNAMEThatCannotBeFollowedKeepsPrevious(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const name = "www.example.invalid"
|
||||
|
||||
w := watcher.NewForTest(
|
||||
nil, nil, resolver.NewFromLogger(slog.Default()), nil, nil, nil,
|
||||
)
|
||||
|
||||
current := hostnameState(map[string]map[string][]string{
|
||||
nsA: {"CNAME": {"target.example.invalid."}},
|
||||
})
|
||||
prev := &state.HostnameState{CNAMEAddresses: []string{oldIP}}
|
||||
|
||||
// The result is the same whether or not live DNS answers, so the
|
||||
// lookup is not retried.
|
||||
_ = livednstest.Run(func(ctx context.Context) error {
|
||||
w.ResolveCNAMEAddresses(ctx, name, current, prev)
|
||||
|
||||
return nil
|
||||
})
|
||||
|
||||
if !slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
|
||||
t.Errorf(
|
||||
"saved %v, want %v",
|
||||
current.CNAMEAddresses, prev.CNAMEAddresses,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCNAMEWhoseNameserversAllFailedKeepsPrevious checks a name none of
|
||||
// whose nameservers answered. The addresses the previous check saved
|
||||
// from following its CNAME are kept, and nothing is looked up: the
|
||||
// watcher has no resolver.
|
||||
func TestCNAMEWhoseNameserversAllFailedKeepsPrevious(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
|
||||
|
||||
current := saved(map[string]*state.NameserverRecordState{
|
||||
nsA: failed(), nsB: failed(),
|
||||
})
|
||||
prev := cnameState(oldIP)
|
||||
|
||||
w.ResolveCNAMEAddresses(t.Context(), host, current, prev)
|
||||
|
||||
if !slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
|
||||
t.Errorf(
|
||||
"saved %v, want %v",
|
||||
current.CNAMEAddresses, prev.CNAMEAddresses,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// cnameState builds the state a check leaves behind for a name whose
|
||||
// nameserver answered with a CNAME and no address, when following the
|
||||
// CNAME found these addresses.
|
||||
func cnameState(addresses ...string) *state.HostnameState {
|
||||
hs := hostnameState(map[string]map[string][]string{
|
||||
nsA: {"CNAME": {"target.example.org."}},
|
||||
})
|
||||
hs.CNAMEAddresses = addresses
|
||||
|
||||
return hs
|
||||
}
|
||||
|
||||
func TestCNAMEAddressChangeAlerts(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Each case is the state saved by the previous check and by the
|
||||
// current one. The name's records are the same in both.
|
||||
tests := []struct {
|
||||
name string
|
||||
prev, current *state.HostnameState
|
||||
want int
|
||||
}{
|
||||
{
|
||||
"same addresses",
|
||||
cnameState(ip1, ip2), cnameState(ip1, ip2), 0,
|
||||
},
|
||||
{
|
||||
"same addresses in another order",
|
||||
cnameState(ip2, ip1), cnameState(ip1, ip2), 0,
|
||||
},
|
||||
{
|
||||
"address replaced",
|
||||
cnameState(ip1), cnameState(ip2), 1,
|
||||
},
|
||||
{
|
||||
"address added",
|
||||
cnameState(ip1), cnameState(ip1, ip2), 1,
|
||||
},
|
||||
{
|
||||
"no address at the end of the chain now",
|
||||
cnameState(ip1), cnameState(), 1,
|
||||
},
|
||||
{
|
||||
"state file from before addresses were saved",
|
||||
cnameState(), cnameState(ip1), 0,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
notifier := &mockNotifier{}
|
||||
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
|
||||
|
||||
w.DetectHostnameChanges(t.Context(), host, tt.prev, tt.current)
|
||||
|
||||
got := len(notifier.getNotifications())
|
||||
if got != tt.want {
|
||||
t.Errorf("sent %d notifications, want %d", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCNAMEAddressChangeAlertNamesHostnameAndAddresses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
notifier := &mockNotifier{}
|
||||
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
|
||||
|
||||
w.DetectHostnameChanges(
|
||||
t.Context(), host, cnameState(ip1), cnameState(ip2, ip3),
|
||||
)
|
||||
|
||||
want := notification{
|
||||
Title: "CNAME Address Change: " + host,
|
||||
Message: "Hostname: " + host +
|
||||
"\nOld: " + ip1 + "\nNew: " + ip2 + ", " + ip3,
|
||||
Priority: "warning",
|
||||
}
|
||||
|
||||
got := notifier.getNotifications()
|
||||
if len(got) != 1 || got[0] != want {
|
||||
t.Errorf("sent %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
@@ -57,6 +57,15 @@ func (w *Watcher) ResolveNameserverAddresses(
|
||||
return w.resolveNameserverAddresses(ctx, nameservers, prev)
|
||||
}
|
||||
|
||||
// ResolveCNAMEAddresses exports resolveCNAMEAddresses for testing.
|
||||
func (w *Watcher) ResolveCNAMEAddresses(
|
||||
ctx context.Context,
|
||||
hostname string,
|
||||
current, prev *state.HostnameState,
|
||||
) {
|
||||
w.resolveCNAMEAddresses(ctx, hostname, current, prev)
|
||||
}
|
||||
|
||||
// DetectNSAddressChanges exports detectNSAddressChanges for testing.
|
||||
func (w *Watcher) DetectNSAddressChanges(
|
||||
ctx context.Context,
|
||||
|
||||
+106
-22
@@ -252,28 +252,9 @@ func (w *Watcher) checkDomain(
|
||||
LastChecked: now,
|
||||
})
|
||||
|
||||
// Also look up A/AAAA records for the apex domain so that
|
||||
// port and TLS checks (which read HostnameState) can find
|
||||
// the domain's IP addresses.
|
||||
results, err := w.resolver.LookupAllRecords(ctx, domain)
|
||||
if err != nil {
|
||||
w.log.Error(
|
||||
"failed to lookup records for domain",
|
||||
"domain", domain,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
newState := buildHostnameState(results, now)
|
||||
|
||||
prevHS, hasPrevHS := w.state.GetHostnameState(domain)
|
||||
if hasPrevHS && !w.firstRun {
|
||||
w.detectHostnameChanges(ctx, domain, prevHS, newState)
|
||||
}
|
||||
|
||||
w.state.SetHostnameState(domain, newState)
|
||||
// The apex domain's records are also checked as a hostname's, so
|
||||
// that the port and TLS checks find its addresses.
|
||||
w.checkHostname(ctx, domain)
|
||||
}
|
||||
|
||||
func (w *Watcher) detectNSChanges(
|
||||
@@ -401,6 +382,9 @@ func (w *Watcher) checkHostname(
|
||||
newState := buildHostnameState(results, time.Now().UTC())
|
||||
|
||||
prev, hasPrev := w.state.GetHostnameState(hostname)
|
||||
|
||||
w.resolveCNAMEAddresses(ctx, hostname, newState, prev)
|
||||
|
||||
if hasPrev && !w.firstRun {
|
||||
w.detectHostnameChanges(ctx, hostname, prev, newState)
|
||||
}
|
||||
@@ -408,6 +392,68 @@ func (w *Watcher) checkHostname(
|
||||
w.state.SetHostnameState(hostname, newState)
|
||||
}
|
||||
|
||||
// resolveCNAMEAddresses saves in current the addresses at the end of
|
||||
// hostname's CNAME chain, when the nameservers' answers in current hold
|
||||
// a CNAME and no address. ResolveIPAddresses looks the name up again
|
||||
// and follows the chain. The addresses saved in prev, which may be nil,
|
||||
// are kept when none of the name's nameservers answered, and when the
|
||||
// chain cannot be followed, as when no nameserver of a zone in it
|
||||
// answers.
|
||||
func (w *Watcher) resolveCNAMEAddresses(
|
||||
ctx context.Context,
|
||||
hostname string,
|
||||
current, prev *state.HostnameState,
|
||||
) {
|
||||
var prevAddresses []string
|
||||
if prev != nil {
|
||||
prevAddresses = prev.CNAMEAddresses
|
||||
}
|
||||
|
||||
answered := false
|
||||
hasCNAME := false
|
||||
|
||||
for _, nsState := range current.RecordsByNameserver {
|
||||
if nsState.Status != statusOK {
|
||||
continue
|
||||
}
|
||||
|
||||
answered = true
|
||||
|
||||
if len(nsState.Records["A"]) > 0 || len(nsState.Records["AAAA"]) > 0 {
|
||||
return
|
||||
}
|
||||
|
||||
if len(nsState.Records["CNAME"]) > 0 {
|
||||
hasCNAME = true
|
||||
}
|
||||
}
|
||||
|
||||
if !answered {
|
||||
current.CNAMEAddresses = prevAddresses
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if !hasCNAME {
|
||||
return
|
||||
}
|
||||
|
||||
ips, err := w.resolver.ResolveIPAddresses(ctx, hostname)
|
||||
if err != nil {
|
||||
w.log.Error(
|
||||
"failed to follow CNAME",
|
||||
"hostname", hostname,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
current.CNAMEAddresses = prevAddresses
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
current.CNAMEAddresses = ips
|
||||
}
|
||||
|
||||
// buildHostnameState saves each nameserver's response. A nameserver
|
||||
// that answered, even with NXDOMAIN or no records, is saved as ok; one
|
||||
// that timed out or failed is saved as error with the reason, and its
|
||||
@@ -451,6 +497,37 @@ func (w *Watcher) detectHostnameChanges(
|
||||
w.detectNSDisappearances(ctx, hostname, prev, current)
|
||||
w.detectNSFailures(ctx, hostname, prev, current)
|
||||
w.detectInconsistencies(ctx, hostname, prev, current)
|
||||
w.detectCNAMEAddressChanges(ctx, hostname, prev, current)
|
||||
}
|
||||
|
||||
// detectCNAMEAddressChanges notifies when the addresses at the end of
|
||||
// hostname's CNAME chain differ from those the previous check saved,
|
||||
// including when there are none now. When the previous check saved
|
||||
// none, as in a state file from before they were saved, nothing is
|
||||
// compared.
|
||||
func (w *Watcher) detectCNAMEAddressChanges(
|
||||
ctx context.Context,
|
||||
hostname string,
|
||||
prev, current *state.HostnameState,
|
||||
) {
|
||||
old, cur := prev.CNAMEAddresses, current.CNAMEAddresses
|
||||
if len(old) == 0 || sliceEqual(old, cur) {
|
||||
return
|
||||
}
|
||||
|
||||
msg := fmt.Sprintf(
|
||||
"Hostname: %s\nOld: %s\nNew: %s",
|
||||
hostname,
|
||||
strings.Join(old, ", "),
|
||||
strings.Join(cur, ", "),
|
||||
)
|
||||
|
||||
w.notify.SendNotification(
|
||||
ctx,
|
||||
"CNAME Address Change: "+hostname,
|
||||
msg,
|
||||
"warning",
|
||||
)
|
||||
}
|
||||
|
||||
// detectRecordChanges compares each nameserver's records with those of
|
||||
@@ -747,6 +824,9 @@ func (w *Watcher) noNameserverAnswered(name string) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// collectIPs returns the addresses saved for hostname: those in its
|
||||
// nameservers' A and AAAA records, and those at the end of its CNAME
|
||||
// chain.
|
||||
func (w *Watcher) collectIPs(hostname string) []string {
|
||||
hs, ok := w.state.GetHostnameState(hostname)
|
||||
if !ok {
|
||||
@@ -765,6 +845,10 @@ func (w *Watcher) collectIPs(hostname string) []string {
|
||||
}
|
||||
}
|
||||
|
||||
for _, ip := range hs.CNAMEAddresses {
|
||||
ipSet[ip] = true
|
||||
}
|
||||
|
||||
result := make([]string, 0, len(ipSet))
|
||||
for ip := range ipSet {
|
||||
result = append(result, ip)
|
||||
|
||||
@@ -315,7 +315,8 @@ func lookupNameservers(t *testing.T, domain string) []string {
|
||||
return nameservers
|
||||
}
|
||||
|
||||
// addresses returns the A and AAAA values saved for a hostname.
|
||||
// addresses returns the A and AAAA values saved for a hostname, and the
|
||||
// addresses saved at the end of its CNAME chain.
|
||||
func addresses(hs *state.HostnameState) []string {
|
||||
var ips []string
|
||||
|
||||
@@ -324,7 +325,7 @@ func addresses(hs *state.HostnameState) []string {
|
||||
ips = append(ips, nsState.Records["AAAA"]...)
|
||||
}
|
||||
|
||||
return ips
|
||||
return append(ips, hs.CNAMEAddresses...)
|
||||
}
|
||||
|
||||
// assertNotified checks that a notification with this title and
|
||||
|
||||
Reference in New Issue
Block a user