6 Commits
Author SHA1 Message Date
sneak 2d9910cb14 resolver, watcher: a record type whose query fails keeps its previous records (closes #231)
check / check (push) Canceled after 0s
The resolver lists in FailedTypes each record type whose query to a
nameserver got no usable reply (none after two tries, an error reply, a
referral, or a truncated reply whose TCP retry failed), and logs it with
the reason. A nameserver that answered no type has failed, as before.
The watcher saves such a type in failedTypes, keeping the previous
check's records, leaves it out of the comparison with other nameservers
on that check, and compares the kept records with the next answer. With
nothing to keep, it is also in unknownTypes and not compared until it
answers. Change messages leave out what was not compared. A nameserver
whose A, AAAA or CNAME query failed is no answer when following a CNAME
or resolving addresses.

Model: opus-5-5
2026-10-02 06:55:24 +00:00
clawbot f99de191c0 watcher: change messages name only the record types that differ (closes #219)
check / check (push) Canceled after 0s
A Record Change notification printed the nameserver's whole old and new
record sets in Go map syntax, and an Inconsistency notification the two
nameservers' whole sets, so a one-address change had to be found by eye
among kilobytes of unchanged TXT, CAA, MX and NS values. Both now list,
in sorted order of type, only the record types whose values differ: a
line naming the type, then each side's values separated by commas, or
none when that side has no records of that type. The dashboard's Recent
alerts shows the same text.

Model: opus-5-5
2026-10-02 08:46:56 +02:00
clawbot 5db5048754 watcher: startup notification no longer says every endpoint works (closes #230)
check / check (push) Canceled after 0s
The startup notification ended "All notification channels are working.",
but it is written once and handed to every notification endpoint before
any delivery has succeeded or failed, so the claim was never checked and
was false whenever one endpoint refused it. It now says only that it is a
test sent to every configured notification endpoint. The startup
notification test checks the whole message.

Model: opus-5-5
2026-10-02 08:42:35 +02:00
clawbot 26c9c74d8e notify: a failed Mattermost delivery's error names Mattermost (closes #227)
check / check (push) Canceled after 0s
Mattermost is sent by the Slack sender, which wrapped every HTTP error
status in ErrSlackFailed, so a Mattermost endpoint answering 503 was
logged as "slack notification failed". The sender now takes the error
to wrap: the Slack endpoint passes ErrSlackFailed and the Mattermost
endpoint passes ErrMattermostFailed, which was defined but unused.

A new delivery test sets both endpoints to a stand-in server answering
503 and checks the error logged for each names its own endpoint.

Model: opus-5-5
2026-10-02 08:40:19 +02:00
clawbot ceb24c5004 log: write durations as text, not nanoseconds (closes #228)
check / check (push) Canceled after 0s
The JSON log wrote a Go duration as a bare count of nanoseconds, so
the watcher starting line showed dnsInterval 120000000000 for 2m and a
delivery retry showed retryIn 1015437050. Each duration logged is now
passed through its String() form: dnsInterval and tlsInterval when the
watcher starts, retryIn on a delivery retry, and latency on a
succeeded port check. A test checks that retryIn is logged as the
text of the wait the retry actually took.

The request log's latency_ms is left as it is: its key names its
unit.

Model: opus-5-5
2026-10-02 08:37:47 +02:00
clawbot ee4cadbd05 watcher: follow a watched name's CNAME for port and TLS checks (closes #203)
check / check (push) Canceled after 0s
When a watched name's nameservers answer with a CNAME and no address,
the DNS check follows every target they gave with ResolveIPAddresses
and saves all addresses found as cnameAddresses in the hostname state,
so nameservers disagreeing on the target do not change them between
checks. Port and TLS checks use them. A change, also from or to none,
is notified as a CNAME address change; the first check from a state
file without them sends none. When a target cannot be followed, or
none of the name's nameservers answered, the last check's addresses
are kept. The domain check now runs the hostname check for the apex
instead of a copy of it.

Model: opus-5-5
2026-10-02 08:26:27 +02:00
24 changed files with 1622 additions and 96 deletions
+58 -11
View File
@@ -80,7 +80,8 @@ notification endpoint set, changes show only on the dashboard; see
different addresses than on the previous check. A nameserver added or different addresses than on the previous check. A nameserver added or
removed gets only the NS change notification. When the lookup of a removed gets only the NS change notification. When the lookup of a
nameserver's addresses fails or finds none, its previous addresses are nameserver's addresses fails or finds none, its previous addresses are
kept and nothing is sent. kept and nothing is sent. The lookup fails when no nameserver it asks
answers every one of its queries, for A, AAAA and CNAME.
### DNS Hostname Monitoring (Subdomains) ### DNS Hostname Monitoring (Subdomains)
@@ -91,6 +92,18 @@ notification endpoint set, changes show only on the dashboard; see
its last two labels (a name under `co.uk`, or in a delegated subdomain). its last two labels (a name under `co.uk`, or in a delegated subdomain).
- Queries **each** authoritative nameserver independently for **all** record - Queries **each** authoritative nameserver independently for **all** record
types: A, AAAA, CNAME, MX, TXT, SRV, CAA, NS. types: A, AAAA, CNAME, MX, TXT, SRV, CAA, NS.
- Each record type is a query of its own. When a nameserver answers some types
but the query for another gets no usable reply (no reply after two tries, an
error reply such as SERVFAIL, a referral, or a reply too large for UDP whose
retry over TCP fails), the failure is logged with the reason, and the type is
listed in the nameserver's `failedTypes` and keeps the records saved for the
nameserver by the previous check. On that check those records are not compared
with the other nameservers', so no record change or inconsistency is reported
for the type; on the next check they are compared with the nameserver's answer
as usual. When there are none to keep, because the nameserver was new or
failing on the previous check, the type is also listed in `unknownTypes` and
left out of every comparison until it answers. A nameserver none of whose
queries got a usable reply has failed (see NS query failure below).
- Stores results **per nameserver**. The state for a hostname is not a merged - Stores results **per nameserver**. The state for a hostname is not a merged
view — it is a map from nameserver to record set. view — it is a map from nameserver to record set.
- DNS names inside record values (CNAME, MX, SRV and NS targets) are stored in - DNS names inside record values (CNAME, MX, SRV and NS targets) are stored in
@@ -119,16 +132,29 @@ notification endpoint set, changes show only on the dashboard; see
they keep disagreeing, including after a restart. A nameserver that was they keep disagreeing, including after a restart. A nameserver that was
not in the previous check (newly added, or back after dropping out), or not in the previous check (newly added, or back after dropping out), or
failed on it, and answers differently is reported on the check where it failed on it, and answers differently is reported on the check where it
answers. If a pair agrees again and later disagrees, the alert is sent answers. So is a pair that differs in a record type whose query to either
again. nameserver failed on the previous check. If a pair agrees again and later
disagrees, the alert is sent again.
- **CNAME address change**: The addresses at the end of a name's CNAME chain
differ from those of the previous check. They are found when its
nameservers answer with a CNAME and no address; a name that answers with
an address has none. A change from or to no addresses is sent too, as when
a name moves between A records and a CNAME. Nothing is sent when the
previous addresses were kept because a chain could not be followed or none
of the name's nameservers answered. The first check after loading a state
file without `cnameAddresses` sends nothing: it saves the addresses it
finds for the next check to compare.
### TCP Port Monitoring ### TCP Port Monitoring
- For every configured domain and hostname, constructs a deduplicated list of - For every configured domain and hostname, constructs a deduplicated list of
the IPv4 and IPv6 addresses in the A and AAAA records its authoritative the IPv4 and IPv6 addresses in the A and AAAA records its authoritative
nameservers returned. A CNAME is not followed: a name whose CNAME points into nameservers returned. When they returned a CNAME and no address, the CNAME
another zone usually has no addresses here, so its ports and certificate are chain is followed and the addresses at its end are used, and a change in those
not checked. is notified as a CNAME address change. When the nameservers gave different
CNAME targets, each is followed and the addresses of all are used. When a
chain cannot be followed, or none of the name's nameservers answered, the
addresses the last check found at its end are used.
- Checks TCP connectivity on ports **80** and **443** for each IP address. - Checks TCP connectivity on ports **80** and **443** for each IP address.
- Every **1 hour** by default, re-checks all ports. - Every **1 hour** by default, re-checks all ports.
- Any change in port availability triggers a notification: - Any change in port availability triggers a notification:
@@ -176,6 +202,8 @@ includes:
- **DNS NS changes**: Which domain, which nameservers were added/removed. - **DNS NS changes**: Which domain, which nameservers were added/removed.
- **NS address changes**: Which domain, which nameserver, its old and new - **NS address changes**: Which domain, which nameserver, its old and new
addresses. addresses.
- **CNAME address changes**: Which hostname, the old and new addresses at the
end of its CNAME chain.
- **NS query failures**: Which nameserver failed, error type (timeout, SERVFAIL, - **NS query failures**: Which nameserver failed, error type (timeout, SERVFAIL,
REFUSED, network error), which hostname/domain affected. REFUSED, network error), which hostname/domain affected.
- **NS recoveries**: Which nameserver recovered, which hostname/domain. - **NS recoveries**: Which nameserver recovered, which hostname/domain.
@@ -420,9 +448,11 @@ This approach ensures:
- Ability to detect split-horizon or inconsistent responses across authoritative - Ability to detect split-horizon or inconsistent responses across authoritative
servers. servers.
CNAME chains are followed (with a depth limit to prevent loops) only to find the A watched name's records are stored as its nameservers return them, CNAME
addresses of nameservers. A watched name's records are stored as its nameservers included. When they return a CNAME and no address, the chain of every CNAME
return them, CNAME included, without following it. target they gave is followed (with a depth limit to prevent loops) to the A and
AAAA records at its end, and the port and TLS checks use those addresses.
Nameservers' addresses are also found by following CNAME chains.
Sending a notification or a Sentry report is the one use of the system's Sending a notification or a Sentry report is the one use of the system's
resolver: the HTTP client looks up the webhook's or Sentry's host name with it. resolver: the HTTP client looks up the webhook's or Sentry's host name with it.
@@ -469,6 +499,7 @@ merged view, to enable inconsistency detection.
"lastChecked": "2026-02-19T12:00:00Z" "lastChecked": "2026-02-19T12:00:00Z"
} }
}, },
"cnameAddresses": [],
"lastChecked": "2026-02-19T12:00:00Z" "lastChecked": "2026-02-19T12:00:00Z"
} }
}, },
@@ -502,7 +533,7 @@ reachability:
| Status | Meaning | | Status | Meaning |
| ------- | -------------------------------------------------------- | | ------- | -------------------------------------------------------- |
| `ok` | Query succeeded, records are current | | `ok` | Query succeeded, records are current except as below |
| `error` | Query failed (timeout, SERVFAIL, REFUSED, network error) | | `error` | Query failed (timeout, SERVFAIL, REFUSED, network error) |
A nameserver that answers NXDOMAIN or with no records has status `ok` and empty A nameserver that answers NXDOMAIN or with no records has status `ok` and empty
@@ -511,10 +542,24 @@ nameservers, has status `error`, empty `records`, and the reason in `error`. A
certificate entry whose TLS connection or handshake failed likewise has status certificate entry whose TLS connection or handshake failed likewise has status
`error`, the reason in `error`, and the certificate fields left empty or zero. `error`, the reason in `error`, and the certificate fields left empty or zero.
A nameserver with status `ok` whose query for one record type failed lists that
type in `failedTypes` and holds its records from the previous check, which may
not be current. When there were none to keep, the type is also listed in
`unknownTypes`, and `records` holds nothing for it. Both lists are left out when
empty.
`nameserverAddresses` lists, by nameserver, the sorted addresses its name `nameserverAddresses` lists, by nameserver, the sorted addresses its name
resolves to. A state file without it loads, and the next check fills it in resolves to. A state file without it loads, and the next check fills it in
without a notification. without a notification.
`cnameAddresses` lists the sorted addresses at the end of the chain of every
CNAME target a hostname's nameservers gave, found when they answered with a
CNAME and no address; it is empty when they answered with an address. When a
chain cannot be followed, or none of the name's nameservers answered its queries
for A, AAAA and CNAME, the previous check's list is kept, or `null` when no
earlier check saved one. A state file without it loads, and the first check
after that saves it without a notification.
A port entry in the older format, with one `hostname` instead of the `hostnames` A port entry in the older format, with one `hostname` instead of the `hostnames`
list, loads as a list of that one name. list, loads as a list of that one name.
@@ -672,7 +717,9 @@ docker run -d \
- Port and TLS checks use the IP addresses found by the DNS phase that - Port and TLS checks use the IP addresses found by the DNS phase that
immediately precedes them. When that phase cannot find a name's immediately precedes them. When that phase cannot find a name's
nameservers at all, the addresses an earlier check saved for the name are nameservers at all, the addresses an earlier check saved for the name are
used. used. When it cannot follow a name's CNAME chain, or none of the name's
nameservers answered, the addresses an earlier check found at the end of
the chain are used.
4. **On change detection**: Send notifications to all configured endpoints, 4. **On change detection**: Send notifications to all configured endpoints,
update in-memory state, persist to disk. update in-memory state, persist to disk.
5. **Shutdown**: The watcher stops checking and saves the final state to disk, 5. **Shutdown**: The watcher stops checking and saves the final state to disk,
+12
View File
@@ -19,6 +19,18 @@ trial run of the finished image: https://git.eeqj.de/sneak/dnswatcher/issues/149
# Completed Steps # Completed Steps
- 2026-10-02: a record type whose query to a nameserver fails keeps its previous
records and alerts nothing; the other types are still saved (closes #231).
- 2026-10-02: Record Change and Inconsistency notifications list only the record
types that differ, each with its values as plain text (closes #219).
- 2026-10-02: the startup notification no longer says every notification
endpoint works; it says it is a test sent to each of them (closes #230).
- 2026-10-02: a Mattermost webhook that answers an HTTP error is logged as
`mattermost notification failed`, not as a Slack failure (closes #227).
- 2026-10-02: durations in the log are written as text such as `2m0s`, not as a
bare count of nanoseconds (closes #228).
- 2026-10-02: a watched name whose nameservers answer with a CNAME and no
address gets port and TLS checks at the end of its CNAME chain (closes #203).
- 2026-10-02: a resolver test that reads one record type from a nameserver's - 2026-10-02: a resolver test that reads one record type from a nameserver's
answer asks again when that type is missing from it (closes #218). answer asks again when that type is missing from it (closes #218).
- 2026-10-02: a plain `docker build .` of a clone stamps its tag or short - 2026-10-02: a plain `docker build .` of a clone stamps its tag or short
+69 -3
View File
@@ -6,9 +6,11 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"io" "io"
"maps"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/url" "net/url"
"strings"
"sync" "sync"
"testing" "testing"
"time" "time"
@@ -413,7 +415,8 @@ func sendSlackInfo(
svc *notify.Service, target *url.URL, svc *notify.Service, target *url.URL,
) error { ) error {
return svc.SendSlack( return svc.SendSlack(
context.Background(), target, "t", "m", prioInfo, context.Background(), target, notify.ErrSlackFailed,
"t", "m", prioInfo,
) )
} }
@@ -506,6 +509,7 @@ func TestSendSlackPayloadFields(t *testing.T) {
err := svc.SendSlack( err := svc.SendSlack(
context.Background(), context.Background(),
webhookURL, webhookURL,
notify.ErrSlackFailed,
"Alert Title", "Alert Title",
"Alert body text", "Alert body text",
"warning", "warning",
@@ -608,7 +612,8 @@ func TestSendSlackAllColors(t *testing.T) {
err := svc.SendSlack( err := svc.SendSlack(
context.Background(), context.Background(),
webhookURL, "t", "m", tc.priority, webhookURL, notify.ErrSlackFailed,
"t", "m", tc.priority,
) )
if err != nil { if err != nil {
t.Fatalf("SendSlack error: %v", err) t.Fatalf("SendSlack error: %v", err)
@@ -659,7 +664,8 @@ func TestSendSlackNetworkError(t *testing.T) {
) )
err := svc.SendSlack( err := svc.SendSlack(
context.Background(), webhookURL, "t", "m", "info", context.Background(), webhookURL, notify.ErrSlackFailed,
"t", "m", "info",
) )
if err == nil { if err == nil {
t.Fatal("expected error for network failure") t.Fatal("expected error for network failure")
@@ -1028,6 +1034,66 @@ func TestSendNotificationMattermostError(t *testing.T) {
) )
} }
// TestSendNotificationErrorNamesEndpoint verifies that, with both
// Slack and Mattermost set, a failed delivery's logged error names
// the endpoint that failed. Both are sent by the Slack sender.
func TestSendNotificationErrorNamesEndpoint(t *testing.T) {
t.Parallel()
srv := httptest.NewServer(
http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusServiceUnavailable)
}),
)
defer srv.Close()
target, _ := url.Parse(srv.URL)
svc, logs := newLoggingService(http.DefaultTransport)
svc.SetSlackWebhookURL(target)
svc.SetMattermostWebhookURL(target)
svc.SetSleepFunc(instantSleep)
svc.SetRetryConfig(notify.RetryConfig{
MaxRetries: 1,
BaseDelay: time.Millisecond,
MaxDelay: time.Millisecond,
})
svc.SendNotification(
context.Background(), "t", "m", prioError,
)
waitForCondition(t, func() bool {
return svc.OutstandingDeliveries() == 0
})
got := map[string]string{}
for line := range strings.Lines(logs.String()) {
var record struct {
Msg string `json:"msg"`
Endpoint string `json:"endpoint"`
Error string `json:"error"`
}
_ = json.Unmarshal([]byte(line), &record)
if record.Msg == "failed to send notification after retries" {
got[record.Endpoint] = record.Error
}
}
want := map[string]string{
"slack": "slack notification failed: status 503",
"mattermost": "mattermost notification failed: status 503",
}
if !maps.Equal(got, want) {
t.Errorf("logged errors = %v, want %v", got, want)
}
}
// ── SlackPayload JSON marshaling ────────────────────────── // ── SlackPayload JSON marshaling ──────────────────────────
func TestSlackPayloadJSON(t *testing.T) { func TestSlackPayloadJSON(t *testing.T) {
+2 -1
View File
@@ -85,10 +85,11 @@ func (svc *Service) SendNtfy(
func (svc *Service) SendSlack( func (svc *Service) SendSlack(
ctx context.Context, ctx context.Context,
webhookURL *url.URL, webhookURL *url.URL,
failed error,
title, message, priority string, title, message, priority string,
) error { ) error {
return svc.sendSlack( return svc.sendSlack(
ctx, webhookURL, title, message, priority, ctx, webhookURL, failed, title, message, priority,
) )
} }
+9 -3
View File
@@ -277,7 +277,8 @@ func (svc *Service) dispatchSlack(
svc.dispatch(ctx, "slack", func(c context.Context) error { svc.dispatch(ctx, "slack", func(c context.Context) error {
return svc.sendSlack( return svc.sendSlack(
c, svc.slackWebhookURL, title, message, priority, c, svc.slackWebhookURL, ErrSlackFailed,
title, message, priority,
) )
}) })
} }
@@ -294,7 +295,7 @@ func (svc *Service) dispatchMattermost(
ctx, "mattermost", ctx, "mattermost",
func(c context.Context) error { func(c context.Context) error {
return svc.sendSlack( return svc.sendSlack(
c, svc.mattermostWebhookURL, c, svc.mattermostWebhookURL, ErrMattermostFailed,
title, message, priority, title, message, priority,
) )
}, },
@@ -370,9 +371,14 @@ type SlackAttachment struct {
Text string `json:"text"` Text string `json:"text"`
} }
// sendSlack posts to a Slack or Mattermost incoming webhook, which
// take the same payload. An HTTP error status is returned wrapped in
// failed, ErrSlackFailed or ErrMattermostFailed, so the error names
// the endpoint.
func (svc *Service) sendSlack( func (svc *Service) sendSlack(
ctx context.Context, ctx context.Context,
webhookURL *url.URL, webhookURL *url.URL,
failed error,
title, message, priority string, title, message, priority string,
) error { ) error {
ctx, cancel := context.WithTimeout( ctx, cancel := context.WithTimeout(
@@ -420,7 +426,7 @@ func (svc *Service) sendSlack(
if resp.StatusCode >= httpStatusClientError { if resp.StatusCode >= httpStatusClientError {
return fmt.Errorf( return fmt.Errorf(
"%w: status %d", "%w: status %d",
ErrSlackFailed, resp.StatusCode, failed, resp.StatusCode,
) )
} }
+3 -1
View File
@@ -115,7 +115,9 @@ func (svc *Service) deliverWithRetry(
"endpoint", endpoint, "endpoint", endpoint,
"attempt", attempt+1, "attempt", attempt+1,
"maxAttempts", cfg.MaxRetries+1, "maxAttempts", cfg.MaxRetries+1,
"retryIn", delay, // As text: the JSON log writes a time.Duration as
// bare nanoseconds.
"retryIn", delay.String(),
"error", lastErr, "error", lastErr,
) )
+45
View File
@@ -2,6 +2,7 @@ package notify_test
import ( import (
"context" "context"
"encoding/json"
"errors" "errors"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
@@ -189,6 +190,50 @@ func TestDeliverWithRetryExhaustsAttempts(t *testing.T) {
} }
} }
// TestDeliverWithRetryLogsRetryInAsText checks that the wait
// before a retry is logged as text such as "1.02s", not as a
// count of nanoseconds.
func TestDeliverWithRetryLogsRetryInAsText(t *testing.T) {
t.Parallel()
svc, logs := newLoggingService(http.DefaultTransport)
svc.SetRetryConfig(notify.RetryConfig{
MaxRetries: 1,
BaseDelay: time.Second,
MaxDelay: time.Second,
})
var waited time.Duration
svc.SetSleepFunc(func(d time.Duration) <-chan time.Time {
waited = d
return instantSleep(d)
})
_ = svc.DeliverWithRetry(
context.Background(), "test",
func(_ context.Context) error {
return errFail
},
)
// With one retry, only the first failure is logged.
var record map[string]any
err := json.Unmarshal([]byte(logs.String()), &record)
if err != nil {
t.Fatalf("log is not one JSON record: %v\n%s", err, logs)
}
if record["retryIn"] != waited.String() {
t.Errorf(
"retryIn logged as %v, want %q",
record["retryIn"], waited.String(),
)
}
}
func TestDeliverWithRetryRespectsContextCancellation( func TestDeliverWithRetryRespectsContextCancellation(
t *testing.T, t *testing.T,
) { ) {
+3 -1
View File
@@ -193,7 +193,9 @@ func (c *Checker) checkConnection(
c.log.Debug( c.log.Debug(
"port check succeeded", "port check succeeded",
"target", target, "target", target,
"latency", latency, // As text: the JSON log writes a time.Duration as bare
// nanoseconds.
"latency", latency.String(),
) )
return &PortResult{ return &PortResult{
+6
View File
@@ -22,6 +22,12 @@ var (
"reply is an error or a referral that leads no closer", "reply is an error or a referral that leads no closer",
) )
// ErrTruncated is the reason given for a reply too large for UDP
// whose retry over TCP failed.
ErrTruncated = errors.New(
"reply truncated and its retry over TCP failed",
)
// ErrIntercepted is returned when every root server refused a // ErrIntercepted is returned when every root server refused a
// query. Root servers refuse no query, so the refusals came from // query. Root servers refuse no query, so the refusals came from
// something on the network answering in their place. // something on the network answering in their place.
+11
View File
@@ -2,10 +2,21 @@ package resolver
import ( import (
"context" "context"
"log/slog"
"time"
"github.com/miekg/dns" "github.com/miekg/dns"
) )
// NewWithFailingTCP returns a Resolver whose TCP client gives up before
// it can connect, so the retry over TCP of every truncated reply fails.
func NewWithFailingTCP(log *slog.Logger) *Resolver {
r := NewFromLogger(log)
r.tcp = &tcpClient{timeout: time.Nanosecond}
return r
}
// ExtractRecordValue exports extractRecordValue for testing. // ExtractRecordValue exports extractRecordValue for testing.
func ExtractRecordValue(rr dns.RR) string { func ExtractRecordValue(rr dns.RR) string {
return extractRecordValue(rr) return extractRecordValue(rr)
+64 -15
View File
@@ -89,6 +89,9 @@ func (r *Resolver) tryExchange(
return resp, err return resp, err
} }
// retryTCP returns the reply to msg over TCP when resp, its reply over
// UDP, is truncated. When that fails it returns resp, still truncated,
// which holds only the records that fit.
func (r *Resolver) retryTCP( func (r *Resolver) retryTCP(
ctx context.Context, ctx context.Context,
msg *dns.Msg, msg *dns.Msg,
@@ -638,8 +641,12 @@ type queryState struct {
gotReferral bool gotReferral bool
netErr error netErr error
hasRecords bool hasRecords bool
answered bool
} }
// queryEachType asks the nameserver at nsIP about hostname once for each
// record type in qtypes, and lists in resp.FailedTypes the types whose
// query got no usable reply, logging each with the reason.
func (r *Resolver) queryEachType( func (r *Resolver) queryEachType(
ctx context.Context, ctx context.Context,
nsIP string, nsIP string,
@@ -654,7 +661,30 @@ func (r *Resolver) queryEachType(
break break
} }
r.querySingleType(ctx, nsIP, hostname, qtype, resp, &state) err := r.querySingleType(ctx, nsIP, hostname, qtype, resp, &state)
if err == nil {
state.answered = true
continue
}
rtype := dns.TypeToString[qtype]
resp.FailedTypes = append(resp.FailedTypes, rtype)
r.log.Warn(
"record type query failed",
"hostname", hostname,
"nameserver", resp.Nameserver,
"type", rtype,
"error", err,
)
}
// The reply about another type can carry the name's CNAME. When the
// query for CNAME itself failed, that is left out too, so Records
// holds nothing for a failed type.
for _, rtype := range resp.FailedTypes {
delete(resp.Records, rtype)
} }
for k := range resp.Records { for k := range resp.Records {
@@ -664,6 +694,9 @@ func (r *Resolver) queryEachType(
return state return state
} }
// querySingleType asks the nameserver at nsIP about hostname's records
// of type qtype. It returns nil when the nameserver answered: with
// records, with none, or with NXDOMAIN; otherwise it returns why not.
func (r *Resolver) querySingleType( func (r *Resolver) querySingleType(
ctx context.Context, ctx context.Context,
nsIP string, nsIP string,
@@ -671,7 +704,7 @@ func (r *Resolver) querySingleType(
qtype uint16, qtype uint16,
resp *NameserverResponse, resp *NameserverResponse,
state *queryState, state *queryState,
) { ) error {
msg, err := r.queryDNS(ctx, nsIP, hostname, qtype) msg, err := r.queryDNS(ctx, nsIP, hostname, qtype)
if err != nil { if err != nil {
switch { switch {
@@ -683,19 +716,19 @@ func (r *Resolver) querySingleType(
state.netErr = err state.netErr = err
} }
return return err
} }
if msg.Rcode == dns.RcodeNameError { if msg.Rcode == dns.RcodeNameError {
state.gotNXDomain = true state.gotNXDomain = true
return return nil
} }
if msg.Rcode == dns.RcodeServerFailure { if msg.Rcode == dns.RcodeServerFailure {
state.gotSERVFAIL = true state.gotSERVFAIL = true
return return fmt.Errorf("server returned SERVFAIL: %w", ErrUnusableReply)
} }
// A reply with no answer that lists other nameservers, from a server // A reply with no answer that lists other nameservers, from a server
@@ -708,10 +741,20 @@ func (r *Resolver) querySingleType(
len(extractNSSet(msg.Ns)) > 0 { len(extractNSSet(msg.Ns)) > 0 {
state.gotReferral = true state.gotReferral = true
return return fmt.Errorf("server returned a referral: %w", ErrUnusableReply)
}
// A reply still truncated is one whose TCP retry failed, and holds
// only the records that fit.
if msg.Truncated {
state.netErr = ErrTruncated
return ErrTruncated
} }
collectAnswerRecords(msg, resp, state) collectAnswerRecords(msg, resp, state)
return nil
} }
func collectAnswerRecords( func collectAnswerRecords(
@@ -743,23 +786,26 @@ func isTimeout(err error) bool {
return false return false
} }
// classifyResponse sets the nameserver's status. One that answered no
// record type has failed, and Error says why; one that answered some has
// the status of those answers.
func classifyResponse(resp *NameserverResponse, state queryState) { func classifyResponse(resp *NameserverResponse, state queryState) {
switch { switch {
case state.gotNXDomain && !state.hasRecords: case state.gotNXDomain && !state.hasRecords:
resp.Status = StatusNXDomain resp.Status = StatusNXDomain
case state.gotTimeout && !state.hasRecords: case state.gotTimeout && !state.answered:
resp.Status = StatusTimeout resp.Status = StatusTimeout
resp.Error = "all queries timed out" resp.Error = "all queries timed out"
case state.gotSERVFAIL && !state.hasRecords: case state.gotSERVFAIL && !state.answered:
resp.Status = StatusError resp.Status = StatusError
resp.Error = "server returned SERVFAIL" resp.Error = "server returned SERVFAIL"
case state.gotRefused && !state.hasRecords: case state.gotRefused && !state.answered:
resp.Status = StatusError resp.Status = StatusError
resp.Error = "server returned REFUSED" resp.Error = "server returned REFUSED"
case state.netErr != nil && !state.hasRecords: case state.netErr != nil && !state.answered:
resp.Status = StatusError resp.Status = StatusError
resp.Error = "network error: " + state.netErr.Error() resp.Error = "network error: " + state.netErr.Error()
case state.gotReferral && !state.hasRecords: case state.gotReferral && !state.answered:
resp.Status = StatusError resp.Status = StatusError
resp.Error = "server returned a referral" resp.Error = "server returned a referral"
case !state.hasRecords && !state.gotNXDomain: case !state.hasRecords && !state.gotNXDomain:
@@ -920,9 +966,11 @@ func (r *Resolver) resolveIPWithCNAME(
} }
// collectIPs returns the addresses in the nameservers' answers and the // collectIPs returns the addresses in the nameservers' answers and the
// first CNAME target among them. It returns ErrNoNameserverAnswered when // first CNAME target among them. A nameserver whose query for one of the
// every nameserver timed out, failed or returned a referral: that is not // types failed gave only part of the addresses, and is left out. It
// a name with no addresses. // returns ErrNoNameserverAnswered when every nameserver timed out,
// failed, returned a referral or was left out: that is not a name with
// no addresses.
func collectIPs( func collectIPs(
results map[string]*NameserverResponse, results map[string]*NameserverResponse,
) ([]string, string, error) { ) ([]string, string, error) {
@@ -935,7 +983,8 @@ func collectIPs(
answered := false answered := false
for _, resp := range results { for _, resp := range results {
if resp.Status == StatusTimeout || resp.Status == StatusError { if resp.Status == StatusTimeout || resp.Status == StatusError ||
len(resp.FailedTypes) > 0 {
continue continue
} }
@@ -0,0 +1,45 @@
package resolver
import (
"testing"
"github.com/stretchr/testify/assert"
)
// TestClassifyResponse sets a nameserver's status from the results of
// its queries, built here. One that answered some record types, even
// with no records, has not failed when its query for another type timed
// out; one whose every query timed out has.
func TestClassifyResponse(t *testing.T) {
t.Parallel()
tests := []struct {
name string
results queryState
wantStatus string
wantError string
}{
{
"some types answered with no records, another timed out",
queryState{answered: true, gotTimeout: true},
StatusNoData, "",
},
{
"every query timed out",
queryState{gotTimeout: true},
StatusTimeout, "all queries timed out",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
resp := &NameserverResponse{Status: StatusOK}
classifyResponse(resp, tt.results)
assert.Equal(t, tt.wantStatus, resp.Status)
assert.Equal(t, tt.wantError, resp.Error)
})
}
}
+19
View File
@@ -43,6 +43,25 @@ func TestCollectIPs_FailedIsNoAnswer(t *testing.T) {
assert.Empty(t, ips) assert.Empty(t, ips)
} }
// TestCollectIPs_FailedTypeIsNoAnswer checks that a nameserver whose
// query for one of the types failed is no answer: its addresses are
// only part of them.
func TestCollectIPs_FailedTypeIsNoAnswer(t *testing.T) {
t.Parallel()
ips, _, err := resolver.CollectIPs(
map[string]*resolver.NameserverResponse{
nsExample1: {
Records: map[string][]string{"A": {"192.0.2.1"}},
FailedTypes: []string{"AAAA"},
Status: resolver.StatusOK,
},
},
)
require.ErrorIs(t, err, resolver.ErrNoNameserverAnswered)
assert.Empty(t, ips)
}
const ( const (
// exampleCom is the zone most cases of TestUsableReply and // exampleCom is the zone most cases of TestUsableReply and
// TestNSSetFrom are about, and wwwExampleCom a name in it. // TestNSSetFrom are about, and wwwExampleCom a name in it.
+8 -4
View File
@@ -31,11 +31,15 @@ type Params struct {
} }
// NameserverResponse holds one nameserver's response for a query. // NameserverResponse holds one nameserver's response for a query.
// FailedTypes lists the record types whose query got no usable reply,
// and Records holds nothing for them: their records are not known. When
// no record type got one, Status and Error say the nameserver failed.
type NameserverResponse struct { type NameserverResponse struct {
Nameserver string Nameserver string
Records map[string][]string Records map[string][]string
Status string FailedTypes []string
Error string Status string
Error string
} }
// Resolver performs iterative DNS resolution from root servers. // Resolver performs iterative DNS resolution from root servers.
+25
View File
@@ -1,6 +1,7 @@
package resolver_test package resolver_test
import ( import (
"bytes"
"context" "context"
"fmt" "fmt"
"log/slog" "log/slog"
@@ -245,6 +246,30 @@ func TestQueryNameserver_TXT(t *testing.T) {
) )
} }
// TestQueryNameserver_TruncatedReplyWhoseTCPRetryFails asks a google.com
// nameserver about google.com with a resolver whose retries over TCP
// fail. google.com's TXT records do not fit in a reply over UDP, so TXT
// is reported as failed, holding none of the records that fit, and
// logged with the reason, while the nameserver, which answered the other
// types, is ok.
func TestQueryNameserver_TruncatedReplyWhoseTCPRetryFails(t *testing.T) {
t.Parallel()
ns := findOneNSForDomain(t, newTestResolver(t), "google.com")
var logs bytes.Buffer
r := resolver.NewWithFailingTCP(slog.New(slog.NewTextHandler(&logs, nil)))
resp := liveQueryNameserver(t, r, ns, "google.com")
assert.Equal(t, resolver.StatusOK, resp.Status)
assert.Contains(t, resp.FailedTypes, "TXT")
assert.NotContains(t, resp.Records, "TXT")
assert.Contains(t, logs.String(),
"hostname=google.com. nameserver="+ns+" type=TXT error=",
)
}
func TestQueryNameserver_NXDomain(t *testing.T) { func TestQueryNameserver_NXDomain(t *testing.T) {
t.Parallel() t.Parallel()
+16 -4
View File
@@ -45,16 +45,28 @@ type DomainState struct {
} }
// NameserverRecordState holds one NS's response for a hostname. // NameserverRecordState holds one NS's response for a hostname.
// FailedTypes lists the record types whose query to the nameserver
// failed on this check: Records holds for them the records saved by the
// previous check, which are kept. UnknownTypes lists those of them whose
// records the previous check did not know either, as when the
// nameserver was new or failing then: Records holds nothing for them.
type NameserverRecordState struct { type NameserverRecordState struct {
Records map[string][]string `json:"records"` Records map[string][]string `json:"records"`
Status string `json:"status"` FailedTypes []string `json:"failedTypes,omitempty"`
Error string `json:"error,omitempty"` UnknownTypes []string `json:"unknownTypes,omitempty"`
LastChecked time.Time `json:"lastChecked"` Status string `json:"status"`
Error string `json:"error,omitempty"`
LastChecked time.Time `json:"lastChecked"`
} }
// HostnameState holds per-nameserver monitoring state for a hostname. // HostnameState holds per-nameserver monitoring state for a hostname.
// CNAMEAddresses holds the sorted addresses at the end of the name's
// CNAME chain, found when its nameservers answered with a CNAME and no
// address; it is empty otherwise. It is nil when they are not known: a
// state file written before it existed loads with it nil.
type HostnameState struct { type HostnameState struct {
RecordsByNameserver map[string]*NameserverRecordState `json:"recordsByNameserver"` RecordsByNameserver map[string]*NameserverRecordState `json:"recordsByNameserver"`
CNAMEAddresses []string `json:"cnameAddresses"`
LastChecked time.Time `json:"lastChecked"` LastChecked time.Time `json:"lastChecked"`
} }
+89
View File
@@ -188,6 +188,95 @@ func TestLoadStateFromBeforeNameserverAddresses(t *testing.T) {
} }
} }
// TestSaveLoadRoundTrip_CNAMEAddresses checks that no addresses at the
// end of a hostname's CNAME chain load as an empty list, and addresses
// that are not known load as nil: the watcher tells the two apart.
func TestSaveLoadRoundTrip_CNAMEAddresses(t *testing.T) {
t.Parallel()
dir := t.TempDir()
s := state.NewForTestWithDataDir(dir)
want := map[string][]string{
"cname.example.com": {testIP},
"none.example.com": {},
"not-known.example.com": nil,
}
for name, addresses := range want {
s.SetHostnameState(name, &state.HostnameState{
CNAMEAddresses: addresses,
})
}
err := s.Save()
if err != nil {
t.Fatalf("Save() error: %v", err)
}
loaded := state.NewForTestWithDataDir(dir)
err = loaded.Load()
if err != nil {
t.Fatalf("Load() error: %v", err)
}
for name, addresses := range want {
hs, ok := loaded.GetHostnameState(name)
if !ok {
t.Fatalf("missing hostname %s", name)
}
if !reflect.DeepEqual(hs.CNAMEAddresses, addresses) {
t.Errorf(
"%s: loaded %#v, want %#v",
name, hs.CNAMEAddresses, addresses,
)
}
}
}
// TestLoadStateFromBeforeCNAMEAddresses loads a state file written
// before the addresses at the end of a hostname's CNAME chain were
// saved. They load as not known (nil), not as none.
func TestLoadStateFromBeforeCNAMEAddresses(t *testing.T) {
t.Parallel()
dir := t.TempDir()
data := []byte(`{
"version": 1,
"lastUpdated": "2026-02-19T12:00:00Z",
"hostnames": {
"www.example.com": {
"recordsByNameserver": {},
"lastChecked": "2026-02-19T12:00:00Z"
}
}
}`)
err := os.WriteFile(filepath.Join(dir, "state.json"), data, 0o600)
if err != nil {
t.Fatalf("writing state file: %v", err)
}
s := state.NewForTestWithDataDir(dir)
err = s.Load()
if err != nil {
t.Fatalf("Load() error: %v", err)
}
hs, ok := s.GetHostnameState(testHostname)
if !ok {
t.Fatal("missing hostname " + testHostname)
}
if hs.CNAMEAddresses != nil {
t.Errorf("CNAME addresses: got %#v, want nil", hs.CNAMEAddresses)
}
}
// TestSaveLoadRoundTrip_Hostnames verifies hostname data survives a save/load cycle. // TestSaveLoadRoundTrip_Hostnames verifies hostname data survives a save/load cycle.
func TestSaveLoadRoundTrip_Hostnames(t *testing.T) { func TestSaveLoadRoundTrip_Hostnames(t *testing.T) {
t.Parallel() t.Parallel()
+372
View File
@@ -0,0 +1,372 @@
package watcher_test
import (
"context"
"log/slog"
"slices"
"testing"
"sneak.berlin/go/dnswatcher/internal/livednstest"
"sneak.berlin/go/dnswatcher/internal/resolver"
"sneak.berlin/go/dnswatcher/internal/state"
"sneak.berlin/go/dnswatcher/internal/watcher"
)
// TestCNAMEIntoAnotherZonePortAndTLSChecks runs the port and TLS
// checks on hostname state built here: the name's nameserver answered
// with a CNAME into another zone, and following it found ip1. Both
// checks must use ip1. They look nothing up, so the watcher has no
// resolver.
func TestCNAMEIntoAnotherZonePortAndTLSChecks(t *testing.T) {
t.Parallel()
cfg := defaultTestConfig(t)
cfg.Hostnames = []string{host}
deps := newTestDeps(t, cfg)
w := watcher.NewForTest(
cfg, deps.state, nil,
deps.portChecker, deps.tlsChecker, deps.notifier,
)
deps.state.SetHostnameState(host, cnameState(ip1))
w.CheckAllPorts(t.Context())
w.RunTLSChecks(t.Context())
snap := deps.state.GetSnapshot()
ps, ok := snap.Ports[ip1+":443"]
if !ok || !slices.Contains(ps.Hostnames, host) {
t.Errorf("no port state for %s at %s:443", host, ip1)
}
certKey := ip1 + ":443:" + host
if _, ok := snap.Certificates[certKey]; !ok {
t.Errorf("no certificate state %s", certKey)
}
}
// TestCNAMEThatCannotBeFollowedKeepsPrevious runs a check of a name, not
// the watcher's first, from the point where its records have been looked
// up: they hold a CNAME to a target under .invalid, whose lookup fails.
// The previous check found the same records, and oldIP at the end of the
// CNAME. The check must keep oldIP and send nothing.
func TestCNAMEThatCannotBeFollowedKeepsPrevious(t *testing.T) {
t.Parallel()
w, deps := newTestWatcher(t, defaultTestConfig(t))
w.SetFirstRun(false)
records := map[string]map[string][]string{
nsA: cnameTo("target.example.invalid."),
}
prev := hostnameState(records)
prev.CNAMEAddresses = []string{oldIP}
deps.state.SetHostnameState(host, prev)
// The result is the same whether or not live DNS answers, so the
// lookup is not retried.
_ = livednstest.Run(func(ctx context.Context) error {
w.UpdateHostnameState(ctx, host, hostnameState(records))
return nil
})
hs, _ := deps.state.GetHostnameState(host)
if !slices.Equal(hs.CNAMEAddresses, prev.CNAMEAddresses) {
t.Errorf(
"saved %v, want %v",
hs.CNAMEAddresses, prev.CNAMEAddresses,
)
}
notifications := deps.notifier.getNotifications()
if len(notifications) != 0 {
t.Errorf("sent %v, want no notifications", notifications)
}
}
// followLive follows in live DNS the CNAMEs in a name's records, built
// from records, and returns the addresses saved for the name. The
// previous check saved oldIP, which is kept when a target cannot be
// followed; that is retried. The tests point CNAMEs only at names in
// zones with two nameservers, to keep queries few (see the top of
// watcher_test.go).
func followLive(
t *testing.T,
records map[string]map[string][]string,
) []string {
t.Helper()
w := watcher.NewForTest(
nil, nil, resolver.NewFromLogger(slog.Default()), nil, nil, nil,
)
prev := cnameState(oldIP)
var current *state.HostnameState
livednstest.Retry(t, "following CNAMEs", func(ctx context.Context) error {
current = hostnameState(records)
w.ResolveCNAMEAddresses(ctx, host, current, prev)
if slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
return livednstest.ErrNoAnswer
}
return nil
})
return current.CNAMEAddresses
}
// TestCNAMEAddressesOfEveryTarget gives a name's two nameservers
// different CNAME targets, as when a secondary still serves an old one.
// The addresses at the end of both are saved, whichever answer is read
// first: one.one.one.one has 1.1.1.1, and dns.adguard-dns.com has
// 94.140.14.14.
func TestCNAMEAddressesOfEveryTarget(t *testing.T) {
t.Parallel()
found := followLive(t, map[string]map[string][]string{
nsA: cnameTo("one.one.one.one."),
nsB: cnameTo("dns.adguard-dns.com."),
})
for _, ip := range []string{"1.1.1.1", "94.140.14.14"} {
if !slices.Contains(found, ip) {
t.Errorf("saved %v, want %s among them", found, ip)
}
}
}
// TestCNAMEChainEndingInNoAddressSavesEmptyList follows a CNAME to a
// name live DNS answers with NXDOMAIN. An empty list is saved, not nil,
// which would mean the addresses are not known.
func TestCNAMEChainEndingInNoAddressSavesEmptyList(t *testing.T) {
t.Parallel()
found := followLive(t, map[string]map[string][]string{
nsA: cnameTo("this-surely-does-not-exist-xyz.example.org."),
})
if found == nil || len(found) != 0 {
t.Errorf("saved %#v, want an empty list", found)
}
}
// TestCNAMEBesideAnAddressNotFollowed gives one nameserver of a name an
// address and another a CNAME. The CNAME is not followed: an empty list
// is saved, not nil, and nothing is looked up, the watcher having no
// resolver.
func TestCNAMEBesideAnAddressNotFollowed(t *testing.T) {
t.Parallel()
w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
current := hostnameState(map[string]map[string][]string{
nsA: {"A": {ip1}},
nsB: cnameTo("target.example.org."),
})
w.ResolveCNAMEAddresses(t.Context(), host, current, nil)
if current.CNAMEAddresses == nil || len(current.CNAMEAddresses) != 0 {
t.Errorf("saved %#v, want an empty list", current.CNAMEAddresses)
}
}
// TestCNAMEWhoseNameserversAllFailedKeepsPrevious checks a name none of
// whose nameservers answered. The addresses the previous check saved
// from following its CNAME are kept, and nothing is looked up: the
// watcher has no resolver.
func TestCNAMEWhoseNameserversAllFailedKeepsPrevious(t *testing.T) {
t.Parallel()
w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
current := saved(map[string]*state.NameserverRecordState{
nsA: failed(), nsB: failed(),
})
prev := cnameState(oldIP)
w.ResolveCNAMEAddresses(t.Context(), host, current, prev)
if !slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
t.Errorf(
"saved %v, want %v",
current.CNAMEAddresses, prev.CNAMEAddresses,
)
}
}
// TestCNAMEWhoseAddressQueryFailedKeepsPrevious checks a name whose
// nameserver answered, but whose query for A, AAAA or CNAME failed with
// nothing kept for it. That is not an answer with no address: the
// addresses the previous check saved from following its CNAME are kept,
// and nothing is looked up, the watcher having no resolver.
func TestCNAMEWhoseAddressQueryFailedKeepsPrevious(t *testing.T) {
t.Parallel()
for _, rtype := range []string{"A", "AAAA", "CNAME"} {
t.Run(rtype, func(t *testing.T) {
t.Parallel()
w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
current := saved(map[string]*state.NameserverRecordState{
nsA: {
Records: map[string][]string{},
FailedTypes: []string{rtype},
UnknownTypes: []string{rtype},
Status: "ok",
},
})
prev := cnameState(oldIP)
w.ResolveCNAMEAddresses(t.Context(), host, current, prev)
if !slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
t.Errorf(
"saved %v, want %v",
current.CNAMEAddresses, prev.CNAMEAddresses,
)
}
})
}
}
// cnameTo builds the records of a nameserver that answered with a CNAME
// to target and no address.
func cnameTo(target string) map[string][]string {
return map[string][]string{"CNAME": {target}}
}
// cnameState builds the state a check leaves behind for a name whose
// nameserver answered with a CNAME and no address, when following the
// CNAME found these addresses, which may be none.
func cnameState(addresses ...string) *state.HostnameState {
hs := hostnameState(map[string]map[string][]string{
nsA: cnameTo("target.example.org."),
})
hs.CNAMEAddresses = append([]string{}, addresses...)
return hs
}
func TestCNAMEAddressChangeAlerts(t *testing.T) {
t.Parallel()
// A state file written before the addresses were saved loads with
// them nil.
olderStateFile := cnameState()
olderStateFile.CNAMEAddresses = nil
// Each case is the state saved by the previous check and by the
// current one. The name's records are the same in both.
tests := []struct {
name string
prev, current *state.HostnameState
want int
}{
{
"same addresses",
cnameState(ip1, ip2), cnameState(ip1, ip2), 0,
},
{
"same addresses in another order",
cnameState(ip2, ip1), cnameState(ip1, ip2), 0,
},
{
"address replaced",
cnameState(ip1), cnameState(ip2), 1,
},
{
"address added",
cnameState(ip1), cnameState(ip1, ip2), 1,
},
{
"no address at the end of the chain now",
cnameState(ip1), cnameState(), 1,
},
{
"addresses at the end of the chain again",
cnameState(), cnameState(ip1), 1,
},
{
"state file from before addresses were saved",
olderStateFile, cnameState(ip1), 0,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
notifier := &mockNotifier{}
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
w.DetectHostnameChanges(t.Context(), host, tt.prev, tt.current)
got := len(notifier.getNotifications())
if got != tt.want {
t.Errorf("sent %d notifications, want %d", got, tt.want)
}
})
}
}
func TestCNAMEAddressChangeAlertNamesHostnameAndAddresses(t *testing.T) {
t.Parallel()
notifier := &mockNotifier{}
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
w.DetectHostnameChanges(
t.Context(), host, cnameState(ip1), cnameState(ip2, ip3),
)
want := notification{
Title: "CNAME Address Change: " + host,
Message: "Hostname: " + host +
"\nOld: " + ip1 + "\nNew: " + ip2 + ", " + ip3,
Priority: "warning",
}
got := notifier.getNotifications()
if len(got) != 1 || got[0] != want {
t.Errorf("sent %v, want %v", got, want)
}
}
// TestNameMovedFromARecordsToCNAMEAlerts checks a name that answers
// with an A record and then with a CNAME whose chain ends in ip2. The
// second check is notified as a CNAME address change from no addresses,
// beside the record change. Nothing is looked up: the watcher has no
// resolver.
func TestNameMovedFromARecordsToCNAMEAlerts(t *testing.T) {
t.Parallel()
notifier := &mockNotifier{}
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
prev := hostnameState(map[string]map[string][]string{
nsA: {"A": {ip1}},
})
w.ResolveCNAMEAddresses(t.Context(), host, prev, nil)
w.DetectHostnameChanges(t.Context(), host, prev, cnameState(ip2))
title := "CNAME Address Change: " + host
message := "Hostname: " + host + "\nOld: \nNew: " + ip2
got := notifier.getNotifications()
if !slices.ContainsFunc(got, func(n notification) bool {
return n.Title == title && n.Message == message
}) {
t.Errorf("sent %v, want %q with %q among them", got, title, message)
}
}
+26 -1
View File
@@ -38,6 +38,21 @@ func NewlyDisagreeingPairs(
return newlyDisagreeingPairs(prev, current) return newlyDisagreeingPairs(prev, current)
} }
// SetFirstRun sets whether the watcher is on its first check, in which
// nothing is compared with the previous check. NewForTest's watcher is.
func (w *Watcher) SetFirstRun(firstRun bool) {
w.firstRun = firstRun
}
// UpdateHostnameState exports updateHostnameState for testing.
func (w *Watcher) UpdateHostnameState(
ctx context.Context,
hostname string,
newState *state.HostnameState,
) {
w.updateHostnameState(ctx, hostname, newState)
}
// DetectHostnameChanges exports detectHostnameChanges for testing. // DetectHostnameChanges exports detectHostnameChanges for testing.
func (w *Watcher) DetectHostnameChanges( func (w *Watcher) DetectHostnameChanges(
ctx context.Context, ctx context.Context,
@@ -57,6 +72,15 @@ func (w *Watcher) ResolveNameserverAddresses(
return w.resolveNameserverAddresses(ctx, nameservers, prev) return w.resolveNameserverAddresses(ctx, nameservers, prev)
} }
// ResolveCNAMEAddresses exports resolveCNAMEAddresses for testing.
func (w *Watcher) ResolveCNAMEAddresses(
ctx context.Context,
hostname string,
current, prev *state.HostnameState,
) {
w.resolveCNAMEAddresses(ctx, hostname, current, prev)
}
// DetectNSAddressChanges exports detectNSAddressChanges for testing. // DetectNSAddressChanges exports detectNSAddressChanges for testing.
func (w *Watcher) DetectNSAddressChanges( func (w *Watcher) DetectNSAddressChanges(
ctx context.Context, ctx context.Context,
@@ -79,7 +103,8 @@ func (w *Watcher) RunTLSChecks(ctx context.Context) {
// BuildHostnameState exports buildHostnameState for testing. // BuildHostnameState exports buildHostnameState for testing.
func BuildHostnameState( func BuildHostnameState(
results map[string]*resolver.NameserverResponse, results map[string]*resolver.NameserverResponse,
prev *state.HostnameState,
now time.Time, now time.Time,
) *state.HostnameState { ) *state.HostnameState {
return buildHostnameState(results, now) return buildHostnameState(results, prev, now)
} }
+364
View File
@@ -0,0 +1,364 @@
package watcher_test
import (
"maps"
"slices"
"testing"
"time"
"sneak.berlin/go/dnswatcher/internal/resolver"
"sneak.berlin/go/dnswatcher/internal/state"
"sneak.berlin/go/dnswatcher/internal/watcher"
)
const (
// txt is the record type whose query fails in these tests.
txt = "TXT"
spf1 = "v=spf1 -all"
spf2 = "v=spf1 include:example.net -all"
)
// response is a nameserver's response with these records, whose queries
// for failedTypes failed.
func response(
records map[string][]string,
failedTypes ...string,
) *resolver.NameserverResponse {
return &resolver.NameserverResponse{
Records: records,
FailedTypes: failedTypes,
Status: resolver.StatusOK,
}
}
// savedChecks saves the state of each check in turn from the
// nameservers' responses, each from the state the check before saved.
func savedChecks(
checks ...map[string]*resolver.NameserverResponse,
) []*state.HostnameState {
states := make([]*state.HostnameState, 0, len(checks))
var prev *state.HostnameState
for _, results := range checks {
prev = watcher.BuildHostnameState(results, prev, time.Now())
states = append(states, prev)
}
return states
}
// TestFailedTypeKeepsPreviousRecords saves a check in which nsA's query
// for TXT failed, after previous checks of several kinds. TXT is always
// saved in FailedTypes, and in UnknownTypes when there was nothing to
// keep.
func TestFailedTypeKeepsPreviousRecords(t *testing.T) {
t.Parallel()
aOnly := map[string][]string{"A": {ip1}}
withTXT := map[string][]string{"A": {ip1}, txt: {spf1}}
txtKept := &state.NameserverRecordState{
Records: withTXT, FailedTypes: []string{txt}, Status: "ok",
}
txtNotKnown := &state.NameserverRecordState{
Records: aOnly,
FailedTypes: []string{txt},
UnknownTypes: []string{txt},
Status: "ok",
}
tests := []struct {
name string
prev *state.HostnameState
wantRecords map[string][]string
wantUnknown []string
}{
{
"previous TXT records are kept",
saved(map[string]*state.NameserverRecordState{nsA: answered(withTXT)}),
withTXT, nil,
},
{
"previous check had no TXT records",
saved(map[string]*state.NameserverRecordState{nsA: answered(aOnly)}),
aOnly, nil,
},
{
"TXT failed on the previous check, which kept its records",
saved(map[string]*state.NameserverRecordState{nsA: txtKept}),
withTXT, nil,
},
{"first check", nil, aOnly, []string{txt}},
{
"nameserver new on this check",
saved(map[string]*state.NameserverRecordState{nsB: answered(withTXT)}),
aOnly, []string{txt},
},
{
"nameserver failed on the previous check",
saved(map[string]*state.NameserverRecordState{nsA: failed()}),
aOnly, []string{txt},
},
{
"TXT failed on the previous check with nothing to keep",
saved(map[string]*state.NameserverRecordState{nsA: txtNotKnown}),
aOnly, []string{txt},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
hs := watcher.BuildHostnameState(
map[string]*resolver.NameserverResponse{
nsA: response(map[string][]string{"A": {ip1}}, txt),
},
tt.prev, time.Now(),
)
got := hs.RecordsByNameserver[nsA]
if got.Status != "ok" ||
!maps.EqualFunc(got.Records, tt.wantRecords, slices.Equal) ||
!slices.Equal(got.FailedTypes, []string{txt}) ||
!slices.Equal(got.UnknownTypes, tt.wantUnknown) {
t.Errorf(
"saved status %q, records %v, failed types %v, "+
"unknown types %v; want ok, %v, [%s], %v",
got.Status, got.Records, got.FailedTypes,
got.UnknownTypes, tt.wantRecords, txt, tt.wantUnknown,
)
}
})
}
}
// TestFailedTypeAlerts saves the checks of each case in turn from the
// nameservers' responses, the first being the state loaded at startup,
// and counts the alerts sent. nsB's TXT query fails on one check, and
// nothing changes.
func TestFailedTypeAlerts(t *testing.T) {
t.Parallel()
records := map[string][]string{"A": {ip1}, txt: {spf1}}
aOnly := map[string][]string{"A": {ip1}}
bothAnswer := map[string]*resolver.NameserverResponse{
nsA: response(records), nsB: response(records),
}
bTXTFails := map[string]*resolver.NameserverResponse{
nsA: response(records), nsB: response(aOnly, txt),
}
onlyA := map[string]*resolver.NameserverResponse{
nsA: response(records),
}
bFails := map[string]*resolver.NameserverResponse{
nsA: response(records),
nsB: {
Records: map[string][]string{},
Status: resolver.StatusTimeout,
Error: "all queries timed out",
},
}
tests := []struct {
name string
checks []map[string]*resolver.NameserverResponse
want alertCounts
}{
{
"type failing at one nameserver alerts nothing, nor its next answer",
[]map[string]*resolver.NameserverResponse{
bothAnswer, bTXTFails, bothAnswer,
},
alertCounts{},
},
{
"type failing on the first check alerts nothing on the next",
[]map[string]*resolver.NameserverResponse{bTXTFails, bothAnswer},
alertCounts{},
},
{
"type failing at a nameserver new on that check alerts nothing",
[]map[string]*resolver.NameserverResponse{
onlyA, bTXTFails, bothAnswer,
},
alertCounts{},
},
{
"type failing at a recovering nameserver alerts the recovery",
[]map[string]*resolver.NameserverResponse{
bFails, bTXTFails, bothAnswer,
},
alertCounts{recoveries: 1},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
states := savedChecks(tt.checks...)
got := countAlerts(t, states[0], states[1:])
if got != tt.want {
t.Errorf("sent %+v, want %+v", got, tt.want)
}
})
}
}
// TestFailedTypeComparedOnceItAnswers saves the checks of each case in
// turn as TestFailedTypeAlerts does. nsB's TXT query fails on one check,
// and the TXT record changes: the change is sent as a Record Change for
// each nameserver on the check where it answers it, and an Inconsistency
// only when nsB still answers the old record.
func TestFailedTypeComparedOnceItAnswers(t *testing.T) {
t.Parallel()
records := map[string][]string{"A": {ip1}, txt: {spf1}}
changed := map[string][]string{"A": {ip1}, txt: {spf2}}
aOnly := map[string][]string{"A": {ip1}}
bothAnswer := map[string]*resolver.NameserverResponse{
nsA: response(records), nsB: response(records),
}
bTXTFails := map[string]*resolver.NameserverResponse{
nsA: response(records), nsB: response(aOnly, txt),
}
bothChange := map[string]*resolver.NameserverResponse{
nsA: response(changed), nsB: response(changed),
}
aChangesBTXTFails := map[string]*resolver.NameserverResponse{
nsA: response(changed), nsB: response(aOnly, txt),
}
bStillOld := map[string]*resolver.NameserverResponse{
nsA: response(changed), nsB: response(records),
}
tests := []struct {
name string
checks []map[string]*resolver.NameserverResponse
want alertCounts
}{
{
"change made while the type failed",
[]map[string]*resolver.NameserverResponse{
bothAnswer, bTXTFails, bothChange,
},
alertCounts{recordChanges: 2},
},
{
"change seen at one nameserver while the other's type failed",
[]map[string]*resolver.NameserverResponse{
bothAnswer, aChangesBTXTFails, bothChange,
},
alertCounts{recordChanges: 2},
},
{
"old record answered after the type failed",
[]map[string]*resolver.NameserverResponse{
bothAnswer, aChangesBTXTFails, bStillOld,
},
alertCounts{recordChanges: 1, inconsistencies: 1},
},
{
"change after the type failed on the first check and answered",
[]map[string]*resolver.NameserverResponse{
bTXTFails, bothAnswer, bothChange,
},
alertCounts{recordChanges: 2},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
states := savedChecks(tt.checks...)
got := countAlerts(t, states[0], states[1:])
if got != tt.want {
t.Errorf("sent %+v, want %+v", got, tt.want)
}
})
}
}
// TestFailedTypeLeftOutOfMessages checks that a Record Change and an
// Inconsistency name only the record types they compared. nsB's TXT
// records are not known on the first check, and on the second either
// answered or still not known; nsB's A record changes, so both alerts
// are sent and name the A record alone.
func TestFailedTypeLeftOutOfMessages(t *testing.T) {
t.Parallel()
withTXT := map[string][]string{"A": {ip1}, txt: {spf1}}
txtNotKnown := func(address string) *state.NameserverRecordState {
return &state.NameserverRecordState{
Records: map[string][]string{"A": {address}},
FailedTypes: []string{txt},
UnknownTypes: []string{txt},
Status: "ok",
}
}
before := saved(map[string]*state.NameserverRecordState{
nsA: answered(withTXT), nsB: txtNotKnown(ip1),
})
tests := []struct {
name string
after *state.HostnameState
}{
{
"TXT answers",
saved(map[string]*state.NameserverRecordState{
nsA: answered(withTXT),
nsB: answered(map[string][]string{"A": {ip2}, txt: {spf1}}),
}),
},
{
"TXT still not known",
saved(map[string]*state.NameserverRecordState{
nsA: answered(withTXT), nsB: txtNotKnown(ip2),
}),
},
}
want := map[string]string{
"Record Change: " + host: "Hostname: " + host +
"\nNameserver: " + nsB + "\nType: A\nOld: " + ip1 + "\nNew: " + ip2,
"Inconsistency: " + host: "Hostname: " + host +
"\nType: A\n" + nsA + ": " + ip1 + "\n" + nsB + ": " + ip2,
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
// The hostname change detection uses only the notifier.
notifier := &mockNotifier{}
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
w.DetectHostnameChanges(t.Context(), host, before, tt.after)
notifications := notifier.getNotifications()
if len(notifications) != len(want) {
t.Fatalf(
"sent %d notifications, want %d: %v",
len(notifications), len(want), notifications,
)
}
for _, n := range notifications {
if n.Message != want[n.Title] {
t.Errorf(
"%s message:\n%s\nwant:\n%s",
n.Title, n.Message, want[n.Title],
)
}
}
})
}
}
+66
View File
@@ -0,0 +1,66 @@
package watcher_test
import (
"testing"
"sneak.berlin/go/dnswatcher/internal/watcher"
)
// When one nameserver's A record changes and its TXT record does not,
// the record change and the inconsistency it starts name the A record
// alone, with its values written as plain text.
func TestChangeMessagesNameTheChangedType(t *testing.T) {
t.Parallel()
// A nameserver's records: this A address and the same TXT record.
records := func(address string) map[string][]string {
return map[string][]string{
"A": {address},
"TXT": {"v=spf1 -all"},
}
}
before := hostnameState(map[string]map[string][]string{
nsA: records(ip1),
nsB: records(ip1),
})
after := hostnameState(map[string]map[string][]string{
nsA: records(ip1),
nsB: records(ip2),
})
// The hostname change detection uses only the notifier.
notifier := &mockNotifier{}
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
w.DetectHostnameChanges(t.Context(), host, before, after)
want := map[string]string{
"Record Change: " + host: `Hostname: www.example.net
Nameserver: b.ns.example.net.
Type: A
Old: 192.0.2.1
New: 192.0.2.2`,
"Inconsistency: " + host: `Hostname: www.example.net
Type: A
a.ns.example.net.: 192.0.2.1
b.ns.example.net.: 192.0.2.2`,
}
notifications := notifier.getNotifications()
if len(notifications) != len(want) {
t.Fatalf(
"sent %d notifications, want %d: %v",
len(notifications), len(want), notifications,
)
}
for _, n := range notifications {
if n.Message != want[n.Title] {
t.Errorf(
"%s message:\n%s\nwant:\n%s",
n.Title, n.Message, want[n.Title],
)
}
}
}
+3 -3
View File
@@ -201,7 +201,7 @@ func TestNameserverThatNeverAnswers(t *testing.T) {
} }
hs := watcher.BuildHostnameState( hs := watcher.BuildHostnameState(
map[string]*resolver.NameserverResponse{nsA: resp}, time.Now(), map[string]*resolver.NameserverResponse{nsA: resp}, nil, time.Now(),
) )
got := hs.RecordsByNameserver[nsA] got := hs.RecordsByNameserver[nsA]
@@ -256,7 +256,7 @@ func TestNameserverThatAnswersNXDOMAIN(t *testing.T) {
} }
hs := watcher.BuildHostnameState( hs := watcher.BuildHostnameState(
map[string]*resolver.NameserverResponse{ns: resp}, time.Now(), map[string]*resolver.NameserverResponse{ns: resp}, nil, time.Now(),
) )
got := hs.RecordsByNameserver[ns] got := hs.RecordsByNameserver[ns]
@@ -320,7 +320,7 @@ func TestNameserverThatRefuses(t *testing.T) {
} }
hs := watcher.BuildHostnameState( hs := watcher.BuildHostnameState(
map[string]*resolver.NameserverResponse{ns: resp}, time.Now(), map[string]*resolver.NameserverResponse{ns: resp}, nil, time.Now(),
) )
got := hs.RecordsByNameserver[ns] got := hs.RecordsByNameserver[ns]
+285 -45
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"fmt" "fmt"
"log/slog" "log/slog"
"maps"
"slices" "slices"
"sort" "sort"
"strings" "strings"
@@ -123,8 +124,10 @@ func (w *Watcher) Run(ctx context.Context) {
"watcher starting", "watcher starting",
"domains", len(w.config.Domains), "domains", len(w.config.Domains),
"hostnames", len(w.config.Hostnames), "hostnames", len(w.config.Hostnames),
"dnsInterval", w.config.DNSInterval, // As text: the JSON log writes a time.Duration as bare
"tlsInterval", w.config.TLSInterval, // nanoseconds.
"dnsInterval", w.config.DNSInterval.String(),
"tlsInterval", w.config.TLSInterval.String(),
) )
w.RunOnce(ctx) w.RunOnce(ctx)
@@ -252,28 +255,9 @@ func (w *Watcher) checkDomain(
LastChecked: now, LastChecked: now,
}) })
// Also look up A/AAAA records for the apex domain so that // The apex domain's records are also checked as a hostname's, so
// port and TLS checks (which read HostnameState) can find // that the port and TLS checks find its addresses.
// the domain's IP addresses. w.checkHostname(ctx, domain)
results, err := w.resolver.LookupAllRecords(ctx, domain)
if err != nil {
w.log.Error(
"failed to lookup records for domain",
"domain", domain,
"error", err,
)
return
}
newState := buildHostnameState(results, now)
prevHS, hasPrevHS := w.state.GetHostnameState(domain)
if hasPrevHS && !w.firstRun {
w.detectHostnameChanges(ctx, domain, prevHS, newState)
}
w.state.SetHostnameState(domain, newState)
} }
func (w *Watcher) detectNSChanges( func (w *Watcher) detectNSChanges(
@@ -398,9 +382,25 @@ func (w *Watcher) checkHostname(
return return
} }
newState := buildHostnameState(results, time.Now().UTC()) prev, _ := w.state.GetHostnameState(hostname)
w.updateHostnameState(
ctx, hostname, buildHostnameState(results, prev, time.Now().UTC()),
)
}
// updateHostnameState finishes a check of hostname from newState, built
// from its nameservers' answers: it follows the CNAME in them, notifies
// what changed since the previous check, and saves newState.
func (w *Watcher) updateHostnameState(
ctx context.Context,
hostname string,
newState *state.HostnameState,
) {
prev, hasPrev := w.state.GetHostnameState(hostname) prev, hasPrev := w.state.GetHostnameState(hostname)
w.resolveCNAMEAddresses(ctx, hostname, newState, prev)
if hasPrev && !w.firstRun { if hasPrev && !w.firstRun {
w.detectHostnameChanges(ctx, hostname, prev, newState) w.detectHostnameChanges(ctx, hostname, prev, newState)
} }
@@ -408,12 +408,88 @@ func (w *Watcher) checkHostname(
w.state.SetHostnameState(hostname, newState) w.state.SetHostnameState(hostname, newState)
} }
// resolveCNAMEAddresses saves in current the addresses at the end of
// hostname's CNAME chain, when the nameservers' answers in current hold
// a CNAME and no address, and an empty list otherwise. Every CNAME
// target the nameservers gave is followed with ResolveIPAddresses and
// the addresses found for all of them are saved, so nameservers that
// disagree on the target do not change the result from check to check.
// The addresses saved in prev, which may be nil, are kept when none of
// the name's nameservers answered its queries for A, AAAA and CNAME,
// and when a target cannot be followed, as when no nameserver of a zone
// in its chain answers.
func (w *Watcher) resolveCNAMEAddresses(
ctx context.Context,
hostname string,
current, prev *state.HostnameState,
) {
var prevAddresses []string
if prev != nil {
prevAddresses = prev.CNAMEAddresses
}
// Empty, not nil: nil means the addresses are not known.
current.CNAMEAddresses = []string{}
answered := false
targets := make(map[string]bool)
for _, nsState := range current.RecordsByNameserver {
if nsState.Status != statusOK ||
slices.Contains(nsState.FailedTypes, "A") ||
slices.Contains(nsState.FailedTypes, "AAAA") ||
slices.Contains(nsState.FailedTypes, "CNAME") {
continue
}
answered = true
if len(nsState.Records["A"]) > 0 || len(nsState.Records["AAAA"]) > 0 {
return
}
for _, target := range nsState.Records["CNAME"] {
targets[target] = true
}
}
if !answered {
current.CNAMEAddresses = prevAddresses
return
}
for target := range targets {
ips, err := w.resolver.ResolveIPAddresses(ctx, target)
if err != nil {
w.log.Error(
"failed to follow CNAME",
"hostname", hostname,
"target", target,
"error", err,
)
current.CNAMEAddresses = prevAddresses
return
}
current.CNAMEAddresses = append(current.CNAMEAddresses, ips...)
}
// Still the empty list when every chain ends in no address.
slices.Sort(current.CNAMEAddresses)
current.CNAMEAddresses = slices.Compact(current.CNAMEAddresses)
}
// buildHostnameState saves each nameserver's response. A nameserver // buildHostnameState saves each nameserver's response. A nameserver
// that answered, even with NXDOMAIN or no records, is saved as ok; one // that answered, even with NXDOMAIN or no records, is saved as ok, with
// that timed out or failed is saved as error with the reason, and its // the record types whose query failed; one that timed out or failed is
// empty record set is not an answer. // saved as error with the reason, and its empty record set is not an
// answer. prev is the hostname's state from the previous check, or nil.
func buildHostnameState( func buildHostnameState(
results map[string]*resolver.NameserverResponse, results map[string]*resolver.NameserverResponse,
prev *state.HostnameState,
now time.Time, now time.Time,
) *state.HostnameState { ) *state.HostnameState {
hs := &state.HostnameState{ hs := &state.HostnameState{
@@ -425,7 +501,7 @@ func buildHostnameState(
for ns, resp := range results { for ns, resp := range results {
nsState := &state.NameserverRecordState{ nsState := &state.NameserverRecordState{
Records: resp.Records, Records: maps.Clone(resp.Records),
Status: statusOK, Status: statusOK,
LastChecked: now, LastChecked: now,
} }
@@ -434,6 +510,15 @@ func buildHostnameState(
resp.Status == resolver.StatusError { resp.Status == resolver.StatusError {
nsState.Status = statusError nsState.Status = statusError
nsState.Error = resp.Error nsState.Error = resp.Error
} else {
nsState.FailedTypes = resp.FailedTypes
var prevNS *state.NameserverRecordState
if prev != nil {
prevNS = prev.RecordsByNameserver[ns]
}
keepFailedTypes(nsState, prevNS)
} }
hs.RecordsByNameserver[ns] = nsState hs.RecordsByNameserver[ns] = nsState
@@ -442,6 +527,27 @@ func buildHostnameState(
return hs return hs
} }
// keepFailedTypes copies into nsState, for each record type in its
// FailedTypes, the records prevNS, the nameserver's state from the
// previous check, holds for that type, which may be none. When prevNS
// does not know them either, because the nameserver was new or failing
// then or the type was in its UnknownTypes, the type goes in
// nsState.UnknownTypes instead.
func keepFailedTypes(nsState, prevNS *state.NameserverRecordState) {
for _, rtype := range nsState.FailedTypes {
if prevNS == nil || prevNS.Status != statusOK ||
slices.Contains(prevNS.UnknownTypes, rtype) {
nsState.UnknownTypes = append(nsState.UnknownTypes, rtype)
continue
}
if records, ok := prevNS.Records[rtype]; ok {
nsState.Records[rtype] = records
}
}
}
func (w *Watcher) detectHostnameChanges( func (w *Watcher) detectHostnameChanges(
ctx context.Context, ctx context.Context,
hostname string, hostname string,
@@ -451,11 +557,45 @@ func (w *Watcher) detectHostnameChanges(
w.detectNSDisappearances(ctx, hostname, prev, current) w.detectNSDisappearances(ctx, hostname, prev, current)
w.detectNSFailures(ctx, hostname, prev, current) w.detectNSFailures(ctx, hostname, prev, current)
w.detectInconsistencies(ctx, hostname, prev, current) w.detectInconsistencies(ctx, hostname, prev, current)
w.detectCNAMEAddressChanges(ctx, hostname, prev, current)
}
// detectCNAMEAddressChanges notifies when the addresses at the end of
// hostname's CNAME chain differ from those the previous check saved,
// including a change from or to none. When the previous addresses are
// not known (nil), as on the first check after loading a state file
// written before they were saved, nothing is compared.
func (w *Watcher) detectCNAMEAddressChanges(
ctx context.Context,
hostname string,
prev, current *state.HostnameState,
) {
old, cur := prev.CNAMEAddresses, current.CNAMEAddresses
if old == nil || sliceEqual(old, cur) {
return
}
msg := fmt.Sprintf(
"Hostname: %s\nOld: %s\nNew: %s",
hostname,
strings.Join(old, ", "),
strings.Join(cur, ", "),
)
w.notify.SendNotification(
ctx,
"CNAME Address Change: "+hostname,
msg,
"warning",
)
} }
// detectRecordChanges compares each nameserver's records with those of // detectRecordChanges compares each nameserver's records with those of
// the previous check. Only answers are compared: a nameserver that // the previous check. Only answers are compared: a nameserver that
// failed on either check has no records to compare. // failed on either check has no records to compare. The records kept
// for a record type whose query failed are compared too, but not those
// of a type in UnknownTypes on either check, which the message leaves
// out as well.
func (w *Watcher) detectRecordChanges( func (w *Watcher) detectRecordChanges(
ctx context.Context, ctx context.Context,
hostname string, hostname string,
@@ -467,15 +607,21 @@ func (w *Watcher) detectRecordChanges(
continue continue
} }
if recordsEqual(prevNS.Records, cur.Records) { unknown := slices.Concat(prevNS.UnknownTypes, cur.UnknownTypes)
oldRecords := withoutTypes(prevNS.Records, unknown)
newRecords := withoutTypes(cur.Records, unknown)
if recordsEqual(oldRecords, newRecords) {
continue continue
} }
msg := fmt.Sprintf( msg := fmt.Sprintf(
"Hostname: %s\nNameserver: %s\n"+ "Hostname: %s\nNameserver: %s\n%s",
"Old: %v\nNew: %v",
hostname, ns, hostname, ns,
prevNS.Records, cur.Records, recordDifferences(
"Old", oldRecords,
"New", newRecords,
),
) )
w.notify.SendNotification( w.notify.SendNotification(
@@ -561,12 +707,20 @@ func (w *Watcher) detectInconsistencies(
) { ) {
for _, pair := range newlyDisagreeingPairs(prev, current) { for _, pair := range newlyDisagreeingPairs(prev, current) {
ns1, ns2 := pair[0], pair[1] ns1, ns2 := pair[0], pair[1]
state1 := current.RecordsByNameserver[ns1]
state2 := current.RecordsByNameserver[ns2]
// The record types left out of the comparison are left out of
// the message too.
failed := slices.Concat(state1.FailedTypes, state2.FailedTypes)
msg := fmt.Sprintf( msg := fmt.Sprintf(
"Hostname: %s\n%s: %v\n%s: %v", "Hostname: %s\n%s",
hostname, hostname,
ns1, current.RecordsByNameserver[ns1].Records, recordDifferences(
ns2, current.RecordsByNameserver[ns2].Records, ns1, withoutTypes(state1.Records, failed),
ns2, withoutTypes(state2.Records, failed),
),
) )
w.notify.SendNotification( w.notify.SendNotification(
@@ -583,7 +737,9 @@ func (w *Watcher) detectInconsistencies(
// except pairs where both nameservers answered in prev and already // except pairs where both nameservers answered in prev and already
// differed there. A nameserver missing from prev, or that failed there, // differed there. A nameserver missing from prev, or that failed there,
// is paired with every nameserver it differs from. A nameserver that // is paired with every nameserver it differs from. A nameserver that
// failed in current has no records to compare and is in no pair. // failed in current has no records to compare and is in no pair. In
// both checks, a record type whose query failed at either nameserver is
// not compared.
func newlyDisagreeingPairs( func newlyDisagreeingPairs(
prev, current *state.HostnameState, prev, current *state.HostnameState,
) [][2]string { ) [][2]string {
@@ -600,9 +756,9 @@ func newlyDisagreeingPairs(
for i, ns1 := range nameservers { for i, ns1 := range nameservers {
for _, ns2 := range nameservers[i+1:] { for _, ns2 := range nameservers[i+1:] {
if recordsEqual( if nameserversAgree(
current.RecordsByNameserver[ns1].Records, current.RecordsByNameserver[ns1],
current.RecordsByNameserver[ns2].Records, current.RecordsByNameserver[ns2],
) { ) {
continue continue
} }
@@ -612,7 +768,7 @@ func newlyDisagreeingPairs(
if ok1 && ok2 && if ok1 && ok2 &&
prev1.Status == statusOK && prev2.Status == statusOK && prev1.Status == statusOK && prev2.Status == statusOK &&
!recordsEqual(prev1.Records, prev2.Records) { !nameserversAgree(prev1, prev2) {
continue continue
} }
@@ -747,6 +903,9 @@ func (w *Watcher) noNameserverAnswered(name string) bool {
return true return true
} }
// collectIPs returns the addresses saved for hostname: those in its
// nameservers' A and AAAA records, and those at the end of its CNAME
// chain.
func (w *Watcher) collectIPs(hostname string) []string { func (w *Watcher) collectIPs(hostname string) []string {
hs, ok := w.state.GetHostnameState(hostname) hs, ok := w.state.GetHostnameState(hostname)
if !ok { if !ok {
@@ -765,6 +924,10 @@ func (w *Watcher) collectIPs(hostname string) []string {
} }
} }
for _, ip := range hs.CNAMEAddresses {
ipSet[ip] = true
}
result := make([]string, 0, len(ipSet)) result := make([]string, 0, len(ipSet))
for ip := range ipSet { for ip := range ipSet {
result = append(result, ip) result = append(result, ip)
@@ -1049,8 +1212,9 @@ func (w *Watcher) saveState() {
// maybeSendTestNotification sends a startup status notification // maybeSendTestNotification sends a startup status notification
// after the first full scan completes, if SEND_TEST_NOTIFICATION // after the first full scan completes, if SEND_TEST_NOTIFICATION
// is enabled. The message is clearly informational ("all ok") // is enabled. The message is informational, not an error or anomaly
// and not an error or anomaly alert. // alert. It is written before it reaches any endpoint, so it claims
// nothing about whether the endpoints work.
func (w *Watcher) maybeSendTestNotification(ctx context.Context) { func (w *Watcher) maybeSendTestNotification(ctx context.Context) {
if !w.config.SendTestNotification { if !w.config.SendTestNotification {
return return
@@ -1062,7 +1226,8 @@ func (w *Watcher) maybeSendTestNotification(ctx context.Context) {
"dnswatcher has started and completed its initial scan.\n"+ "dnswatcher has started and completed its initial scan.\n"+
"Monitoring %d domain(s) and %d hostname(s).\n"+ "Monitoring %d domain(s) and %d hostname(s).\n"+
"Tracking %d port endpoint(s) and %d TLS certificate(s).\n"+ "Tracking %d port endpoint(s) and %d TLS certificate(s).\n"+
"All notification channels are working.", "This is a test notification, sent to every configured "+
"notification endpoint.",
len(snap.Domains), len(snap.Domains),
len(snap.Hostnames), len(snap.Hostnames),
len(snap.Ports), len(snap.Ports),
@@ -1090,6 +1255,33 @@ func toSet(items []string) map[string]bool {
return set return set
} }
// nameserversAgree reports whether two nameservers' states from the same
// check hold the same records, leaving out the record types either lists
// in FailedTypes: the records held for those are kept from an earlier
// check, or not known.
func nameserversAgree(a, b *state.NameserverRecordState) bool {
failed := slices.Concat(a.FailedTypes, b.FailedTypes)
return recordsEqual(
withoutTypes(a.Records, failed), withoutTypes(b.Records, failed),
)
}
// withoutTypes returns a copy of records without the record types in
// types.
func withoutTypes(
records map[string][]string,
types []string,
) map[string][]string {
records = maps.Clone(records)
for _, rtype := range types {
delete(records, rtype)
}
return records
}
func recordsEqual( func recordsEqual(
a, b map[string][]string, a, b map[string][]string,
) bool { ) bool {
@@ -1107,6 +1299,54 @@ func recordsEqual(
return true return true
} }
// recordDifferences describes, in sorted order of type, each record
// type whose values differ between a and b: a line naming the type,
// then a line with a's values after labelA and one with b's after
// labelB. Types with the same values in both are left out.
func recordDifferences(
labelA string, a map[string][]string,
labelB string, b map[string][]string,
) string {
types := make([]string, 0, len(a)+len(b))
for recordType := range a {
types = append(types, recordType)
}
for recordType := range b {
if _, ok := a[recordType]; !ok {
types = append(types, recordType)
}
}
sort.Strings(types)
var lines []string
for _, recordType := range types {
if sliceEqual(a[recordType], b[recordType]) {
continue
}
lines = append(lines,
"Type: "+recordType,
labelA+": "+joinValues(a[recordType]),
labelB+": "+joinValues(b[recordType]),
)
}
return strings.Join(lines, "\n")
}
// joinValues lists record values separated by commas, or says none.
func joinValues(values []string) string {
if len(values) == 0 {
return "none"
}
return strings.Join(values, ", ")
}
func sliceEqual(a, b []string) bool { func sliceEqual(a, b []string) bool {
if len(a) != len(b) { if len(a) != len(b) {
return false return false
+22 -4
View File
@@ -312,7 +312,8 @@ func lookupNameservers(t *testing.T, name string) []string {
return nameservers return nameservers
} }
// addresses returns the A and AAAA values saved for a hostname. // addresses returns the A and AAAA values saved for a hostname, and the
// addresses saved at the end of its CNAME chain.
func addresses(hs *state.HostnameState) []string { func addresses(hs *state.HostnameState) []string {
var ips []string var ips []string
@@ -321,7 +322,7 @@ func addresses(hs *state.HostnameState) []string {
ips = append(ips, nsState.Records["AAAA"]...) ips = append(ips, nsState.Records["AAAA"]...)
} }
return ips return append(ips, hs.CNAMEAddresses...)
} }
// assertNotified checks that a notification with this title and // assertNotified checks that a notification with this title and
@@ -371,6 +372,14 @@ func TestFirstRunBaseline(t *testing.T) {
assertNoNotifications(t, deps) assertNoNotifications(t, deps)
assertStatePopulated(t, deps) assertStatePopulated(t, deps)
// testHost answers with an address, so the check saves an empty list
// of CNAME addresses for it; nil would mean the check did not look
// at whether to follow a CNAME.
hs, _ := deps.state.GetHostnameState(testHost)
if hs.CNAMEAddresses == nil || len(hs.CNAMEAddresses) != 0 {
t.Errorf("saved CNAME addresses %#v, want []", hs.CNAMEAddresses)
}
} }
func assertNoNotifications( func assertNoNotifications(
@@ -866,18 +875,27 @@ func TestSendTestNotification_ViaRun(t *testing.T) {
notifications := deps.notifier.getNotifications() notifications := deps.notifier.getNotifications()
// No names are configured, so every count is 0.
wantMessage := "dnswatcher has started and completed its initial scan.\n" +
"Monitoring 0 domain(s) and 0 hostname(s).\n" +
"Tracking 0 port endpoint(s) and 0 TLS certificate(s).\n" +
"This is a test notification, sent to every configured " +
"notification endpoint."
found := false found := false
for _, n := range notifications { for _, n := range notifications {
if n.Priority == "success" && if n.Priority == "success" &&
n.Title == "✅ dnswatcher startup complete" { n.Title == "✅ dnswatcher startup complete" &&
n.Message == wantMessage {
found = true found = true
} }
} }
if !found { if !found {
t.Errorf( t.Errorf(
"expected startup test notification, got: %v", "expected startup test notification with message %q, got: %v",
wantMessage,
notifications, notifications,
) )
} }