9 Commits
Author SHA1 Message Date
clawbot 661ef8d937 resolver: ask a referral's nameservers that come without addresses (closes #221)
check / check (push) Canceled after 0s
Looking up a nameserver's own address followed only the addresses a
referral gave, so a nameserver whose zone is delegated without them,
such as a.ntpns.org of pool.ntp.org, never resolved. The walk to a
name's nameservers looked addresses up only when a referral gave none.
Both now ask the nameservers whose addresses the referral gives first
and, if none of them gives a usable reply, look up and ask the others;
with no addresses given, all are looked up, as before. maxLookupDepth
stops lookups three deep, so delegations that point at each other
still end. g.ntpns.org's address needs all three when anyns.pch.net
gives the referral to g.ntpns.org without addresses.

Model: opus-5-5
2026-10-02 07:16:14 +00:00
clawbot 250f3dd687 dashboard and status API show why a check failed (closes #225)
check / check (push) Canceled after 0s
/api/v1/status now gives `error` for each nameserver entry and
certificate entry whose status is `error`, copied from the state, which
already kept it. The dashboard shows that reason in place of the records
for a failed nameserver, which used to show the same `-` as one that
answered with no records, and across the CN, issuer and expiry cells
for a failed certificate, wrapped at a width of 20rem so the long TLS
error does not narrow the Endpoint column. The dashboard stylesheet is a
trimmed build, so the new markup uses only classes the page already had.
README Web Dashboard and HTTP API say so.

Model: opus-5-5
2026-10-02 09:14:28 +02:00
clawbot c07976a73a resolver: store a name's CNAME once per nameserver (closes #220)
check / check (push) Canceled after 0s
For a name with a CNAME, a nameserver answers a query of any type with
that CNAME, and the records of every answer were added, so the CNAME
was stored once for each of the eight record types asked for.
collectAnswerRecords now adds each value once per record type.

A state file saved before this holds the repeated values. Load keeps
each record value once, so the first check after upgrading sees no
record change and notifies nothing for them.

Model: opus-5-5
2026-10-02 09:09:51 +02:00
clawbot b047c3c64c watcher: a lookup cut short by shutdown is not logged as an error (closes #229)
check / check (push) Canceled after 0s
Stopping dnswatcher during a DNS check logged every lookup the stop cut
short as an error, "context canceled". The four places in the watcher
that log a failed lookup, checkDomain, checkHostname,
resolveNameserverAddresses and resolveCNAMEAddresses, now do it through
logFailedLookup, which logs nothing when the watcher's context was
cancelled. It asks the context, not the lookup's error: the resolver
reports a cancelled lookup with its own error, which does not wrap
context.Canceled. A context whose deadline passed is not cancelled, so a
lookup that ran out of time is still logged. The tests run each of these
lookups on a cancelled context and on one whose deadline passed; neither
sends a query.

Model: opus-5-5
2026-10-02 08:56:52 +02:00
clawbot f99de191c0 watcher: change messages name only the record types that differ (closes #219)
check / check (push) Canceled after 0s
A Record Change notification printed the nameserver's whole old and new
record sets in Go map syntax, and an Inconsistency notification the two
nameservers' whole sets, so a one-address change had to be found by eye
among kilobytes of unchanged TXT, CAA, MX and NS values. Both now list,
in sorted order of type, only the record types whose values differ: a
line naming the type, then each side's values separated by commas, or
none when that side has no records of that type. The dashboard's Recent
alerts shows the same text.

Model: opus-5-5
2026-10-02 08:46:56 +02:00
clawbot 5db5048754 watcher: startup notification no longer says every endpoint works (closes #230)
check / check (push) Canceled after 0s
The startup notification ended "All notification channels are working.",
but it is written once and handed to every notification endpoint before
any delivery has succeeded or failed, so the claim was never checked and
was false whenever one endpoint refused it. It now says only that it is a
test sent to every configured notification endpoint. The startup
notification test checks the whole message.

Model: opus-5-5
2026-10-02 08:42:35 +02:00
clawbot 26c9c74d8e notify: a failed Mattermost delivery's error names Mattermost (closes #227)
check / check (push) Canceled after 0s
Mattermost is sent by the Slack sender, which wrapped every HTTP error
status in ErrSlackFailed, so a Mattermost endpoint answering 503 was
logged as "slack notification failed". The sender now takes the error
to wrap: the Slack endpoint passes ErrSlackFailed and the Mattermost
endpoint passes ErrMattermostFailed, which was defined but unused.

A new delivery test sets both endpoints to a stand-in server answering
503 and checks the error logged for each names its own endpoint.

Model: opus-5-5
2026-10-02 08:40:19 +02:00
clawbot ceb24c5004 log: write durations as text, not nanoseconds (closes #228)
check / check (push) Canceled after 0s
The JSON log wrote a Go duration as a bare count of nanoseconds, so
the watcher starting line showed dnsInterval 120000000000 for 2m and a
delivery retry showed retryIn 1015437050. Each duration logged is now
passed through its String() form: dnsInterval and tlsInterval when the
watcher starts, retryIn on a delivery retry, and latency on a
succeeded port check. A test checks that retryIn is logged as the
text of the wait the retry actually took.

The request log's latency_ms is left as it is: its key names its
unit.

Model: opus-5-5
2026-10-02 08:37:47 +02:00
clawbot ee4cadbd05 watcher: follow a watched name's CNAME for port and TLS checks (closes #203)
check / check (push) Canceled after 0s
When a watched name's nameservers answer with a CNAME and no address,
the DNS check follows every target they gave with ResolveIPAddresses
and saves all addresses found as cnameAddresses in the hostname state,
so nameservers disagreeing on the target do not change them between
checks. Port and TLS checks use them. A change, also from or to none,
is notified as a CNAME address change; the first check from a state
file without them sends none. When a target cannot be followed, or
none of the name's nameservers answered, the last check's addresses
are kept. The domain check now runs the hostname check for the apex
instead of a copy of it.

Model: opus-5-5
2026-10-02 08:26:27 +02:00
25 changed files with 1451 additions and 80 deletions
+44 -11
View File
@@ -121,14 +121,26 @@ notification endpoint set, changes show only on the dashboard; see
failed on it, and answers differently is reported on the check where it
answers. If a pair agrees again and later disagrees, the alert is sent
again.
- **CNAME address change**: The addresses at the end of a name's CNAME chain
differ from those of the previous check. They are found when its
nameservers answer with a CNAME and no address; a name that answers with
an address has none. A change from or to no addresses is sent too, as when
a name moves between A records and a CNAME. Nothing is sent when the
previous addresses were kept because a chain could not be followed or none
of the name's nameservers answered. The first check after loading a state
file without `cnameAddresses` sends nothing: it saves the addresses it
finds for the next check to compare.
### TCP Port Monitoring
- For every configured domain and hostname, constructs a deduplicated list of
the IPv4 and IPv6 addresses in the A and AAAA records its authoritative
nameservers returned. A CNAME is not followed: a name whose CNAME points into
another zone usually has no addresses here, so its ports and certificate are
not checked.
nameservers returned. When they returned a CNAME and no address, the CNAME
chain is followed and the addresses at its end are used, and a change in those
is notified as a CNAME address change. When the nameservers gave different
CNAME targets, each is followed and the addresses of all are used. When a
chain cannot be followed, or none of the name's nameservers answered, the
addresses the last check found at its end are used.
- Checks TCP connectivity on ports **80** and **443** for each IP address.
- Every **1 hour** by default, re-checks all ports.
- Any change in port availability triggers a notification:
@@ -176,6 +188,8 @@ includes:
- **DNS NS changes**: Which domain, which nameservers were added/removed.
- **NS address changes**: Which domain, which nameserver, its old and new
addresses.
- **CNAME address changes**: Which hostname, the old and new addresses at the
end of its CNAME chain.
- **NS query failures**: Which nameserver failed, error type (timeout, SERVFAIL,
REFUSED, network error), which hostname/domain affected.
- **NS recoveries**: Which nameserver recovered, which hostname/domain.
@@ -216,9 +230,11 @@ dnswatcher includes an unauthenticated, read-only web dashboard at the root URL
- **Summary counts** for monitored domains, hostnames, ports, and certificates.
- **Domains** with their discovered nameservers.
- **Hostnames** with per-nameserver DNS records and status.
- **Hostnames** with per-nameserver DNS records and status. For a nameserver
whose query failed, the reason is shown in place of the records.
- **Ports** with open/closed state and associated hostnames.
- **TLS certificates** with CN, issuer, expiry, and status.
- **TLS certificates** with CN, issuer, expiry, and status. For a failed check,
the reason is shown in place of CN, issuer and expiry.
- **Recent alerts** (last 100 notifications sent since the process started),
displayed in reverse chronological order.
@@ -245,6 +261,10 @@ dnswatcher exposes a lightweight HTTP API for operational visibility:
| `GET /api/v1/status` | Current monitoring state |
| `GET /metrics` | Prometheus metrics, see below |
In `/api/v1/status`, each nameserver entry and certificate entry whose `status`
is `error` also has `error`, the reason, as in the state file (see State File
Format).
`/metrics` is served only when `DNSWATCHER_METRICS_USERNAME` is set, behind
Basic Auth. It has the Prometheus Go client's default metrics only (Go runtime,
process, and counts of `/metrics` requests); dnswatcher records no metrics of
@@ -416,8 +436,8 @@ all of them are looked up, so that each can be asked. When it gives addresses
for only some of them, those are asked first, and the others are looked up and
asked only if none of those gives a usable reply. Both hold in the walk to a
name's nameservers and in the lookup of a nameserver's own address. Such a
lookup can need others in turn; lookups go at most two deep, one inside another,
so delegations that point at each other still end.
lookup can need others in turn; lookups go at most three deep, one inside
another, so delegations that point at each other still end.
This approach ensures:
@@ -425,9 +445,11 @@ This approach ensures:
- Ability to detect split-horizon or inconsistent responses across authoritative
servers.
CNAME chains are followed (with a depth limit to prevent loops) only to find the
addresses of nameservers. A watched name's records are stored as its nameservers
return them, CNAME included, without following it.
A watched name's records are stored as its nameservers return them, CNAME
included. When they return a CNAME and no address, the chain of every CNAME
target they gave is followed (with a depth limit to prevent loops) to the A and
AAAA records at its end, and the port and TLS checks use those addresses.
Nameservers' addresses are also found by following CNAME chains.
Sending a notification or a Sentry report is the one use of the system's
resolver: the HTTP client looks up the webhook's or Sentry's host name with it.
@@ -474,6 +496,7 @@ merged view, to enable inconsistency detection.
"lastChecked": "2026-02-19T12:00:00Z"
}
},
"cnameAddresses": [],
"lastChecked": "2026-02-19T12:00:00Z"
}
},
@@ -520,6 +543,14 @@ certificate entry whose TLS connection or handshake failed likewise has status
resolves to. A state file without it loads, and the next check fills it in
without a notification.
`cnameAddresses` lists the sorted addresses at the end of the chain of every
CNAME target a hostname's nameservers gave, found when they answered with a
CNAME and no address; it is empty when they answered with an address. When a
chain cannot be followed, or none of the name's nameservers answered, the
previous check's list is kept, or `null` when no earlier check saved one. A
state file without it loads, and the first check after that saves it without a
notification.
A port entry in the older format, with one `hostname` instead of the `hostnames`
list, loads as a list of that one name.
@@ -677,7 +708,9 @@ docker run -d \
- Port and TLS checks use the IP addresses found by the DNS phase that
immediately precedes them. When that phase cannot find a name's
nameservers at all, the addresses an earlier check saved for the name are
used.
used. When it cannot follow a name's CNAME chain, or none of the name's
nameservers answered, the addresses an earlier check found at the end of
the chain are used.
4. **On change detection**: Send notifications to all configured endpoints,
update in-memory state, persist to disk.
5. **Shutdown**: The watcher stops checking and saves the final state to disk,
+18 -2
View File
@@ -19,8 +19,24 @@ trial run of the finished image: https://git.eeqj.de/sneak/dnswatcher/issues/149
# Completed Steps
- 2026-10-02: nameservers a referral gives no address for are looked up when
needed, two deep at most; `pool.ntp.org`'s nameservers resolve (closes #221).
- 2026-10-02: nameservers a referral names without addresses are looked up,
three deep at most; `pool.ntp.org`'s nameservers resolve (closes #221).
- 2026-10-02: the dashboard and `/api/v1/status` show why a nameserver query or
a certificate check failed, which only the state file showed (closes #225).
- 2026-10-02: a name's CNAME is stored once per nameserver, not once per record
type asked for; a state file with repeats loads each value once (closes #220).
- 2026-10-02: a DNS lookup that shutdown cuts short logs no error; one that
fails otherwise, or runs out of time, still does (closes #229).
- 2026-10-02: Record Change and Inconsistency notifications list only the record
types that differ, each with its values as plain text (closes #219).
- 2026-10-02: the startup notification no longer says every notification
endpoint works; it says it is a test sent to each of them (closes #230).
- 2026-10-02: a Mattermost webhook that answers an HTTP error is logged as
`mattermost notification failed`, not as a Slack failure (closes #227).
- 2026-10-02: durations in the log are written as text such as `2m0s`, not as a
bare count of nanoseconds (closes #228).
- 2026-10-02: a watched name whose nameservers answer with a CNAME and no
address gets port and TLS checks at the end of its CNAME chain (closes #203).
- 2026-10-02: a resolver test that reads one record type from a nameserver's
answer asks again when that type is missing from it (closes #218).
- 2026-10-02: a plain `docker build .` of a clone stamps its tag or short
+37
View File
@@ -1,6 +1,7 @@
package handlers_test
import (
"strings"
"testing"
"time"
@@ -78,3 +79,39 @@ func TestFormatRecords(t *testing.T) {
t.Errorf("unexpected format: %q", got)
}
}
// dashboardRow returns the table row of page that contains name.
func dashboardRow(t *testing.T, page string, name string) string {
t.Helper()
for row := range strings.SplitSeq(page, "<tr") {
if strings.Contains(row, name) {
return row
}
}
t.Fatalf("dashboard has no row containing %q", name)
return ""
}
// TestDashboardShowsFailureReasons checks that the dashboard shows the
// reason in the row of a failed nameserver and of a failed certificate,
// and not in the row of a nameserver that answered.
func TestDashboardShowsFailureReasons(t *testing.T) {
t.Parallel()
page := get(t, newHandlersWithFailures(t).HandleDashboard())
if !strings.Contains(dashboardRow(t, page, failedNS), nsFailureReason) {
t.Errorf("row of %s does not show %q", failedNS, nsFailureReason)
}
if strings.Contains(dashboardRow(t, page, answeringNS), nsFailureReason) {
t.Errorf("row of %s shows %q", answeringNS, nsFailureReason)
}
if !strings.Contains(dashboardRow(t, page, certKey), certFailedReason) {
t.Errorf("row of %s does not show %q", certKey, certFailedReason)
}
}
+4
View File
@@ -18,6 +18,7 @@ type statusDomainInfo struct {
type statusHostnameNSInfo struct {
Records map[string][]string `json:"records"`
Status string `json:"status"`
Error string `json:"error,omitempty"`
LastChecked time.Time `json:"lastChecked"`
}
@@ -41,6 +42,7 @@ type statusCertificateInfo struct {
NotAfter time.Time `json:"notAfter"`
SubjectAlternativeNames []string `json:"subjectAlternativeNames"`
Status string `json:"status"`
Error string `json:"error,omitempty"`
LastChecked time.Time `json:"lastChecked"`
}
@@ -144,6 +146,7 @@ func buildHostnames(
info.Nameservers[ns] = &statusHostnameNSInfo{
Records: recs,
Status: nsState.Status,
Error: nsState.Error,
LastChecked: nsState.LastChecked,
}
}
@@ -183,6 +186,7 @@ func buildCertificates(
NotAfter: cs.NotAfter,
SubjectAlternativeNames: sans,
Status: cs.Status,
Error: cs.Error,
LastChecked: cs.LastChecked,
}
}
+158
View File
@@ -0,0 +1,158 @@
package handlers_test
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"go.uber.org/fx/fxtest"
"sneak.berlin/go/dnswatcher/internal/config"
"sneak.berlin/go/dnswatcher/internal/globals"
"sneak.berlin/go/dnswatcher/internal/handlers"
"sneak.berlin/go/dnswatcher/internal/logger"
"sneak.berlin/go/dnswatcher/internal/notify"
"sneak.berlin/go/dnswatcher/internal/state"
)
// The state the handler tests serve: www.example.com has one nameserver
// that answered and one whose query failed, and its certificate check
// failed.
const (
testHostname = "www.example.com"
answeringNS = "ns1.example.com."
failedNS = "ns2.example.com."
nsFailureReason = "server returned a referral"
certKey = "192.0.2.1:443:www.example.com"
certFailedReason = "x509: certificate has expired or is not yet valid"
)
// newHandlersWithFailures builds real Handlers whose state holds the
// entries described above.
func newHandlersWithFailures(t *testing.T) *handlers.Handlers {
t.Helper()
glob, err := globals.New(nil)
if err != nil {
t.Fatalf("globals.New: %v", err)
}
log, err := logger.New(nil, logger.Params{Globals: glob})
if err != nil {
t.Fatalf("logger.New: %v", err)
}
notifier, err := notify.New(fxtest.NewLifecycle(t), notify.Params{
Logger: log,
Config: &config.Config{},
})
if err != nil {
t.Fatalf("notify.New: %v", err)
}
st, err := state.New(fxtest.NewLifecycle(t), state.Params{
Logger: log,
Config: &config.Config{DataDir: t.TempDir()},
})
if err != nil {
t.Fatalf("state.New: %v", err)
}
now := time.Now()
st.SetHostnameState(testHostname, &state.HostnameState{
RecordsByNameserver: map[string]*state.NameserverRecordState{
answeringNS: {
Records: map[string][]string{"A": {"192.0.2.1"}},
Status: "ok",
LastChecked: now,
},
failedNS: {
Records: map[string][]string{},
Status: "error",
Error: nsFailureReason,
LastChecked: now,
},
},
LastChecked: now,
})
st.SetCertificateState(certKey, &state.CertificateState{
Status: "error",
Error: certFailedReason,
LastChecked: now,
})
hnd, err := handlers.New(nil, handlers.Params{
Logger: log,
Globals: glob,
State: st,
Notify: notifier,
})
if err != nil {
t.Fatalf("handlers.New: %v", err)
}
return hnd
}
// get serves one GET request to handler and returns the response body.
func get(t *testing.T, handler http.HandlerFunc) string {
t.Helper()
rec := httptest.NewRecorder()
req := httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/", nil,
)
handler(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rec.Code)
}
return rec.Body.String()
}
// TestStatusGivesFailureReasons checks that /api/v1/status gives the
// reason for a failed nameserver entry and a failed certificate entry,
// and no error for a nameserver that answered.
func TestStatusGivesFailureReasons(t *testing.T) {
t.Parallel()
body := get(t, newHandlersWithFailures(t).HandleStatus())
var resp struct {
Hostnames map[string]struct {
Nameservers map[string]map[string]any `json:"nameservers"`
} `json:"hostnames"`
Certificates map[string]map[string]any `json:"certificates"`
}
err := json.Unmarshal([]byte(body), &resp)
if err != nil {
t.Fatalf("decoding response: %v", err)
}
nameservers := resp.Hostnames[testHostname].Nameservers
got := nameservers[failedNS]["error"]
if got != nsFailureReason {
t.Errorf("failed nameserver error = %v, want %q",
got, nsFailureReason)
}
_, has := nameservers[answeringNS]["error"]
if has {
t.Errorf("answering nameserver has an error field: %v",
nameservers[answeringNS])
}
got = resp.Certificates[certKey]["error"]
if got != certFailedReason {
t.Errorf("failed certificate error = %v, want %q",
got, certFailedReason)
}
}
@@ -146,7 +146,11 @@
<td
class="py-2 px-3 text-slate-400 break-all max-w-xs"
>
{{ if $nsr.Error }}
<span class="text-red-400">{{ $nsr.Error }}</span>
{{ else }}
{{ formatRecords $nsr.Records }}
{{ end }}
</td>
<td class="py-2 px-3 text-slate-500 whitespace-nowrap">
{{ relTime $nsr.LastChecked }}
@@ -258,6 +262,11 @@
>
{{ end }}
</td>
{{ if $cs.Error }}
<td colspan="3" class="py-2 px-3 text-red-400 break-all">
<div class="max-w-xs">{{ $cs.Error }}</div>
</td>
{{ else }}
<td class="py-2 px-3 text-slate-200">
{{ $cs.CommonName }}
</td>
@@ -285,6 +294,7 @@
{{ end }}
{{ end }}
</td>
{{ end }}
<td class="py-2 px-3 text-slate-500 whitespace-nowrap">
{{ relTime $cs.LastChecked }}
</td>
+69 -3
View File
@@ -6,9 +6,11 @@ import (
"encoding/json"
"errors"
"io"
"maps"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync"
"testing"
"time"
@@ -413,7 +415,8 @@ func sendSlackInfo(
svc *notify.Service, target *url.URL,
) error {
return svc.SendSlack(
context.Background(), target, "t", "m", prioInfo,
context.Background(), target, notify.ErrSlackFailed,
"t", "m", prioInfo,
)
}
@@ -506,6 +509,7 @@ func TestSendSlackPayloadFields(t *testing.T) {
err := svc.SendSlack(
context.Background(),
webhookURL,
notify.ErrSlackFailed,
"Alert Title",
"Alert body text",
"warning",
@@ -608,7 +612,8 @@ func TestSendSlackAllColors(t *testing.T) {
err := svc.SendSlack(
context.Background(),
webhookURL, "t", "m", tc.priority,
webhookURL, notify.ErrSlackFailed,
"t", "m", tc.priority,
)
if err != nil {
t.Fatalf("SendSlack error: %v", err)
@@ -659,7 +664,8 @@ func TestSendSlackNetworkError(t *testing.T) {
)
err := svc.SendSlack(
context.Background(), webhookURL, "t", "m", "info",
context.Background(), webhookURL, notify.ErrSlackFailed,
"t", "m", "info",
)
if err == nil {
t.Fatal("expected error for network failure")
@@ -1028,6 +1034,66 @@ func TestSendNotificationMattermostError(t *testing.T) {
)
}
// TestSendNotificationErrorNamesEndpoint verifies that, with both
// Slack and Mattermost set, a failed delivery's logged error names
// the endpoint that failed. Both are sent by the Slack sender.
func TestSendNotificationErrorNamesEndpoint(t *testing.T) {
t.Parallel()
srv := httptest.NewServer(
http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusServiceUnavailable)
}),
)
defer srv.Close()
target, _ := url.Parse(srv.URL)
svc, logs := newLoggingService(http.DefaultTransport)
svc.SetSlackWebhookURL(target)
svc.SetMattermostWebhookURL(target)
svc.SetSleepFunc(instantSleep)
svc.SetRetryConfig(notify.RetryConfig{
MaxRetries: 1,
BaseDelay: time.Millisecond,
MaxDelay: time.Millisecond,
})
svc.SendNotification(
context.Background(), "t", "m", prioError,
)
waitForCondition(t, func() bool {
return svc.OutstandingDeliveries() == 0
})
got := map[string]string{}
for line := range strings.Lines(logs.String()) {
var record struct {
Msg string `json:"msg"`
Endpoint string `json:"endpoint"`
Error string `json:"error"`
}
_ = json.Unmarshal([]byte(line), &record)
if record.Msg == "failed to send notification after retries" {
got[record.Endpoint] = record.Error
}
}
want := map[string]string{
"slack": "slack notification failed: status 503",
"mattermost": "mattermost notification failed: status 503",
}
if !maps.Equal(got, want) {
t.Errorf("logged errors = %v, want %v", got, want)
}
}
// ── SlackPayload JSON marshaling ──────────────────────────
func TestSlackPayloadJSON(t *testing.T) {
+2 -1
View File
@@ -85,10 +85,11 @@ func (svc *Service) SendNtfy(
func (svc *Service) SendSlack(
ctx context.Context,
webhookURL *url.URL,
failed error,
title, message, priority string,
) error {
return svc.sendSlack(
ctx, webhookURL, title, message, priority,
ctx, webhookURL, failed, title, message, priority,
)
}
+9 -3
View File
@@ -277,7 +277,8 @@ func (svc *Service) dispatchSlack(
svc.dispatch(ctx, "slack", func(c context.Context) error {
return svc.sendSlack(
c, svc.slackWebhookURL, title, message, priority,
c, svc.slackWebhookURL, ErrSlackFailed,
title, message, priority,
)
})
}
@@ -294,7 +295,7 @@ func (svc *Service) dispatchMattermost(
ctx, "mattermost",
func(c context.Context) error {
return svc.sendSlack(
c, svc.mattermostWebhookURL,
c, svc.mattermostWebhookURL, ErrMattermostFailed,
title, message, priority,
)
},
@@ -370,9 +371,14 @@ type SlackAttachment struct {
Text string `json:"text"`
}
// sendSlack posts to a Slack or Mattermost incoming webhook, which
// take the same payload. An HTTP error status is returned wrapped in
// failed, ErrSlackFailed or ErrMattermostFailed, so the error names
// the endpoint.
func (svc *Service) sendSlack(
ctx context.Context,
webhookURL *url.URL,
failed error,
title, message, priority string,
) error {
ctx, cancel := context.WithTimeout(
@@ -420,7 +426,7 @@ func (svc *Service) sendSlack(
if resp.StatusCode >= httpStatusClientError {
return fmt.Errorf(
"%w: status %d",
ErrSlackFailed, resp.StatusCode,
failed, resp.StatusCode,
)
}
+3 -1
View File
@@ -115,7 +115,9 @@ func (svc *Service) deliverWithRetry(
"endpoint", endpoint,
"attempt", attempt+1,
"maxAttempts", cfg.MaxRetries+1,
"retryIn", delay,
// As text: the JSON log writes a time.Duration as
// bare nanoseconds.
"retryIn", delay.String(),
"error", lastErr,
)
+45
View File
@@ -2,6 +2,7 @@ package notify_test
import (
"context"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
@@ -189,6 +190,50 @@ func TestDeliverWithRetryExhaustsAttempts(t *testing.T) {
}
}
// TestDeliverWithRetryLogsRetryInAsText checks that the wait
// before a retry is logged as text such as "1.02s", not as a
// count of nanoseconds.
func TestDeliverWithRetryLogsRetryInAsText(t *testing.T) {
t.Parallel()
svc, logs := newLoggingService(http.DefaultTransport)
svc.SetRetryConfig(notify.RetryConfig{
MaxRetries: 1,
BaseDelay: time.Second,
MaxDelay: time.Second,
})
var waited time.Duration
svc.SetSleepFunc(func(d time.Duration) <-chan time.Time {
waited = d
return instantSleep(d)
})
_ = svc.DeliverWithRetry(
context.Background(), "test",
func(_ context.Context) error {
return errFail
},
)
// With one retry, only the first failure is logged.
var record map[string]any
err := json.Unmarshal([]byte(logs.String()), &record)
if err != nil {
t.Fatalf("log is not one JSON record: %v\n%s", err, logs)
}
if record["retryIn"] != waited.String() {
t.Errorf(
"retryIn logged as %v, want %q",
record["retryIn"], waited.String(),
)
}
}
func TestDeliverWithRetryRespectsContextCancellation(
t *testing.T,
) {
+3 -1
View File
@@ -193,7 +193,9 @@ func (c *Checker) checkConnection(
c.log.Debug(
"port check succeeded",
"target", target,
"latency", latency,
// As text: the JSON log writes a time.Duration as bare
// nanoseconds.
"latency", latency.String(),
)
return &PortResult{
+7
View File
@@ -11,6 +11,13 @@ func ExtractRecordValue(rr dns.RR) string {
return extractRecordValue(rr)
}
// CollectAnswerRecords exports collectAnswerRecords for testing.
func CollectAnswerRecords(msg *dns.Msg, resp *NameserverResponse) {
var state queryState
collectAnswerRecords(msg, resp, &state)
}
// UsableReply exports usableReply for testing.
func UsableReply(resp *dns.Msg, zone string, name string) bool {
return usableReply(resp, zone, name)
+14 -5
View File
@@ -25,8 +25,10 @@ const (
// can meet a referral that names nameservers without their
// addresses, which are then looked up in turn; without a limit,
// delegations that point at each other would never end. Each level
// multiplies the queries sent.
maxLookupDepth = 2
// multiplies the queries sent. pool.ntp.org needs three: the
// address of its nameserver g.ntpns.org can need a.ntpns.org's,
// which needs a bitnames.com nameserver's.
maxLookupDepth = 3
)
// ErrRefused is returned when a DNS server refuses a query.
@@ -771,6 +773,10 @@ func (r *Resolver) querySingleType(
collectAnswerRecords(msg, resp, state)
}
// collectAnswerRecords adds the records in msg's answer to resp, each
// value once per record type. For a name with a CNAME, a nameserver
// answers a query of any type with that CNAME, so the same value comes
// in the answer to every type asked for.
func collectAnswerRecords(
msg *dns.Msg,
resp *NameserverResponse,
@@ -783,9 +789,12 @@ func collectAnswerRecords(
}
typeName := dns.TypeToString[rr.Header().Rrtype]
resp.Records[typeName] = append(
resp.Records[typeName], val,
)
if !slices.Contains(resp.Records[typeName], val) {
resp.Records[typeName] = append(
resp.Records[typeName], val,
)
}
state.hasRecords = true
}
}
+32
View File
@@ -238,6 +238,38 @@ func TestExtractRecordValue_LetterCase(t *testing.T) {
}
}
// TestCollectAnswerRecords_CNAMEOnce collects the answers a nameserver
// gives for a name with a CNAME, one for each record type a check asks
// for. Each answer holds the CNAME, which must be stored once.
func TestCollectAnswerRecords_CNAMEOnce(t *testing.T) {
t.Parallel()
cname := &dns.CNAME{
Hdr: dns.RR_Header{
Name: "git.eeqj.de.", Rrtype: dns.TypeCNAME, Class: dns.ClassINET,
},
Target: "fsn1app1.datavi.be.",
}
resp := &resolver.NameserverResponse{Records: map[string][]string{}}
for _, qtype := range []uint16{
dns.TypeA, dns.TypeAAAA, dns.TypeCNAME, dns.TypeMX,
dns.TypeTXT, dns.TypeSRV, dns.TypeCAA, dns.TypeNS,
} {
msg := new(dns.Msg)
msg.SetQuestion("git.eeqj.de.", qtype)
msg.Answer = []dns.RR{cname}
resolver.CollectAnswerRecords(msg, resp)
}
assert.Equal(t,
map[string][]string{"CNAME": {"fsn1app1.datavi.be."}},
resp.Records,
)
}
// TestShuffled shuffles the root servers with many seeds. Every order
// must hold each root server once, so each is tried before a
// resolution fails; each root server must come first for some seed, so
+38 -12
View File
@@ -210,33 +210,44 @@ func TestQueryNameservers_GivenAddressesFail(t *testing.T) {
assert.NotEmpty(t, resolver.NSSetFrom(resp, "pool.ntp.org."))
}
// TestQueryNameservers_LookupDepth asks the servers of desec.io about
// ns1.desec.io, giving no address and naming ns1.desec.io without one.
// Below maxLookupDepth its address is looked up and it is asked. At the
// limit it is not, so nothing can be asked: this is where lookups of
// nameserver addresses stop when delegations point at each other.
// TestQueryNameservers_LookupDepth asks the servers of g.ntpns.org, a
// nameserver of pool.ntp.org, for its address, as looking that address
// up does when anyns.pch.net, one of the servers of ntpns.org, gives the
// referral to g.ntpns.org without addresses. Their addresses are looked
// up (here only a.ntpns.org's), and that needs a bitnames.com
// nameserver's address, as the org servers delegate ntpns.org without
// addresses. From depth 1, where looking up g.ntpns.org's address
// starts, that makes three lookups and the address is found. From one
// below maxLookupDepth, the bitnames.com lookup would be past the limit,
// so nothing can be asked.
func TestQueryNameservers_LookupDepth(t *testing.T) {
t.Parallel()
r := newTestResolver(t)
withoutAddresses := []string{"ns1.desec.io."}
withoutAddresses := []string{"a.ntpns.org."}
var resp *dns.Msg
livednstest.Retry(
t,
"QueryNameservers(ns1.desec.io without its address, ns1.desec.io)",
"QueryNameservers(a.ntpns.org without its address, g.ntpns.org)",
func(ctx context.Context) error {
_, err := r.QueryNameservers(
ctx, nil, withoutAddresses, "desec.io.", "ns1.desec.io.",
dns.TypeA, 0,
var err error
resp, err = r.QueryNameservers(
ctx, nil, withoutAddresses, "g.ntpns.org.", "g.ntpns.org.",
dns.TypeA, 1,
)
return err
},
)
assert.NotEmpty(t, resp.Answer)
_, err := r.QueryNameservers(
t.Context(), nil, withoutAddresses, "desec.io.", "ns1.desec.io.",
dns.TypeA, resolver.MaxLookupDepth,
t.Context(), nil, withoutAddresses, "g.ntpns.org.", "g.ntpns.org.",
dns.TypeA, resolver.MaxLookupDepth-1,
)
require.ErrorIs(t, err, resolver.ErrNoNameservers)
}
@@ -760,6 +771,21 @@ func TestLookupNS_MatchesFindAuthoritative(t *testing.T) {
assert.Equal(t, fromFind, fromLookup)
}
// TestLookupNS_ParentZoneDelegatedWithoutAddresses looks up the
// nameservers of g.ntpns.org. The org servers delegate its parent zone,
// ntpns.org, without the addresses of its nameservers, so the walk has
// to look them up to ask them. If it did not, the walk for g.ntpns.org
// would fail and LookupNS would return the nameservers of ntpns.org,
// which a.ntpns.org is not one of.
func TestLookupNS_ParentZoneDelegatedWithoutAddresses(t *testing.T) {
t.Parallel()
r := newTestResolver(t)
nameservers := liveLookupNS(t, r, "g.ntpns.org")
assert.Contains(t, nameservers, "a.ntpns.org.")
}
// ----------------------------------------------------------------
// ResolveIPAddresses tests
// ----------------------------------------------------------------
+19
View File
@@ -8,6 +8,7 @@ import (
"log/slog"
"os"
"path/filepath"
"slices"
"sync"
"time"
@@ -53,8 +54,13 @@ type NameserverRecordState struct {
}
// HostnameState holds per-nameserver monitoring state for a hostname.
// CNAMEAddresses holds the sorted addresses at the end of the name's
// CNAME chain, found when its nameservers answered with a CNAME and no
// address; it is empty otherwise. It is nil when they are not known: a
// state file written before it existed loads with it nil.
type HostnameState struct {
RecordsByNameserver map[string]*NameserverRecordState `json:"recordsByNameserver"`
CNAMEAddresses []string `json:"cnameAddresses"`
LastChecked time.Time `json:"lastChecked"`
}
@@ -196,6 +202,19 @@ func (s *State) Load() error {
return fmt.Errorf("parsing state file: %w", err)
}
// A state file saved before each record value was stored once can
// hold a hostname's CNAME once for every record type asked for.
// Each value is kept once, so the first check does not see a
// record change.
for _, hs := range snapshot.Hostnames {
for _, ns := range hs.RecordsByNameserver {
for recordType, values := range ns.Records {
slices.Sort(values)
ns.Records[recordType] = slices.Compact(values)
}
}
}
s.snapshot = &snapshot
s.log.Info("loaded state from disk", "path", path)
+142
View File
@@ -188,6 +188,148 @@ func TestLoadStateFromBeforeNameserverAddresses(t *testing.T) {
}
}
// TestSaveLoadRoundTrip_CNAMEAddresses checks that no addresses at the
// end of a hostname's CNAME chain load as an empty list, and addresses
// that are not known load as nil: the watcher tells the two apart.
func TestSaveLoadRoundTrip_CNAMEAddresses(t *testing.T) {
t.Parallel()
dir := t.TempDir()
s := state.NewForTestWithDataDir(dir)
want := map[string][]string{
"cname.example.com": {testIP},
"none.example.com": {},
"not-known.example.com": nil,
}
for name, addresses := range want {
s.SetHostnameState(name, &state.HostnameState{
CNAMEAddresses: addresses,
})
}
err := s.Save()
if err != nil {
t.Fatalf("Save() error: %v", err)
}
loaded := state.NewForTestWithDataDir(dir)
err = loaded.Load()
if err != nil {
t.Fatalf("Load() error: %v", err)
}
for name, addresses := range want {
hs, ok := loaded.GetHostnameState(name)
if !ok {
t.Fatalf("missing hostname %s", name)
}
if !reflect.DeepEqual(hs.CNAMEAddresses, addresses) {
t.Errorf(
"%s: loaded %#v, want %#v",
name, hs.CNAMEAddresses, addresses,
)
}
}
}
// TestLoadStateFromBeforeCNAMEAddresses loads a state file written
// before the addresses at the end of a hostname's CNAME chain were
// saved. They load as not known (nil), not as none.
func TestLoadStateFromBeforeCNAMEAddresses(t *testing.T) {
t.Parallel()
dir := t.TempDir()
data := []byte(`{
"version": 1,
"lastUpdated": "2026-02-19T12:00:00Z",
"hostnames": {
"www.example.com": {
"recordsByNameserver": {},
"lastChecked": "2026-02-19T12:00:00Z"
}
}
}`)
err := os.WriteFile(filepath.Join(dir, "state.json"), data, 0o600)
if err != nil {
t.Fatalf("writing state file: %v", err)
}
s := state.NewForTestWithDataDir(dir)
err = s.Load()
if err != nil {
t.Fatalf("Load() error: %v", err)
}
hs, ok := s.GetHostnameState(testHostname)
if !ok {
t.Fatal("missing hostname " + testHostname)
}
if hs.CNAMEAddresses != nil {
t.Errorf("CNAME addresses: got %#v, want nil", hs.CNAMEAddresses)
}
}
// TestLoadStateWithRepeatedValues loads a state file saved when a
// hostname's CNAME was stored once for every record type asked for.
// Each value must load once, and every different value must load.
func TestLoadStateWithRepeatedValues(t *testing.T) {
t.Parallel()
dir := t.TempDir()
data := []byte(`{
"version": 1,
"hostnames": {
"www.example.com": {
"recordsByNameserver": {
"ns1.example.com.": {
"records": {
"A": ["192.0.2.2", "192.0.2.1", "192.0.2.2", "192.0.2.1"],
"CNAME": ["a.example.net.", "a.example.net.", "a.example.net."]
},
"status": "ok"
}
}
}
}
}`)
err := os.WriteFile(filepath.Join(dir, "state.json"), data, 0o600)
if err != nil {
t.Fatalf("writing state file: %v", err)
}
s := state.NewForTestWithDataDir(dir)
err = s.Load()
if err != nil {
t.Fatalf("Load() error: %v", err)
}
hs, ok := s.GetHostnameState(testHostname)
if !ok {
t.Fatal("missing hostname " + testHostname)
}
want := map[string][]string{
"A": {"192.0.2.1", "192.0.2.2"},
"CNAME": {"a.example.net."},
}
got := hs.RecordsByNameserver[testNS1].Records
if !reflect.DeepEqual(got, want) {
t.Errorf("records: got %v, want %v", got, want)
}
}
// TestSaveLoadRoundTrip_Hostnames verifies hostname data survives a save/load cycle.
func TestSaveLoadRoundTrip_Hostnames(t *testing.T) {
t.Parallel()
+72
View File
@@ -1,10 +1,13 @@
package watcher_test
import (
"bytes"
"context"
"log/slog"
"reflect"
"strings"
"testing"
"time"
"sneak.berlin/go/dnswatcher/internal/portcheck"
"sneak.berlin/go/dnswatcher/internal/resolver"
@@ -79,3 +82,72 @@ func TestCancelledCheckSavesNothing(t *testing.T) {
t.Errorf("sent %v, want no notifications", notifications)
}
}
// newLoggingWatcher returns a watcher for a domain and a hostname, with
// the real resolver, that writes what it logs at warning level or above
// into the returned buffer.
func newLoggingWatcher(t *testing.T) (*watcher.Watcher, *bytes.Buffer) {
t.Helper()
cfg := defaultTestConfig(t)
cfg.Domains = []string{testSmallDomain}
cfg.Hostnames = []string{host}
w, _ := newTestWatcher(t, cfg)
logs := &bytes.Buffer{}
w.SetLogger(slog.New(slog.NewJSONHandler(
logs, &slog.HandlerOptions{Level: slog.LevelWarn},
)))
return w, logs
}
// TestLookupCutShortIsNotLogged checks a domain and a hostname, looks
// up a nameserver's addresses and follows a CNAME, with the context
// cancelled, as shutdown leaves it. The real resolver fails each lookup
// without sending a query. Shutdown cutting a lookup short is not a
// failure, so nothing may be logged at warning level or above.
func TestLookupCutShortIsNotLogged(t *testing.T) {
t.Parallel()
w, logs := newLoggingWatcher(t)
ctx, cancel := context.WithCancel(t.Context())
cancel()
w.RunOnce(ctx)
w.ResolveNameserverAddresses(ctx, []string{nsA}, nil)
w.ResolveCNAMEAddresses(ctx, host, cnameState(), nil)
if logs.Len() > 0 {
t.Errorf("logged at warning level or above:\n%s", logs)
}
}
// TestLookupOutOfTimeIsLoggedAsError does what
// TestLookupCutShortIsNotLogged does, with the context's deadline passed
// instead. A lookup that ran out of time did fail, so the domain's NS
// lookup, the hostname's lookup, the nameserver's address lookup and the
// CNAME's are each logged as an error.
func TestLookupOutOfTimeIsLoggedAsError(t *testing.T) {
t.Parallel()
w, logs := newLoggingWatcher(t)
ctx, cancel := context.WithDeadline(t.Context(), time.Now())
t.Cleanup(cancel)
w.RunOnce(ctx)
w.ResolveNameserverAddresses(ctx, []string{nsA}, nil)
w.ResolveCNAMEAddresses(ctx, host, cnameState(), nil)
const want = 4
lines := strings.Count(logs.String(), "\n")
errorLines := strings.Count(logs.String(), `"level":"ERROR"`)
if lines != want || errorLines != want {
t.Errorf("logged:\n%s\nwant %d lines, each at error level", logs, want)
}
}
+336
View File
@@ -0,0 +1,336 @@
package watcher_test
import (
"context"
"log/slog"
"slices"
"testing"
"sneak.berlin/go/dnswatcher/internal/livednstest"
"sneak.berlin/go/dnswatcher/internal/resolver"
"sneak.berlin/go/dnswatcher/internal/state"
"sneak.berlin/go/dnswatcher/internal/watcher"
)
// TestCNAMEIntoAnotherZonePortAndTLSChecks runs the port and TLS
// checks on hostname state built here: the name's nameserver answered
// with a CNAME into another zone, and following it found ip1. Both
// checks must use ip1. They look nothing up, so the watcher has no
// resolver.
func TestCNAMEIntoAnotherZonePortAndTLSChecks(t *testing.T) {
t.Parallel()
cfg := defaultTestConfig(t)
cfg.Hostnames = []string{host}
deps := newTestDeps(t, cfg)
w := watcher.NewForTest(
cfg, deps.state, nil,
deps.portChecker, deps.tlsChecker, deps.notifier,
)
deps.state.SetHostnameState(host, cnameState(ip1))
w.CheckAllPorts(t.Context())
w.RunTLSChecks(t.Context())
snap := deps.state.GetSnapshot()
ps, ok := snap.Ports[ip1+":443"]
if !ok || !slices.Contains(ps.Hostnames, host) {
t.Errorf("no port state for %s at %s:443", host, ip1)
}
certKey := ip1 + ":443:" + host
if _, ok := snap.Certificates[certKey]; !ok {
t.Errorf("no certificate state %s", certKey)
}
}
// TestCNAMEThatCannotBeFollowedKeepsPrevious runs a check of a name, not
// the watcher's first, from the point where its records have been looked
// up: they hold a CNAME to a target under .invalid, whose lookup fails.
// The previous check found the same records, and oldIP at the end of the
// CNAME. The check must keep oldIP and send nothing.
func TestCNAMEThatCannotBeFollowedKeepsPrevious(t *testing.T) {
t.Parallel()
w, deps := newTestWatcher(t, defaultTestConfig(t))
w.SetFirstRun(false)
records := map[string]map[string][]string{
nsA: cnameTo("target.example.invalid."),
}
prev := hostnameState(records)
prev.CNAMEAddresses = []string{oldIP}
deps.state.SetHostnameState(host, prev)
// The result is the same whether or not live DNS answers, so the
// lookup is not retried.
_ = livednstest.Run(func(ctx context.Context) error {
w.UpdateHostnameState(ctx, host, hostnameState(records))
return nil
})
hs, _ := deps.state.GetHostnameState(host)
if !slices.Equal(hs.CNAMEAddresses, prev.CNAMEAddresses) {
t.Errorf(
"saved %v, want %v",
hs.CNAMEAddresses, prev.CNAMEAddresses,
)
}
notifications := deps.notifier.getNotifications()
if len(notifications) != 0 {
t.Errorf("sent %v, want no notifications", notifications)
}
}
// followLive follows in live DNS the CNAMEs in a name's records, built
// from records, and returns the addresses saved for the name. The
// previous check saved oldIP, which is kept when a target cannot be
// followed; that is retried. The tests point CNAMEs only at names in
// zones with two nameservers, to keep queries few (see the top of
// watcher_test.go).
func followLive(
t *testing.T,
records map[string]map[string][]string,
) []string {
t.Helper()
w := watcher.NewForTest(
nil, nil, resolver.NewFromLogger(slog.Default()), nil, nil, nil,
)
prev := cnameState(oldIP)
var current *state.HostnameState
livednstest.Retry(t, "following CNAMEs", func(ctx context.Context) error {
current = hostnameState(records)
w.ResolveCNAMEAddresses(ctx, host, current, prev)
if slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
return livednstest.ErrNoAnswer
}
return nil
})
return current.CNAMEAddresses
}
// TestCNAMEAddressesOfEveryTarget gives a name's two nameservers
// different CNAME targets, as when a secondary still serves an old one.
// The addresses at the end of both are saved, whichever answer is read
// first: one.one.one.one has 1.1.1.1, and dns.adguard-dns.com has
// 94.140.14.14.
func TestCNAMEAddressesOfEveryTarget(t *testing.T) {
t.Parallel()
found := followLive(t, map[string]map[string][]string{
nsA: cnameTo("one.one.one.one."),
nsB: cnameTo("dns.adguard-dns.com."),
})
for _, ip := range []string{"1.1.1.1", "94.140.14.14"} {
if !slices.Contains(found, ip) {
t.Errorf("saved %v, want %s among them", found, ip)
}
}
}
// TestCNAMEChainEndingInNoAddressSavesEmptyList follows a CNAME to a
// name live DNS answers with NXDOMAIN. An empty list is saved, not nil,
// which would mean the addresses are not known.
func TestCNAMEChainEndingInNoAddressSavesEmptyList(t *testing.T) {
t.Parallel()
found := followLive(t, map[string]map[string][]string{
nsA: cnameTo("this-surely-does-not-exist-xyz.example.org."),
})
if found == nil || len(found) != 0 {
t.Errorf("saved %#v, want an empty list", found)
}
}
// TestCNAMEBesideAnAddressNotFollowed gives one nameserver of a name an
// address and another a CNAME. The CNAME is not followed: an empty list
// is saved, not nil, and nothing is looked up, the watcher having no
// resolver.
func TestCNAMEBesideAnAddressNotFollowed(t *testing.T) {
t.Parallel()
w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
current := hostnameState(map[string]map[string][]string{
nsA: {"A": {ip1}},
nsB: cnameTo("target.example.org."),
})
w.ResolveCNAMEAddresses(t.Context(), host, current, nil)
if current.CNAMEAddresses == nil || len(current.CNAMEAddresses) != 0 {
t.Errorf("saved %#v, want an empty list", current.CNAMEAddresses)
}
}
// TestCNAMEWhoseNameserversAllFailedKeepsPrevious checks a name none of
// whose nameservers answered. The addresses the previous check saved
// from following its CNAME are kept, and nothing is looked up: the
// watcher has no resolver.
func TestCNAMEWhoseNameserversAllFailedKeepsPrevious(t *testing.T) {
t.Parallel()
w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
current := saved(map[string]*state.NameserverRecordState{
nsA: failed(), nsB: failed(),
})
prev := cnameState(oldIP)
w.ResolveCNAMEAddresses(t.Context(), host, current, prev)
if !slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
t.Errorf(
"saved %v, want %v",
current.CNAMEAddresses, prev.CNAMEAddresses,
)
}
}
// cnameTo builds the records of a nameserver that answered with a CNAME
// to target and no address.
func cnameTo(target string) map[string][]string {
return map[string][]string{"CNAME": {target}}
}
// cnameState builds the state a check leaves behind for a name whose
// nameserver answered with a CNAME and no address, when following the
// CNAME found these addresses, which may be none.
func cnameState(addresses ...string) *state.HostnameState {
hs := hostnameState(map[string]map[string][]string{
nsA: cnameTo("target.example.org."),
})
hs.CNAMEAddresses = append([]string{}, addresses...)
return hs
}
func TestCNAMEAddressChangeAlerts(t *testing.T) {
t.Parallel()
// A state file written before the addresses were saved loads with
// them nil.
olderStateFile := cnameState()
olderStateFile.CNAMEAddresses = nil
// Each case is the state saved by the previous check and by the
// current one. The name's records are the same in both.
tests := []struct {
name string
prev, current *state.HostnameState
want int
}{
{
"same addresses",
cnameState(ip1, ip2), cnameState(ip1, ip2), 0,
},
{
"same addresses in another order",
cnameState(ip2, ip1), cnameState(ip1, ip2), 0,
},
{
"address replaced",
cnameState(ip1), cnameState(ip2), 1,
},
{
"address added",
cnameState(ip1), cnameState(ip1, ip2), 1,
},
{
"no address at the end of the chain now",
cnameState(ip1), cnameState(), 1,
},
{
"addresses at the end of the chain again",
cnameState(), cnameState(ip1), 1,
},
{
"state file from before addresses were saved",
olderStateFile, cnameState(ip1), 0,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
notifier := &mockNotifier{}
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
w.DetectHostnameChanges(t.Context(), host, tt.prev, tt.current)
got := len(notifier.getNotifications())
if got != tt.want {
t.Errorf("sent %d notifications, want %d", got, tt.want)
}
})
}
}
func TestCNAMEAddressChangeAlertNamesHostnameAndAddresses(t *testing.T) {
t.Parallel()
notifier := &mockNotifier{}
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
w.DetectHostnameChanges(
t.Context(), host, cnameState(ip1), cnameState(ip2, ip3),
)
want := notification{
Title: "CNAME Address Change: " + host,
Message: "Hostname: " + host +
"\nOld: " + ip1 + "\nNew: " + ip2 + ", " + ip3,
Priority: "warning",
}
got := notifier.getNotifications()
if len(got) != 1 || got[0] != want {
t.Errorf("sent %v, want %v", got, want)
}
}
// TestNameMovedFromARecordsToCNAMEAlerts checks a name that answers
// with an A record and then with a CNAME whose chain ends in ip2. The
// second check is notified as a CNAME address change from no addresses,
// beside the record change. Nothing is looked up: the watcher has no
// resolver.
func TestNameMovedFromARecordsToCNAMEAlerts(t *testing.T) {
t.Parallel()
notifier := &mockNotifier{}
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
prev := hostnameState(map[string]map[string][]string{
nsA: {"A": {ip1}},
})
w.ResolveCNAMEAddresses(t.Context(), host, prev, nil)
w.DetectHostnameChanges(t.Context(), host, prev, cnameState(ip2))
title := "CNAME Address Change: " + host
message := "Hostname: " + host + "\nOld: \nNew: " + ip2
got := notifier.getNotifications()
if !slices.ContainsFunc(got, func(n notification) bool {
return n.Title == title && n.Message == message
}) {
t.Errorf("sent %v, want %q with %q among them", got, title, message)
}
}
+30
View File
@@ -31,6 +31,12 @@ func NewForTest(
}
}
// SetLogger replaces the watcher's logger, so a test can read what it
// logs.
func (w *Watcher) SetLogger(log *slog.Logger) {
w.log = log
}
// NewlyDisagreeingPairs exports newlyDisagreeingPairs for testing.
func NewlyDisagreeingPairs(
prev, current *state.HostnameState,
@@ -38,6 +44,21 @@ func NewlyDisagreeingPairs(
return newlyDisagreeingPairs(prev, current)
}
// SetFirstRun sets whether the watcher is on its first check, in which
// nothing is compared with the previous check. NewForTest's watcher is.
func (w *Watcher) SetFirstRun(firstRun bool) {
w.firstRun = firstRun
}
// UpdateHostnameState exports updateHostnameState for testing.
func (w *Watcher) UpdateHostnameState(
ctx context.Context,
hostname string,
newState *state.HostnameState,
) {
w.updateHostnameState(ctx, hostname, newState)
}
// DetectHostnameChanges exports detectHostnameChanges for testing.
func (w *Watcher) DetectHostnameChanges(
ctx context.Context,
@@ -57,6 +78,15 @@ func (w *Watcher) ResolveNameserverAddresses(
return w.resolveNameserverAddresses(ctx, nameservers, prev)
}
// ResolveCNAMEAddresses exports resolveCNAMEAddresses for testing.
func (w *Watcher) ResolveCNAMEAddresses(
ctx context.Context,
hostname string,
current, prev *state.HostnameState,
) {
w.resolveCNAMEAddresses(ctx, hostname, current, prev)
}
// DetectNSAddressChanges exports detectNSAddressChanges for testing.
func (w *Watcher) DetectNSAddressChanges(
ctx context.Context,
+55
View File
@@ -183,3 +183,58 @@ func TestInconsistencyAlert(t *testing.T) {
})
}
}
// TestFirstCheckAfterRepeatedValuesLoaded saves a state file holding a
// hostname's CNAME once for every record type asked for, as checks did
// before each value was stored once, and two addresses each repeated,
// and loads it. A check that then finds each value once at each
// nameserver must notify nothing.
func TestFirstCheckAfterRepeatedValuesLoaded(t *testing.T) {
t.Parallel()
const (
cnameType = "CNAME"
cname = "c.example.net."
)
cfg := defaultTestConfig(t)
repeated := map[string][]string{
"A": {ip2, ip1, ip2, ip1},
cnameType: {cname, cname, cname, cname, cname, cname, cname, cname},
}
once := map[string][]string{"A": {ip1, ip2}, cnameType: {cname}}
saved := newTestDeps(t, cfg).state
saved.SetHostnameState(host, hostnameState(map[string]map[string][]string{
nsA: repeated, nsB: repeated,
}))
err := saved.Save()
if err != nil {
t.Fatalf("saving the state file: %v", err)
}
deps := newTestDeps(t, cfg)
err = deps.state.Load()
if err != nil {
t.Fatalf("loading the state file: %v", err)
}
prev, ok := deps.state.GetHostnameState(host)
if !ok {
t.Fatal("the state file has no state for " + host)
}
current := hostnameState(map[string]map[string][]string{
nsA: once, nsB: once,
})
// The hostname change detection uses only the notifier.
w := watcher.NewForTest(nil, nil, nil, nil, nil, deps.notifier)
w.DetectHostnameChanges(t.Context(), host, prev, current)
if got := deps.notifier.getNotifications(); len(got) != 0 {
t.Errorf("sent %v, want no notification", got)
}
}
+66
View File
@@ -0,0 +1,66 @@
package watcher_test
import (
"testing"
"sneak.berlin/go/dnswatcher/internal/watcher"
)
// When one nameserver's A record changes and its TXT record does not,
// the record change and the inconsistency it starts name the A record
// alone, with its values written as plain text.
func TestChangeMessagesNameTheChangedType(t *testing.T) {
t.Parallel()
// A nameserver's records: this A address and the same TXT record.
records := func(address string) map[string][]string {
return map[string][]string{
"A": {address},
"TXT": {"v=spf1 -all"},
}
}
before := hostnameState(map[string]map[string][]string{
nsA: records(ip1),
nsB: records(ip1),
})
after := hostnameState(map[string]map[string][]string{
nsA: records(ip1),
nsB: records(ip2),
})
// The hostname change detection uses only the notifier.
notifier := &mockNotifier{}
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
w.DetectHostnameChanges(t.Context(), host, before, after)
want := map[string]string{
"Record Change: " + host: `Hostname: www.example.net
Nameserver: b.ns.example.net.
Type: A
Old: 192.0.2.1
New: 192.0.2.2`,
"Inconsistency: " + host: `Hostname: www.example.net
Type: A
a.ns.example.net.: 192.0.2.1
b.ns.example.net.: 192.0.2.2`,
}
notifications := notifier.getNotifications()
if len(notifications) != len(want) {
t.Fatalf(
"sent %d notifications, want %d: %v",
len(notifications), len(want), notifications,
)
}
for _, n := range notifications {
if n.Message != want[n.Title] {
t.Errorf(
"%s message:\n%s\nwant:\n%s",
n.Title, n.Message, want[n.Title],
)
}
}
}
+216 -37
View File
@@ -2,6 +2,7 @@ package watcher
import (
"context"
"errors"
"fmt"
"log/slog"
"slices"
@@ -123,8 +124,10 @@ func (w *Watcher) Run(ctx context.Context) {
"watcher starting",
"domains", len(w.config.Domains),
"hostnames", len(w.config.Hostnames),
"dnsInterval", w.config.DNSInterval,
"tlsInterval", w.config.TLSInterval,
// As text: the JSON log writes a time.Duration as bare
// nanoseconds.
"dnsInterval", w.config.DNSInterval.String(),
"tlsInterval", w.config.TLSInterval.String(),
)
w.RunOnce(ctx)
@@ -211,13 +214,29 @@ func (w *Watcher) runDNSChecks(ctx context.Context) {
}
}
// logFailedLookup logs a failed DNS lookup at error level, unless ctx
// was cancelled: shutdown cancels it, and a lookup it cut short did not
// fail. A lookup that ran out of time did fail, so it is logged.
func (w *Watcher) logFailedLookup(
ctx context.Context,
msg string,
args ...any,
) {
if errors.Is(ctx.Err(), context.Canceled) {
return
}
w.log.Error(msg, args...)
}
func (w *Watcher) checkDomain(
ctx context.Context,
domain string,
) {
nameservers, err := w.resolver.LookupNS(ctx, domain)
if err != nil {
w.log.Error(
w.logFailedLookup(
ctx,
"failed to lookup NS",
"domain", domain,
"error", err,
@@ -252,28 +271,9 @@ func (w *Watcher) checkDomain(
LastChecked: now,
})
// Also look up A/AAAA records for the apex domain so that
// port and TLS checks (which read HostnameState) can find
// the domain's IP addresses.
results, err := w.resolver.LookupAllRecords(ctx, domain)
if err != nil {
w.log.Error(
"failed to lookup records for domain",
"domain", domain,
"error", err,
)
return
}
newState := buildHostnameState(results, now)
prevHS, hasPrevHS := w.state.GetHostnameState(domain)
if hasPrevHS && !w.firstRun {
w.detectHostnameChanges(ctx, domain, prevHS, newState)
}
w.state.SetHostnameState(domain, newState)
// The apex domain's records are also checked as a hostname's, so
// that the port and TLS checks find its addresses.
w.checkHostname(ctx, domain)
}
func (w *Watcher) detectNSChanges(
@@ -337,7 +337,8 @@ func (w *Watcher) resolveNameserverAddresses(
continue
}
w.log.Error(
w.logFailedLookup(
ctx,
"no addresses found for nameserver",
"nameserver", ns,
"error", err,
@@ -389,7 +390,8 @@ func (w *Watcher) checkHostname(
) {
results, err := w.resolver.LookupAllRecords(ctx, hostname)
if err != nil {
w.log.Error(
w.logFailedLookup(
ctx,
"failed to lookup records",
"hostname", hostname,
"error", err,
@@ -398,9 +400,23 @@ func (w *Watcher) checkHostname(
return
}
newState := buildHostnameState(results, time.Now().UTC())
w.updateHostnameState(
ctx, hostname, buildHostnameState(results, time.Now().UTC()),
)
}
// updateHostnameState finishes a check of hostname from newState, built
// from its nameservers' answers: it follows the CNAME in them, notifies
// what changed since the previous check, and saves newState.
func (w *Watcher) updateHostnameState(
ctx context.Context,
hostname string,
newState *state.HostnameState,
) {
prev, hasPrev := w.state.GetHostnameState(hostname)
w.resolveCNAMEAddresses(ctx, hostname, newState, prev)
if hasPrev && !w.firstRun {
w.detectHostnameChanges(ctx, hostname, prev, newState)
}
@@ -408,6 +424,77 @@ func (w *Watcher) checkHostname(
w.state.SetHostnameState(hostname, newState)
}
// resolveCNAMEAddresses saves in current the addresses at the end of
// hostname's CNAME chain, when the nameservers' answers in current hold
// a CNAME and no address, and an empty list otherwise. Every CNAME
// target the nameservers gave is followed with ResolveIPAddresses and
// the addresses found for all of them are saved, so nameservers that
// disagree on the target do not change the result from check to check.
// The addresses saved in prev, which may be nil, are kept when none of
// the name's nameservers answered, and when a target cannot be
// followed, as when no nameserver of a zone in its chain answers.
func (w *Watcher) resolveCNAMEAddresses(
ctx context.Context,
hostname string,
current, prev *state.HostnameState,
) {
var prevAddresses []string
if prev != nil {
prevAddresses = prev.CNAMEAddresses
}
// Empty, not nil: nil means the addresses are not known.
current.CNAMEAddresses = []string{}
answered := false
targets := make(map[string]bool)
for _, nsState := range current.RecordsByNameserver {
if nsState.Status != statusOK {
continue
}
answered = true
if len(nsState.Records["A"]) > 0 || len(nsState.Records["AAAA"]) > 0 {
return
}
for _, target := range nsState.Records["CNAME"] {
targets[target] = true
}
}
if !answered {
current.CNAMEAddresses = prevAddresses
return
}
for target := range targets {
ips, err := w.resolver.ResolveIPAddresses(ctx, target)
if err != nil {
w.logFailedLookup(
ctx,
"failed to follow CNAME",
"hostname", hostname,
"target", target,
"error", err,
)
current.CNAMEAddresses = prevAddresses
return
}
current.CNAMEAddresses = append(current.CNAMEAddresses, ips...)
}
// Still the empty list when every chain ends in no address.
slices.Sort(current.CNAMEAddresses)
current.CNAMEAddresses = slices.Compact(current.CNAMEAddresses)
}
// buildHostnameState saves each nameserver's response. A nameserver
// that answered, even with NXDOMAIN or no records, is saved as ok; one
// that timed out or failed is saved as error with the reason, and its
@@ -451,6 +538,37 @@ func (w *Watcher) detectHostnameChanges(
w.detectNSDisappearances(ctx, hostname, prev, current)
w.detectNSFailures(ctx, hostname, prev, current)
w.detectInconsistencies(ctx, hostname, prev, current)
w.detectCNAMEAddressChanges(ctx, hostname, prev, current)
}
// detectCNAMEAddressChanges notifies when the addresses at the end of
// hostname's CNAME chain differ from those the previous check saved,
// including a change from or to none. When the previous addresses are
// not known (nil), as on the first check after loading a state file
// written before they were saved, nothing is compared.
func (w *Watcher) detectCNAMEAddressChanges(
ctx context.Context,
hostname string,
prev, current *state.HostnameState,
) {
old, cur := prev.CNAMEAddresses, current.CNAMEAddresses
if old == nil || sliceEqual(old, cur) {
return
}
msg := fmt.Sprintf(
"Hostname: %s\nOld: %s\nNew: %s",
hostname,
strings.Join(old, ", "),
strings.Join(cur, ", "),
)
w.notify.SendNotification(
ctx,
"CNAME Address Change: "+hostname,
msg,
"warning",
)
}
// detectRecordChanges compares each nameserver's records with those of
@@ -472,10 +590,12 @@ func (w *Watcher) detectRecordChanges(
}
msg := fmt.Sprintf(
"Hostname: %s\nNameserver: %s\n"+
"Old: %v\nNew: %v",
"Hostname: %s\nNameserver: %s\n%s",
hostname, ns,
prevNS.Records, cur.Records,
recordDifferences(
"Old", prevNS.Records,
"New", cur.Records,
),
)
w.notify.SendNotification(
@@ -563,10 +683,12 @@ func (w *Watcher) detectInconsistencies(
ns1, ns2 := pair[0], pair[1]
msg := fmt.Sprintf(
"Hostname: %s\n%s: %v\n%s: %v",
"Hostname: %s\n%s",
hostname,
ns1, current.RecordsByNameserver[ns1].Records,
ns2, current.RecordsByNameserver[ns2].Records,
recordDifferences(
ns1, current.RecordsByNameserver[ns1].Records,
ns2, current.RecordsByNameserver[ns2].Records,
),
)
w.notify.SendNotification(
@@ -747,6 +869,9 @@ func (w *Watcher) noNameserverAnswered(name string) bool {
return true
}
// collectIPs returns the addresses saved for hostname: those in its
// nameservers' A and AAAA records, and those at the end of its CNAME
// chain.
func (w *Watcher) collectIPs(hostname string) []string {
hs, ok := w.state.GetHostnameState(hostname)
if !ok {
@@ -765,6 +890,10 @@ func (w *Watcher) collectIPs(hostname string) []string {
}
}
for _, ip := range hs.CNAMEAddresses {
ipSet[ip] = true
}
result := make([]string, 0, len(ipSet))
for ip := range ipSet {
result = append(result, ip)
@@ -1049,8 +1178,9 @@ func (w *Watcher) saveState() {
// maybeSendTestNotification sends a startup status notification
// after the first full scan completes, if SEND_TEST_NOTIFICATION
// is enabled. The message is clearly informational ("all ok")
// and not an error or anomaly alert.
// is enabled. The message is informational, not an error or anomaly
// alert. It is written before it reaches any endpoint, so it claims
// nothing about whether the endpoints work.
func (w *Watcher) maybeSendTestNotification(ctx context.Context) {
if !w.config.SendTestNotification {
return
@@ -1062,7 +1192,8 @@ func (w *Watcher) maybeSendTestNotification(ctx context.Context) {
"dnswatcher has started and completed its initial scan.\n"+
"Monitoring %d domain(s) and %d hostname(s).\n"+
"Tracking %d port endpoint(s) and %d TLS certificate(s).\n"+
"All notification channels are working.",
"This is a test notification, sent to every configured "+
"notification endpoint.",
len(snap.Domains),
len(snap.Hostnames),
len(snap.Ports),
@@ -1107,6 +1238,54 @@ func recordsEqual(
return true
}
// recordDifferences describes, in sorted order of type, each record
// type whose values differ between a and b: a line naming the type,
// then a line with a's values after labelA and one with b's after
// labelB. Types with the same values in both are left out.
func recordDifferences(
labelA string, a map[string][]string,
labelB string, b map[string][]string,
) string {
types := make([]string, 0, len(a)+len(b))
for recordType := range a {
types = append(types, recordType)
}
for recordType := range b {
if _, ok := a[recordType]; !ok {
types = append(types, recordType)
}
}
sort.Strings(types)
var lines []string
for _, recordType := range types {
if sliceEqual(a[recordType], b[recordType]) {
continue
}
lines = append(lines,
"Type: "+recordType,
labelA+": "+joinValues(a[recordType]),
labelB+": "+joinValues(b[recordType]),
)
}
return strings.Join(lines, "\n")
}
// joinValues lists record values separated by commas, or says none.
func joinValues(values []string) string {
if len(values) == 0 {
return "none"
}
return strings.Join(values, ", ")
}
func sliceEqual(a, b []string) bool {
if len(a) != len(b) {
return false
+22 -4
View File
@@ -312,7 +312,8 @@ func lookupNameservers(t *testing.T, name string) []string {
return nameservers
}
// addresses returns the A and AAAA values saved for a hostname.
// addresses returns the A and AAAA values saved for a hostname, and the
// addresses saved at the end of its CNAME chain.
func addresses(hs *state.HostnameState) []string {
var ips []string
@@ -321,7 +322,7 @@ func addresses(hs *state.HostnameState) []string {
ips = append(ips, nsState.Records["AAAA"]...)
}
return ips
return append(ips, hs.CNAMEAddresses...)
}
// assertNotified checks that a notification with this title and
@@ -371,6 +372,14 @@ func TestFirstRunBaseline(t *testing.T) {
assertNoNotifications(t, deps)
assertStatePopulated(t, deps)
// testHost answers with an address, so the check saves an empty list
// of CNAME addresses for it; nil would mean the check did not look
// at whether to follow a CNAME.
hs, _ := deps.state.GetHostnameState(testHost)
if hs.CNAMEAddresses == nil || len(hs.CNAMEAddresses) != 0 {
t.Errorf("saved CNAME addresses %#v, want []", hs.CNAMEAddresses)
}
}
func assertNoNotifications(
@@ -866,18 +875,27 @@ func TestSendTestNotification_ViaRun(t *testing.T) {
notifications := deps.notifier.getNotifications()
// No names are configured, so every count is 0.
wantMessage := "dnswatcher has started and completed its initial scan.\n" +
"Monitoring 0 domain(s) and 0 hostname(s).\n" +
"Tracking 0 port endpoint(s) and 0 TLS certificate(s).\n" +
"This is a test notification, sent to every configured " +
"notification endpoint."
found := false
for _, n := range notifications {
if n.Priority == "success" &&
n.Title == "✅ dnswatcher startup complete" {
n.Title == "✅ dnswatcher startup complete" &&
n.Message == wantMessage {
found = true
}
}
if !found {
t.Errorf(
"expected startup test notification, got: %v",
"expected startup test notification with message %q, got: %v",
wantMessage,
notifications,
)
}