Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
661ef8d937 | ||
|
|
250f3dd687 | ||
|
|
c07976a73a | ||
|
|
b047c3c64c | ||
|
|
f99de191c0 | ||
|
|
5db5048754 | ||
|
|
26c9c74d8e | ||
|
|
ceb24c5004 | ||
|
|
ee4cadbd05 |
@@ -121,14 +121,26 @@ notification endpoint set, changes show only on the dashboard; see
|
||||
failed on it, and answers differently is reported on the check where it
|
||||
answers. If a pair agrees again and later disagrees, the alert is sent
|
||||
again.
|
||||
- **CNAME address change**: The addresses at the end of a name's CNAME chain
|
||||
differ from those of the previous check. They are found when its
|
||||
nameservers answer with a CNAME and no address; a name that answers with
|
||||
an address has none. A change from or to no addresses is sent too, as when
|
||||
a name moves between A records and a CNAME. Nothing is sent when the
|
||||
previous addresses were kept because a chain could not be followed or none
|
||||
of the name's nameservers answered. The first check after loading a state
|
||||
file without `cnameAddresses` sends nothing: it saves the addresses it
|
||||
finds for the next check to compare.
|
||||
|
||||
### TCP Port Monitoring
|
||||
|
||||
- For every configured domain and hostname, constructs a deduplicated list of
|
||||
the IPv4 and IPv6 addresses in the A and AAAA records its authoritative
|
||||
nameservers returned. A CNAME is not followed: a name whose CNAME points into
|
||||
another zone usually has no addresses here, so its ports and certificate are
|
||||
not checked.
|
||||
nameservers returned. When they returned a CNAME and no address, the CNAME
|
||||
chain is followed and the addresses at its end are used, and a change in those
|
||||
is notified as a CNAME address change. When the nameservers gave different
|
||||
CNAME targets, each is followed and the addresses of all are used. When a
|
||||
chain cannot be followed, or none of the name's nameservers answered, the
|
||||
addresses the last check found at its end are used.
|
||||
- Checks TCP connectivity on ports **80** and **443** for each IP address.
|
||||
- Every **1 hour** by default, re-checks all ports.
|
||||
- Any change in port availability triggers a notification:
|
||||
@@ -176,6 +188,8 @@ includes:
|
||||
- **DNS NS changes**: Which domain, which nameservers were added/removed.
|
||||
- **NS address changes**: Which domain, which nameserver, its old and new
|
||||
addresses.
|
||||
- **CNAME address changes**: Which hostname, the old and new addresses at the
|
||||
end of its CNAME chain.
|
||||
- **NS query failures**: Which nameserver failed, error type (timeout, SERVFAIL,
|
||||
REFUSED, network error), which hostname/domain affected.
|
||||
- **NS recoveries**: Which nameserver recovered, which hostname/domain.
|
||||
@@ -216,9 +230,11 @@ dnswatcher includes an unauthenticated, read-only web dashboard at the root URL
|
||||
|
||||
- **Summary counts** for monitored domains, hostnames, ports, and certificates.
|
||||
- **Domains** with their discovered nameservers.
|
||||
- **Hostnames** with per-nameserver DNS records and status.
|
||||
- **Hostnames** with per-nameserver DNS records and status. For a nameserver
|
||||
whose query failed, the reason is shown in place of the records.
|
||||
- **Ports** with open/closed state and associated hostnames.
|
||||
- **TLS certificates** with CN, issuer, expiry, and status.
|
||||
- **TLS certificates** with CN, issuer, expiry, and status. For a failed check,
|
||||
the reason is shown in place of CN, issuer and expiry.
|
||||
- **Recent alerts** (last 100 notifications sent since the process started),
|
||||
displayed in reverse chronological order.
|
||||
|
||||
@@ -245,6 +261,10 @@ dnswatcher exposes a lightweight HTTP API for operational visibility:
|
||||
| `GET /api/v1/status` | Current monitoring state |
|
||||
| `GET /metrics` | Prometheus metrics, see below |
|
||||
|
||||
In `/api/v1/status`, each nameserver entry and certificate entry whose `status`
|
||||
is `error` also has `error`, the reason, as in the state file (see State File
|
||||
Format).
|
||||
|
||||
`/metrics` is served only when `DNSWATCHER_METRICS_USERNAME` is set, behind
|
||||
Basic Auth. It has the Prometheus Go client's default metrics only (Go runtime,
|
||||
process, and counts of `/metrics` requests); dnswatcher records no metrics of
|
||||
@@ -416,8 +436,8 @@ all of them are looked up, so that each can be asked. When it gives addresses
|
||||
for only some of them, those are asked first, and the others are looked up and
|
||||
asked only if none of those gives a usable reply. Both hold in the walk to a
|
||||
name's nameservers and in the lookup of a nameserver's own address. Such a
|
||||
lookup can need others in turn; lookups go at most two deep, one inside another,
|
||||
so delegations that point at each other still end.
|
||||
lookup can need others in turn; lookups go at most three deep, one inside
|
||||
another, so delegations that point at each other still end.
|
||||
|
||||
This approach ensures:
|
||||
|
||||
@@ -425,9 +445,11 @@ This approach ensures:
|
||||
- Ability to detect split-horizon or inconsistent responses across authoritative
|
||||
servers.
|
||||
|
||||
CNAME chains are followed (with a depth limit to prevent loops) only to find the
|
||||
addresses of nameservers. A watched name's records are stored as its nameservers
|
||||
return them, CNAME included, without following it.
|
||||
A watched name's records are stored as its nameservers return them, CNAME
|
||||
included. When they return a CNAME and no address, the chain of every CNAME
|
||||
target they gave is followed (with a depth limit to prevent loops) to the A and
|
||||
AAAA records at its end, and the port and TLS checks use those addresses.
|
||||
Nameservers' addresses are also found by following CNAME chains.
|
||||
|
||||
Sending a notification or a Sentry report is the one use of the system's
|
||||
resolver: the HTTP client looks up the webhook's or Sentry's host name with it.
|
||||
@@ -474,6 +496,7 @@ merged view, to enable inconsistency detection.
|
||||
"lastChecked": "2026-02-19T12:00:00Z"
|
||||
}
|
||||
},
|
||||
"cnameAddresses": [],
|
||||
"lastChecked": "2026-02-19T12:00:00Z"
|
||||
}
|
||||
},
|
||||
@@ -520,6 +543,14 @@ certificate entry whose TLS connection or handshake failed likewise has status
|
||||
resolves to. A state file without it loads, and the next check fills it in
|
||||
without a notification.
|
||||
|
||||
`cnameAddresses` lists the sorted addresses at the end of the chain of every
|
||||
CNAME target a hostname's nameservers gave, found when they answered with a
|
||||
CNAME and no address; it is empty when they answered with an address. When a
|
||||
chain cannot be followed, or none of the name's nameservers answered, the
|
||||
previous check's list is kept, or `null` when no earlier check saved one. A
|
||||
state file without it loads, and the first check after that saves it without a
|
||||
notification.
|
||||
|
||||
A port entry in the older format, with one `hostname` instead of the `hostnames`
|
||||
list, loads as a list of that one name.
|
||||
|
||||
@@ -677,7 +708,9 @@ docker run -d \
|
||||
- Port and TLS checks use the IP addresses found by the DNS phase that
|
||||
immediately precedes them. When that phase cannot find a name's
|
||||
nameservers at all, the addresses an earlier check saved for the name are
|
||||
used.
|
||||
used. When it cannot follow a name's CNAME chain, or none of the name's
|
||||
nameservers answered, the addresses an earlier check found at the end of
|
||||
the chain are used.
|
||||
4. **On change detection**: Send notifications to all configured endpoints,
|
||||
update in-memory state, persist to disk.
|
||||
5. **Shutdown**: The watcher stops checking and saves the final state to disk,
|
||||
|
||||
@@ -19,8 +19,24 @@ trial run of the finished image: https://git.eeqj.de/sneak/dnswatcher/issues/149
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-02: nameservers a referral gives no address for are looked up when
|
||||
needed, two deep at most; `pool.ntp.org`'s nameservers resolve (closes #221).
|
||||
- 2026-10-02: nameservers a referral names without addresses are looked up,
|
||||
three deep at most; `pool.ntp.org`'s nameservers resolve (closes #221).
|
||||
- 2026-10-02: the dashboard and `/api/v1/status` show why a nameserver query or
|
||||
a certificate check failed, which only the state file showed (closes #225).
|
||||
- 2026-10-02: a name's CNAME is stored once per nameserver, not once per record
|
||||
type asked for; a state file with repeats loads each value once (closes #220).
|
||||
- 2026-10-02: a DNS lookup that shutdown cuts short logs no error; one that
|
||||
fails otherwise, or runs out of time, still does (closes #229).
|
||||
- 2026-10-02: Record Change and Inconsistency notifications list only the record
|
||||
types that differ, each with its values as plain text (closes #219).
|
||||
- 2026-10-02: the startup notification no longer says every notification
|
||||
endpoint works; it says it is a test sent to each of them (closes #230).
|
||||
- 2026-10-02: a Mattermost webhook that answers an HTTP error is logged as
|
||||
`mattermost notification failed`, not as a Slack failure (closes #227).
|
||||
- 2026-10-02: durations in the log are written as text such as `2m0s`, not as a
|
||||
bare count of nanoseconds (closes #228).
|
||||
- 2026-10-02: a watched name whose nameservers answer with a CNAME and no
|
||||
address gets port and TLS checks at the end of its CNAME chain (closes #203).
|
||||
- 2026-10-02: a resolver test that reads one record type from a nameserver's
|
||||
answer asks again when that type is missing from it (closes #218).
|
||||
- 2026-10-02: a plain `docker build .` of a clone stamps its tag or short
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -78,3 +79,39 @@ func TestFormatRecords(t *testing.T) {
|
||||
t.Errorf("unexpected format: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// dashboardRow returns the table row of page that contains name.
|
||||
func dashboardRow(t *testing.T, page string, name string) string {
|
||||
t.Helper()
|
||||
|
||||
for row := range strings.SplitSeq(page, "<tr") {
|
||||
if strings.Contains(row, name) {
|
||||
return row
|
||||
}
|
||||
}
|
||||
|
||||
t.Fatalf("dashboard has no row containing %q", name)
|
||||
|
||||
return ""
|
||||
}
|
||||
|
||||
// TestDashboardShowsFailureReasons checks that the dashboard shows the
|
||||
// reason in the row of a failed nameserver and of a failed certificate,
|
||||
// and not in the row of a nameserver that answered.
|
||||
func TestDashboardShowsFailureReasons(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
page := get(t, newHandlersWithFailures(t).HandleDashboard())
|
||||
|
||||
if !strings.Contains(dashboardRow(t, page, failedNS), nsFailureReason) {
|
||||
t.Errorf("row of %s does not show %q", failedNS, nsFailureReason)
|
||||
}
|
||||
|
||||
if strings.Contains(dashboardRow(t, page, answeringNS), nsFailureReason) {
|
||||
t.Errorf("row of %s shows %q", answeringNS, nsFailureReason)
|
||||
}
|
||||
|
||||
if !strings.Contains(dashboardRow(t, page, certKey), certFailedReason) {
|
||||
t.Errorf("row of %s does not show %q", certKey, certFailedReason)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18,6 +18,7 @@ type statusDomainInfo struct {
|
||||
type statusHostnameNSInfo struct {
|
||||
Records map[string][]string `json:"records"`
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
LastChecked time.Time `json:"lastChecked"`
|
||||
}
|
||||
|
||||
@@ -41,6 +42,7 @@ type statusCertificateInfo struct {
|
||||
NotAfter time.Time `json:"notAfter"`
|
||||
SubjectAlternativeNames []string `json:"subjectAlternativeNames"`
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
LastChecked time.Time `json:"lastChecked"`
|
||||
}
|
||||
|
||||
@@ -144,6 +146,7 @@ func buildHostnames(
|
||||
info.Nameservers[ns] = &statusHostnameNSInfo{
|
||||
Records: recs,
|
||||
Status: nsState.Status,
|
||||
Error: nsState.Error,
|
||||
LastChecked: nsState.LastChecked,
|
||||
}
|
||||
}
|
||||
@@ -183,6 +186,7 @@ func buildCertificates(
|
||||
NotAfter: cs.NotAfter,
|
||||
SubjectAlternativeNames: sans,
|
||||
Status: cs.Status,
|
||||
Error: cs.Error,
|
||||
LastChecked: cs.LastChecked,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"go.uber.org/fx/fxtest"
|
||||
|
||||
"sneak.berlin/go/dnswatcher/internal/config"
|
||||
"sneak.berlin/go/dnswatcher/internal/globals"
|
||||
"sneak.berlin/go/dnswatcher/internal/handlers"
|
||||
"sneak.berlin/go/dnswatcher/internal/logger"
|
||||
"sneak.berlin/go/dnswatcher/internal/notify"
|
||||
"sneak.berlin/go/dnswatcher/internal/state"
|
||||
)
|
||||
|
||||
// The state the handler tests serve: www.example.com has one nameserver
|
||||
// that answered and one whose query failed, and its certificate check
|
||||
// failed.
|
||||
const (
|
||||
testHostname = "www.example.com"
|
||||
answeringNS = "ns1.example.com."
|
||||
failedNS = "ns2.example.com."
|
||||
nsFailureReason = "server returned a referral"
|
||||
certKey = "192.0.2.1:443:www.example.com"
|
||||
certFailedReason = "x509: certificate has expired or is not yet valid"
|
||||
)
|
||||
|
||||
// newHandlersWithFailures builds real Handlers whose state holds the
|
||||
// entries described above.
|
||||
func newHandlersWithFailures(t *testing.T) *handlers.Handlers {
|
||||
t.Helper()
|
||||
|
||||
glob, err := globals.New(nil)
|
||||
if err != nil {
|
||||
t.Fatalf("globals.New: %v", err)
|
||||
}
|
||||
|
||||
log, err := logger.New(nil, logger.Params{Globals: glob})
|
||||
if err != nil {
|
||||
t.Fatalf("logger.New: %v", err)
|
||||
}
|
||||
|
||||
notifier, err := notify.New(fxtest.NewLifecycle(t), notify.Params{
|
||||
Logger: log,
|
||||
Config: &config.Config{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("notify.New: %v", err)
|
||||
}
|
||||
|
||||
st, err := state.New(fxtest.NewLifecycle(t), state.Params{
|
||||
Logger: log,
|
||||
Config: &config.Config{DataDir: t.TempDir()},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("state.New: %v", err)
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
|
||||
st.SetHostnameState(testHostname, &state.HostnameState{
|
||||
RecordsByNameserver: map[string]*state.NameserverRecordState{
|
||||
answeringNS: {
|
||||
Records: map[string][]string{"A": {"192.0.2.1"}},
|
||||
Status: "ok",
|
||||
LastChecked: now,
|
||||
},
|
||||
failedNS: {
|
||||
Records: map[string][]string{},
|
||||
Status: "error",
|
||||
Error: nsFailureReason,
|
||||
LastChecked: now,
|
||||
},
|
||||
},
|
||||
LastChecked: now,
|
||||
})
|
||||
|
||||
st.SetCertificateState(certKey, &state.CertificateState{
|
||||
Status: "error",
|
||||
Error: certFailedReason,
|
||||
LastChecked: now,
|
||||
})
|
||||
|
||||
hnd, err := handlers.New(nil, handlers.Params{
|
||||
Logger: log,
|
||||
Globals: glob,
|
||||
State: st,
|
||||
Notify: notifier,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("handlers.New: %v", err)
|
||||
}
|
||||
|
||||
return hnd
|
||||
}
|
||||
|
||||
// get serves one GET request to handler and returns the response body.
|
||||
func get(t *testing.T, handler http.HandlerFunc) string {
|
||||
t.Helper()
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, "/", nil,
|
||||
)
|
||||
|
||||
handler(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", rec.Code)
|
||||
}
|
||||
|
||||
return rec.Body.String()
|
||||
}
|
||||
|
||||
// TestStatusGivesFailureReasons checks that /api/v1/status gives the
|
||||
// reason for a failed nameserver entry and a failed certificate entry,
|
||||
// and no error for a nameserver that answered.
|
||||
func TestStatusGivesFailureReasons(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
body := get(t, newHandlersWithFailures(t).HandleStatus())
|
||||
|
||||
var resp struct {
|
||||
Hostnames map[string]struct {
|
||||
Nameservers map[string]map[string]any `json:"nameservers"`
|
||||
} `json:"hostnames"`
|
||||
Certificates map[string]map[string]any `json:"certificates"`
|
||||
}
|
||||
|
||||
err := json.Unmarshal([]byte(body), &resp)
|
||||
if err != nil {
|
||||
t.Fatalf("decoding response: %v", err)
|
||||
}
|
||||
|
||||
nameservers := resp.Hostnames[testHostname].Nameservers
|
||||
|
||||
got := nameservers[failedNS]["error"]
|
||||
if got != nsFailureReason {
|
||||
t.Errorf("failed nameserver error = %v, want %q",
|
||||
got, nsFailureReason)
|
||||
}
|
||||
|
||||
_, has := nameservers[answeringNS]["error"]
|
||||
if has {
|
||||
t.Errorf("answering nameserver has an error field: %v",
|
||||
nameservers[answeringNS])
|
||||
}
|
||||
|
||||
got = resp.Certificates[certKey]["error"]
|
||||
if got != certFailedReason {
|
||||
t.Errorf("failed certificate error = %v, want %q",
|
||||
got, certFailedReason)
|
||||
}
|
||||
}
|
||||
@@ -146,7 +146,11 @@
|
||||
<td
|
||||
class="py-2 px-3 text-slate-400 break-all max-w-xs"
|
||||
>
|
||||
{{ if $nsr.Error }}
|
||||
<span class="text-red-400">{{ $nsr.Error }}</span>
|
||||
{{ else }}
|
||||
{{ formatRecords $nsr.Records }}
|
||||
{{ end }}
|
||||
</td>
|
||||
<td class="py-2 px-3 text-slate-500 whitespace-nowrap">
|
||||
{{ relTime $nsr.LastChecked }}
|
||||
@@ -258,6 +262,11 @@
|
||||
>
|
||||
{{ end }}
|
||||
</td>
|
||||
{{ if $cs.Error }}
|
||||
<td colspan="3" class="py-2 px-3 text-red-400 break-all">
|
||||
<div class="max-w-xs">{{ $cs.Error }}</div>
|
||||
</td>
|
||||
{{ else }}
|
||||
<td class="py-2 px-3 text-slate-200">
|
||||
{{ $cs.CommonName }}
|
||||
</td>
|
||||
@@ -285,6 +294,7 @@
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
</td>
|
||||
{{ end }}
|
||||
<td class="py-2 px-3 text-slate-500 whitespace-nowrap">
|
||||
{{ relTime $cs.LastChecked }}
|
||||
</td>
|
||||
|
||||
@@ -6,9 +6,11 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"maps"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -413,7 +415,8 @@ func sendSlackInfo(
|
||||
svc *notify.Service, target *url.URL,
|
||||
) error {
|
||||
return svc.SendSlack(
|
||||
context.Background(), target, "t", "m", prioInfo,
|
||||
context.Background(), target, notify.ErrSlackFailed,
|
||||
"t", "m", prioInfo,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -506,6 +509,7 @@ func TestSendSlackPayloadFields(t *testing.T) {
|
||||
err := svc.SendSlack(
|
||||
context.Background(),
|
||||
webhookURL,
|
||||
notify.ErrSlackFailed,
|
||||
"Alert Title",
|
||||
"Alert body text",
|
||||
"warning",
|
||||
@@ -608,7 +612,8 @@ func TestSendSlackAllColors(t *testing.T) {
|
||||
|
||||
err := svc.SendSlack(
|
||||
context.Background(),
|
||||
webhookURL, "t", "m", tc.priority,
|
||||
webhookURL, notify.ErrSlackFailed,
|
||||
"t", "m", tc.priority,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("SendSlack error: %v", err)
|
||||
@@ -659,7 +664,8 @@ func TestSendSlackNetworkError(t *testing.T) {
|
||||
)
|
||||
|
||||
err := svc.SendSlack(
|
||||
context.Background(), webhookURL, "t", "m", "info",
|
||||
context.Background(), webhookURL, notify.ErrSlackFailed,
|
||||
"t", "m", "info",
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("expected error for network failure")
|
||||
@@ -1028,6 +1034,66 @@ func TestSendNotificationMattermostError(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
// TestSendNotificationErrorNamesEndpoint verifies that, with both
|
||||
// Slack and Mattermost set, a failed delivery's logged error names
|
||||
// the endpoint that failed. Both are sent by the Slack sender.
|
||||
func TestSendNotificationErrorNamesEndpoint(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
srv := httptest.NewServer(
|
||||
http.HandlerFunc(
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusServiceUnavailable)
|
||||
}),
|
||||
)
|
||||
defer srv.Close()
|
||||
|
||||
target, _ := url.Parse(srv.URL)
|
||||
|
||||
svc, logs := newLoggingService(http.DefaultTransport)
|
||||
svc.SetSlackWebhookURL(target)
|
||||
svc.SetMattermostWebhookURL(target)
|
||||
svc.SetSleepFunc(instantSleep)
|
||||
svc.SetRetryConfig(notify.RetryConfig{
|
||||
MaxRetries: 1,
|
||||
BaseDelay: time.Millisecond,
|
||||
MaxDelay: time.Millisecond,
|
||||
})
|
||||
|
||||
svc.SendNotification(
|
||||
context.Background(), "t", "m", prioError,
|
||||
)
|
||||
|
||||
waitForCondition(t, func() bool {
|
||||
return svc.OutstandingDeliveries() == 0
|
||||
})
|
||||
|
||||
got := map[string]string{}
|
||||
|
||||
for line := range strings.Lines(logs.String()) {
|
||||
var record struct {
|
||||
Msg string `json:"msg"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
|
||||
_ = json.Unmarshal([]byte(line), &record)
|
||||
|
||||
if record.Msg == "failed to send notification after retries" {
|
||||
got[record.Endpoint] = record.Error
|
||||
}
|
||||
}
|
||||
|
||||
want := map[string]string{
|
||||
"slack": "slack notification failed: status 503",
|
||||
"mattermost": "mattermost notification failed: status 503",
|
||||
}
|
||||
|
||||
if !maps.Equal(got, want) {
|
||||
t.Errorf("logged errors = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SlackPayload JSON marshaling ──────────────────────────
|
||||
|
||||
func TestSlackPayloadJSON(t *testing.T) {
|
||||
|
||||
@@ -85,10 +85,11 @@ func (svc *Service) SendNtfy(
|
||||
func (svc *Service) SendSlack(
|
||||
ctx context.Context,
|
||||
webhookURL *url.URL,
|
||||
failed error,
|
||||
title, message, priority string,
|
||||
) error {
|
||||
return svc.sendSlack(
|
||||
ctx, webhookURL, title, message, priority,
|
||||
ctx, webhookURL, failed, title, message, priority,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -277,7 +277,8 @@ func (svc *Service) dispatchSlack(
|
||||
|
||||
svc.dispatch(ctx, "slack", func(c context.Context) error {
|
||||
return svc.sendSlack(
|
||||
c, svc.slackWebhookURL, title, message, priority,
|
||||
c, svc.slackWebhookURL, ErrSlackFailed,
|
||||
title, message, priority,
|
||||
)
|
||||
})
|
||||
}
|
||||
@@ -294,7 +295,7 @@ func (svc *Service) dispatchMattermost(
|
||||
ctx, "mattermost",
|
||||
func(c context.Context) error {
|
||||
return svc.sendSlack(
|
||||
c, svc.mattermostWebhookURL,
|
||||
c, svc.mattermostWebhookURL, ErrMattermostFailed,
|
||||
title, message, priority,
|
||||
)
|
||||
},
|
||||
@@ -370,9 +371,14 @@ type SlackAttachment struct {
|
||||
Text string `json:"text"`
|
||||
}
|
||||
|
||||
// sendSlack posts to a Slack or Mattermost incoming webhook, which
|
||||
// take the same payload. An HTTP error status is returned wrapped in
|
||||
// failed, ErrSlackFailed or ErrMattermostFailed, so the error names
|
||||
// the endpoint.
|
||||
func (svc *Service) sendSlack(
|
||||
ctx context.Context,
|
||||
webhookURL *url.URL,
|
||||
failed error,
|
||||
title, message, priority string,
|
||||
) error {
|
||||
ctx, cancel := context.WithTimeout(
|
||||
@@ -420,7 +426,7 @@ func (svc *Service) sendSlack(
|
||||
if resp.StatusCode >= httpStatusClientError {
|
||||
return fmt.Errorf(
|
||||
"%w: status %d",
|
||||
ErrSlackFailed, resp.StatusCode,
|
||||
failed, resp.StatusCode,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -115,7 +115,9 @@ func (svc *Service) deliverWithRetry(
|
||||
"endpoint", endpoint,
|
||||
"attempt", attempt+1,
|
||||
"maxAttempts", cfg.MaxRetries+1,
|
||||
"retryIn", delay,
|
||||
// As text: the JSON log writes a time.Duration as
|
||||
// bare nanoseconds.
|
||||
"retryIn", delay.String(),
|
||||
"error", lastErr,
|
||||
)
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ package notify_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@@ -189,6 +190,50 @@ func TestDeliverWithRetryExhaustsAttempts(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestDeliverWithRetryLogsRetryInAsText checks that the wait
|
||||
// before a retry is logged as text such as "1.02s", not as a
|
||||
// count of nanoseconds.
|
||||
func TestDeliverWithRetryLogsRetryInAsText(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, logs := newLoggingService(http.DefaultTransport)
|
||||
svc.SetRetryConfig(notify.RetryConfig{
|
||||
MaxRetries: 1,
|
||||
BaseDelay: time.Second,
|
||||
MaxDelay: time.Second,
|
||||
})
|
||||
|
||||
var waited time.Duration
|
||||
|
||||
svc.SetSleepFunc(func(d time.Duration) <-chan time.Time {
|
||||
waited = d
|
||||
|
||||
return instantSleep(d)
|
||||
})
|
||||
|
||||
_ = svc.DeliverWithRetry(
|
||||
context.Background(), "test",
|
||||
func(_ context.Context) error {
|
||||
return errFail
|
||||
},
|
||||
)
|
||||
|
||||
// With one retry, only the first failure is logged.
|
||||
var record map[string]any
|
||||
|
||||
err := json.Unmarshal([]byte(logs.String()), &record)
|
||||
if err != nil {
|
||||
t.Fatalf("log is not one JSON record: %v\n%s", err, logs)
|
||||
}
|
||||
|
||||
if record["retryIn"] != waited.String() {
|
||||
t.Errorf(
|
||||
"retryIn logged as %v, want %q",
|
||||
record["retryIn"], waited.String(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeliverWithRetryRespectsContextCancellation(
|
||||
t *testing.T,
|
||||
) {
|
||||
|
||||
@@ -193,7 +193,9 @@ func (c *Checker) checkConnection(
|
||||
c.log.Debug(
|
||||
"port check succeeded",
|
||||
"target", target,
|
||||
"latency", latency,
|
||||
// As text: the JSON log writes a time.Duration as bare
|
||||
// nanoseconds.
|
||||
"latency", latency.String(),
|
||||
)
|
||||
|
||||
return &PortResult{
|
||||
|
||||
@@ -11,6 +11,13 @@ func ExtractRecordValue(rr dns.RR) string {
|
||||
return extractRecordValue(rr)
|
||||
}
|
||||
|
||||
// CollectAnswerRecords exports collectAnswerRecords for testing.
|
||||
func CollectAnswerRecords(msg *dns.Msg, resp *NameserverResponse) {
|
||||
var state queryState
|
||||
|
||||
collectAnswerRecords(msg, resp, &state)
|
||||
}
|
||||
|
||||
// UsableReply exports usableReply for testing.
|
||||
func UsableReply(resp *dns.Msg, zone string, name string) bool {
|
||||
return usableReply(resp, zone, name)
|
||||
|
||||
@@ -25,8 +25,10 @@ const (
|
||||
// can meet a referral that names nameservers without their
|
||||
// addresses, which are then looked up in turn; without a limit,
|
||||
// delegations that point at each other would never end. Each level
|
||||
// multiplies the queries sent.
|
||||
maxLookupDepth = 2
|
||||
// multiplies the queries sent. pool.ntp.org needs three: the
|
||||
// address of its nameserver g.ntpns.org can need a.ntpns.org's,
|
||||
// which needs a bitnames.com nameserver's.
|
||||
maxLookupDepth = 3
|
||||
)
|
||||
|
||||
// ErrRefused is returned when a DNS server refuses a query.
|
||||
@@ -771,6 +773,10 @@ func (r *Resolver) querySingleType(
|
||||
collectAnswerRecords(msg, resp, state)
|
||||
}
|
||||
|
||||
// collectAnswerRecords adds the records in msg's answer to resp, each
|
||||
// value once per record type. For a name with a CNAME, a nameserver
|
||||
// answers a query of any type with that CNAME, so the same value comes
|
||||
// in the answer to every type asked for.
|
||||
func collectAnswerRecords(
|
||||
msg *dns.Msg,
|
||||
resp *NameserverResponse,
|
||||
@@ -783,9 +789,12 @@ func collectAnswerRecords(
|
||||
}
|
||||
|
||||
typeName := dns.TypeToString[rr.Header().Rrtype]
|
||||
resp.Records[typeName] = append(
|
||||
resp.Records[typeName], val,
|
||||
)
|
||||
if !slices.Contains(resp.Records[typeName], val) {
|
||||
resp.Records[typeName] = append(
|
||||
resp.Records[typeName], val,
|
||||
)
|
||||
}
|
||||
|
||||
state.hasRecords = true
|
||||
}
|
||||
}
|
||||
|
||||
@@ -238,6 +238,38 @@ func TestExtractRecordValue_LetterCase(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestCollectAnswerRecords_CNAMEOnce collects the answers a nameserver
|
||||
// gives for a name with a CNAME, one for each record type a check asks
|
||||
// for. Each answer holds the CNAME, which must be stored once.
|
||||
func TestCollectAnswerRecords_CNAMEOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cname := &dns.CNAME{
|
||||
Hdr: dns.RR_Header{
|
||||
Name: "git.eeqj.de.", Rrtype: dns.TypeCNAME, Class: dns.ClassINET,
|
||||
},
|
||||
Target: "fsn1app1.datavi.be.",
|
||||
}
|
||||
|
||||
resp := &resolver.NameserverResponse{Records: map[string][]string{}}
|
||||
|
||||
for _, qtype := range []uint16{
|
||||
dns.TypeA, dns.TypeAAAA, dns.TypeCNAME, dns.TypeMX,
|
||||
dns.TypeTXT, dns.TypeSRV, dns.TypeCAA, dns.TypeNS,
|
||||
} {
|
||||
msg := new(dns.Msg)
|
||||
msg.SetQuestion("git.eeqj.de.", qtype)
|
||||
msg.Answer = []dns.RR{cname}
|
||||
|
||||
resolver.CollectAnswerRecords(msg, resp)
|
||||
}
|
||||
|
||||
assert.Equal(t,
|
||||
map[string][]string{"CNAME": {"fsn1app1.datavi.be."}},
|
||||
resp.Records,
|
||||
)
|
||||
}
|
||||
|
||||
// TestShuffled shuffles the root servers with many seeds. Every order
|
||||
// must hold each root server once, so each is tried before a
|
||||
// resolution fails; each root server must come first for some seed, so
|
||||
|
||||
@@ -210,33 +210,44 @@ func TestQueryNameservers_GivenAddressesFail(t *testing.T) {
|
||||
assert.NotEmpty(t, resolver.NSSetFrom(resp, "pool.ntp.org."))
|
||||
}
|
||||
|
||||
// TestQueryNameservers_LookupDepth asks the servers of desec.io about
|
||||
// ns1.desec.io, giving no address and naming ns1.desec.io without one.
|
||||
// Below maxLookupDepth its address is looked up and it is asked. At the
|
||||
// limit it is not, so nothing can be asked: this is where lookups of
|
||||
// nameserver addresses stop when delegations point at each other.
|
||||
// TestQueryNameservers_LookupDepth asks the servers of g.ntpns.org, a
|
||||
// nameserver of pool.ntp.org, for its address, as looking that address
|
||||
// up does when anyns.pch.net, one of the servers of ntpns.org, gives the
|
||||
// referral to g.ntpns.org without addresses. Their addresses are looked
|
||||
// up (here only a.ntpns.org's), and that needs a bitnames.com
|
||||
// nameserver's address, as the org servers delegate ntpns.org without
|
||||
// addresses. From depth 1, where looking up g.ntpns.org's address
|
||||
// starts, that makes three lookups and the address is found. From one
|
||||
// below maxLookupDepth, the bitnames.com lookup would be past the limit,
|
||||
// so nothing can be asked.
|
||||
func TestQueryNameservers_LookupDepth(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
r := newTestResolver(t)
|
||||
withoutAddresses := []string{"ns1.desec.io."}
|
||||
withoutAddresses := []string{"a.ntpns.org."}
|
||||
|
||||
var resp *dns.Msg
|
||||
|
||||
livednstest.Retry(
|
||||
t,
|
||||
"QueryNameservers(ns1.desec.io without its address, ns1.desec.io)",
|
||||
"QueryNameservers(a.ntpns.org without its address, g.ntpns.org)",
|
||||
func(ctx context.Context) error {
|
||||
_, err := r.QueryNameservers(
|
||||
ctx, nil, withoutAddresses, "desec.io.", "ns1.desec.io.",
|
||||
dns.TypeA, 0,
|
||||
var err error
|
||||
|
||||
resp, err = r.QueryNameservers(
|
||||
ctx, nil, withoutAddresses, "g.ntpns.org.", "g.ntpns.org.",
|
||||
dns.TypeA, 1,
|
||||
)
|
||||
|
||||
return err
|
||||
},
|
||||
)
|
||||
|
||||
assert.NotEmpty(t, resp.Answer)
|
||||
|
||||
_, err := r.QueryNameservers(
|
||||
t.Context(), nil, withoutAddresses, "desec.io.", "ns1.desec.io.",
|
||||
dns.TypeA, resolver.MaxLookupDepth,
|
||||
t.Context(), nil, withoutAddresses, "g.ntpns.org.", "g.ntpns.org.",
|
||||
dns.TypeA, resolver.MaxLookupDepth-1,
|
||||
)
|
||||
require.ErrorIs(t, err, resolver.ErrNoNameservers)
|
||||
}
|
||||
@@ -760,6 +771,21 @@ func TestLookupNS_MatchesFindAuthoritative(t *testing.T) {
|
||||
assert.Equal(t, fromFind, fromLookup)
|
||||
}
|
||||
|
||||
// TestLookupNS_ParentZoneDelegatedWithoutAddresses looks up the
|
||||
// nameservers of g.ntpns.org. The org servers delegate its parent zone,
|
||||
// ntpns.org, without the addresses of its nameservers, so the walk has
|
||||
// to look them up to ask them. If it did not, the walk for g.ntpns.org
|
||||
// would fail and LookupNS would return the nameservers of ntpns.org,
|
||||
// which a.ntpns.org is not one of.
|
||||
func TestLookupNS_ParentZoneDelegatedWithoutAddresses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
r := newTestResolver(t)
|
||||
nameservers := liveLookupNS(t, r, "g.ntpns.org")
|
||||
|
||||
assert.Contains(t, nameservers, "a.ntpns.org.")
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
// ResolveIPAddresses tests
|
||||
// ----------------------------------------------------------------
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -53,8 +54,13 @@ type NameserverRecordState struct {
|
||||
}
|
||||
|
||||
// HostnameState holds per-nameserver monitoring state for a hostname.
|
||||
// CNAMEAddresses holds the sorted addresses at the end of the name's
|
||||
// CNAME chain, found when its nameservers answered with a CNAME and no
|
||||
// address; it is empty otherwise. It is nil when they are not known: a
|
||||
// state file written before it existed loads with it nil.
|
||||
type HostnameState struct {
|
||||
RecordsByNameserver map[string]*NameserverRecordState `json:"recordsByNameserver"`
|
||||
CNAMEAddresses []string `json:"cnameAddresses"`
|
||||
LastChecked time.Time `json:"lastChecked"`
|
||||
}
|
||||
|
||||
@@ -196,6 +202,19 @@ func (s *State) Load() error {
|
||||
return fmt.Errorf("parsing state file: %w", err)
|
||||
}
|
||||
|
||||
// A state file saved before each record value was stored once can
|
||||
// hold a hostname's CNAME once for every record type asked for.
|
||||
// Each value is kept once, so the first check does not see a
|
||||
// record change.
|
||||
for _, hs := range snapshot.Hostnames {
|
||||
for _, ns := range hs.RecordsByNameserver {
|
||||
for recordType, values := range ns.Records {
|
||||
slices.Sort(values)
|
||||
ns.Records[recordType] = slices.Compact(values)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
s.snapshot = &snapshot
|
||||
s.log.Info("loaded state from disk", "path", path)
|
||||
|
||||
|
||||
@@ -188,6 +188,148 @@ func TestLoadStateFromBeforeNameserverAddresses(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestSaveLoadRoundTrip_CNAMEAddresses checks that no addresses at the
|
||||
// end of a hostname's CNAME chain load as an empty list, and addresses
|
||||
// that are not known load as nil: the watcher tells the two apart.
|
||||
func TestSaveLoadRoundTrip_CNAMEAddresses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
s := state.NewForTestWithDataDir(dir)
|
||||
|
||||
want := map[string][]string{
|
||||
"cname.example.com": {testIP},
|
||||
"none.example.com": {},
|
||||
"not-known.example.com": nil,
|
||||
}
|
||||
|
||||
for name, addresses := range want {
|
||||
s.SetHostnameState(name, &state.HostnameState{
|
||||
CNAMEAddresses: addresses,
|
||||
})
|
||||
}
|
||||
|
||||
err := s.Save()
|
||||
if err != nil {
|
||||
t.Fatalf("Save() error: %v", err)
|
||||
}
|
||||
|
||||
loaded := state.NewForTestWithDataDir(dir)
|
||||
|
||||
err = loaded.Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error: %v", err)
|
||||
}
|
||||
|
||||
for name, addresses := range want {
|
||||
hs, ok := loaded.GetHostnameState(name)
|
||||
if !ok {
|
||||
t.Fatalf("missing hostname %s", name)
|
||||
}
|
||||
|
||||
if !reflect.DeepEqual(hs.CNAMEAddresses, addresses) {
|
||||
t.Errorf(
|
||||
"%s: loaded %#v, want %#v",
|
||||
name, hs.CNAMEAddresses, addresses,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoadStateFromBeforeCNAMEAddresses loads a state file written
|
||||
// before the addresses at the end of a hostname's CNAME chain were
|
||||
// saved. They load as not known (nil), not as none.
|
||||
func TestLoadStateFromBeforeCNAMEAddresses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
|
||||
data := []byte(`{
|
||||
"version": 1,
|
||||
"lastUpdated": "2026-02-19T12:00:00Z",
|
||||
"hostnames": {
|
||||
"www.example.com": {
|
||||
"recordsByNameserver": {},
|
||||
"lastChecked": "2026-02-19T12:00:00Z"
|
||||
}
|
||||
}
|
||||
}`)
|
||||
|
||||
err := os.WriteFile(filepath.Join(dir, "state.json"), data, 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("writing state file: %v", err)
|
||||
}
|
||||
|
||||
s := state.NewForTestWithDataDir(dir)
|
||||
|
||||
err = s.Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error: %v", err)
|
||||
}
|
||||
|
||||
hs, ok := s.GetHostnameState(testHostname)
|
||||
if !ok {
|
||||
t.Fatal("missing hostname " + testHostname)
|
||||
}
|
||||
|
||||
if hs.CNAMEAddresses != nil {
|
||||
t.Errorf("CNAME addresses: got %#v, want nil", hs.CNAMEAddresses)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoadStateWithRepeatedValues loads a state file saved when a
|
||||
// hostname's CNAME was stored once for every record type asked for.
|
||||
// Each value must load once, and every different value must load.
|
||||
func TestLoadStateWithRepeatedValues(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
|
||||
data := []byte(`{
|
||||
"version": 1,
|
||||
"hostnames": {
|
||||
"www.example.com": {
|
||||
"recordsByNameserver": {
|
||||
"ns1.example.com.": {
|
||||
"records": {
|
||||
"A": ["192.0.2.2", "192.0.2.1", "192.0.2.2", "192.0.2.1"],
|
||||
"CNAME": ["a.example.net.", "a.example.net.", "a.example.net."]
|
||||
},
|
||||
"status": "ok"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}`)
|
||||
|
||||
err := os.WriteFile(filepath.Join(dir, "state.json"), data, 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("writing state file: %v", err)
|
||||
}
|
||||
|
||||
s := state.NewForTestWithDataDir(dir)
|
||||
|
||||
err = s.Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error: %v", err)
|
||||
}
|
||||
|
||||
hs, ok := s.GetHostnameState(testHostname)
|
||||
if !ok {
|
||||
t.Fatal("missing hostname " + testHostname)
|
||||
}
|
||||
|
||||
want := map[string][]string{
|
||||
"A": {"192.0.2.1", "192.0.2.2"},
|
||||
"CNAME": {"a.example.net."},
|
||||
}
|
||||
|
||||
got := hs.RecordsByNameserver[testNS1].Records
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("records: got %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSaveLoadRoundTrip_Hostnames verifies hostname data survives a save/load cycle.
|
||||
func TestSaveLoadRoundTrip_Hostnames(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -1,10 +1,13 @@
|
||||
package watcher_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"log/slog"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/dnswatcher/internal/portcheck"
|
||||
"sneak.berlin/go/dnswatcher/internal/resolver"
|
||||
@@ -79,3 +82,72 @@ func TestCancelledCheckSavesNothing(t *testing.T) {
|
||||
t.Errorf("sent %v, want no notifications", notifications)
|
||||
}
|
||||
}
|
||||
|
||||
// newLoggingWatcher returns a watcher for a domain and a hostname, with
|
||||
// the real resolver, that writes what it logs at warning level or above
|
||||
// into the returned buffer.
|
||||
func newLoggingWatcher(t *testing.T) (*watcher.Watcher, *bytes.Buffer) {
|
||||
t.Helper()
|
||||
|
||||
cfg := defaultTestConfig(t)
|
||||
cfg.Domains = []string{testSmallDomain}
|
||||
cfg.Hostnames = []string{host}
|
||||
|
||||
w, _ := newTestWatcher(t, cfg)
|
||||
|
||||
logs := &bytes.Buffer{}
|
||||
w.SetLogger(slog.New(slog.NewJSONHandler(
|
||||
logs, &slog.HandlerOptions{Level: slog.LevelWarn},
|
||||
)))
|
||||
|
||||
return w, logs
|
||||
}
|
||||
|
||||
// TestLookupCutShortIsNotLogged checks a domain and a hostname, looks
|
||||
// up a nameserver's addresses and follows a CNAME, with the context
|
||||
// cancelled, as shutdown leaves it. The real resolver fails each lookup
|
||||
// without sending a query. Shutdown cutting a lookup short is not a
|
||||
// failure, so nothing may be logged at warning level or above.
|
||||
func TestLookupCutShortIsNotLogged(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
w, logs := newLoggingWatcher(t)
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
cancel()
|
||||
|
||||
w.RunOnce(ctx)
|
||||
w.ResolveNameserverAddresses(ctx, []string{nsA}, nil)
|
||||
w.ResolveCNAMEAddresses(ctx, host, cnameState(), nil)
|
||||
|
||||
if logs.Len() > 0 {
|
||||
t.Errorf("logged at warning level or above:\n%s", logs)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLookupOutOfTimeIsLoggedAsError does what
|
||||
// TestLookupCutShortIsNotLogged does, with the context's deadline passed
|
||||
// instead. A lookup that ran out of time did fail, so the domain's NS
|
||||
// lookup, the hostname's lookup, the nameserver's address lookup and the
|
||||
// CNAME's are each logged as an error.
|
||||
func TestLookupOutOfTimeIsLoggedAsError(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
w, logs := newLoggingWatcher(t)
|
||||
|
||||
ctx, cancel := context.WithDeadline(t.Context(), time.Now())
|
||||
t.Cleanup(cancel)
|
||||
|
||||
w.RunOnce(ctx)
|
||||
w.ResolveNameserverAddresses(ctx, []string{nsA}, nil)
|
||||
w.ResolveCNAMEAddresses(ctx, host, cnameState(), nil)
|
||||
|
||||
const want = 4
|
||||
|
||||
lines := strings.Count(logs.String(), "\n")
|
||||
errorLines := strings.Count(logs.String(), `"level":"ERROR"`)
|
||||
|
||||
if lines != want || errorLines != want {
|
||||
t.Errorf("logged:\n%s\nwant %d lines, each at error level", logs, want)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,336 @@
|
||||
package watcher_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/dnswatcher/internal/livednstest"
|
||||
"sneak.berlin/go/dnswatcher/internal/resolver"
|
||||
"sneak.berlin/go/dnswatcher/internal/state"
|
||||
"sneak.berlin/go/dnswatcher/internal/watcher"
|
||||
)
|
||||
|
||||
// TestCNAMEIntoAnotherZonePortAndTLSChecks runs the port and TLS
|
||||
// checks on hostname state built here: the name's nameserver answered
|
||||
// with a CNAME into another zone, and following it found ip1. Both
|
||||
// checks must use ip1. They look nothing up, so the watcher has no
|
||||
// resolver.
|
||||
func TestCNAMEIntoAnotherZonePortAndTLSChecks(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := defaultTestConfig(t)
|
||||
cfg.Hostnames = []string{host}
|
||||
|
||||
deps := newTestDeps(t, cfg)
|
||||
w := watcher.NewForTest(
|
||||
cfg, deps.state, nil,
|
||||
deps.portChecker, deps.tlsChecker, deps.notifier,
|
||||
)
|
||||
|
||||
deps.state.SetHostnameState(host, cnameState(ip1))
|
||||
|
||||
w.CheckAllPorts(t.Context())
|
||||
w.RunTLSChecks(t.Context())
|
||||
|
||||
snap := deps.state.GetSnapshot()
|
||||
|
||||
ps, ok := snap.Ports[ip1+":443"]
|
||||
if !ok || !slices.Contains(ps.Hostnames, host) {
|
||||
t.Errorf("no port state for %s at %s:443", host, ip1)
|
||||
}
|
||||
|
||||
certKey := ip1 + ":443:" + host
|
||||
if _, ok := snap.Certificates[certKey]; !ok {
|
||||
t.Errorf("no certificate state %s", certKey)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCNAMEThatCannotBeFollowedKeepsPrevious runs a check of a name, not
|
||||
// the watcher's first, from the point where its records have been looked
|
||||
// up: they hold a CNAME to a target under .invalid, whose lookup fails.
|
||||
// The previous check found the same records, and oldIP at the end of the
|
||||
// CNAME. The check must keep oldIP and send nothing.
|
||||
func TestCNAMEThatCannotBeFollowedKeepsPrevious(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
w, deps := newTestWatcher(t, defaultTestConfig(t))
|
||||
w.SetFirstRun(false)
|
||||
|
||||
records := map[string]map[string][]string{
|
||||
nsA: cnameTo("target.example.invalid."),
|
||||
}
|
||||
|
||||
prev := hostnameState(records)
|
||||
prev.CNAMEAddresses = []string{oldIP}
|
||||
deps.state.SetHostnameState(host, prev)
|
||||
|
||||
// The result is the same whether or not live DNS answers, so the
|
||||
// lookup is not retried.
|
||||
_ = livednstest.Run(func(ctx context.Context) error {
|
||||
w.UpdateHostnameState(ctx, host, hostnameState(records))
|
||||
|
||||
return nil
|
||||
})
|
||||
|
||||
hs, _ := deps.state.GetHostnameState(host)
|
||||
if !slices.Equal(hs.CNAMEAddresses, prev.CNAMEAddresses) {
|
||||
t.Errorf(
|
||||
"saved %v, want %v",
|
||||
hs.CNAMEAddresses, prev.CNAMEAddresses,
|
||||
)
|
||||
}
|
||||
|
||||
notifications := deps.notifier.getNotifications()
|
||||
if len(notifications) != 0 {
|
||||
t.Errorf("sent %v, want no notifications", notifications)
|
||||
}
|
||||
}
|
||||
|
||||
// followLive follows in live DNS the CNAMEs in a name's records, built
|
||||
// from records, and returns the addresses saved for the name. The
|
||||
// previous check saved oldIP, which is kept when a target cannot be
|
||||
// followed; that is retried. The tests point CNAMEs only at names in
|
||||
// zones with two nameservers, to keep queries few (see the top of
|
||||
// watcher_test.go).
|
||||
func followLive(
|
||||
t *testing.T,
|
||||
records map[string]map[string][]string,
|
||||
) []string {
|
||||
t.Helper()
|
||||
|
||||
w := watcher.NewForTest(
|
||||
nil, nil, resolver.NewFromLogger(slog.Default()), nil, nil, nil,
|
||||
)
|
||||
prev := cnameState(oldIP)
|
||||
|
||||
var current *state.HostnameState
|
||||
|
||||
livednstest.Retry(t, "following CNAMEs", func(ctx context.Context) error {
|
||||
current = hostnameState(records)
|
||||
|
||||
w.ResolveCNAMEAddresses(ctx, host, current, prev)
|
||||
|
||||
if slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
|
||||
return livednstest.ErrNoAnswer
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
|
||||
return current.CNAMEAddresses
|
||||
}
|
||||
|
||||
// TestCNAMEAddressesOfEveryTarget gives a name's two nameservers
|
||||
// different CNAME targets, as when a secondary still serves an old one.
|
||||
// The addresses at the end of both are saved, whichever answer is read
|
||||
// first: one.one.one.one has 1.1.1.1, and dns.adguard-dns.com has
|
||||
// 94.140.14.14.
|
||||
func TestCNAMEAddressesOfEveryTarget(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
found := followLive(t, map[string]map[string][]string{
|
||||
nsA: cnameTo("one.one.one.one."),
|
||||
nsB: cnameTo("dns.adguard-dns.com."),
|
||||
})
|
||||
|
||||
for _, ip := range []string{"1.1.1.1", "94.140.14.14"} {
|
||||
if !slices.Contains(found, ip) {
|
||||
t.Errorf("saved %v, want %s among them", found, ip)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCNAMEChainEndingInNoAddressSavesEmptyList follows a CNAME to a
|
||||
// name live DNS answers with NXDOMAIN. An empty list is saved, not nil,
|
||||
// which would mean the addresses are not known.
|
||||
func TestCNAMEChainEndingInNoAddressSavesEmptyList(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
found := followLive(t, map[string]map[string][]string{
|
||||
nsA: cnameTo("this-surely-does-not-exist-xyz.example.org."),
|
||||
})
|
||||
|
||||
if found == nil || len(found) != 0 {
|
||||
t.Errorf("saved %#v, want an empty list", found)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCNAMEBesideAnAddressNotFollowed gives one nameserver of a name an
|
||||
// address and another a CNAME. The CNAME is not followed: an empty list
|
||||
// is saved, not nil, and nothing is looked up, the watcher having no
|
||||
// resolver.
|
||||
func TestCNAMEBesideAnAddressNotFollowed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
|
||||
|
||||
current := hostnameState(map[string]map[string][]string{
|
||||
nsA: {"A": {ip1}},
|
||||
nsB: cnameTo("target.example.org."),
|
||||
})
|
||||
|
||||
w.ResolveCNAMEAddresses(t.Context(), host, current, nil)
|
||||
|
||||
if current.CNAMEAddresses == nil || len(current.CNAMEAddresses) != 0 {
|
||||
t.Errorf("saved %#v, want an empty list", current.CNAMEAddresses)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCNAMEWhoseNameserversAllFailedKeepsPrevious checks a name none of
|
||||
// whose nameservers answered. The addresses the previous check saved
|
||||
// from following its CNAME are kept, and nothing is looked up: the
|
||||
// watcher has no resolver.
|
||||
func TestCNAMEWhoseNameserversAllFailedKeepsPrevious(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
w := watcher.NewForTest(nil, nil, nil, nil, nil, nil)
|
||||
|
||||
current := saved(map[string]*state.NameserverRecordState{
|
||||
nsA: failed(), nsB: failed(),
|
||||
})
|
||||
prev := cnameState(oldIP)
|
||||
|
||||
w.ResolveCNAMEAddresses(t.Context(), host, current, prev)
|
||||
|
||||
if !slices.Equal(current.CNAMEAddresses, prev.CNAMEAddresses) {
|
||||
t.Errorf(
|
||||
"saved %v, want %v",
|
||||
current.CNAMEAddresses, prev.CNAMEAddresses,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// cnameTo builds the records of a nameserver that answered with a CNAME
|
||||
// to target and no address.
|
||||
func cnameTo(target string) map[string][]string {
|
||||
return map[string][]string{"CNAME": {target}}
|
||||
}
|
||||
|
||||
// cnameState builds the state a check leaves behind for a name whose
|
||||
// nameserver answered with a CNAME and no address, when following the
|
||||
// CNAME found these addresses, which may be none.
|
||||
func cnameState(addresses ...string) *state.HostnameState {
|
||||
hs := hostnameState(map[string]map[string][]string{
|
||||
nsA: cnameTo("target.example.org."),
|
||||
})
|
||||
|
||||
hs.CNAMEAddresses = append([]string{}, addresses...)
|
||||
|
||||
return hs
|
||||
}
|
||||
|
||||
func TestCNAMEAddressChangeAlerts(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// A state file written before the addresses were saved loads with
|
||||
// them nil.
|
||||
olderStateFile := cnameState()
|
||||
olderStateFile.CNAMEAddresses = nil
|
||||
|
||||
// Each case is the state saved by the previous check and by the
|
||||
// current one. The name's records are the same in both.
|
||||
tests := []struct {
|
||||
name string
|
||||
prev, current *state.HostnameState
|
||||
want int
|
||||
}{
|
||||
{
|
||||
"same addresses",
|
||||
cnameState(ip1, ip2), cnameState(ip1, ip2), 0,
|
||||
},
|
||||
{
|
||||
"same addresses in another order",
|
||||
cnameState(ip2, ip1), cnameState(ip1, ip2), 0,
|
||||
},
|
||||
{
|
||||
"address replaced",
|
||||
cnameState(ip1), cnameState(ip2), 1,
|
||||
},
|
||||
{
|
||||
"address added",
|
||||
cnameState(ip1), cnameState(ip1, ip2), 1,
|
||||
},
|
||||
{
|
||||
"no address at the end of the chain now",
|
||||
cnameState(ip1), cnameState(), 1,
|
||||
},
|
||||
{
|
||||
"addresses at the end of the chain again",
|
||||
cnameState(), cnameState(ip1), 1,
|
||||
},
|
||||
{
|
||||
"state file from before addresses were saved",
|
||||
olderStateFile, cnameState(ip1), 0,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
notifier := &mockNotifier{}
|
||||
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
|
||||
|
||||
w.DetectHostnameChanges(t.Context(), host, tt.prev, tt.current)
|
||||
|
||||
got := len(notifier.getNotifications())
|
||||
if got != tt.want {
|
||||
t.Errorf("sent %d notifications, want %d", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCNAMEAddressChangeAlertNamesHostnameAndAddresses(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
notifier := &mockNotifier{}
|
||||
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
|
||||
|
||||
w.DetectHostnameChanges(
|
||||
t.Context(), host, cnameState(ip1), cnameState(ip2, ip3),
|
||||
)
|
||||
|
||||
want := notification{
|
||||
Title: "CNAME Address Change: " + host,
|
||||
Message: "Hostname: " + host +
|
||||
"\nOld: " + ip1 + "\nNew: " + ip2 + ", " + ip3,
|
||||
Priority: "warning",
|
||||
}
|
||||
|
||||
got := notifier.getNotifications()
|
||||
if len(got) != 1 || got[0] != want {
|
||||
t.Errorf("sent %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNameMovedFromARecordsToCNAMEAlerts checks a name that answers
|
||||
// with an A record and then with a CNAME whose chain ends in ip2. The
|
||||
// second check is notified as a CNAME address change from no addresses,
|
||||
// beside the record change. Nothing is looked up: the watcher has no
|
||||
// resolver.
|
||||
func TestNameMovedFromARecordsToCNAMEAlerts(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
notifier := &mockNotifier{}
|
||||
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
|
||||
|
||||
prev := hostnameState(map[string]map[string][]string{
|
||||
nsA: {"A": {ip1}},
|
||||
})
|
||||
w.ResolveCNAMEAddresses(t.Context(), host, prev, nil)
|
||||
|
||||
w.DetectHostnameChanges(t.Context(), host, prev, cnameState(ip2))
|
||||
|
||||
title := "CNAME Address Change: " + host
|
||||
message := "Hostname: " + host + "\nOld: \nNew: " + ip2
|
||||
|
||||
got := notifier.getNotifications()
|
||||
if !slices.ContainsFunc(got, func(n notification) bool {
|
||||
return n.Title == title && n.Message == message
|
||||
}) {
|
||||
t.Errorf("sent %v, want %q with %q among them", got, title, message)
|
||||
}
|
||||
}
|
||||
@@ -31,6 +31,12 @@ func NewForTest(
|
||||
}
|
||||
}
|
||||
|
||||
// SetLogger replaces the watcher's logger, so a test can read what it
|
||||
// logs.
|
||||
func (w *Watcher) SetLogger(log *slog.Logger) {
|
||||
w.log = log
|
||||
}
|
||||
|
||||
// NewlyDisagreeingPairs exports newlyDisagreeingPairs for testing.
|
||||
func NewlyDisagreeingPairs(
|
||||
prev, current *state.HostnameState,
|
||||
@@ -38,6 +44,21 @@ func NewlyDisagreeingPairs(
|
||||
return newlyDisagreeingPairs(prev, current)
|
||||
}
|
||||
|
||||
// SetFirstRun sets whether the watcher is on its first check, in which
|
||||
// nothing is compared with the previous check. NewForTest's watcher is.
|
||||
func (w *Watcher) SetFirstRun(firstRun bool) {
|
||||
w.firstRun = firstRun
|
||||
}
|
||||
|
||||
// UpdateHostnameState exports updateHostnameState for testing.
|
||||
func (w *Watcher) UpdateHostnameState(
|
||||
ctx context.Context,
|
||||
hostname string,
|
||||
newState *state.HostnameState,
|
||||
) {
|
||||
w.updateHostnameState(ctx, hostname, newState)
|
||||
}
|
||||
|
||||
// DetectHostnameChanges exports detectHostnameChanges for testing.
|
||||
func (w *Watcher) DetectHostnameChanges(
|
||||
ctx context.Context,
|
||||
@@ -57,6 +78,15 @@ func (w *Watcher) ResolveNameserverAddresses(
|
||||
return w.resolveNameserverAddresses(ctx, nameservers, prev)
|
||||
}
|
||||
|
||||
// ResolveCNAMEAddresses exports resolveCNAMEAddresses for testing.
|
||||
func (w *Watcher) ResolveCNAMEAddresses(
|
||||
ctx context.Context,
|
||||
hostname string,
|
||||
current, prev *state.HostnameState,
|
||||
) {
|
||||
w.resolveCNAMEAddresses(ctx, hostname, current, prev)
|
||||
}
|
||||
|
||||
// DetectNSAddressChanges exports detectNSAddressChanges for testing.
|
||||
func (w *Watcher) DetectNSAddressChanges(
|
||||
ctx context.Context,
|
||||
|
||||
@@ -183,3 +183,58 @@ func TestInconsistencyAlert(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestFirstCheckAfterRepeatedValuesLoaded saves a state file holding a
|
||||
// hostname's CNAME once for every record type asked for, as checks did
|
||||
// before each value was stored once, and two addresses each repeated,
|
||||
// and loads it. A check that then finds each value once at each
|
||||
// nameserver must notify nothing.
|
||||
func TestFirstCheckAfterRepeatedValuesLoaded(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
cnameType = "CNAME"
|
||||
cname = "c.example.net."
|
||||
)
|
||||
|
||||
cfg := defaultTestConfig(t)
|
||||
repeated := map[string][]string{
|
||||
"A": {ip2, ip1, ip2, ip1},
|
||||
cnameType: {cname, cname, cname, cname, cname, cname, cname, cname},
|
||||
}
|
||||
once := map[string][]string{"A": {ip1, ip2}, cnameType: {cname}}
|
||||
|
||||
saved := newTestDeps(t, cfg).state
|
||||
saved.SetHostnameState(host, hostnameState(map[string]map[string][]string{
|
||||
nsA: repeated, nsB: repeated,
|
||||
}))
|
||||
|
||||
err := saved.Save()
|
||||
if err != nil {
|
||||
t.Fatalf("saving the state file: %v", err)
|
||||
}
|
||||
|
||||
deps := newTestDeps(t, cfg)
|
||||
|
||||
err = deps.state.Load()
|
||||
if err != nil {
|
||||
t.Fatalf("loading the state file: %v", err)
|
||||
}
|
||||
|
||||
prev, ok := deps.state.GetHostnameState(host)
|
||||
if !ok {
|
||||
t.Fatal("the state file has no state for " + host)
|
||||
}
|
||||
|
||||
current := hostnameState(map[string]map[string][]string{
|
||||
nsA: once, nsB: once,
|
||||
})
|
||||
|
||||
// The hostname change detection uses only the notifier.
|
||||
w := watcher.NewForTest(nil, nil, nil, nil, nil, deps.notifier)
|
||||
w.DetectHostnameChanges(t.Context(), host, prev, current)
|
||||
|
||||
if got := deps.notifier.getNotifications(); len(got) != 0 {
|
||||
t.Errorf("sent %v, want no notification", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
package watcher_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/dnswatcher/internal/watcher"
|
||||
)
|
||||
|
||||
// When one nameserver's A record changes and its TXT record does not,
|
||||
// the record change and the inconsistency it starts name the A record
|
||||
// alone, with its values written as plain text.
|
||||
func TestChangeMessagesNameTheChangedType(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// A nameserver's records: this A address and the same TXT record.
|
||||
records := func(address string) map[string][]string {
|
||||
return map[string][]string{
|
||||
"A": {address},
|
||||
"TXT": {"v=spf1 -all"},
|
||||
}
|
||||
}
|
||||
|
||||
before := hostnameState(map[string]map[string][]string{
|
||||
nsA: records(ip1),
|
||||
nsB: records(ip1),
|
||||
})
|
||||
after := hostnameState(map[string]map[string][]string{
|
||||
nsA: records(ip1),
|
||||
nsB: records(ip2),
|
||||
})
|
||||
|
||||
// The hostname change detection uses only the notifier.
|
||||
notifier := &mockNotifier{}
|
||||
w := watcher.NewForTest(nil, nil, nil, nil, nil, notifier)
|
||||
|
||||
w.DetectHostnameChanges(t.Context(), host, before, after)
|
||||
|
||||
want := map[string]string{
|
||||
"Record Change: " + host: `Hostname: www.example.net
|
||||
Nameserver: b.ns.example.net.
|
||||
Type: A
|
||||
Old: 192.0.2.1
|
||||
New: 192.0.2.2`,
|
||||
"Inconsistency: " + host: `Hostname: www.example.net
|
||||
Type: A
|
||||
a.ns.example.net.: 192.0.2.1
|
||||
b.ns.example.net.: 192.0.2.2`,
|
||||
}
|
||||
|
||||
notifications := notifier.getNotifications()
|
||||
if len(notifications) != len(want) {
|
||||
t.Fatalf(
|
||||
"sent %d notifications, want %d: %v",
|
||||
len(notifications), len(want), notifications,
|
||||
)
|
||||
}
|
||||
|
||||
for _, n := range notifications {
|
||||
if n.Message != want[n.Title] {
|
||||
t.Errorf(
|
||||
"%s message:\n%s\nwant:\n%s",
|
||||
n.Title, n.Message, want[n.Title],
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
+216
-37
@@ -2,6 +2,7 @@ package watcher
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"slices"
|
||||
@@ -123,8 +124,10 @@ func (w *Watcher) Run(ctx context.Context) {
|
||||
"watcher starting",
|
||||
"domains", len(w.config.Domains),
|
||||
"hostnames", len(w.config.Hostnames),
|
||||
"dnsInterval", w.config.DNSInterval,
|
||||
"tlsInterval", w.config.TLSInterval,
|
||||
// As text: the JSON log writes a time.Duration as bare
|
||||
// nanoseconds.
|
||||
"dnsInterval", w.config.DNSInterval.String(),
|
||||
"tlsInterval", w.config.TLSInterval.String(),
|
||||
)
|
||||
|
||||
w.RunOnce(ctx)
|
||||
@@ -211,13 +214,29 @@ func (w *Watcher) runDNSChecks(ctx context.Context) {
|
||||
}
|
||||
}
|
||||
|
||||
// logFailedLookup logs a failed DNS lookup at error level, unless ctx
|
||||
// was cancelled: shutdown cancels it, and a lookup it cut short did not
|
||||
// fail. A lookup that ran out of time did fail, so it is logged.
|
||||
func (w *Watcher) logFailedLookup(
|
||||
ctx context.Context,
|
||||
msg string,
|
||||
args ...any,
|
||||
) {
|
||||
if errors.Is(ctx.Err(), context.Canceled) {
|
||||
return
|
||||
}
|
||||
|
||||
w.log.Error(msg, args...)
|
||||
}
|
||||
|
||||
func (w *Watcher) checkDomain(
|
||||
ctx context.Context,
|
||||
domain string,
|
||||
) {
|
||||
nameservers, err := w.resolver.LookupNS(ctx, domain)
|
||||
if err != nil {
|
||||
w.log.Error(
|
||||
w.logFailedLookup(
|
||||
ctx,
|
||||
"failed to lookup NS",
|
||||
"domain", domain,
|
||||
"error", err,
|
||||
@@ -252,28 +271,9 @@ func (w *Watcher) checkDomain(
|
||||
LastChecked: now,
|
||||
})
|
||||
|
||||
// Also look up A/AAAA records for the apex domain so that
|
||||
// port and TLS checks (which read HostnameState) can find
|
||||
// the domain's IP addresses.
|
||||
results, err := w.resolver.LookupAllRecords(ctx, domain)
|
||||
if err != nil {
|
||||
w.log.Error(
|
||||
"failed to lookup records for domain",
|
||||
"domain", domain,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
newState := buildHostnameState(results, now)
|
||||
|
||||
prevHS, hasPrevHS := w.state.GetHostnameState(domain)
|
||||
if hasPrevHS && !w.firstRun {
|
||||
w.detectHostnameChanges(ctx, domain, prevHS, newState)
|
||||
}
|
||||
|
||||
w.state.SetHostnameState(domain, newState)
|
||||
// The apex domain's records are also checked as a hostname's, so
|
||||
// that the port and TLS checks find its addresses.
|
||||
w.checkHostname(ctx, domain)
|
||||
}
|
||||
|
||||
func (w *Watcher) detectNSChanges(
|
||||
@@ -337,7 +337,8 @@ func (w *Watcher) resolveNameserverAddresses(
|
||||
continue
|
||||
}
|
||||
|
||||
w.log.Error(
|
||||
w.logFailedLookup(
|
||||
ctx,
|
||||
"no addresses found for nameserver",
|
||||
"nameserver", ns,
|
||||
"error", err,
|
||||
@@ -389,7 +390,8 @@ func (w *Watcher) checkHostname(
|
||||
) {
|
||||
results, err := w.resolver.LookupAllRecords(ctx, hostname)
|
||||
if err != nil {
|
||||
w.log.Error(
|
||||
w.logFailedLookup(
|
||||
ctx,
|
||||
"failed to lookup records",
|
||||
"hostname", hostname,
|
||||
"error", err,
|
||||
@@ -398,9 +400,23 @@ func (w *Watcher) checkHostname(
|
||||
return
|
||||
}
|
||||
|
||||
newState := buildHostnameState(results, time.Now().UTC())
|
||||
w.updateHostnameState(
|
||||
ctx, hostname, buildHostnameState(results, time.Now().UTC()),
|
||||
)
|
||||
}
|
||||
|
||||
// updateHostnameState finishes a check of hostname from newState, built
|
||||
// from its nameservers' answers: it follows the CNAME in them, notifies
|
||||
// what changed since the previous check, and saves newState.
|
||||
func (w *Watcher) updateHostnameState(
|
||||
ctx context.Context,
|
||||
hostname string,
|
||||
newState *state.HostnameState,
|
||||
) {
|
||||
prev, hasPrev := w.state.GetHostnameState(hostname)
|
||||
|
||||
w.resolveCNAMEAddresses(ctx, hostname, newState, prev)
|
||||
|
||||
if hasPrev && !w.firstRun {
|
||||
w.detectHostnameChanges(ctx, hostname, prev, newState)
|
||||
}
|
||||
@@ -408,6 +424,77 @@ func (w *Watcher) checkHostname(
|
||||
w.state.SetHostnameState(hostname, newState)
|
||||
}
|
||||
|
||||
// resolveCNAMEAddresses saves in current the addresses at the end of
|
||||
// hostname's CNAME chain, when the nameservers' answers in current hold
|
||||
// a CNAME and no address, and an empty list otherwise. Every CNAME
|
||||
// target the nameservers gave is followed with ResolveIPAddresses and
|
||||
// the addresses found for all of them are saved, so nameservers that
|
||||
// disagree on the target do not change the result from check to check.
|
||||
// The addresses saved in prev, which may be nil, are kept when none of
|
||||
// the name's nameservers answered, and when a target cannot be
|
||||
// followed, as when no nameserver of a zone in its chain answers.
|
||||
func (w *Watcher) resolveCNAMEAddresses(
|
||||
ctx context.Context,
|
||||
hostname string,
|
||||
current, prev *state.HostnameState,
|
||||
) {
|
||||
var prevAddresses []string
|
||||
if prev != nil {
|
||||
prevAddresses = prev.CNAMEAddresses
|
||||
}
|
||||
|
||||
// Empty, not nil: nil means the addresses are not known.
|
||||
current.CNAMEAddresses = []string{}
|
||||
|
||||
answered := false
|
||||
targets := make(map[string]bool)
|
||||
|
||||
for _, nsState := range current.RecordsByNameserver {
|
||||
if nsState.Status != statusOK {
|
||||
continue
|
||||
}
|
||||
|
||||
answered = true
|
||||
|
||||
if len(nsState.Records["A"]) > 0 || len(nsState.Records["AAAA"]) > 0 {
|
||||
return
|
||||
}
|
||||
|
||||
for _, target := range nsState.Records["CNAME"] {
|
||||
targets[target] = true
|
||||
}
|
||||
}
|
||||
|
||||
if !answered {
|
||||
current.CNAMEAddresses = prevAddresses
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
for target := range targets {
|
||||
ips, err := w.resolver.ResolveIPAddresses(ctx, target)
|
||||
if err != nil {
|
||||
w.logFailedLookup(
|
||||
ctx,
|
||||
"failed to follow CNAME",
|
||||
"hostname", hostname,
|
||||
"target", target,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
current.CNAMEAddresses = prevAddresses
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
current.CNAMEAddresses = append(current.CNAMEAddresses, ips...)
|
||||
}
|
||||
|
||||
// Still the empty list when every chain ends in no address.
|
||||
slices.Sort(current.CNAMEAddresses)
|
||||
current.CNAMEAddresses = slices.Compact(current.CNAMEAddresses)
|
||||
}
|
||||
|
||||
// buildHostnameState saves each nameserver's response. A nameserver
|
||||
// that answered, even with NXDOMAIN or no records, is saved as ok; one
|
||||
// that timed out or failed is saved as error with the reason, and its
|
||||
@@ -451,6 +538,37 @@ func (w *Watcher) detectHostnameChanges(
|
||||
w.detectNSDisappearances(ctx, hostname, prev, current)
|
||||
w.detectNSFailures(ctx, hostname, prev, current)
|
||||
w.detectInconsistencies(ctx, hostname, prev, current)
|
||||
w.detectCNAMEAddressChanges(ctx, hostname, prev, current)
|
||||
}
|
||||
|
||||
// detectCNAMEAddressChanges notifies when the addresses at the end of
|
||||
// hostname's CNAME chain differ from those the previous check saved,
|
||||
// including a change from or to none. When the previous addresses are
|
||||
// not known (nil), as on the first check after loading a state file
|
||||
// written before they were saved, nothing is compared.
|
||||
func (w *Watcher) detectCNAMEAddressChanges(
|
||||
ctx context.Context,
|
||||
hostname string,
|
||||
prev, current *state.HostnameState,
|
||||
) {
|
||||
old, cur := prev.CNAMEAddresses, current.CNAMEAddresses
|
||||
if old == nil || sliceEqual(old, cur) {
|
||||
return
|
||||
}
|
||||
|
||||
msg := fmt.Sprintf(
|
||||
"Hostname: %s\nOld: %s\nNew: %s",
|
||||
hostname,
|
||||
strings.Join(old, ", "),
|
||||
strings.Join(cur, ", "),
|
||||
)
|
||||
|
||||
w.notify.SendNotification(
|
||||
ctx,
|
||||
"CNAME Address Change: "+hostname,
|
||||
msg,
|
||||
"warning",
|
||||
)
|
||||
}
|
||||
|
||||
// detectRecordChanges compares each nameserver's records with those of
|
||||
@@ -472,10 +590,12 @@ func (w *Watcher) detectRecordChanges(
|
||||
}
|
||||
|
||||
msg := fmt.Sprintf(
|
||||
"Hostname: %s\nNameserver: %s\n"+
|
||||
"Old: %v\nNew: %v",
|
||||
"Hostname: %s\nNameserver: %s\n%s",
|
||||
hostname, ns,
|
||||
prevNS.Records, cur.Records,
|
||||
recordDifferences(
|
||||
"Old", prevNS.Records,
|
||||
"New", cur.Records,
|
||||
),
|
||||
)
|
||||
|
||||
w.notify.SendNotification(
|
||||
@@ -563,10 +683,12 @@ func (w *Watcher) detectInconsistencies(
|
||||
ns1, ns2 := pair[0], pair[1]
|
||||
|
||||
msg := fmt.Sprintf(
|
||||
"Hostname: %s\n%s: %v\n%s: %v",
|
||||
"Hostname: %s\n%s",
|
||||
hostname,
|
||||
ns1, current.RecordsByNameserver[ns1].Records,
|
||||
ns2, current.RecordsByNameserver[ns2].Records,
|
||||
recordDifferences(
|
||||
ns1, current.RecordsByNameserver[ns1].Records,
|
||||
ns2, current.RecordsByNameserver[ns2].Records,
|
||||
),
|
||||
)
|
||||
|
||||
w.notify.SendNotification(
|
||||
@@ -747,6 +869,9 @@ func (w *Watcher) noNameserverAnswered(name string) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// collectIPs returns the addresses saved for hostname: those in its
|
||||
// nameservers' A and AAAA records, and those at the end of its CNAME
|
||||
// chain.
|
||||
func (w *Watcher) collectIPs(hostname string) []string {
|
||||
hs, ok := w.state.GetHostnameState(hostname)
|
||||
if !ok {
|
||||
@@ -765,6 +890,10 @@ func (w *Watcher) collectIPs(hostname string) []string {
|
||||
}
|
||||
}
|
||||
|
||||
for _, ip := range hs.CNAMEAddresses {
|
||||
ipSet[ip] = true
|
||||
}
|
||||
|
||||
result := make([]string, 0, len(ipSet))
|
||||
for ip := range ipSet {
|
||||
result = append(result, ip)
|
||||
@@ -1049,8 +1178,9 @@ func (w *Watcher) saveState() {
|
||||
|
||||
// maybeSendTestNotification sends a startup status notification
|
||||
// after the first full scan completes, if SEND_TEST_NOTIFICATION
|
||||
// is enabled. The message is clearly informational ("all ok")
|
||||
// and not an error or anomaly alert.
|
||||
// is enabled. The message is informational, not an error or anomaly
|
||||
// alert. It is written before it reaches any endpoint, so it claims
|
||||
// nothing about whether the endpoints work.
|
||||
func (w *Watcher) maybeSendTestNotification(ctx context.Context) {
|
||||
if !w.config.SendTestNotification {
|
||||
return
|
||||
@@ -1062,7 +1192,8 @@ func (w *Watcher) maybeSendTestNotification(ctx context.Context) {
|
||||
"dnswatcher has started and completed its initial scan.\n"+
|
||||
"Monitoring %d domain(s) and %d hostname(s).\n"+
|
||||
"Tracking %d port endpoint(s) and %d TLS certificate(s).\n"+
|
||||
"All notification channels are working.",
|
||||
"This is a test notification, sent to every configured "+
|
||||
"notification endpoint.",
|
||||
len(snap.Domains),
|
||||
len(snap.Hostnames),
|
||||
len(snap.Ports),
|
||||
@@ -1107,6 +1238,54 @@ func recordsEqual(
|
||||
return true
|
||||
}
|
||||
|
||||
// recordDifferences describes, in sorted order of type, each record
|
||||
// type whose values differ between a and b: a line naming the type,
|
||||
// then a line with a's values after labelA and one with b's after
|
||||
// labelB. Types with the same values in both are left out.
|
||||
func recordDifferences(
|
||||
labelA string, a map[string][]string,
|
||||
labelB string, b map[string][]string,
|
||||
) string {
|
||||
types := make([]string, 0, len(a)+len(b))
|
||||
|
||||
for recordType := range a {
|
||||
types = append(types, recordType)
|
||||
}
|
||||
|
||||
for recordType := range b {
|
||||
if _, ok := a[recordType]; !ok {
|
||||
types = append(types, recordType)
|
||||
}
|
||||
}
|
||||
|
||||
sort.Strings(types)
|
||||
|
||||
var lines []string
|
||||
|
||||
for _, recordType := range types {
|
||||
if sliceEqual(a[recordType], b[recordType]) {
|
||||
continue
|
||||
}
|
||||
|
||||
lines = append(lines,
|
||||
"Type: "+recordType,
|
||||
labelA+": "+joinValues(a[recordType]),
|
||||
labelB+": "+joinValues(b[recordType]),
|
||||
)
|
||||
}
|
||||
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// joinValues lists record values separated by commas, or says none.
|
||||
func joinValues(values []string) string {
|
||||
if len(values) == 0 {
|
||||
return "none"
|
||||
}
|
||||
|
||||
return strings.Join(values, ", ")
|
||||
}
|
||||
|
||||
func sliceEqual(a, b []string) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
|
||||
@@ -312,7 +312,8 @@ func lookupNameservers(t *testing.T, name string) []string {
|
||||
return nameservers
|
||||
}
|
||||
|
||||
// addresses returns the A and AAAA values saved for a hostname.
|
||||
// addresses returns the A and AAAA values saved for a hostname, and the
|
||||
// addresses saved at the end of its CNAME chain.
|
||||
func addresses(hs *state.HostnameState) []string {
|
||||
var ips []string
|
||||
|
||||
@@ -321,7 +322,7 @@ func addresses(hs *state.HostnameState) []string {
|
||||
ips = append(ips, nsState.Records["AAAA"]...)
|
||||
}
|
||||
|
||||
return ips
|
||||
return append(ips, hs.CNAMEAddresses...)
|
||||
}
|
||||
|
||||
// assertNotified checks that a notification with this title and
|
||||
@@ -371,6 +372,14 @@ func TestFirstRunBaseline(t *testing.T) {
|
||||
|
||||
assertNoNotifications(t, deps)
|
||||
assertStatePopulated(t, deps)
|
||||
|
||||
// testHost answers with an address, so the check saves an empty list
|
||||
// of CNAME addresses for it; nil would mean the check did not look
|
||||
// at whether to follow a CNAME.
|
||||
hs, _ := deps.state.GetHostnameState(testHost)
|
||||
if hs.CNAMEAddresses == nil || len(hs.CNAMEAddresses) != 0 {
|
||||
t.Errorf("saved CNAME addresses %#v, want []", hs.CNAMEAddresses)
|
||||
}
|
||||
}
|
||||
|
||||
func assertNoNotifications(
|
||||
@@ -866,18 +875,27 @@ func TestSendTestNotification_ViaRun(t *testing.T) {
|
||||
|
||||
notifications := deps.notifier.getNotifications()
|
||||
|
||||
// No names are configured, so every count is 0.
|
||||
wantMessage := "dnswatcher has started and completed its initial scan.\n" +
|
||||
"Monitoring 0 domain(s) and 0 hostname(s).\n" +
|
||||
"Tracking 0 port endpoint(s) and 0 TLS certificate(s).\n" +
|
||||
"This is a test notification, sent to every configured " +
|
||||
"notification endpoint."
|
||||
|
||||
found := false
|
||||
|
||||
for _, n := range notifications {
|
||||
if n.Priority == "success" &&
|
||||
n.Title == "✅ dnswatcher startup complete" {
|
||||
n.Title == "✅ dnswatcher startup complete" &&
|
||||
n.Message == wantMessage {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
|
||||
if !found {
|
||||
t.Errorf(
|
||||
"expected startup test notification, got: %v",
|
||||
"expected startup test notification with message %q, got: %v",
|
||||
wantMessage,
|
||||
notifications,
|
||||
)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user