notify: fix flaky drain timing assertions, drop false abandon warn (closes #106)

The drain tests measured elapsed time from an instant captured after
the clock they compared it against had already started, so the lower
bounds were structurally unreachable and passed only when the gap
between the two statements rounded to zero. TestDrainBoundedByContext-
Deadline failed the Docker gate outright (49.9ms against its own 50ms
deadline) and roughly 1 run in 12 locally.

- TestDrainBoundedByContextDeadline: capture start before
  context.WithTimeout, so the measured interval is a superset of the
  deadline interval and only an early return can fail the lower bound.
  The upper bound moves to a watchdog around the drain, which turns an
  unbounded drain into a prompt failure instead of a package-timeout
  hang.
- TestDrainWaitsForInFlightDelivery: same ordering fix, ahead of the
  timer that releases the held delivery.
- TestDrainWithoutDeliveriesReturnsImmediately: its 50ms ceiling was
  under the observed cost of the goroutine hop through inFlight.Wait()
  on a loaded box (57ms), and failed once in 20 runs. It now bounds the
  idle drain at 500ms, still well under the 2s deadline a stalled drain
  would hit.
- drain: an OnStop context already expired on entry with nothing
  outstanding logged a WARN about abandoning deliveries with
  abandoned=0 and closed the abandon channel for no reason. The timeout
  branch now reports at debug level when the outstanding count is zero,
  and warns only when deliveries genuinely are abandoned.
  TestDrainWithCancelledContextDoesNotWarn covers it.

Verified: script/cibuild passes; 25 consecutive cache-bypassed
make test runs under -race, all clean; make check green at 5.2s.
Both corrected assertions were confirmed non-vacuous by temporarily
breaking drain and watching them fail.

.golangci.yml

@@ -1,5 +1,9 @@
 version: "2"
 
+# Config schema uses the golangci-lint v2 layout (settings live under
+# linters.settings, not top-level linters-settings) so that the
+# thresholds below are actually applied by golangci-lint >= v2.
+
 run:
   timeout: 5m
   modules-download-mode: readonly
@@ -14,8 +18,7 @@ linters:
     - wsl              # Deprecated, replaced by wsl_v5
     - wrapcheck        # Too verbose for internal packages
     - varnamelen       # Short names like db, id are idiomatic Go
-
+  settings:
-linters-settings:
     lll:
       line-length: 88
     funlen:
@@ -27,6 +30,5 @@ linters-settings:
       threshold: 100
 
 issues:
-  exclude-use-default: false
   max-issues-per-linter: 0
   max-same-issues: 0

README.md

@@ -17,6 +17,26 @@ without requiring an external database.
 
 ---
 
+## No DNS mocking. Ever.
+
+**DNS is never mocked in this project — not in tests, not anywhere else.**
+No mock resolvers, no fake DNS servers, no stubbed lookups.
+
+dnswatcher's entire purpose is correct behavior against the real DNS.
+Tests exercise real iterative resolution against live nameservers by
+design; a test suite that passes against a mock proves nothing about the
+one thing this program exists to do.
+
+When live tests are flaky, that is a robustness problem, and it gets
+fixed with robustness: retries with backoff, querying multiple
+independent nameservers, longer timeouts — or explicit opt-in gating
+decided by the project owner. Never with mocks.
+
+Contributions that introduce mocked, faked, or stubbed DNS will be
+rejected.
+
+---
+
 ## Features
 
 ### DNS Domain Monitoring (Apex Domains)
@@ -198,7 +218,8 @@ internal/
 - **Structured logging**: All logs use `log/slog` with JSON output in
   production (TTY detection for development).
 - **Graceful shutdown**: All background goroutines respect context
-  cancellation and the fx lifecycle.
+  cancellation and the fx lifecycle. In-flight notification deliveries
+  are drained on shutdown, bounded by the shutdown timeout.
 
 ---
 
@@ -432,8 +453,14 @@ docker run -d \
      from a previous cycle.
 4. **On change detection**: Send notifications to all configured
    endpoints, update in-memory state, persist to disk.
-5. **Shutdown**: Persist final state to disk, complete in-flight
+5. **Shutdown**: Persist final state to disk, wait for in-flight
-   notifications, stop gracefully.
+   notification deliveries to complete, stop gracefully. The wait is
+   bounded by the fx shutdown timeout (15s by default): deliveries still
+   retrying against an unreachable endpoint when that expires are
+   abandoned, and the number abandoned is logged at warn level rather
+   than dropped silently. Notifications generated after shutdown has
+   begun are refused and logged, so a late burst cannot extend the
+   shutdown.
 
 ---
 

TODO.md

@@ -25,9 +25,23 @@ confirm make check still passes.
 
 # Completed Steps
 
+- 2026-08-09: in-flight notification deliveries are now drained at
+  shutdown (#106): `notify.New` registers an fx `OnStop` hook that waits
+  on a `sync.WaitGroup` of tracked delivery goroutines, bounded by the
+  `OnStop` context; on expiry the outstanding count is logged at warn
+  level and parked retry backoffs are released instead of being dropped
+  silently, and deliveries submitted after the drain begins are refused
+  so shutdown cannot be extended indefinitely; an `OnStop` context that
+  is already expired on entry with nothing outstanding drains quietly
+  rather than warning about deliveries that were never abandoned
 - 2026-08-07: golangci-lint bumped to v2.12.2 (commit-pinned installs
-  in `Dockerfile` and `script/bootstrap`); fixed the resulting
+  in `Dockerfile` and `script/bootstrap`); `.golangci.yml` set to the
-  `goconst` findings. `.golangci.yml` unchanged (canonical)
+  org-standard v2-schema config used across the org's repos
+  (owner-authorized; same file is being landed as canonical via prompts
+  PR #24), with settings under `linters.settings` so the
+  lll/funlen/cyclop/dupl thresholds apply; fixed the resulting
+  `goconst`, `dupl`, and `lll` findings; the informational `gomodguard`
+  deprecation warning under this config is accepted
 - 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
   Makefile shims, README Entrypoints section
 - 2026-02-20: iterative DNS resolver implemented; tests made hermetic

internal/config/classify_test.go

@@ -17,13 +17,33 @@ func TestClassifyDNSName(t *testing.T) {
 	}{
 		{name: "apex domain simple", input: "example.com", want: config.DNSNameTypeDomain},
 		{name: "hostname simple", input: "www.example.com", want: config.DNSNameTypeHostname},
-		{name: "apex domain multi-part TLD", input: "example.co.uk", want: config.DNSNameTypeDomain},
+		{
-		{name: "hostname multi-part TLD", input: "api.example.co.uk", want: config.DNSNameTypeHostname},
+			name:  "apex domain multi-part TLD",
+			input: "example.co.uk",
+			want:  config.DNSNameTypeDomain,
+		},
+		{
+			name:  "hostname multi-part TLD",
+			input: "api.example.co.uk",
+			want:  config.DNSNameTypeHostname,
+		},
 		{name: "public suffix itself", input: "co.uk", wantErr: true},
 		{name: "empty string", input: "", wantErr: true},
-		{name: "deeply nested hostname", input: "a.b.c.example.com", want: config.DNSNameTypeHostname},
+		{
-		{name: "trailing dot stripped", input: "example.com.", want: config.DNSNameTypeDomain},
+			name:  "deeply nested hostname",
-		{name: "uppercase normalized", input: "WWW.Example.COM", want: config.DNSNameTypeHostname},
+			input: "a.b.c.example.com",
+			want:  config.DNSNameTypeHostname,
+		},
+		{
+			name:  "trailing dot stripped",
+			input: "example.com.",
+			want:  config.DNSNameTypeDomain,
+		},
+		{
+			name:  "uppercase normalized",
+			input: "WWW.Example.COM",
+			want:  config.DNSNameTypeHostname,
+		},
 	}
 
 	for _, tt := range tests {

internal/notify/delivery_test.go

@@ -25,18 +25,20 @@ const (
 	colorDefault = "#6c757d"
 )
 
-// Priority and fixture values shared across tests.
+// Priority strings used across multiple tests.
 const (
 	prioError   = "error"
 	prioWarning = "warning"
-	prioInfo    = "info"
 	prioSuccess = "success"
+	prioInfo    = "info"
 	prioUnknown = "unknown"
-	ntfyUrgent  = "urgent"
+	prioDefault = "default"
-	ntfyDefault = "default"
+	prioUrgent  = "urgent"
-	testHost    = "example.com"
 )
 
+// testHost is the hostname used in request construction tests.
+const testHost = "example.com"
+
 // errSimulated is a static error for transport failures.
 var errSimulated = errors.New("simulated transport failure")
 
@@ -113,13 +115,13 @@ func TestNtfyPriority(t *testing.T) {
 		input string
 		want  string
 	}{
-		{prioError, ntfyUrgent},
+		{prioError, prioUrgent},
 		{prioWarning, "high"},
-		{prioSuccess, ntfyDefault},
+		{prioSuccess, prioDefault},
 		{prioInfo, "low"},
-		{"", ntfyDefault},
+		{"", prioDefault},
-		{prioUnknown, ntfyDefault},
+		{prioUnknown, prioDefault},
-		{"critical", ntfyDefault},
+		{"critical", prioDefault},
 	}
 
 	for _, tc := range cases {
@@ -301,10 +303,10 @@ func TestSendNtfyHeaders(t *testing.T) {
 		)
 	}
 
-	if captured.priority != ntfyUrgent {
+	if captured.priority != prioUrgent {
 		t.Errorf(
 			"Priority header = %q, want %q",
-			captured.priority, ntfyUrgent,
+			captured.priority, prioUrgent,
 		)
 	}
 
@@ -323,9 +325,9 @@ func TestSendNtfyAllPriorities(t *testing.T) {
 		input string
 		want  string
 	}{
-		{prioError, ntfyUrgent},
+		{prioError, prioUrgent},
 		{prioWarning, "high"},
-		{prioSuccess, ntfyDefault},
+		{prioSuccess, prioDefault},
 		{prioInfo, "low"},
 	}
 
@@ -368,56 +370,69 @@ func TestSendNtfyAllPriorities(t *testing.T) {
 	}
 }
 
-func TestSendNtfyClientError(t *testing.T) {
+// assertSendStatusError verifies that send returns an error
-	t.Parallel()
+// wrapping wantErr when the server responds with status.
+func assertSendStatusError(
+	t *testing.T,
+	status int,
+	wantErr error,
+	send func(*notify.Service, *url.URL) error,
+) {
+	t.Helper()
 
 	srv := httptest.NewServer(
 		http.HandlerFunc(
 			func(w http.ResponseWriter, _ *http.Request) {
-				w.WriteHeader(http.StatusForbidden)
+				w.WriteHeader(status)
 			}),
 	)
 	defer srv.Close()
 
 	svc := notify.NewTestService(srv.Client().Transport)
-	topicURL, _ := url.Parse(srv.URL)
+	target, _ := url.Parse(srv.URL)
 
-	err := svc.SendNtfy(
+	err := send(svc, target)
-		context.Background(), topicURL, "t", "m", "info",
-	)
 	if err == nil {
-		t.Fatal("expected error for 403 response")
+		t.Fatalf("expected error for %d response", status)
 	}
 
-	if !errors.Is(err, notify.ErrNtfyFailed) {
+	if !errors.Is(err, wantErr) {
-		t.Errorf("error = %v, want ErrNtfyFailed", err)
+		t.Errorf("error = %v, want %v", err, wantErr)
 	}
 }
 
+func sendNtfyInfo(
+	svc *notify.Service, target *url.URL,
+) error {
+	return svc.SendNtfy(
+		context.Background(), target, "t", "m", prioInfo,
+	)
+}
+
+func sendSlackInfo(
+	svc *notify.Service, target *url.URL,
+) error {
+	return svc.SendSlack(
+		context.Background(), target, "t", "m", prioInfo,
+	)
+}
+
+func TestSendNtfyClientError(t *testing.T) {
+	t.Parallel()
+
+	assertSendStatusError(
+		t, http.StatusForbidden,
+		notify.ErrNtfyFailed, sendNtfyInfo,
+	)
+}
+
 func TestSendNtfyServerError(t *testing.T) {
 	t.Parallel()
 
-	srv := httptest.NewServer(
+	assertSendStatusError(
-		http.HandlerFunc(
+		t, http.StatusInternalServerError,
-			func(w http.ResponseWriter, _ *http.Request) {
+		notify.ErrNtfyFailed, sendNtfyInfo,
-				w.WriteHeader(http.StatusInternalServerError)
-			}),
 	)
-	defer srv.Close()
-
-	svc := notify.NewTestService(srv.Client().Transport)
-	topicURL, _ := url.Parse(srv.URL)
-
-	err := svc.SendNtfy(
-		context.Background(), topicURL, "t", "m", "info",
-	)
-	if err == nil {
-		t.Fatal("expected error for 500 response")
-	}
-
-	if !errors.Is(err, notify.ErrNtfyFailed) {
-		t.Errorf("error = %v, want ErrNtfyFailed", err)
-	}
 }
 
 func TestSendNtfySuccess(t *testing.T) {
@@ -618,53 +633,19 @@ func TestSendSlackAllColors(t *testing.T) {
 func TestSendSlackClientError(t *testing.T) {
 	t.Parallel()
 
-	srv := httptest.NewServer(
+	assertSendStatusError(
-		http.HandlerFunc(
+		t, http.StatusBadRequest,
-			func(w http.ResponseWriter, _ *http.Request) {
+		notify.ErrSlackFailed, sendSlackInfo,
-				w.WriteHeader(http.StatusBadRequest)
-			}),
 	)
-	defer srv.Close()
-
-	svc := notify.NewTestService(srv.Client().Transport)
-	webhookURL, _ := url.Parse(srv.URL)
-
-	err := svc.SendSlack(
-		context.Background(), webhookURL, "t", "m", "info",
-	)
-	if err == nil {
-		t.Fatal("expected error for 400 response")
-	}
-
-	if !errors.Is(err, notify.ErrSlackFailed) {
-		t.Errorf("error = %v, want ErrSlackFailed", err)
-	}
 }
 
 func TestSendSlackServerError(t *testing.T) {
 	t.Parallel()
 
-	srv := httptest.NewServer(
+	assertSendStatusError(
-		http.HandlerFunc(
+		t, http.StatusBadGateway,
-			func(w http.ResponseWriter, _ *http.Request) {
+		notify.ErrSlackFailed, sendSlackInfo,
-				w.WriteHeader(http.StatusBadGateway)
-			}),
 	)
-	defer srv.Close()
-
-	svc := notify.NewTestService(srv.Client().Transport)
-	webhookURL, _ := url.Parse(srv.URL)
-
-	err := svc.SendSlack(
-		context.Background(), webhookURL, "t", "m", "error",
-	)
-	if err == nil {
-		t.Fatal("expected error for 502 response")
-	}
-
-	if !errors.Is(err, notify.ErrSlackFailed) {
-		t.Errorf("error = %v, want ErrSlackFailed", err)
-	}
 }
 
 func TestSendSlackNetworkError(t *testing.T) {
@@ -989,74 +970,62 @@ func TestSendNotificationMattermostOnly(t *testing.T) {
 	}
 }
 
-func TestSendNotificationNtfyError(t *testing.T) {
+// assertSendNotificationTolerates verifies SendNotification
-	t.Parallel()
+// neither panics nor blocks when the endpoint configured by
+// setURL responds with status.
+func assertSendNotificationTolerates(
+	t *testing.T,
+	status int,
+	priority string,
+	setURL func(*notify.Service, *url.URL),
+) {
+	t.Helper()
 
 	srv := httptest.NewServer(
 		http.HandlerFunc(
 			func(w http.ResponseWriter, _ *http.Request) {
-				w.WriteHeader(http.StatusInternalServerError)
+				w.WriteHeader(status)
 			}),
 	)
 	defer srv.Close()
 
-	ntfyURL, _ := url.Parse(srv.URL)
+	target, _ := url.Parse(srv.URL)
 
 	svc := notify.NewTestService(http.DefaultTransport)
-	svc.SetNtfyURL(ntfyURL)
+	setURL(svc, target)
 
-	// Should not panic or block.
 	svc.SendNotification(
-		context.Background(), "t", "m", "error",
+		context.Background(), "t", "m", priority,
 	)
 
 	time.Sleep(100 * time.Millisecond)
 }
 
+func TestSendNotificationNtfyError(t *testing.T) {
+	t.Parallel()
+
+	assertSendNotificationTolerates(
+		t, http.StatusInternalServerError, prioError,
+		(*notify.Service).SetNtfyURL,
+	)
+}
+
 func TestSendNotificationSlackError(t *testing.T) {
 	t.Parallel()
 
-	srv := httptest.NewServer(
+	assertSendNotificationTolerates(
-		http.HandlerFunc(
+		t, http.StatusForbidden, prioError,
-			func(w http.ResponseWriter, _ *http.Request) {
+		(*notify.Service).SetSlackWebhookURL,
-				w.WriteHeader(http.StatusForbidden)
-			}),
 	)
-	defer srv.Close()
-
-	slackURL, _ := url.Parse(srv.URL)
-
-	svc := notify.NewTestService(http.DefaultTransport)
-	svc.SetSlackWebhookURL(slackURL)
-
-	svc.SendNotification(
-		context.Background(), "t", "m", "error",
-	)
-
-	time.Sleep(100 * time.Millisecond)
 }
 
 func TestSendNotificationMattermostError(t *testing.T) {
 	t.Parallel()
 
-	srv := httptest.NewServer(
+	assertSendNotificationTolerates(
-		http.HandlerFunc(
+		t, http.StatusBadGateway, prioWarning,
-			func(w http.ResponseWriter, _ *http.Request) {
+		(*notify.Service).SetMattermostWebhookURL,
-				w.WriteHeader(http.StatusBadGateway)
-			}),
 	)
-	defer srv.Close()
-
-	mmURL, _ := url.Parse(srv.URL)
-
-	svc := notify.NewTestService(http.DefaultTransport)
-	svc.SetMattermostWebhookURL(mmURL)
-
-	svc.SendNotification(
-		context.Background(), "t", "m", "warning",
-	)
-
-	time.Sleep(100 * time.Millisecond)
 }
 
 // ── SlackPayload JSON marshaling ──────────────────────────

internal/notify/export_test.go

@@ -32,11 +32,27 @@ func NewRequestForTest(
 // NewTestService creates a Service suitable for unit testing.
 // It discards log output and uses the given transport.
 func NewTestService(transport http.RoundTripper) *Service {
-	return &Service{
+	return newService(slog.New(slog.DiscardHandler), transport)
-		log:       slog.New(slog.DiscardHandler),
+}
-		transport: transport,
+
-		history:   NewAlertHistory(),
+// NewTestServiceWithLogger creates a Service that writes to the
-	}
+// given handler, so tests can assert on emitted log records.
+func NewTestServiceWithLogger(
+	transport http.RoundTripper,
+	handler slog.Handler,
+) *Service {
+	return newService(slog.New(handler), transport)
+}
+
+// Drain exports drain for testing.
+func (svc *Service) Drain(ctx context.Context) {
+	svc.drain(ctx)
+}
+
+// OutstandingDeliveries reports how many delivery goroutines
+// are currently tracked as in flight.
+func (svc *Service) OutstandingDeliveries() int64 {
+	return svc.outstanding.Load()
 }
 
 // SetNtfyURL sets the ntfy URL on a Service for testing.

internal/notify/notify.go

@@ -12,6 +12,8 @@ import (
 	"log/slog"
 	"net/http"
 	"net/url"
+	"sync"
+	"sync/atomic"
 	"time"
 
 	"go.uber.org/fx"
@@ -115,19 +117,41 @@ type Service struct {
 	history              *AlertHistory
 	retryConfig          RetryConfig
 	sleepFn              func(time.Duration) <-chan time.Time
+
+	// Shutdown draining state.  drainMu guards draining and
+	// serialises it against the counter increment in
+	// startDelivery; inFlight tracks the delivery goroutines
+	// themselves and outstanding mirrors its count so a timed
+	// out drain can report how many were abandoned.
+	drainMu     sync.Mutex
+	draining    bool
+	inFlight    sync.WaitGroup
+	outstanding atomic.Int64
+	abandon     chan struct{}
+	abandonOnce sync.Once
+}
+
+// newService builds a Service with the fields every Service
+// needs regardless of how it was constructed.
+func newService(
+	log *slog.Logger,
+	transport http.RoundTripper,
+) *Service {
+	return &Service{
+		log:       log,
+		transport: transport,
+		history:   NewAlertHistory(),
+		abandon:   make(chan struct{}),
+	}
 }
 
 // New creates a new notify Service.
 func New(
-	_ fx.Lifecycle,
+	lifecycle fx.Lifecycle,
 	params Params,
 ) (*Service, error) {
-	svc := &Service{
+	svc := newService(params.Logger.Get(), http.DefaultTransport)
-		log:       params.Logger.Get(),
+	svc.config = params.Config
-		transport: http.DefaultTransport,
-		config:    params.Config,
-		history:   NewAlertHistory(),
-	}
 
 	if params.Config.NtfyTopic != "" {
 		u, err := ValidateWebhookURL(
@@ -168,6 +192,14 @@ func New(
 		svc.mattermostWebhookURL = u
 	}
 
+	lifecycle.Append(fx.Hook{
+		OnStop: func(ctx context.Context) error {
+			svc.drain(ctx)
+
+			return nil
+		},
+	})
+
 	return svc, nil
 }
 
@@ -194,6 +226,32 @@ func (svc *Service) SendNotification(
 	svc.dispatchMattermost(ctx, title, message, priority)
 }
 
+// dispatch delivers a notification to one endpoint on a
+// tracked background goroutine.
+//
+// The delivery context is detached from ctx with
+// context.WithoutCancel so that a cancelled caller does not
+// kill a delivery already under way; the shutdown drain, not
+// the caller, decides how long deliveries may keep running.
+func (svc *Service) dispatch(
+	ctx context.Context,
+	endpoint string,
+	send func(context.Context) error,
+) {
+	notifyCtx := context.WithoutCancel(ctx)
+
+	svc.startDelivery(endpoint, func() {
+		err := svc.deliverWithRetry(notifyCtx, endpoint, send)
+		if err != nil {
+			svc.log.Error(
+				"failed to send notification after retries",
+				"endpoint", endpoint,
+				"error", err,
+			)
+		}
+	})
+}
+
 func (svc *Service) dispatchNtfy(
 	ctx context.Context,
 	title, message, priority string,
@@ -202,26 +260,11 @@ func (svc *Service) dispatchNtfy(
 		return
 	}
 
-	go func() {
+	svc.dispatch(ctx, "ntfy", func(c context.Context) error {
-		notifyCtx := context.WithoutCancel(ctx)
-
-		err := svc.deliverWithRetry(
-			notifyCtx, "ntfy",
-			func(c context.Context) error {
 		return svc.sendNtfy(
-					c, svc.ntfyURL,
+			c, svc.ntfyURL, title, message, priority,
-					title, message, priority,
 		)
-			},
+	})
-		)
-		if err != nil {
-			svc.log.Error(
-				"failed to send ntfy notification "+
-					"after retries",
-				"error", err,
-			)
-		}
-	}()
 }
 
 func (svc *Service) dispatchSlack(
@@ -232,26 +275,11 @@ func (svc *Service) dispatchSlack(
 		return
 	}
 
-	go func() {
+	svc.dispatch(ctx, "slack", func(c context.Context) error {
-		notifyCtx := context.WithoutCancel(ctx)
-
-		err := svc.deliverWithRetry(
-			notifyCtx, "slack",
-			func(c context.Context) error {
 		return svc.sendSlack(
-					c, svc.slackWebhookURL,
+			c, svc.slackWebhookURL, title, message, priority,
-					title, message, priority,
 		)
-			},
+	})
-		)
-		if err != nil {
-			svc.log.Error(
-				"failed to send slack notification "+
-					"after retries",
-				"error", err,
-			)
-		}
-	}()
 }
 
 func (svc *Service) dispatchMattermost(
@@ -262,11 +290,8 @@ func (svc *Service) dispatchMattermost(
 		return
 	}
 
-	go func() {
+	svc.dispatch(
-		notifyCtx := context.WithoutCancel(ctx)
+		ctx, "mattermost",
-
-		err := svc.deliverWithRetry(
-			notifyCtx, "mattermost",
 		func(c context.Context) error {
 			return svc.sendSlack(
 				c, svc.mattermostWebhookURL,
@@ -274,14 +299,6 @@ func (svc *Service) dispatchMattermost(
 			)
 		},
 	)
-		if err != nil {
-			svc.log.Error(
-				"failed to send mattermost notification "+
-					"after retries",
-				"error", err,
-			)
-		}
-	}()
 }
 
 func (svc *Service) sendNtfy(

internal/notify/retry.go

@@ -2,6 +2,7 @@ package notify
 
 import (
 	"context"
+	"fmt"
 	"math"
 	"math/rand/v2"
 	"time"
@@ -69,7 +70,7 @@ func (rc RetryConfig) backoff(attempt int) time.Duration {
 	lo := raw * (1 - jitterFraction)
 	hi := raw * (1 + jitterFraction)
 
-	jittered := lo + rand.Float64()*(hi-lo) //nolint:gosec // jitter does not need crypto/rand
+	jittered := lo + rand.Float64()*(hi-lo) //nolint:gosec // jitter needs no crypto/rand
 
 	return time.Duration(jittered)
 }
@@ -121,6 +122,14 @@ func (svc *Service) deliverWithRetry(
 		select {
 		case <-ctx.Done():
 			return ctx.Err()
+		case <-svc.abandon:
+			// Shutdown drained past its deadline; stop
+			// sleeping rather than outlive the process.
+			// A nil channel (Service built without a
+			// constructor) simply never fires.
+			return fmt.Errorf(
+				"%w: %s", ErrDeliveryAbandoned, endpoint,
+			)
 		case <-svc.sleepFunc(delay):
 		}
 	}

internal/notify/shutdown.go

@@ -0,0 +1,119 @@
+package notify
+
+import (
+	"context"
+	"errors"
+)
+
+// ErrDeliveryAbandoned is returned by a retry loop that was
+// cut short because shutdown drained past its deadline.
+var ErrDeliveryAbandoned = errors.New(
+	"notification delivery abandoned at shutdown",
+)
+
+// startDelivery runs fn on its own goroutine while tracking it,
+// so that drain can wait for it during shutdown.
+//
+// The WaitGroup counter is incremented here, on the caller's
+// goroutine, before the worker exists: incrementing it inside
+// the worker would race with drain's Wait and could let
+// shutdown sail past a delivery that had not started yet.
+//
+// Once draining has begun the delivery is refused outright
+// rather than queued, so a steady stream of newly submitted
+// notifications cannot keep extending the drain.
+func (svc *Service) startDelivery(endpoint string, fn func()) {
+	svc.drainMu.Lock()
+
+	if svc.draining {
+		svc.drainMu.Unlock()
+
+		svc.log.Warn(
+			"notification not dispatched: shutdown in progress",
+			"endpoint", endpoint,
+		)
+
+		return
+	}
+
+	svc.outstanding.Add(1)
+
+	// WaitGroup.Go increments the counter synchronously, here,
+	// and only then starts the goroutine.
+	svc.inFlight.Go(func() {
+		// Runs before the WaitGroup counter is decremented, so
+		// a drain that times out reports an accurate count.
+		defer svc.outstanding.Add(-1)
+
+		fn()
+	})
+
+	svc.drainMu.Unlock()
+}
+
+// drain waits for in-flight notification deliveries to finish.
+//
+// It first stops accepting new deliveries, then waits until
+// either every outstanding delivery has completed or ctx
+// expires — whichever comes first.  ctx is the context fx
+// passes to the OnStop hook, so a permanently dead webhook
+// cannot hang shutdown indefinitely.
+//
+// When the deadline arrives with deliveries still outstanding,
+// the count is logged at warn level and the abandon channel is
+// closed, which releases any retry loop sleeping in backoff.
+// Deliveries already inside an HTTP round trip are bounded by
+// the existing httpClientTimeout instead.
+//
+// A ctx that is already expired on entry is not by itself cause
+// for alarm: if nothing is outstanding there is nothing to
+// abandon, and the drain says so at debug level rather than
+// warning about deliveries that do not exist.
+func (svc *Service) drain(ctx context.Context) {
+	svc.drainMu.Lock()
+	svc.draining = true
+	svc.drainMu.Unlock()
+
+	done := make(chan struct{})
+
+	go func() {
+		svc.inFlight.Wait()
+		close(done)
+	}()
+
+	select {
+	case <-done:
+		svc.log.Debug(
+			"all in-flight notifications completed",
+		)
+	case <-ctx.Done():
+		// outstanding is decremented before the WaitGroup
+		// counter, and startDelivery can no longer add to it
+		// now that draining is set, so a zero here means every
+		// delivery really did finish.  ctx expiring in that
+		// state (an OnStop context that was already cancelled
+		// on entry is the usual way) abandons nothing, so it
+		// must not close abandon or warn about it.
+		abandoned := svc.outstanding.Load()
+		if abandoned == 0 {
+			svc.log.Debug(
+				"all in-flight notifications completed",
+			)
+
+			return
+		}
+
+		svc.abandonOnce.Do(func() {
+			if svc.abandon != nil {
+				close(svc.abandon)
+			}
+		})
+
+		svc.log.Warn(
+			"shutdown deadline reached with notifications "+
+				"still in flight; abandoning them",
+			"abandoned", abandoned,
+			"error", ctx.Err(),
+		)
+	}
+}

internal/notify/shutdown_test.go

@@ -0,0 +1,531 @@
+package notify_test
+
+import (
+	"bytes"
+	"context"
+	"log/slog"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"go.uber.org/fx"
+
+	"sneak.berlin/go/dnswatcher/internal/config"
+	"sneak.berlin/go/dnswatcher/internal/globals"
+	"sneak.berlin/go/dnswatcher/internal/logger"
+	"sneak.berlin/go/dnswatcher/internal/notify"
+)
+
+// Timings used by the drain tests.  They stay in the same
+// 10-100ms band as the retry tests so the suite never waits on
+// a real backoff delay.
+const (
+	// inFlightHold is how long a delivery is kept mid-request
+	// before the handler is released.
+	inFlightHold = 30 * time.Millisecond
+
+	// drainDeadline bounds a drain that is expected to time
+	// out.
+	drainDeadline = 50 * time.Millisecond
+
+	// drainSlack is the upper bound on how long a bounded
+	// drain may take; generous enough for a loaded CI box,
+	// still far below the 20s test ceiling.
+	drainSlack = 2 * time.Second
+
+	// settleDelay is how long to wait before asserting that
+	// something did *not* happen.
+	settleDelay = 50 * time.Millisecond
+
+	// idleDrainBound is the upper bound on a drain that has
+	// nothing in flight.  It is deliberately far above the cost
+	// of the goroutine hop through inFlight.Wait() — which
+	// reached 57ms on a loaded box under -race with the package's
+	// parallel tests — and far below drainSlack, the deadline
+	// such a drain is given.  A drain that blocked until its
+	// deadline instead of returning on the WaitGroup therefore
+	// still fails this bound, but scheduling delay alone cannot.
+	idleDrainBound = 500 * time.Millisecond
+)
+
+// syncBuffer is an io.Writer safe for concurrent use, so log
+// output written from delivery goroutines can be inspected.
+type syncBuffer struct {
+	mu  sync.Mutex
+	buf bytes.Buffer
+}
+
+func (sb *syncBuffer) Write(p []byte) (int, error) {
+	sb.mu.Lock()
+	defer sb.mu.Unlock()
+
+	return sb.buf.Write(p) //nolint:wrapcheck // test helper
+}
+
+func (sb *syncBuffer) String() string {
+	sb.mu.Lock()
+	defer sb.mu.Unlock()
+
+	return sb.buf.String()
+}
+
+// newLoggingService returns a Service writing JSON logs into
+// the returned buffer.
+func newLoggingService(
+	transport http.RoundTripper,
+) (*notify.Service, *syncBuffer) {
+	logs := &syncBuffer{}
+	handler := slog.NewJSONHandler(logs, nil)
+
+	return notify.NewTestServiceWithLogger(transport, handler),
+		logs
+}
+
+// blockingNtfyServer returns a server whose handler signals on
+// entered, waits for release, and then responds 200.
+func blockingNtfyServer(
+	entered chan<- struct{},
+	release <-chan struct{},
+	served *atomic.Bool,
+) *httptest.Server {
+	var once sync.Once
+
+	return httptest.NewServer(
+		http.HandlerFunc(
+			func(w http.ResponseWriter, _ *http.Request) {
+				once.Do(func() { close(entered) })
+				<-release
+
+				served.Store(true)
+
+				w.WriteHeader(http.StatusOK)
+			}),
+	)
+}
+
+// TestDrainWaitsForInFlightDelivery verifies that a delivery
+// already under way when shutdown starts is allowed to finish.
+func TestDrainWaitsForInFlightDelivery(t *testing.T) {
+	t.Parallel()
+
+	var served atomic.Bool
+
+	entered := make(chan struct{})
+	release := make(chan struct{})
+
+	srv := blockingNtfyServer(entered, release, &served)
+	defer srv.Close()
+
+	topicURL, _ := url.Parse(srv.URL)
+
+	svc := notify.NewTestService(http.DefaultTransport)
+	svc.SetNtfyURL(topicURL)
+
+	svc.SendNotification(
+		context.Background(), "t", "m", prioInfo,
+	)
+
+	// Make sure the delivery really is mid-request before the
+	// drain begins.
+	select {
+	case <-entered:
+	case <-time.After(drainSlack):
+		t.Fatal("delivery never reached the endpoint")
+	}
+
+	// As in TestDrainBoundedByContextDeadline: start is captured
+	// before the clock it is compared against, here the timer
+	// holding the delivery open, so elapsed covers the whole hold
+	// and the lower bound cannot come out short from scheduling
+	// delay alone.
+	start := time.Now()
+
+	timer := time.AfterFunc(inFlightHold, func() {
+		close(release)
+	})
+	defer timer.Stop()
+
+	ctx, cancel := context.WithTimeout(
+		context.Background(), drainSlack,
+	)
+	defer cancel()
+
+	svc.Drain(ctx)
+
+	elapsed := time.Since(start)
+
+	if !served.Load() {
+		t.Error(
+			"drain returned before the in-flight delivery " +
+				"completed",
+		)
+	}
+
+	if elapsed < inFlightHold {
+		t.Errorf(
+			"drain took %v, want at least %v",
+			elapsed, inFlightHold,
+		)
+	}
+
+	if got := svc.OutstandingDeliveries(); got != 0 {
+		t.Errorf("outstanding deliveries = %d, want 0", got)
+	}
+}
+
+// neverFires returns a channel that never delivers, standing in
+// for a long backoff sleep without actually sleeping.
+func neverFires(_ time.Duration) <-chan time.Time {
+	return make(chan time.Time)
+}
+
+// TestDrainBoundedByContextDeadline verifies that a delivery
+// stuck retrying against a dead endpoint does not hold shutdown
+// past the OnStop context deadline, and that the abandoned
+// deliveries are logged at warn level rather than dropped
+// silently.
+func TestDrainBoundedByContextDeadline(t *testing.T) {
+	t.Parallel()
+
+	var requests atomic.Int64
+
+	srv := httptest.NewServer(
+		http.HandlerFunc(
+			func(w http.ResponseWriter, _ *http.Request) {
+				requests.Add(1)
+
+				w.WriteHeader(http.StatusInternalServerError)
+			}),
+	)
+	defer srv.Close()
+
+	topicURL, _ := url.Parse(srv.URL)
+
+	svc, logs := newLoggingService(http.DefaultTransport)
+	svc.SetNtfyURL(topicURL)
+	// Never let the backoff sleep complete: the delivery is
+	// parked in its retry wait until shutdown releases it.
+	svc.SetSleepFunc(neverFires)
+	svc.SetRetryConfig(notify.RetryConfig{
+		MaxRetries: 5,
+		BaseDelay:  time.Hour,
+		MaxDelay:   time.Hour,
+	})
+
+	svc.SendNotification(
+		context.Background(), "t", "m", prioError,
+	)
+
+	waitForCondition(t, func() bool {
+		return requests.Load() >= 1 &&
+			svc.OutstandingDeliveries() == 1
+	})
+
+	// start must be captured *before* the deadline clock starts,
+	// so that the measured interval is a superset of the deadline
+	// interval.  Capturing it after context.WithTimeout would
+	// make elapsed structurally smaller than drainDeadline and
+	// the lower bound below unfalsifiable-by-luck: it would fail
+	// whenever the two statements were separated by any
+	// scheduling delay, and pass otherwise, regardless of what
+	// the drain did.
+	start := time.Now()
+
+	ctx, cancel := context.WithTimeout(
+		context.Background(), drainDeadline,
+	)
+	defer cancel()
+
+	// The upper bound is enforced by a watchdog rather than by
+	// measuring after the fact: a drain that is not bounded at
+	// all never returns here (the delivery is parked in a backoff
+	// that never fires), so an unbounded drain must fail this
+	// test promptly instead of hanging the package until the test
+	// binary's 30s timeout.
+	returned := make(chan struct{})
+
+	go func() {
+		defer close(returned)
+
+		svc.Drain(ctx)
+	}()
+
+	select {
+	case <-returned:
+	case <-time.After(drainSlack):
+		t.Fatalf(
+			"drain did not return within %v; its %v deadline "+
+				"did not bound it",
+			drainSlack, drainDeadline,
+		)
+	}
+
+	// The lower bound is the real assertion: the drain must have
+	// waited for its whole deadline rather than giving up on the
+	// outstanding delivery early.  With start captured above, an
+	// early return is the only thing that can make it fail.
+	if elapsed := time.Since(start); elapsed < drainDeadline {
+		t.Errorf(
+			"drain returned after %v, before its %v deadline",
+			elapsed, drainDeadline,
+		)
+	}
+
+	assertAbandonLogged(t, logs.String())
+
+	// The abandoned delivery must stop retrying rather than
+	// outlive the drain.
+	waitForCondition(t, func() bool {
+		return svc.OutstandingDeliveries() == 0
+	})
+}
+
+// assertAbandonLogged checks that the drain logged the
+// abandoned deliveries at warn level with a count.
+func assertAbandonLogged(t *testing.T, output string) {
+	t.Helper()
+
+	if !strings.Contains(output, `"level":"WARN"`) {
+		t.Errorf(
+			"abandoned deliveries not logged at warn level; "+
+				"log output: %s",
+			output,
+		)
+	}
+
+	if !strings.Contains(output, `"abandoned":1`) {
+		t.Errorf(
+			"abandoned delivery count not logged; "+
+				"log output: %s",
+			output,
+		)
+	}
+}
+
+// TestDrainRefusesNewDeliveries verifies that notifications
+// submitted after the drain has begun are refused and logged,
+// so a stream of new work cannot extend shutdown indefinitely.
+func TestDrainRefusesNewDeliveries(t *testing.T) {
+	t.Parallel()
+
+	var requests atomic.Int64
+
+	srv := httptest.NewServer(
+		http.HandlerFunc(
+			func(w http.ResponseWriter, _ *http.Request) {
+				requests.Add(1)
+
+				w.WriteHeader(http.StatusOK)
+			}),
+	)
+	defer srv.Close()
+
+	target, _ := url.Parse(srv.URL)
+
+	svc, logs := newLoggingService(http.DefaultTransport)
+	svc.SetNtfyURL(target)
+	svc.SetSlackWebhookURL(target)
+	svc.SetMattermostWebhookURL(target)
+
+	ctx, cancel := context.WithTimeout(
+		context.Background(), drainSlack,
+	)
+	defer cancel()
+
+	// Nothing is in flight, so this returns immediately and
+	// leaves the service refusing further deliveries.
+	svc.Drain(ctx)
+
+	for range 3 {
+		svc.SendNotification(
+			context.Background(), "t", "m", prioInfo,
+		)
+	}
+
+	time.Sleep(settleDelay)
+
+	if got := requests.Load(); got != 0 {
+		t.Errorf(
+			"%d requests reached the endpoint after drain, "+
+				"want 0",
+			got,
+		)
+	}
+
+	if got := svc.OutstandingDeliveries(); got != 0 {
+		t.Errorf("outstanding deliveries = %d, want 0", got)
+	}
+
+	output := logs.String()
+	if !strings.Contains(output, "shutdown in progress") {
+		t.Errorf(
+			"refused deliveries not logged; log output: %s",
+			output,
+		)
+	}
+}
+
+// recordingLifecycle is a minimal fx.Lifecycle that records the
+// hooks appended to it, so the wiring done by notify.New can be
+// inspected without standing up a whole fx application.
+type recordingLifecycle struct {
+	hooks []fx.Hook
+}
+
+func (l *recordingLifecycle) Append(hook fx.Hook) {
+	l.hooks = append(l.hooks, hook)
+}
+
+// newNotifyService builds a Service through the real
+// constructor, wired to the given lifecycle.
+func newNotifyService(
+	t *testing.T,
+	lifecycle fx.Lifecycle,
+	ntfyTopic string,
+) *notify.Service {
+	t.Helper()
+
+	g, err := globals.New(nil)
+	if err != nil {
+		t.Fatalf("globals.New: %v", err)
+	}
+
+	log, err := logger.New(nil, logger.Params{Globals: g})
+	if err != nil {
+		t.Fatalf("logger.New: %v", err)
+	}
+
+	svc, err := notify.New(lifecycle, notify.Params{
+		Logger: log,
+		Config: &config.Config{NtfyTopic: ntfyTopic},
+	})
+	if err != nil {
+		t.Fatalf("notify.New: %v", err)
+	}
+
+	return svc
+}
+
+// TestNewRegistersDrainingStopHook verifies that notify.New
+// wires an OnStop hook into the fx lifecycle and that the hook
+// waits for in-flight deliveries.
+func TestNewRegistersDrainingStopHook(t *testing.T) {
+	t.Parallel()
+
+	var served atomic.Bool
+
+	entered := make(chan struct{})
+	release := make(chan struct{})
+
+	srv := blockingNtfyServer(entered, release, &served)
+	defer srv.Close()
+
+	lifecycle := &recordingLifecycle{}
+	svc := newNotifyService(t, lifecycle, srv.URL)
+
+	if len(lifecycle.hooks) != 1 {
+		t.Fatalf(
+			"appended %d lifecycle hooks, want 1",
+			len(lifecycle.hooks),
+		)
+	}
+
+	stop := lifecycle.hooks[0].OnStop
+	if stop == nil {
+		t.Fatal("lifecycle hook has no OnStop function")
+	}
+
+	svc.SendNotification(
+		context.Background(), "t", "m", prioInfo,
+	)
+
+	select {
+	case <-entered:
+	case <-time.After(drainSlack):
+		t.Fatal("delivery never reached the endpoint")
+	}
+
+	timer := time.AfterFunc(inFlightHold, func() {
+		close(release)
+	})
+	defer timer.Stop()
+
+	ctx, cancel := context.WithTimeout(
+		context.Background(), drainSlack,
+	)
+	defer cancel()
+
+	err := stop(ctx)
+	if err != nil {
+		t.Fatalf("OnStop returned error: %v", err)
+	}
+
+	if !served.Load() {
+		t.Error(
+			"OnStop returned before the in-flight delivery " +
+				"completed",
+		)
+	}
+}
+
+// TestDrainWithoutDeliveriesReturnsImmediately verifies the
+// common case: nothing in flight, shutdown is not delayed.
+func TestDrainWithoutDeliveriesReturnsImmediately(t *testing.T) {
+	t.Parallel()
+
+	svc := notify.NewTestService(http.DefaultTransport)
+
+	// Captured before the deadline clock, as elsewhere in this
+	// file; for an upper bound that is the conservative
+	// direction, since the measured interval can then only be
+	// longer than the drain itself.
+	start := time.Now()
+
+	ctx, cancel := context.WithTimeout(
+		context.Background(), drainSlack,
+	)
+	defer cancel()
+
+	svc.Drain(ctx)
+
+	if elapsed := time.Since(start); elapsed > idleDrainBound {
+		t.Errorf(
+			"drain of an idle service took %v, want well "+
+				"under its %v deadline",
+			elapsed, drainSlack,
+		)
+	}
+}
+
+// TestDrainWithCancelledContextDoesNotWarn verifies that an
+// OnStop context that is already dead on entry does not produce
+// an "abandoning them" warning when there was nothing in flight
+// to abandon.  The expired context wins the select immediately,
+// so only the outstanding count can tell the difference between
+// a genuine timeout and a shutdown that had simply already run
+// out of time with no work left.
+func TestDrainWithCancelledContextDoesNotWarn(t *testing.T) {
+	t.Parallel()
+
+	svc, logs := newLoggingService(http.DefaultTransport)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+
+	svc.Drain(ctx)
+
+	if output := logs.String(); strings.Contains(
+		output, `"level":"WARN"`,
+	) {
+		t.Errorf(
+			"drain with nothing in flight warned about "+
+				"abandoned deliveries; log output: %s",
+			output,
+		)
+	}
+}

internal/state/state_test.go

@@ -223,7 +223,8 @@ func TestSaveLoadRoundTrip_Ports(t *testing.T) {
 	}
 }
 
-// TestSaveLoadRoundTrip_Certificates verifies certificate data survives a save/load cycle.
+// TestSaveLoadRoundTrip_Certificates verifies certificate data
+// survives a save/load cycle.
 func TestSaveLoadRoundTrip_Certificates(t *testing.T) {
 	t.Parallel()
 
@@ -1072,7 +1073,8 @@ func TestConcurrentGetSet(t *testing.T) {
 	wg.Wait()
 }
 
-// runConcurrentOps performs a series of get/set/delete operations for concurrency testing.
+// runConcurrentOps performs a series of get/set/delete
+// operations for concurrency testing.
 func runConcurrentOps(s *state.State, key string, now time.Time) {
 	const iterations = 50
 

internal/watcher/watcher.go

@@ -26,8 +26,11 @@ const tlsPort = 443
 // hoursPerDay converts days to hours for duration calculations.
 const hoursPerDay = 24
 
-// statusError is the status value recorded for failed checks.
+// Status values recorded for nameserver and certificate checks.
-const statusError = "error"
+const (
+	statusOK    = "ok"
+	statusError = "error"
+)
 
 // Params contains dependencies for Watcher.
 type Params struct {
@@ -347,7 +350,7 @@ func buildHostnameState(
 	for ns, recs := range records {
 		hs.RecordsByNameserver[ns] = &state.NameserverRecordState{
 			Records:     recs,
-			Status:      "ok",
+			Status:      statusOK,
 			LastChecked: now,
 		}
 	}
@@ -405,7 +408,7 @@ func (w *Watcher) detectNSDisappearances(
 	current map[string]map[string][]string,
 ) {
 	for ns, prevNS := range prev.RecordsByNameserver {
-		if _, ok := current[ns]; ok || prevNS.Status != "ok" {
+		if _, ok := current[ns]; ok || prevNS.Status != statusOK {
 			continue
 		}
 
@@ -708,7 +711,7 @@ func (w *Watcher) handleTLSError(
 	now time.Time,
 	err error,
 ) {
-	if hasPrev && !w.firstRun && prev.Status == "ok" {
+	if hasPrev && !w.firstRun && prev.Status == statusOK {
 		msg := fmt.Sprintf(
 			"Host: %s\nIP: %s\nError: %s",
 			hostname, ip, err,
@@ -751,7 +754,7 @@ func (w *Watcher) handleTLSSuccess(
 			Issuer:                  cert.Issuer,
 			NotAfter:                cert.NotAfter,
 			SubjectAlternativeNames: cert.SubjectAlternativeNames,
-			Status:                  "ok",
+			Status:                  statusOK,
 			LastChecked:             now,
 		},
 	)