Add a Download button that exports a database target's archive as gzipped JSON (closes #374)

Each database target on the webhook page links to
/hook/ID/targets/TARGETID/download, which streams the target's archive
as archive-WEBHOOK-TARGET-YYYYMMDDTHHMMSSZ.json.gz: the webhook, the
target, exported_at, and archived_events, one object per row keyed by
column, a body that is not valid UTF-8 in base64 with body_encoding.
The export reads one row at a time on its own connection inside a
read-only transaction: one snapshot, and no write lock. The handler
holds the rename lock only while it reads the names and opens the file.
A missing archive exports empty and is not created.

Model: opus-5-5

.gitea/workflows/check.yml

@@ -33,5 +33,5 @@ jobs:
         # report success from cache.
         run: git rev-parse HEAD > .ci-fingerprint
 
-      - name: Build Docker image (runs make fmt-check, golangci-lint, make test, make build)
+      - name: Build Docker image (runs make check)
         run: script/cibuild

README.md

@@ -19,8 +19,8 @@ before deploying one.
 ### Prerequisites
 
 - Go 1.26.1+ (the version in `go.mod`)
-- Docker (for `make lint` and so for `make check`, for the CI gate, and
-  for containerized deployment)
+- Docker (for linting, for the test stage of the CI gate, and for
+  containerized deployment)
 
 golangci-lint is not a prerequisite and must not be installed on the
 host: `script/bootstrap` does not install it, and `make lint` runs the
@@ -456,19 +456,6 @@ Your proxy must therefore **append** the peer address to
 `option forwardfor`, Caddy and AWS ALB by default), and must append a
 bare address with no port.
 
-Every log line that names a client carries two addresses: `remoteIP`,
-the connecting peer, which behind a proxy is the proxy; and `clientIP`,
-the client the rate limiters identify by the rules above, which is the
-field to read when tracing who sent what. Those lines are the
-`http request` access log line, the rate-limit rejection lines
-(`login failure limit exceeded` among them), the
-`csrf: token validation failed` warning and the receiver's
-`webhook request received` line. `clientIP` is only as trustworthy as
-`TRUSTED_PROXIES`: for a request from a peer inside the list, it is
-read out of the `X-Forwarded-For` that peer sent, so a peer that does
-not belong in the list can make it name any address it likes. For a
-request from any other peer, both fields name the peer.
-
 #### Sessions
 
 Sessions are bounded by two independent clocks, and end at whichever
@@ -853,9 +840,9 @@ reports.
    was given, so on any port other than 443 `$host` makes every form
    POST — including login — fail with `403 origin invalid`, with
    nothing in the error naming the cause.
-5. **Keep the proxy's access log.** webhooker's own access log names
-   the client in its `clientIP` field only while `TRUSTED_PROXIES`
-   covers the proxy; the proxy's log names it regardless. nginx's
+5. **Keep the proxy's access log.** webhooker's own access log records
+   the peer address, which behind a proxy is always the proxy. The
+   proxy's log is the only record of which client sent what. nginx's
    default `combined` format already logs `$remote_addr`; do not
    replace it with one that drops the client address, and retain those
    logs as long as you would want to answer a question about traffic.
@@ -884,8 +871,9 @@ server {
     # webhooker's message.
     client_max_body_size 1m;
 
-    # $remote_addr is the client. webhooker's own log names it, as
-    # clientIP, only while TRUSTED_PROXIES covers this proxy.
+    # $remote_addr is the client. webhooker's own log records this
+    # proxy and nothing else, so this file is the only place the
+    # client's address is written down.
     access_log /var/log/nginx/webhooker.access.log combined;
 
     location / {
@@ -2004,6 +1992,30 @@ Because each `database` target has its own archive file, a target's
 webhook with different expiries keep two archives, each pruned on its
 own schedule.
 
+Each `database` target on the webhook page has a **Download** button,
+which returns its archive as one gzipped JSON file,
+`archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`, the
+names made safe as above and the time in UTC. The file holds one
+object: `webhook` and `target`, each an `id` and a `name`;
+`exported_at`; and `archived_events`, one object per archived row with
+every column, keyed by column name. A body that is not valid UTF-8 is
+written in base64, with `"body_encoding": "base64"` beside it. An
+archive that does not exist yet, or was moved away, downloads with an
+empty `archived_events`; the download never creates the file.
+
+The download streams: each row is read and written out compressed
+before the next is read, so neither the archive nor the JSON is held in
+memory. It reads on a connection of its own, inside one read-only
+transaction, so the file holds the archive as it stood when the
+download started, and archive writes go on meanwhile, since under WAL a
+reader never blocks a writer. While it runs, the `-wal` cannot be
+checkpointed past what it reads, so a long download lets the `-wal`
+grow. It finds the file by the stored names under the lock that webhook
+edits, target edits and target creation hold, and lets go once the file
+is open: a rename during the download moves the file without affecting
+it. Like every request it is cut off after 60 seconds, which leaves the
+file incomplete and failing to decompress.
+
 Deleting a webhook releases its archives: the delivery engine's cached
 archive writers are dropped and their file handles closed, so nothing
 lingers after the webhook is gone. The archive **files themselves are
@@ -2460,20 +2472,20 @@ trade.
 Net: **one `INFO` line per request, of at most 2,560 bytes.** That
 ceiling is arithmetic, not an observation: 3 × (512 + 11) for `url`,
 `useragent` and `referer`, plus 128 + 11 for `request_id`, plus 32 + 11
-for `method`, plus a 405-byte fixed portion (the field names, the
-punctuation, both timestamps at their longest, `remoteIP` and
-`clientIP` each charged as an IPv6 address with a zone, the status and
-the latency) — 2,156 bytes, stated at 2,560 so the figure has headroom.
-`internal/middleware/accesslog_test.go` asserts it against 8 KB of
-client-chosen text in the path, in the query, and in each of
-`User-Agent`, `Referer` and `X-Request-Id`,
+for `method`, plus a 336-byte fixed portion (the field names, the
+punctuation, both timestamps at their longest, an IPv6 `remoteIP` with
+a zone, the status and the latency) — 2,087 bytes, stated at 2,560 so
+the figure has headroom. `internal/middleware/accesslog_test.go`
+asserts it against 8 KB of client-chosen text in the path, in the
+query, and in each of `User-Agent`, `Referer` and `X-Request-Id`,
 including cases built from the characters the handlers escape, and
 against the widest access log line the service can be made to write: a
 5xx that keeps its concrete path while all three header fields are also
 at their budget. Every case runs through both handlers
 `internal/logger` can select — the JSON one and the text one it installs
 on a tty — since the two do not escape alike and the ceiling is quoted
-unqualified.
+unqualified. Measured over a real connection, the widest access log line
+is 1,972 bytes.
 
 Multiply that ceiling by the request rate to size log storage. Note
 that the rate is not bounded by the limits above on every route:
@@ -2811,9 +2823,9 @@ remedies are to block the source at the reverse proxy, or to
 rate-limit `POST /pages/login` there — the one place a limit can be
 applied without reintroducing the lockout, because the proxy sees the
 real client address. `TRUSTED_PROXIES` does not stop the saturation.
-The flood's source is in the `clientIP` field of webhooker's access
-log while `TRUSTED_PROXIES` covers the proxy, and in the proxy's own
-access log either way (see [Trusted proxies](#trusted-proxies)).
+The flood's source is in the proxy's access log: webhooker's own logs
+record the proxy's address, not the client's (see
+[Deployment behind a reverse proxy](#deployment-behind-a-reverse-proxy)).
 
 Finer-grained per-webhook rate limits (configured in the web UI and
 enforced in the webhook handler) can layer on top of this env-level
@@ -2866,6 +2878,7 @@ returns to the page that was asked for.
 | `POST` | `/hook/{id}/targets`     | Add target to webhook |
 | `GET`  | `/hook/{id}/targets/{targetID}/edit` | Edit target form. The one page that renders a target's destination URL and header values in full, rather than masked |
 | `POST` | `/hook/{id}/targets/{targetID}/edit` | Edit target submission |
+| `GET`  | `/hook/{id}/targets/{targetID}/download` | Download a `database` target's archive as one gzipped JSON file. See [Database Architecture](#database-architecture) |
 | `POST` | `/hook/{id}/targets/{targetID}/delete` | Delete a target |
 | `POST` | `/hook/{id}/targets/{targetID}/toggle` | Enable or disable a target |
 
@@ -2946,6 +2959,7 @@ webhooker/
 │   │   ├── target_slack.go    # Slack/Mattermost incoming-webhook target
 │   │   ├── target_database.go # Database archive target
 │   │   ├── target_database_archive.go  # Archive file lifecycle and pruning
+│   │   ├── target_database_export.go   # Archive download as gzipped JSON
 │   │   ├── target_log.go      # Log target (stdout)
 │   │   ├── target_config_view.go  # Masked target config for templates
 │   │   ├── archive_sweeper.go # Periodic pruning of idle archives
@@ -3050,7 +3064,7 @@ Applied to all routes in this order:
    (HSTS, X-Content-Type-Options, X-Frame-Options, CSP, Referrer-Policy,
    Permissions-Policy)
 3. **Logging** — Structured request logging (method, URL, status,
-   latency, remote IP, client IP, user agent, request ID)
+   latency, remote IP, user agent, request ID)
 4. **Metrics** — Prometheus HTTP metrics (if `METRICS_USERNAME` and
    `METRICS_PASSWORD` are both set)
 5. **CORS** — Cross-origin resource sharing headers
@@ -3347,9 +3361,8 @@ linked, which is what lets it run on the Alpine runtime image.
 inside the image, so a build that succeeds is a repo that is formatted,
 linted, tested and compiled. `script/lint` also uses Docker
 (`Dockerfile.lint`, see Linting above), so `make lint` and `make check`
-run the same pinned linter version the gate does; of the steps
-`make check` runs, only `script/test` and `script/fmt-check` run on the
-host.
+run the same pinned linter version the gate does; only `script/test`
+and `script/fmt-check` run on the host.
 
 #### CI gate honesty
 

internal/delivery/target_database.go

@@ -3,7 +3,6 @@ package delivery
 import (
 	"context"
 	"fmt"
-	"path/filepath"
 	"strings"
 	"sync"
 	"time"
@@ -277,10 +276,9 @@ func (t *databaseTarget) releaseSweepWriter(
 }
 
 // newWriter builds the writer for a database target's archive. The
-// file lives beside the webhook's event database in the data
-// directory and is named for the webhook and the target as the main
-// database has them now; from then on only rename changes the name
-// the writer uses. It does not touch the archive file.
+// file is the one ArchivePath gives for the webhook and the target as
+// the main database names them now; from then on only rename changes
+// the name the writer uses. It does not touch the archive file.
 func (t *databaseTarget) newWriter(
 	targetID string,
 ) (*archiveWriter, error) {
@@ -299,12 +297,10 @@ func (t *databaseTarget) newWriter(
 		)
 	}
 
-	dir := filepath.Dir(t.eng.dbManager.DBPath(target.WebhookID))
-	name := ArchiveFileName(
-		target.Webhook.Name, target.Name, target.ID,
+	w := newArchiveWriter(
+		ArchivePath(t.eng.dbManager, &target.Webhook, &target),
+		t.eng.log,
 	)
-
-	w := newArchiveWriter(filepath.Join(dir, name), t.eng.log)
 	w.webhookID = target.WebhookID
 
 	return w, nil

internal/delivery/target_database_export.go

@@ -0,0 +1,270 @@
+package delivery
+
+import (
+	"compress/gzip"
+	"context"
+	"database/sql"
+	"encoding/base64"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"path/filepath"
+	"time"
+	"unicode/utf8"
+
+	"sneak.berlin/go/webhooker/internal/database"
+)
+
+// archiveTableQuery counts the archive's table: 0 when the archive
+// writer has created the file but not yet the table in it.
+const archiveTableQuery = "SELECT count(*) FROM sqlite_master " +
+	"WHERE type = 'table' AND name = 'archived_events'"
+
+// archiveNextRowQuery reads every column of the first archived row
+// after a given id. An export reads the archive a row at a time this
+// way rather than through one cursor, because the Scan check in
+// internal/gormlog accepts only a Scan straight on QueryRowContext's
+// result.
+const archiveNextRowQuery = "SELECT id, event_id, webhook_id, " +
+	"entrypoint_id, method, headers, body, content_type, archived_at " +
+	"FROM archived_events WHERE id > ? ORDER BY id LIMIT 1"
+
+// ArchivePath returns where a database target's archive file is: in
+// the data directory, beside the webhook's event database, under the
+// name ArchiveFileName gives it.
+func ArchivePath(
+	dbMgr *database.WebhookDBManager,
+	webhook *database.Webhook,
+	target *database.Target,
+) string {
+	return filepath.Join(
+		filepath.Dir(dbMgr.DBPath(webhook.ID)),
+		ArchiveFileName(webhook.Name, target.Name, target.ID),
+	)
+}
+
+// ArchiveExportFileName returns the name a database target's archive
+// downloads under:
+// archive-WEBHOOKNAME-TARGETNAME-YYYYMMDDTHHMMSSZ.json.gz, the names
+// made safe as in ArchiveFileName and the time in UTC.
+func ArchiveExportFileName(
+	webhookName, targetName string, at time.Time,
+) string {
+	return "archive-" + archiveNamePart(webhookName) + "-" +
+		archiveNamePart(targetName) + "-" +
+		at.UTC().Format("20060102T150405Z") + ".json.gz"
+}
+
+// ArchiveExport is a database target's archive opened for download.
+// It reads the file on its own connection, inside one read-only
+// transaction, so it writes out the archive as it stood when
+// OpenArchiveExport returned.
+//
+// Archives are in WAL mode, where a reader works from a snapshot and
+// never blocks a writer: archive writes go on while an export is open,
+// and the export does not see them. SQLite cannot checkpoint the -wal
+// past an open snapshot, so the -wal grows until the export is closed.
+type ArchiveExport struct {
+	db *sql.DB
+	tx *sql.Tx
+
+	// empty is true when there is nothing to read: no file, or a file
+	// without the archive's table yet.
+	empty bool
+}
+
+// exportedName is how an export names its webhook and its target.
+type exportedName struct {
+	ID   string `json:"id"`
+	Name string `json:"name"`
+}
+
+// OpenArchiveExport opens the archive file at path for export and
+// takes the snapshot the export reads. It never creates the file: with
+// no file at path, the export has no rows.
+//
+// Once it has returned, the file is open, so a rename or a move of it
+// does not affect the export, which reads the same file under its new
+// name.
+func OpenArchiveExport(
+	ctx context.Context, path string,
+) (*ArchiveExport, error) {
+	if !fileExists(path) {
+		return &ArchiveExport{empty: true}, nil
+	}
+
+	db, err := database.OpenSQLite(path, archiveModeExisting)
+	if err != nil {
+		return nil, fmt.Errorf("opening archive %s: %w", path, err)
+	}
+
+	// ReadOnly makes the driver begin a deferred transaction in place
+	// of the BEGIN IMMEDIATE the connection string asks for, so the
+	// export never takes the archive's write lock.
+	tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
+	if err != nil {
+		_ = db.Close()
+
+		return nil, fmt.Errorf("reading archive %s: %w", path, err)
+	}
+
+	// The transaction's first read is what takes the snapshot.
+	var tables int
+
+	err = tx.QueryRowContext(ctx, archiveTableQuery).Scan(&tables)
+	if err != nil {
+		_ = tx.Rollback()
+		_ = db.Close()
+
+		return nil, fmt.Errorf("reading archive %s: %w", path, err)
+	}
+
+	return &ArchiveExport{db: db, tx: tx, empty: tables == 0}, nil
+}
+
+// WriteGzipJSON writes the export to w as one gzipped JSON object:
+// webhook and target, each an id and a name; exported_at; and
+// archived_events, one object per archived row, keyed by column name.
+// A body that is not valid UTF-8 cannot be a JSON string, so it is
+// written in base64, with "body_encoding": "base64" beside it.
+//
+// Each row is written out before the next is read, so neither the
+// archive nor its JSON is ever held in memory whole. After an error
+// the gzip stream is left unfinished, so what was written does not
+// decompress as a whole file.
+func (x *ArchiveExport) WriteGzipJSON(
+	ctx context.Context,
+	w io.Writer,
+	webhook *database.Webhook,
+	target *database.Target,
+	exportedAt time.Time,
+) error {
+	head, err := json.Marshal(map[string]any{
+		"webhook":     exportedName{ID: webhook.ID, Name: webhook.Name},
+		"target":      exportedName{ID: target.ID, Name: target.Name},
+		"exported_at": exportedAt.UTC(),
+	})
+	if err != nil {
+		return fmt.Errorf("encoding archive export: %w", err)
+	}
+
+	zw := gzip.NewWriter(w)
+
+	err = x.writeJSON(ctx, zw, head)
+	if err != nil {
+		return fmt.Errorf("writing archive export: %w", err)
+	}
+
+	return zw.Close()
+}
+
+// Close ends the export's transaction and closes its connection.
+func (x *ArchiveExport) Close() error {
+	if x.db == nil {
+		return nil
+	}
+
+	_ = x.tx.Rollback()
+
+	return x.db.Close()
+}
+
+// writeJSON writes head with archived_events added as its last key,
+// the rows going into it one at a time.
+func (x *ArchiveExport) writeJSON(
+	ctx context.Context, w io.Writer, head []byte,
+) error {
+	// head goes out without its closing brace, so that
+	// archived_events can follow it.
+	_, err := w.Write(head[:len(head)-1])
+	if err != nil {
+		return err
+	}
+
+	_, err = io.WriteString(w, `,"archived_events":[`)
+	if err != nil {
+		return err
+	}
+
+	err = x.writeRows(ctx, w)
+	if err != nil {
+		return err
+	}
+
+	_, err = io.WriteString(w, "\n]}\n")
+
+	return err
+}
+
+// writeRows writes each archived row to w, oldest first, one per line,
+// separated by commas.
+func (x *ArchiveExport) writeRows(ctx context.Context, w io.Writer) error {
+	if x.empty {
+		return nil
+	}
+
+	next, err := x.tx.PrepareContext(ctx, archiveNextRowQuery)
+	if err != nil {
+		return err
+	}
+
+	defer func() { _ = next.Close() }()
+
+	var ev archivedEvent
+
+	for sep := "\n"; ; sep = ",\n" {
+		err = next.QueryRowContext(ctx, ev.ID).Scan(
+			&ev.ID, &ev.EventID, &ev.WebhookID, &ev.EntrypointID,
+			&ev.Method, &ev.Headers, &ev.Body, &ev.ContentType,
+			&ev.ArchivedAt,
+		)
+		if errors.Is(err, sql.ErrNoRows) {
+			return nil
+		}
+
+		if err != nil {
+			return err
+		}
+
+		_, err = io.WriteString(w, sep)
+		if err != nil {
+			return err
+		}
+
+		err = writeRow(w, &ev)
+		if err != nil {
+			return err
+		}
+	}
+}
+
+// writeRow writes an archived row to w as a JSON object keyed by
+// column name, its body in base64 when it is not valid UTF-8.
+func writeRow(w io.Writer, ev *archivedEvent) error {
+	row := map[string]any{
+		"id":            ev.ID,
+		"event_id":      ev.EventID,
+		"webhook_id":    ev.WebhookID,
+		"entrypoint_id": ev.EntrypointID,
+		"method":        ev.Method,
+		"headers":       ev.Headers,
+		"body":          ev.Body,
+		"content_type":  ev.ContentType,
+		"archived_at":   ev.ArchivedAt.UTC(),
+	}
+
+	if !utf8.ValidString(ev.Body) {
+		row["body"] = base64.StdEncoding.EncodeToString([]byte(ev.Body))
+		row["body_encoding"] = "base64"
+	}
+
+	line, err := json.Marshal(row)
+	if err != nil {
+		return err
+	}
+
+	_, err = w.Write(line)
+
+	return err
+}

internal/delivery/target_database_export_test.go

@@ -0,0 +1,373 @@
+package delivery_test
+
+import (
+	"bytes"
+	"compress/gzip"
+	"encoding/base64"
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"runtime"
+	"runtime/debug"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"sneak.berlin/go/webhooker/internal/database"
+	"sneak.berlin/go/webhooker/internal/delivery"
+)
+
+// The webhook and the target the export tests' archives belong to.
+const (
+	exportWebhookID   = "wh-export"
+	exportWebhookName = "Orders (EU)"
+	exportTargetID    = "tgt-export"
+	exportTargetName  = "Long-term archive"
+)
+
+const (
+	// binaryBody is a body that is not valid UTF-8.
+	binaryBody = "\xff\xfe\x00\x01binary\x80"
+
+	// openedEventID is the event the snapshot tests archive before
+	// they open the export.
+	openedEventID = "opened"
+)
+
+// writeExportTo writes export to w as the archive of the export tests'
+// webhook and target, exported at 2026-10-02T12:03:04Z.
+func writeExportTo(
+	t *testing.T, export *delivery.ArchiveExport, w io.Writer,
+) error {
+	t.Helper()
+
+	return export.WriteGzipJSON(
+		t.Context(), w,
+		&database.Webhook{
+			BaseModel: database.BaseModel{ID: exportWebhookID},
+			Name:      exportWebhookName,
+		},
+		&database.Target{
+			BaseModel: database.BaseModel{ID: exportTargetID},
+			Name:      exportTargetName,
+		},
+		time.Date(2026, 10, 2, 12, 3, 4, 0, time.UTC),
+	)
+}
+
+// exportArchive runs a whole export of the archive at path and returns
+// its JSON, decompressed and parsed.
+func exportArchive(t *testing.T, path string) map[string]any {
+	t.Helper()
+
+	export, err := delivery.OpenArchiveExport(t.Context(), path)
+	require.NoError(t, err)
+
+	defer func() { require.NoError(t, export.Close()) }()
+
+	return writeExport(t, export)
+}
+
+// writeExport writes an opened export and returns its JSON,
+// decompressed and parsed. Reading to the end makes the gzip reader
+// check that the stream was finished.
+func writeExport(
+	t *testing.T, export *delivery.ArchiveExport,
+) map[string]any {
+	t.Helper()
+
+	var buf bytes.Buffer
+
+	require.NoError(t, writeExportTo(t, export, &buf))
+
+	zr, err := gzip.NewReader(&buf)
+	require.NoError(t, err)
+
+	raw, err := io.ReadAll(zr)
+	require.NoError(t, err)
+
+	var got map[string]any
+
+	require.NoError(t, json.Unmarshal(raw, &got))
+
+	return got
+}
+
+// exportedEvents returns an export's archived_events.
+func exportedEvents(t *testing.T, got map[string]any) []map[string]any {
+	t.Helper()
+
+	list, ok := got["archived_events"].([]any)
+	require.True(t, ok, "archived_events must be an array: %v", got)
+
+	events := make([]map[string]any, len(list))
+
+	for i, v := range list {
+		events[i], ok = v.(map[string]any)
+		require.True(t, ok, "an archived event must be an object: %v", v)
+	}
+
+	return events
+}
+
+// exportedEventIDs returns the event_id of each of an export's
+// archived_events.
+func exportedEventIDs(t *testing.T, got map[string]any) []string {
+	t.Helper()
+
+	events := exportedEvents(t, got)
+	ids := make([]string, 0, len(events))
+
+	for _, ev := range events {
+		ids = append(ids, fmt.Sprint(ev["event_id"]))
+	}
+
+	return ids
+}
+
+// TestArchiveExport_MatchesStoredRows proves an export holds the
+// webhook, the target, the time, and every column of every stored
+// row: a body that is valid UTF-8 as a string, and one that is not in
+// base64, marked as such.
+func TestArchiveExport_MatchesStoredRows(t *testing.T) {
+	t.Parallel()
+
+	path := filepath.Join(t.TempDir(), "archive.db")
+	w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
+	bodies := []string{`{"order":1}`, "plain text", "", binaryBody}
+
+	for i, body := range bodies {
+		require.NoError(t, w.Write(delivery.ExportArchivedEvent{
+			EventID:      fmt.Sprintf("ev-%d", i),
+			WebhookID:    exportWebhookID,
+			EntrypointID: "ep-1",
+			Method:       "POST",
+			Headers:      `{"X-Test":["yes"]}`,
+			Body:         body,
+			ContentType:  testContentType,
+		}, 0))
+	}
+
+	var stored []delivery.ExportArchivedEvent
+
+	require.NoError(t, openArchiveDBForRead(t, path).
+		Order("id").Find(&stored).Error)
+
+	got := exportArchive(t, path)
+
+	assert.Equal(t,
+		map[string]any{"id": exportWebhookID, "name": exportWebhookName},
+		got["webhook"],
+	)
+	assert.Equal(t,
+		map[string]any{"id": exportTargetID, "name": exportTargetName},
+		got["target"],
+	)
+	assert.Equal(t, "2026-10-02T12:03:04Z", got["exported_at"])
+
+	events := exportedEvents(t, got)
+	require.Len(t, events, len(bodies))
+
+	for i, row := range stored {
+		assertExportedRow(t, row, events[i])
+	}
+}
+
+// assertExportedRow checks that ev, from an export, holds every column
+// of the stored row.
+func assertExportedRow(
+	t *testing.T, row delivery.ExportArchivedEvent, ev map[string]any,
+) {
+	t.Helper()
+
+	archivedAt, err := time.Parse(
+		time.RFC3339Nano, fmt.Sprint(ev["archived_at"]),
+	)
+	require.NoError(t, err)
+	assert.True(t, archivedAt.Equal(row.ArchivedAt))
+
+	assert.EqualValues(t, row.ID, ev["id"])
+	assert.Equal(t, row.EventID, ev["event_id"])
+	assert.Equal(t, row.WebhookID, ev["webhook_id"])
+	assert.Equal(t, row.EntrypointID, ev["entrypoint_id"])
+	assert.Equal(t, row.Method, ev["method"])
+	assert.Equal(t, row.Headers, ev["headers"])
+	assert.Equal(t, row.ContentType, ev["content_type"])
+
+	if row.Body != binaryBody {
+		assert.Equal(t, row.Body, ev["body"])
+		assert.Len(t, ev, 9, "the nine columns and nothing else: %v", ev)
+
+		return
+	}
+
+	body, err := base64.StdEncoding.DecodeString(fmt.Sprint(ev["body"]))
+	require.NoError(t, err)
+	assert.Equal(t, binaryBody, string(body))
+	assert.Equal(t, "base64", ev["body_encoding"])
+	assert.Len(t, ev, 10, "the nine columns and body_encoding: %v", ev)
+}
+
+// TestArchiveExport_Empty proves an archive with nothing in it exports
+// as an empty archived_events: no file, which the export must not
+// create; a file the archive writer has not yet put its table in; and
+// a table with no rows.
+func TestArchiveExport_Empty(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	missing := filepath.Join(dir, "missing.db")
+	noTable := filepath.Join(dir, "no-table.db")
+	noRows := filepath.Join(dir, "no-rows.db")
+
+	require.NoError(t, os.WriteFile(noTable, nil, 0o600))
+	require.NoError(t,
+		delivery.NewExportArchiveWriter(noRows, archiveTestLogger(), 0).
+			Open(0),
+	)
+
+	for _, path := range []string{missing, noTable, noRows} {
+		assert.Empty(t, exportedEvents(t, exportArchive(t, path)), path)
+	}
+
+	for _, suffix := range archiveFileSuffixes() {
+		assert.NoFileExists(t, missing+suffix)
+	}
+}
+
+// TestArchiveExport_ReadsOneSnapshot proves an export writes the
+// archive as it was when it was opened, and holds up no archive
+// write: a row written while the export is open is stored, and is not
+// in the export. A write held up for the whole busy timeout would
+// fail.
+func TestArchiveExport_ReadsOneSnapshot(t *testing.T) {
+	t.Parallel()
+
+	path := filepath.Join(t.TempDir(), "archive.db")
+	w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
+
+	require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: openedEventID}, 0))
+
+	export, err := delivery.OpenArchiveExport(t.Context(), path)
+	require.NoError(t, err)
+
+	defer func() { require.NoError(t, export.Close()) }()
+
+	require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "during"}, 0))
+
+	assert.Equal(t,
+		[]string{openedEventID}, exportedEventIDs(t, writeExport(t, export)),
+	)
+
+	var stored int64
+
+	require.NoError(t, openArchiveDBForRead(t, path).
+		Model(&delivery.ExportArchivedEvent{}).Count(&stored).Error)
+	assert.Equal(t, int64(2), stored)
+}
+
+// TestArchiveExport_SurvivesRename proves that renaming the archive
+// while an export of it is open, as renaming its webhook or target
+// does, leaves the export reading the same file.
+func TestArchiveExport_SurvivesRename(t *testing.T) {
+	t.Parallel()
+
+	path := filepath.Join(t.TempDir(), "archive-old.db")
+	w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
+
+	require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: openedEventID}, 0))
+
+	export, err := delivery.OpenArchiveExport(t.Context(), path)
+	require.NoError(t, err)
+
+	defer func() { require.NoError(t, export.Close()) }()
+
+	require.NoError(t, w.Rename("archive-new.db"))
+	require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "after"}, 0))
+	require.NoFileExists(t, path)
+
+	assert.Equal(t,
+		[]string{openedEventID}, exportedEventIDs(t, writeExport(t, export)),
+	)
+}
+
+// heapPeak is an io.Writer that discards what it is given and records
+// the largest heap it saw at a write.
+type heapPeak struct {
+	max uint64
+}
+
+func (p *heapPeak) Write(b []byte) (int, error) {
+	var m runtime.MemStats
+
+	runtime.ReadMemStats(&m)
+	p.max = max(p.max, m.HeapAlloc)
+
+	return len(b), nil
+}
+
+// TestArchiveExport_Streams proves an export holds neither the archive
+// nor its JSON in memory whole: exporting a 16 MiB archive grows the
+// heap by less than half of that. An export that read every row before
+// writing, or built the JSON before writing it, would hold all 16 MiB
+// at a write.
+//
+//nolint:paralleltest // It measures the heap, which tests share.
+func TestArchiveExport_Streams(t *testing.T) {
+	const (
+		rows     = 64
+		bodySize = 256 << 10
+		limit    = rows * bodySize / 2
+	)
+
+	path := filepath.Join(t.TempDir(), "archive.db")
+	w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
+	body := strings.Repeat("x", bodySize)
+
+	for range rows {
+		require.NoError(t, w.Write(delivery.ExportArchivedEvent{Body: body}, 0))
+	}
+
+	export, err := delivery.OpenArchiveExport(t.Context(), path)
+	require.NoError(t, err)
+
+	defer func() { require.NoError(t, export.Close()) }()
+
+	// A low GC target collects garbage soon after it is made, so the
+	// heap at each write is close to what the export is holding.
+	defer debug.SetGCPercent(debug.SetGCPercent(10))
+
+	runtime.GC()
+
+	var start runtime.MemStats
+
+	runtime.ReadMemStats(&start)
+
+	peak := &heapPeak{}
+
+	require.NoError(t, writeExportTo(t, export, peak))
+	assert.Less(t, peak.max, start.HeapAlloc+limit,
+		"heap at the start %d, at its peak %d", start.HeapAlloc, peak.max,
+	)
+}
+
+// TestArchiveExportFileName proves the download is named for the
+// webhook and the target, with the names made safe as for the archive
+// file, and the export time in UTC.
+func TestArchiveExportFileName(t *testing.T) {
+	t.Parallel()
+
+	cest := time.FixedZone("CEST", int((2 * time.Hour).Seconds()))
+
+	assert.Equal(t,
+		"archive-orders-eu-long-term-archive-20261002T120304Z.json.gz",
+		delivery.ArchiveExportFileName(
+			exportWebhookName, exportTargetName,
+			time.Date(2026, 10, 2, 14, 3, 4, 0, cest),
+		),
+	)
+}

internal/handlers/handlers.go

@@ -97,7 +97,8 @@ type Handlers struct {
 	// names through the archive rename, the save and any move back.
 	// Interleaved, one could rename an archive between another's
 	// rename and save, leaving the file named for one edit and the
-	// stored names from the other.
+	// stored names from the other. An archive download holds it while
+	// it reads the stored names and opens the file they give.
 	renameMu sync.Mutex
 
 	// dummyVerifications counts the equivalent-cost verifications

internal/handlers/target_download.go

@@ -0,0 +1,82 @@
+package handlers
+
+import (
+	"net/http"
+	"time"
+
+	"sneak.berlin/go/webhooker/internal/database"
+	"sneak.berlin/go/webhooker/internal/delivery"
+)
+
+// HandleTargetDownload serves a database target's archive as one
+// gzipped JSON file, named for the webhook, the target and the time;
+// see delivery.ArchiveExport.WriteGzipJSON for what it holds. Other
+// target types have no archive and are a 404.
+func (h *Handlers) HandleTargetDownload() http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		webhook, target, export, ok := h.openTargetArchive(w, r)
+		if !ok {
+			return
+		}
+
+		defer func() { _ = export.Close() }()
+
+		now := time.Now()
+
+		w.Header().Set("Content-Type", "application/gzip")
+		w.Header().Set(
+			"Content-Disposition",
+			`attachment; filename="`+delivery.ArchiveExportFileName(
+				webhook.Name, target.Name, now,
+			)+`"`,
+		)
+
+		err := export.WriteGzipJSON(r.Context(), w, &webhook, target, now)
+		if err != nil {
+			// The 200 has gone out. The client is left with a file
+			// that does not decompress; the log is the record.
+			h.log.Error(
+				"failed to export archive",
+				"target_id", target.ID,
+				"error", err,
+			)
+		}
+	}
+}
+
+// openTargetArchive opens the archive of the request's database target
+// for export. It reports false once it has written the response.
+//
+// It holds renameMu, which every archive rename runs under, while it
+// reads the stored names and opens the file, so the file it opens is
+// the one those names give. It lets go before the export is streamed:
+// once the file is open, a rename does not affect the export.
+func (h *Handlers) openTargetArchive(
+	w http.ResponseWriter,
+	r *http.Request,
+) (database.Webhook, *database.Target, *delivery.ArchiveExport, bool) {
+	h.renameMu.Lock()
+	defer h.renameMu.Unlock()
+
+	webhook, target, ok := h.ownedTarget(w, r)
+	if !ok {
+		return database.Webhook{}, nil, nil, false
+	}
+
+	if target.Type != database.TargetTypeDatabase {
+		h.renderError(w, r, http.StatusNotFound)
+
+		return database.Webhook{}, nil, nil, false
+	}
+
+	export, err := delivery.OpenArchiveExport(
+		r.Context(), delivery.ArchivePath(h.dbMgr, &webhook, target),
+	)
+	if err != nil {
+		h.serverError(w, r, "failed to open archive for export", err)
+
+		return database.Webhook{}, nil, nil, false
+	}
+
+	return webhook, target, export, true
+}

internal/handlers/target_download_test.go

@@ -0,0 +1,185 @@
+package handlers_test
+
+import (
+	"compress/gzip"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"sneak.berlin/go/webhooker/internal/database"
+)
+
+// downloadPath is the archive download route of a target.
+func downloadPath(webhookID, targetID string) string {
+	return "/hook/" + webhookID + "/targets/" + targetID + "/download"
+}
+
+// renameTarget submits the edit form renaming a target to Renamed.
+func renameTarget(
+	env *sourceTestEnv, webhookID, targetID string,
+) *httptest.ResponseRecorder {
+	form := url.Values{}
+	form.Set("name", "Renamed")
+
+	return submitTargetEdit(env, webhookID, targetID, form)
+}
+
+// TestHandleTargetDownload proves a database target's archive
+// downloads as a gzipped JSON attachment named for the webhook, the
+// target and the time, here with no archive file yet, so with no
+// rows; and that a target of another type has no download.
+func TestHandleTargetDownload(t *testing.T) {
+	t.Parallel()
+
+	env := setupSourceTest(t)
+	wh := seedWebhookWithRetention(t, env.db, 7)
+	archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
+	logTarget := seedTarget(t, env.db, wh.ID, database.TargetTypeLog)
+
+	w := serveTarget(
+		env, http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
+	)
+	require.Equal(t, http.StatusOK, w.Code, w.Body.String())
+	assert.Equal(t, "application/gzip", w.Header().Get("Content-Type"))
+	assert.Regexp(t,
+		`^attachment; filename="archive-seeded-t-database-`+
+			`\d{8}T\d{6}Z\.json\.gz"$`,
+		w.Header().Get("Content-Disposition"),
+	)
+
+	zr, err := gzip.NewReader(w.Body)
+	require.NoError(t, err)
+
+	var got map[string]json.RawMessage
+
+	require.NoError(t, json.NewDecoder(zr).Decode(&got))
+	assert.JSONEq(t,
+		`{"id":"`+archive.ID+`","name":"t-database"}`,
+		string(got["target"]),
+	)
+	assert.JSONEq(t, `[]`, string(got["archived_events"]))
+
+	w = serveTarget(
+		env, http.MethodGet, downloadPath(wh.ID, logTarget.ID), nil,
+	)
+	assert.Equal(t, http.StatusNotFound, w.Code)
+}
+
+// TestHandleTargetDownload_WaitsForRename proves a download reads the
+// target's names and opens its archive under the lock a rename holds:
+// started while an edit is renaming the archive, it waits, and is
+// named for the target's new name.
+func TestHandleTargetDownload_WaitsForRename(t *testing.T) {
+	t.Parallel()
+
+	env := setupSourceTest(t)
+	wh := seedWebhookWithRetention(t, env.db, 7)
+	archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
+
+	renaming, release := env.archives.BlockNextRename()
+	edited := make(chan *httptest.ResponseRecorder, 1)
+
+	go func() {
+		edited <- renameTarget(env, wh.ID, archive.ID)
+	}()
+
+	<-renaming
+
+	downloaded := make(chan *httptest.ResponseRecorder, 1)
+
+	go func() {
+		downloaded <- serveTarget(
+			env, http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
+		)
+	}()
+
+	select {
+	case <-downloaded:
+		release()
+		t.Fatal("the download did not wait for the rename")
+	case <-time.After(100 * time.Millisecond):
+	}
+
+	release()
+	require.Equal(t, http.StatusSeeOther, (<-edited).Code)
+
+	w := <-downloaded
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Contains(t,
+		w.Header().Get("Content-Disposition"), "archive-seeded-renamed-",
+	)
+}
+
+// stalledWriter is a response writer whose first write waits until
+// resume is closed, closing writing when it starts to wait.
+type stalledWriter struct {
+	*httptest.ResponseRecorder
+
+	once    sync.Once
+	writing chan struct{}
+	resume  chan struct{}
+}
+
+func (s *stalledWriter) Write(b []byte) (int, error) {
+	s.once.Do(func() {
+		close(s.writing)
+		<-s.resume
+	})
+
+	return s.ResponseRecorder.Write(b)
+}
+
+// TestHandleTargetDownload_StreamsWithoutTheLock proves a download
+// lets go of the rename lock once its archive is open: while the
+// download is stalled writing, an edit can still rename the target.
+func TestHandleTargetDownload_StreamsWithoutTheLock(t *testing.T) {
+	t.Parallel()
+
+	env := setupSourceTest(t)
+	wh := seedWebhookWithRetention(t, env.db, 7)
+	archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
+
+	req := httptest.NewRequestWithContext(
+		t.Context(), http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
+	)
+	for _, c := range env.cookies {
+		req.AddCookie(c)
+	}
+
+	sw := &stalledWriter{
+		ResponseRecorder: httptest.NewRecorder(),
+		writing:          make(chan struct{}),
+		resume:           make(chan struct{}),
+	}
+	downloaded := make(chan struct{})
+
+	go func() {
+		targetRouter(env).ServeHTTP(sw, req)
+		close(downloaded)
+	}()
+
+	<-sw.writing
+
+	edited := make(chan *httptest.ResponseRecorder, 1)
+
+	go func() {
+		edited <- renameTarget(env, wh.ID, archive.ID)
+	}()
+
+	select {
+	case w := <-edited:
+		assert.Equal(t, http.StatusSeeOther, w.Code)
+	case <-time.After(10 * time.Second):
+		t.Error("the rename waited for the download")
+	}
+
+	close(sw.resume)
+	<-downloaded
+	assert.Equal(t, http.StatusOK, sw.Code)
+}

internal/handlers/target_edit_test.go

@@ -37,14 +37,18 @@ const (
 	editAuthHeader   = "Authorization: Bearer " + editBearerSecret
 )
 
-// targetRouter mounts the target create and edit routes on a chi
-// router so the handlers see the URL parameters they read.
+// targetRouter mounts the target create, edit and download routes on
+// a chi router so the handlers see the URL parameters they read.
 func targetRouter(env *sourceTestEnv) *chi.Mux {
 	router := chi.NewRouter()
 	router.Post(
 		"/hook/{sourceID}/targets",
 		env.handlers.HandleTargetCreate(),
 	)
+	router.Get(
+		"/hook/{sourceID}/targets/{targetID}/download",
+		env.handlers.HandleTargetDownload(),
+	)
 	router.Get(
 		"/hook/{sourceID}/targets/{targetID}/edit",
 		env.handlers.HandleTargetEdit(),

internal/handlers/webhook.go

@@ -11,7 +11,6 @@ import (
 	"sneak.berlin/go/webhooker/internal/database"
 	"sneak.berlin/go/webhooker/internal/delivery"
 	"sneak.berlin/go/webhooker/internal/logfield"
-	"sneak.berlin/go/webhooker/internal/middleware"
 )
 
 const (
@@ -58,8 +57,7 @@ func (h *Handlers) HandleWebhook() http.HandlerFunc {
 		h.log.Info("webhook request received",
 			"entrypoint_uuid", entrypointUUID,
 			"method", r.Method,
-			"remoteIP", middleware.RemoteIP(r),
-			"clientIP", middleware.ClientIP(r),
+			"remote_addr", r.RemoteAddr,
 		)
 
 		if !entrypoint.Active {

internal/handlers/webhook_test.go

@@ -1,124 +0,0 @@
-package handlers_test
-
-import (
-	"bytes"
-	"context"
-	"encoding/json"
-	"log/slog"
-	"net/http"
-	"net/http/httptest"
-	"net/netip"
-	"strings"
-	"testing"
-
-	"github.com/go-chi/chi"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"sneak.berlin/go/webhooker/internal/config"
-	"sneak.berlin/go/webhooker/internal/database"
-	"sneak.berlin/go/webhooker/internal/handlers"
-	"sneak.berlin/go/webhooker/internal/middleware"
-)
-
-// TestHandleWebhook_LogsClientNextToThePeer checks that the
-// receiver's "webhook request received" line carries both addresses:
-// remoteIP, the connecting peer, and clientIP, the client the access
-// log attributes the request to.
-func TestHandleWebhook_LogsClientNextToThePeer(t *testing.T) {
-	t.Parallel()
-
-	// untrustedPeer is outside the trusted 10.0.0.0/8, so its
-	// X-Forwarded-For is ignored and it is the client.
-	const untrustedPeer = "192.0.2.10"
-
-	cases := map[string]struct {
-		peer       string
-		wantRemote string
-		wantClient string
-	}{
-		"trusted proxy with a forwarded chain": {
-			peer:       "10.0.0.1:44444",
-			wantRemote: "10.0.0.1",
-			wantClient: "198.51.100.7",
-		},
-		"untrusted peer": {
-			peer:       untrustedPeer + ":5555",
-			wantRemote: untrustedPeer,
-			wantClient: untrustedPeer,
-		},
-	}
-
-	for name, tc := range cases {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			var (
-				h  *handlers.Handlers
-				mw *middleware.Middleware
-				db *database.Database
-			)
-
-			app := newTestAppWithConfig(t, &config.Config{
-				DataDir: t.TempDir(),
-				TrustedProxies: []netip.Prefix{
-					netip.MustParsePrefix("10.0.0.0/8"),
-				},
-			}, &h, &mw, &db)
-			app.RequireStart()
-
-			t.Cleanup(app.RequireStop)
-
-			buf := new(bytes.Buffer)
-			h.SetLogForTest(slog.New(slog.NewJSONHandler(buf, nil)))
-
-			webhook := seedWebhook(t, db)
-			seedEntrypoint(t, db, webhook.ID)
-
-			// Logging is what works the client address out, so the
-			// request goes through it as it does in production.
-			router := chi.NewRouter()
-			router.Use(mw.Logging())
-			router.Post("/h/{uuid}", h.HandleWebhook())
-
-			req := httptest.NewRequestWithContext(
-				context.Background(), http.MethodPost,
-				"/h/ep-"+webhook.ID, strings.NewReader("{}"),
-			)
-			req.RemoteAddr = tc.peer
-			req.Header.Set("X-Forwarded-For", "198.51.100.7, 10.0.0.2")
-
-			w := httptest.NewRecorder()
-			router.ServeHTTP(w, req)
-
-			require.Equal(t, http.StatusOK, w.Code)
-
-			line := receivedLine(t, buf)
-			assert.Equal(t, tc.wantRemote, line["remoteIP"])
-			assert.Equal(t, tc.wantClient, line["clientIP"])
-		})
-	}
-}
-
-// receivedLine returns the one "webhook request received" line in the
-// captured JSON log.
-func receivedLine(t *testing.T, buf *bytes.Buffer) map[string]any {
-	t.Helper()
-
-	var found []map[string]any
-
-	for line := range strings.SplitSeq(
-		strings.TrimSpace(buf.String()), "\n",
-	) {
-		var entry map[string]any
-
-		require.NoError(t, json.Unmarshal([]byte(line), &entry))
-
-		if entry["msg"] == "webhook request received" {
-			found = append(found, entry)
-		}
-	}
-
-	require.Len(t, found, 1)
-
-	return found[0]
-}

internal/middleware/accesslog_test.go

@@ -648,8 +648,7 @@ func TestAccessLog_RetainsEveryOtherField(t *testing.T) {
 
 	for _, key := range []string{
 		"request_start", "method", "url", "useragent", "request_id",
-		"referer", "proto", "remoteIP", "clientIP", "status",
-		"latency_ms",
+		"referer", "proto", "remoteIP", "status", "latency_ms",
 	} {
 		assert.Contains(t, entries[0], key)
 	}

internal/middleware/clientip_test.go

@@ -1,176 +0,0 @@
-package middleware_test
-
-import (
-	"bytes"
-	"context"
-	"log/slog"
-	"net/http"
-	"net/http/httptest"
-	"testing"
-
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"sneak.berlin/go/webhooker/internal/config"
-	"sneak.berlin/go/webhooker/internal/middleware"
-)
-
-const (
-	// forwardedChain is the X-Forwarded-For a request arrives with:
-	// the client, then a second proxy inside trustedProxyCIDR that the
-	// request passed through before reaching trustedPeer.
-	forwardedChain = clientIPv4 + ", 10.0.0.2"
-
-	// untrustedPeer is a peer outside trustedProxyCIDR, so its
-	// X-Forwarded-For is ignored and the peer is the client.
-	untrustedPeer = "192.0.2.10:5555"
-
-	// oneRequestPerMinute is the receiver limit these tests install:
-	// the second request on a path is rejected, and the aggregate
-	// limit is ReceiverAggregateMultiplierConst.
-	oneRequestPerMinute = 1
-)
-
-// clientLogSite is one log line that names the client. build wraps the
-// middleware that writes it around a handler, and requests is how many
-// identical requests it takes before the line is written.
-type clientLogSite struct {
-	build    func(m *middleware.Middleware) http.Handler
-	requests int
-}
-
-// clientLogSites maps the message of each line that names the client
-// to the way to make it be written.
-func clientLogSites() map[string]clientLogSite {
-	served := func(*middleware.Middleware) http.Handler {
-		return okHandler()
-	}
-
-	receiver := func(m *middleware.Middleware) http.Handler {
-		return m.ReceiverRateLimit()(okHandler())
-	}
-
-	login := func(m *middleware.Middleware) http.Handler {
-		return http.HandlerFunc(
-			func(w http.ResponseWriter, r *http.Request) {
-				m.RecordLoginFailure(r, "someone")
-				w.WriteHeader(http.StatusUnauthorized)
-			},
-		)
-	}
-
-	csrf := func(m *middleware.Middleware) http.Handler {
-		return m.CSRF(http.HandlerFunc(forbidden))(okHandler())
-	}
-
-	return map[string]clientLogSite{
-		"http request": {
-			build:    served,
-			requests: 1,
-		},
-		"webhook receiver rate limit exceeded": {
-			build:    receiver,
-			requests: oneRequestPerMinute + 1,
-		},
-		// The aggregate limit sits in front of the per-entrypoint
-		// one, so the requests that one rejects count towards it.
-		"webhook receiver aggregate rate limit exceeded": {
-			build: receiver,
-			requests: middleware.ReceiverAggregateMultiplierConst*
-				oneRequestPerMinute + 1,
-		},
-		"login failure limit exceeded": {
-			build:    login,
-			requests: middleware.LoginRateLimitConst + 1,
-		},
-		"csrf: token validation failed": {
-			build:    csrf,
-			requests: 1,
-		},
-	}
-}
-
-// clientLogLines sends the site's requests from peer, each carrying
-// forwardedChain, through Logging and then the site, as production
-// does, and returns the logged lines whose message is msg.
-func clientLogLines(
-	t *testing.T, site clientLogSite, msg, peer string,
-) []map[string]any {
-	t.Helper()
-
-	buf := new(bytes.Buffer)
-	log := slog.New(slog.NewJSONHandler(
-		buf,
-		&slog.HandlerOptions{Level: slog.LevelDebug},
-	))
-
-	cfg := &config.Config{
-		Environment:       config.EnvironmentDev,
-		ReceiverRateLimit: oneRequestPerMinute,
-		TrustedProxies:    trustedProxies(trustedProxyCIDR),
-	}
-
-	m := middleware.NewForTest(
-		log, cfg, newTestSessionManager(cfg, log, nil),
-	)
-	handler := m.Logging()(site.build(m))
-
-	for range site.requests {
-		req := httptest.NewRequestWithContext(
-			context.Background(), http.MethodPost, "/h/x", nil,
-		)
-		req.RemoteAddr = peer
-		req.Header.Set(headerXFF, forwardedChain)
-
-		handler.ServeHTTP(httptest.NewRecorder(), req)
-	}
-
-	var lines []map[string]any
-
-	for _, entry := range accessLogEntries(t, buf) {
-		if entry["msg"] == msg {
-			lines = append(lines, entry)
-		}
-	}
-
-	return lines
-}
-
-// TestClientIP_LoggedNextToThePeer checks that every line that names
-// the client carries both addresses: remoteIP, the connecting peer,
-// and clientIP, the client the rate limiters key on.
-func TestClientIP_LoggedNextToThePeer(t *testing.T) {
-	t.Parallel()
-
-	cases := map[string]struct {
-		peer       string
-		wantRemote string
-		wantClient string
-	}{
-		"trusted proxy with a forwarded chain": {
-			peer:       trustedPeer,
-			wantRemote: "10.0.0.1",
-			wantClient: clientIPv4,
-		},
-		"untrusted peer": {
-			peer:       untrustedPeer,
-			wantRemote: "192.0.2.10",
-			wantClient: "192.0.2.10",
-		},
-	}
-
-	for msg, site := range clientLogSites() {
-		for name, tc := range cases {
-			t.Run(msg+"/"+name, func(t *testing.T) {
-				t.Parallel()
-
-				lines := clientLogLines(t, site, msg, tc.peer)
-				require.NotEmpty(t, lines, "%q was never logged", msg)
-
-				for _, line := range lines {
-					assert.Equal(t, tc.wantRemote, line["remoteIP"])
-					assert.Equal(t, tc.wantClient, line["clientIP"])
-				}
-			})
-		}
-	}
-}

internal/middleware/csrf.go

@@ -45,10 +45,10 @@ func (m *Middleware) CSRF(
 		// unauthenticated client: a POST with no token to
 		// /hook/<any length of any text>/edit lands here. The
 		// method and path are capped against the same budgets as
-		// the access log. remoteIP and clientIP are the same
-		// addresses the access log carries, and
+		// the access log. remote_addr is set by net/http from the
+		// accepted connection rather than by the client, and
 		// csrf.FailureReason returns one of gorilla/csrf's own
-		// fixed error values, so none of them is client-sized.
+		// fixed error values, so neither is client-sized.
 		m.log.Warn("csrf: token validation failed",
 			"method", logfield.Truncate(
 				r.Method, maxLogMethodBytes,
@@ -56,8 +56,7 @@ func (m *Middleware) CSRF(
 			"path", logfield.Truncate(
 				r.URL.Path, logfield.MaxBytes,
 			),
-			"remoteIP", RemoteIP(r),
-			"clientIP", ClientIP(r),
+			"remote_addr", r.RemoteAddr,
 			"reason", csrf.FailureReason(r),
 		)
 		forbidden.ServeHTTP(w, r)

internal/middleware/loginguard.go

@@ -385,8 +385,6 @@ func (m *Middleware) RecordLoginFailure(
 			"path", logfield.Truncate(
 				r.URL.Path, logfield.MaxBytes,
 			),
-			"remoteIP", RemoteIP(r),
-			"clientIP", ClientIP(r),
 		)
 	}
 

internal/middleware/middleware.go

@@ -3,7 +3,6 @@
 package middleware
 
 import (
-	"context"
 	"log/slog"
 	"net"
 	"net/http"
@@ -70,19 +69,18 @@ const (
 	//	url, useragent, referer   3*(512+11) = 1569
 	//	request_id                    128+11 =  139
 	//	method                         32+11 =   43
-	//	fixed portion                        =  405
+	//	fixed portion                        =  336
 	//	                                        ----
-	//	                                        2156
+	//	                                        2087
 	//
 	// The 512 is logfield.MaxBytes; the 11 is the truncation marker,
 	// charged on top of each budget rather than inside it.
 	//
 	// The fixed portion is the JSON punctuation, the field names, the
-	// level and the message, both timestamps at their longest, remoteIP
-	// and clientIP each charged as an IPv6 address with a zone, a
-	// three-digit status and a full-width int64 latency. Stated at 2560
-	// so the figure carries headroom rather than sitting on the
-	// arithmetic.
+	// level and the message, both timestamps at their longest, an IPv6
+	// remoteIP with a zone, a three-digit status and a full-width int64
+	// latency. Stated at 2560 so the figure carries headroom rather
+	// than sitting on the arithmetic.
 	//
 	// The tty text handler in internal/logger is covered by the same
 	// figure. logfield.EncodedBytes charges every rune at least what
@@ -90,8 +88,8 @@ const (
 	// bytes strconv.Quote spends on a non-printable rune at or above
 	// U+10000, which is four more than the JSON handler ever spends —
 	// so each budget bounds the encoded field under either handler.
-	// The text handler's fixed portion is 351, the smaller of the two,
-	// which puts its worst case at 2102.
+	// The text handler's fixed portion is 286, the smaller of the two,
+	// which puts its worst case at 2037.
 	//
 	// It is also the ceiling on every OTHER line this service writes
 	// THROUGH SLOG that carries text an UNAUTHENTICATED client
@@ -217,28 +215,6 @@ func ipFromHostPort(hp string) string {
 	return h
 }
 
-// RemoteIP returns the address of the connecting peer, without its
-// port. Behind a reverse proxy it is the proxy. Every log line that
-// names the client logs it as remoteIP, next to clientIP.
-func RemoteIP(r *http.Request) string {
-	return ipFromHostPort(r.RemoteAddr)
-}
-
-// clientIPKey is the request context key under which Logging stores
-// the value ClientIP returns.
-type clientIPKey struct{}
-
-// ClientIP returns the address the request is attributed to, which
-// Logging works out once per request with clientAddr in ratelimit.go
-// and logs as clientIP. The other lines that name the client read it
-// from here, so all of them agree. It is empty for a request Logging
-// has not seen.
-func ClientIP(r *http.Request) string {
-	ip, _ := r.Context().Value(clientIPKey{}).(string)
-
-	return ip
-}
-
 type loggingResponseWriter struct {
 	http.ResponseWriter
 
@@ -340,13 +316,6 @@ func (s *Middleware) Logging() func(http.Handler) http.Handler {
 			lrw := newLoggingResponseWriter(w)
 			ctx := r.Context()
 
-			// When RemoteAddr is not an address, the peer's own
-			// text is all the request can be attributed to.
-			clientIP := RemoteIP(r)
-			if addr, ok := s.clientAddr(r); ok {
-				clientIP = addr.String()
-			}
-
 			defer func() {
 				latency := time.Since(start)
 				requestID := ""
@@ -381,16 +350,13 @@ func (s *Middleware) Logging() func(http.Handler) http.Handler {
 						r.Referer(), logfield.MaxBytes,
 					),
 					"proto", r.Proto,
-					"remoteIP", RemoteIP(r),
-					"clientIP", clientIP,
+					"remoteIP", ipFromHostPort(r.RemoteAddr),
 					"status", lrw.statusCode,
 					"latency_ms", latency.Milliseconds(),
 				)
 			}()
 
-			next.ServeHTTP(lrw, r.WithContext(
-				context.WithValue(ctx, clientIPKey{}, clientIP),
-			))
+			next.ServeHTTP(lrw, r)
 		})
 	}
 }

internal/middleware/ratelimit.go

@@ -202,17 +202,24 @@ func (m *Middleware) forwardedClientAddr(
 }
 
 // rateLimitKey is the client identity every rate limiter in this
-// package buckets on: the address clientAddr attributes the request
-// to, reduced to a bucket by bucketKey — full address for IPv4, /64
-// prefix for IPv6.
+// package buckets on. Forwarded headers are honoured only when the
+// direct peer (RemoteAddr) is inside the configured trusted-proxy
+// set; otherwise the peer address itself is the key. Without that
+// gate any client could mint a fresh bucket per request, or starve
+// another client's bucket, by picking an X-Forwarded-For value —
+// which makes every limit here decorative against a deliberate
+// attacker.
+//
+// The address that identifies the client is then reduced to a bucket
+// by bucketKey: full address for IPv4, /64 prefix for IPv6.
 func (m *Middleware) rateLimitKey(r *http.Request) (string, error) {
 	return m.clientKey(r), nil
 }
 
 // clientKey computes the bucket key described on rateLimitKey.
 func (m *Middleware) clientKey(r *http.Request) string {
-	addr, ok := m.clientAddr(r)
-	if !ok {
+	peer, err := netip.ParseAddr(ipFromHostPort(r.RemoteAddr))
+	if err != nil {
 		// Not an address we can reason about; key on the raw
 		// value, the most specific identity left. Distinct
 		// RemoteAddr values stay in distinct buckets, so this
@@ -223,36 +230,16 @@ func (m *Middleware) clientKey(r *http.Request) string {
 		return r.RemoteAddr
 	}
 
-	return bucketKey(addr)
-}
-
-// clientAddr is the address a request is attributed to. The rate
-// limiters key on it and the logs name it as clientIP.
-//
-// Forwarded headers are honoured only when the direct peer
-// (RemoteAddr) is inside the configured trusted-proxy set; otherwise
-// the peer address itself is the client. Without that gate any client
-// could mint a fresh bucket per request, or starve another client's
-// bucket, by picking an X-Forwarded-For value — which makes every
-// limit here decorative against a deliberate attacker.
-//
-// ok is false when RemoteAddr is not an address at all.
-func (m *Middleware) clientAddr(r *http.Request) (netip.Addr, bool) {
-	peer, err := netip.ParseAddr(ipFromHostPort(r.RemoteAddr))
-	if err != nil {
-		return netip.Addr{}, false
-	}
-
 	peer = normalizeAddr(peer)
 	if !m.isTrustedProxy(peer) {
-		return peer, true
+		return bucketKey(peer)
 	}
 
 	if addr, ok := m.forwardedClientAddr(r); ok {
-		return addr, true
+		return bucketKey(addr)
 	}
 
-	return peer, true
+	return bucketKey(peer)
 }
 
 // tooManyRequests returns the 429 handler used by the
@@ -275,8 +262,6 @@ func (m *Middleware) tooManyRequests(
 			"path", logfield.Truncate(
 				r.URL.Path, logfield.MaxBytes,
 			),
-			"remoteIP", RemoteIP(r),
-			"clientIP", ClientIP(r),
 		)
 		http.Error(w, responseMessage, http.StatusTooManyRequests)
 	}
@@ -301,12 +286,8 @@ func (m *Middleware) tooManyRequests(
 func (m *Middleware) floodTooManyRequests(
 	logMessage, responseMessage string,
 ) http.HandlerFunc {
-	return func(w http.ResponseWriter, r *http.Request) {
-		m.log.Debug(
-			logMessage,
-			"remoteIP", RemoteIP(r),
-			"clientIP", ClientIP(r),
-		)
+	return func(w http.ResponseWriter, _ *http.Request) {
+		m.log.Debug(logMessage)
 		http.Error(w, responseMessage, http.StatusTooManyRequests)
 	}
 }

internal/server/archive_download_test.go

@@ -0,0 +1,69 @@
+package server_test
+
+import (
+	"compress/gzip"
+	"encoding/json"
+	"net/http"
+	"regexp"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"gorm.io/gorm/clause"
+	"sneak.berlin/go/webhooker/internal/database"
+)
+
+// TestHook_DownloadArchive follows the Download link the webhook page
+// shows for a database target, and only for it, and gets the archive
+// as a gzipped JSON file. Signed out, the link leads to the login page.
+func TestHook_DownloadArchive(t *testing.T) {
+	t.Parallel()
+
+	env := newTestEnv(t)
+
+	userID, _ := env.seedUser(t, "archivist", "somepassword")
+	cookies := env.authCookies(t, userID, "archivist")
+	wh := env.seedWebhook(t, userID)
+	env.seedTarget(t, wh.ID)
+
+	archive := &database.Target{
+		WebhookID: wh.ID,
+		Name:      "kept",
+		Type:      database.TargetTypeDatabase,
+		Active:    true,
+	}
+	require.NoError(t,
+		env.db.DB().Omit(clause.Associations).Create(archive).Error,
+	)
+
+	page := env.get("/hook/"+wh.ID, cookies)
+	require.Equal(t, http.StatusOK, page.Code)
+
+	links := regexp.MustCompile(
+		`href="(/hook/[^/"]+/targets/[^/"]+/download)"`,
+	).FindAllStringSubmatch(page.Body.String(), -1)
+	require.Len(t, links, 1, "only the database target has a Download")
+
+	link := links[0][1]
+	assert.Equal(t,
+		"/hook/"+wh.ID+"/targets/"+archive.ID+"/download", link,
+	)
+
+	w := env.get(link, cookies)
+	require.Equal(t, http.StatusOK, w.Code)
+	assert.Equal(t, "application/gzip", w.Header().Get("Content-Type"))
+
+	zr, err := gzip.NewReader(w.Body)
+	require.NoError(t, err)
+
+	var got map[string]json.RawMessage
+
+	require.NoError(t, json.NewDecoder(zr).Decode(&got))
+	assert.JSONEq(t,
+		`{"id":"`+wh.ID+`","name":"routed"}`, string(got["webhook"]),
+	)
+
+	w = env.get(link, nil)
+	assert.Equal(t, http.StatusSeeOther, w.Code)
+	assert.Contains(t, w.Header().Get("Location"), "/pages/login")
+}

internal/server/routes.go

@@ -312,6 +312,10 @@ func (s *Server) setupSourceRoutes() {
 			"/targets/{targetID}/edit",
 			s.h.HandleTargetEditSubmit(),
 		)
+		r.Get(
+			"/targets/{targetID}/download",
+			s.h.HandleTargetDownload(),
+		)
 		r.Post(
 			"/targets/{targetID}/delete",
 			s.h.HandleTargetDelete(),

templates/source_detail.html

@@ -157,6 +157,9 @@
                             {{else}}
                             <span class="badge-error">Inactive</span>
                             {{end}}
+                            {{if eq .Type "database"}}
+                            <a href="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/download" class="text-xs text-gray-500 hover:text-primary-600" title="Download the archive as gzipped JSON">Download</a>
+                            {{end}}
                             <a href="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/edit" class="text-xs text-gray-500 hover:text-primary-600" title="Edit">Edit</a>
                             <form method="POST" action="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/toggle" class="inline">
                                 <input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">