2 Commits
Author SHA1 Message Date
clawbot c83ccca336 Show a database target's archive file, size and last write in the target list (closes #397)
check / check (push) Successful in 3m21s
Each database target's row on the webhook page names its archive
file, as delivery.ArchivePath finds it, and shows its size on disk and
when it was last written, or "not created yet" when there is no file.
The figures come from the metadata of the file and its -wal only: the
size is the two together, and the last write the later of their
modification times, since a write lands in the -wal first.

Model: opus-5-5
2026-10-02 18:58:01 +00:00
clawbot 35d2f28c67 Name each embedded static file so a missing Alpine.js fails the build (closes #166)
check / check (push) Successful in 3m20s
static/static.go embedded the css and js directories, so a build without the extracted Alpine.js file compiled and produced a binary whose admin pages silently had no Alpine. It now names the four files the pages load, so such a build fails naming js/alpine.min.js; make build extracts the file first, so the failure shows when that step is skipped. Both lint stages extract Alpine.js before linting, since the static package no longer compiles without it, and the README's lint stage says so. static/css/input.css, the Tailwind source no page loads, is no longer embedded or served.

Model: opus-5-5
2026-10-02 20:56:35 +02:00
10 changed files with 290 additions and 7 deletions

No files matched your search

+1
View File
@@ -25,6 +25,7 @@ COPY . .
# would need a docker daemon inside the build. Keep these steps in step with # would need a docker daemon inside the build. Keep these steps in step with
# Dockerfile.lint, including --network=none (see its header for why). # Dockerfile.lint, including --network=none (see its header for why).
RUN make fmt-check RUN make fmt-check
RUN script/assets
RUN --network=none golangci-lint config verify --config .golangci.yml RUN --network=none golangci-lint config verify --config .golangci.yml
RUN --network=none golangci-lint run --config .golangci.yml --build-tags browser ./... RUN --network=none golangci-lint run --config .golangci.yml --build-tags browser ./...
+4
View File
@@ -31,6 +31,10 @@ FROM deps AS lint
COPY . . COPY . .
# static/static.go embeds the Alpine.js file this extracts from 3p/; without
# it the static package does not compile and cannot be linted.
RUN script/assets
# `run` silently ignores config keys it does not recognize, so a typo would # `run` silently ignores config keys it does not recognize, so a typo would
# disable a setting without a word. `config verify` is what catches that. # disable a setting without a word. `config verify` is what catches that.
RUN --network=none golangci-lint config verify --config .golangci.yml RUN --network=none golangci-lint config verify --config .golangci.yml
+16 -4
View File
@@ -1050,7 +1050,8 @@ Archive databases are the one exception the service is built for: the
archive writer closes and reopens its handle around writes (debounced archive writer closes and reopens its handle around writes (debounced
to at most one reopen per second), so an operator can move an to at most one reopen per second), so an operator can move an
`archive-….db` away for offline retention while the service runs, `archive-….db` away for offline retention while the service runs,
and it is recreated on the next write. See and it is recreated on the next write. The webhook page names each
`database` target's archive file. See
[Database Architecture](#database-architecture). That is a [Database Architecture](#database-architecture). That is a
move-the-file-away workflow, not a substitute for the backup procedures move-the-file-away workflow, not a substitute for the backup procedures
above. above.
@@ -1348,7 +1349,9 @@ apply. The directory is `3p/` rather than `vendor/` because Go treats a root
where `go:embed` picks it up. `script/test`, `make build` and `make dev` run where `go:embed` picks it up. `script/test`, `make build` and `make dev` run
it first, and the Dockerfile builds through `make test` and `make build`, so it first, and the Dockerfile builds through `make test` and `make build`, so
nothing downloads Alpine.js. The extracted file is not committed, and nothing downloads Alpine.js. The extracted file is not committed, and
`.dockerignore` keeps any host copy out of the build context. `.dockerignore` keeps any host copy out of the build context. `static/static.go`
names every file it embeds, so a build that skips the extraction, such as a
bare `go build`, fails with an error naming `js/alpine.min.js`.
To move to a new version: download To move to a new version: download
`https://registry.npmjs.org/@alpinejs/csp/-/csp-<version>.tgz`, check it against `https://registry.npmjs.org/@alpinejs/csp/-/csp-<version>.tgz`, check it against
@@ -2031,6 +2034,14 @@ Because each `database` target has its own archive file, a target's
webhook with different expiries keep two archives, each pruned on its webhook with different expiries keep two archives, each pruned on its
own schedule. own schedule.
The webhook page shows, for each `database` target, its archive file's
name, its size on disk and when it was last written. The size counts
the `.db` and its `-wal` together, and the last write is the later of
their two modification times, since a write lands in the `-wal` first.
Both are read from the files' metadata; the archive is never opened.
Before the first write, and after the file has been moved away, the page
shows `not created yet` beside the name.
Each `database` target on the webhook page has a **Download** button, Each `database` target on the webhook page has a **Download** button,
which returns its archive as one gzipped JSON file, which returns its archive as one gzipped JSON file,
`archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`, the `archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`, the
@@ -3379,8 +3390,9 @@ version is fixed independently of the compiler's:
1. **Lint stage** (`golangci/golangci-lint:v2.12.2`, Debian-based) — 1. **Lint stage** (`golangci/golangci-lint:v2.12.2`, Debian-based) —
installs `make`, downloads dependencies, copies the source, and runs installs `make`, downloads dependencies, copies the source, and runs
`make fmt-check`, then `golangci-lint config verify` and `make fmt-check`, then `script/assets` to extract Alpine.js from
`golangci-lint run`, both with `--network=none`. `3p/`, then `golangci-lint config verify` and `golangci-lint run`,
both with `--network=none`.
2. **Builder stage** (`golang:1.26.1-bookworm`) — depends on the lint 2. **Builder stage** (`golang:1.26.1-bookworm`) — depends on the lint
stage passing (it copies a file from it), runs `make test` and stage passing (it copies a file from it), runs `make test` and
`make build` (both extract Alpine.js from `3p/` first), and finally `make build` (both extract Alpine.js from `3p/` first), and finally
@@ -515,6 +515,47 @@ func (w *archiveWriter) prune(expiry time.Duration) {
} }
} }
// ArchiveFileInfo is what the metadata of a database target's archive
// file says about it.
type ArchiveFileInfo struct {
// Size is the bytes on disk of the file and its -wal together.
Size int64
// Written is when the file or its -wal was last modified, whichever
// is later: a write lands in the -wal first.
Written time.Time
}
// StatArchive reads the metadata of the archive file at path and of
// its -wal, without opening the archive. With no file at path, which is
// so before the first write and after the operator moved it away, the
// error wraps fs.ErrNotExist.
func StatArchive(path string) (ArchiveFileInfo, error) {
file, err := os.Stat(path)
if err != nil {
return ArchiveFileInfo{}, err
}
info := ArchiveFileInfo{Size: file.Size(), Written: file.ModTime()}
wal, err := os.Stat(path + "-wal")
if errors.Is(err, fs.ErrNotExist) {
return info, nil
}
if err != nil {
return ArchiveFileInfo{}, err
}
info.Size += wal.Size()
if wal.ModTime().After(info.Written) {
info.Written = wal.ModTime()
}
return info, nil
}
// fileExists reports whether a path currently exists. // fileExists reports whether a path currently exists.
func fileExists(path string) bool { func fileExists(path string) bool {
_, err := os.Stat(path) _, err := os.Stat(path)
+44
View File
@@ -3,6 +3,7 @@ package delivery_test
import ( import (
"database/sql" "database/sql"
"fmt" "fmt"
"io/fs"
"log/slog" "log/slog"
"os" "os"
"path/filepath" "path/filepath"
@@ -183,6 +184,49 @@ func TestArchiveWriter_RecreatesAfterRemoval(
assert.Equal(t, "b", got[0].EventID) assert.Equal(t, "b", got[0].EventID)
} }
// TestStatArchive proves StatArchive finds no file before the first
// write; after a write still held in the -wal, counts the -wal in the
// size and takes its later time as the last write; and finds no file
// again once the file has been moved away.
func TestStatArchive(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
_, err := delivery.StatArchive(path)
require.ErrorIs(t, err, fs.ErrNotExist)
// With the clock stopped, the reopen debounce never passes, so
// the handle stays open after the write.
stopped := time.Now()
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
w.SetNow(func() time.Time { return stopped })
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "a"}, 0))
written := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
earlier := written.Add(-time.Hour)
require.NoError(t, os.Chtimes(path, earlier, earlier))
require.NoError(t, os.Chtimes(path+"-wal", written, written))
file, err := os.Stat(path)
require.NoError(t, err)
wal, err := os.Stat(path + "-wal")
require.NoError(t, err)
require.Positive(t, wal.Size())
got, err := delivery.StatArchive(path)
require.NoError(t, err)
assert.Equal(t, file.Size()+wal.Size(), got.Size)
assert.True(t, written.Equal(got.Written), got.Written)
removeArchiveFiles(t, path)
_, err = delivery.StatArchive(path)
require.ErrorIs(t, err, fs.ErrNotExist)
}
func TestArchiveWriter_ReopenDebounce(t *testing.T) { func TestArchiveWriter_ReopenDebounce(t *testing.T) {
t.Parallel() t.Parallel()
+1 -1
View File
@@ -521,7 +521,7 @@ func (h *Handlers) renderSourceDetail(
// target's stored config blob holds a credential, and it // target's stored config blob holds a credential, and it
// must never reach a template. // must never reach a template.
"Entrypoints": NewEntrypointViews(entrypoints), "Entrypoints": NewEntrypointViews(entrypoints),
"Targets": delivery.NewTargetViews(targets), "Targets": h.targetRows(&webhook, targets),
"Events": events, "Events": events,
"BaseURL": baseURL, "BaseURL": baseURL,
"Stats": h.loadWebhookStats(webhook.ID, entrypoints, targets), "Stats": h.loadWebhookStats(webhook.ID, entrypoints, targets),
+90
View File
@@ -0,0 +1,90 @@
package handlers
import (
"errors"
"io/fs"
"path/filepath"
"time"
"github.com/dustin/go-humanize"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// TargetRowView is one row of the target list on a webhook's page.
type TargetRowView struct {
delivery.TargetView
// Archive is a database target's archive file, and nil for a target
// of any other type.
Archive *ArchiveFileView
}
// ArchiveFileView is what a database target's row shows about its
// archive file.
type ArchiveFileView struct {
Name string
// Note stands in for the size and the last write when there are
// none to show, and is empty when there are.
Note string
// Size is the size on disk. Written is how long ago the file was
// last written, and WrittenUTC the full time the page shows on
// hover.
Size string
Written string
WrittenUTC string
}
// targetRows projects a webhook's targets for the target list on its
// page.
func (h *Handlers) targetRows(
webhook *database.Webhook, targets []database.Target,
) []TargetRowView {
views := delivery.NewTargetViews(targets)
rows := make([]TargetRowView, len(views))
// NewTargetViews returns one view per target, in order.
for i := range views {
rows[i].TargetView = views[i]
if targets[i].Type == database.TargetTypeDatabase {
rows[i].Archive = h.archiveFileView(webhook, &targets[i])
}
}
return rows
}
// archiveFileView describes a database target's archive file from the
// file's metadata alone; the archive is never opened. The file is found
// by the name the archive writer uses, so it follows a rename of the
// webhook or the target.
func (h *Handlers) archiveFileView(
webhook *database.Webhook, target *database.Target,
) *ArchiveFileView {
path := delivery.ArchivePath(h.dbMgr, webhook, target)
view := &ArchiveFileView{Name: filepath.Base(path)}
file, err := delivery.StatArchive(path)
switch {
case errors.Is(err, fs.ErrNotExist):
view.Note = "not created yet"
case err != nil:
h.log.Error(
"failed to read archive file metadata",
"target_id", target.ID,
"error", err,
)
view.Note = "could not be read"
default:
view.Size = humanize.Bytes(uint64(file.Size)) //nolint:gosec // never negative
view.Written = humanize.Time(file.Written)
view.WrittenUTC = file.Written.UTC().Format(time.DateTime) + " UTC"
}
return view
}
+71
View File
@@ -0,0 +1,71 @@
package handlers_test
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// TestHandleSourceDetail_ShowsArchiveFile proves a database target's
// row names its archive file and says "not created yet" before the
// first write, adds the file's size and last write once it has one row,
// and says "not created yet" again once the file has been moved away.
// A target of another type shows no archive file.
func TestHandleSourceDetail_ShowsArchiveFile(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
archive := seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
seedTarget(t, db, wh.ID, database.TargetTypeLog)
path := delivery.ArchivePath(dbMgr, wh, archive)
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.Equal(t, 1, strings.Count(body, "Archive File:"))
assert.Contains(t, body, filepath.Base(path))
assert.Contains(t, body, "not created yet")
assert.NotContains(t, body, "Archive Size:")
seedArchive(t, path, 1, 100)
file, err := os.Stat(path)
require.NoError(t, err)
body = renderSourceDetailPage(t, h, sess, wh.ID)
assert.Contains(t, body, filepath.Base(path))
assert.NotContains(t, body, "not created yet")
assert.Regexp(t,
`Archive Size:</span>\s*<span>[1-9][0-9.]* [kM]?B</span>`, body,
)
assert.Contains(t, body,
`title="`+file.ModTime().UTC().Format(time.DateTime)+` UTC"`,
)
require.NoError(t, os.Rename(path, filepath.Join(t.TempDir(), "moved.db")))
body = renderSourceDetailPage(t, h, sess, wh.ID)
assert.Contains(t, body, filepath.Base(path))
assert.Contains(t, body, "not created yet")
assert.NotContains(t, body, "Archive Size:")
}
+5 -2
View File
@@ -5,7 +5,10 @@ import (
"embed" "embed"
) )
// Static holds the embedded CSS and JavaScript files for the web UI. // Static holds the CSS and JavaScript files the web UI's pages load. They
// are named one by one so that a missing js/alpine.min.js, which make
// assets extracts and git does not track, fails the build instead of
// leaving the pages without Alpine.js.
// //
//go:embed css js //go:embed css/tailwind.css css/style.css js/app.js js/alpine.min.js
var Static embed.FS var Static embed.FS
+17
View File
@@ -179,6 +179,23 @@
<span>{{.Value}}</span> <span>{{.Value}}</span>
</div> </div>
{{end}} {{end}}
{{with .Archive}}
<div class="text-xs text-gray-500 mt-1">
<span class="font-medium text-gray-700">Archive File:</span>
<span class="break-all">{{.Name}}</span>
{{with .Note}}<span>({{.}})</span>{{end}}
</div>
{{if .Size}}
<div class="text-xs text-gray-500 mt-1">
<span class="font-medium text-gray-700">Archive Size:</span>
<span>{{.Size}}</span>
</div>
<div class="text-xs text-gray-500 mt-1">
<span class="font-medium text-gray-700">Last Written:</span>
<span title="{{.WrittenUTC}}">{{.Written}}</span>
</div>
{{end}}
{{end}}
</div> </div>
{{else}} {{else}}
<div class="p-4 text-sm text-gray-500">No targets configured.</div> <div class="p-4 text-sm text-gray-500">No targets configured.</div>