Rotate a database target's archive monthly, daily or hourly (closes #379)
check / check (push) Waiting to run
check / check (push) Waiting to run
A database target's rotation (none, monthly, daily or hourly) puts the UTC period of each event's receive time in its archive file name, so each file holds exactly its period's events. It is on the new webhook page, the add target form and the target edit form, and shown in the target list. Renames move every one of a target's files and move them back if one fails. The sweep prunes every file, one at a time under the target's lock, and deletes a rotated file it leaves empty. Download lists the files, then opens one at a time, oldest first, finding each again under the target's current names, and gives each row its period. The target list names the current file and totals the size of all of them. Model: opus-5-5
This commit is contained in:
@@ -736,7 +736,8 @@ The app runs as a non-root user (`webhooker`, UID 1000), exposes port
|
||||
The `/var/lib/webhooker` volume holds all SQLite databases: the main
|
||||
application database (`webhooker.db`), the per-webhook event databases
|
||||
(`events-{uuid}.db`), and any archive databases written by `database`
|
||||
targets (`archive-{webhook_name}-{target_name}-{target_uuid}.db`). Mount
|
||||
targets (`archive-{webhook_name}-{target_name}-{target_uuid}.db`, with
|
||||
`-{period}` before `.db` for a target that rotates). Mount
|
||||
this as a persistent volume to
|
||||
preserve data across container restarts.
|
||||
|
||||
@@ -976,9 +977,11 @@ is both the simplest and the only complete rule:
|
||||
- `events-{webhook_uuid}.db` — **one per webhook**. Events, deliveries,
|
||||
delivery results.
|
||||
- `archive-{webhook_name}-{target_name}-{target_uuid}.db` — **one per
|
||||
`database` target**. Archived events. The two names are made safe for
|
||||
a file name, and the file is renamed when the webhook or the target is
|
||||
(see [Database Architecture](#database-architecture)).
|
||||
`database` target**, or one per month, day or hour for a target that
|
||||
rotates, with `-{period}` before `.db`. Archived events. The two names
|
||||
are made safe for a file name, and the files are renamed when the
|
||||
webhook or the target is (see
|
||||
[Database Architecture](#database-architecture)).
|
||||
|
||||
`{webhook_uuid}` and `{target_uuid}` are UUID primary keys in their
|
||||
canonical 36-character hyphenated form, so a real filename looks like
|
||||
@@ -1611,8 +1614,9 @@ The new webhook form can also give the webhook its first targets: an
|
||||
optional HTTP target URL creates an `http` target named `HTTP`, and the
|
||||
archive checkbox creates a `database` target named `Archive` whose
|
||||
`expiry` is the pruning chosen beside it (never, 1h, 12h, 24h, 30d, 90d
|
||||
or 365d). Both are validated as on the add target form, and the webhook
|
||||
and its targets are created together or not at all.
|
||||
or 365d) and whose `rotation` is the rotation chosen below that (none,
|
||||
monthly, daily or hourly). Both are validated as on the add target form,
|
||||
and the webhook and its targets are created together or not at all.
|
||||
|
||||
| Field | Type | Description |
|
||||
| ---------------- | ------- | ----------- |
|
||||
@@ -1721,12 +1725,14 @@ events should be forwarded.
|
||||
own archive database
|
||||
(`archive-{webhook_name}-{target_name}-{target_uuid}.db`) for long-term
|
||||
retention, with an optional creation-validated expiry (default: keep
|
||||
forever). The new webhook form, the add target form and the target edit
|
||||
form all offer the same expiries: never, 1h, 12h, 24h, 30d, 90d or 365d.
|
||||
The target list shows the expiry in plain units, such as "30 days". No
|
||||
external delivery and no retries; an archive write failure fails the
|
||||
delivery. See the database target section under
|
||||
"Per-Webhook Event Databases" for the full semantics.
|
||||
forever) and rotation (default: none, one file). The new webhook form,
|
||||
the add target form and the target edit form all offer the same
|
||||
expiries: never, 1h, 12h, 24h, 30d, 90d or 365d, and the same
|
||||
rotations: none, monthly, daily or hourly. The target list shows the
|
||||
expiry in plain units, such as "30 days", and the rotation. No external
|
||||
delivery and no retries; an archive write failure fails the delivery.
|
||||
See the database target section under "Per-Webhook Event Databases"
|
||||
for the full semantics.
|
||||
- **`log`** — Write the event to the application log (stdout). Useful
|
||||
for debugging.
|
||||
|
||||
@@ -2097,15 +2103,31 @@ single `-`, no `-` at either end, cut to 40 characters, and `unnamed`
|
||||
when nothing is left. The target UUID keeps the file name unique. A
|
||||
webhook named `Orders (EU)` with a target named `Long-term archive`
|
||||
archives into `archive-orders-eu-long-term-archive-{target_uuid}.db`.
|
||||
Renaming the webhook or the target renames the file, under the same
|
||||
lock the archive writes and the archive sweeper take. Webhook edits,
|
||||
target edits and target creation run one at a time, so no edit can
|
||||
rename the file between another's rename and save, and the name on disk
|
||||
matches the UI. A rename never replaces a file: if one already has
|
||||
the new name, the edit is refused with an error naming that file, and
|
||||
the stored name stays. If the archive is not there (the operator moved
|
||||
it away), the rename is not an error, and the next write creates the
|
||||
file under the new name.
|
||||
|
||||
An optional `rotation` in the target's config JSON (e.g.
|
||||
`{"rotation":"daily"}`) is `none`, the default, which keeps the one
|
||||
file, or `monthly`, `daily` or `hourly`. A target that rotates writes
|
||||
each event to a file named for the period of the event's receive time,
|
||||
in UTC, put before the `.db`:
|
||||
`archive-orders-eu-long-term-archive-{target_uuid}-2026-10.db` monthly,
|
||||
`…-2026-10-01.db` daily and `…-2026-10-01-19.db` hourly. Each file holds
|
||||
exactly its period's events, and the first event of a new period starts
|
||||
the next file, so a finished period's file can be moved away like any
|
||||
archive. A changed rotation applies from the next event: the files
|
||||
already written keep their names and stay, pruned, shown and downloaded
|
||||
with the rest, since every file named for the target is its archive,
|
||||
whichever rotation wrote it.
|
||||
|
||||
Renaming the webhook or the target renames every one of the target's
|
||||
files, each keeping its period, under the same lock the archive writes
|
||||
and the archive sweeper take. Webhook edits, target edits and target
|
||||
creation run one at a time, so no edit can rename the files between
|
||||
another's rename and save, and the names on disk match the UI. A rename
|
||||
never replaces a file: if one already has a new name, the edit is
|
||||
refused with an error naming that file, nothing is moved, and the
|
||||
stored name stays. If the archive is not there (the operator moved it
|
||||
away), the rename is not an error, and the next write creates the file
|
||||
under the new name.
|
||||
|
||||
The file is moved just before the new name is saved. If the process
|
||||
stops between the two, the archive is left under the new name while the
|
||||
@@ -2144,18 +2166,29 @@ interleave with a write, and it leaves the archive closed afterwards so
|
||||
the move-the-file-away workflow keeps working. Archives with no expiry,
|
||||
or the expiry `never`, are not touched by the sweep at all.
|
||||
|
||||
For a target with several files, the write path prunes only the file it
|
||||
writes to, and the sweep prunes every one of them, taking the target's
|
||||
lock for one file at a time, so a write to the target waits for at most
|
||||
one file's prune. A file that is gone by the time the sweep reaches it
|
||||
is skipped. A file named for a period that the sweep leaves empty is
|
||||
deleted, with any `-wal` and `-shm` beside it; the file without a period
|
||||
is kept even when empty, as it always has been.
|
||||
|
||||
Because each `database` target has its own archive file, a target's
|
||||
`expiry` governs only its own archive. Two `database` targets on one
|
||||
webhook with different expiries keep two archives, each pruned on its
|
||||
own schedule.
|
||||
|
||||
The webhook page shows, for each `database` target, its archive file's
|
||||
name, its size on disk and when it was last written. The size counts
|
||||
the `.db` and its `-wal` together, and the last write is the later of
|
||||
their two modification times, since a write lands in the `-wal` first.
|
||||
Both are read from the files' metadata; the archive is never opened.
|
||||
Before the first write, and after the file has been moved away, the page
|
||||
shows `not created yet` beside the name.
|
||||
The webhook page shows, for each `database` target, the name of the
|
||||
archive file an event received now would go to, the size on disk of all
|
||||
the target's archive files together, how many there are when there is
|
||||
more than one, and when the latest of them was last written. The size
|
||||
counts each `.db` and its `-wal` together, and the last write is the
|
||||
latest of their modification times, since a write lands in the `-wal`
|
||||
first. All are read from the files' metadata; the archive is never
|
||||
opened. While the named file does not exist — before the first write,
|
||||
before the first event of a new period, and after the file has been
|
||||
moved away — the page shows `not created yet` beside the name.
|
||||
|
||||
Each `database` target on the webhook page has a **Download** button,
|
||||
which returns its archive as one gzipped JSON file,
|
||||
@@ -2163,22 +2196,31 @@ which returns its archive as one gzipped JSON file,
|
||||
names made safe as above and the time in UTC. The file holds one
|
||||
object: `webhook` and `target`, each an `id` and a `name`;
|
||||
`exported_at`; and `archived_events`, one object per archived row with
|
||||
every column, keyed by column name. A body that is not valid UTF-8 is
|
||||
written in base64, with `"body_encoding": "base64"` beside it. An
|
||||
archive that does not exist yet, or was moved away, downloads with an
|
||||
empty `archived_events`; the download never creates the file.
|
||||
every column, keyed by column name. The rows come from every one of the
|
||||
target's files: the file without a period first, then the others in
|
||||
the order of their periods, oldest first, and each row from a file named
|
||||
for a period has that `period` beside its columns. A body that is not
|
||||
valid UTF-8 is written in base64, with `"body_encoding": "base64"`
|
||||
beside it. An archive that does not exist yet, or was moved away,
|
||||
downloads with an empty `archived_events`; the download never creates a
|
||||
file.
|
||||
|
||||
The download streams: each row is read and written out compressed
|
||||
before the next is read, so neither the archive nor the JSON is held in
|
||||
memory. It reads on a connection of its own, inside one read-only
|
||||
transaction, so the file holds the archive as it stood when the
|
||||
download started, and archive writes go on meanwhile, since under WAL a
|
||||
reader never blocks a writer. While it runs, the `-wal` cannot be
|
||||
checkpointed past what it reads, so a long download lets the `-wal`
|
||||
grow. It finds the file by the stored names under the lock that webhook
|
||||
edits, target edits and target creation hold, and lets go once the file
|
||||
is open: a rename during the download moves the file without affecting
|
||||
it.
|
||||
memory. When it starts it lists the target's files by the stored names,
|
||||
under the lock that webhook edits, target edits and target creation
|
||||
hold, and lets go. It then opens one file at a time, only when its rows
|
||||
are about to be written out, and closes it before it opens the next, so
|
||||
it never has more than one of the target's files open. To open each, it
|
||||
takes the lock again just long enough to find the file by its period
|
||||
under the names stored then, so a rename during the download loses no
|
||||
file; a file that is gone by then, emptied by the sweep or moved away,
|
||||
is skipped. Each file is read on a connection of its own inside one
|
||||
read-only transaction, so it is written out as it stood when it was
|
||||
opened, and archive writes go on meanwhile, since under WAL a reader
|
||||
never blocks a writer. Until the open file is closed its `-wal` cannot
|
||||
be checkpointed past what the download reads, so a long download lets
|
||||
that `-wal` grow.
|
||||
|
||||
Deleting a webhook releases its archives: the delivery engine's cached
|
||||
archive writers are dropped and their file handles closed, so nothing
|
||||
@@ -3144,6 +3186,7 @@ webhooker/
|
||||
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
|
||||
│ │ ├── target_database.go # Database archive target
|
||||
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
|
||||
│ │ ├── target_database_rotation.go # Archive rotation and file names
|
||||
│ │ ├── target_database_export.go # Archive download as gzipped JSON
|
||||
│ │ ├── target_log.go # Log target (stdout)
|
||||
│ │ ├── target_config_view.go # Masked target config for templates
|
||||
|
||||
@@ -163,6 +163,17 @@ func (env *archiveEnv) seedArchiveRows(
|
||||
t.Helper()
|
||||
|
||||
path := env.archivePath(tgt)
|
||||
seedArchiveFile(t, path, tgt.WebhookID, archivedAt...)
|
||||
|
||||
return path
|
||||
}
|
||||
|
||||
// seedArchiveFile creates the archive file at path and inserts one row
|
||||
// per supplied archived-at timestamp, as seedArchiveRows does.
|
||||
func seedArchiveFile(
|
||||
t *testing.T, path, webhookID string, archivedAt ...time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
sqlDB, err := sql.Open(
|
||||
"sqlite", fmt.Sprintf("file:%s?mode=rwc", path),
|
||||
@@ -182,7 +193,7 @@ func (env *archiveEnv) seedArchiveRows(
|
||||
for i, at := range archivedAt {
|
||||
row := delivery.ExportArchivedEvent{
|
||||
EventID: fmt.Sprintf("ev-%d", i),
|
||||
WebhookID: tgt.WebhookID,
|
||||
WebhookID: webhookID,
|
||||
Method: http.MethodPost,
|
||||
Body: `{"seeded":true}`,
|
||||
ArchivedAt: at,
|
||||
@@ -191,8 +202,6 @@ func (env *archiveEnv) seedArchiveRows(
|
||||
}
|
||||
|
||||
require.NoError(t, sqlDB.Close())
|
||||
|
||||
return path
|
||||
}
|
||||
|
||||
// archivedEventIDs returns the event ids currently stored in an
|
||||
|
||||
@@ -295,12 +295,13 @@ func (e *Engine) EvictTarget(targetID string) {
|
||||
e.dbTarget.evict(targetID)
|
||||
}
|
||||
|
||||
// Rename implements Archives. It renames a database target's
|
||||
// archive file to ArchiveFileName(webhookName, targetName,
|
||||
// targetID), under the lock the target's archive writes and the
|
||||
// idle sweep take. It never replaces a file: if one already has the
|
||||
// new name, the error is ErrArchiveNameTaken. The caller renames
|
||||
// before it saves the new name: see databaseTarget.rename.
|
||||
// Rename implements Archives. It renames every one of a database
|
||||
// target's archive files to ArchiveFileName(webhookName, targetName,
|
||||
// targetID), each keeping the period in its name, under the lock the
|
||||
// target's archive writes and the idle sweep take. It never replaces
|
||||
// a file: if one already has a new name, the error is
|
||||
// ErrArchiveNameTaken. The caller renames before it saves the new
|
||||
// name: see databaseTarget.rename.
|
||||
func (e *Engine) Rename(
|
||||
targetID, webhookName, targetName string,
|
||||
) error {
|
||||
|
||||
@@ -501,23 +501,32 @@ func NewExportArchiveWriter(
|
||||
return &ExportArchiveWriter{w: w}
|
||||
}
|
||||
|
||||
// Write archives a row through the writer.
|
||||
// Write archives a row through the writer, into the file named
|
||||
// without a period.
|
||||
func (e *ExportArchiveWriter) Write(
|
||||
row ExportArchivedEvent, expiry time.Duration,
|
||||
) error {
|
||||
return e.w.write(row, expiry)
|
||||
return e.w.write(row, expiry, "")
|
||||
}
|
||||
|
||||
// WritePeriod archives a row through the writer, into the file for
|
||||
// period.
|
||||
func (e *ExportArchiveWriter) WritePeriod(
|
||||
row ExportArchivedEvent, expiry time.Duration, period string,
|
||||
) error {
|
||||
return e.w.write(row, expiry, period)
|
||||
}
|
||||
|
||||
// Open opens the archive file, pruning when expiry is positive.
|
||||
func (e *ExportArchiveWriter) Open(expiry time.Duration) error {
|
||||
return e.w.open(expiry)
|
||||
return e.w.open(e.w.path, expiry)
|
||||
}
|
||||
|
||||
// Reopen closes and reopens the archive file.
|
||||
func (e *ExportArchiveWriter) Reopen(
|
||||
expiry time.Duration,
|
||||
) error {
|
||||
return e.w.reopen(expiry)
|
||||
return e.w.reopen(e.w.path, expiry)
|
||||
}
|
||||
|
||||
// SetNow replaces the clock the writer measures its reopen
|
||||
@@ -547,7 +556,7 @@ func (e *ExportArchiveWriter) Path() string {
|
||||
func (e *ExportArchiveWriter) OpenExisting(
|
||||
expiry time.Duration,
|
||||
) error {
|
||||
return e.w.openMode(archiveModeExisting, expiry)
|
||||
return e.w.openMode(e.w.path, archiveModeExisting, expiry)
|
||||
}
|
||||
|
||||
// SweepExpired runs an idle sweep of the archive.
|
||||
|
||||
@@ -41,6 +41,8 @@ type TargetConfigForm struct {
|
||||
Timeout string
|
||||
// Expiry is the database (archive) target's row expiry.
|
||||
Expiry string
|
||||
// Rotation is the database (archive) target's rotation.
|
||||
Rotation string
|
||||
}
|
||||
|
||||
// NewTargetConfigForm parses a target's stored configuration into
|
||||
@@ -85,11 +87,13 @@ func NewTargetConfigForm(
|
||||
}
|
||||
}
|
||||
|
||||
// databaseConfigForm parses an archive target's optional expiry.
|
||||
// An absent, empty or never expiry yields an empty expiry, on which
|
||||
// the edit form starts at never; saving it unchanged stores never,
|
||||
// which means the same as an empty expiry. An expiry that is set
|
||||
// but not a valid duration is an error, not a blank field.
|
||||
// databaseConfigForm parses an archive target's optional expiry and
|
||||
// rotation. An absent, empty or never expiry yields an empty expiry,
|
||||
// on which the edit form starts at never; saving it unchanged stores
|
||||
// never, which means the same as an empty expiry. An absent rotation
|
||||
// is empty too, and the form starts at none. An expiry that is set
|
||||
// but not a valid duration, or a rotation that is not one of the
|
||||
// four, is an error, not a blank field.
|
||||
func databaseConfigForm(
|
||||
configJSON string,
|
||||
) (TargetConfigForm, error) {
|
||||
@@ -106,8 +110,15 @@ func databaseConfigForm(
|
||||
)
|
||||
}
|
||||
|
||||
err = ValidateArchiveRotation(cfg.Rotation)
|
||||
if err != nil {
|
||||
return TargetConfigForm{}, err
|
||||
}
|
||||
|
||||
form := TargetConfigForm{Rotation: cfg.Rotation}
|
||||
|
||||
if cfg.Expiry == "" || cfg.Expiry == archiveExpiryNever {
|
||||
return TargetConfigForm{}, nil
|
||||
return form, nil
|
||||
}
|
||||
|
||||
err = ValidateArchiveExpiry(cfg.Expiry)
|
||||
@@ -115,5 +126,7 @@ func databaseConfigForm(
|
||||
return TargetConfigForm{}, err
|
||||
}
|
||||
|
||||
return TargetConfigForm{Expiry: cfg.Expiry}, nil
|
||||
form.Expiry = cfg.Expiry
|
||||
|
||||
return form, nil
|
||||
}
|
||||
|
||||
@@ -192,8 +192,9 @@ func maxRetriesField(t *database.Target) ConfigField {
|
||||
|
||||
// databaseConfigFields describes an archive target by its
|
||||
// expiry in plain units, such as "30 days", or "never" when
|
||||
// the archive is kept forever. An expiry that is set but not
|
||||
// a valid duration is reported as unavailable rather than
|
||||
// the archive is kept forever, and by its rotation. An expiry
|
||||
// that is set but not a valid duration, or a rotation that is
|
||||
// not one of the four, is reported as unavailable rather than
|
||||
// echoed back.
|
||||
func databaseConfigFields(configJSON string) []ConfigField {
|
||||
expiry, err := parseArchiveExpiry(configJSON)
|
||||
@@ -201,6 +202,11 @@ func databaseConfigFields(configJSON string) []ConfigField {
|
||||
return unavailableConfigFields()
|
||||
}
|
||||
|
||||
rotation, err := parseArchiveRotation(configJSON)
|
||||
if err != nil {
|
||||
return unavailableConfigFields()
|
||||
}
|
||||
|
||||
value := archiveExpiryNever
|
||||
if expiry > 0 {
|
||||
value = plainDuration(expiry)
|
||||
@@ -209,6 +215,9 @@ func databaseConfigFields(configJSON string) []ConfigField {
|
||||
return []ConfigField{{
|
||||
Label: "Archive Expiry",
|
||||
Value: value,
|
||||
}, {
|
||||
Label: "Archive Rotation",
|
||||
Value: rotation,
|
||||
}}
|
||||
}
|
||||
|
||||
|
||||
@@ -328,13 +328,49 @@ func TestNewTargetViews_Database(t *testing.T) {
|
||||
|
||||
assert.Equal(
|
||||
t,
|
||||
map[string]string{"Archive Expiry": tc.want},
|
||||
map[string]string{
|
||||
"Archive Expiry": tc.want,
|
||||
"Archive Rotation": rotationNone,
|
||||
},
|
||||
fieldMap(view.Config),
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewTargetViews_DatabaseRotation proves the target list shows a
|
||||
// database target's rotation, none when it has none stored.
|
||||
func TestNewTargetViews_DatabaseRotation(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Each stored config, and the rotation the list shows for it.
|
||||
tests := map[string]string{
|
||||
"": rotationNone,
|
||||
`{"rotation":""}`: rotationNone,
|
||||
}
|
||||
|
||||
for _, rotation := range []string{
|
||||
rotationNone, rotationMonthly, rotationDaily, rotationHourly,
|
||||
} {
|
||||
tests[`{"rotation":"`+rotation+`"}`] = rotation
|
||||
}
|
||||
|
||||
for config, want := range tests {
|
||||
t.Run(config, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
view := viewFor(t, database.Target{
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: config,
|
||||
})
|
||||
|
||||
assert.Equal(
|
||||
t, want, fieldMap(view.Config)["Archive Rotation"],
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewTargetViews_Log(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -382,6 +418,10 @@ func TestNewTargetViews_Unpresentable(t *testing.T) {
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: `{"expiry":"a fortnight"}`,
|
||||
},
|
||||
"invalid archive rotation": {
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: weeklyConfig,
|
||||
},
|
||||
}
|
||||
|
||||
for name, target := range tests {
|
||||
|
||||
@@ -20,7 +20,8 @@ const archiveNameMaxLen = 40
|
||||
// from the per-webhook event database. The event is already
|
||||
// persisted in the per-webhook event DB by the time delivery runs;
|
||||
// the database target additionally writes a durable long-term copy
|
||||
// into the file ArchiveFileName names and then records a single
|
||||
// into the file ArchiveFileName names, with a period added when the
|
||||
// target rotates (see archivePeriodPath), and then records a single
|
||||
// attempt whose outcome reflects whether the archive write
|
||||
// succeeded. See archiveWriter for the close/reopen, auto-recreate,
|
||||
// and expiry semantics.
|
||||
@@ -146,8 +147,9 @@ func (t *databaseTarget) Deliver(
|
||||
}
|
||||
|
||||
// archive writes the full event as a row into the target's
|
||||
// archive database, honouring the optional per-target expiry
|
||||
// parsed from the target config JSON.
|
||||
// archive database, honouring the optional per-target expiry and
|
||||
// rotation parsed from the target config JSON. With rotation, the
|
||||
// event goes to the file for the period of its receive time.
|
||||
func (t *databaseTarget) archive(d *database.Delivery) error {
|
||||
webhookID := d.Event.WebhookID
|
||||
if webhookID == "" {
|
||||
@@ -159,6 +161,19 @@ func (t *databaseTarget) archive(d *database.Delivery) error {
|
||||
return err
|
||||
}
|
||||
|
||||
rotation, err := parseArchiveRotation(d.Target.Config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// An event whose stored row was gone before its delivery ran has
|
||||
// no receive time (see Engine.hydrateEvent), and goes to the
|
||||
// file for now.
|
||||
receivedAt := d.Event.CreatedAt
|
||||
if receivedAt.IsZero() {
|
||||
receivedAt = time.Now()
|
||||
}
|
||||
|
||||
w, err := t.writerFor(d.TargetID)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -174,7 +189,7 @@ func (t *databaseTarget) archive(d *database.Delivery) error {
|
||||
ContentType: d.Event.ContentType,
|
||||
}
|
||||
|
||||
return w.write(row, expiry)
|
||||
return w.write(row, expiry, archivePeriod(rotation, receivedAt))
|
||||
}
|
||||
|
||||
// writerFor returns the archive writer for a database target,
|
||||
@@ -275,10 +290,10 @@ func (t *databaseTarget) releaseSweepWriter(
|
||||
delete(t.writers, targetID)
|
||||
}
|
||||
|
||||
// newWriter builds the writer for a database target's archive. The
|
||||
// file is the one ArchivePath gives for the webhook and the target as
|
||||
// newWriter builds the writer for a database target's archive. Its
|
||||
// path is the one ArchivePath gives for the webhook and the target as
|
||||
// the main database names them now; from then on only rename changes
|
||||
// the name the writer uses. It does not touch the archive file.
|
||||
// the name the writer uses. It does not touch the archive files.
|
||||
func (t *databaseTarget) newWriter(
|
||||
targetID string,
|
||||
) (*archiveWriter, error) {
|
||||
@@ -306,10 +321,10 @@ func (t *databaseTarget) newWriter(
|
||||
return w, nil
|
||||
}
|
||||
|
||||
// rename moves a database target's archive file to the name for
|
||||
// webhookName and targetName. It goes through the target's writer,
|
||||
// so the move holds the lock that writes and the idle sweep take,
|
||||
// and later writes use the new name.
|
||||
// rename moves every one of a database target's archive files to the
|
||||
// name for webhookName and targetName. It goes through the target's
|
||||
// writer, so the move holds the lock that writes and the idle sweep
|
||||
// take, and later writes use the new name.
|
||||
//
|
||||
// The writer is created if there is none, and it stays cached. The
|
||||
// handlers rename before they save the new name, so until the save
|
||||
|
||||
@@ -85,6 +85,10 @@ type databaseTargetConfig struct {
|
||||
// archived rows are pruned, or "never" (the default) to
|
||||
// keep them forever.
|
||||
Expiry string `json:"expiry"`
|
||||
|
||||
// Rotation is none (the default), monthly, daily or hourly: see
|
||||
// archivePeriod.
|
||||
Rotation string `json:"rotation"`
|
||||
}
|
||||
|
||||
// archivedEvent is one fully captured webhook event stored in a
|
||||
@@ -178,7 +182,7 @@ func ValidateArchiveExpiry(expiry string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// archiveWriter owns one database target's archive SQLite file.
|
||||
// archiveWriter owns one database target's archive SQLite files.
|
||||
// It serialises writes, and after each write closes and reopens
|
||||
// the file (debounced to at most once per debounce window) so
|
||||
// an operator can move the file away for offline archiving. The
|
||||
@@ -186,8 +190,16 @@ func ValidateArchiveExpiry(expiry string) error {
|
||||
// file is opened create-if-missing and its schema is migrated
|
||||
// on every open.
|
||||
type archiveWriter struct {
|
||||
mu sync.Mutex
|
||||
path string
|
||||
mu sync.Mutex
|
||||
|
||||
// path is the target's archive file as ArchivePath names it. A
|
||||
// target that rotates writes to the files archivePeriodPath names
|
||||
// for path and a period instead.
|
||||
path string
|
||||
|
||||
// current is the file db is open on.
|
||||
current string
|
||||
|
||||
log *slog.Logger
|
||||
debounce time.Duration
|
||||
db *gorm.DB
|
||||
@@ -236,12 +248,14 @@ func newArchiveWriter(
|
||||
}
|
||||
}
|
||||
|
||||
// write appends the event as a row, then applies the debounced
|
||||
// close/reopen. It recreates the archive file if it was moved
|
||||
// or removed since the last open. A positive expiry prunes rows
|
||||
// older than it on each (re)open.
|
||||
// write appends the event as a row to the archive file for period
|
||||
// (see archivePeriodPath), then applies the debounced close/reopen.
|
||||
// When period names a different file from the one open, the open one
|
||||
// is closed first. It recreates the archive file if it was moved or
|
||||
// removed since the last open. A positive expiry prunes rows older
|
||||
// than it on each (re)open.
|
||||
func (w *archiveWriter) write(
|
||||
row archivedEvent, expiry time.Duration,
|
||||
row archivedEvent, expiry time.Duration, period string,
|
||||
) error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
@@ -252,8 +266,10 @@ func (w *archiveWriter) write(
|
||||
)
|
||||
}
|
||||
|
||||
if w.db == nil || !fileExists(w.path) {
|
||||
err := w.reopen(expiry)
|
||||
file := archivePeriodPath(w.path, period)
|
||||
|
||||
if w.db == nil || w.current != file || !fileExists(file) {
|
||||
err := w.reopen(file, expiry)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -264,41 +280,41 @@ func (w *archiveWriter) write(
|
||||
err := w.db.Create(&row).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"archiving event to %s: %w", w.path, err,
|
||||
"archiving event to %s: %w", file, err,
|
||||
)
|
||||
}
|
||||
|
||||
if w.now().Sub(w.lastReopen) >= w.debounce {
|
||||
return w.reopen(expiry)
|
||||
return w.reopen(file, expiry)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// open opens (creating if missing) the archive file, migrates
|
||||
// open opens (creating if missing) an archive file, migrates
|
||||
// its schema, records the reopen time, and prunes expired rows
|
||||
// when expiry is positive.
|
||||
func (w *archiveWriter) open(expiry time.Duration) error {
|
||||
return w.openMode(archiveModeCreate, expiry)
|
||||
func (w *archiveWriter) open(file string, expiry time.Duration) error {
|
||||
return w.openMode(file, archiveModeCreate, expiry)
|
||||
}
|
||||
|
||||
// openMode opens the archive file with the given SQLite URI
|
||||
// openMode opens an archive file with the given SQLite URI
|
||||
// mode, migrates its schema, records the reopen time, and
|
||||
// prunes expired rows when expiry is positive. The write path
|
||||
// passes archiveModeCreate so a missing file is recreated; the
|
||||
// idle sweep passes archiveModeExisting so a missing file is an
|
||||
// error rather than a newly conjured empty archive.
|
||||
func (w *archiveWriter) openMode(
|
||||
mode string, expiry time.Duration,
|
||||
file, mode string, expiry time.Duration,
|
||||
) error {
|
||||
// Opened through database.OpenSQLite so an archive file carries
|
||||
// the same WAL journaling, busy timeout, immediate-transaction
|
||||
// locking, and pool bounds as every other database file. See
|
||||
// internal/database/sqlite_open.go.
|
||||
sqlDB, err := database.OpenSQLite(w.path, mode)
|
||||
sqlDB, err := database.OpenSQLite(file, mode)
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"opening archive database %s: %w", w.path, err,
|
||||
"opening archive database %s: %w", file, err,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -314,7 +330,7 @@ func (w *archiveWriter) openMode(
|
||||
|
||||
return fmt.Errorf(
|
||||
"connecting to archive database %s: %w",
|
||||
w.path, err,
|
||||
file, err,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -323,11 +339,12 @@ func (w *archiveWriter) openMode(
|
||||
_ = sqlDB.Close()
|
||||
|
||||
return fmt.Errorf(
|
||||
"migrating archive database %s: %w", w.path, err,
|
||||
"migrating archive database %s: %w", file, err,
|
||||
)
|
||||
}
|
||||
|
||||
w.db = gdb
|
||||
w.current = file
|
||||
w.lastReopen = w.now()
|
||||
w.reopens++
|
||||
|
||||
@@ -338,12 +355,12 @@ func (w *archiveWriter) openMode(
|
||||
return nil
|
||||
}
|
||||
|
||||
// reopen closes any open handle and opens the file afresh. The
|
||||
// reopen closes any open handle and opens file afresh. The
|
||||
// fresh open recreates the file if it was moved away.
|
||||
func (w *archiveWriter) reopen(expiry time.Duration) error {
|
||||
func (w *archiveWriter) reopen(file string, expiry time.Duration) error {
|
||||
w.close()
|
||||
|
||||
return w.open(expiry)
|
||||
return w.open(file, expiry)
|
||||
}
|
||||
|
||||
// close closes the underlying handle, if any.
|
||||
@@ -360,22 +377,56 @@ func (w *archiveWriter) close() {
|
||||
w.db = nil
|
||||
}
|
||||
|
||||
// sweepExpired prunes an archive that may have gone idle, with
|
||||
// no write to trigger the usual on-reopen prune. It takes the
|
||||
// writer's own mutex for the whole operation, so a sweep is
|
||||
// ordered against concurrent writes rather than reaching around
|
||||
// them to the file.
|
||||
// sweepExpired prunes the target's archive files, which may have
|
||||
// gone idle, with no write to trigger the usual on-reopen prune. It
|
||||
// lists the files under the writer's own mutex, then takes the mutex
|
||||
// again for one file at a time, so a write waits for at most one
|
||||
// file's prune, and each prune is ordered against concurrent writes
|
||||
// rather than reaching around them to the file.
|
||||
//
|
||||
// It never creates the archive file: a missing file is skipped,
|
||||
// and the reopen uses archiveModeExisting so SQLite itself
|
||||
// refuses to create one if the file disappears between the
|
||||
// check and the open.
|
||||
// It never creates an archive file: it prunes only the files
|
||||
// archiveFiles lists, skips one that is gone by the time it is
|
||||
// reached (moved away, or renamed since the listing), and opens each
|
||||
// with archiveModeExisting so SQLite itself refuses to create one if
|
||||
// the file disappears between the check and the open. A file named
|
||||
// for a period that the prune leaves empty is deleted.
|
||||
//
|
||||
// The archive is left CLOSED afterwards. An idle archive holding
|
||||
// no handle is what keeps the operator's move-the-file-away
|
||||
// workflow working; the next write reopens (and recreates) the
|
||||
// file as it always has.
|
||||
func (w *archiveWriter) sweepExpired(expiry time.Duration) error {
|
||||
w.mu.Lock()
|
||||
files, err := archiveFiles(w.path)
|
||||
w.mu.Unlock()
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var errs []error
|
||||
|
||||
for _, file := range files {
|
||||
err = w.sweepFile(file, expiry)
|
||||
if errors.Is(err, errArchiveWriterEvicted) {
|
||||
return err
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
errs = append(errs, err)
|
||||
}
|
||||
}
|
||||
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
|
||||
// sweepFile prunes one of the target's archive files for sweepExpired,
|
||||
// holding w.mu while it does. It skips a file that is gone, and deletes
|
||||
// the file, with its -wal and -shm, when it is named for a period and
|
||||
// the prune leaves it empty.
|
||||
func (w *archiveWriter) sweepFile(
|
||||
file archiveFile, expiry time.Duration,
|
||||
) error {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
|
||||
@@ -385,7 +436,7 @@ func (w *archiveWriter) sweepExpired(expiry time.Duration) error {
|
||||
)
|
||||
}
|
||||
|
||||
if !fileExists(w.path) {
|
||||
if !fileExists(file.path) {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -393,26 +444,56 @@ func (w *archiveWriter) sweepExpired(expiry time.Duration) error {
|
||||
// freshly opened file, matching the write path's semantics.
|
||||
w.close()
|
||||
|
||||
err := w.openMode(archiveModeExisting, expiry)
|
||||
err := w.openMode(file.path, archiveModeExisting, expiry)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if file.period == "" {
|
||||
w.close()
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
var rows int64
|
||||
|
||||
err = w.db.Model(&archivedEvent{}).Count(&rows).Error
|
||||
|
||||
w.close()
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"counting rows in archive %s: %w", file.path, err,
|
||||
)
|
||||
}
|
||||
|
||||
if rows > 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
for _, suffix := range []string{"", "-wal", "-shm"} {
|
||||
err = os.Remove(file.path + suffix)
|
||||
if err != nil && !errors.Is(err, fs.ErrNotExist) {
|
||||
return fmt.Errorf("deleting empty archive file: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
w.log.Info("deleted empty archive file", "path", file.path)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// rename gives the archive file a new name in the same directory,
|
||||
// and the writer uses the file under that name from now on. The
|
||||
// handle is closed first, which folds the -wal into the .db; any
|
||||
// -wal or -shm still beside the file (left by a crash) is moved with
|
||||
// it, because SQLite finds them by name. A missing file is not an
|
||||
// error: the operator may have moved it away, and the next write
|
||||
// creates it under the new name.
|
||||
// rename gives every one of the target's archive files the new
|
||||
// name, keeping the period in the name of each (see
|
||||
// archivePeriodPath), and the writer uses the files under that name
|
||||
// from now on. The handle is closed first, which folds the -wal into
|
||||
// the .db; any -wal or -shm still beside a file (left by a crash) is
|
||||
// moved with it, because SQLite finds them by name. A target with no
|
||||
// files is not an error: the operator may have moved them away, and
|
||||
// the next write creates its file under the new name.
|
||||
//
|
||||
// If a file already has the new name, nothing is moved and the
|
||||
// error is ErrArchiveNameTaken. If one file fails to move, those
|
||||
// If a file already has one of the new names, nothing is moved and
|
||||
// the error is ErrArchiveNameTaken. If one file fails to move, those
|
||||
// already moved are moved back before the error is returned, so the
|
||||
// archive is never split across two names.
|
||||
func (w *archiveWriter) rename(name string) error {
|
||||
@@ -430,38 +511,53 @@ func (w *archiveWriter) rename(name string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
suffixes := []string{"", "-wal", "-shm"}
|
||||
files, err := archiveFiles(w.path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, suffix := range suffixes {
|
||||
if fileExists(path + suffix) {
|
||||
// from[i] moves to to[i].
|
||||
var from, to []string
|
||||
|
||||
for _, file := range files {
|
||||
renamed := archivePeriodPath(path, file.period)
|
||||
|
||||
for _, suffix := range []string{"", "-wal", "-shm"} {
|
||||
from = append(from, file.path+suffix)
|
||||
to = append(to, renamed+suffix)
|
||||
}
|
||||
}
|
||||
|
||||
for _, taken := range to {
|
||||
if fileExists(taken) {
|
||||
return fmt.Errorf(
|
||||
"%w: %s", ErrArchiveNameTaken, name+suffix,
|
||||
"%w: %s", ErrArchiveNameTaken, filepath.Base(taken),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
w.close()
|
||||
|
||||
for i, suffix := range suffixes {
|
||||
err := os.Rename(w.path+suffix, path+suffix)
|
||||
for i := range from {
|
||||
err = os.Rename(from[i], to[i])
|
||||
if err == nil || errors.Is(err, fs.ErrNotExist) {
|
||||
continue
|
||||
}
|
||||
|
||||
for _, moved := range suffixes[:i] {
|
||||
backErr := os.Rename(path+moved, w.path+moved)
|
||||
for j := range i {
|
||||
backErr := os.Rename(to[j], from[j])
|
||||
if backErr != nil && !errors.Is(backErr, fs.ErrNotExist) {
|
||||
w.log.Error(
|
||||
"failed to move archive file back",
|
||||
"from", path+moved,
|
||||
"to", w.path+moved,
|
||||
"from", to[j],
|
||||
"to", from[j],
|
||||
"error", backErr,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
return fmt.Errorf(
|
||||
"renaming archive %s to %s: %w", w.path+suffix, path+suffix, err,
|
||||
"renaming archive %s to %s: %w", from[i], to[i], err,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -499,7 +595,7 @@ func (w *archiveWriter) prune(expiry time.Duration) {
|
||||
if res.Error != nil {
|
||||
w.log.Error(
|
||||
"failed to prune expired archive rows",
|
||||
"path", w.path,
|
||||
"path", w.current,
|
||||
"error", res.Error,
|
||||
)
|
||||
|
||||
@@ -509,48 +605,76 @@ func (w *archiveWriter) prune(expiry time.Duration) {
|
||||
if res.RowsAffected > 0 {
|
||||
w.log.Info(
|
||||
"pruned expired archive rows",
|
||||
"path", w.path,
|
||||
"path", w.current,
|
||||
"rows_deleted", res.RowsAffected,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// ArchiveFileInfo is what the metadata of a database target's archive
|
||||
// file says about it.
|
||||
// files says about them.
|
||||
type ArchiveFileInfo struct {
|
||||
// Size is the bytes on disk of the file and its -wal together.
|
||||
// Files counts the files.
|
||||
Files int
|
||||
|
||||
// Size is the bytes on disk of the files and their -wal together.
|
||||
Size int64
|
||||
|
||||
// Written is when the file or its -wal was last modified, whichever
|
||||
// is later: a write lands in the -wal first.
|
||||
// Written is when a file or a -wal was last modified, whichever is
|
||||
// latest: a write lands in the -wal first.
|
||||
Written time.Time
|
||||
}
|
||||
|
||||
// StatArchive reads the metadata of the archive file at path and of
|
||||
// its -wal, without opening the archive. With no file at path, which is
|
||||
// so before the first write and after the operator moved it away, the
|
||||
// StatArchive reads the metadata of a database target's archive
|
||||
// files, given the path ArchivePath gives it (see archiveFiles), and
|
||||
// of their -wal, without opening them. With no files, which is so
|
||||
// before the first write and after the operator moved them away, the
|
||||
// error wraps fs.ErrNotExist.
|
||||
func StatArchive(path string) (ArchiveFileInfo, error) {
|
||||
file, err := os.Stat(path)
|
||||
files, err := archiveFiles(path)
|
||||
if err != nil {
|
||||
return ArchiveFileInfo{}, err
|
||||
}
|
||||
|
||||
info := ArchiveFileInfo{Size: file.Size(), Written: file.ModTime()}
|
||||
var info ArchiveFileInfo
|
||||
|
||||
wal, err := os.Stat(path + "-wal")
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return info, nil
|
||||
for _, file := range files {
|
||||
db, err := os.Stat(file.path)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
continue
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return ArchiveFileInfo{}, err
|
||||
}
|
||||
|
||||
info.Files++
|
||||
info.Size += db.Size()
|
||||
|
||||
if db.ModTime().After(info.Written) {
|
||||
info.Written = db.ModTime()
|
||||
}
|
||||
|
||||
wal, err := os.Stat(file.path + "-wal")
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
continue
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return ArchiveFileInfo{}, err
|
||||
}
|
||||
|
||||
info.Size += wal.Size()
|
||||
|
||||
if wal.ModTime().After(info.Written) {
|
||||
info.Written = wal.ModTime()
|
||||
}
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return ArchiveFileInfo{}, err
|
||||
}
|
||||
|
||||
info.Size += wal.Size()
|
||||
|
||||
if wal.ModTime().After(info.Written) {
|
||||
info.Written = wal.ModTime()
|
||||
if info.Files == 0 {
|
||||
return ArchiveFileInfo{}, fmt.Errorf(
|
||||
"no archive file for %s: %w", path, fs.ErrNotExist,
|
||||
)
|
||||
}
|
||||
|
||||
return info, nil
|
||||
|
||||
@@ -6,10 +6,14 @@ import (
|
||||
"database/sql"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
@@ -50,21 +54,46 @@ func ArchiveExportFileName(
|
||||
at.UTC().Format("20060102T150405Z") + ".json.gz"
|
||||
}
|
||||
|
||||
// ArchiveExport is a database target's archive opened for download.
|
||||
// It reads the file on its own connection, inside one read-only
|
||||
// transaction, so it writes out the archive as it stood when
|
||||
// OpenArchiveExport returned.
|
||||
// ArchiveExport is a database target's archive listed for download. It
|
||||
// opens one of the target's files at a time, only when its rows are
|
||||
// about to be written out, and closes it before it opens the next, so
|
||||
// an export holds at most one file open however many the target has.
|
||||
//
|
||||
// Archives are in WAL mode, where a reader works from a snapshot and
|
||||
// never blocks a writer: archive writes go on while an export is open,
|
||||
// and the export does not see them. SQLite cannot checkpoint the -wal
|
||||
// past an open snapshot, so the -wal grows until the export is closed.
|
||||
// Each file is read on its own connection inside one read-only
|
||||
// transaction, so its rows are written out as the file stood when it
|
||||
// was opened. Archives are in WAL mode, where a reader works from a
|
||||
// snapshot and never blocks a writer: archive writes go on while a file
|
||||
// is open, and the export does not see them. SQLite cannot checkpoint
|
||||
// a -wal past an open snapshot, so the open file's -wal grows until the
|
||||
// export has written that file out.
|
||||
type ArchiveExport struct {
|
||||
// periods are the periods of the target's files when the export
|
||||
// was listed, "" for the file without one, in the order
|
||||
// archiveFiles lists them.
|
||||
periods []string
|
||||
|
||||
// lock is held while currentPath is called and a file is opened,
|
||||
// so that a rename, which holds it too, cannot move the file in
|
||||
// between.
|
||||
lock sync.Locker
|
||||
|
||||
// currentPath returns the path ArchivePath gives the target under
|
||||
// the names stored for it now, which a rename may have changed since
|
||||
// the export was listed.
|
||||
currentPath func() (string, error)
|
||||
|
||||
log *slog.Logger
|
||||
}
|
||||
|
||||
// exportFile is one archive file opened for an export.
|
||||
type exportFile struct {
|
||||
db *sql.DB
|
||||
tx *gorm.DB
|
||||
|
||||
// empty is true when there is nothing to read: no file, or a file
|
||||
// without the archive's table yet.
|
||||
// period is the period in the file's name, "" for none.
|
||||
period string
|
||||
|
||||
// empty is true for a file without the archive's table yet.
|
||||
empty bool
|
||||
}
|
||||
|
||||
@@ -74,26 +103,46 @@ type exportedName struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// OpenArchiveExport opens the archive file at path for export and
|
||||
// takes the snapshot the export reads. It never creates the file: with
|
||||
// no file at path, the export has no rows.
|
||||
// NewArchiveExport lists a database target's archive files for export,
|
||||
// given the path ArchivePath gives it (see archiveFiles). It opens none
|
||||
// of them. Its caller holds lock, which every rename of the target's
|
||||
// files runs under, from reading the names path is made of until it
|
||||
// returns, so the files it lists are the ones those names give.
|
||||
//
|
||||
// Once it has returned, the file is open, so a rename or a move of it
|
||||
// does not affect the export, which reads the same file under its new
|
||||
// name.
|
||||
//
|
||||
// The transaction lasts as long as ctx does, so ctx must last for the
|
||||
// whole export.
|
||||
func OpenArchiveExport(
|
||||
ctx context.Context, path string, log *slog.Logger,
|
||||
// WriteGzipJSON, called without lock held, finds each file again by its
|
||||
// period under the path currentPath gives, holding lock while it does
|
||||
// and while it opens the file, so a rename during the export loses no
|
||||
// file. A file that is gone by then, emptied by the sweep or moved
|
||||
// away, is skipped. The export never creates a file: with no files, it
|
||||
// has no rows.
|
||||
func NewArchiveExport(
|
||||
path string,
|
||||
lock sync.Locker,
|
||||
currentPath func() (string, error),
|
||||
log *slog.Logger,
|
||||
) (*ArchiveExport, error) {
|
||||
if !fileExists(path) {
|
||||
return &ArchiveExport{empty: true}, nil
|
||||
files, err := archiveFiles(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
db, err := database.OpenSQLite(path, archiveModeExisting)
|
||||
x := &ArchiveExport{lock: lock, currentPath: currentPath, log: log}
|
||||
|
||||
for _, file := range files {
|
||||
x.periods = append(x.periods, file.period)
|
||||
}
|
||||
|
||||
return x, nil
|
||||
}
|
||||
|
||||
// openExportFile opens one archive file for an export and takes its
|
||||
// snapshot. The transaction lasts as long as ctx does.
|
||||
func openExportFile(
|
||||
ctx context.Context, file archiveFile, log *slog.Logger,
|
||||
) (*exportFile, error) {
|
||||
db, err := database.OpenSQLite(file.path, archiveModeExisting)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("opening archive %s: %w", path, err)
|
||||
return nil, fmt.Errorf("opening archive %s: %w", file.path, err)
|
||||
}
|
||||
|
||||
gdb, err := gorm.Open(
|
||||
@@ -106,7 +155,7 @@ func OpenArchiveExport(
|
||||
if err != nil {
|
||||
_ = db.Close()
|
||||
|
||||
return nil, fmt.Errorf("opening archive %s: %w", path, err)
|
||||
return nil, fmt.Errorf("opening archive %s: %w", file.path, err)
|
||||
}
|
||||
|
||||
// ReadOnly makes the driver begin a deferred transaction in place
|
||||
@@ -116,7 +165,9 @@ func OpenArchiveExport(
|
||||
if tx.Error != nil {
|
||||
_ = db.Close()
|
||||
|
||||
return nil, fmt.Errorf("reading archive %s: %w", path, tx.Error)
|
||||
return nil, fmt.Errorf(
|
||||
"reading archive %s: %w", file.path, tx.Error,
|
||||
)
|
||||
}
|
||||
|
||||
// The transaction's first read is what takes the snapshot.
|
||||
@@ -127,22 +178,27 @@ func OpenArchiveExport(
|
||||
_ = tx.Rollback()
|
||||
_ = db.Close()
|
||||
|
||||
return nil, fmt.Errorf("reading archive %s: %w", path, err)
|
||||
return nil, fmt.Errorf("reading archive %s: %w", file.path, err)
|
||||
}
|
||||
|
||||
return &ArchiveExport{db: db, tx: tx, empty: tables == 0}, nil
|
||||
return &exportFile{
|
||||
db: db, tx: tx, period: file.period, empty: tables == 0,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// WriteGzipJSON writes the export to w as one gzipped JSON object:
|
||||
// webhook and target, each an id and a name; exported_at; and
|
||||
// archived_events, one object per archived row, keyed by column name.
|
||||
// A body that is not valid UTF-8 cannot be a JSON string, so it is
|
||||
// written in base64, with "body_encoding": "base64" beside it.
|
||||
// archived_events, one object per archived row, keyed by column name,
|
||||
// the files in the order archiveFiles lists them. A row from a file
|
||||
// named for a period has "period" beside its columns. A body that is
|
||||
// not valid UTF-8 cannot be a JSON string, so it is written in base64,
|
||||
// with "body_encoding": "base64" beside it.
|
||||
//
|
||||
// Each row is written out before the next is read, so neither the
|
||||
// archive nor its JSON is ever held in memory whole. After an error
|
||||
// the gzip stream is left unfinished, so what was written does not
|
||||
// decompress as a whole file.
|
||||
// archive nor its JSON is ever held in memory whole, and each file is
|
||||
// closed once its rows are written, before the next is opened. When it
|
||||
// returns, no file is open. After an error the gzip stream is left
|
||||
// unfinished, so what was written does not decompress as a whole file.
|
||||
func (x *ArchiveExport) WriteGzipJSON(
|
||||
ctx context.Context,
|
||||
w io.Writer,
|
||||
@@ -169,15 +225,35 @@ func (x *ArchiveExport) WriteGzipJSON(
|
||||
return zw.Close()
|
||||
}
|
||||
|
||||
// Close ends the export's transaction and closes its connection.
|
||||
func (x *ArchiveExport) Close() error {
|
||||
if x.db == nil {
|
||||
return nil
|
||||
// openFile finds the target's archive file for period under the path
|
||||
// currentPath gives now, and opens it for the export, holding x.lock
|
||||
// for both. For a file that is gone, the error wraps fs.ErrNotExist.
|
||||
func (x *ArchiveExport) openFile(
|
||||
ctx context.Context, period string,
|
||||
) (*exportFile, error) {
|
||||
x.lock.Lock()
|
||||
defer x.lock.Unlock()
|
||||
|
||||
path, err := x.currentPath()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("finding archive file: %w", err)
|
||||
}
|
||||
|
||||
_ = x.tx.Rollback()
|
||||
file := archiveFile{path: archivePeriodPath(path, period), period: period}
|
||||
|
||||
return x.db.Close()
|
||||
_, err = os.Stat(file.path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return openExportFile(ctx, file, x.log)
|
||||
}
|
||||
|
||||
// close ends the file's transaction and closes its connection.
|
||||
func (f *exportFile) close() error {
|
||||
_ = f.tx.Rollback()
|
||||
|
||||
return f.db.Close()
|
||||
}
|
||||
|
||||
// writeJSON writes head with archived_events added as its last key,
|
||||
@@ -207,46 +283,77 @@ func (x *ArchiveExport) writeJSON(
|
||||
return err
|
||||
}
|
||||
|
||||
// writeRows writes each archived row to w, oldest first, one per line,
|
||||
// separated by commas.
|
||||
// writeRows writes the archived rows of each file to w, one per line,
|
||||
// separated by commas, opening each file in turn and closing it once
|
||||
// its rows are written.
|
||||
func (x *ArchiveExport) writeRows(ctx context.Context, w io.Writer) error {
|
||||
if x.empty {
|
||||
return nil
|
||||
sep := "\n"
|
||||
|
||||
for _, period := range x.periods {
|
||||
f, err := x.openFile(ctx, period)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
continue
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
sep, err = f.writeRows(ctx, w, sep)
|
||||
|
||||
err = errors.Join(err, f.close())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
rows, err := x.tx.WithContext(ctx).
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeRows writes the file's archived rows to w, oldest first, the
|
||||
// first after sep and each other after ",\n". It returns what goes
|
||||
// before the next row: sep again when the file had no rows.
|
||||
func (f *exportFile) writeRows(
|
||||
ctx context.Context, w io.Writer, sep string,
|
||||
) (string, error) {
|
||||
if f.empty {
|
||||
return sep, nil
|
||||
}
|
||||
|
||||
rows, err := f.tx.WithContext(ctx).
|
||||
Model(&archivedEvent{}).Order("id").Rows()
|
||||
if err != nil {
|
||||
return err
|
||||
return "", err
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
for sep := "\n"; rows.Next(); sep = ",\n" {
|
||||
for ; rows.Next(); sep = ",\n" {
|
||||
var ev archivedEvent
|
||||
|
||||
err = x.tx.ScanRows(rows, &ev)
|
||||
err = f.tx.ScanRows(rows, &ev)
|
||||
if err != nil {
|
||||
return err
|
||||
return "", err
|
||||
}
|
||||
|
||||
_, err = io.WriteString(w, sep)
|
||||
if err != nil {
|
||||
return err
|
||||
return "", err
|
||||
}
|
||||
|
||||
err = writeRow(w, &ev)
|
||||
err = writeRow(w, &ev, f.period)
|
||||
if err != nil {
|
||||
return err
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
|
||||
return rows.Err()
|
||||
return sep, rows.Err()
|
||||
}
|
||||
|
||||
// writeRow writes an archived row to w as a JSON object keyed by
|
||||
// column name, its body in base64 when it is not valid UTF-8.
|
||||
func writeRow(w io.Writer, ev *archivedEvent) error {
|
||||
// column name, its body in base64 when it is not valid UTF-8, with
|
||||
// the period of its file beside them unless that is "".
|
||||
func writeRow(w io.Writer, ev *archivedEvent, period string) error {
|
||||
row := map[string]any{
|
||||
"id": ev.ID,
|
||||
"event_id": ev.EventID,
|
||||
@@ -264,6 +371,10 @@ func writeRow(w io.Writer, ev *archivedEvent) error {
|
||||
row["body_encoding"] = "base64"
|
||||
}
|
||||
|
||||
if period != "" {
|
||||
row["period"] = period
|
||||
}
|
||||
|
||||
line, err := json.Marshal(row)
|
||||
if err != nil {
|
||||
return err
|
||||
|
||||
@@ -12,6 +12,8 @@ import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -29,14 +31,8 @@ const (
|
||||
exportTargetName = "Long-term archive"
|
||||
)
|
||||
|
||||
const (
|
||||
// binaryBody is a body that is not valid UTF-8.
|
||||
binaryBody = "\xff\xfe\x00\x01binary\x80"
|
||||
|
||||
// openedEventID is the event the snapshot tests archive before
|
||||
// they open the export.
|
||||
openedEventID = "opened"
|
||||
)
|
||||
// binaryBody is a body that is not valid UTF-8.
|
||||
const binaryBody = "\xff\xfe\x00\x01binary\x80"
|
||||
|
||||
// writeExportTo writes export to w as the archive of the export tests'
|
||||
// webhook and target, exported at 2026-10-02T12:03:04Z.
|
||||
@@ -59,19 +55,41 @@ func writeExportTo(
|
||||
)
|
||||
}
|
||||
|
||||
// listExport lists the archive at path for export, as the archive of a
|
||||
// target whose names do not change.
|
||||
func listExport(t *testing.T, path string) *delivery.ArchiveExport {
|
||||
t.Helper()
|
||||
|
||||
return newExport(t, path, &sync.Mutex{}, func() (string, error) {
|
||||
return path, nil
|
||||
})
|
||||
}
|
||||
|
||||
// newExport lists the archive at path for export, to find each file
|
||||
// again under the path currentPath gives, holding lock while it does.
|
||||
// Nothing else takes lock while it lists, so it does not hold lock.
|
||||
func newExport(
|
||||
t *testing.T,
|
||||
path string,
|
||||
lock sync.Locker,
|
||||
currentPath func() (string, error),
|
||||
) *delivery.ArchiveExport {
|
||||
t.Helper()
|
||||
|
||||
export, err := delivery.NewArchiveExport(
|
||||
path, lock, currentPath, archiveTestLogger(),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
return export
|
||||
}
|
||||
|
||||
// exportArchive runs a whole export of the archive at path and returns
|
||||
// its JSON, decompressed and parsed.
|
||||
func exportArchive(t *testing.T, path string) map[string]any {
|
||||
t.Helper()
|
||||
|
||||
export, err := delivery.OpenArchiveExport(
|
||||
t.Context(), path, archiveTestLogger(),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { require.NoError(t, export.Close()) }()
|
||||
|
||||
return writeExport(t, export)
|
||||
return writeExport(t, listExport(t, path))
|
||||
}
|
||||
|
||||
// writeExport writes an opened export and returns its JSON,
|
||||
@@ -241,66 +259,199 @@ func TestArchiveExport_Empty(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestArchiveExport_ReadsOneSnapshot proves an export writes the
|
||||
// archive as it was when it was opened, and holds up no archive
|
||||
// write: a row written while the export is open is stored, and is not
|
||||
// in the export. A write held up for the whole busy timeout would
|
||||
// fail.
|
||||
// unlockHook is a sync.Locker that runs fn each time it is unlocked. An
|
||||
// export unlocks its lock right after it opens a file.
|
||||
type unlockHook struct {
|
||||
sync.Mutex
|
||||
|
||||
fn func()
|
||||
}
|
||||
|
||||
func (u *unlockHook) Unlock() {
|
||||
u.Mutex.Unlock()
|
||||
u.fn()
|
||||
}
|
||||
|
||||
// TestArchiveExport_ReadsOneSnapshot proves an export writes a file
|
||||
// out as it was when the export opened it, and holds up no archive
|
||||
// write: a row written after the export was listed but before the file
|
||||
// was opened is in the export, and one written while the file is open
|
||||
// is stored, and is not. A write held up for the whole busy timeout
|
||||
// would fail.
|
||||
func TestArchiveExport_ReadsOneSnapshot(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: openedEventID}, 0))
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "listed"}, 0))
|
||||
|
||||
export, err := delivery.OpenArchiveExport(
|
||||
t.Context(), path, archiveTestLogger(),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
opened := &unlockHook{fn: func() {
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "during"}, 0))
|
||||
}}
|
||||
export := newExport(t, path, opened, func() (string, error) {
|
||||
return path, nil
|
||||
})
|
||||
|
||||
defer func() { require.NoError(t, export.Close()) }()
|
||||
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "during"}, 0))
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "before-open"}, 0))
|
||||
|
||||
assert.Equal(t,
|
||||
[]string{openedEventID}, exportedEventIDs(t, writeExport(t, export)),
|
||||
[]string{"listed", "before-open"},
|
||||
exportedEventIDs(t, writeExport(t, export)),
|
||||
)
|
||||
|
||||
var stored int64
|
||||
|
||||
require.NoError(t, openArchiveDBForRead(t, path).
|
||||
Model(&delivery.ExportArchivedEvent{}).Count(&stored).Error)
|
||||
assert.Equal(t, int64(2), stored)
|
||||
assert.Equal(t, int64(3), stored)
|
||||
}
|
||||
|
||||
// TestArchiveExport_SurvivesRename proves that renaming the archive
|
||||
// while an export of it is open, as renaming its webhook or target
|
||||
// does, leaves the export reading the same file.
|
||||
func TestArchiveExport_SurvivesRename(t *testing.T) {
|
||||
// TestArchiveExport_FindsFilesAfterRename proves that renaming the
|
||||
// archive after an export has listed it, as renaming its webhook or
|
||||
// target does, loses no file: the export finds each file again by its
|
||||
// period under the new name. A file moved away by then is skipped.
|
||||
func TestArchiveExport_FindsFilesAfterRename(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-old.db")
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "archive-old.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: openedEventID}, 0))
|
||||
for _, period := range []string{"", dayPeriod, hourPeriod} {
|
||||
require.NoError(t, w.WritePeriod(
|
||||
delivery.ExportArchivedEvent{EventID: "in-" + period}, 0, period,
|
||||
))
|
||||
}
|
||||
|
||||
export, err := delivery.OpenArchiveExport(
|
||||
t.Context(), path, archiveTestLogger(),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { require.NoError(t, export.Close()) }()
|
||||
current := path
|
||||
export := newExport(t, path, &sync.Mutex{}, func() (string, error) {
|
||||
return current, nil
|
||||
})
|
||||
|
||||
require.NoError(t, w.Rename("archive-new.db"))
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "after"}, 0))
|
||||
require.NoFileExists(t, path)
|
||||
|
||||
current = filepath.Join(dir, "archive-new.db")
|
||||
|
||||
removeArchiveFiles(t, periodPath(current, dayPeriod))
|
||||
|
||||
assert.Equal(t,
|
||||
[]string{openedEventID}, exportedEventIDs(t, writeExport(t, export)),
|
||||
[]string{"in-", "in-" + hourPeriod},
|
||||
exportedEventIDs(t, writeExport(t, export)),
|
||||
)
|
||||
}
|
||||
|
||||
// TestArchiveExport_EveryFileOldestFirst writes a row to a target's
|
||||
// file without a period and to its files for a month, an hour and a
|
||||
// day, and proves the export holds every row: the file without a
|
||||
// period first, then the others oldest period first, each row from a
|
||||
// file named for a period carrying that period. A file made after the
|
||||
// export was listed is not in it.
|
||||
func TestArchiveExport_EveryFileOldestFirst(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-wh.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
|
||||
// Written in an order that is not the export's.
|
||||
for _, period := range []string{nextDayPeriod, "", hourPeriod, "2026-03"} {
|
||||
require.NoError(t, w.WritePeriod(
|
||||
delivery.ExportArchivedEvent{EventID: "in-" + period}, 0, period,
|
||||
))
|
||||
}
|
||||
|
||||
export := listExport(t, path)
|
||||
|
||||
require.NoError(t, w.WritePeriod(
|
||||
delivery.ExportArchivedEvent{EventID: "later"}, 0, "2026-03-06",
|
||||
))
|
||||
|
||||
events := exportedEvents(t, writeExport(t, export))
|
||||
ids := make([]string, 0, len(events))
|
||||
periods := make([]any, 0, len(events))
|
||||
|
||||
for _, ev := range events {
|
||||
ids = append(ids, fmt.Sprint(ev["event_id"]))
|
||||
periods = append(periods, ev["period"])
|
||||
}
|
||||
|
||||
assert.Equal(t,
|
||||
[]string{"in-", "in-2026-03", "in-" + hourPeriod, "in-" + nextDayPeriod},
|
||||
ids,
|
||||
)
|
||||
assert.Equal(t,
|
||||
[]any{nil, "2026-03", hourPeriod, nextDayPeriod}, periods,
|
||||
)
|
||||
assert.NotContains(t, events[0], "period",
|
||||
"a row from the file without a period has no period")
|
||||
}
|
||||
|
||||
// openFilesPeak is an io.Writer that discards what it is given and
|
||||
// records the most archive files in dir the process had open at any
|
||||
// write, as /proc/self/fd lists the files a process has open.
|
||||
type openFilesPeak struct {
|
||||
dir string
|
||||
max int
|
||||
}
|
||||
|
||||
func (p *openFilesPeak) Write(b []byte) (int, error) {
|
||||
fds, err := os.ReadDir("/proc/self/fd")
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
open := map[string]bool{}
|
||||
|
||||
for _, fd := range fds {
|
||||
file, err := os.Readlink(filepath.Join("/proc/self/fd", fd.Name()))
|
||||
if err == nil && filepath.Dir(file) == p.dir &&
|
||||
strings.HasSuffix(file, ".db") {
|
||||
open[file] = true
|
||||
}
|
||||
}
|
||||
|
||||
p.max = max(p.max, len(open))
|
||||
|
||||
return len(b), nil
|
||||
}
|
||||
|
||||
// TestArchiveExport_OneFileOpenAtATime exports a target with a file for
|
||||
// each of 24 hours and proves the export never had more than one of
|
||||
// them open, and had one open while it wrote. Each file holds a row of
|
||||
// 48 KiB of random base64, which gzip shrinks little, so the export
|
||||
// writes output while it reads each file.
|
||||
func TestArchiveExport_OneFileOpenAtATime(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
if runtime.GOOS != "linux" {
|
||||
t.Skip("only Linux lists a process's open files in /proc/self/fd")
|
||||
}
|
||||
|
||||
// Readlink gives each open file's path with no symbolic link in it.
|
||||
dir, err := filepath.EvalSymlinks(t.TempDir())
|
||||
require.NoError(t, err)
|
||||
|
||||
path := filepath.Join(dir, "archive-wh.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
random := make([]byte, 36<<10)
|
||||
|
||||
for hour := range 24 {
|
||||
_, _ = rand.Read(random)
|
||||
|
||||
require.NoError(t, w.WritePeriod(delivery.ExportArchivedEvent{
|
||||
Body: base64.StdEncoding.EncodeToString(random),
|
||||
}, 0, fmt.Sprintf("2026-10-01-%02d", hour)))
|
||||
}
|
||||
|
||||
// The writer's own handle on the last file is not the export's.
|
||||
w.Evict()
|
||||
|
||||
peak := &openFilesPeak{dir: dir}
|
||||
|
||||
require.NoError(t, writeExportTo(t, listExport(t, path), peak))
|
||||
assert.Equal(t, 1, peak.max)
|
||||
}
|
||||
|
||||
// heapPeak is an io.Writer that discards what it is given and records
|
||||
// the largest heap it saw at a write. It collects garbage before each
|
||||
// reading, so the heap it reads is what is still held.
|
||||
@@ -338,12 +489,7 @@ func exportHeapGrowth(t *testing.T, rows, bodySize int) uint64 {
|
||||
}, 0))
|
||||
}
|
||||
|
||||
export, err := delivery.OpenArchiveExport(
|
||||
t.Context(), path, archiveTestLogger(),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { require.NoError(t, export.Close()) }()
|
||||
export := listExport(t, path)
|
||||
|
||||
runtime.GC()
|
||||
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
package delivery
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// The archive rotations: how often a database target starts a new
|
||||
// archive file. Every rotation but none puts the period of an event's
|
||||
// receive time, in UTC, in the name of the file the event goes to,
|
||||
// written in the layout beside it.
|
||||
const (
|
||||
archiveRotationNone = "none"
|
||||
archiveRotationMonthly = "monthly"
|
||||
archiveRotationDaily = "daily"
|
||||
archiveRotationHourly = "hourly"
|
||||
|
||||
archiveMonthLayout = "2006-01"
|
||||
archiveDayLayout = "2006-01-02"
|
||||
archiveHourLayout = "2006-01-02-15"
|
||||
)
|
||||
|
||||
// errArchiveRotationUnknown is returned for a rotation that is not one
|
||||
// of the four.
|
||||
var errArchiveRotationUnknown = errors.New(
|
||||
"rotation must be none, monthly, daily or hourly",
|
||||
)
|
||||
|
||||
// archiveFile is one of a database target's archive files, and the
|
||||
// period in its name: "" for the file named without one.
|
||||
type archiveFile struct {
|
||||
path string
|
||||
period string
|
||||
}
|
||||
|
||||
// ValidateArchiveRotation checks a user-supplied archive rotation for
|
||||
// a database target: empty or none (both meaning one file), monthly,
|
||||
// daily or hourly.
|
||||
func ValidateArchiveRotation(rotation string) error {
|
||||
switch rotation {
|
||||
case "", archiveRotationNone, archiveRotationMonthly,
|
||||
archiveRotationDaily, archiveRotationHourly:
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("%w: %q", errArchiveRotationUnknown, rotation)
|
||||
}
|
||||
}
|
||||
|
||||
// parseArchiveRotation reads the rotation from a database target's
|
||||
// config JSON. An empty config or an empty rotation is none.
|
||||
func parseArchiveRotation(configJSON string) (string, error) {
|
||||
if configJSON == "" {
|
||||
return archiveRotationNone, nil
|
||||
}
|
||||
|
||||
var cfg databaseTargetConfig
|
||||
|
||||
err := json.Unmarshal([]byte(configJSON), &cfg)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("parsing database target config: %w", err)
|
||||
}
|
||||
|
||||
err = ValidateArchiveRotation(cfg.Rotation)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
if cfg.Rotation == "" {
|
||||
return archiveRotationNone, nil
|
||||
}
|
||||
|
||||
return cfg.Rotation, nil
|
||||
}
|
||||
|
||||
// archivePeriod returns the period, in UTC, that a rotation puts an
|
||||
// event received at receivedAt in: "2026-10" for monthly,
|
||||
// "2026-10-01" for daily, "2026-10-01-19" for hourly, and "" for none.
|
||||
func archivePeriod(rotation string, receivedAt time.Time) string {
|
||||
switch rotation {
|
||||
case archiveRotationMonthly:
|
||||
return receivedAt.UTC().Format(archiveMonthLayout)
|
||||
case archiveRotationDaily:
|
||||
return receivedAt.UTC().Format(archiveDayLayout)
|
||||
case archiveRotationHourly:
|
||||
return receivedAt.UTC().Format(archiveHourLayout)
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// archivePeriodPath returns the path of a database target's archive
|
||||
// file for a period: path, as ArchivePath gives it, with "-" and the
|
||||
// period put before its ".db". The period "" gives path itself.
|
||||
func archivePeriodPath(path, period string) string {
|
||||
if period == "" {
|
||||
return path
|
||||
}
|
||||
|
||||
return strings.TrimSuffix(path, ".db") + "-" + period + ".db"
|
||||
}
|
||||
|
||||
// ArchivePathAt returns the archive file a database target writes an
|
||||
// event received at receivedAt to: ArchivePath's file, with the
|
||||
// period in its name when the target rotates.
|
||||
func ArchivePathAt(
|
||||
dbMgr *database.WebhookDBManager,
|
||||
webhook *database.Webhook,
|
||||
target *database.Target,
|
||||
receivedAt time.Time,
|
||||
) (string, error) {
|
||||
rotation, err := parseArchiveRotation(target.Config)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return archivePeriodPath(
|
||||
ArchivePath(dbMgr, webhook, target),
|
||||
archivePeriod(rotation, receivedAt),
|
||||
), nil
|
||||
}
|
||||
|
||||
// archiveFiles lists the database target's archive files that exist,
|
||||
// given the path ArchivePath gives it: the file at path, then each
|
||||
// file archivePeriodPath names for path and a period, oldest period
|
||||
// first. Which rotation wrote a file does not matter, so the files of
|
||||
// an earlier rotation setting are listed too.
|
||||
func archiveFiles(path string) ([]archiveFile, error) {
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing archive files: %w", err)
|
||||
}
|
||||
|
||||
stem := strings.TrimSuffix(filepath.Base(path), ".db")
|
||||
|
||||
var files []archiveFile
|
||||
|
||||
for _, entry := range entries {
|
||||
period, ok := archiveFilePeriod(stem, entry.Name())
|
||||
if ok {
|
||||
files = append(files, archiveFile{
|
||||
path: filepath.Join(dir, entry.Name()),
|
||||
period: period,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A month sorts before the days and hours in it.
|
||||
slices.SortFunc(files, func(a, b archiveFile) int {
|
||||
return strings.Compare(a.period, b.period)
|
||||
})
|
||||
|
||||
return files, nil
|
||||
}
|
||||
|
||||
// archiveFilePeriod reports whether name is the name of an archive
|
||||
// file of the target whose file name without a period is stem+".db",
|
||||
// and the period in it.
|
||||
func archiveFilePeriod(stem, name string) (string, bool) {
|
||||
rest, ok := strings.CutPrefix(name, stem)
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
|
||||
rest, ok = strings.CutSuffix(rest, ".db")
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
|
||||
if rest == "" {
|
||||
return "", true
|
||||
}
|
||||
|
||||
period, ok := strings.CutPrefix(rest, "-")
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
|
||||
for _, layout := range []string{
|
||||
archiveMonthLayout, archiveDayLayout, archiveHourLayout,
|
||||
} {
|
||||
_, err := time.Parse(layout, period)
|
||||
if err == nil {
|
||||
return period, true
|
||||
}
|
||||
}
|
||||
|
||||
return "", false
|
||||
}
|
||||
@@ -0,0 +1,389 @@
|
||||
package delivery_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// The archive rotations, and the configs of a daily target and of one
|
||||
// whose rotation is not one of the four.
|
||||
const (
|
||||
rotationNone = "none"
|
||||
rotationMonthly = "monthly"
|
||||
rotationDaily = "daily"
|
||||
rotationHourly = "hourly"
|
||||
|
||||
dailyConfig = `{"rotation":"daily"}`
|
||||
weeklyConfig = `{"rotation":"weekly"}`
|
||||
)
|
||||
|
||||
// The periods the tests archive into most: two days, and an hour of
|
||||
// the first.
|
||||
const (
|
||||
dayPeriod = "2026-03-04"
|
||||
nextDayPeriod = "2026-03-05"
|
||||
hourPeriod = "2026-03-04-05"
|
||||
)
|
||||
|
||||
// periodPath returns the archive file for a period of the target whose
|
||||
// file without a period is path.
|
||||
func periodPath(path, period string) string {
|
||||
return strings.TrimSuffix(path, ".db") + "-" + period + ".db"
|
||||
}
|
||||
|
||||
// deliverReceivedAt delivers to a database target an event whose
|
||||
// receive time is receivedAt, and returns the event's id. The receive
|
||||
// time is what decides a rotated archive's file, so setting it is how
|
||||
// these tests move the clock across a period boundary.
|
||||
func (env *archiveEnv) deliverReceivedAt(
|
||||
t *testing.T, tgt *database.Target, receivedAt time.Time,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
webhookDB := testWebhookDB(t)
|
||||
event := seedEvent(t, webhookDB, `{"n":1}`)
|
||||
event.CreatedAt = receivedAt
|
||||
|
||||
env.eng.ExportDeliverDatabase(
|
||||
webhookDB, seedDatabaseTargetDelivery(t, webhookDB, event, tgt),
|
||||
)
|
||||
|
||||
return event.ID
|
||||
}
|
||||
|
||||
// TestDeliverDatabase_RotatesAtEachPeriodBoundary delivers, for each
|
||||
// rotation, an event received in the last second of a period and one
|
||||
// received in the first second of the next, and checks each lands in
|
||||
// the file named for its own period, in UTC. Rotation none keeps both
|
||||
// in the one file.
|
||||
func TestDeliverDatabase_RotatesAtEachPeriodBoundary(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
berlin := time.FixedZone("CEST", 2*60*60)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
rotation string
|
||||
before, after time.Time
|
||||
// periods are the periods of before and after.
|
||||
periods [2]string
|
||||
}{
|
||||
{
|
||||
rotationMonthly, rotationMonthly,
|
||||
time.Date(2026, 1, 31, 23, 59, 59, 0, time.UTC),
|
||||
time.Date(2026, 2, 1, 0, 0, 0, 0, time.UTC),
|
||||
[2]string{"2026-01", "2026-02"},
|
||||
},
|
||||
{
|
||||
rotationDaily, rotationDaily,
|
||||
time.Date(2026, 3, 4, 23, 59, 59, 0, time.UTC),
|
||||
time.Date(2026, 3, 5, 0, 0, 0, 0, time.UTC),
|
||||
[2]string{dayPeriod, nextDayPeriod},
|
||||
},
|
||||
{
|
||||
// The same instants, received in a zone two hours ahead
|
||||
// of UTC, where they fall on 5 March: the period is UTC's.
|
||||
"daily in another zone", rotationDaily,
|
||||
time.Date(2026, 3, 5, 1, 59, 59, 0, berlin),
|
||||
time.Date(2026, 3, 5, 2, 0, 0, 0, berlin),
|
||||
[2]string{dayPeriod, nextDayPeriod},
|
||||
},
|
||||
{
|
||||
rotationHourly, rotationHourly,
|
||||
time.Date(2026, 3, 4, 5, 59, 59, 0, time.UTC),
|
||||
time.Date(2026, 3, 4, 6, 0, 0, 0, time.UTC),
|
||||
[2]string{hourPeriod, "2026-03-04-06"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, `{"rotation":"`+tc.rotation+`"}`)
|
||||
path := env.archivePath(tgt)
|
||||
|
||||
first := env.deliverReceivedAt(t, tgt, tc.before)
|
||||
second := env.deliverReceivedAt(t, tgt, tc.after)
|
||||
|
||||
assert.Equal(t, []string{first},
|
||||
archivedEventIDs(t, periodPath(path, tc.periods[0])))
|
||||
assert.Equal(t, []string{second},
|
||||
archivedEventIDs(t, periodPath(path, tc.periods[1])))
|
||||
assert.NoFileExists(t, path,
|
||||
"a rotated target never writes the file without a period")
|
||||
})
|
||||
}
|
||||
|
||||
t.Run(rotationNone, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, `{"rotation":"`+rotationNone+`"}`)
|
||||
|
||||
first := env.deliverReceivedAt(t, tgt, cases[0].before)
|
||||
second := env.deliverReceivedAt(t, tgt, cases[0].after)
|
||||
|
||||
assert.ElementsMatch(t, []string{first, second},
|
||||
archivedEventIDs(t, env.archivePath(tgt)))
|
||||
})
|
||||
}
|
||||
|
||||
// TestDeliverDatabase_EventWithoutReceiveTime proves an event whose
|
||||
// receive time is not known goes to the file for the time it is
|
||||
// archived, rather than to one for the year 1.
|
||||
func TestDeliverDatabase_EventWithoutReceiveTime(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, `{"rotation":"`+rotationMonthly+`"}`)
|
||||
path := env.archivePath(tgt)
|
||||
|
||||
before := time.Now().UTC().Format("2006-01")
|
||||
id := env.deliverReceivedAt(t, tgt, time.Time{})
|
||||
|
||||
file := periodPath(path, time.Now().UTC().Format("2006-01"))
|
||||
|
||||
_, err := os.Stat(file)
|
||||
if err != nil {
|
||||
// The month turned during the delivery.
|
||||
file = periodPath(path, before)
|
||||
}
|
||||
|
||||
assert.Equal(t, []string{id}, archivedEventIDs(t, file))
|
||||
assert.NoFileExists(t, periodPath(path, "0001-01"))
|
||||
}
|
||||
|
||||
// TestDeliverDatabase_RotationChangeKeepsOldFiles changes a target's
|
||||
// rotation from none to daily between two events, and checks the
|
||||
// second goes to the daily file while the first stays where it was.
|
||||
func TestDeliverDatabase_RotationChangeKeepsOldFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
path := env.archivePath(tgt)
|
||||
at := time.Date(2026, 3, 4, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
first := env.deliverReceivedAt(t, tgt, at)
|
||||
|
||||
tgt.Config = dailyConfig
|
||||
second := env.deliverReceivedAt(t, tgt, at)
|
||||
|
||||
assert.Equal(t, []string{first}, archivedEventIDs(t, path))
|
||||
assert.Equal(t, []string{second},
|
||||
archivedEventIDs(t, periodPath(path, dayPeriod)))
|
||||
}
|
||||
|
||||
// TestArchiveSweep_PrunesEveryFile gives a daily target three files:
|
||||
// the file without a period, left from before it rotated, and two
|
||||
// daily files. Each holds a row older than the expiry, and one daily
|
||||
// file also a newer row. The sweep prunes the old row from every file,
|
||||
// deletes the daily file it leaves empty, and keeps the file without a
|
||||
// period although it is empty too.
|
||||
func TestArchiveSweep_PrunesEveryFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(
|
||||
t, `{"expiry":"1h","rotation":"`+rotationDaily+`"}`,
|
||||
)
|
||||
path := env.archivePath(tgt)
|
||||
emptied := periodPath(path, dayPeriod)
|
||||
kept := periodPath(path, nextDayPeriod)
|
||||
|
||||
now := time.Now()
|
||||
old := now.Add(-48 * time.Hour)
|
||||
|
||||
seedArchiveFile(t, path, tgt.WebhookID, old)
|
||||
seedArchiveFile(t, emptied, tgt.WebhookID, old)
|
||||
seedArchiveFile(t, kept, tgt.WebhookID, old, now.Add(-time.Minute))
|
||||
|
||||
env.sweeper.ExportSweep(t.Context())
|
||||
|
||||
assert.Empty(t, archivedEventIDs(t, path))
|
||||
assert.Equal(t, []string{sweepRowNew}, archivedEventIDs(t, kept))
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
assert.NoFileExists(t, emptied+suffix)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRename_MovesEveryFile renames a daily target that also has a
|
||||
// file without a period, and checks every file moves to the new name
|
||||
// with its period, rows and all, and that a later write uses the new
|
||||
// name.
|
||||
func TestRename_MovesEveryFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, "")
|
||||
oldPath := env.archivePath(tgt)
|
||||
day := time.Date(2026, 3, 4, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
unrotated := env.deliverReceivedAt(t, tgt, day)
|
||||
|
||||
tgt.Config = dailyConfig
|
||||
first := env.deliverReceivedAt(t, tgt, day)
|
||||
second := env.deliverReceivedAt(t, tgt, day.Add(24*time.Hour))
|
||||
|
||||
require.NoError(t, env.eng.Rename(tgt.ID, "Orders", "Long Term"))
|
||||
|
||||
newPath := filepath.Join(
|
||||
env.dataDir, "archive-orders-long-term-"+tgt.ID+".db",
|
||||
)
|
||||
|
||||
for _, old := range []string{
|
||||
oldPath,
|
||||
periodPath(oldPath, dayPeriod),
|
||||
periodPath(oldPath, nextDayPeriod),
|
||||
} {
|
||||
assert.NoFileExists(t, old)
|
||||
}
|
||||
|
||||
assert.Equal(t, []string{unrotated}, archivedEventIDs(t, newPath))
|
||||
assert.Equal(t, []string{first},
|
||||
archivedEventIDs(t, periodPath(newPath, dayPeriod)))
|
||||
assert.Equal(t, []string{second},
|
||||
archivedEventIDs(t, periodPath(newPath, nextDayPeriod)))
|
||||
|
||||
third := env.deliverReceivedAt(t, tgt, day.Add(48*time.Hour))
|
||||
assert.Equal(t, []string{third},
|
||||
archivedEventIDs(t, periodPath(newPath, "2026-03-06")))
|
||||
}
|
||||
|
||||
// TestRename_NeverReplacesARotatedFile plants a file at the new name
|
||||
// of a target's daily file, and proves the rename is refused and moves
|
||||
// none of the target's files.
|
||||
func TestRename_NeverReplacesARotatedFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupArchiveTest(t)
|
||||
tgt := env.seedDatabaseTarget(t, dailyConfig)
|
||||
oldPath := env.archivePath(tgt)
|
||||
day := time.Date(2026, 3, 4, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
env.deliverReceivedAt(t, tgt, day)
|
||||
env.deliverReceivedAt(t, tgt, day.Add(24*time.Hour))
|
||||
|
||||
newPath := filepath.Join(
|
||||
env.dataDir, "archive-orders-long-term-"+tgt.ID+".db",
|
||||
)
|
||||
planted := periodPath(newPath, nextDayPeriod)
|
||||
require.NoError(t, os.WriteFile(planted, []byte("planted"), 0o600))
|
||||
|
||||
require.ErrorIs(
|
||||
t, env.eng.Rename(tgt.ID, "Orders", "Long Term"),
|
||||
delivery.ErrArchiveNameTaken,
|
||||
)
|
||||
|
||||
assert.FileExists(t, periodPath(oldPath, dayPeriod))
|
||||
assert.FileExists(t, periodPath(oldPath, nextDayPeriod))
|
||||
assert.NoFileExists(t, periodPath(newPath, dayPeriod))
|
||||
}
|
||||
|
||||
// TestStatArchive_EveryFile proves StatArchive counts and adds up every
|
||||
// one of a target's files, takes the latest write of any of them, and
|
||||
// leaves out files whose names only look like the target's.
|
||||
func TestStatArchive_EveryFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "archive-wh.db")
|
||||
files := []string{
|
||||
path, periodPath(path, "2026-03"), periodPath(path, hourPeriod),
|
||||
}
|
||||
written := time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC)
|
||||
|
||||
var size int64
|
||||
|
||||
for i, file := range files {
|
||||
require.NoError(t, os.WriteFile(file, make([]byte, 100*(i+1)), 0o600))
|
||||
|
||||
size += int64(100 * (i + 1))
|
||||
at := written.Add(-time.Duration(i) * time.Hour)
|
||||
require.NoError(t, os.Chtimes(file, at, at))
|
||||
}
|
||||
|
||||
for _, other := range []string{
|
||||
"archive-wh-2026-13.db", "archive-wh-2026-3.db",
|
||||
"archive-wh-other.db", "archive-wh-2026-03.json",
|
||||
"archive-whx.db",
|
||||
} {
|
||||
require.NoError(t,
|
||||
os.WriteFile(filepath.Join(dir, other), []byte("x"), 0o600))
|
||||
}
|
||||
|
||||
got, err := delivery.StatArchive(path)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, len(files), got.Files)
|
||||
assert.Equal(t, size, got.Size)
|
||||
assert.True(t, written.Equal(got.Written), got.Written)
|
||||
}
|
||||
|
||||
// TestArchivePathAt names the file each rotation writes an event to.
|
||||
func TestArchivePathAt(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := t.TempDir()
|
||||
dbMgr := database.NewTestWebhookDBManager(dataDir)
|
||||
webhook := &database.Webhook{
|
||||
BaseModel: database.BaseModel{ID: "wh-id"}, Name: "Orders",
|
||||
}
|
||||
at := time.Date(2026, 10, 1, 19, 30, 0, 0, time.UTC)
|
||||
|
||||
cases := map[string]string{
|
||||
"": "",
|
||||
`{"rotation":"` + rotationNone + `"}`: "",
|
||||
`{"rotation":"` + rotationMonthly + `"}`: "-2026-10",
|
||||
dailyConfig: "-2026-10-01",
|
||||
`{"rotation":"` + rotationHourly + `"}`: "-2026-10-01-19",
|
||||
}
|
||||
|
||||
for config, period := range cases {
|
||||
target := &database.Target{
|
||||
BaseModel: database.BaseModel{ID: "tgt-id"},
|
||||
Name: "Archive",
|
||||
Config: config,
|
||||
}
|
||||
|
||||
got, err := delivery.ArchivePathAt(dbMgr, webhook, target, at)
|
||||
require.NoError(t, err, config)
|
||||
assert.Equal(t,
|
||||
filepath.Join(
|
||||
dataDir, "archive-orders-archive-tgt-id"+period+".db",
|
||||
),
|
||||
got, config,
|
||||
)
|
||||
}
|
||||
|
||||
_, err := delivery.ArchivePathAt(dbMgr, webhook, &database.Target{
|
||||
Config: weeklyConfig,
|
||||
}, at)
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
// TestValidateArchiveRotation accepts the four rotations, and empty,
|
||||
// and refuses anything else.
|
||||
func TestValidateArchiveRotation(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, ok := range []string{
|
||||
"", rotationNone, rotationMonthly, rotationDaily, rotationHourly,
|
||||
} {
|
||||
require.NoError(t, delivery.ValidateArchiveRotation(ok), ok)
|
||||
}
|
||||
|
||||
for _, bad := range []string{"weekly", "Daily", "hourly "} {
|
||||
require.Error(t, delivery.ValidateArchiveRotation(bad), bad)
|
||||
}
|
||||
}
|
||||
@@ -218,6 +218,7 @@ func TestStatArchive(t *testing.T) {
|
||||
|
||||
got, err := delivery.StatArchive(path)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, got.Files)
|
||||
assert.Equal(t, file.Size()+wal.Size(), got.Size)
|
||||
assert.True(t, written.Equal(got.Written), got.Written)
|
||||
|
||||
@@ -720,3 +721,41 @@ func TestArchiveWriter_RenameMovesBackOnFailure(t *testing.T) {
|
||||
assert.NoFileExists(t, filepath.Join(dir, newName))
|
||||
assert.Equal(t, oldPath, w.Path())
|
||||
}
|
||||
|
||||
// TestArchiveWriter_RenameMovesBackEveryFile renames a target with a
|
||||
// file without a period and a file for a month, and proves that when
|
||||
// the month's file fails to move, the file already moved is moved back:
|
||||
// both files are under the old name with their rows, and nothing is
|
||||
// under the new name. The new name is 251 bytes, so the file without a
|
||||
// period, with its -wal and -shm, can take it, but the month's file,
|
||||
// eight bytes longer, cannot.
|
||||
func TestArchiveWriter_RenameMovesBackEveryFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
oldPath := filepath.Join(dir, "archive-old.db")
|
||||
monthPath := filepath.Join(dir, "archive-old-2026-03.db")
|
||||
w := delivery.NewExportArchiveWriter(oldPath, archiveTestLogger(), 0)
|
||||
|
||||
require.NoError(t, w.WritePeriod(
|
||||
delivery.ExportArchivedEvent{EventID: "in-none"}, 0, "",
|
||||
))
|
||||
require.NoError(t, w.WritePeriod(
|
||||
delivery.ExportArchivedEvent{EventID: "in-month"}, 0, "2026-03",
|
||||
))
|
||||
|
||||
require.Error(t, w.Rename(strings.Repeat("a", 248)+".db"))
|
||||
|
||||
assert.Equal(t, []string{"in-none"}, archivedEventIDs(t, oldPath))
|
||||
assert.Equal(t, []string{"in-month"}, archivedEventIDs(t, monthPath))
|
||||
|
||||
entries, err := os.ReadDir(dir)
|
||||
require.NoError(t, err)
|
||||
|
||||
for _, entry := range entries {
|
||||
assert.True(t, strings.HasPrefix(entry.Name(), "archive-old"),
|
||||
"%s is not under the old name", entry.Name())
|
||||
}
|
||||
|
||||
assert.Equal(t, oldPath, w.Path())
|
||||
}
|
||||
|
||||
@@ -204,10 +204,11 @@ func TestNewTargetConfigForm(t *testing.T) {
|
||||
|
||||
form, err = delivery.NewTargetConfigForm(&database.Target{
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: `{"expiry":"720h"}`,
|
||||
Config: `{"expiry":"720h","rotation":"daily"}`,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "720h", form.Expiry)
|
||||
assert.Equal(t, rotationDaily, form.Rotation)
|
||||
|
||||
form, err = delivery.NewTargetConfigForm(&database.Target{
|
||||
Type: database.TargetTypeLog,
|
||||
@@ -248,6 +249,10 @@ func TestNewTargetConfigForm_UnreadableConfigErrors(t *testing.T) {
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: `{"expiry":"soon"}`,
|
||||
},
|
||||
{
|
||||
Type: database.TargetTypeDatabase,
|
||||
Config: weeklyConfig,
|
||||
},
|
||||
{Type: database.TargetType("nope")},
|
||||
}
|
||||
|
||||
|
||||
@@ -10,10 +10,10 @@ const (
|
||||
tmplKeyArchiveExpiryChoices = "ArchiveExpiryChoices"
|
||||
)
|
||||
|
||||
// archiveExpiryChoice is one entry of a database target's archive
|
||||
// expiry select: the expiry stored, the label shown, and whether the
|
||||
// select starts on it.
|
||||
type archiveExpiryChoice struct {
|
||||
// archiveChoice is one entry of a database target's archive expiry
|
||||
// or archive rotation select: the value stored, the label shown, and
|
||||
// whether the select starts on it.
|
||||
type archiveChoice struct {
|
||||
Value string
|
||||
Label string
|
||||
Selected bool
|
||||
@@ -21,8 +21,8 @@ type archiveExpiryChoice struct {
|
||||
|
||||
// archiveExpiryChoices lists the archive expiries offered by the new
|
||||
// webhook page, the add target form and the target edit form.
|
||||
func archiveExpiryChoices() []archiveExpiryChoice {
|
||||
return []archiveExpiryChoice{
|
||||
func archiveExpiryChoices() []archiveChoice {
|
||||
return []archiveChoice{
|
||||
{Value: archiveExpiryNever, Label: archiveExpiryNever},
|
||||
{Value: "1h", Label: "1h"},
|
||||
{Value: "12h", Label: "12h"},
|
||||
@@ -37,7 +37,7 @@ func archiveExpiryChoices() []archiveExpiryChoice {
|
||||
// empty expiry selects never. An expiry that is not one of the
|
||||
// choices comes first as its own selected entry, so saving the form
|
||||
// unchanged keeps it.
|
||||
func archiveExpiryOptions(expiry string) []archiveExpiryChoice {
|
||||
func archiveExpiryOptions(expiry string) []archiveChoice {
|
||||
if expiry == "" {
|
||||
expiry = archiveExpiryNever
|
||||
}
|
||||
@@ -52,7 +52,7 @@ func archiveExpiryOptions(expiry string) []archiveExpiryChoice {
|
||||
}
|
||||
}
|
||||
|
||||
own := archiveExpiryChoice{Value: expiry, Label: expiry, Selected: true}
|
||||
own := archiveChoice{Value: expiry, Label: expiry, Selected: true}
|
||||
|
||||
return append([]archiveExpiryChoice{own}, options...)
|
||||
return append([]archiveChoice{own}, options...)
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -49,21 +50,39 @@ func expiryShown(
|
||||
}
|
||||
|
||||
// expirySelected returns the target edit page and the expiries its
|
||||
// select starts on.
|
||||
// expiry select starts on.
|
||||
func expirySelected(
|
||||
t *testing.T, env *sourceTestEnv, webhookID, targetID string,
|
||||
) (string, []string) {
|
||||
t.Helper()
|
||||
|
||||
page := targetEditPage(t, env, webhookID, targetID)
|
||||
|
||||
return page, selectedIn(page, "expiry")
|
||||
}
|
||||
|
||||
// targetEditPage returns a target's edit page.
|
||||
func targetEditPage(
|
||||
t *testing.T, env *sourceTestEnv, webhookID, targetID string,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodGet,
|
||||
"/hook/"+webhookID+"/targets/"+targetID+"/edit", nil,
|
||||
)
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
|
||||
page := w.Body.String()
|
||||
return w.Body.String()
|
||||
}
|
||||
|
||||
return page, matched(`<option value="([^"]*)" selected>`, page)
|
||||
// selectedIn returns the values the select named name on page starts
|
||||
// on.
|
||||
func selectedIn(page, name string) []string {
|
||||
_, rest, _ := strings.Cut(page, `<select id="`+name+`" name="`+name+`"`)
|
||||
options, _, _ := strings.Cut(rest, "</select>")
|
||||
|
||||
return matched(`<option value="([^"]*)" selected>`, options)
|
||||
}
|
||||
|
||||
// TestArchiveExpiryChoices adds a database target with each archive
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
package handlers
|
||||
|
||||
const (
|
||||
// archiveRotationNone is the archive rotation that keeps a
|
||||
// database target's archive in one file. A stored empty rotation
|
||||
// means the same.
|
||||
archiveRotationNone = "none"
|
||||
|
||||
// tmplKeyArchiveRotationChoices is the template data key for the
|
||||
// entries of a page's archive rotation select.
|
||||
tmplKeyArchiveRotationChoices = "ArchiveRotationChoices"
|
||||
)
|
||||
|
||||
// archiveRotationChoices lists the archive rotations offered by the
|
||||
// new webhook page, the add target form and the target edit form.
|
||||
func archiveRotationChoices() []archiveChoice {
|
||||
return []archiveChoice{
|
||||
{Value: archiveRotationNone, Label: archiveRotationNone},
|
||||
{Value: "monthly", Label: "monthly"},
|
||||
{Value: "daily", Label: "daily"},
|
||||
{Value: "hourly", Label: "hourly"},
|
||||
}
|
||||
}
|
||||
|
||||
// archiveRotationOptions returns the choices with rotation selected;
|
||||
// an empty rotation, or one that is not a choice, selects none. A
|
||||
// stored rotation is always a choice: the forms refuse any other.
|
||||
func archiveRotationOptions(rotation string) []archiveChoice {
|
||||
options := archiveRotationChoices()
|
||||
|
||||
for i := range options {
|
||||
if options[i].Value == rotation {
|
||||
options[i].Selected = true
|
||||
|
||||
return options
|
||||
}
|
||||
}
|
||||
|
||||
options[0].Selected = true
|
||||
|
||||
return options
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// rotationNone is the archive rotation that keeps one file.
|
||||
const rotationNone = "none"
|
||||
|
||||
// rotationShown returns the archive rotations the webhook page's
|
||||
// target list shows.
|
||||
func rotationShown(
|
||||
t *testing.T, env *sourceTestEnv, webhookID string,
|
||||
) []string {
|
||||
t.Helper()
|
||||
|
||||
return matched(
|
||||
`Archive Rotation:</span>\s*<span>([^<]*)</span>`,
|
||||
renderedPage(t, env, webhookID),
|
||||
)
|
||||
}
|
||||
|
||||
// TestArchiveRotationChoices adds a database target with each archive
|
||||
// rotation the forms offer, and checks that it is stored as chosen,
|
||||
// shown in the target list, and that the target edit form starts on
|
||||
// it. It then edits the target to hourly, keeping its expiry.
|
||||
func TestArchiveRotationChoices(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
for _, rotation := range []string{rotationNone, "monthly", "daily", "hourly"} {
|
||||
t.Run(rotation, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("name", "archive")
|
||||
form.Set("type", string(database.TargetTypeDatabase))
|
||||
form.Set("expiry", "720h")
|
||||
form.Set("rotation", rotation)
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodPost, "/hook/"+webhook.ID+"/targets", form,
|
||||
)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
|
||||
targets := targetsForWebhook(t, env.db, webhook.ID)
|
||||
require.Len(t, targets, 1)
|
||||
assert.JSONEq(t,
|
||||
`{"expiry":"720h","rotation":"`+rotation+`"}`,
|
||||
targets[0].Config,
|
||||
)
|
||||
|
||||
assert.Equal(t,
|
||||
[]string{rotation}, rotationShown(t, env, webhook.ID))
|
||||
|
||||
page := targetEditPage(t, env, webhook.ID, targets[0].ID)
|
||||
assert.Equal(t, []string{rotation}, selectedIn(page, "rotation"))
|
||||
|
||||
form = url.Values{}
|
||||
form.Set("name", "archive")
|
||||
form.Set("expiry", "720h")
|
||||
form.Set("rotation", "hourly")
|
||||
|
||||
w = submitTargetEdit(env, webhook.ID, targets[0].ID, form)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
assert.JSONEq(t,
|
||||
`{"expiry":"720h","rotation":"hourly"}`,
|
||||
storedTarget(t, env, targets[0].ID).Config,
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestArchiveRotationEditStartsOnNone checks the edit form of a
|
||||
// database target with no rotation stored starts on none.
|
||||
func TestArchiveRotationEditStartsOnNone(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
target := seedConfiguredTarget(
|
||||
t, env.db, webhook.ID, database.TargetTypeDatabase, "",
|
||||
)
|
||||
|
||||
page := targetEditPage(t, env, webhook.ID, target.ID)
|
||||
assert.Equal(t, []string{rotationNone}, selectedIn(page, "rotation"))
|
||||
assert.Equal(t, []string{rotationNone}, rotationShown(t, env, webhook.ID))
|
||||
}
|
||||
|
||||
// TestArchiveRotationRefused proves a rotation that is not one of the
|
||||
// four is refused on the add target form and the target edit form, and
|
||||
// that nothing is stored.
|
||||
func TestArchiveRotationRefused(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("name", "archive")
|
||||
form.Set("type", string(database.TargetTypeDatabase))
|
||||
form.Set("rotation", "weekly")
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodPost, "/hook/"+webhook.ID+"/targets", form,
|
||||
)
|
||||
assert.Equal(t, http.StatusBadRequest, w.Code)
|
||||
assert.Contains(t, w.Body.String(), "Invalid archive rotation")
|
||||
assert.Empty(t, targetsForWebhook(t, env.db, webhook.ID))
|
||||
|
||||
target := seedConfiguredTarget(
|
||||
t, env.db, webhook.ID, database.TargetTypeDatabase,
|
||||
`{"rotation":"daily"}`,
|
||||
)
|
||||
|
||||
form.Del("type")
|
||||
|
||||
w = submitTargetEdit(env, webhook.ID, target.ID, form)
|
||||
assert.Equal(t, http.StatusBadRequest, w.Code)
|
||||
assert.Contains(t, w.Body.String(), "Invalid archive rotation")
|
||||
assert.JSONEq(t,
|
||||
`{"rotation":"daily"}`, storedTarget(t, env, target.ID).Config,
|
||||
)
|
||||
}
|
||||
|
||||
// TestHandleSourceCreateSubmit_ArchiveRotation proves the new webhook
|
||||
// page's archive rotation is stored on the archive target it creates.
|
||||
func TestHandleSourceCreateSubmit_ArchiveRotation(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("name", "rotated")
|
||||
form.Set("archive", "on")
|
||||
form.Set("archive_expiry", "720h")
|
||||
form.Set("archive_rotation", "daily")
|
||||
|
||||
w := submitCreateForm(env, form)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
|
||||
var webhook database.Webhook
|
||||
|
||||
require.NoError(t, env.db.DB().
|
||||
Where("name = ?", "rotated").First(&webhook).Error)
|
||||
|
||||
targets := targetsForWebhook(t, env.db, webhook.ID)
|
||||
require.Len(t, targets, 1)
|
||||
assert.JSONEq(t,
|
||||
`{"expiry":"720h","rotation":"daily"}`, targets[0].Config,
|
||||
)
|
||||
}
|
||||
|
||||
// TestArchiveFileView_Rotated describes a daily target's archive files
|
||||
// at two times. On a day that has a file, the view names that file;
|
||||
// on the next, before any event, it names the file the next event
|
||||
// will go to, not created yet. Both times the size is of every file
|
||||
// together and the last write the latest of them.
|
||||
func TestArchiveFileView_Rotated(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
target := seedConfiguredTarget(
|
||||
t, env.db, webhook.ID, database.TargetTypeDatabase,
|
||||
`{"rotation":"daily"}`,
|
||||
)
|
||||
|
||||
path := delivery.ArchivePath(env.dbMgr, &webhook, target)
|
||||
stem := strings.TrimSuffix(path, ".db")
|
||||
written := time.Date(2026, 10, 2, 9, 0, 0, 0, time.UTC)
|
||||
|
||||
for i, day := range []string{"2026-10-01", "2026-10-02"} {
|
||||
file := stem + "-" + day + ".db"
|
||||
require.NoError(t, os.WriteFile(file, make([]byte, 1000), 0o600))
|
||||
|
||||
at := written.Add(time.Duration(i-1) * 24 * time.Hour)
|
||||
require.NoError(t, os.Chtimes(file, at, at))
|
||||
}
|
||||
|
||||
view := env.handlers.ArchiveFileViewForTest(
|
||||
&webhook, target, time.Date(2026, 10, 2, 23, 0, 0, 0, time.UTC),
|
||||
)
|
||||
assert.Equal(t, filepath.Base(stem)+"-2026-10-02.db", view.Name)
|
||||
assert.Empty(t, view.Note)
|
||||
assert.Equal(t, 2, view.Files)
|
||||
assert.Equal(t, "2.0 kB", view.Size)
|
||||
assert.Equal(t, "2026-10-02 09:00:00 UTC", view.WrittenUTC)
|
||||
|
||||
view = env.handlers.ArchiveFileViewForTest(
|
||||
&webhook, target, time.Date(2026, 10, 3, 0, 0, 0, 0, time.UTC),
|
||||
)
|
||||
assert.Equal(t, filepath.Base(stem)+"-2026-10-03.db", view.Name)
|
||||
assert.Equal(t, "not created yet", view.Note)
|
||||
assert.Equal(t, 2, view.Files)
|
||||
assert.Equal(t, "2.0 kB", view.Size)
|
||||
|
||||
page := targetList(t, renderedPage(t, env, webhook.ID))
|
||||
assert.Contains(t, page, "Archive Size: 2.0 kB in 2 files")
|
||||
}
|
||||
|
||||
// renderedPage returns the webhook page.
|
||||
func renderedPage(t *testing.T, env *sourceTestEnv, webhookID string) string {
|
||||
t.Helper()
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
env.handlers.HandleSourceDetail().ServeHTTP(w, getRequest(
|
||||
t, "/hook/"+webhookID, env.cookies,
|
||||
map[string]string{sourceIDParam: webhookID},
|
||||
))
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
|
||||
return w.Body.String()
|
||||
}
|
||||
@@ -167,7 +167,16 @@ func (s *Handlers) BuildHTTPTargetConfigForTest(
|
||||
// buildDatabaseTargetConfig for use in the handlers_test
|
||||
// package.
|
||||
func BuildDatabaseTargetConfigForTest(
|
||||
expiry string,
|
||||
expiry, rotation string,
|
||||
) (string, string, error) {
|
||||
return buildDatabaseTargetConfig(expiry)
|
||||
return buildDatabaseTargetConfig(expiry, rotation)
|
||||
}
|
||||
|
||||
// ArchiveFileViewForTest exposes archiveFileView, which describes a
|
||||
// database target's archive files as the target list shows them at
|
||||
// now.
|
||||
func (s *Handlers) ArchiveFileViewForTest(
|
||||
webhook *database.Webhook, target *database.Target, now time.Time,
|
||||
) *ArchiveFileView {
|
||||
return s.archiveFileView(webhook, target, now)
|
||||
}
|
||||
|
||||
@@ -100,7 +100,9 @@ type Handlers struct {
|
||||
// Interleaved, one could rename an archive between another's
|
||||
// rename and save, leaving the file named for one edit and the
|
||||
// stored names from the other. An archive download holds it while
|
||||
// it reads the stored names and opens the file they give.
|
||||
// it reads the stored names and lists the files they give, and
|
||||
// again for each file while it finds the file under the names
|
||||
// stored then and opens it.
|
||||
renameMu sync.Mutex
|
||||
|
||||
// dummyVerifications counts the equivalent-cost verifications
|
||||
|
||||
@@ -477,23 +477,37 @@ func TestRenderTemplateMidRenderErrorSendsNoPartialBody(t *testing.T) {
|
||||
func TestBuildDatabaseTargetConfig_Valid(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// Empty expiry: the keep-forever default, empty config.
|
||||
cfg, errMsg, err := handlers.BuildDatabaseTargetConfigForTest("")
|
||||
// Empty expiry and rotation: the keep-forever, one-file default,
|
||||
// empty config.
|
||||
cfg, errMsg, err := handlers.BuildDatabaseTargetConfigForTest("", "")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, errMsg)
|
||||
assert.Empty(t, cfg)
|
||||
|
||||
// Explicit never is stored as config.
|
||||
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest("never")
|
||||
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest("never", "")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, errMsg)
|
||||
assert.JSONEq(t, `{"expiry":"never"}`, cfg)
|
||||
|
||||
// A positive duration is stored as config.
|
||||
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest("720h")
|
||||
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest("720h", "")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, errMsg)
|
||||
assert.JSONEq(t, `{"expiry":"720h"}`, cfg)
|
||||
|
||||
// A rotation is stored as config, with or without an expiry.
|
||||
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest("", "daily")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, errMsg)
|
||||
assert.JSONEq(t, `{"rotation":"daily"}`, cfg)
|
||||
|
||||
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest(
|
||||
"720h", "hourly",
|
||||
)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, errMsg)
|
||||
assert.JSONEq(t, `{"expiry":"720h","rotation":"hourly"}`, cfg)
|
||||
}
|
||||
|
||||
func TestBuildDatabaseTargetConfig_RejectsBadExpiry(
|
||||
@@ -502,7 +516,7 @@ func TestBuildDatabaseTargetConfig_RejectsBadExpiry(
|
||||
t.Parallel()
|
||||
|
||||
for _, bad := range []string{"nonsense", "7d", "-5h"} {
|
||||
cfg, errMsg, err := handlers.BuildDatabaseTargetConfigForTest(bad)
|
||||
cfg, errMsg, err := handlers.BuildDatabaseTargetConfigForTest(bad, "")
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Contains(
|
||||
@@ -512,3 +526,20 @@ func TestBuildDatabaseTargetConfig_RejectsBadExpiry(
|
||||
assert.Empty(t, cfg)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildDatabaseTargetConfig_RejectsBadRotation(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
for _, bad := range []string{"weekly", "Daily", " none"} {
|
||||
cfg, errMsg, err := handlers.BuildDatabaseTargetConfigForTest("", bad)
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Contains(
|
||||
t, errMsg, "Invalid archive rotation",
|
||||
"rotation %q should be refused", bad,
|
||||
)
|
||||
assert.Empty(t, cfg)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -120,7 +120,7 @@ func TestHandleSourceCreateSubmit_CreatesRequestedTargets(t *testing.T) {
|
||||
// retention, each with archive on. Nothing is created, and the form
|
||||
// comes back with the reason and every value entered: name,
|
||||
// description, retention, URL, the checked archive box and the pruning
|
||||
// choice.
|
||||
// and rotation choices.
|
||||
func TestHandleSourceCreateSubmit_RefusedFormKeepsEveryValue(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -152,6 +152,7 @@ func TestHandleSourceCreateSubmit_RefusedFormKeepsEveryValue(
|
||||
form.Set("http_url", tc.httpURL)
|
||||
form.Set("archive", "on")
|
||||
form.Set("archive_expiry", "2160h")
|
||||
form.Set("archive_rotation", "hourly")
|
||||
|
||||
w := submitCreateForm(env, form)
|
||||
require.Equal(t, http.StatusBadRequest, w.Code)
|
||||
@@ -166,6 +167,7 @@ func TestHandleSourceCreateSubmit_RefusedFormKeepsEveryValue(
|
||||
assert.Contains(t, page, `name="archive" value="on" checked`)
|
||||
assert.Contains(t, page, `x-data="collapsible" data-open`)
|
||||
assert.Contains(t, page, `<option value="2160h" selected>`)
|
||||
assert.Contains(t, page, `<option value="hourly" selected>`)
|
||||
|
||||
assertNothingCreated(t, env.db)
|
||||
})
|
||||
|
||||
@@ -300,9 +300,10 @@ type sourceFormInput struct {
|
||||
// destination.
|
||||
HTTPURL string
|
||||
// Archive asks for a database (archive) target, whose rows expire
|
||||
// after ArchiveExpiry.
|
||||
Archive bool
|
||||
ArchiveExpiry string
|
||||
// after ArchiveExpiry and whose files rotate by ArchiveRotation.
|
||||
Archive bool
|
||||
ArchiveExpiry string
|
||||
ArchiveRotation string
|
||||
}
|
||||
|
||||
// newSourceFormData builds the template data for the webhook creation
|
||||
@@ -319,6 +320,9 @@ func newSourceFormData(
|
||||
tmplKeyArchiveExpiryChoices: archiveExpiryOptions(
|
||||
in.ArchiveExpiry,
|
||||
),
|
||||
tmplKeyArchiveRotationChoices: archiveRotationOptions(
|
||||
in.ArchiveRotation,
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -345,12 +349,13 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
|
||||
}
|
||||
|
||||
in := sourceFormInput{
|
||||
Name: r.PostFormValue("name"),
|
||||
Description: r.PostFormValue("description"),
|
||||
RetentionDays: r.PostFormValue("retention_days"),
|
||||
HTTPURL: r.PostFormValue("http_url"),
|
||||
Archive: r.PostFormValue("archive") != "",
|
||||
ArchiveExpiry: r.PostFormValue("archive_expiry"),
|
||||
Name: r.PostFormValue("name"),
|
||||
Description: r.PostFormValue("description"),
|
||||
RetentionDays: r.PostFormValue("retention_days"),
|
||||
HTTPURL: r.PostFormValue("http_url"),
|
||||
Archive: r.PostFormValue("archive") != "",
|
||||
ArchiveExpiry: r.PostFormValue("archive_expiry"),
|
||||
ArchiveRotation: r.PostFormValue("archive_rotation"),
|
||||
}
|
||||
|
||||
refuse := func(errMsg string) {
|
||||
@@ -421,9 +426,10 @@ func (h *Handlers) newWebhookTargets(
|
||||
|
||||
if in.Archive {
|
||||
requested = append(requested, targetFormInput{
|
||||
Name: "Archive",
|
||||
Type: database.TargetTypeDatabase,
|
||||
Expiry: in.ArchiveExpiry,
|
||||
Name: "Archive",
|
||||
Type: database.TargetTypeDatabase,
|
||||
Expiry: in.ArchiveExpiry,
|
||||
Rotation: in.ArchiveRotation,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -630,9 +636,10 @@ func (h *Handlers) renderSourceDetail(
|
||||
"Stats": h.loadWebhookStats(webhook.ID, entrypoints, targets),
|
||||
tmplKeyTargetForm: targetForm,
|
||||
"TargetError": targetErr,
|
||||
// The add target form's select starts on its expiry
|
||||
// through Alpine, so no choice is selected here.
|
||||
tmplKeyArchiveExpiryChoices: archiveExpiryChoices(),
|
||||
// The add target form's selects start on its expiry and
|
||||
// rotation through Alpine, so no choice is selected here.
|
||||
tmplKeyArchiveExpiryChoices: archiveExpiryChoices(),
|
||||
tmplKeyArchiveRotationChoices: archiveRotationChoices(),
|
||||
}
|
||||
|
||||
status := http.StatusOK
|
||||
@@ -1800,6 +1807,8 @@ type targetFormInput struct {
|
||||
MaxRetries string
|
||||
// Expiry is a database (archive) target's row expiry.
|
||||
Expiry string
|
||||
// Rotation is a database (archive) target's rotation.
|
||||
Rotation string
|
||||
}
|
||||
|
||||
// targetFormInputFrom reads a target form from a request body. The
|
||||
@@ -1823,6 +1832,7 @@ func targetFormInputFrom(r *http.Request) targetFormInput {
|
||||
Timeout: r.PostFormValue("timeout"),
|
||||
MaxRetries: r.PostFormValue("max_retries"),
|
||||
Expiry: r.PostFormValue("expiry"),
|
||||
Rotation: r.PostFormValue("rotation"),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1843,7 +1853,7 @@ func (h *Handlers) buildTargetConfig(
|
||||
case database.TargetTypeSlack:
|
||||
return h.buildSlackTargetConfig(ctx, in.URL)
|
||||
case database.TargetTypeDatabase:
|
||||
return buildDatabaseTargetConfig(in.Expiry)
|
||||
return buildDatabaseTargetConfig(in.Expiry, in.Rotation)
|
||||
case database.TargetTypeLog:
|
||||
return "", "", nil
|
||||
default:
|
||||
@@ -1964,22 +1974,41 @@ func marshalTargetConfig(cfg any) (string, error) {
|
||||
}
|
||||
|
||||
// buildDatabaseTargetConfig builds config JSON for a database
|
||||
// (archive) target. The optional expiry is validated here, at
|
||||
// creation time, so an unparseable value is refused instead of
|
||||
// failing every subsequent delivery. An empty expiry yields an
|
||||
// empty config (the keep-forever default).
|
||||
func buildDatabaseTargetConfig(expiry string) (string, string, error) {
|
||||
// (archive) target. The optional expiry and rotation are validated
|
||||
// here, at creation time, so a bad value is refused instead of
|
||||
// failing every subsequent delivery. Each is stored only when set,
|
||||
// and with neither the config is empty (the keep-forever, one-file
|
||||
// default).
|
||||
func buildDatabaseTargetConfig(
|
||||
expiry, rotation string,
|
||||
) (string, string, error) {
|
||||
expiry = strings.TrimSpace(expiry)
|
||||
if expiry == "" {
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
err := delivery.ValidateArchiveExpiry(expiry)
|
||||
if err != nil {
|
||||
return "", fmt.Sprintf("Invalid archive expiry: %v", err), nil
|
||||
}
|
||||
|
||||
configJSON, err := marshalTargetConfig(map[string]any{"expiry": expiry})
|
||||
err = delivery.ValidateArchiveRotation(rotation)
|
||||
if err != nil {
|
||||
return "", fmt.Sprintf("Invalid archive rotation: %v", err), nil
|
||||
}
|
||||
|
||||
cfg := map[string]any{}
|
||||
|
||||
if expiry != "" {
|
||||
cfg["expiry"] = expiry
|
||||
}
|
||||
|
||||
if rotation != "" {
|
||||
cfg["rotation"] = rotation
|
||||
}
|
||||
|
||||
if len(cfg) == 0 {
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
configJSON, err := marshalTargetConfig(cfg)
|
||||
|
||||
return configJSON, "", err
|
||||
}
|
||||
|
||||
@@ -27,13 +27,11 @@ func (h *Handlers) HandleTargetDownload() http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
ctx := context.WithoutCancel(r.Context())
|
||||
|
||||
webhook, target, export, ok := h.openTargetArchive(ctx, w, r)
|
||||
webhook, target, export, ok := h.listTargetArchive(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
defer func() { _ = export.Close() }()
|
||||
|
||||
now := time.Now()
|
||||
|
||||
w.Header().Set("Content-Type", "application/gzip")
|
||||
@@ -81,16 +79,15 @@ func (d downloadWriter) Write(b []byte) (int, error) {
|
||||
return d.w.Write(b)
|
||||
}
|
||||
|
||||
// openTargetArchive opens the archive of the request's database target
|
||||
// for export, with its reads under ctx. It reports false once it has
|
||||
// written the response.
|
||||
// listTargetArchive lists the archive files of the request's database
|
||||
// target for export. It reports false once it has written the response.
|
||||
//
|
||||
// It holds renameMu, which every archive rename runs under, while it
|
||||
// reads the stored names and opens the file, so the file it opens is
|
||||
// the one those names give. It lets go before the export is streamed:
|
||||
// once the file is open, a rename does not affect the export.
|
||||
func (h *Handlers) openTargetArchive(
|
||||
ctx context.Context,
|
||||
// reads the stored names and lists the files, so the files it lists
|
||||
// are the ones those names give. It lets go before the export is
|
||||
// streamed, which takes renameMu again for each file only while it
|
||||
// finds the file under the names stored then and opens it.
|
||||
func (h *Handlers) listTargetArchive(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
) (database.Webhook, *database.Target, *delivery.ArchiveExport, bool) {
|
||||
@@ -108,14 +105,43 @@ func (h *Handlers) openTargetArchive(
|
||||
return database.Webhook{}, nil, nil, false
|
||||
}
|
||||
|
||||
export, err := delivery.OpenArchiveExport(
|
||||
ctx, delivery.ArchivePath(h.dbMgr, &webhook, target), h.log,
|
||||
export, err := delivery.NewArchiveExport(
|
||||
delivery.ArchivePath(h.dbMgr, &webhook, target),
|
||||
&h.renameMu,
|
||||
func() (string, error) {
|
||||
return h.storedArchivePath(webhook.ID, target.ID)
|
||||
},
|
||||
h.log,
|
||||
)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to open archive for export", err)
|
||||
h.serverError(w, r, "failed to list archive for export", err)
|
||||
|
||||
return database.Webhook{}, nil, nil, false
|
||||
}
|
||||
|
||||
return webhook, target, export, true
|
||||
}
|
||||
|
||||
// storedArchivePath returns the path delivery.ArchivePath gives a
|
||||
// database target under the names stored for it and its webhook now.
|
||||
// Its caller holds renameMu. A webhook or target deleted since is still
|
||||
// found, since deleting one leaves its archive files under their names.
|
||||
func (h *Handlers) storedArchivePath(
|
||||
webhookID, targetID string,
|
||||
) (string, error) {
|
||||
var webhook database.Webhook
|
||||
|
||||
err := h.db.DB().Unscoped().First(&webhook, "id = ?", webhookID).Error
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
var target database.Target
|
||||
|
||||
err = h.db.DB().Unscoped().First(&target, "id = ?", targetID).Error
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return delivery.ArchivePath(h.dbMgr, &webhook, &target), nil
|
||||
}
|
||||
|
||||
@@ -13,6 +13,8 @@ import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -85,7 +87,7 @@ func TestHandleTargetDownload(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestHandleTargetDownload_WaitsForRename proves a download reads the
|
||||
// target's names and opens its archive under the lock a rename holds:
|
||||
// target's names and lists its archive under the lock a rename holds:
|
||||
// started while an edit is renaming the archive, it waits, and is
|
||||
// named for the target's new name.
|
||||
func TestHandleTargetDownload_WaitsForRename(t *testing.T) {
|
||||
@@ -148,18 +150,17 @@ func (s *stalledWriter) Write(b []byte) (int, error) {
|
||||
return s.ResponseRecorder.Write(b)
|
||||
}
|
||||
|
||||
// TestHandleTargetDownload_StreamsWithoutTheLock proves a download
|
||||
// lets go of the rename lock once its archive is open: while the
|
||||
// download is stalled writing, an edit can still rename the target.
|
||||
func TestHandleTargetDownload_StreamsWithoutTheLock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
// startStalledDownload starts a download of the target and returns once
|
||||
// it is stalled at its first write, which comes before it opens any
|
||||
// archive file. Closing the writer's resume lets it go on; the returned
|
||||
// channel is closed when it has finished.
|
||||
func startStalledDownload(
|
||||
t *testing.T, env *sourceTestEnv, webhookID, targetID string,
|
||||
) (*stalledWriter, <-chan struct{}) {
|
||||
t.Helper()
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
|
||||
t.Context(), http.MethodGet, downloadPath(webhookID, targetID), nil,
|
||||
)
|
||||
for _, c := range env.cookies {
|
||||
req.AddCookie(c)
|
||||
@@ -179,6 +180,21 @@ func TestHandleTargetDownload_StreamsWithoutTheLock(t *testing.T) {
|
||||
|
||||
<-sw.writing
|
||||
|
||||
return sw, downloaded
|
||||
}
|
||||
|
||||
// TestHandleTargetDownload_StreamsWithoutTheLock proves a download
|
||||
// lets go of the rename lock once it has listed its archive: while the
|
||||
// download is stalled writing, an edit can still rename the target.
|
||||
func TestHandleTargetDownload_StreamsWithoutTheLock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
|
||||
sw, downloaded := startStalledDownload(t, env, wh.ID, archive.ID)
|
||||
|
||||
edited := make(chan *httptest.ResponseRecorder, 1)
|
||||
|
||||
go func() {
|
||||
@@ -197,6 +213,62 @@ func TestHandleTargetDownload_StreamsWithoutTheLock(t *testing.T) {
|
||||
assert.Equal(t, http.StatusOK, sw.Code)
|
||||
}
|
||||
|
||||
// TestHandleTargetDownload_FindsFilesAfterRename proves a download
|
||||
// finds each of the target's files again under the names stored when
|
||||
// it reaches the file: the target is renamed while the download is
|
||||
// stalled before it has opened any file, and the rows of both its files
|
||||
// are in the download.
|
||||
func TestHandleTargetDownload_FindsFilesAfterRename(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
oldPath := delivery.ArchivePath(env.dbMgr, &wh, archive)
|
||||
|
||||
month := func(path string) string {
|
||||
return strings.TrimSuffix(path, ".db") + "-2026-10.db"
|
||||
}
|
||||
|
||||
seedArchive(t, oldPath, 1, 16)
|
||||
seedArchive(t, month(oldPath), 1, 16)
|
||||
|
||||
sw, downloaded := startStalledDownload(t, env, wh.ID, archive.ID)
|
||||
|
||||
require.Equal(t,
|
||||
http.StatusSeeOther, renameTarget(env, wh.ID, archive.ID).Code,
|
||||
)
|
||||
|
||||
// The test's archives record a rename without moving any file, so
|
||||
// the files are moved here, as the delivery engine moves them.
|
||||
var renamed database.Target
|
||||
|
||||
require.NoError(t, env.db.DB().First(&renamed, "id = ?", archive.ID).Error)
|
||||
|
||||
newPath := delivery.ArchivePath(env.dbMgr, &wh, &renamed)
|
||||
|
||||
require.NoError(t, os.Rename(oldPath, newPath))
|
||||
require.NoError(t, os.Rename(month(oldPath), month(newPath)))
|
||||
|
||||
close(sw.resume)
|
||||
<-downloaded
|
||||
require.Equal(t, http.StatusOK, sw.Code)
|
||||
|
||||
zr, err := gzip.NewReader(sw.Body)
|
||||
require.NoError(t, err)
|
||||
|
||||
var (
|
||||
got map[string]json.RawMessage
|
||||
events []map[string]any
|
||||
)
|
||||
|
||||
require.NoError(t, json.NewDecoder(zr).Decode(&got))
|
||||
require.NoError(t, json.Unmarshal(got["archived_events"], &events))
|
||||
require.Len(t, events, 2)
|
||||
assert.NotContains(t, events[0], "period")
|
||||
assert.Equal(t, "2026-10", events[1]["period"])
|
||||
}
|
||||
|
||||
// seedArchive writes rows to the archive file at path, each with a
|
||||
// body of bodySize random bytes, which do not compress. Its table has
|
||||
// only the columns the test fills; an export writes the others empty.
|
||||
|
||||
@@ -83,6 +83,7 @@ func (h *Handlers) HandleTargetEdit() http.HandlerFunc {
|
||||
Timeout: cfg.Timeout,
|
||||
MaxRetries: strconv.Itoa(target.MaxRetries),
|
||||
Expiry: cfg.Expiry,
|
||||
Rotation: cfg.Rotation,
|
||||
}
|
||||
|
||||
h.renderTargetEdit(
|
||||
@@ -242,6 +243,9 @@ func (h *Handlers) renderTargetEdit(
|
||||
tmplKeyMaxTimeout: delivery.MaxTargetTimeoutSeconds,
|
||||
tmplKeyError: errMsg,
|
||||
tmplKeyArchiveExpiryChoices: archiveExpiryOptions(form.Expiry),
|
||||
tmplKeyArchiveRotationChoices: archiveRotationOptions(
|
||||
form.Rotation,
|
||||
),
|
||||
}
|
||||
|
||||
h.renderTemplateStatus(w, r, targetEditTemplate, data, status)
|
||||
|
||||
@@ -591,7 +591,8 @@ func TestHandleTargetEditSubmit_RefusedFormComesBack(t *testing.T) {
|
||||
"Invalid max retries",
|
||||
},
|
||||
{
|
||||
database.TargetTypeDatabase, "name=edited&expiry=7d",
|
||||
database.TargetTypeDatabase,
|
||||
"name=edited&expiry=7d&rotation=daily",
|
||||
"Invalid archive expiry",
|
||||
},
|
||||
{database.TargetTypeLog, "name=", "Name is required"},
|
||||
@@ -613,15 +614,15 @@ func TestHandleTargetEditSubmit_RefusedFormComesBack(t *testing.T) {
|
||||
page := w.Body.String()
|
||||
assert.Contains(t, page, `class="alert-error">`+tc.reason)
|
||||
|
||||
// headers is the form's one textarea and expiry its one
|
||||
// select; every other field is an input.
|
||||
// headers is the form's one textarea, and expiry and
|
||||
// rotation its selects; every other field is an input.
|
||||
for field := range form {
|
||||
shown := `name="` + field + `" value="` + form.Get(field) + `"`
|
||||
|
||||
switch field {
|
||||
case "headers":
|
||||
shown = ">" + form.Get(field) + "</textarea>"
|
||||
case "expiry":
|
||||
case "expiry", "rotation":
|
||||
shown = `<option value="` + form.Get(field) + `" selected>`
|
||||
}
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
@@ -21,8 +22,8 @@ type TargetRowView struct {
|
||||
// and is nil when the webhook's event database could not be read.
|
||||
Deliveries *TargetDeliveries
|
||||
|
||||
// Archive is a database target's archive file, and nil for a target
|
||||
// of any other type.
|
||||
// Archive is a database target's archive files, and nil for a
|
||||
// target of any other type.
|
||||
Archive *ArchiveFileView
|
||||
|
||||
// Paused is set while the target's circuit breaker is turning its
|
||||
@@ -117,17 +118,23 @@ type TargetDeliveries struct {
|
||||
}
|
||||
|
||||
// ArchiveFileView is what a database target's row shows about its
|
||||
// archive file.
|
||||
// archive files.
|
||||
type ArchiveFileView struct {
|
||||
// Name is the name of the file an event received now goes to.
|
||||
Name string
|
||||
|
||||
// Note stands in for the size and the last write when there are
|
||||
// none to show, and is empty when there are.
|
||||
// Note says that file does not exist yet, or that the files could
|
||||
// not be read, and is empty otherwise.
|
||||
Note string
|
||||
|
||||
// Size is the size on disk. Written is how long ago the file was
|
||||
// last written, and WrittenUTC the full time the page shows on
|
||||
// hover.
|
||||
// Files counts the target's archive files, and is 0 when there are
|
||||
// none, or they could not be read, and so no size or last write to
|
||||
// show.
|
||||
Files int
|
||||
|
||||
// Size is the size on disk of all the files together. Written is
|
||||
// how long ago the latest of them was last written, and WrittenUTC
|
||||
// the full time the page shows on hover.
|
||||
Size string
|
||||
Written string
|
||||
WrittenUTC string
|
||||
@@ -140,6 +147,7 @@ func (h *Handlers) targetRows(
|
||||
) []TargetRowView {
|
||||
views := delivery.NewTargetViews(targets)
|
||||
rows := make([]TargetRowView, len(views))
|
||||
now := time.Now()
|
||||
|
||||
deliveries, err := h.loadTargetDeliveries(webhook.ID)
|
||||
if err != nil {
|
||||
@@ -160,7 +168,7 @@ func (h *Handlers) targetRows(
|
||||
}
|
||||
|
||||
if targets[i].Type == database.TargetTypeDatabase {
|
||||
rows[i].Archive = h.archiveFileView(webhook, &targets[i])
|
||||
rows[i].Archive = h.archiveFileView(webhook, &targets[i], now)
|
||||
}
|
||||
|
||||
rows[i].Paused = h.pausedView(targets[i].ID)
|
||||
@@ -227,34 +235,57 @@ func readTargetDeliveries(
|
||||
return byTarget, nil
|
||||
}
|
||||
|
||||
// archiveFileView describes a database target's archive file from the
|
||||
// file's metadata alone; the archive is never opened. The file is found
|
||||
// by the name the archive writer uses, so it follows a rename of the
|
||||
// webhook or the target.
|
||||
// archiveFileView describes a database target's archive files from
|
||||
// their metadata alone; the archive is never opened. It names the file
|
||||
// an event received at now goes to. The files are found by the name
|
||||
// the archive writer uses, so they follow a rename of the webhook or
|
||||
// the target.
|
||||
func (h *Handlers) archiveFileView(
|
||||
webhook *database.Webhook, target *database.Target,
|
||||
webhook *database.Webhook, target *database.Target, now time.Time,
|
||||
) *ArchiveFileView {
|
||||
path := delivery.ArchivePath(h.dbMgr, webhook, target)
|
||||
view := &ArchiveFileView{Name: filepath.Base(path)}
|
||||
current, err := delivery.ArchivePathAt(h.dbMgr, webhook, target, now)
|
||||
if err != nil {
|
||||
return h.archiveUnreadable(&ArchiveFileView{}, target, err)
|
||||
}
|
||||
|
||||
file, err := delivery.StatArchive(path)
|
||||
view := &ArchiveFileView{Name: filepath.Base(current)}
|
||||
|
||||
_, statErr := os.Stat(current)
|
||||
if errors.Is(statErr, fs.ErrNotExist) {
|
||||
view.Note = "not created yet"
|
||||
}
|
||||
|
||||
files, err := delivery.StatArchive(
|
||||
delivery.ArchivePath(h.dbMgr, webhook, target),
|
||||
)
|
||||
|
||||
switch {
|
||||
case errors.Is(err, fs.ErrNotExist):
|
||||
view.Note = "not created yet"
|
||||
// No files: no size or last write to show.
|
||||
case err != nil:
|
||||
h.log.Error(
|
||||
"failed to read archive file metadata",
|
||||
"target_id", target.ID,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
view.Note = "could not be read"
|
||||
return h.archiveUnreadable(view, target, err)
|
||||
default:
|
||||
view.Size = humanize.Bytes(uint64(file.Size)) //nolint:gosec // never negative
|
||||
view.Written = humanize.Time(file.Written)
|
||||
view.WrittenUTC = file.Written.UTC().Format(time.DateTime) + " UTC"
|
||||
view.Files = files.Files
|
||||
view.Size = humanize.Bytes(uint64(files.Size)) //nolint:gosec // never negative
|
||||
view.Written = humanize.Time(files.Written)
|
||||
view.WrittenUTC = files.Written.UTC().Format(time.DateTime) + " UTC"
|
||||
}
|
||||
|
||||
return view
|
||||
}
|
||||
|
||||
// archiveUnreadable logs why a database target's archive files could
|
||||
// not be described, and returns view saying so.
|
||||
func (h *Handlers) archiveUnreadable(
|
||||
view *ArchiveFileView, target *database.Target, err error,
|
||||
) *ArchiveFileView {
|
||||
h.log.Error(
|
||||
"failed to read archive file metadata",
|
||||
"target_id", target.ID,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
view.Note = "could not be read"
|
||||
|
||||
return view
|
||||
}
|
||||
|
||||
@@ -107,7 +107,8 @@ func TestPausedTarget_ShownUntilBreakerCloses(t *testing.T) {
|
||||
assert.Contains(t, list, "t-http http Active Edit Deactivate Delete "+
|
||||
"Deliveries Paused: held while one delivery tests whether the "+
|
||||
"target has recovered")
|
||||
assert.NotContains(t, list, "UTC")
|
||||
// Not the whole list: the add target form above the rows says UTC.
|
||||
assert.NotContains(t, targetRow(list, "t-http", "t-log"), "UTC")
|
||||
|
||||
assertRetryingNotWaiting(t, h, sess, wh.ID, retrying, backedOff)
|
||||
|
||||
@@ -119,6 +120,15 @@ func TestPausedTarget_ShownUntilBreakerCloses(t *testing.T) {
|
||||
assertRetryingNotWaiting(t, h, sess, wh.ID, retrying, backedOff)
|
||||
}
|
||||
|
||||
// targetRow returns the row of the target named name in a targetList:
|
||||
// from its name to the name of the target listed after it, next.
|
||||
func targetRow(list, name, next string) string {
|
||||
_, row, _ := strings.Cut(list, name+" ")
|
||||
row, _, _ = strings.Cut(row, next+" ")
|
||||
|
||||
return row
|
||||
}
|
||||
|
||||
// assertRetryingNotWaiting checks that the event log and each event's
|
||||
// page show the http target's delivery of the event as retrying, and
|
||||
// none of them as waiting.
|
||||
|
||||
@@ -84,7 +84,7 @@ func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
|
||||
// this order. A new check is one more line here.
|
||||
checkAddEntrypoint(ctx, t, page)
|
||||
checkAddEachTargetType(ctx, t, page)
|
||||
checkArchiveExpiry(ctx, t, page)
|
||||
checkArchiveChoices(ctx, t, page)
|
||||
checkRefusedTarget(ctx, t, page)
|
||||
checkTargetDeliveries(ctx, t, page, target.Name,
|
||||
"0 in total, 0 in the last 24 hours",
|
||||
@@ -334,8 +334,8 @@ func checkAddEachTargetType(ctx context.Context, t *testing.T, url string) {
|
||||
map[string]string{"url": publicTargetURL},
|
||||
},
|
||||
{
|
||||
"database", "csrf_token name type expiry",
|
||||
map[string]string{"expiry": "720h"},
|
||||
"database", "csrf_token name type expiry rotation",
|
||||
map[string]string{"expiry": "720h", "rotation": "daily"},
|
||||
},
|
||||
{"log", "csrf_token name type", nil},
|
||||
}
|
||||
@@ -435,14 +435,18 @@ func chooseTargetType(ctx context.Context, t *testing.T, targetType string) {
|
||||
"%s: Add still shows while the form is open", targetType)
|
||||
}
|
||||
|
||||
// checkArchiveExpiry loads a webhook page and checks that the add
|
||||
// target form's archive expiry starts on never, that the database
|
||||
// target checkAddTarget added with 720h is listed as 30 days, and that
|
||||
// its edit form starts on 720h.
|
||||
func checkArchiveExpiry(ctx context.Context, t *testing.T, url string) {
|
||||
// checkArchiveChoices loads a webhook page and checks that the add
|
||||
// target form's archive expiry starts on never and its archive
|
||||
// rotation on none, that the database target checkAddTarget added with
|
||||
// 720h and daily is listed as 30 days and daily, and that its edit
|
||||
// form starts on 720h and daily.
|
||||
func checkArchiveChoices(ctx context.Context, t *testing.T, url string) {
|
||||
t.Helper()
|
||||
|
||||
const expiry = `form[action$="/targets"] select[name="expiry"]`
|
||||
const (
|
||||
expiry = `form[action$="/targets"] select[name="expiry"]`
|
||||
rotation = `form[action$="/targets"] select[name="rotation"]`
|
||||
)
|
||||
|
||||
row := `//span[text()="added-database"]/ancestor::div[@class="p-4"][1]`
|
||||
|
||||
@@ -450,26 +454,36 @@ func checkArchiveExpiry(ctx context.Context, t *testing.T, url string) {
|
||||
|
||||
chooseTargetType(ctx, t, "database")
|
||||
|
||||
var start, edited string
|
||||
var startExpiry, startRotation, editedExpiry, editedRotation string
|
||||
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx, chromedp.Value(expiry, &start, chromedp.ByQuery),
|
||||
ctx,
|
||||
chromedp.Value(expiry, &startExpiry, chromedp.ByQuery),
|
||||
chromedp.Value(rotation, &startRotation, chromedp.ByQuery),
|
||||
))
|
||||
assert.Equal(t, "never", start,
|
||||
assert.Equal(t, "never", startExpiry,
|
||||
"the add target form's archive expiry does not start on never")
|
||||
assert.Equal(t, "none", startRotation,
|
||||
"the add target form's archive rotation does not start on none")
|
||||
|
||||
assert.True(t, shown(ctx, row+`//span[text()="Archive Expiry:"]`+
|
||||
`/following-sibling::span[text()="30 days"]`),
|
||||
"a database target added with 720h is not listed as 30 days")
|
||||
assert.True(t, shown(ctx, row+`//span[text()="Archive Rotation:"]`+
|
||||
`/following-sibling::span[text()="daily"]`),
|
||||
"a database target added with daily is not listed as daily")
|
||||
|
||||
click(ctx, t, row+`//a[text()="Edit"]`)
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx,
|
||||
chromedp.WaitReady("#expiry", chromedp.ByQuery),
|
||||
chromedp.Value("#expiry", &edited, chromedp.ByQuery),
|
||||
chromedp.Value("#expiry", &editedExpiry, chromedp.ByQuery),
|
||||
chromedp.Value("#rotation", &editedRotation, chromedp.ByQuery),
|
||||
))
|
||||
assert.Equal(t, "720h", edited,
|
||||
assert.Equal(t, "720h", editedExpiry,
|
||||
"the edit form does not start on the stored archive expiry")
|
||||
assert.Equal(t, "daily", editedRotation,
|
||||
"the edit form does not start on the stored archive rotation")
|
||||
}
|
||||
|
||||
// checkRefusedTarget submits an http target the server refuses, a
|
||||
@@ -1145,7 +1159,9 @@ func checkNewWebhookTargets(
|
||||
pruningChoice, expiry, chromedp.BySearch,
|
||||
)))
|
||||
|
||||
want[database.TargetTypeDatabase] = `{"expiry":"` + expiry + `"}`
|
||||
// The rotation choice is left on none.
|
||||
want[database.TargetTypeDatabase] = `{"expiry":"` + expiry +
|
||||
`","rotation":"none"}`
|
||||
}
|
||||
|
||||
click(ctx, t, createButton)
|
||||
@@ -1186,8 +1202,8 @@ func targetConfigs(
|
||||
// checkRefusedNewWebhook submits the new webhook page with archive
|
||||
// checked and an HTTP target URL the server refuses, a loopback
|
||||
// destination, and checks that the page comes back with the reason and
|
||||
// every value entered, archive still checked and its pruning choice
|
||||
// showing.
|
||||
// every value entered, archive still checked and its pruning and
|
||||
// rotation choices showing.
|
||||
func checkRefusedNewWebhook(ctx context.Context, t *testing.T, url string) {
|
||||
t.Helper()
|
||||
|
||||
@@ -1204,17 +1220,19 @@ func checkRefusedNewWebhook(ctx context.Context, t *testing.T, url string) {
|
||||
click(ctx, t, archiveBox)
|
||||
require.True(t, shown(ctx, pruningChoice),
|
||||
"checking archive does not show the pruning choice")
|
||||
require.NoError(t, chromedp.Run(ctx, chromedp.SetValue(
|
||||
pruningChoice, "2160h", chromedp.BySearch,
|
||||
)))
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx,
|
||||
chromedp.SetValue(pruningChoice, "2160h", chromedp.BySearch),
|
||||
chromedp.SetValue("#archive_rotation", "monthly", chromedp.ByQuery),
|
||||
))
|
||||
click(ctx, t, createButton)
|
||||
|
||||
assert.True(t, shown(ctx, `//div[@class="alert-error"]`),
|
||||
"a refused webhook does not show the reason")
|
||||
|
||||
var (
|
||||
name, description, retention, typed, expiry string
|
||||
checked bool
|
||||
name, description, retention, typed, expiry, rotation string
|
||||
checked bool
|
||||
)
|
||||
|
||||
require.NoError(t, chromedp.Run(
|
||||
@@ -1224,6 +1242,7 @@ func checkRefusedNewWebhook(ctx context.Context, t *testing.T, url string) {
|
||||
chromedp.Value("#retention_days", &retention, chromedp.ByQuery),
|
||||
chromedp.Value("#http_url", &typed, chromedp.ByQuery),
|
||||
chromedp.Value("#archive_expiry", &expiry, chromedp.ByQuery),
|
||||
chromedp.Value("#archive_rotation", &rotation, chromedp.ByQuery),
|
||||
chromedp.Evaluate(archiveIsOn, &checked),
|
||||
))
|
||||
|
||||
@@ -1234,6 +1253,7 @@ func checkRefusedNewWebhook(ctx context.Context, t *testing.T, url string) {
|
||||
assert.True(t, checked, "archive is no longer checked")
|
||||
assert.True(t, shown(ctx, pruningChoice), "the pruning choice is hidden")
|
||||
assert.Equal(t, "2160h", expiry, "the pruning chosen is lost")
|
||||
assert.Equal(t, "monthly", rotation, "the rotation chosen is lost")
|
||||
}
|
||||
|
||||
// checkMobileMenu loads a page in a phone-sized window and checks that
|
||||
|
||||
File diff suppressed because one or more lines are too long
+5
-2
@@ -72,8 +72,8 @@ document.addEventListener("alpine:init", function () {
|
||||
// Something a click shows and hides: the mobile menu, an add form,
|
||||
// an entrypoint's edit form, an event in the event log or in the
|
||||
// recent events, a delivery's attempts, the new webhook page's
|
||||
// archive pruning choice. It starts hidden, or shown when its
|
||||
// element has the data-open attribute.
|
||||
// archive pruning and rotation choices. It starts hidden, or shown
|
||||
// when its element has the data-open attribute.
|
||||
window.Alpine.data("collapsible", function () {
|
||||
return {
|
||||
open: false,
|
||||
@@ -140,6 +140,7 @@ document.addEventListener("alpine:init", function () {
|
||||
timeout: "",
|
||||
maxRetries: "",
|
||||
expiry: "",
|
||||
rotation: "",
|
||||
init() {
|
||||
const refused = this.$root.dataset;
|
||||
|
||||
@@ -151,6 +152,7 @@ document.addEventListener("alpine:init", function () {
|
||||
this.timeout = refused.timeout;
|
||||
this.maxRetries = refused.maxRetries;
|
||||
this.expiry = refused.expiry;
|
||||
this.rotation = refused.rotation;
|
||||
},
|
||||
add() {
|
||||
this.choosing = true;
|
||||
@@ -169,6 +171,7 @@ document.addEventListener("alpine:init", function () {
|
||||
this.timeout = "";
|
||||
this.maxRetries = "";
|
||||
this.expiry = "";
|
||||
this.rotation = "";
|
||||
this.$refs.form.reset();
|
||||
},
|
||||
get filling() {
|
||||
|
||||
@@ -129,7 +129,8 @@
|
||||
data-headers="{{.TargetForm.Headers}}"
|
||||
data-timeout="{{.TargetForm.Timeout}}"
|
||||
data-max-retries="{{.TargetForm.MaxRetries}}"
|
||||
data-expiry="{{.TargetForm.Expiry}}">
|
||||
data-expiry="{{.TargetForm.Expiry}}"
|
||||
data-rotation="{{.TargetForm.Rotation}}">
|
||||
<div class="p-4 border-b border-gray-200 flex justify-between items-center">
|
||||
<h2 class="text-lg font-medium text-gray-900">Targets</h2>
|
||||
<button type="button" @click="add" x-show="closed" class="btn-small">
|
||||
@@ -201,17 +202,30 @@
|
||||
</div>
|
||||
</template>
|
||||
<template x-if="isDatabase">
|
||||
<div>
|
||||
<div class="space-y-3">
|
||||
<input type="hidden" name="type" value="database">
|
||||
<div class="flex gap-2 items-center">
|
||||
<label class="text-sm text-gray-700">Archive expiry:</label>
|
||||
<select name="expiry" :value="expiry" class="input text-sm w-24">
|
||||
{{range .ArchiveExpiryChoices}}
|
||||
<option value="{{.Value}}">{{.Label}}</option>
|
||||
{{end}}
|
||||
</select>
|
||||
<div>
|
||||
<div class="flex gap-2 items-center">
|
||||
<label class="text-sm text-gray-700">Archive expiry:</label>
|
||||
<select name="expiry" :value="expiry" class="input text-sm w-24">
|
||||
{{range .ArchiveExpiryChoices}}
|
||||
<option value="{{.Value}}">{{.Label}}</option>
|
||||
{{end}}
|
||||
</select>
|
||||
</div>
|
||||
<p class="text-xs text-gray-500 mt-1">Archived events older than this are deleted from the archive; never keeps them all.</p>
|
||||
</div>
|
||||
<div>
|
||||
<div class="flex gap-2 items-center">
|
||||
<label class="text-sm text-gray-700">Archive rotation:</label>
|
||||
<select name="rotation" :value="rotation" class="input text-sm w-28">
|
||||
{{range .ArchiveRotationChoices}}
|
||||
<option value="{{.Value}}">{{.Label}}</option>
|
||||
{{end}}
|
||||
</select>
|
||||
</div>
|
||||
<p class="text-xs text-gray-500 mt-1">Starts a new archive file each month, day or hour (UTC), named for it; none keeps one file.</p>
|
||||
</div>
|
||||
<p class="text-xs text-gray-500 mt-1">Archived events older than this are deleted from the archive; never keeps them all.</p>
|
||||
</div>
|
||||
</template>
|
||||
<template x-if="isLog">
|
||||
@@ -273,10 +287,10 @@
|
||||
<span class="break-all">{{.Name}}</span>
|
||||
{{with .Note}}<span>({{.}})</span>{{end}}
|
||||
</div>
|
||||
{{if .Size}}
|
||||
{{if .Files}}
|
||||
<div class="text-xs text-gray-500 mt-1">
|
||||
<span class="font-medium text-gray-700">Archive Size:</span>
|
||||
<span>{{.Size}}</span>
|
||||
<span>{{.Size}}{{if gt .Files 1}} in {{.Files}} files{{end}}</span>
|
||||
</div>
|
||||
<div class="text-xs text-gray-500 mt-1">
|
||||
<span class="font-medium text-gray-700">Last Written:</span>
|
||||
|
||||
@@ -38,9 +38,9 @@
|
||||
<p class="text-xs text-gray-500 mt-1">Optional. When filled in, the webhook is created with an HTTP target that delivers each event to this URL.</p>
|
||||
</div>
|
||||
|
||||
<!-- The checkbox shows the pruning choice while checked. With
|
||||
autocomplete="off", going back to the page does not
|
||||
check the box again with the choice hidden. -->
|
||||
<!-- The checkbox shows the pruning and rotation choices while
|
||||
checked. With autocomplete="off", going back to the page
|
||||
does not check the box again with the choices hidden. -->
|
||||
<div class="form-group" x-data="collapsible"{{if .Form.Archive}} data-open{{end}}>
|
||||
<label class="flex items-center gap-2 text-sm font-medium text-gray-700">
|
||||
<input type="checkbox" name="archive" value="on"{{if .Form.Archive}} checked{{end}} autocomplete="off" @change="toggle" class="h-4 w-4">
|
||||
@@ -55,6 +55,15 @@
|
||||
{{end}}
|
||||
</select>
|
||||
<p class="text-xs text-gray-500 mt-1">Archived events older than this are deleted from the archive; never keeps them all.</p>
|
||||
<div class="mt-3">
|
||||
<label for="archive_rotation" class="label">Archive rotation</label>
|
||||
<select id="archive_rotation" name="archive_rotation" class="input">
|
||||
{{range .ArchiveRotationChoices}}
|
||||
<option value="{{.Value}}"{{if .Selected}} selected{{end}}>{{.Label}}</option>
|
||||
{{end}}
|
||||
</select>
|
||||
<p class="text-xs text-gray-500 mt-1">Starts a new archive file each month, day or hour (UTC), named for it; none keeps one file.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -67,6 +67,16 @@
|
||||
</select>
|
||||
<p class="text-xs text-gray-500 mt-1">Archived events older than this are deleted from the archive; never keeps them all.</p>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="rotation" class="label">Archive Rotation</label>
|
||||
<select id="rotation" name="rotation" class="input">
|
||||
{{range .ArchiveRotationChoices}}
|
||||
<option value="{{.Value}}"{{if .Selected}} selected{{end}}>{{.Label}}</option>
|
||||
{{end}}
|
||||
</select>
|
||||
<p class="text-xs text-gray-500 mt-1">Starts a new archive file each month, day or hour (UTC), named for it; none keeps one file. A change applies from the next event; existing files keep their names.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
{{if or (eq .Target.Type "http") (eq .Target.Type "slack")}}
|
||||
|
||||
Reference in New Issue
Block a user