Rotate a database target's archive monthly, daily or hourly (closes #379)
check / check (push) Successful in 3m31s

A database target's rotation (none, monthly, daily or hourly) puts the
UTC period of each event's receive time in its archive file name, so
each file holds exactly its period's events. It is on the new webhook
page, the add target form and the target edit form, and shown in the
target list.

Renames move every one of a target's files and move them back if one
fails. The sweep prunes every file, one at a time under the target's
lock, and deletes a rotated file it leaves empty. Download lists the
files, then opens one at a time, oldest first, finding each again under
the target's current names, and gives each row its period. The target
list names the current file and totals the size of all of them.

Model: opus-5-5
This commit is contained in:
2026-10-03 00:47:31 +00:00
parent bcdd4791ec
commit e55134532c
36 changed files with 2120 additions and 406 deletions
+26 -11
View File
@@ -20,7 +20,8 @@ const archiveNameMaxLen = 40
// from the per-webhook event database. The event is already
// persisted in the per-webhook event DB by the time delivery runs;
// the database target additionally writes a durable long-term copy
// into the file ArchiveFileName names and then records a single
// into the file ArchiveFileName names, with a period added when the
// target rotates (see archivePeriodPath), and then records a single
// attempt whose outcome reflects whether the archive write
// succeeded. See archiveWriter for the close/reopen, auto-recreate,
// and expiry semantics.
@@ -146,8 +147,9 @@ func (t *databaseTarget) Deliver(
}
// archive writes the full event as a row into the target's
// archive database, honouring the optional per-target expiry
// parsed from the target config JSON.
// archive database, honouring the optional per-target expiry and
// rotation parsed from the target config JSON. With rotation, the
// event goes to the file for the period of its receive time.
func (t *databaseTarget) archive(d *database.Delivery) error {
webhookID := d.Event.WebhookID
if webhookID == "" {
@@ -159,6 +161,19 @@ func (t *databaseTarget) archive(d *database.Delivery) error {
return err
}
rotation, err := parseArchiveRotation(d.Target.Config)
if err != nil {
return err
}
// An event whose stored row was gone before its delivery ran has
// no receive time (see Engine.hydrateEvent), and goes to the
// file for now.
receivedAt := d.Event.CreatedAt
if receivedAt.IsZero() {
receivedAt = time.Now()
}
w, err := t.writerFor(d.TargetID)
if err != nil {
return err
@@ -174,7 +189,7 @@ func (t *databaseTarget) archive(d *database.Delivery) error {
ContentType: d.Event.ContentType,
}
return w.write(row, expiry)
return w.write(row, expiry, archivePeriod(rotation, receivedAt))
}
// writerFor returns the archive writer for a database target,
@@ -275,10 +290,10 @@ func (t *databaseTarget) releaseSweepWriter(
delete(t.writers, targetID)
}
// newWriter builds the writer for a database target's archive. The
// file is the one ArchivePath gives for the webhook and the target as
// newWriter builds the writer for a database target's archive. Its
// path is the one ArchivePath gives for the webhook and the target as
// the main database names them now; from then on only rename changes
// the name the writer uses. It does not touch the archive file.
// the name the writer uses. It does not touch the archive files.
func (t *databaseTarget) newWriter(
targetID string,
) (*archiveWriter, error) {
@@ -306,10 +321,10 @@ func (t *databaseTarget) newWriter(
return w, nil
}
// rename moves a database target's archive file to the name for
// webhookName and targetName. It goes through the target's writer,
// so the move holds the lock that writes and the idle sweep take,
// and later writes use the new name.
// rename moves every one of a database target's archive files to the
// name for webhookName and targetName. It goes through the target's
// writer, so the move holds the lock that writes and the idle sweep
// take, and later writes use the new name.
//
// The writer is created if there is none, and it stays cached. The
// handlers rename before they save the new name, so until the save