From 7489d2159c0883b62a13e4322a9bdcd5e2c12f17 Mon Sep 17 00:00:00 2001 From: Caret Date: Mon, 6 Apr 2026 13:16:50 +0000 Subject: [PATCH] phase1: FEATURE-PARITY-TESTS.md (74 tests) + DEPENDENCIES.md (28 items) --- DEPENDENCIES.md | 222 ++++++++++++++++++++++++++++++++++++++++ FEATURE-PARITY-TESTS.md | 126 +++++++++++++++++++++++ 2 files changed, 348 insertions(+) create mode 100644 DEPENDENCIES.md create mode 100644 FEATURE-PARITY-TESTS.md diff --git a/DEPENDENCIES.md b/DEPENDENCIES.md new file mode 100644 index 0000000..372f915 --- /dev/null +++ b/DEPENDENCIES.md @@ -0,0 +1,222 @@ +# Openclaw dependency matrix + +Every openclaw-provided capability that the current gitea-webhooks pipeline uses, and what Caret's replacement does with it. Source citations: R01 = RESEARCH-01, R02 = RESEARCH-02, R03 = RESEARCH-03, PLAN = PLAN.md. + +Categories: +- **REMOVE** — Caret's replacement doesn't need this at all +- **REPLACE** — Caret builds a standalone equivalent +- **KEEP** — Caret continues depending on openclaw for this (and why that's safe) +- **BLOCKING** — Caret cannot move forward without resolving this first (Rooh action required) + +## By capability + +### 1. HTTP webhook ingress (port 18789, `/hooks/agent`) +**Category:** REPLACE +**What Caret does:** Stand up a bun HTTP listener at a Caret-owned port (e.g. 18790 or unix socket), with its own path. Mirrors openclaw's `/hooks/agent` shape so existing scripts can be reused. [R01 §Ingress path, R02 §HTTP endpoints] +**Risk if kept:** Hard-couples Caret's lifecycle to the openclaw gateway container. Defeats the entire migration. +**Effort:** 1 day. Stateless request/response, well-understood. [R02 difficulty matrix: Easy] + +### 2. HMAC verification (currently NOT done by openclaw) +**Category:** REPLACE (net-new — openclaw never had it) +**What Caret does:** Real HMAC-SHA256 of the raw body using `X-Gitea-Signature` header, timing-safe compare, raw body preserved before JSON parse. [R01 §HMAC recipe, R03 confirms all 4 known webhooks have `Secret: NOT SET`] +**Risk if kept:** No HMAC at all today; protection is bearer + nginx ACL only. Caret should do better. +**Effort:** 0.5 day. ~30 lines. +**Depends on BLOCKING #1 (webhook secret provisioning).** + +### 3. Bearer token authentication (`OPENCLAW_HOOKS_TOKEN`) +**Category:** REPLACE +**What Caret does:** Own bearer token (`CARET_HOOKS_TOKEN`) stored in Caret's credentials dir, validated on each request as a second factor alongside HMAC. [R01 §Ingress path layer 3] +**Risk if kept:** Sharing the openclaw token cross-couples secret rotation. +**Effort:** Trivial. + +### 4. nginx TLS termination and path rewriting +**Category:** KEEP (with new location block) +**What Caret does:** Adds a new nginx `location /hooks/caret` block that proxies to the Caret listener, same TLS cert. Reuses existing Let's Encrypt setup. [R01 §Ingress path] +**Risk if kept:** None — nginx is host-level infra, not openclaw-specific. +**Effort:** 1 hour, but requires host root access. +**See BLOCKING #2.** + +### 5. Event dedup cache (`X-Gitea-Delivery` 24h) +**Category:** REPLACE +**What Caret does:** Own JSON-on-disk dedup cache at `/host/root/.caret/state/dedup-cache.json`, identical 24h TTL semantics. [R01 §Ingress path layer 4, R01 gotcha 4] +**Risk if kept:** Sharing openclaw's cache file is fragile and couples crash recovery. +**Effort:** 0.5 day. + +### 6. Rate limiting (5 concurrent per agent) +**Category:** REPLACE +**What Caret does:** Per-route concurrency limiter in the listener. Caret's routing is simpler (one "agent") so this is mostly a global semaphore. [R01 gotcha 7] +**Risk if kept:** N/A. +**Effort:** Trivial. + +### 7. Session lock manager (`hooks/locks/`) +**Category:** REPLACE +**What Caret does:** Own lock dir at `/host/root/.caret/state/locks/`, same 2h TTL, same file naming `{owner}-{repo}-{issue}`, same IS_DONE 5min grace. [R01 §Session lock] +**Risk if kept:** Two writers to one lock dir = race. +**Effort:** 0.5 day. + +### 8. Queue daemon (`queue-inbox/` + `queue-daemon.js`) +**Category:** REPLACE +**What Caret does:** In-process queue inside the listener (or a sidecar bun script reading from `/host/root/.caret/state/queue/`). Re-verifies spawn signatures off the hot path. [R01 gotcha 12, R03 §queue-daemon.js] +**Risk if kept:** Couples Caret's spawn pipeline to openclaw container restart. +**Effort:** 1 day. + +### 9. `sessions_spawn` agent orchestration primitive +**Category:** REPLACE (with Claude Code Channels plugin) +**What Caret does:** Build a Channels plugin at `/host/root/.caret/channels/gitea-judgment/` that receives an HTTP POST and starts a Claude Code session with the payload as initial prompt. [R02 §Caret's conclusion, PLAN §Phase 3 J3.1] +**Risk if kept:** Openclaw's `sessions_spawn` is the deepest coupling — see R02 difficulty matrix "Hard" rating. Keeping it means Caret can never fully cut over. +**Effort:** 2-3 days for the minimum viable plugin. Full parity (wakeMode, thinking, model selection) is more like 1 week. + +### 10. Tool allowlist enforcement per agent +**Category:** KEEP (judgment path runs inside Claude Code, which has its own tool config) +**What Caret does:** Caret's deterministic path runs scripts directly with no tools concept. The judgment path is a Claude Code session whose tool allowlist is configured in Claude Code's settings.json (not openclaw's). [R02 §Tool policy resolution] +**Risk if kept:** None — Caret never touches openclaw's tool policy resolver. +**Effort:** 0 (already separated). + +### 11. Plugin SDK (`src/plugin-sdk/`) +**Category:** REMOVE +**What Caret does:** Doesn't load openclaw plugins. Caret's listener is a plain bun script with no plugin loader. Channels plugin uses Claude Code's plugin mechanism, not openclaw's. [R02 §Don't reinvent #2] +**Risk if kept:** N/A. +**Effort:** 0. + +### 12. Delivery system (Telegram, Mattermost, Discord) +**Category:** REPLACE (use tg-stream + Mattermost direct calls) +**What Caret does:** Calls tg-stream HTTP API for Telegram alerts and Mattermost webhook URL directly for incident posts. No multi-channel abstraction layer. [R02 §Don't reinvent #4 warns it's tightly coupled — we don't replicate, we use simpler primitives] +**Risk if kept:** Hard coupling to openclaw outbound system. +**Effort:** 0.5 day (just curl calls). + +### 13. Session storage (JSONL transcripts at `~/.openclaw/sessions/`) +**Category:** KEEP (judgment path only — Claude Code's own session store) +**What Caret does:** Deterministic path has no sessions. Judgment path uses Claude Code's native session JSONL under Claude Code's config dir, NOT openclaw's. [R02 §Don't reinvent #1] +**Risk if kept:** None — Caret never reads/writes openclaw's session files. +**Effort:** 0. + +### 14. Heartbeat scheduler (`heartbeat-runner.ts`) +**Category:** REPLACE (use Claude Code `schedule` skill / cron) +**What Caret does:** Use systemd timer in the Caret container OR Claude Code's schedule mechanism for the 6h policy sweep and 15min webhook audit. No 28-item checklist — Caret's heartbeat is much smaller. [R02 §Heartbeat loop, R03 §Heartbeat checklists] +**Risk if kept:** Couples to openclaw gateway uptime, which R03 shows is currently degraded. +**Effort:** 0.5 day for the timer setup. + +### 15. Cron service (`CronService` in gateway) +**Category:** REPLACE (same as #14 — single mechanism) +**What Caret does:** Same as heartbeat — systemd timers or Claude Code schedule. [R02 §Cron service, R03 §Active cron jobs shows 8 of 12 currently failing] +**Risk if kept:** R03 shows openclaw cron is currently degraded; depending on it inherits the breakage. +**Effort:** Combined with #14. + +### 16. Gitea API integration (token + curl wrappers) +**Category:** KEEP +**What Caret does:** Continue using the same `GITEA_TOKEN` and stateless curl calls. Wrapper scripts ported into `/host/root/.caret/tools/`. [R01 §Tools fan-out, PLAN B2.3] +**Risk if kept:** None — Gitea API is a third-party service, openclaw is not in the path. +**Effort:** Path-strip and copy, 1 hour. +**Depends on BLOCKING #3 (admin scope).** + +### 17. Workspace directory (`/root/.openclaw/workspace/`) +**Category:** KEEP (155 projects stay owned by openclaw/Xen) +**What Caret does:** Caret's migration scope is the Gitea-facing slice ONLY. The 155-project workspace, the PROJ-XXX-* hierarchy, and Xen's project ownership stay as-is. Caret has its own `/host/root/.caret/workspace/` for migration artifacts only. [R03 §What the migration actually has to replace, PLAN §Goal] +**Risk if kept:** None for the in-scope migration. +**Effort:** 0. + +### 18. Project registry (`projects/registry.json`) +**Category:** KEEP +**What Caret does:** Read-only access if needed (e.g., to look up which manager owns a project for a given issue). No writes. [R03 §Shared state] +**Risk if kept:** Stale reads possible but acceptable. +**Effort:** 0. + +### 19. Memory system (`memory/`) +**Category:** REMOVE (from migration scope) +**What Caret does:** Caret's own memory is in `/root/.claude/projects/-app/memory/`. Doesn't touch openclaw's memory dir. [R03 §Shared state] +**Risk if kept:** N/A. +**Effort:** 0. + +### 20. Credentials store (`credentials/`) +**Category:** KEEP (read-only for shared tokens like GITEA_TOKEN) +**What Caret does:** Read GITEA_TOKEN from openclaw credentials OR copy to Caret credentials store. Caret-owned secrets (HMAC webhook secret, CARET_HOOKS_TOKEN) live in `/host/root/.caret/credentials/`. [R03 §Shared state credentials] +**Risk if kept:** Cross-coupling on token rotation. Acceptable short-term. +**Effort:** Trivial. +**See BLOCKING #4 (secret rotation story).** + +### 21. `post-repo-audit.sh` +**Category:** REPLACE (port the script verbatim) +**What Caret does:** Copy to `/host/root/.caret/tools/post-repo-audit.sh`, strip openclaw path prefixes, ship as-is. Pure script, zero tokens. [R01 §Tools fan-out, PLAN B2.3] +**Risk if kept:** Couples to openclaw tools dir lifecycle. +**Effort:** 1 hour. + +### 22. `audit-repo-policies.sh` +**Category:** REPLACE (port verbatim) +**What Caret does:** Copy to Caret tools dir, point at sol/repo-policies template repo, run with `--fix` from Caret's heartbeat. [R01 §Tools fan-out] +**Risk if kept:** Same as #21. +**Effort:** 1 hour. + +### 23. `spawn-manager.sh` +**Category:** REPLACE (port verbatim, but only after #9 is solved) +**What Caret does:** Copy script. Generates Manager spawn JSON from issue body, creates project workspace. Caret's listener calls it via execSync with same 30s timeout pattern. [R01 §Tools fan-out, R01 gotcha 9] +**Risk if kept:** Tightly coupled to openclaw workspace path conventions — depends on whether Caret keeps openclaw's PROJ-XXX scheme (R01 advises yes). +**Effort:** 0.5 day. + +### 24. `create-implement-issue.sh` +**Category:** REPLACE (port verbatim) +**What Caret does:** Copy script. Creates signed `[IMPLEMENT]` issue with HMAC spawn signature using `/host/root/.caret/credentials/spawn-secret`. [R01 §Tools fan-out] +**Risk if kept:** Spawn secret coupling. +**Effort:** 1 hour. +**Depends on BLOCKING #5 (spawn secret ownership).** + +### 25. `secret-scan.sh` +**Category:** REPLACE (port verbatim) +**What Caret does:** Copy script, used by both CI (`make check`) inside repos and Caret's push handler. [R01 §Tools fan-out] +**Risk if kept:** N/A. +**Effort:** 1 hour. + +### 26. `check-implement-orphans.sh` +**Category:** REPLACE (port verbatim) +**What Caret does:** Copy script, run from Caret's 15min heartbeat to detect stale pending spawns and orphaned managers. [R01 §Tools fan-out] +**Risk if kept:** N/A. +**Effort:** 1 hour. + +### 27. `auditLog → logs/audit.jsonl` +**Category:** REPLACE +**What Caret does:** Caret's listener writes to `/host/root/.caret/log/audit.jsonl`, same JSONL line schema. Rotation by line count. [R01 §Logging, PLAN B2.5] +**Risk if kept:** Two writers to one file = corruption. +**Effort:** 0.5 day. + +### 28. Incident flagging (`logs/incidents.jsonl`) +**Category:** REPLACE +**What Caret does:** Caret writes to `/host/root/.caret/log/incidents.jsonl`. Critical incidents also fan out to Telegram via tg-stream. [R01 §Logging, T2.23] +**Risk if kept:** Same as #27. +**Effort:** 0.5 day (combined with #27). + +## Summary by category + +| Category | Count | +|---|---| +| REMOVE | 2 (#11, #19) | +| REPLACE | 19 (#1, #2, #3, #5, #6, #7, #8, #9, #12, #14, #15, #21, #22, #23, #24, #25, #26, #27, #28) | +| KEEP | 6 (#4, #10, #13, #16, #17, #18, #20) — note #20 partially | +| BLOCKING | 5 (see below) | + +## BLOCKING items — Rooh action required + +These five items must be resolved by Rooh before Caret can write the production code. They are not engineering decisions; they are policy / access decisions only Rooh can make. + +### BLOCKING #1 — Webhook secret provisioning and storage location +**Question:** Where is the new HMAC webhook secret stored? `/host/root/.caret/credentials/webhook-secret`? A vault entry? Reused from an existing openclaw secret? +**Why it blocks:** Cannot ship HMAC verification (T1.06–T1.11) without the secret being authoritative somewhere. Cannot register webhooks on sol/* repos without knowing what secret to set on the Gitea side. +**Reference:** R01 §HMAC recipe, R03 confirms all current webhooks have `Secret: NOT SET`. + +### BLOCKING #2 — nginx config write access for `/hooks/caret` location +**Question:** Does Caret have permission to edit `/host/etc/nginx/...` and reload nginx, or does Rooh do that one-time setup? +**Why it blocks:** Without an nginx route, Gitea cannot reach the Caret listener via HTTPS. Direct port exposure is the only alternative and that needs firewall changes. +**Reference:** R01 §Ingress path nginx termination, dependency #4. + +### BLOCKING #3 — Gitea token admin scope OR manual system-webhook registration +**Question:** Will Rooh elevate the sol token to include `read:admin` + `write:admin`, or will Rooh manually register the system-level webhook one time? +**Why it blocks:** Without admin scope Caret cannot list system-level webhooks (R03 confirmed) and cannot create them. The sol/* per-repo webhooks can be registered with the existing token, but for new repo bootstrap to register Caret's webhook automatically (T1.02), the admin scope or manual setup is required. +**Reference:** R03 §Registered Gitea webhooks, PLAN §Dependencies token scope. + +### BLOCKING #4 — Secret rotation story across openclaw + Caret +**Question:** When `GITEA_TOKEN` or the webhook secret rotates, what's the rollout? Does Caret read from openclaw's credentials dir live, or get its own copy that needs separate rotation? Who is responsible for rotation drills? +**Why it blocks:** PLAN §Risks #1 explicitly calls this out as a first-class deliverable. Without a documented rotation procedure, T3.13 cannot pass and the migration leaves a security debt. +**Reference:** PLAN §Risks #1, dependency #20. + +### BLOCKING #5 — Spawn signature secret ownership (`/root/.openclaw/hooks/spawn-secret`) +**Question:** Does Caret get its own spawn secret (and `create-implement-issue.sh` is updated to use it), or does Caret read openclaw's existing secret? If the latter, what happens when openclaw rotates it? +**Why it blocks:** T1.15–T1.17 (spawn signature verification) require Caret and `create-implement-issue.sh` to share a secret. Cross-system sharing is the cleaner short-term answer but locks Caret to openclaw's lifecycle until cut-over completes. +**Reference:** R01 §Ingress path "spawn signatures", dependency #24. diff --git a/FEATURE-PARITY-TESTS.md b/FEATURE-PARITY-TESTS.md new file mode 100644 index 0000000..adaa9ad --- /dev/null +++ b/FEATURE-PARITY-TESTS.md @@ -0,0 +1,126 @@ +# Feature parity test list + +Every test must pass on the Caret replacement before cut-over (Phase 5). Tests are ordered by criticality — tier 1 is "must pass", tier 2 is "should pass", tier 3 is "nice to have". Sources cited inline: R01 = RESEARCH-01-gitea-webhooks-deep-read.md, R02 = RESEARCH-02-gateway-internals.md, R03 = RESEARCH-03-live-state-audit.md. + +## Tier 1 — mandatory + +### Repo creation / bootstrap + +- [ ] T1.01 new sol/* repo created → Rooh (user id 29) added as admin collaborator within 10s [R01 §Tools fan-out, post-repo-audit.sh] +- [ ] T1.02 new sol/* repo created → Caret webhook registered on the repo within 10s, URL points to Caret listener, content-type application/json [R01 §Tools fan-out, R03 §Registered Gitea webhooks] +- [ ] T1.03 new sol/* repo created → required policy files (Makefile, .editorconfig, .prettierrc, .prettierignore, .dockerignore, .gitignore, tools/secret-scan.sh) present on default branch within 30s [R01 audit-repo-policies.sh] +- [ ] T1.04 repo creation handler is idempotent — re-firing the same `repository.create` event does NOT create duplicate collaborator entries or duplicate webhooks [PLAN §Risks idempotency] +- [ ] T1.05 repo creation handler runs as a pure script with zero LLM tokens consumed (asserted by zero entries in any token-spend log for that delivery id) [R01 post-repo-audit.sh "zero tokens"] + +### Authentication / ingress + +- [ ] T1.06 HTTPS POST to listener with valid HMAC-SHA256 over the raw body using the configured webhook secret → 200, processed [R01 §HMAC recipe] +- [ ] T1.07 POST with wrong HMAC signature → rejected 403 within 50ms, body parsing skipped, "hmac_failed" line in log [R01 §HMAC recipe] +- [ ] T1.08 POST missing `X-Gitea-Signature` header → rejected 403 with "missing_signature" log line [R01 §HMAC recipe] +- [ ] T1.09 POST with valid HMAC but non-Gitea content-type → rejected 415 [R03 webhook content-type] +- [ ] T1.10 raw body must be available to verifier — listener does NOT JSON-parse before HMAC verification (asserted by sending malformed JSON with valid HMAC and observing 200 + "unparseable_body" log) [R01 §HMAC recipe gotcha] +- [ ] T1.11 timing-safe HMAC compare — sending one-byte-off signatures over 1000 requests shows constant-time response (variance < 5ms) [R01 §HMAC recipe `timingSafeEqual`] +- [ ] T1.12 listener bound only to localhost OR fronted by nginx ACL — direct connection from non-allowlisted host refused [R01 §Ingress path] + +### Event routing + +- [ ] T1.13 `push` event to non-default branch → recorded in log, no enforcement scripts fired [R01 §Transform phases push] +- [ ] T1.14 `push` to main/master → triggers secret-scan and policy re-check [R01 §Transform phases] +- [ ] T1.15 `issues.opened` with title prefix `[IMPLEMENT]` from sol with valid `` → spawn signature verified, dispatch enqueued [R01 §Transform phases issues.opened] +- [ ] T1.16 `issues.opened` `[IMPLEMENT]` from sol with stale (>2h) signature → rejected with "spawn_sig_expired" [R01 gotcha 3] +- [ ] T1.17 `issues.opened` `[IMPLEMENT]` from sol with invalid HMAC → rejected with "spawn_sig_invalid", incident written to incidents.jsonl [R01 §Transform phases] +- [ ] T1.18 `issue_comment` containing approval word from Rooh (id 29) → lock acquired, EXECUTE_PLAN dispatched [R01 §Transform phases issue_comment] +- [ ] T1.19 `issue_comment` approval word from non-Rooh → ignored, log line "approval_ignored_non_owner" [R01 §Trust level] +- [ ] T1.20 `issue_comment` approval word from sol account → ignored (sol is contributor only) [R01 gotcha 6] +- [ ] T1.21 events from `clawbot` sender → silently dropped (loop prevention) [R01 §Validation gates] +- [ ] T1.22 issue body containing `` (or Caret equivalent) → silently dropped (echo suppression) [R01 §Validation gates] + +### Idempotency / dedup + +- [ ] T1.23 duplicate delivery (same `X-Gitea-Delivery` id within 24h) → returns 200 but skips processing, log line "dedup_hit" [R01 §Ingress dedup, R01 gotcha 4] +- [ ] T1.24 dedup cache persisted to disk and survives listener restart [R01 §Ingress dedup] +- [ ] T1.25 dedup cache trims entries older than 24h on each write [R01 §Ingress dedup] + +### Locking / concurrency + +- [ ] T1.26 concurrent `issue_comment` events on the same `(owner, repo, issue)` → only one acquires the lock; second sees "lock_held" and is queued or dropped per policy [R01 §Session lock] +- [ ] T1.27 lock file TTL is 2h; expired lock is reclaimable [R01 §Session lock] +- [ ] T1.28 issue closed while locked → transitions to IS_DONE with 5min grace before lock release [R01 §Transform phases, R01 gotcha 2] +- [ ] T1.29 rate limit: 6th concurrent agent dispatch for the same agent id → rejected with "rate_limited" [R01 gotcha 7] + +### Observability / audit + +- [ ] T1.30 every accepted event produces exactly one line in `audit.jsonl` containing `{ts, delivery_id, event, repo, sender, decision}` [R01 §Logging] +- [ ] T1.31 every rejected event also produces an audit line with `decision: rejected` and reason [R01 §Logging, PLAN §Risks silent drops] +- [ ] T1.32 incidents (HMAC fail, spawn sig fail, script error) appended to `incidents.jsonl` [R01 §Logging] + +### Rollback + +- [ ] T1.33 disabling Caret listener and re-enabling openclaw gateway restores end-to-end pipeline within 60s, verified by canary repo create [PLAN §Phase 5 C5.3] +- [ ] T1.34 Caret listener can be stopped with one command and pending in-flight requests drained or 503'd cleanly [PLAN §Phase 5] + +## Tier 2 — should pass + +### Push / scan / policy + +- [ ] T2.01 push to main triggers `secret-scan.sh`; a planted private-key blob in the diff creates a Gitea issue labeled `security` within 30s [R01 secret-scan.sh] +- [ ] T2.02 push to main with no findings → exit 0, audit line "scan_clean", no issue created [R01 secret-scan.sh] +- [ ] T2.03 secret-scan respects `.secret-scan-allowlist` — allowlisted hash is not flagged [R01 secret-scan.sh] +- [ ] T2.04 `audit-repo-policies.sh --fix` re-applies missing files on a repo where one was deleted, within next 6h heartbeat [R01 audit-repo-policies.sh] +- [ ] T2.05 `audit-webhooks.sh --fix` recreates a deleted webhook within next 15min check [R01 audit-webhooks.sh] +- [ ] T2.06 push that touches `tools/secret-scan.sh` itself → policy re-check still passes (file is part of policy template) [R01 audit-repo-policies.sh] + +### Spawn pipeline + +- [ ] T2.07 `[IMPLEMENT]` issue with valid sig → `spawn-manager.sh` invoked, project workspace `PROJ-XXX-*` created [R01 spawn-manager.sh] +- [ ] T2.08 `precomputeSpawnParams` execSync timeout (30s) → falls back to text-directive without crashing the listener [R01 gotcha 9] +- [ ] T2.09 `asyncDispatchToSpawner` failure (spawner down) → dispatch failure recorded in incidents.jsonl, NOT a 500 to Gitea [R01 gotcha 8] +- [ ] T2.10 `check-implement-orphans.sh` equivalent runs every 15min and detects stale pending spawn files older than 2h [R01 check-implement-orphans.sh] + +### Trust levels + +- [ ] T2.11 sender id 29 → trust=owner; collaborator → trust=contributor; unknown → trust=readonly [R01 §Trust level detection] +- [ ] T2.12 readonly sender events → no script fan-out, only audit log [R01 §Trust level detection] + +### Heartbeat / cron + +- [ ] T2.13 6h policy sweep job runs on schedule and exits 0 [PLAN §Phase 4, R03 webhook-verify] +- [ ] T2.14 15min webhook-audit job runs on schedule and exits 0 [R01 audit-webhooks.sh] +- [ ] T2.15 cron job failure increments a consecutive-failure counter; ≥3 consecutive failures posts a Telegram alert [R03 §Critical finding cron failures] +- [ ] T2.16 webhook-verify E2E canary (synthetic event roundtrip) succeeds every 6h [R03 webhook-verify] + +### Gitea API hygiene + +- [ ] T2.17 Gitea API call failure (5xx) → retried with exponential backoff up to 3 attempts before recording incident [PLAN §Risks] +- [ ] T2.18 Gitea API rate-limit response (429) → backs off per Retry-After header, no incident on first occurrence [R02 §Replacement difficulty] +- [ ] T2.19 token rotation: changing the Gitea token in config and SIGHUP'ing the listener takes effect without restart [PLAN §Risks HMAC secret management] + +### Concurrency edges + +- [ ] T2.20 two `repository.create` events for the same repo arriving within 100ms → exactly one bootstrap run, second deduped [T1.04 + dedup] +- [ ] T2.21 listener under 50 req/s sustained for 60s → no dropped events, p99 latency < 500ms [R02 §Hook API] +- [ ] T2.22 lock acquisition under contention is fair-ish — no event waits >5min behind a single 2h lock when policy says to drop, not queue [R01 §Session lock] + +### Delivery / alerting + +- [ ] T2.23 critical incident (HMAC failure storm: >10 in 1min) → Telegram alert posted via tg-stream within 30s [PLAN §Phase 3] +- [ ] T2.24 alert delivery failure (Telegram down) → fallback to Mattermost, then to local incidents.jsonl [R02 §Delivery system] +- [ ] T2.25 listener exposes a metrics counter for events_total, events_rejected_total, hmac_failures_total [R02 §Hook API observability] + +## Tier 3 — nice to have + +- [ ] T3.01 listener exposes `/health` returning `{"ok":true,"uptime_s":N,"last_event_ms":N}` [R02 §HTTP endpoints health] +- [ ] T3.02 listener exposes `/ready` returning 503 until dedup cache loaded and Gitea token validated [R02 §HTTP endpoints] +- [ ] T3.03 audit.jsonl rotated by line count (configurable, default 100k lines) — rotation happens without losing in-flight writes [PLAN B2.5] +- [ ] T3.04 audit.jsonl old segments gzipped after rotation [PLAN B2.5] +- [ ] T3.05 listener restart replays no events (dedup cache prevents) and reports start time in /health [R01 §Ingress dedup persisted] +- [ ] T3.06 structured log lines parseable as single-line JSON, no multi-line stack traces [PLAN B2.5] +- [ ] T3.07 listener handles SIGTERM gracefully — finishes in-flight, refuses new, exits within 10s [PLAN §Phase 5] +- [ ] T3.08 dry-run mode (`CARET_DRYRUN=1`) logs the script that *would* run without executing [R02 §Tools invoke dryRun] +- [ ] T3.09 admin API endpoint to list registered Gitea webhooks across all sol/* repos in one call (requires admin token) [R03 §Registered Gitea webhooks token scope] +- [ ] T3.10 manual replay endpoint: POST `/replay/{delivery_id}` re-processes a stored event bypassing dedup [PLAN §Risks idempotency] +- [ ] T3.11 listener config hot-reload on SIGHUP without restart [R02 §Hot-reload config] +- [ ] T3.12 PR (`pull_request`) events trigger same policy checks as push to main [R01 §Transform phases] +- [ ] T3.13 webhook secret rotation runbook: rotate, deploy, verify, rollback documented and tested [PLAN §Risks HMAC secret management] +- [ ] T3.14 listener supports both `Authorization: Bearer` AND `X-Gitea-Signature` simultaneously for migration window [R01 §Ingress path layered auth] +- [ ] T3.15 cost report: weekly summary of LLM tokens spent by judgment-path wakeups vs deterministic-path zero-cost runs [PLAN §Phase 3 cost hygiene]