openclaw-to-caret-migration/research/RESEARCH-03-live-state-audit.md

Research 03 — live openclaw state audit

Subagent: abf0cb0928d823a0b (Explore) Completed: 2026-04-06 12:50 UTC Status: openclaw system currently DEGRADED — multiple cron timeouts and model misconfiguration.

🚨 CRITICAL FINDING — openclaw is degraded right now

The subagent discovered the openclaw system is already unhealthy:

1. Model misconfiguration. openclaw.json line 138 references claudehack/claude-sonnet-4-6, which does not exist in the models provider list. All heartbeat jobs fail with FailoverError: Unknown model before any LLM call is made. This matches the errors I saw in the gateway logs earlier today.
2. Cron job failures. 8 of 12 scheduled cron jobs are in error state with 4-20 consecutive failures each:
   • self-review-3day, compress-daily-notes, project-archive, rooh-style-review, rooh-style-reply-handler (20 consecutive failures), queue-doctor-6h, ws-sync, webhook-verify — all failing
   • Only rooh-style-learner is currently succeeding
3. API pressure. Failover chains exhaust within 10-11s per attempt: Anthropic Opus/Sonnet timeout, OpenAI Codex rate-limited, openai/gpt-5.4 has no API key in this env.
4. Stale data. db/repos.json is stale — the ws-sync cron has been failing since at least 2026-04-05, so the cached repo list is 48+ hours old.
5. Webhook E2E test failing every 6h. The webhook-verify cron that's supposed to do a full pipeline delivery test hasn't succeeded for multiple cycles.

Implication for the migration: this is not a "replace a working system" project. It's a "replace a system that's already showing cracks". The migration deadline gets more urgent because the current pipeline is on thin ice.

Registered Gitea webhooks

The sol token lacks read:admin scope so system-level webhooks can't be listed. But the per-repo webhooks paint a clear picture:

Repo	URL	Events	Secret	Content-Type
gitea-webhooks	https://slack.solio.tech/hooks/gitea	issues, issue_comment, issue_label, issue_assign, issue_milestone	NOT SET	application/json
openclaw-mattermost	https://slack.solio.tech/hooks/gitea	(same)	NOT SET	application/json
openclaw-commands	https://slack.solio.tech/hooks/gitea	(same)	NOT SET	application/json
e2e-ticket-system	https://slack.solio.tech/hooks/gitea	(same)	NOT SET	application/json

Pattern: every repo has a webhook that posts to the same single endpoint. HMAC secret is not set on any of them — confirming Research 01's finding that openclaw's pipeline is authenticated via bearer token + nginx ACL, not Gitea HMAC.

Event set is uniform: the five issue/PR-style events. Notably MISSING from the event list: push, repository, create, fork. Either those are configured elsewhere (system-level admin webhooks, which I can't see) or the post-repo-audit.sh pipeline only fires when a human manually triggers it.

Running cron / timer infrastructure

• No systemd timers (systemctl not in container)
• No classic crontab
• All scheduling is openclaw's internal scheduler — queue-daemon.js running as PID 4 inside the gateway container, reading ~/.openclaw/cron/jobs.json
• Jobs are in-process timers managed by the gateway's CronService

Active cron jobs (from jobs.json)

Job ID	Schedule (UTC)	Status	Last result
self-review-3day	0 6 */3 * *	Enabled	Error (timeout)
compress-daily-notes	0 4 * * *	Enabled	Error (timeout)
project-archive	30 3 * * *	Enabled	Error (timeout)
rooh-style-review	0 */4 * * *	Enabled	Error (timeout)
rooh-style-reply-handler	*/5 * * * *	Enabled	Error (timeout, 20 consecutive)
queue-doctor-6h	0 */6 * * *	Enabled	Error (timeout)
rooh-style-learner	0 */2 * * *	Enabled	OK ✓
ws-sync	daily	Enabled	Error (timeout)
webhook-verify	0 */6 * * *	Enabled	Error (timeout)

Daemon: /root/.openclaw/hooks/queue-daemon.js (42KB, active). PID tracked at /root/.openclaw/hooks/queue-daemon.pid.

Heartbeat checklists

• Main agent (/root/.openclaw/workspace/HEARTBEAT.md): comprehensive, 28 checks across 7 intervals, managed by tools/heartbeat-scheduler.sh with state at memory/heartbeat-state.json. Last check 2026-04-05 15:18 UTC. Every-heartbeat items include protocol re-read, project DB validation, incident checking, webhook auditing. Every-6h includes repo policy audit, webhook E2E verification.
• Xen agent (/root/.openclaw/agents/xen/workspace/HEARTBEAT.md): intentionally empty (comment-only). No periodic tasks for Xen.
• Spawner, gitea-worker, coder-agent, god-agent, global-calendar, gym-designer, nutrition, particles-ai: all have empty or task-driven heartbeat configs, not time-driven.

Pattern: only the main agent runs the comprehensive heartbeat. Everything else is event-driven.

Live agent processes

Agent	Runtime	Status
main	Embedded in openclaw gateway	Running (part of the gateway process)
xen	Directory config only	Configured, not actively running
spawner	Directory config only	Configured, not actively running
gitea-worker	Directory config only	Configured, event-driven (spawned by queue daemon)
coder-agent	Directory config only	Not running
god-agent	Workspace only	Not running
global-calendar	Workspace only	Not running
gym-designer	Workspace only	Not running
nutrition-agent	Shares gym workspace	Not running
particles-ai	Directory config only	Not running

Key discovery — main agent is NOT a separate process

The main agent ("Xen") isn't a long-lived process. It's embedded in the openclaw-openclaw-gateway-1 container, running as in-process Claude sessions triggered by user input and heartbeat events. The gateway was last restarted ~5 hours ago.

Configuration:

• Primary model: anthropic/claude-opus-4-6
• Fallback chain: Sonnet, gpt-5.4, gpt-5.3-codex
• Workspace: /root/.openclaw/workspace (155 active projects)

claude-worker (me) is NOT alive right now

The subagent checked the process list for tmux sessions matching the claude-worker startup scripts (/root/start-claude-worker.sh, claude-worker-watcher.sh) and found no such session alive. My current running session (this conversation) was started directly, not via those scripts.

Shared state and data stores

All under /root/.openclaw/:

Store	Path	Purpose	Freshness
Project registry	workspace/projects/registry.json	Master index of 155+ projects, status, repo links, manager assignments	Real-time, updated on lifecycle events
Repo database	workspace/db/repos.json	Cached list of sol/* Gitea repos, metadata	STALE (48+ hours, ws-sync failing)
Infrastructure registry	workspace/db/infra.json	Running containers, scripts, transforms, cron, sandboxes	Last good scan 2026-04-05 15:04 UTC
Agent source mapping	workspace/db/agent-sources.json	Which repo built which agent	Manual
Memory system	workspace/memory/	Daily logs, lessons, errors, checklist state, heartbeat timestamps, rooh-translator staging	Real-time
Projects DB (text)	workspace/PROJECTS_DB.md	Human-readable project summary	Manual
Cron jobs state	cron/jobs.json	12 scheduled jobs, execution state, error counts	Very high churn (failing jobs retrying)
Cron logs	hooks/logs/audit.jsonl, incidents.jsonl	JSONL audit trail	Real-time
Queue inbox	hooks/queue-inbox/	Temporary staging for webhook payloads	Ephemeral
Vault	vault-data/	Encrypted configuration, secret state	Sandbox-managed
Credentials	credentials/	API tokens, OAuth profiles, Anthropic keys (perms 700)	Updated by sync-oauth-token.sh
Agent auth profiles	agents/*/agent/auth-profiles.json	Per-agent Anthropic, Google, OpenAI, Codex creds	Last morning sync 2026-04-06 07:11

Architecture: event-driven with scheduled backup scans. Real-time data in workspace/ is Git-versioned (branch main). Calculated data in db/ is cached output of pure bash+jq scripts. No central database; on-disk state is source of truth.

What the migration actually has to replace

A single main agent (Opus, embedded in gateway) managing 155 projects across 47 sol/* Gitea repos with a distributed sub-agent orchestration pattern. The agent operates on:

• Inbound: Gitea webhooks → slack.solio.tech/hooks/gitea → queue daemon → gitea-worker transform → agent spawn
• Outbound: Cron jobs (12 scheduled, 8 failing) → delivery queue → agent heartbeats → Mattermost DMs and Gitea issue updates
• State: Git-versioned /root/.openclaw/workspace/ with 5-level agent hierarchy and a 28-item heartbeat checklist

The migration only needs to replace the Gitea-facing slice (webhook ingress, event router, script fan-out, repo policy enforcement). The rest of the system — the 155 projects, the workspace, the memory system, the sub-agent orchestration — stays owned by openclaw unless Rooh expands the scope.

Caret's immediate observations

1. The model misconfiguration needs to be fixed NOW. Even if I don't touch the migration yet, the claudehack/claude-sonnet-4-6 reference is breaking the heartbeat. That's Xen's production pipeline. Worth flagging to Rooh.
2. The admin scope issue is real. I cannot list system-level webhooks. Any migration requires either token elevation or Rooh manually registering the new endpoint.
3. Current security is flimsier than the code looks. All four known webhooks have no HMAC secret. The protection is bearer token + nginx ACL. My replacement should do better — implement real HMAC on the raw body.
4. The migration scope is smaller than I feared. I don't need to replicate the 28-item heartbeat or the 155-project workspace. I only need the Gitea-facing slice. That's a few days of work, not 4-8 weeks.
5. Rooh's earlier 6-hour cron I set up (eaeef6ff) is the right shape for the basic policy sweep. It doesn't need to be elaborate.