sol/openclaw-to-caret-migration
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

research: Phase 0 reports 2 and 3 — gateway internals + live state audit

Browse Source
This commit is contained in:
Caret
2026-04-06 12:50:22 +00:00
parent f3296d136a
commit d36fa6538a
2 changed files with 295 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

132
research/RESEARCH-03-live-state-audit.md Normal file
View File

@@ -0,0 +1,132 @@
# Research 03 — live openclaw state audit
**Subagent:** `abf0cb0928d823a0b` (Explore)
**Completed:** 2026-04-06 12:50 UTC
**Status:** openclaw system currently DEGRADED — multiple cron timeouts and model misconfiguration.
## 🚨 CRITICAL FINDING — openclaw is degraded right now
The subagent discovered the openclaw system is already unhealthy:
1. **Model misconfiguration.** `openclaw.json` line 138 references `claudehack/claude-sonnet-4-6`, which does not exist in the models provider list. All heartbeat jobs fail with `FailoverError: Unknown model` before any LLM call is made. This matches the errors I saw in the gateway logs earlier today.
2. **Cron job failures.** 8 of 12 scheduled cron jobs are in error state with 4-20 consecutive failures each:
- `self-review-3day`, `compress-daily-notes`, `project-archive`, `rooh-style-review`, `rooh-style-reply-handler` (20 consecutive failures), `queue-doctor-6h`, `ws-sync`, `webhook-verify` — all failing
- Only `rooh-style-learner` is currently succeeding
3. **API pressure.** Failover chains exhaust within 10-11s per attempt: Anthropic Opus/Sonnet timeout, OpenAI Codex rate-limited, openai/gpt-5.4 has no API key in this env.
4. **Stale data.** `db/repos.json` is stale — the `ws-sync` cron has been failing since at least 2026-04-05, so the cached repo list is 48+ hours old.
5. **Webhook E2E test failing every 6h.** The `webhook-verify` cron that's supposed to do a full pipeline delivery test hasn't succeeded for multiple cycles.
**Implication for the migration:** this is not a "replace a working system" project. It's a "replace a system that's already showing cracks". The migration deadline gets more urgent because the current pipeline is on thin ice.
## Registered Gitea webhooks
The sol token lacks `read:admin` scope so system-level webhooks can't be listed. But the per-repo webhooks paint a clear picture:
| Repo | URL | Events | Secret | Content-Type |
|----------------------|----------------------------------------|---------------------------------------------------------------------|-----------|--------------------|
| gitea-webhooks | https://slack.solio.tech/hooks/gitea | issues, issue_comment, issue_label, issue_assign, issue_milestone | NOT SET | application/json |
| openclaw-mattermost | https://slack.solio.tech/hooks/gitea | (same) | NOT SET | application/json |
| openclaw-commands | https://slack.solio.tech/hooks/gitea | (same) | NOT SET | application/json |
| e2e-ticket-system | https://slack.solio.tech/hooks/gitea | (same) | NOT SET | application/json |
**Pattern:** every repo has a webhook that posts to the same single endpoint. `HMAC secret is not set on any of them` — confirming Research 01's finding that openclaw's pipeline is authenticated via bearer token + nginx ACL, not Gitea HMAC.
Event set is uniform: the five issue/PR-style events. Notably MISSING from the event list: `push`, `repository`, `create`, `fork`. Either those are configured elsewhere (system-level admin webhooks, which I can't see) or the `post-repo-audit.sh` pipeline only fires when a human manually triggers it.
## Running cron / timer infrastructure
- **No systemd timers** (systemctl not in container)
- **No classic crontab**
- **All scheduling is openclaw's internal scheduler** — `queue-daemon.js` running as PID 4 inside the gateway container, reading `~/.openclaw/cron/jobs.json`
- Jobs are in-process timers managed by the gateway's `CronService`
### Active cron jobs (from jobs.json)
| Job ID | Schedule (UTC) | Status | Last result |
|-----------------------------|---------------------|----------|---------------------------------|
| self-review-3day | `0 6 */3 * *` | Enabled | Error (timeout) |
| compress-daily-notes | `0 4 * * *` | Enabled | Error (timeout) |
| project-archive | `30 3 * * *` | Enabled | Error (timeout) |
| rooh-style-review | `0 */4 * * *` | Enabled | Error (timeout) |
| rooh-style-reply-handler | `*/5 * * * *` | Enabled | Error (timeout, 20 consecutive)|
| queue-doctor-6h | `0 */6 * * *` | Enabled | Error (timeout) |
| rooh-style-learner | `0 */2 * * *` | Enabled | OK ✓ |
| ws-sync | daily | Enabled | Error (timeout) |
| webhook-verify | `0 */6 * * *` | Enabled | Error (timeout) |
Daemon: `/root/.openclaw/hooks/queue-daemon.js` (42KB, active). PID tracked at `/root/.openclaw/hooks/queue-daemon.pid`.
## Heartbeat checklists
- **Main agent** (`/root/.openclaw/workspace/HEARTBEAT.md`): comprehensive, 28 checks across 7 intervals, managed by `tools/heartbeat-scheduler.sh` with state at `memory/heartbeat-state.json`. Last check 2026-04-05 15:18 UTC. Every-heartbeat items include protocol re-read, project DB validation, incident checking, webhook auditing. Every-6h includes repo policy audit, webhook E2E verification.
- **Xen agent** (`/root/.openclaw/agents/xen/workspace/HEARTBEAT.md`): intentionally empty (comment-only). **No periodic tasks for Xen.**
- **Spawner, gitea-worker, coder-agent, god-agent, global-calendar, gym-designer, nutrition, particles-ai**: all have empty or task-driven heartbeat configs, not time-driven.
**Pattern:** only the main agent runs the comprehensive heartbeat. Everything else is event-driven.
## Live agent processes
| Agent | Runtime | Status |
|------------------|--------------------------------------|--------------------------------------------------------|
| main | Embedded in openclaw gateway | Running (part of the gateway process) |
| xen | Directory config only | Configured, not actively running |
| spawner | Directory config only | Configured, not actively running |
| gitea-worker | Directory config only | Configured, event-driven (spawned by queue daemon) |
| coder-agent | Directory config only | Not running |
| god-agent | Workspace only | Not running |
| global-calendar | Workspace only | Not running |
| gym-designer | Workspace only | Not running |
| nutrition-agent | Shares gym workspace | Not running |
| particles-ai | Directory config only | Not running |
### Key discovery — main agent is NOT a separate process
The main agent ("Xen") isn't a long-lived process. It's **embedded in the openclaw-openclaw-gateway-1 container**, running as in-process Claude sessions triggered by user input and heartbeat events. The gateway was last restarted ~5 hours ago.
Configuration:
- Primary model: `anthropic/claude-opus-4-6`
- Fallback chain: Sonnet, gpt-5.4, gpt-5.3-codex
- Workspace: `/root/.openclaw/workspace` (155 active projects)
### claude-worker (me) is NOT alive right now
The subagent checked the process list for `tmux` sessions matching the claude-worker startup scripts (`/root/start-claude-worker.sh`, `claude-worker-watcher.sh`) and **found no such session alive**. My current running session (this conversation) was started directly, not via those scripts.
## Shared state and data stores
All under `/root/.openclaw/`:
| Store | Path | Purpose | Freshness |
|-------------------------|--------------------------------------|--------------------------------------------------------|----------------------------------|
| Project registry | `workspace/projects/registry.json` | Master index of 155+ projects, status, repo links, manager assignments | Real-time, updated on lifecycle events |
| Repo database | `workspace/db/repos.json` | Cached list of sol/* Gitea repos, metadata | **STALE** (48+ hours, ws-sync failing) |
| Infrastructure registry | `workspace/db/infra.json` | Running containers, scripts, transforms, cron, sandboxes | Last good scan 2026-04-05 15:04 UTC |
| Agent source mapping | `workspace/db/agent-sources.json` | Which repo built which agent | Manual |
| Memory system | `workspace/memory/` | Daily logs, lessons, errors, checklist state, heartbeat timestamps, rooh-translator staging | Real-time |
| Projects DB (text) | `workspace/PROJECTS_DB.md` | Human-readable project summary | Manual |
| Cron jobs state | `cron/jobs.json` | 12 scheduled jobs, execution state, error counts | Very high churn (failing jobs retrying) |
| Cron logs | `hooks/logs/audit.jsonl`, `incidents.jsonl` | JSONL audit trail | Real-time |
| Queue inbox | `hooks/queue-inbox/` | Temporary staging for webhook payloads | Ephemeral |
| Vault | `vault-data/` | Encrypted configuration, secret state | Sandbox-managed |
| Credentials | `credentials/` | API tokens, OAuth profiles, Anthropic keys (perms 700)| Updated by sync-oauth-token.sh |
| Agent auth profiles | `agents/*/agent/auth-profiles.json` | Per-agent Anthropic, Google, OpenAI, Codex creds | Last morning sync 2026-04-06 07:11 |
**Architecture:** event-driven with scheduled backup scans. Real-time data in `workspace/` is Git-versioned (branch `main`). Calculated data in `db/` is cached output of pure bash+jq scripts. No central database; on-disk state is source of truth.
## What the migration actually has to replace
A **single main agent** (Opus, embedded in gateway) managing 155 projects across 47 sol/* Gitea repos with a distributed sub-agent orchestration pattern. The agent operates on:
- **Inbound:** Gitea webhooks → `slack.solio.tech/hooks/gitea` → queue daemon → gitea-worker transform → agent spawn
- **Outbound:** Cron jobs (12 scheduled, 8 failing) → delivery queue → agent heartbeats → Mattermost DMs and Gitea issue updates
- **State:** Git-versioned `/root/.openclaw/workspace/` with 5-level agent hierarchy and a 28-item heartbeat checklist
The migration only needs to replace the **Gitea-facing slice** (webhook ingress, event router, script fan-out, repo policy enforcement). The rest of the system — the 155 projects, the workspace, the memory system, the sub-agent orchestration — stays owned by openclaw unless Rooh expands the scope.
## Caret's immediate observations
1. **The model misconfiguration needs to be fixed NOW.** Even if I don't touch the migration yet, the `claudehack/claude-sonnet-4-6` reference is breaking the heartbeat. That's Xen's production pipeline. Worth flagging to Rooh.
2. **The admin scope issue is real.** I cannot list system-level webhooks. Any migration requires either token elevation or Rooh manually registering the new endpoint.
3. **Current security is flimsier than the code looks.** All four known webhooks have no HMAC secret. The protection is bearer token + nginx ACL. My replacement should do better — implement real HMAC on the raw body.
4. **The migration scope is smaller than I feared.** I don't need to replicate the 28-item heartbeat or the 155-project workspace. I only need the Gitea-facing slice. That's a few days of work, not 4-8 weeks.
5. **Rooh's earlier 6-hour cron I set up (`eaeef6ff`) is the right shape** for the basic policy sweep. It doesn't need to be elaborate.