From c700721806373364035dc7e5056d8cf7fa56dfad Mon Sep 17 00:00:00 2001 From: clawbot Date: Sat, 28 Feb 2026 02:03:52 -0800 Subject: [PATCH] add system overview: how and why the pieces fit together --- OPENCLAW_TRICKS.md | 419 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 419 insertions(+) diff --git a/OPENCLAW_TRICKS.md b/OPENCLAW_TRICKS.md index a79371a..e2b3a68 100644 --- a/OPENCLAW_TRICKS.md +++ b/OPENCLAW_TRICKS.md @@ -6,6 +6,425 @@ production setup that's been running since early 2026. --- +## How This System Evolved + +This section explains what we built, why, and how the pieces fit together. The +rest of the document has the specific prompts and schemas — this part gives you +the big picture so you understand the design decisions. + +### The Starting Point + +OpenClaw gives you a persistent workspace directory and the ability to run +tools (shell, web, browser, messaging). Out of the box, the agent wakes up +fresh every session with no memory of what happened before. The core challenge +is: **how do you turn a stateless LLM into a stateful personal assistant that +knows who you are, where you are, what you need, and what it was working on?** + +The answer is files. The workspace IS the agent's brain. Every piece of state, +every rule, every memory lives in files that the agent reads on startup and +updates as things change. + +### The Workspace Files — Separation of Concerns + +We started with one big AGENTS.md that had everything: rules, procedures, +medication instructions, git workflow, travel preferences. It worked at first +but quickly became unwieldy. The context window was getting eaten by +instructions the agent didn't need for most tasks. + +The solution was **factoring out rulesets into focused files:** + +- **AGENTS.md** — the master file. Responsibilities, session startup + procedure, high-level rules. Think of it as the table of contents that + points to everything else. +- **SOUL.md** — personality, tone, values. Separated because it's + philosophical, not procedural. Also: it's fun to let the agent evolve this + one over time. +- **USER.md** — info about the human. Timezone, communication preferences, + basics the agent needs every session. +- **MEMORY.md** — curated long-term memory. Only loaded in private sessions + (security — prevents leaking personal context in group chats). +- **HEARTBEAT.md** — what to check on periodic heartbeat polls. Kept small + intentionally to minimize token burn on frequent checks. +- **TOOLS.md** — environment-specific notes (hostnames, device names, channel + IDs). Separated from skills because this is YOUR setup, skills are shared. + +Then the procedural rulesets got their own files too: + +- **`memory/medications-instructions.md`** — full medication protocol. Dosages, + schedules, safety rules, overdose prevention logic. Only loaded when + medication tasks come up. +- **`memory/checklist-pr.md`** — PR quality gate. What to verify before pushing, + after sub-agent pushes, before assigning to the human for review. +- **`memory/checklist-flights.md`** — flight time verification, prep block + creation, landing checklist triggers. +- **`memory/checklist-medications.md`** — pre-action verification before + reporting medication status or sending reminders. +- **`memory/checklist-messaging.md`** — rules for every outgoing message: URL + verification, timezone conversion, status claim verification. +- **`memory/landing-checklist.md`** — post-flight procedures (update location, + timezone, check meds, sync calendar). + +The key insight: **MEMORY.md has a "checklists" section at the very top** that +says "before doing X, read file Y." The agent reads MEMORY.md at session start, +sees the checklist index, and knows which file to consult before any action. +This way the detailed instructions aren't always in context — they're loaded +on-demand. + +```markdown +## ⛔ CHECKLISTS (read the relevant file before acting) + +- **Any PR/code work** → `memory/checklist-pr.md` +- **Any message to the user** → `memory/checklist-messaging.md` +- **Medications** → `memory/checklist-medications.md` +- **Flights/travel** → `memory/checklist-flights.md` +``` + +### The Daily Context State File + +This was probably the single most impactful addition. It's a JSON file +(`memory/daily-context.json`) that every session reads on every message. It +tracks the current state of the human: where they are, what timezone they're +in, whether they're sleeping, when they last took meds, when they last sent a +message. + +Why JSON instead of markdown? Because it's machine-readable. The agent can +update individual fields programmatically, and the structure is unambiguous. +Markdown is great for instructions; JSON is great for state. + +The daily context file means: +- The agent always knows the human's timezone (critical for a frequent + traveler) +- Medication reminders fire based on actual timestamps, not guesses +- Sleep predictions inform when to send alerts vs stay quiet +- Location tracking happens automatically + +### The Two-Tier Memory System + +Daily files (`memory/YYYY-MM-DD.md`) are raw logs — what happened each day. +Topic tables, decisions, lessons, notes. Think of these as a daily journal. + +MEMORY.md is curated long-term memory — the distilled essence. Standing rules, +key facts, lessons learned. Think of this as what a human would "just know" +about their life. + +During heartbeats, the agent periodically reviews recent daily files and +promotes significant items to MEMORY.md, and removes stale info. It's +explicitly modeled on how human memory works: raw experience gets processed +into lasting knowledge, and irrelevant details fade. + +The security model: MEMORY.md is only loaded in private (1:1 DM) sessions. +In group chats, the agent works from daily files and daily-context.json only. +This prevents personal context from leaking into shared channels. + +### Medication Tracking + +This is a safety-critical system. The design is deliberately paranoid: + +- **CSV as source of truth.** Not the daily-context boolean, not the agent's + memory — the CSV log file is authoritative. +- **Double-verification before any action.** The daily context has a + `hasTakenDailyMedsToday` boolean AND a `dailyMedsTimestamp`. A midnight + cron resets the boolean. But if the human is in a timezone ahead of the + server, the reset happens at the wrong time. So the rule is: always verify + the timestamp falls on today's date in the human's timezone. The boolean + is a convenience hint, never the source of truth. +- **Interval medications anchored to actual doses.** Some meds are "every 5 + days" or "every 14 days." The next dose date is calculated from the last + actual dose timestamp, not from the intended schedule. If a dose is missed, + the next date is unknown until the missed dose is taken and logged. +- **Overdose prevention.** The agent blocks logging if it detects a same-day + duplicate batch. This is the highest-priority safety rule — a duplicate + daily batch of certain medications could cause cardiac arrest. +- **Escalating alerts.** If daily meds are >26h since last dose, the agent + escalates aggressively — ntfy push notification if chat messages go + unacknowledged. + +The medication instructions file is loaded on-demand (not every session), +keeping context costs low for non-medication conversations. + +### Sleep Tracking + +The agent infers sleep from activity gaps rather than requiring explicit +"I'm going to sleep" statements. On every message, it checks: was there a +gap since the last message that overlaps with the predicted sleep window? If +so, log sleep start = last activity before gap, wake = first activity after. + +Activity isn't just chat messages — it includes Gitea commits, comments, +scheduled flight departures, and any other observable actions. + +Sleep predictions drift over time (the human in this setup tends to sleep +~30min later each day), so the agent tracks the trend and extrapolates. +Caffeine intake adjusts predictions forward. The predictions feed into: +- When to send alerts vs stay quiet +- Sitrep "sleep conflict" warnings (appointment during predicted sleep) +- General awareness of the human's schedule + +### Location & Timezone Tracking + +The agent always knows where the human is. This matters because: +- Times must always be displayed in the human's local timezone, not the + server's timezone +- "Today" and "tomorrow" mean different things in different timezones +- Medication timing needs to account for timezone changes +- Flight prep needs local airport info + +The landing checklist (triggered automatically after every flight) updates +location, timezone, nearest airport, and lodging in the daily context file. +It also checks if any cron jobs have hardcoded timezones that need updating. + +### The Gitea Notification Poller + +OpenClaw has heartbeats, but those are periodic (every ~30min). For Gitea +issues and PRs, we wanted near-realtime response. The solution: a tiny Python +script that polls the Gitea notifications API every 2 seconds and wakes the +agent via OpenClaw's `/hooks/wake` endpoint when new notifications arrive. + +Key design decisions: +- **The poller never marks notifications as read.** That's the agent's job + after it processes them. This prevents the poller and agent from racing. +- **It tracks notification IDs, not counts.** This way it only fires on + genuinely new notifications, not re-reads of existing ones. +- **The wake message tells the agent to route output to Gitea/Mattermost, + not to DM.** This prevents chatty notification processing from disturbing + the human. +- **Zero dependencies.** Just Python stdlib (`urllib`, `json`, `time`). Runs + anywhere. + +Here's the full source: + +```python +#!/usr/bin/env python3 +""" +Gitea notification poller. +Polls for unread notifications and wakes OpenClaw when the count +changes. The AGENT marks notifications as read after processing — +the poller never marks anything as read. + +Required env vars: + GITEA_URL - Gitea instance URL + GITEA_TOKEN - Gitea API token + HOOK_TOKEN - OpenClaw hooks auth token + +Optional env vars: + GATEWAY_URL - OpenClaw gateway URL (default: http://127.0.0.1:18789) + POLL_DELAY - Delay between polls in seconds (default: 2) +""" + +import json +import os +import sys +import time +import urllib.request +import urllib.error + +GITEA_URL = os.environ.get("GITEA_URL", "").rstrip("/") +GITEA_TOKEN = os.environ.get("GITEA_TOKEN", "") +GATEWAY_URL = os.environ.get("GATEWAY_URL", "http://127.0.0.1:18789").rstrip( + "/" +) +HOOK_TOKEN = os.environ.get("HOOK_TOKEN", "") +POLL_DELAY = int(os.environ.get("POLL_DELAY", "2")) + + +def check_config(): + missing = [] + if not GITEA_URL: + missing.append("GITEA_URL") + if not GITEA_TOKEN: + missing.append("GITEA_TOKEN") + if not HOOK_TOKEN: + missing.append("HOOK_TOKEN") + if missing: + print( + f"ERROR: Missing required env vars: {', '.join(missing)}", + file=sys.stderr, + ) + sys.exit(1) + + +def gitea_unread_ids(): + """Return set of unread notification IDs.""" + req = urllib.request.Request( + f"{GITEA_URL}/api/v1/notifications?status-types=unread", + headers={"Authorization": f"token {GITEA_TOKEN}"}, + ) + try: + with urllib.request.urlopen(req, timeout=10) as resp: + notifs = json.loads(resp.read()) + return {n["id"] for n in notifs} + except Exception as e: + print( + f"WARN: Gitea API failed: {e}", file=sys.stderr, flush=True + ) + return set() + + +def wake_openclaw(count): + text = ( + f"[Gitea Notification] {count} new notification(s). " + "Check your Gitea notification inbox via API, process them, " + "and mark as read when done. " + "Route all output to Gitea comments or Mattermost #git/#claw. " + "Do NOT reply to this session — respond with NO_REPLY." + ) + payload = json.dumps({"text": text, "mode": "now"}).encode() + req = urllib.request.Request( + f"{GATEWAY_URL}/hooks/wake", + data=payload, + headers={ + "Authorization": f"Bearer {HOOK_TOKEN}", + "Content-Type": "application/json", + }, + method="POST", + ) + try: + with urllib.request.urlopen(req, timeout=5) as resp: + status = resp.status + print(f" Wake responded: {status}", flush=True) + return True + except Exception as e: + print( + f"WARN: Failed to wake OpenClaw: {e}", + file=sys.stderr, + flush=True, + ) + return False + + +def main(): + check_config() + print( + f"Gitea notification poller started (delay={POLL_DELAY}s)", + flush=True, + ) + + last_seen_ids = gitea_unread_ids() + print( + f"Initial unread: {len(last_seen_ids)} notification(s)", flush=True + ) + + while True: + time.sleep(POLL_DELAY) + + current_ids = gitea_unread_ids() + new_ids = current_ids - last_seen_ids + + if not new_ids: + last_seen_ids = current_ids + continue + + ts = time.strftime("%H:%M:%S") + print( + f"[{ts}] {len(new_ids)} new notification(s) " + f"({len(current_ids)} total unread), waking agent", + flush=True, + ) + + wake_openclaw(len(new_ids)) + last_seen_ids = current_ids + + +if __name__ == "__main__": + main() +``` + +Run it as a background service (launchd on macOS, systemd on Linux) with the +env vars set. It's intentionally simple — no frameworks, no async, no +dependencies. + +### The Daily Diary + +Every day gets a `memory/YYYY-MM-DD.md` file. The agent appends a topic +summary to a table after every meaningful conversation: + +```markdown +## Topics + +| Time (TZ) | Topic | +| ---------- | ------------------------------------------------- | +| 14:30 | Discussed PR review workflow | +| 16:00 | Medication logged | +| 18:45 | Flight prep blocks created for tomorrow's flight | +``` + +This serves multiple purposes: +- Any session can see what's been discussed today without loading full + conversation history +- The agent can use `sessions_history` to get details on a specific topic + if needed +- During memory maintenance, the agent reviews these to decide what's worth + promoting to MEMORY.md +- It's a simple audit trail of what happened + +### The Requirement Capture Rule + +One of the most important rules: **every single requirement, preference, or +instruction the human provides MUST be captured in a file immediately.** Not +"mentally noted" — written to disk. Because the agent wakes up fresh every +session, a "mental note" is worthless. If it's not in a file, it didn't +happen. + +This applies to everything: project rules ("no mocks in tests"), workflow +preferences ("fewer PRs, don't over-split"), corrections, new policies. +Immediate write to the daily file, and to MEMORY.md if it's a standing rule. + +### PII-Aware Output Routing + +A lesson learned the hard way: **the audience determines what you can say, +not who asked.** If the human asks for a medication status report in a group +channel, the agent can't just dump it there — other people can read it. The +rule: if the output would contain PII and the channel isn't private, redirect +to DM and reply in-channel with "sent privately." + +This is enforced at multiple levels: +- AGENTS.md has a warning banner at the top +- The checklist system catches it before action +- Channel-specific rule files (like our `memory/JAMES_CHAT.md`) define what's + shareable per-channel + +### Sub-Agent Isolation + +When spawning coding sub-agents for PR work, each one MUST clone to a fresh +temporary directory. Never share git clones between agents — dirty working +directories cause false CI results, merge conflicts, and wrong file state. +The rule: `cd $(mktemp -d) && git clone . && ...` + +### The Heartbeat System + +OpenClaw polls the agent periodically with a configurable prompt. The agent +reads HEARTBEAT.md and decides what to do. We keep HEARTBEAT.md small +(focused checklist) to minimize token burn on these frequent checks. + +The heartbeat handles: +- Gitea inbox triage (check for new assignments) +- Flight prep block creation (look ahead 7 days) +- Open project review (can I take a step on anything?) +- Workspace sync (commit and push changes) +- Periodic memory maintenance + +State tracking in `memory/heartbeat-state.json` prevents redundant checks +(e.g., don't re-check email if you checked 10 minutes ago). + +The key output rule: heartbeats should either be `HEARTBEAT_OK` (nothing to +do) or a direct alert. Work narration goes to a designated status channel, +never to the human's DM. + +### Putting It All Together + +The system works as a loop: + +1. **Session starts** → read SOUL, USER, daily-context, today's daily file +2. **Message arrives** → check daily-context for state changes (sleep gap? + location change? meds overdue?), respond to the message, update state +3. **Heartbeat fires** → check inbox, projects, flights, sync workspace +4. **External event** (Gitea poller wake) → process notification, respond + via appropriate channel +5. **Session ends** → state persists in files for next session + +The files are the continuity. The agent is stateless; the workspace is not. + +--- + ## Table of Contents 1. [Workspace Bootstrapping](#workspace-bootstrapping)