add system overview: how and why the pieces fit together

OPENCLAW_TRICKS.md

@@ -6,6 +6,425 @@ production setup that's been running since early 2026.
 
 ---
 
+## How This System Evolved
+
+This section explains what we built, why, and how the pieces fit together. The
+rest of the document has the specific prompts and schemas — this part gives you
+the big picture so you understand the design decisions.
+
+### The Starting Point
+
+OpenClaw gives you a persistent workspace directory and the ability to run
+tools (shell, web, browser, messaging). Out of the box, the agent wakes up
+fresh every session with no memory of what happened before. The core challenge
+is: **how do you turn a stateless LLM into a stateful personal assistant that
+knows who you are, where you are, what you need, and what it was working on?**
+
+The answer is files. The workspace IS the agent's brain. Every piece of state,
+every rule, every memory lives in files that the agent reads on startup and
+updates as things change.
+
+### The Workspace Files — Separation of Concerns
+
+We started with one big AGENTS.md that had everything: rules, procedures,
+medication instructions, git workflow, travel preferences. It worked at first
+but quickly became unwieldy. The context window was getting eaten by
+instructions the agent didn't need for most tasks.
+
+The solution was **factoring out rulesets into focused files:**
+
+- **AGENTS.md** — the master file. Responsibilities, session startup
+  procedure, high-level rules. Think of it as the table of contents that
+  points to everything else.
+- **SOUL.md** — personality, tone, values. Separated because it's
+  philosophical, not procedural. Also: it's fun to let the agent evolve this
+  one over time.
+- **USER.md** — info about the human. Timezone, communication preferences,
+  basics the agent needs every session.
+- **MEMORY.md** — curated long-term memory. Only loaded in private sessions
+  (security — prevents leaking personal context in group chats).
+- **HEARTBEAT.md** — what to check on periodic heartbeat polls. Kept small
+  intentionally to minimize token burn on frequent checks.
+- **TOOLS.md** — environment-specific notes (hostnames, device names, channel
+  IDs). Separated from skills because this is YOUR setup, skills are shared.
+
+Then the procedural rulesets got their own files too:
+
+- **`memory/medications-instructions.md`** — full medication protocol. Dosages,
+  schedules, safety rules, overdose prevention logic. Only loaded when
+  medication tasks come up.
+- **`memory/checklist-pr.md`** — PR quality gate. What to verify before pushing,
+  after sub-agent pushes, before assigning to the human for review.
+- **`memory/checklist-flights.md`** — flight time verification, prep block
+  creation, landing checklist triggers.
+- **`memory/checklist-medications.md`** — pre-action verification before
+  reporting medication status or sending reminders.
+- **`memory/checklist-messaging.md`** — rules for every outgoing message: URL
+  verification, timezone conversion, status claim verification.
+- **`memory/landing-checklist.md`** — post-flight procedures (update location,
+  timezone, check meds, sync calendar).
+
+The key insight: **MEMORY.md has a "checklists" section at the very top** that
+says "before doing X, read file Y." The agent reads MEMORY.md at session start,
+sees the checklist index, and knows which file to consult before any action.
+This way the detailed instructions aren't always in context — they're loaded
+on-demand.
+
+```markdown
+## ⛔ CHECKLISTS (read the relevant file before acting)
+
+- **Any PR/code work** → `memory/checklist-pr.md`
+- **Any message to the user** → `memory/checklist-messaging.md`
+- **Medications** → `memory/checklist-medications.md`
+- **Flights/travel** → `memory/checklist-flights.md`
+```
+
+### The Daily Context State File
+
+This was probably the single most impactful addition. It's a JSON file
+(`memory/daily-context.json`) that every session reads on every message. It
+tracks the current state of the human: where they are, what timezone they're
+in, whether they're sleeping, when they last took meds, when they last sent a
+message.
+
+Why JSON instead of markdown? Because it's machine-readable. The agent can
+update individual fields programmatically, and the structure is unambiguous.
+Markdown is great for instructions; JSON is great for state.
+
+The daily context file means:
+- The agent always knows the human's timezone (critical for a frequent
+  traveler)
+- Medication reminders fire based on actual timestamps, not guesses
+- Sleep predictions inform when to send alerts vs stay quiet
+- Location tracking happens automatically
+
+### The Two-Tier Memory System
+
+Daily files (`memory/YYYY-MM-DD.md`) are raw logs — what happened each day.
+Topic tables, decisions, lessons, notes. Think of these as a daily journal.
+
+MEMORY.md is curated long-term memory — the distilled essence. Standing rules,
+key facts, lessons learned. Think of this as what a human would "just know"
+about their life.
+
+During heartbeats, the agent periodically reviews recent daily files and
+promotes significant items to MEMORY.md, and removes stale info. It's
+explicitly modeled on how human memory works: raw experience gets processed
+into lasting knowledge, and irrelevant details fade.
+
+The security model: MEMORY.md is only loaded in private (1:1 DM) sessions.
+In group chats, the agent works from daily files and daily-context.json only.
+This prevents personal context from leaking into shared channels.
+
+### Medication Tracking
+
+This is a safety-critical system. The design is deliberately paranoid:
+
+- **CSV as source of truth.** Not the daily-context boolean, not the agent's
+  memory — the CSV log file is authoritative.
+- **Double-verification before any action.** The daily context has a
+  `hasTakenDailyMedsToday` boolean AND a `dailyMedsTimestamp`. A midnight
+  cron resets the boolean. But if the human is in a timezone ahead of the
+  server, the reset happens at the wrong time. So the rule is: always verify
+  the timestamp falls on today's date in the human's timezone. The boolean
+  is a convenience hint, never the source of truth.
+- **Interval medications anchored to actual doses.** Some meds are "every 5
+  days" or "every 14 days." The next dose date is calculated from the last
+  actual dose timestamp, not from the intended schedule. If a dose is missed,
+  the next date is unknown until the missed dose is taken and logged.
+- **Overdose prevention.** The agent blocks logging if it detects a same-day
+  duplicate batch. This is the highest-priority safety rule — a duplicate
+  daily batch of certain medications could cause cardiac arrest.
+- **Escalating alerts.** If daily meds are >26h since last dose, the agent
+  escalates aggressively — ntfy push notification if chat messages go
+  unacknowledged.
+
+The medication instructions file is loaded on-demand (not every session),
+keeping context costs low for non-medication conversations.
+
+### Sleep Tracking
+
+The agent infers sleep from activity gaps rather than requiring explicit
+"I'm going to sleep" statements. On every message, it checks: was there a
+gap since the last message that overlaps with the predicted sleep window? If
+so, log sleep start = last activity before gap, wake = first activity after.
+
+Activity isn't just chat messages — it includes Gitea commits, comments,
+scheduled flight departures, and any other observable actions.
+
+Sleep predictions drift over time (the human in this setup tends to sleep
+~30min later each day), so the agent tracks the trend and extrapolates.
+Caffeine intake adjusts predictions forward. The predictions feed into:
+- When to send alerts vs stay quiet
+- Sitrep "sleep conflict" warnings (appointment during predicted sleep)
+- General awareness of the human's schedule
+
+### Location & Timezone Tracking
+
+The agent always knows where the human is. This matters because:
+- Times must always be displayed in the human's local timezone, not the
+  server's timezone
+- "Today" and "tomorrow" mean different things in different timezones
+- Medication timing needs to account for timezone changes
+- Flight prep needs local airport info
+
+The landing checklist (triggered automatically after every flight) updates
+location, timezone, nearest airport, and lodging in the daily context file.
+It also checks if any cron jobs have hardcoded timezones that need updating.
+
+### The Gitea Notification Poller
+
+OpenClaw has heartbeats, but those are periodic (every ~30min). For Gitea
+issues and PRs, we wanted near-realtime response. The solution: a tiny Python
+script that polls the Gitea notifications API every 2 seconds and wakes the
+agent via OpenClaw's `/hooks/wake` endpoint when new notifications arrive.
+
+Key design decisions:
+- **The poller never marks notifications as read.** That's the agent's job
+  after it processes them. This prevents the poller and agent from racing.
+- **It tracks notification IDs, not counts.** This way it only fires on
+  genuinely new notifications, not re-reads of existing ones.
+- **The wake message tells the agent to route output to Gitea/Mattermost,
+  not to DM.** This prevents chatty notification processing from disturbing
+  the human.
+- **Zero dependencies.** Just Python stdlib (`urllib`, `json`, `time`). Runs
+  anywhere.
+
+Here's the full source:
+
+```python
+#!/usr/bin/env python3
+"""
+Gitea notification poller.
+Polls for unread notifications and wakes OpenClaw when the count
+changes. The AGENT marks notifications as read after processing —
+the poller never marks anything as read.
+
+Required env vars:
+  GITEA_URL        - Gitea instance URL
+  GITEA_TOKEN      - Gitea API token
+  HOOK_TOKEN       - OpenClaw hooks auth token
+
+Optional env vars:
+  GATEWAY_URL      - OpenClaw gateway URL (default: http://127.0.0.1:18789)
+  POLL_DELAY       - Delay between polls in seconds (default: 2)
+"""
+
+import json
+import os
+import sys
+import time
+import urllib.request
+import urllib.error
+
+GITEA_URL = os.environ.get("GITEA_URL", "").rstrip("/")
+GITEA_TOKEN = os.environ.get("GITEA_TOKEN", "")
+GATEWAY_URL = os.environ.get("GATEWAY_URL", "http://127.0.0.1:18789").rstrip(
+    "/"
+)
+HOOK_TOKEN = os.environ.get("HOOK_TOKEN", "")
+POLL_DELAY = int(os.environ.get("POLL_DELAY", "2"))
+
+
+def check_config():
+    missing = []
+    if not GITEA_URL:
+        missing.append("GITEA_URL")
+    if not GITEA_TOKEN:
+        missing.append("GITEA_TOKEN")
+    if not HOOK_TOKEN:
+        missing.append("HOOK_TOKEN")
+    if missing:
+        print(
+            f"ERROR: Missing required env vars: {', '.join(missing)}",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+
+def gitea_unread_ids():
+    """Return set of unread notification IDs."""
+    req = urllib.request.Request(
+        f"{GITEA_URL}/api/v1/notifications?status-types=unread",
+        headers={"Authorization": f"token {GITEA_TOKEN}"},
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=10) as resp:
+            notifs = json.loads(resp.read())
+            return {n["id"] for n in notifs}
+    except Exception as e:
+        print(
+            f"WARN: Gitea API failed: {e}", file=sys.stderr, flush=True
+        )
+        return set()
+
+
+def wake_openclaw(count):
+    text = (
+        f"[Gitea Notification] {count} new notification(s). "
+        "Check your Gitea notification inbox via API, process them, "
+        "and mark as read when done. "
+        "Route all output to Gitea comments or Mattermost #git/#claw. "
+        "Do NOT reply to this session — respond with NO_REPLY."
+    )
+    payload = json.dumps({"text": text, "mode": "now"}).encode()
+    req = urllib.request.Request(
+        f"{GATEWAY_URL}/hooks/wake",
+        data=payload,
+        headers={
+            "Authorization": f"Bearer {HOOK_TOKEN}",
+            "Content-Type": "application/json",
+        },
+        method="POST",
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=5) as resp:
+            status = resp.status
+        print(f"  Wake responded: {status}", flush=True)
+        return True
+    except Exception as e:
+        print(
+            f"WARN: Failed to wake OpenClaw: {e}",
+            file=sys.stderr,
+            flush=True,
+        )
+        return False
+
+
+def main():
+    check_config()
+    print(
+        f"Gitea notification poller started (delay={POLL_DELAY}s)",
+        flush=True,
+    )
+
+    last_seen_ids = gitea_unread_ids()
+    print(
+        f"Initial unread: {len(last_seen_ids)} notification(s)", flush=True
+    )
+
+    while True:
+        time.sleep(POLL_DELAY)
+
+        current_ids = gitea_unread_ids()
+        new_ids = current_ids - last_seen_ids
+
+        if not new_ids:
+            last_seen_ids = current_ids
+            continue
+
+        ts = time.strftime("%H:%M:%S")
+        print(
+            f"[{ts}] {len(new_ids)} new notification(s) "
+            f"({len(current_ids)} total unread), waking agent",
+            flush=True,
+        )
+
+        wake_openclaw(len(new_ids))
+        last_seen_ids = current_ids
+
+
+if __name__ == "__main__":
+    main()
+```
+
+Run it as a background service (launchd on macOS, systemd on Linux) with the
+env vars set. It's intentionally simple — no frameworks, no async, no
+dependencies.
+
+### The Daily Diary
+
+Every day gets a `memory/YYYY-MM-DD.md` file. The agent appends a topic
+summary to a table after every meaningful conversation:
+
+```markdown
+## Topics
+
+| Time (TZ) | Topic                                              |
+| ---------- | ------------------------------------------------- |
+| 14:30      | Discussed PR review workflow                      |
+| 16:00      | Medication logged                                 |
+| 18:45      | Flight prep blocks created for tomorrow's flight  |
+```
+
+This serves multiple purposes:
+- Any session can see what's been discussed today without loading full
+  conversation history
+- The agent can use `sessions_history` to get details on a specific topic
+  if needed
+- During memory maintenance, the agent reviews these to decide what's worth
+  promoting to MEMORY.md
+- It's a simple audit trail of what happened
+
+### The Requirement Capture Rule
+
+One of the most important rules: **every single requirement, preference, or
+instruction the human provides MUST be captured in a file immediately.** Not
+"mentally noted" — written to disk. Because the agent wakes up fresh every
+session, a "mental note" is worthless. If it's not in a file, it didn't
+happen.
+
+This applies to everything: project rules ("no mocks in tests"), workflow
+preferences ("fewer PRs, don't over-split"), corrections, new policies.
+Immediate write to the daily file, and to MEMORY.md if it's a standing rule.
+
+### PII-Aware Output Routing
+
+A lesson learned the hard way: **the audience determines what you can say,
+not who asked.** If the human asks for a medication status report in a group
+channel, the agent can't just dump it there — other people can read it. The
+rule: if the output would contain PII and the channel isn't private, redirect
+to DM and reply in-channel with "sent privately."
+
+This is enforced at multiple levels:
+- AGENTS.md has a warning banner at the top
+- The checklist system catches it before action
+- Channel-specific rule files (like our `memory/JAMES_CHAT.md`) define what's
+  shareable per-channel
+
+### Sub-Agent Isolation
+
+When spawning coding sub-agents for PR work, each one MUST clone to a fresh
+temporary directory. Never share git clones between agents — dirty working
+directories cause false CI results, merge conflicts, and wrong file state.
+The rule: `cd $(mktemp -d) && git clone <url> . && ...`
+
+### The Heartbeat System
+
+OpenClaw polls the agent periodically with a configurable prompt. The agent
+reads HEARTBEAT.md and decides what to do. We keep HEARTBEAT.md small
+(focused checklist) to minimize token burn on these frequent checks.
+
+The heartbeat handles:
+- Gitea inbox triage (check for new assignments)
+- Flight prep block creation (look ahead 7 days)
+- Open project review (can I take a step on anything?)
+- Workspace sync (commit and push changes)
+- Periodic memory maintenance
+
+State tracking in `memory/heartbeat-state.json` prevents redundant checks
+(e.g., don't re-check email if you checked 10 minutes ago).
+
+The key output rule: heartbeats should either be `HEARTBEAT_OK` (nothing to
+do) or a direct alert. Work narration goes to a designated status channel,
+never to the human's DM.
+
+### Putting It All Together
+
+The system works as a loop:
+
+1. **Session starts** → read SOUL, USER, daily-context, today's daily file
+2. **Message arrives** → check daily-context for state changes (sleep gap?
+   location change? meds overdue?), respond to the message, update state
+3. **Heartbeat fires** → check inbox, projects, flights, sync workspace
+4. **External event** (Gitea poller wake) → process notification, respond
+   via appropriate channel
+5. **Session ends** → state persists in files for next session
+
+The files are the continuity. The agent is stateless; the workspace is not.
+
+---
+
 ## Table of Contents
 
 1. [Workspace Bootstrapping](#workspace-bootstrapping)