Caty Agent Harness
<div align="center"
🇺🇸 English ｜ 🇯🇵 日本語 ｜ 🇨🇳 简体中文 ｜ 🇹🇭 ไทย
Notice (2026-08-24): We found a gap between this README and the shipped behaviour (#144): part of the learning loop described below — "the method is saved only as a lesson at first, and becomes a rule after passing verification again on a different job. Procedures that keep coming up are reviewed by a different AI than the one that wrote them; only the ones that pass are stored as skills" — is designed but not yet implemented. We are implementing it now (#147, #148, #149) and will adjust and republish these docs once it ships. Tracking: #146.
<subCI evidence (2026-08-15 UTC): matrix 7/7 green ・ 60 independent runs of one SHA, 0 flakes. Weekly schedule — GitHub pauses schedules after 60 days of repo inactivity, so mind the run date.</sub <br<sub WSL2 is a verified-with-conditions tier on one dated VM, not a CI-tested tier; see the WSL2 support note.</sub
Re-explaining everything. Context that vanishes. A cheerful "done!" with nothing to show for it.<br Caty Agent Harness fixes those — with plain text files and real checks.<br Not magic. The machinery remembers, drives, and checks, so the AI can focus on thinking —<br a small system, wrapped around the AI you already use.
A system that doesn't take the AI's "done!" at face value — it runs work in stages, verifies "done" against evidence, and publishes its own scorecard, losses included.
What actually gets better — measured, not promised:
- Fake "done" stops on a weak model — the verified completion rate triples. On Claude Haiku 4.5 (a lower-capability model), "done" claims with the work left unread fell from 98% to 8% of completion claims (222/226 → 2/26; 30 runs/arm, M/L-size jobs) — and verified completions rose from 13% to 43% (full numbers).
- Strong models keep their accuracy — and cut tokens on big jobs by 58% (sonnet) / 31% (opus). sonnet and opus scored the same on the hidden answer key with and without the harness — on large jobs, running without it used 2.40× (sonnet) / 1.46× (opus) as many tokens (= 58% / 31% fewer with the harness; L-band medians — the P1 table). On sonnet's smaller jobs in these tests the harness used more, not fewer — use it where the work is big. And on the largest measured jobs — ≈2.4M tokens, about 2.4× the context window — where opus without the harness read 2–4% of the files and kept submitting, opus under the harness finished verified with 100% of the files read, 2/2 in both overflow bands (the two arms ran under different budget envelopes).
- The results show its limits too. On search-style runtimes (e.g. Codex) the overflow guard never fired — by design, there was nothing to rescue — and on one runtime it fired but used more tokens than running without it (per-model profiles). sonnet's overflow cells stayed 0/2: all four runs delivered, 20/20-correct with valid quotes, while skipping ~10–12% of the files — and the verification gate refused "done" each time, listing the unread files (the pre-registered honest-failure branch). Every losing result is published next to the wins.
<subFour sealed, pre-registered, machine-scored experiment families (2026-08) — every number, method, and caveat, including where the harness costs extra tokens: benchmark ・ Try it in 2 minutes — one pasted prompt into the AI tool you already use.</sub
🔧 Engineering guide ｜ 📘 Reference
</div <!-- repo-state:begin (generated; do not edit) -- <p align="center"<subgeneration: <codeed18d59</code (2026-09-18T11:26:53Z) · verify: <a href="https://api.github.com/repos/caty-ai/caty-agent-harness/commits/main"API HEAD</a · <a href="./status.json"status.json</a</sub</p <!-- repo-state:end --
- Does this sound familiar?
- What you get
- What you need
- Get started
- Why it's safe to try
- Which models benefit
- Dig deeper
- Part of Family OS
- License
---
<a id="problems"</a
Does this sound familiar?
The more work you hand to an AI, the more often these moments show up.