caty-ai/caty-agent-harness

Caty Agent Harness

<div align="center"

🇺🇸 English ｜ 🇯🇵 日本語 ｜ 🇨🇳 简体中文 ｜ 🇹🇭 ไทย

Notice (2026-08-24): We found a gap between this README and the shipped behaviour (#144): part of the learning loop described below — "the method is saved only as a lesson at first, and becomes a rule after passing verification again on a different job. Procedures that keep coming up are reviewed by a different AI than the one that wrote them; only the ones that pass are stored as skills" — is designed but not yet implemented. We are implementing it now (#147, #148, #149) and will adjust and republish these docs once it ships. Tracking: #146.

<subCI evidence (2026-08-15 UTC): matrix 7/7 green ・ 60 independent runs of one SHA, 0 flakes. Weekly schedule — GitHub pauses schedules after 60 days of repo inactivity, so mind the run date.</sub <br<sub WSL2 is a verified-with-conditions tier on one dated VM, not a CI-tested tier; see the WSL2 support note.</sub

Re-explaining everything. Context that vanishes. A cheerful "done!" with nothing to show for it.<br Caty Agent Harness fixes those — with plain text files and real checks.<br Not magic. The machinery remembers, drives, and checks, so the AI can focus on thinking —<br a small system, wrapped around the AI you already use.

A system that doesn't take the AI's "done!" at face value — it runs work in stages, verifies "done" against evidence, and publishes its own scorecard, losses included.

What actually gets better — measured, not promised:

• Fake "done" stops on a weak model — the verified completion rate triples. On Claude Haiku 4.5 (a lower-capability model), "done" claims with the work left unread fell from 98% to 8% of completion claims (222/226 → 2/26; 30 runs/arm, M/L-size jobs) — and verified completions rose from 13% to 43% (full numbers).
• Strong models keep their accuracy — and cut tokens on big jobs by 58% (sonnet) / 31% (opus). sonnet and opus scored the same on the hidden answer key with and without the harness — on large jobs, running without it used 2.40× (sonnet) / 1.46× (opus) as many tokens (= 58% / 31% fewer with the harness; L-band medians — the P1 table). On sonnet's smaller jobs in these tests the harness used more, not fewer — use it where the work is big. And on the largest measured jobs — ≈2.4M tokens, about 2.4× the context window — where opus without the harness read 2–4% of the files and kept submitting, opus under the harness finished verified with 100% of the files read, 2/2 in both overflow bands (the two arms ran under different budget envelopes).
• The results show its limits too. On search-style runtimes (e.g. Codex) the overflow guard never fired — by design, there was nothing to rescue — and on one runtime it fired but used more tokens than running without it (per-model profiles). sonnet's overflow cells stayed 0/2: all four runs delivered, 20/20-correct with valid quotes, while skipping ~10–12% of the files — and the verification gate refused "done" each time, listing the unread files (the pre-registered honest-failure branch). Every losing result is published next to the wins.

<subFour sealed, pre-registered, machine-scored experiment families (2026-08) — every number, method, and caveat, including where the harness costs extra tokens: benchmark ・ Try it in 2 minutes — one pasted prompt into the AI tool you already use.</sub

🔧 Engineering guide ｜ 📘 Reference

</div <!-- repo-state:begin (generated; do not edit) -- <p align="center"<subgeneration: <codeed18d59</code (2026-09-18T11:26:53Z) · verify: <a href="https://api.github.com/repos/caty-ai/caty-agent-harness/commits/main"API HEAD</a · <a href="./status.json"status.json</a</sub</p <!-- repo-state:end --

• Does this sound familiar?
• What you get
• What you need
• Get started
• Why it's safe to try
• Which models benefit

• Dig deeper
• Part of Family OS
• License

---

<a id="problems"</a

Does this sound familiar?

The more work you hand to an AI, the more often these moments show up.