Behavior Evaluation

When deterministic checks pass but behavior is still uncertain, the user needs a bounded way to compare observed intentful behavior. Using the cautilus evaluate CLI command and the cautilus-agent skill, a user can evaluate behavior across dev/repo, dev/skill, app/chat, and app/prompt surfaces without turning the host repo's runners, prompts, or policy into Cautilus-owned state. Both dev coding-agent surfaces (dev/repo routing and dev/skill orientation) are proven live on demand, with the dev-surface judges load-bearing as regression guards: the an always-sound judge FAILS every decomposed claim invariant (scripts/agent-runtime/reasoning-soundness-judge.test.mjs) fails if a judge stops discriminating, and the same composite catches a deliberately-worse routing variant across three distinct pinned behaviors. A population of natural unsound cases is impractical to harvest on these single-turn surfaces — ~44 real harvested responses across two tiers produced zero natural semantic unsound, a property of current capable models rather than a weakness of the judge — so the maintainer accepted constructed-control reject-capability plus the natural-sound harvest as the proven standard for the dev surfaces (2026-06-19), keeping the natural-population bar as a known limitation. That harvest is positive context whose raw responses live in session transcripts (prompts and objective truths pre-registered in the frontier evidence); the load-bearing gate here is the checked-in always-sound-judge-fails invariant plus the two on-demand live proofs, not the harvest count. The app/chat surface is now evaluated on an anonymized private external product-log replay graded by a load-bearing blind intent judge (external validity and the intent judge, with the agent run replayed from the production log), and it includes natural sound secret-handling, memory-continuity, and clarification-first captures plus a natural unsound artifact-fidelity capture. app/prompt now has a fresh backend probe plus a load-bearing blind intent judge over that probe, but it still needs product-runner proof before it can be treated like product-path app evidence.

Governed by governed-by::Reviewable Artifacts, governed-by::Packet Freshness, governed-by::Cost And Proof Freshness, and governed-by::Host-Owned Execution. Implemented by implemented-by::Evaluation Surfaces And Runners, implemented-by::Reporting And Review Variants, and implemented-by::Live Invocation Runtime.