Testing and evals

	Checks	Costs
A test	that a job's code does the thing	nothing: no model is asked
An eval	that a prompt led the model to decide well	a model call per case

Jobs are tested

A test sits beside its job, named <job>.test.ts, with cases from @chloejs/core/test. Test the rules, which is the part that is code:

// The one rule in this job, checked without a database and without a model.
import { about, is } from "@chloejs/core/test";

import { stuck, GRACE_HOURS } from "./stuck-orders.ts";
import type { Order } from "../services/ordersService.ts";

about("which orders are worth chasing");

const now = Date.parse("2026-09-20T12:00:00Z");
const hours = (n: number) => new Date(now + n * 60 * 60 * 1000).toISOString();
const order = (extra: Partial<Order>): Order => ({
  id: "A-1", customer: "c-1", placed: hours(-200), paid: true, promised: hours(-48), total: 1000, lines: [], ...extra,
});

is("paid, unshipped and two days late is stuck", stuck(order({}), now), true);
is("the same order once it ships is not", stuck(order({ shipped: hours(-1) }), now), false);
is("an unpaid one is not ours to chase", stuck(order({ paid: false }), now), false);
is("one still inside the grace period is not yet late", stuck(order({ promised: hours(-GRACE_HOURS + 1) }), now), false);
is("and one due tomorrow is not late at all", stuck(order({ promised: hours(24) }), now), false);

node agents/<id>/jobs/<job>.test.ts runs one and prints each case. The more of a job that moves from a model step into code, the more of it a test covers.

Prompts are scored

npx chloe evals <agent>

An eval file in the agent's evals/ folder is one job, with a case per situation: what each tool answers, and what the agent should and should not have done. Run it before and after you change a purpose or a skill, because words are the easiest thing to make worse without noticing.

Nothing in a case runs for real. A tool the case does not answer is refused, so an eval cannot send an email or ship an order. Add a tool to the agent and its cases fail until they answer it.

A tool's answer matches only the exact arguments it lists. Mark one "anyArgs": true when the arguments do not change it, and give "times" when one answer covers several calls.