@chloejs/core/scorers

export function calls(result: Result, expected: Expected): Mark

Scores a run on the tools it called, against expected. It only counts the calls and asks no model, so it is free and gives the same answer every time.

The score is 1 if every rule is kept, and 0 if any rule is broken. The reason lists what went wrong and every tool the run used.

export async function expectations(
  prompt: string,
  result: Result,
  expected: Expected,
  model: string,
  purpose?: string,
): Promise<Mark>

Scores a run against expected by asking a model to judge it. Each line in should and shouldNot is checked one by one, and the score is the share of lines that were met (from 0 to 1).

• prompt: what the agent was asked.
• result: the run, with its reply and its tool calls.
• model: the model that judges, such as "anthropic/claude-haiku-4.5".
• purpose: the agent's own purpose. Optional. Give them when an expectation depends on them, such as "invents nothing the purpose do not say".

Each call costs one model call, or two if the judge's first answer is not valid JSON. With no lines in should or shouldNot, it returns 1 and asks no model.

export interface Expected {
  /** Tools the run must call at least once. */
  mustCall?: string[];
  /** Tools the run must never call. */
  mustNotCall?: string[];
  /** Tools the run may call, but no more than once. Useful for a tool that should not run twice, such as a restart. */
  atMostOnce?: string[];
}

The tools a run should use, must not use, and may use only once. Each list holds tool names, as the model calls them. Imported as ExpectedCalls.

export interface Expected {
  /** What was true at the time, in words, such as "The site has been down for an hour." Shown to the judging model. */
  situation?: string;
  /** Things the agent should have said or done, one per line, such as "says which site is down". */
  should?: string[];
  /** Things the agent should not have said or done, one per line. */
  shouldNot?: string[];
}

What was true when the run happened, and what the agent should and should not have done about it. Imported as ExpectedOutcome.

export interface Mark {
  /** From 0 (failed) to 1 (passed). */
  score: number;
  /** Why the run got this score, in words. */
  reason: string;
}

A score from 0 to 1, and the reason for it.