chloejs/scorers

export function calls(result: Result, expected: Expected): Mark

Marks a run on the tools it used, which is arithmetic and asks nobody.

export async function expectations(
  prompt: string,
  result: Result,
  expected: Expected,
  model: string,
): Promise<Mark>

Marks what a run said against what it should have said, by asking a model to judge it.

export interface Expected {
  mustCall?: string[];
  mustNotCall?: string[];
  /** Tools it may use, but only once in a run. Restarting twice is why this exists. */
  atMostOnce?: string[];
}

Which tools a run should have used, must not have used, and may use only once.

export interface Expected {
  situation?: string;
  should?: string[];
  shouldNot?: string[];
}

The situation a case sets up, and what the agent should and should not have done about it.

export interface Mark {
  score: number;
  reason: string;
}

A score between zero and one, and why it came out that way.