Isomer evals

@elastic/isomer-evals scores whether a model produces valid, well-chosen compositions from a pack's authoring context. On Node it needs 22.13.0 or later, like the SDK it depends on.

Conformance answers "does this primitive render?" This answers the other half: "does an agent handed the catalog reach for the right one?" Catalog useWhen / avoidWhen copy is the tunable, and without a measurement it is unfalsifiable.

The library holds no corpus, no goldens, no credentials, and no model client; the recorded corpus under corpus/ is a test fixture, not part of the build. It reads nothing from the environment and writes nothing to disk. Everything arrives as an argument:

import { formatReport, runEvals } from '@elastic/isomer-evals';

const report = await runEvals({
  runtime,
  corpus,
  generate: async ({ prompt, evalCase, previousErrors }) =>
    callYourModel(
      [
        prompt,
        `## Request\n${evalCase.prompt}`,
        ...(previousErrors
          ? [`## Previous errors\n${previousErrors.join('\n')}`]
          : []),
      ].join('\n\n')
    ),
  concurrency: 4,
});

console.log(formatReport(report));
		

1. your pack's assembled runtime
2. your prompts, with optional goldens
3. cases in flight at once; defaults to 1

prompt contains shared authoring instructions, not the case request or retry feedback. Send evalCase.prompt on every attempt and previousErrors when present.

generate is the only way a model is ever reached. Pass one that replays recorded output and the run is deterministic — which is how this belongs in CI.

runtime is an EvalRuntime, the slice of a runtime the harness reads (primitives, getAuthoringContext, parse, validate, and the text surface), declared structurally so a pack can pass a stub; a runtime from createIsomerRuntime satisfies it, which run.test.ts proves against the runtime's own type.

The prompt is one buildAuthoringPrompt call over the runtime's authoring context, with no host examples: each catalog entry already shows its own shape.

Three of the four need no model at all. A pack can run validity, selection, and payload with no credentials and no flakiness; answerability is opt-in and skipped when judge is omitted.

A failed attempt is handed its own validation errors and tried once more. The gap between valid and validAfterRetry measures how actionable the validator's messages are — a property of Isomer, not of the model. A wide gap is a bug report about error copy.

Answering a three-stat question with one statGroup should score differently from answering it with three separate nodes. A set would score both 1.0.

Selection counts every node, reached through each primitive's child hook the same way scoreExpectedPrimitives and payload.unknownTypes reach them, so a data field named type is not a node. A pack whose frame wraps every answer in one container, as the slides pack wraps each in a slideFrame, puts the choice that matters below the top level; counting only body[].type would score whether the model used a frame. The wrapper still counts once on each side, so two answers that share only it score above zero.

formatReport prints EvalReport.totals; results[] carries the per-case scores behind each line.

Per case, payload.rawBytes is the model's output as sent and payload.sanitizedBytes is the copy validate checked, 0 when the first attempt did not validate. payload.unknownTypes lists node types the catalog does not have — any entry is either a catalog gap or the model reaching past it. Payload scores describe the first attempt, including when a retry succeeds. If a malformed container makes its child hook throw, its descendants cannot be inspected; scoring continues with the other branches, so an empty unknownTypes list does not prove the payload is valid.

The package ships none. What a pack should score depends on its catalog, its corpus, and the model under test, so record a baseline run first and gate on that. One expect per axis, with floors a pack might set after a baseline:

const { totals, results } = await runEvals({ runtime, corpus, generate, judge });

expect(totals.parsed).toBe(totals.cases);
expect(totals.valid / totals.cases).toBeGreaterThanOrEqual(0.9);
expect(totals.validAfterRetry).toBe(totals.cases);
expect(totals.meanSelectionF1).toBeGreaterThanOrEqual(0.8);
for (const { payload } of results) {
  expect(payload.unknownTypes).toEqual([]);
  expect(payload.sizeVsGolden ?? 1).toBeLessThanOrEqual(1.5);
}
expect(totals.answerability?.no ?? 0).toBe(0);
		

The values are illustrative, not recommended: 0.8 F1 tolerates one substituted primitive in a five-node body, and 1.5 tolerates half again the golden's bytes. A pack replaces them with what its own baseline supports.

An EvalCase is an id, a prompt, and optionally the composition a good answer resembles (golden) and the node types any one of which a good answer uses (expected). Both harnesses take the same EvalCase. The node types below are illustrative; a corpus uses the types of the pack under test:

const corpus = [
  {
    id: 'queue-health',
    prompt: 'How many items are open right now?',
    golden: { type: 'view', title: 'Queue health', body: [{ type: 'stat', … }] },
    expected: ['stat', 'statGroup'],
  },
];
		

A case without a golden still scores validity, payload, and answerability — only selection F1 needs one, and only the expected-primitive hit needs expected. Selection and answerability run only when the first attempt or retry validates; a case that remains invalid still reports validity and payload scores.

corpus/slides/ replays a small recorded corpus against the in-repo slides pack, so a change to a scorer, the harness, or the pack that moves a score shows up as a diff in review. src/slides_corpus.test.ts runs it in the normal pnpm test, with no model and no credentials.

The test fails when the report, the prompt, or the formatted text differs from what is committed. It also records, in baseline.json, a hash of the prompt, any attempt the harness asked for that has no recording (missingRecordings; the replay hands back an empty answer, which fails to parse), and any case whose text rendering no longer matches the one its verdict was given for (staleJudgments; the recorded verdict is still returned). Either list going non-empty means the recordings no longer match the harness and need re-recording.

To accept a change that moves the numbers, regenerate the baseline and commit it with the change, so the diff shows what moved:

pnpm exec vitest run packages/isomer-evals/src/slides_corpus.test.ts -u
		

The requests, goldens, answers, and verdicts were all written by Claude (model id claude-opus-5-5) through Claude Code on 2026-10-29, not by a production model, and are for replay only. Each answer was written from prompt.txt followed by ## Request and the case prompt, the way the example generate above assembles it; the one retry also had ## Previous errors and the first attempt's errors. No answer was edited after it was scored. Because one model wrote the goldens and the answers, selection and expected-primitive scores share its priors: treat the baseline as a regression signal for the scorer and the pack, not as a measurement of any model.

1. Keep corpus.json, or replace its cases; each id names its answer files.
2. Run the same runEvals options the test passes, with a generate that calls the production model and writes each attempt to answers/<id>.<attempt>.txt before returning it.
3. Re-judge each validated case from its request and text rendering, and record the verdict and the rendering in judgments.json.
4. Update the provenance blocks in corpus.json and judgments.json to name the model, version, date, and any sampling settings.
5. Regenerate the baseline with the command above and commit everything together.

The axes are exported individually, so a pack can assert on one without running a corpus. scoreValidity is pure: it takes attempts already checked with checkAttempt. scorePrimitiveSelection and scoreExpectedPrimitives read only runtime.primitives, to walk the tree; scorePayload and scoreAnswerability call the runtime you hand them:

import { scorePrimitiveSelection } from '@elastic/isomer-evals';

expect(scorePrimitiveSelection(runtime, generated, golden).missing).toEqual([]);
		

checkAttempt(runtime, parseGenerated(raw)) checks the attempt's value once, through the SDK's checkComposition — runEvals reads that one check for the retry decision, scoreValidity, and scorePayload. A value that matches the schema but fails a semantic rule (duplicate ids, for instance) therefore never reads as a good composition. scorePayload reads the first attempt only, so a retry changes validity and not payload.

runEvals hands the model the whole catalog in one prompt. An agent instead discovers primitives through tools, and what it spends doing so is the cost to measure. runAgentEvals drives that loop: it offers the tools, runs the calls the model makes, and scores the answer the model ends on.

import { formatAgentReport, runAgentEvals } from '@elastic/isomer-evals';

const report = await runAgentEvals({
  runtime,
  tools,
  corpus,
  step: async ({ system, tools, messages }) => callYourModel(system, tools, messages),
  repeat: 3,
});

console.log(formatAgentReport(report));
		

1. e.g. createIsomerTools(runtime) from agent-tools
2. cases with optional expected node types

step is one model turn, and the only way a model is reached. It receives the system prompt, each tool as { name, description, parameters } with parameters as JSON Schema, and the conversation so far, and it returns the turn's text, its tool calls, and the tokens it spent. A turn with no tool calls is the answer. A provider adapter translates AgentMessage to its own wire format; the raw it returns on an assistant turn is handed back to it verbatim, for providers that need their own content replayed.

A tool is an AgentEvalTool: a name, a description, a Zod inputSchema, offered to the model as the JSON Schema of the input it accepts, and a handler returning text content, the same shape @elastic/isomer-agent-tools produces. Input the schema refuses, an unknown tool name, and a throwing handler are each answered to the model as a failed result rather than ending the run.

A case's expected node types are scored as in runEvals: scoreExpectedPrimitives counts a hit when any of them appears as a node anywhere in the answer's tree. Unlike runEvals, an agent answer that parsed but failed validation is still scored for it. AgentEvalReport.totals reports finished and valid runs, expected-primitive hits, and per-run means of turns, input and output tokens, tool-result bytes, and calls to each tool.

Types: EvalCase, EvalCaseResult, EvalReport, EvalRuntime, RunEvalsOptions, Generate, GenerateRequest, Judge, JudgeRequest, AnswerabilityVerdict, EvalFrame, and the score shapes ValidityScore, PrimitiveSelectionScore, ExpectedPrimitivesScore, PayloadScore, AnswerabilityScore. Agent runs add AgentEvalCaseResult, AgentEvalCall, AgentEvalReport, AgentEvalTool, AgentMessage, AgentStep, AgentStepRequest, AgentStepResult, AgentToolCall, AgentToolResult, AgentToolSpec, AgentUsage, and RunAgentEvalsOptions. src/api_reference.test.ts fails when a name exported from src/index.ts is missing from this page.

This package is private and not published to npm; it runs from this repository.

It depends on @elastic/isomer-sdk at its own version, and react and zod are required peers, because the SDK's root and ./author entries load them; a pack's runtime brings both already. One entry point, built as ESM for bundlers and CommonJS, which Node loads. The runtime and the slides pack are development dependencies, for the structural-contract test and the recorded corpus; only tests import them.