> ## Documentation Index
> Fetch the complete documentation index at: https://growthx-chore-remove-output-wrapper.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Test workflow quality across datasets without modifying your workflow code

Your workflows have [evaluator steps](/evaluators/evaluator-step) that power generate-evaluate-retry loops in production. Evaluation workflows answer a different question: "across a set of known inputs, does my workflow still produce acceptable output?"

Workflow evaluators run *outside* your workflow. You define evaluators with `verify()`, write test cases as YAML datasets, and run them with the CLI. The framework feeds each dataset case through your evaluators and reports pass/partial/fail verdicts. Nothing in your production workflow changes.

This is how you catch regressions when you update a prompt, swap a model, or refactor a step. Run evals before deploying and you'll know if quality dropped.

## Evaluator Step vs Evaluation Workflow

These serve different purposes and live in different places:

|                    | Evaluator Step                        | Evaluation Workflow                              |
| ------------------ | ------------------------------------- | ------------------------------------------------ |
| **Purpose**        | Control workflow flow at runtime      | Test quality across datasets                     |
| **Runs**           | Inside production workflows           | Outside, via CLI                                 |
| **Created with**   | `evaluator()` from `@outputai/core`   | `verify()` from `@outputai/evals`                |
| **File location**  | `src/workflows/<name>/evaluators.ts`  | `src/workflows/<name>/tests/evals/evaluators.ts` |
| **Affects output** | Yes — retry, branch, gate             | No — reports results only                        |
| **Uses**           | Quality gates, self-correcting agents | Regression testing, prompt comparison, CI/CD     |

Evaluator steps are documented in [Evaluator Step](/evaluators/evaluator-step). This section covers evaluation workflows.

## Directory Structure

Eval files live alongside your workflow in a `tests/` directory:

```text theme={null}
src/workflows/
  blog_generator/
    workflow.ts
    steps.ts
    evaluators.ts              # Inline evaluators (production)
    types.ts
    prompts/
      generate_blog@v1.prompt
    tests/
      evals/
        workflow.ts            # evalWorkflow() definition
        evaluators.ts          # Workflow evaluators (verify + Verdict)
        judge_topic@v1.prompt  # Judge prompt files
        judge_quality@v1.prompt
      datasets/
        happy_path.yml         # Test cases
        edge_case.yml
```

The eval workflow file (`tests/evals/workflow.ts`) is discovered automatically by the worker alongside your regular workflow.

### Nested workflow folders

Workflow folders nest to any depth. A workflow at `src/workflows/content/writing/style_anchor/` registered as `content_writing_style_anchor` keeps its evals and datasets in its own `tests/` directory, and the CLI resolves them by the registered name — no symlink and no flat folder required.

<Warning>
  The eval workflow only registers if `tests/evals/workflow.ts` compiles to `dist` — the worker loads workflows from the compiled output, not from source. A `tsconfig` that excludes the whole `src/**/tests` tree silently drops it, and `output workflow test` then can't find `<workflow>_eval`. Exclude test files individually instead:

  ```json tsconfig.json theme={null}
  {
    "exclude": ["node_modules", "dist", "**/*.spec.ts", "**/*.test.ts"]
  }
  ```

  When the eval source exists but isn't registered, `output workflow test` says so and points at this fix instead of failing with a generic "workflow not found".
</Warning>

## Writing Evaluators with verify()

`verify()` creates a typed evaluator that receives the workflow's input, output, and optional ground truth from the dataset. It wraps `evaluator()` from `@outputai/core`, so it integrates with both the eval workflow and the Temporal worker.

```typescript tests/evals/evaluators.ts theme={null}
import { verify, Verdict } from '@outputai/evals';
import { z } from '@outputai/core';

export const evaluateSum = verify(
  {
    name: 'evaluate_sum',
    input: z.object( { values: z.array( z.number() ) } ),
    output: z.object( { result: z.number() } )
  },
  ( { input, output } ) =>
    Verdict.equals( output.result, input.values.reduce( ( a, b ) => a + b, 0 ) )
);
```

The check function receives a `CheckContext` with three fields:

| Field                  | Type                      | Description                                      |
| ---------------------- | ------------------------- | ------------------------------------------------ |
| `input`                | `TInput`                  | The workflow input from the dataset              |
| `output`               | `TOutput`                 | The workflow output (cached or freshly executed) |
| `context.ground_truth` | `Record<string, unknown>` | Ground truth values from the dataset YAML        |

The `input` and `output` schemas are optional — they default to `z.any()` if omitted. When provided, they give you type safety inside the check function.

### Using Ground Truth

Evaluators can read expected values from the dataset's `ground_truth` field. This lets you define per-case expectations without hardcoding them:

```typescript tests/evals/evaluators.ts theme={null}
import { verify, Verdict } from '@outputai/evals';
import { z } from '@outputai/core';

export const lengthOfOutput = verify(
  {
    name: 'length_of_output',
    input: z.object( { topic: z.string() } ),
    output: z.object( { title: z.string(), blog_post: z.string() } )
  },
  ( { output, context } ) =>
    Verdict.gte( output.blog_post.length, Number( context.ground_truth.min_length ?? 100 ) )
);
```

The `min_length` value comes from the dataset YAML — see [Ground Truth Structure](/evaluators/datasets#ground-truth-structure) for how this works.

## Verdict Helpers

The `Verdict` object provides helpers for returning evaluation results. There are three categories:

* **Deterministic assertions** — `equals`, `gte`, `contains`, `matches`, etc. Return `EvaluationBooleanResult` with `confidence: 1.0`.
* **Manual verdicts** — `pass()`, `partial()`, `fail()`. Return `EvaluationVerdictResult` for custom logic.
* **LLM result wrappers** — `fromJudge()`, `score()`, `label()`. Wrap LLM output with `confidence: 0.9`.

```typescript tests/evals/evaluators.ts theme={null}
import { verify, Verdict } from '@outputai/evals';
import { z } from '@outputai/core';

export const evaluateContent = verify(
  {
    name: 'evaluate_content',
    input: z.object( { topic: z.string() } ),
    output: z.object( { title: z.string(), blog_post: z.string() } )
  },
  ( { output, context } ) => {
    const required = String( context.ground_truth.required_content ?? '' );
    if ( !required ) {
      return Verdict.isTrue( true );
    }
    return Verdict.contains( output.blog_post, required );
  }
);
```

For the complete reference with signatures, examples, and edge cases for every Verdict method, see [Verdict Helpers](/evaluators/verdicts).

## LLM Judge Functions

For subjective evaluation — "is this blog post on-topic?", "rate the quality" — use the judge functions. They load a `.prompt` file, call the LLM, and return a typed evaluation result.

Four judge functions are available:

| Function       | Returns                     | Use for                  |
| -------------- | --------------------------- | ------------------------ |
| `judgeVerdict` | `pass` / `partial` / `fail` | Subjective quality gates |
| `judgeScore`   | `number`                    | Numeric ratings          |
| `judgeBoolean` | `true` / `false`            | Binary subjective checks |
| `judgeLabel`   | `string`                    | Classifications          |

```typescript tests/evals/evaluators.ts theme={null}
import { verify, judgeVerdict, judgeScore, judgeLabel } from '@outputai/evals';
import { z } from '@outputai/core';

const blogInput = z.object( { topic: z.string() } );
const blogOutput = z.object( { title: z.string(), blog_post: z.string() } );

export const evaluateTopic = verify(
  { name: 'evaluate_topic', input: blogInput, output: blogOutput },
  async ( { input, output, context } ) =>
    judgeVerdict( {
      prompt: 'judge_topic@v1',
      variables: {
        blog_title: output.title,
        blog_post: output.blog_post,
        required_topic: String( context.ground_truth.required_topic ?? input.topic )
      }
    } )
);

export const evaluateQuality = verify(
  { name: 'evaluate_quality', input: blogInput, output: blogOutput },
  async ( { input, output } ) =>
    judgeScore( {
      prompt: 'judge_quality@v1',
      variables: {
        blog_title: output.title,
        blog_post: output.blog_post,
        topic: input.topic
      }
    } )
);

export const evaluateTone = verify(
  { name: 'evaluate_tone', input: blogInput, output: blogOutput },
  async ( { output } ) =>
    judgeLabel( {
      prompt: 'judge_tone@v1',
      variables: {
        blog_title: output.title,
        blog_post: output.blog_post
      }
    } )
);
```

All judge functions accept a `JudgeArgs` object:

| Field       | Type                                          | Description                                           |
| ----------- | --------------------------------------------- | ----------------------------------------------------- |
| `prompt`    | `string`                                      | Prompt filename (e.g., `'judge_topic@v1'`)            |
| `variables` | `Record<string, string \| number \| boolean>` | Template variables                                    |
| `schema`    | `ZodType`                                     | Optional custom output schema (overrides the default) |

### Writing Judge Prompts

Judge `.prompt` files live in the same `tests/evals/` directory as your evaluators. Each judge function expects a specific output schema from the LLM:

| Function       | Expected JSON fields                                        |
| -------------- | ----------------------------------------------------------- |
| `judgeVerdict` | `{ verdict: 'pass'\|'partial'\|'fail', reasoning: string }` |
| `judgeScore`   | `{ score: number, reasoning: string }`                      |
| `judgeBoolean` | `{ result: boolean, reasoning: string }`                    |
| `judgeLabel`   | `{ label: string, reasoning: string }`                      |

Here's a judge prompt for topic evaluation:

```yaml tests/evals/judge_topic@v1.prompt theme={null}
---
provider: anthropic
model: claude-haiku-4-5-20251001
temperature: 0
maxTokens: 1000
---

<system>
You are an evaluation judge. Assess whether a blog post is faithfully about the required topic.

Return a JSON object with:
- verdict: "pass" if the blog clearly focuses on the topic, "partial" if it mentions the topic but lacks depth, "fail" if it is not about the topic
- reasoning: a brief explanation of your judgment
</system>

<user>
Required topic: {{ required_topic }}

Blog title: {{ blog_title }}

Blog post:
{{ blog_post }}

Judge whether this blog post is faithfully about the required topic.
</user>
```

Use `temperature: 0` for judge prompts — you want consistency, not creativity. For best practices on writing effective judge prompts, see [LLM-as-a-Judge Best Practices](/evaluators/best-practices).

### LLM Result Wrappers

If you're calling the LLM yourself instead of using the judge functions, the `Verdict` object provides wrappers to convert raw LLM output into evaluation results (with confidence `0.9`). See [LLM Result Wrappers](/evaluators/verdicts#llm-result-wrappers) for the full reference.

## What's Next

* [Verdict Helpers](/evaluators/verdicts) — Complete reference for all Verdict methods with signatures and examples
* [Datasets](/evaluators/datasets) — Defining test cases with inputs and ground truth
* [Running Eval Workflows](/evaluators/datasets-and-cli) — Wire evaluators into an eval workflow and run from the CLI
* [LLM-as-a-Judge Best Practices](/evaluators/best-practices) — Writing effective judge prompts and choosing grading scales
* [`@outputai/evals` API Reference](/packages/evals) — Complete package reference