> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mutagent.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations Overview

> Measure and improve prompt quality systematically

# Evaluations

Evaluations define how your prompts are measured against test datasets. An evaluation is a **definition entity** that specifies what to evaluate (prompt + dataset) and how to evaluate it (criteria and LLM configuration). Evaluation results are produced when the optimization loop runs your evaluation against dataset items.

## Why Evaluate?

<CardGroup cols={2}>
  <Card title="Quality Assurance" icon="shield-check">
    Verify prompts meet quality standards before deployment. Catch issues early with field-level criteria.
  </Card>

  <Card title="Regression Detection" icon="bug">
    Automatically detect when changes break existing behavior. Compare scores across prompt versions.
  </Card>

  <Card title="Optimization Feedback" icon="chart-line">
    Provide scoring criteria that guide the automated optimization engine toward better prompts.
  </Card>

  <Card title="Version Comparison" icon="code-compare">
    Objectively compare different prompt versions using consistent metrics and datasets.
  </Card>
</CardGroup>

## How It Works

<Mermaid>
  flowchart LR
  subgraph Definition\["Evaluation Definition"]
  direction TB
  P\["Prompt"]
  D\["Dataset"]
  C\["Criteria"]
  end

  Definition --> OPT\["Optimization Loop<br />1. Substitute vars<br />2. Call LLM<br />3. Score against criteria<br />4. Record results"]

  OPT --> Results\["Results<br />Score: 0.87<br />47/50 passed"]

  style P fill:#7C3AED,color:#fff
  style D fill:#06B6D4,color:#000
  style C fill:#F59E0B,color:#000
  style Results fill:#10B981,color:#000
</Mermaid>

Evaluations are **created as definitions**, then executed automatically within the optimization loop or via the dashboard. There is no separate "run evaluation" CLI command -- evaluation execution happens as part of prompt optimization.

## Creating an Evaluation

An evaluation links a prompt to a dataset with evaluation criteria.

<CodeGroup>
  ```typescript TypeScript theme={null}
  import { Mutagent } from '@mutagent/sdk';

  const client = new Mutagent({ apiKey: process.env.MUTAGENT_API_KEY });

  // Create an evaluation definition
  const evaluation = await client.promptEvaluations.createEvaluation({
    promptId: 123,
    datasetId: 456,
    name: 'Customer Support Quality Eval',
    description: 'Evaluate response quality for support prompts',
    evalConfig: {
      criteria: [
        { field: 'output', metric: 'g_eval', weight: 0.4 },
        { field: 'output', metric: 'semantic_similarity', weight: 0.3 },
        { field: 'output', metric: 'contains', params: { required: ['refund policy'] }, weight: 0.3 },
      ],
      threshold: 0.8,
    },
    llmConfig: {
      model: 'claude-sonnet-4-6',
      temperature: 0.7,
    },
  });

  console.log('Evaluation created:', evaluation.id);
  ```

  ```bash CLI theme={null}
  # Create an evaluation with criteria from a JSON file
  mutagent prompts evaluation create 123 \
    --name "Customer Support Quality Eval" \
    --file criteria.json

  # List evaluations for a prompt
  mutagent prompts evaluation list --prompt-id 123
  ```

  ```bash cURL theme={null}
  # Create evaluation definition
  curl -X POST https://api.mutagent.io/api/prompts/evaluations \
    -H "x-api-key: mt_xxxx" \
    -H "x-workspace-id: ws_xxxx" \
    -H "Content-Type: application/json" \
    -d '{
      "promptId": 123,
      "datasetId": 456,
      "name": "Customer Support Quality Eval",
      "evalConfig": {
        "criteria": [
          { "field": "output", "metric": "g_eval", "weight": 0.4 },
          { "field": "output", "metric": "semantic_similarity", "weight": 0.3 }
        ],
        "threshold": 0.8
      },
      "llmConfig": {
        "model": "claude-sonnet-4-6",
        "temperature": 0.7
      }
    }'

  # Get evaluation results
  curl https://api.mutagent.io/api/prompts/evaluations/1/result \
    -H "x-api-key: mt_xxxx"
  ```
</CodeGroup>

## Available Metrics

| Metric                  | Type          | Description                                     | Best For                   |
| ----------------------- | ------------- | ----------------------------------------------- | -------------------------- |
| **G-Eval**              | LLM-based     | AI judge assesses quality, relevance, coherence | General quality            |
| **Semantic Similarity** | Embedding     | Cosine similarity between output and expected   | Meaning preservation       |
| **Exact Match**         | Deterministic | Binary match against expected output            | Classification, structured |
| **Contains**            | Deterministic | Checks for required substrings                  | Key information            |
| **Regex Match**         | Deterministic | Pattern matching against output                 | Format validation          |
| **Custom**              | LLM-based     | Your own evaluation criteria                    | Domain-specific            |

<Note>
  Combine multiple metrics for comprehensive evaluation. G-Eval catches quality issues; semantic similarity catches meaning drift; deterministic metrics catch format errors.
</Note>

## Evaluation Configuration

The `evalConfig` object defines the criteria for scoring. Criteria are field-level and can target input or output:

```typescript theme={null}
interface EvalConfig {
  criteria: Array<{
    field: 'input' | 'output';       // Which field to evaluate
    metric: string;                   // Metric name (g_eval, semantic_similarity, etc.)
    weight?: number;                  // Importance weight (default: 1.0)
    params?: Record<string, any>;     // Metric-specific parameters
  }>;
  threshold?: number;                 // Minimum acceptable score (0.0 - 1.0)
}
```

### Criteria Examples

<AccordionGroup>
  <Accordion title="Development Evaluation">
    Quick feedback during prompt iteration:

    * Small dataset (5-10 items)
    * Single metric for fast turnaround

    ```json theme={null}
    {
      "criteria": [
        { "field": "output", "metric": "g_eval", "weight": 1.0 }
      ],
      "threshold": 0.7
    }
    ```
  </Accordion>

  <Accordion title="Pre-deployment Evaluation">
    Comprehensive check before publishing:

    * Full dataset (50+ items)
    * Multiple metrics with weighted scoring
    * Higher quality threshold

    ```json theme={null}
    {
      "criteria": [
        { "field": "output", "metric": "g_eval", "weight": 0.4 },
        { "field": "output", "metric": "semantic_similarity", "weight": 0.3 },
        { "field": "output", "metric": "contains", "params": { "required": ["disclaimer"] }, "weight": 0.3 }
      ],
      "threshold": 0.85
    }
    ```
  </Accordion>

  <Accordion title="Regression Evaluation">
    Compare new version against baseline:

    * Same dataset, different prompt versions
    * Consistent criteria across evaluations
    * Track improvement over time

    ```json theme={null}
    {
      "criteria": [
        { "field": "output", "metric": "g_eval", "weight": 0.5 },
        { "field": "output", "metric": "semantic_similarity", "weight": 0.5 }
      ],
      "threshold": 0.8
    }
    ```
  </Accordion>
</AccordionGroup>

## Evaluation Results

Results are generated when the optimization loop executes your evaluation. Retrieve them via the API:

```typescript theme={null}
// Get results for an evaluation
const result = await client.promptEvaluations.getEvaluationResult({ id: 1 });

console.log('Score:', result.score);
console.log('Success:', result.success);
console.log('Metric Results:', result.metricResults);
console.log('Execution Time:', result.executionTime, 'ms');
```

### Result Structure

```typescript theme={null}
interface EvaluationResult {
  id: number;
  evaluationId: number;           // Parent evaluation definition
  actualOutput: object;           // LLM output as JSON
  success: boolean;               // Whether the evaluation passed
  score: number | null;           // Numeric score (0.0 - 1.0)
  metricResults: object;          // Per-metric results (e.g., {"g_eval": 0.95, "semantic_similarity": 0.82})
  executionTime: number | null;   // Execution time in milliseconds
  createdAt: string;              // Result timestamp
}
```

## Quality Gates

Use evaluations as quality gates in your workflow:

```typescript theme={null}
async function deployPrompt(promptId: number) {
  // Get evaluation results
  const result = await client.promptEvaluations.getEvaluationResult({ id: evalId });

  // Enforce quality thresholds
  const QUALITY_THRESHOLD = 0.85;

  if (!result.success || (result.score && result.score < QUALITY_THRESHOLD)) {
    throw new Error(
      `Quality gate failed: score ${result.score} < ${QUALITY_THRESHOLD}`
    );
  }

  // Safe to deploy
  console.log('Quality gate passed, deploying prompt');
}
```

## What's Next?

<CardGroup cols={2}>
  <Card title="Evaluation Metrics" icon="ruler" href="/platform/evaluations/metrics">
    Deep dive into available metrics and when to use each
  </Card>

  <Card title="Running Evaluations" icon="play" href="/platform/evaluations/running">
    Learn how evaluations execute within the optimization workflow
  </Card>
</CardGroup>