> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mutagent.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Running Evaluations

> How evaluations execute and produce results

# Running Evaluations

Evaluations in MutagenT are **definition entities** -- they specify what to evaluate and how. Evaluation execution happens within the optimization loop or via the dashboard. This guide covers how to create evaluations, retrieve results, and interpret scores.

<Note>
  There is no separate "run evaluation" CLI command. Evaluations execute as part of the optimization workflow or through the dashboard UI. The CLI is used to create evaluation definitions and retrieve results.
</Note>

## Create an Evaluation

### Via Dashboard

1. Navigate to a prompt's **Evaluations** tab
2. Click **New Evaluation**
3. Configure your evaluation:
   * Select the dataset to test against
   * Define evaluation criteria (metrics, weights, thresholds)
   * Configure LLM settings for the judge model
4. Click **Create Evaluation**
5. The evaluation will execute when optimization runs or when triggered from the dashboard

### Via CLI

```bash theme={null}
# Create an evaluation definition with criteria from a JSON file
mutagent prompts evaluation create 123 \
  --name "Quality Check" \
  --file criteria.json

# List evaluations for a prompt
mutagent prompts evaluation list --prompt-id 123

# Get evaluation details
mutagent prompts evaluation get 1
```

The `criteria.json` file contains your evaluation configuration:

```json theme={null}
{
  "datasetId": 456,
  "evalConfig": {
    "criteria": [
      { "field": "output", "metric": "g_eval", "weight": 0.5 },
      { "field": "output", "metric": "semantic_similarity", "weight": 0.5 }
    ],
    "threshold": 0.8
  },
  "llmConfig": {
    "model": "claude-sonnet-4-6",
    "temperature": 0.7
  }
}
```

### Via SDK

<CodeGroup>
  ```typescript TypeScript theme={null}
  import { Mutagent } from '@mutagent/sdk';

  const client = new Mutagent({ apiKey: process.env.MUTAGENT_API_KEY });

  // Create evaluation definition
  const evaluation = await client.promptEvaluations.createEvaluation({
    promptId: 123,
    datasetId: 456,
    name: 'Customer Support Quality',
    evalConfig: {
      criteria: [
        { field: 'output', metric: 'g_eval', weight: 0.5 },
        { field: 'output', metric: 'semantic_similarity', weight: 0.5 },
      ],
      threshold: 0.8,
    },
    llmConfig: {
      model: 'claude-sonnet-4-6',
      temperature: 0.7,
    },
  });

  console.log('Evaluation created:', evaluation.id);
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.mutagent.io/api/prompts/evaluations \
    -H "x-api-key: mt_xxxx" \
    -H "x-workspace-id: ws_xxxx" \
    -H "Content-Type: application/json" \
    -d '{
      "promptId": 123,
      "datasetId": 456,
      "name": "Customer Support Quality",
      "evalConfig": {
        "criteria": [
          { "field": "output", "metric": "g_eval", "weight": 0.5 },
          { "field": "output", "metric": "semantic_similarity", "weight": 0.5 }
        ],
        "threshold": 0.8
      },
      "llmConfig": {
        "model": "claude-sonnet-4-6",
        "temperature": 0.7
      }
    }'
  ```
</CodeGroup>

## How Evaluations Execute

Evaluations produce results through two paths:

<Mermaid>
  flowchart LR
  E\["Evaluation<br />Definition"] --> P1\["Dashboard<br />Manual Trigger"]
  E --> P2\["Optimization Loop<br />Automatic"]

  P1 --> EX\["Execute:<br />1. Load dataset items<br />2. Run prompt per item<br />3. Score with criteria<br />4. Aggregate results"]
  P2 --> EX

  EX --> R\["Results<br />Score + Metrics"]

  style E fill:#7C3AED,color:#fff
  style R fill:#10B981,color:#000
</Mermaid>

1. **Dashboard** - Trigger evaluation execution manually from the prompt evaluations page
2. **Optimization Loop** - Evaluations run automatically during each optimization iteration to score prompt variants

## Get Results

Retrieve evaluation results once execution completes:

```typescript theme={null}
// Get results for a specific evaluation
const result = await client.promptEvaluations.getEvaluationResult({ id: 1 });

console.log('=== Evaluation Results ===');
console.log('Score:', result.score);
console.log('Success:', result.success);
console.log('Execution Time:', result.executionTime, 'ms');

// Per-metric breakdown
console.log('Metric Results:');
for (const [metric, score] of Object.entries(result.metricResults)) {
  console.log(`  ${metric}: ${score}`);
}
```

### Via CLI

```bash theme={null}
# Get evaluation details (use 'get' subcommand — 'result' is not a documented subcommand)
mutagent prompts evaluation get 1
```

<Note>
  To retrieve evaluation results programmatically, use the SDK's `client.promptEvaluations.getEvaluationResult({ id })` method or the REST endpoint `GET /api/prompts/evaluations/:id/result`. The CLI `prompts evaluation` command supports: `list`, `get`, `create`, `delete`.
</Note>

### Via cURL

```bash theme={null}
curl https://api.mutagent.io/api/prompts/evaluations/1/result \
  -H "x-api-key: mt_xxxx"
```

## Interpreting Scores

### Score Thresholds

| Score Range | Quality Level | Recommendation         |
| ----------- | ------------- | ---------------------- |
| 0.95 - 1.00 | Excellent     | Production ready       |
| 0.85 - 0.94 | Good          | Deploy with monitoring |
| 0.75 - 0.84 | Fair          | Consider improvements  |
| 0.65 - 0.74 | Poor          | Needs significant work |
| \< 0.65     | Critical      | Do not deploy          |

### Analyzing Low Scores

When scores are lower than expected, examine the per-metric breakdown in `metricResults`:

```typescript theme={null}
const result = await client.promptEvaluations.getEvaluationResult({ id: evalId });

if (result.score && result.score < 0.8) {
  console.log('Low score detected. Metric breakdown:');
  for (const [metric, score] of Object.entries(result.metricResults)) {
    const status = typeof score === 'number' && score < 0.7 ? 'FAILING' : 'OK';
    console.log(`  ${metric}: ${score} [${status}]`);
  }
}
```

## Comparing Evaluations

Compare results across different prompt versions by creating evaluations with the same dataset and criteria for each version:

```typescript theme={null}
// Create evaluations for two prompt versions (same dataset, same criteria)
const evalConfig = {
  criteria: [
    { field: 'output', metric: 'g_eval', weight: 0.5 },
    { field: 'output', metric: 'semantic_similarity', weight: 0.5 },
  ],
};

const evalV1 = await client.promptEvaluations.createEvaluation({
  promptId: promptV1Id,
  datasetId: goldenDatasetId,
  name: 'V1 Evaluation',
  evalConfig,
});

const evalV2 = await client.promptEvaluations.createEvaluation({
  promptId: promptV2Id,
  datasetId: goldenDatasetId,
  name: 'V2 Evaluation',
  evalConfig,
});

// After optimization runs, compare results
const resultV1 = await client.promptEvaluations.getEvaluationResult({ id: evalV1.id });
const resultV2 = await client.promptEvaluations.getEvaluationResult({ id: evalV2.id });

if (resultV1.score && resultV2.score) {
  const improvement = resultV2.score - resultV1.score;
  console.log(`V1 Score: ${resultV1.score.toFixed(2)}`);
  console.log(`V2 Score: ${resultV2.score.toFixed(2)}`);
  console.log(`Change: ${improvement > 0 ? '+' : ''}${(improvement * 100).toFixed(1)}%`);
}
```

## Handling Failures

### Common Failure Causes

<AccordionGroup>
  <Accordion title="Provider errors">
    LLM API returned an error. Check provider configuration in **Settings > Providers** and verify rate limits.
  </Accordion>

  <Accordion title="Timeout">
    Evaluation took too long. Try a smaller dataset or check if the LLM provider is experiencing high latency.
  </Accordion>

  <Accordion title="Invalid prompt">
    Prompt template has syntax errors or missing variables. Verify the prompt variables match the dataset schema.
  </Accordion>

  <Accordion title="Dataset issues">
    Dataset items have invalid or missing required fields. Ensure all dataset items have the fields referenced in your criteria.
  </Accordion>
</AccordionGroup>

## Best Practices

<AccordionGroup>
  <Accordion title="Use consistent datasets">
    Always compare evaluations using the same dataset to ensure valid comparisons.
  </Accordion>

  <Accordion title="Set appropriate thresholds">
    Define minimum scores based on your quality requirements and adjust as you learn.
  </Accordion>

  <Accordion title="Combine metrics strategically">
    Use G-Eval for quality, semantic similarity for meaning, and deterministic metrics for format validation.
  </Accordion>

  <Accordion title="Track trends over time">
    Create evaluations with the same criteria for each prompt version to monitor quality trends.
  </Accordion>
</AccordionGroup>