Evaluations

Evaluations define how your prompts are measured against test datasets. An evaluation is a definition entity that specifies what to evaluate (prompt + dataset) and how to evaluate it (criteria and LLM configuration). Evaluation results are produced when the optimization loop runs your evaluation against dataset items.

Why Evaluate?

Quality Assurance

Verify prompts meet quality standards before deployment. Catch issues early with field-level criteria.

Regression Detection

Automatically detect when changes break existing behavior. Compare scores across prompt versions.

Optimization Feedback

Provide scoring criteria that guide the automated optimization engine toward better prompts.

Version Comparison

Objectively compare different prompt versions using consistent metrics and datasets.

How It Works

Evaluations are created as definitions, then executed automatically within the optimization loop or via the dashboard. There is no separate “run evaluation” CLI command — evaluation execution happens as part of prompt optimization.

Creating an Evaluation

An evaluation links a prompt to a dataset with evaluation criteria.

import { Mutagent } from '@mutagent/sdk';

const client = new Mutagent({ apiKey: process.env.MUTAGENT_API_KEY });

// Create an evaluation definition
const evaluation = await client.promptEvaluations.createEvaluation({
  promptId: 123,
  datasetId: 456,
  name: 'Customer Support Quality Eval',
  description: 'Evaluate response quality for support prompts',
  evalConfig: {
    criteria: [
      { field: 'output', metric: 'g_eval', weight: 0.4 },
      { field: 'output', metric: 'semantic_similarity', weight: 0.3 },
      { field: 'output', metric: 'contains', params: { required: ['refund policy'] }, weight: 0.3 },
    ],
    threshold: 0.8,
  },
  llmConfig: {
    model: 'claude-sonnet-4-6',
    temperature: 0.7,
  },
});

console.log('Evaluation created:', evaluation.id);

# Create an evaluation with criteria from a JSON file
mutagent prompts evaluation create 123 \
  --name "Customer Support Quality Eval" \
  --file criteria.json

# List evaluations for a prompt
mutagent prompts evaluation list --prompt-id 123

# Create evaluation definition
curl -X POST https://api.mutagent.io/api/prompts/evaluations \
  -H "x-api-key: mt_xxxx" \
  -H "x-workspace-id: ws_xxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "promptId": 123,
    "datasetId": 456,
    "name": "Customer Support Quality Eval",
    "evalConfig": {
      "criteria": [
        { "field": "output", "metric": "g_eval", "weight": 0.4 },
        { "field": "output", "metric": "semantic_similarity", "weight": 0.3 }
      ],
      "threshold": 0.8
    },
    "llmConfig": {
      "model": "claude-sonnet-4-6",
      "temperature": 0.7
    }
  }'

# Get evaluation results
curl https://api.mutagent.io/api/prompts/evaluations/1/result \
  -H "x-api-key: mt_xxxx"

Available Metrics

Metric	Type	Description	Best For
G-Eval	LLM-based	AI judge assesses quality, relevance, coherence	General quality
Semantic Similarity	Embedding	Cosine similarity between output and expected	Meaning preservation
Exact Match	Deterministic	Binary match against expected output	Classification, structured
Contains	Deterministic	Checks for required substrings	Key information
Regex Match	Deterministic	Pattern matching against output	Format validation
Custom	LLM-based	Your own evaluation criteria	Domain-specific

Combine multiple metrics for comprehensive evaluation. G-Eval catches quality issues; semantic similarity catches meaning drift; deterministic metrics catch format errors.

Evaluation Configuration

The evalConfig object defines the criteria for scoring. Criteria are field-level and can target input or output:

interface EvalConfig {
  criteria: Array<{
    field: 'input' | 'output';       // Which field to evaluate
    metric: string;                   // Metric name (g_eval, semantic_similarity, etc.)
    weight?: number;                  // Importance weight (default: 1.0)
    params?: Record<string, any>;     // Metric-specific parameters
  }>;
  threshold?: number;                 // Minimum acceptable score (0.0 - 1.0)
}

Criteria Examples

Development Evaluation

Quick feedback during prompt iteration:

Small dataset (5-10 items)
Single metric for fast turnaround

{
  "criteria": [
    { "field": "output", "metric": "g_eval", "weight": 1.0 }
  ],
  "threshold": 0.7
}

Pre-deployment Evaluation

Comprehensive check before publishing:

Full dataset (50+ items)
Multiple metrics with weighted scoring
Higher quality threshold

{
  "criteria": [
    { "field": "output", "metric": "g_eval", "weight": 0.4 },
    { "field": "output", "metric": "semantic_similarity", "weight": 0.3 },
    { "field": "output", "metric": "contains", "params": { "required": ["disclaimer"] }, "weight": 0.3 }
  ],
  "threshold": 0.85
}

Regression Evaluation

Compare new version against baseline:

Same dataset, different prompt versions
Consistent criteria across evaluations
Track improvement over time

{
  "criteria": [
    { "field": "output", "metric": "g_eval", "weight": 0.5 },
    { "field": "output", "metric": "semantic_similarity", "weight": 0.5 }
  ],
  "threshold": 0.8
}

Evaluation Results

Results are generated when the optimization loop executes your evaluation. Retrieve them via the API:

// Get results for an evaluation
const result = await client.promptEvaluations.getEvaluationResult({ id: 1 });

console.log('Score:', result.score);
console.log('Success:', result.success);
console.log('Metric Results:', result.metricResults);
console.log('Execution Time:', result.executionTime, 'ms');

Result Structure

interface EvaluationResult {
  id: number;
  evaluationId: number;           // Parent evaluation definition
  actualOutput: object;           // LLM output as JSON
  success: boolean;               // Whether the evaluation passed
  score: number | null;           // Numeric score (0.0 - 1.0)
  metricResults: object;          // Per-metric results (e.g., {"g_eval": 0.95, "semantic_similarity": 0.82})
  executionTime: number | null;   // Execution time in milliseconds
  createdAt: string;              // Result timestamp
}

Quality Gates

Use evaluations as quality gates in your workflow:

async function deployPrompt(promptId: number) {
  // Get evaluation results
  const result = await client.promptEvaluations.getEvaluationResult({ id: evalId });

  // Enforce quality thresholds
  const QUALITY_THRESHOLD = 0.85;

  if (!result.success || (result.score && result.score < QUALITY_THRESHOLD)) {
    throw new Error(
      `Quality gate failed: score ${result.score} < ${QUALITY_THRESHOLD}`
    );
  }

  // Safe to deploy
  console.log('Quality gate passed, deploying prompt');
}

Evaluations Overview

Evaluations

Why Evaluate?

Quality Assurance

Regression Detection

Optimization Feedback

Version Comparison

How It Works

Creating an Evaluation

Available Metrics

Evaluation Configuration

Criteria Examples

Evaluation Results

Result Structure

Quality Gates

What’s Next?

Evaluation Metrics

Running Evaluations

​Evaluations

​Why Evaluate?

Quality Assurance

Regression Detection

Optimization Feedback

Version Comparison

​How It Works

​Creating an Evaluation

​Available Metrics

​Evaluation Configuration

​Criteria Examples

​Evaluation Results

​Result Structure

​Quality Gates

​What’s Next?

Evaluation Metrics

Running Evaluations

Evaluations

Why Evaluate?

How It Works

Creating an Evaluation

Available Metrics

Evaluation Configuration

Criteria Examples

Evaluation Results

Result Structure

Quality Gates

What’s Next?