> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mutagent.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets Overview

> Test cases for prompt evaluation and optimization

# Datasets

Datasets are collections of test cases used to evaluate and optimize your prompts. They provide the ground truth against which your prompts are measured, enabling systematic quality assurance and continuous improvement.

## What is a Dataset?

A dataset is **scoped to a prompt** (linked via `promptGroupId`) and contains test case items with:

* **Input**: JSON object of variable values matching the prompt's `inputSchema`
* **Expected Output**: (Optional) JSON object of reference responses matching the prompt's `outputSchema`
* **Name**: (Optional) Human-readable test case name
* **Labels**: (Optional) ML-style labels for categorization (e.g., `["edge-case", "regression"]`)
* **Metadata**: (Optional) Arbitrary JSON for filtering and organization

```typescript theme={null}
interface Dataset {
  id: number;
  name: string;
  description?: string;
  promptGroupId: string;          // UUID linking to the prompt family
  labels?: string[];              // ML-style labels
  metadata?: Record<string, any>; // Custom fields
  createdBy?: string;             // Creator email
  createdAt: string;
  updatedAt: string;
}

interface DatasetItem {
  id: number;
  datasetId: number;
  name?: string;                       // Human-readable test case name
  input: Record<string, any>;          // Variable values (JSON object)
  expectedOutput?: Record<string, any>; // Expected response (JSON object)
  userFeedback?: string;               // Human feedback
  systemFeedback?: string;             // Automated feedback
  labels?: string[];                   // Test case labels
  metadata?: Record<string, any>;      // Custom fields
  createdAt: string;
  updatedAt: string;
}
```

## Use Cases

<CardGroup cols={2}>
  <Card title="Quality Testing" icon="flask">
    Verify prompts produce expected results across diverse inputs. Catch issues before deployment.
  </Card>

  <Card title="Regression Testing" icon="rotate-left">
    Ensure changes don't break existing behavior. Run the same tests after each prompt update.
  </Card>

  <Card title="Optimization Training" icon="chart-line">
    Provide training data for the optimization engine. Better datasets lead to better optimized prompts.
  </Card>

  <Card title="Benchmarking" icon="gauge">
    Compare different prompt versions objectively. Track improvement over time with consistent test cases.
  </Card>
</CardGroup>

## How Datasets Work

<Mermaid>
  flowchart LR
  Input\["Dataset Item<br />input JSON"] --> Merge\["Merge"]
  Template\["Prompt<br />+ variables"] --> Merge
  Merge --> LLM\["LLM"]
  LLM --> Output\["Response"]
  Output --> Compare\["Compare"]
  Expected\["expectedOutput"] --> Compare
  Compare --> Score\["Score"]

  style Input fill:#06B6D4,color:#000
  style Template fill:#7C3AED,color:#fff
  style Score fill:#10B981,color:#000
</Mermaid>

## Two-Step Creation

Datasets follow a two-step creation pattern:

1. **Create dataset metadata** -- name, description, and association to a prompt
2. **Bulk insert items** -- upload test case items via file or inline JSON

<Mermaid>
  flowchart LR
  A\["Step 1<br />Create Metadata"] --> B\["Step 2<br />Bulk Insert Items"]
  B --> C\["Dataset Ready<br />for Evaluation"]

  style A fill:#06B6D4,color:#000
  style B fill:#7C3AED,color:#fff
  style C fill:#10B981,color:#000
</Mermaid>

## Quick Example

<CodeGroup>
  ```typescript SDK theme={null}
  import { Mutagent } from '@mutagent/sdk';

  const client = new Mutagent({ apiKey: process.env.MUTAGENT_API_KEY });

  // Step 1: Create dataset metadata (scoped to a prompt)
  const dataset = await client.promptDatasets.createPromptDataset({
    id: 123,  // prompt ID
    body: {
      name: 'Support Scenarios',
      description: 'Common customer support questions and expected answers',
    },
  });

  // Step 2: Bulk insert test case items
  await client.promptDatasetItems.bulkCreatePromptDatasetItems({
    id: dataset.id,
    body: { items: [
      {
        input: {
          customer_name: 'John',
          question: 'How do I reset my password?',
        },
        expectedOutput: {
          response: 'To reset your password, go to Settings > Security > Reset Password.',
        },
        name: 'Password reset - happy path',
        labels: ['authentication', 'happy-path'],
      },
      {
        input: {
          customer_name: 'Sarah',
          question: 'What are your business hours?',
        },
        expectedOutput: {
          response: 'We are open Monday through Friday, 9am to 5pm EST.',
        },
        labels: ['general', 'happy-path'],
      },
    ]},
  });
  ```

  ```bash CLI theme={null}
  # One-step: create dataset + upload items from a file
  mutagent prompts dataset add <prompt-id> \
    --file dataset.json \
    --name "Support Scenarios"

  # Upload from JSONL file
  mutagent prompts dataset add <prompt-id> \
    --file dataset.jsonl \
    --name "Edge Cases"

  # Upload from CSV
  mutagent prompts dataset add <prompt-id> \
    --file dataset.csv \
    --name "Production Data"

  # Inline JSON data
  mutagent prompts dataset add <prompt-id> \
    -d '[{"input":{"question":"How do I reset?"},"expectedOutput":{"response":"Go to Settings..."}}]'
  ```

  ```bash cURL theme={null}
  # Step 1: Create dataset metadata
  curl -X POST https://api.mutagent.io/api/prompt/123/datasets \
    -H "x-api-key: mt_xxxx" \
    -H "Content-Type: application/json" \
    -d '{
      "name": "Support Scenarios",
      "description": "Common customer support questions"
    }'

  # Step 2: Bulk insert items
  curl -X POST https://api.mutagent.io/api/prompts/datasets/456/items/bulk \
    -H "x-api-key: mt_xxxx" \
    -H "Content-Type: application/json" \
    -d '{
      "items": [
        {
          "input": {"question": "How do I reset my password?"},
          "expectedOutput": {"response": "Go to Settings > Security > Reset Password"},
          "name": "Password reset",
          "labels": ["authentication"]
        }
      ]
    }'
  ```
</CodeGroup>

## Dataset Types

<AccordionGroup>
  <Accordion title="Golden Datasets">
    High-quality, human-verified test cases that serve as the benchmark:

    * Carefully curated inputs
    * Expert-written expected outputs
    * Used for critical quality gates
  </Accordion>

  <Accordion title="Synthetic Datasets">
    Generated test cases for broader coverage:

    * Created from templates or rules
    * Useful for stress testing
    * May not have expected outputs
  </Accordion>

  <Accordion title="Production Datasets">
    Real-world data captured from production:

    * Actual user inputs
    * Representative of real usage
    * May require anonymization
  </Accordion>
</AccordionGroup>

## What's Next?

<CardGroup cols={2}>
  <Card title="Creating Datasets" icon="plus" href="/platform/datasets/creating">
    Learn how to build datasets with items, imports, and the CLI
  </Card>

  <Card title="Best Practices" icon="check" href="/platform/datasets/best-practices">
    Guidelines for effective dataset design and management
  </Card>
</CardGroup>