> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mutagent.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Creating Datasets

> Build evaluation datasets for your prompts

# Creating Datasets

Build comprehensive datasets to evaluate and optimize your prompts. Datasets are scoped to prompts and follow a two-step creation pattern: create the metadata, then bulk insert items.

## CLI: One-Step Creation

The CLI combines both steps into a single command. It creates the dataset metadata and uploads items from a file or inline JSON in one call.

<CodeGroup>
  ```bash JSON file theme={null}
  # Upload from a JSON array file
  mutagent prompts dataset add <prompt-id> \
    --file dataset.json \
    --name "Customer Support Test Cases"
  ```

  ```bash JSONL file theme={null}
  # Upload from a JSONL file (one JSON object per line)
  mutagent prompts dataset add <prompt-id> \
    --file dataset.jsonl \
    --name "Edge Cases"
  ```

  ```bash CSV file theme={null}
  # Upload from a CSV file
  mutagent prompts dataset add <prompt-id> \
    --file dataset.csv \
    --name "Production Data"
  ```

  ```bash Inline JSON theme={null}
  # Provide items directly as inline JSON
  mutagent prompts dataset add <prompt-id> \
    -d '[{"input":{"text":"hello"},"expectedOutput":{"result":"world"}}]' \
    --name "Quick Test"
  ```
</CodeGroup>

<Note>
  The `--file` and `-d` flags are mutually exclusive. Use one or the other. If no `--name` is provided, the CLI generates a timestamped name automatically.
</Note>

## SDK: Two-Step Creation

With the SDK, you first create the dataset metadata, then bulk insert items:

```typescript theme={null}
import { Mutagent } from '@mutagent/sdk';

const client = new Mutagent({ apiKey: process.env.MUTAGENT_API_KEY });

// Step 1: Create dataset metadata (scoped to a prompt)
const dataset = await client.promptDatasets.createPromptDataset({
  id: 123,  // prompt ID
  body: {
    name: 'Customer Support Test Cases',
    description: 'Common support questions and expected answers for Q1 2026',
  },
});

console.log('Dataset ID:', dataset.id);

// Step 2: Bulk insert items
await client.promptDatasetItems.bulkCreatePromptDatasetItems({
  id: dataset.id,
  body: { items: [
    {
      input: {
        customer_name: 'John',
        question: 'How do I cancel my subscription?',
      },
      expectedOutput: {
        response: 'To cancel, go to Account > Subscription > Cancel.',
      },
      name: 'Cancel subscription - happy path',
      labels: ['billing', 'happy-path'],
      metadata: {
        source: 'support_tickets',
        difficulty: 'easy',
      },
    },
    {
      input: {
        customer_name: 'Sarah',
        question: 'Can I get a refund for last month?',
      },
      expectedOutput: {
        response: 'Refunds are available within 30 days of purchase.',
      },
      name: 'Refund request',
      labels: ['billing', 'edge-case'],
    },
  ]},
});
```

## API: Two-Step Creation

```bash theme={null}
# Step 1: Create dataset metadata
curl -X POST https://api.mutagent.io/api/prompt/123/datasets \
  -H "x-api-key: mt_xxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Customer Support Test Cases",
    "description": "Common support questions and expected answers"
  }'

# Step 2: Bulk insert items (use the dataset ID from step 1)
curl -X POST https://api.mutagent.io/api/prompts/datasets/456/items/bulk \
  -H "x-api-key: mt_xxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {
        "input": {"question": "How do I cancel?"},
        "expectedOutput": {"response": "Go to Account > Subscription > Cancel."},
        "name": "Cancel subscription",
        "labels": ["billing"]
      }
    ]
  }'
```

## Supported File Formats

### JSON Array

A JSON file containing an array of item objects:

```json theme={null}
[
  {
    "input": {
      "customer_name": "Alice",
      "question": "How do I upgrade my account?"
    },
    "expectedOutput": {
      "response": "Visit Account Settings and click Upgrade Plan."
    },
    "name": "Upgrade account",
    "labels": ["account"],
    "metadata": {
      "priority": "high"
    }
  },
  {
    "input": {
      "customer_name": "Bob",
      "question": "Can I get a refund?"
    },
    "expectedOutput": {
      "response": "Refunds are available within 30 days of purchase."
    },
    "name": "Refund request",
    "labels": ["billing"]
  }
]
```

### JSONL (Newline-Delimited JSON)

One JSON object per line -- useful for large datasets:

```jsonl theme={null}
{"input":{"question":"How do I upgrade?"},"expectedOutput":{"response":"Visit Account Settings."},"name":"Upgrade"}
{"input":{"question":"Can I get a refund?"},"expectedOutput":{"response":"Within 30 days."},"name":"Refund"}
{"input":{"question":"Reset password?"},"expectedOutput":{"response":"Click Forgot Password."},"name":"Password reset"}
```

### CSV

Comma-separated values with a header row. The CLI passes CSV content directly to the API for parsing:

```csv theme={null}
question,expected_answer,category
"How do I upgrade?","Visit Account Settings and click Upgrade Plan.",account
"Can I get a refund?","Refunds are available within 30 days.",billing
"Reset my password?","Click Forgot Password on the login page.",authentication
```

## Dataset Item Structure

Each item in a dataset follows this structure:

```typescript theme={null}
{
  // Required
  input: Record<string, any>;           // Variable values matching prompt's inputSchema

  // Optional
  expectedOutput?: Record<string, any>;  // Expected LLM response (for scoring)
  name?: string;                         // Human-readable test case name
  userFeedback?: string;                 // Human feedback notes
  systemFeedback?: string;               // Automated feedback
  labels?: string[];                     // ML-style labels for categorization
  metadata?: Record<string, any>;        // Arbitrary metadata
}
```

<Tip>
  Always include `expectedOutput` when you want to use the dataset for evaluation and optimization. Without expected outputs, only reference-free metrics (like coherence) can be used.
</Tip>

## List Datasets

View all datasets associated with a prompt:

<CodeGroup>
  ```bash CLI theme={null}
  # List datasets for a prompt
  mutagent prompts dataset list <prompt-id>

  # Machine-readable output
  mutagent prompts dataset list <prompt-id> --json
  ```

  ```typescript SDK theme={null}
  const datasets = await client.promptDatasets.listDatasetsForPrompt({ id: 123 });

  datasets.forEach(d => {
    console.log(`${d.name} (ID: ${d.id})`);
  });
  ```
</CodeGroup>

## Remove a Dataset

<CodeGroup>
  ```bash CLI theme={null}
  mutagent prompts dataset delete <prompt-id> <dataset-id>
  ```

  ```bash cURL theme={null}
  curl -X DELETE https://api.mutagent.io/api/prompts/datasets/456 \
    -H "x-api-key: mt_xxxx"
  ```
</CodeGroup>

## Clone Existing Dataset

Create a copy of an existing dataset, optionally targeting a different prompt:

```bash theme={null}
curl -X POST https://api.mutagent.io/api/prompts/datasets/456/clone \
  -H "x-api-key: mt_xxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "targetPromptId": 789,
    "newName": "Support Cases v2"
  }'
```

## Export Dataset

Export a dataset with all items for backup or analysis:

```bash theme={null}
curl https://api.mutagent.io/api/prompts/datasets/456/export \
  -H "x-api-key: mt_xxxx" \
  -o dataset-export.json
```

The export includes:

* Dataset metadata
* Prompt group information (promptGroupId, latest version, total versions)
* All items with their input/output data

## Best Practices for Creation

<AccordionGroup>
  <Accordion title="Start small, iterate">
    Begin with 10-20 high-quality items and expand based on evaluation results.
  </Accordion>

  <Accordion title="Use descriptive names">
    Name both the dataset and individual items descriptively:

    ```json theme={null}
    {
      "name": "Angry customer refund request",
      "input": {"question": "I want my money back NOW!"},
      "expectedOutput": {"response": "I understand your frustration..."}
    }
    ```

    Avoid generic names like "test-1" or "item-42".
  </Accordion>

  <Accordion title="Include expectedOutput for optimization">
    Datasets used with the optimizer require `expectedOutput` to measure quality. Always include expected outputs for datasets you plan to use in evaluation or optimization.
  </Accordion>

  <Accordion title="Use labels for categorization">
    Labels help filter and organize test cases:

    ```json theme={null}
    {
      "labels": ["edge-case", "billing", "regression-test"]
    }
    ```
  </Accordion>

  <Accordion title="Use consistent input formats">
    Ensure all items use the same variable names and data types as your prompt's `inputSchema`.
  </Accordion>
</AccordionGroup>

<Note>
  Datasets are associated with a prompt's `promptGroupId`. This means a dataset works across all versions of the same prompt, so you can test new versions against the same dataset.
</Note>