Datasets SDK

Create and manage evaluation datasets programmatically. Datasets are collections of test cases used for prompt evaluation, each containing input variables and expected outputs.

Dataset Architecture

Datasets are scoped to a prompt group (all versions of a prompt share the same promptGroupId). This means a dataset created for prompt v1.0.0 is automatically available when testing v2.0.0.

List All Datasets

const result = await client.promptDatasets.listPromptDatasets({
  limit: 20,
  offset: 0,
});

for await (const page of result) {
  const { data, total, hasNext } = page.result;
  data.forEach(d => console.log(d.name, d.promptGroupId));
}

You can filter by promptId, promptGroupId, name, or createdBy:

const result = await client.promptDatasets.listPromptDatasets({
  promptId: 123,
  name: 'baseline',
});

List Datasets for a Prompt

const datasets = await client.promptDatasets.listDatasetsForPrompt({
  id: 123,  // Prompt ID
});

datasets.forEach(d => console.log(d.name, `labels: ${d.labels}`));

Create Dataset

Datasets are created for a specific prompt:

const dataset = await client.promptDatasets.createPromptDataset({
  id: 123,  // Prompt ID
  body: {
    name: 'Support Scenarios',
    description: 'Common customer support scenarios',
    labels: ['baseline', 'v1'],
  },
});

console.log('Created dataset:', dataset.id);

Get Dataset

const dataset = await client.promptDatasets.getPromptDataset({
  id: 456,  // Dataset ID
});

console.log(dataset.name, dataset.promptGroupId);

Update Dataset

const updated = await client.promptDatasets.updatePromptDataset({
  id: 456,
  body: {
    name: 'Updated Name',
    description: 'New description',
  },
});

Delete Dataset

Use force: true to delete a dataset that still contains items:

await client.promptDatasets.deletePromptDataset({
  id: 456,
  force: true,
});

Clone Dataset

Clone a dataset to another prompt. Provide either targetPromptId or targetPromptGroupId:

const cloned = await client.promptDatasets.clonePromptDataset({
  id: 456,
  body: {
    targetPromptId: 789,
    newName: 'Support Scenarios (Copy)',
  },
});

Export Dataset

Export a dataset with all items and metadata:

const exported = await client.promptDatasets.exportPromptDataset({
  id: 456,
});

console.log('Dataset:', exported.dataset.name);
console.log('Items:', exported.items.length);
console.log('Exported at:', exported.exportedAt);

Dataset Items

List Items

const items = await client.promptDatasetItems.listPromptDatasetItems({
  id: 456,  // Dataset ID
});

items.forEach(item => {
  console.log('Input:', item.input);
  console.log('Expected:', item.expectedOutput);
});

Add Single Item

const item = await client.promptDatasetItems.createPromptDatasetItem({
  id: 456,  // Dataset ID
  body: {
    input: {
      company: 'Acme Inc',
      question: 'How do I reset my password?',
    },
    expectedOutput: {
      answer: 'To reset your password, go to Settings > Security...',
    },
    name: 'Password Reset',
    labels: ['auth', 'common'],
  },
});

Bulk Add Items

const result = await client.promptDatasetItems.bulkCreatePromptDatasetItems({
  id: 456,  // Dataset ID
  body: {
    items: [
      {
        input: { company: 'Acme', question: 'Pricing?' },
        expectedOutput: { answer: 'Our pricing starts at...' },
        name: 'Pricing Inquiry',
      },
      {
        input: { company: 'Acme', question: 'Refund policy?' },
        expectedOutput: { answer: 'We offer 30-day refunds...' },
        name: 'Refund Question',
      },
    ],
  },
});

console.log(`Added ${result.length} items`);

Get Item

const item = await client.promptDatasetItems.getPromptDatasetItem({
  id: 789,  // Item ID
});

Update Item

await client.promptDatasetItems.updatePromptDatasetItem({
  id: 789,  // Item ID
  body: {
    expectedOutput: { answer: 'Updated expected output...' },
    labels: ['updated'],
  },
});

Delete Item

await client.promptDatasetItems.deletePromptDatasetItem({
  id: 789,  // Item ID
});

Two-Step Upload Pattern

For uploading datasets from files (JSON, JSONL, CSV), use the two-step pattern: create dataset metadata first, then bulk insert items.

import { readFileSync } from 'fs';

// Step 1: Create dataset metadata
const dataset = await client.promptDatasets.createPromptDataset({
  id: promptId,
  body: { name: 'Imported Dataset' },
});

// Step 2: Parse file and bulk insert items
const raw = JSON.parse(readFileSync('test-cases.json', 'utf-8'));
const items = raw.map((row: any) => ({
  input: row.input,
  expectedOutput: row.expectedOutput,
  name: row.name,
  labels: row.labels,
}));

await client.promptDatasetItems.bulkCreatePromptDatasetItems({
  id: dataset.id,
  body: { items },
});

Type Definitions

interface PromptDataset {
  id: number;
  name: string;
  description: string | null;
  promptGroupId: string;
  metadata: any;
  createdAt: Date | null;
  updatedAt: Date | null;
  createdBy: string | null;
  labels: string[] | null;
}

interface PromptDatasetItem {
  id: number;
  datasetId: number;
  name: string | null;
  input: any;
  expectedOutput: any;
  userFeedback: string | null;
  systemFeedback: string | null;
  labels: string[] | null;
  metadata: any;
  createdAt: Date | null;
  updatedAt: Date | null;
}

interface PromptDatasetExport {
  dataset: PromptDataset;
  promptGroup: {
    promptGroupId: string;
    latestPrompt: any;
    totalVersions: number;
  };
  items: PromptDatasetItem[];
  exportedAt: Date;
}

Method Reference

Dataset Methods (`client.promptDatasets`)

Method	Description	Returns
`listPromptDatasets(request?)`	List all datasets (paginated)	`PageIterator<ListPromptDatasetsResponse>`
`listDatasetsForPrompt({ id })`	List datasets for a prompt	`ListDatasetsForPromptResponse[]`
`createPromptDataset({ id, body })`	Create dataset for prompt	`PromptDataset`
`getPromptDataset({ id })`	Get dataset by ID	`PromptDataset`
`updatePromptDataset({ id, body })`	Update dataset	`PromptDataset`
`deletePromptDataset({ id, force? })`	Delete dataset	`PromptSuccessResponse`
`clonePromptDataset({ id, body })`	Clone dataset to another prompt	`PromptDataset`
`exportPromptDataset({ id })`	Export dataset with all items	`PromptDatasetExport`

Item Methods (`client.promptDatasetItems`)

Method	Description	Returns
`listPromptDatasetItems({ id })`	List dataset items	`ListPromptDatasetItemsResponse[]`
`createPromptDatasetItem({ id, body })`	Add single item	`PromptDatasetItem`
`bulkCreatePromptDatasetItems({ id, body })`	Add multiple items	`BulkCreatePromptDatasetItemsResponse[]`
`getPromptDatasetItem({ id })`	Get item by ID	`PromptDatasetItem`
`updatePromptDatasetItem({ id, body })`	Update item	`PromptDatasetItem`
`deletePromptDatasetItem({ id })`	Delete item	`PromptSuccessResponse`

Documentation Index

​Datasets SDK

​Dataset Architecture

​List All Datasets

​List Datasets for a Prompt

​Create Dataset

​Get Dataset

​Update Dataset

​Delete Dataset

​Clone Dataset

​Export Dataset

​Dataset Items

​List Items

​Add Single Item

​Bulk Add Items

​Get Item

​Update Item

​Delete Item

​Two-Step Upload Pattern

​Type Definitions

​Method Reference

​Dataset Methods (client.promptDatasets)

​Item Methods (client.promptDatasetItems)