Skip to main content

​
Datasets SDK

Create and manage evaluation datasets programmatically. Datasets are collections of test cases used for prompt evaluation, each containing input variables and expected outputs.

​
Dataset Architecture

Datasets are scoped to a prompt group (all versions of a prompt share the same promptGroupId). This means a dataset created for prompt v1.0.0 is automatically available when testing v2.0.0.

​
List All Datasets

const result = await client.promptDatasets.listPromptDatasets({
  limit: 20,
  offset: 0,
});

for await (const page of result) {
  const { data, total, hasNext } = page.result;
  data.forEach(d => console.log(d.name, d.promptGroupId));
}
You can filter by promptId, promptGroupId, name, or createdBy: 
const result = await client.promptDatasets.listPromptDatasets({
  promptId: 123,
  name: 'baseline',
});

​
List Datasets for a Prompt

const datasets = await client.promptDatasets.listDatasetsForPrompt({
  id: 123,  // Prompt ID
});

datasets.forEach(d => console.log(d.name, `labels: ${d.labels}`));

​
Create Dataset

Datasets are created for a specific prompt: 
const dataset = await client.promptDatasets.createPromptDataset({
  id: 123,  // Prompt ID
  body: {
    name: 'Support Scenarios',
    description: 'Common customer support scenarios',
    labels: ['baseline', 'v1'],
  },
});

console.log('Created dataset:', dataset.id);

​
Get Dataset

const dataset = await client.promptDatasets.getPromptDataset({
  id: 456,  // Dataset ID
});

console.log(dataset.name, dataset.promptGroupId);

​
Update Dataset

const updated = await client.promptDatasets.updatePromptDataset({
  id: 456,
  body: {
    name: 'Updated Name',
    description: 'New description',
  },
});

​
Delete Dataset

Use force: true to delete a dataset that still contains items: 
await client.promptDatasets.deletePromptDataset({
  id: 456,
  force: true,
});

​
Clone Dataset

Clone a dataset to another prompt. Provide either targetPromptId or targetPromptGroupId: 
const cloned = await client.promptDatasets.clonePromptDataset({
  id: 456,
  body: {
    targetPromptId: 789,
    newName: 'Support Scenarios (Copy)',
  },
});

​
Export Dataset

Export a dataset with all items and metadata: 
const exported = await client.promptDatasets.exportPromptDataset({
  id: 456,
});

console.log('Dataset:', exported.dataset.name);
console.log('Items:', exported.items.length);
console.log('Exported at:', exported.exportedAt);

​
Dataset Items

​
List Items

const items = await client.promptDatasetItems.listPromptDatasetItems({
  id: 456,  // Dataset ID
});

items.forEach(item => {
  console.log('Input:', item.input);
  console.log('Expected:', item.expectedOutput);
});

​
Add Single Item

const item = await client.promptDatasetItems.createPromptDatasetItem({
  id: 456,  // Dataset ID
  body: {
    input: {
      company: 'Acme Inc',
      question: 'How do I reset my password?',
    },
    expectedOutput: {
      answer: 'To reset your password, go to Settings > Security...',
    },
    name: 'Password Reset',
    labels: ['auth', 'common'],
  },
});

​
Bulk Add Items

const result = await client.promptDatasetItems.bulkCreatePromptDatasetItems({
  id: 456,  // Dataset ID
  body: {
    items: [
      {
        input: { company: 'Acme', question: 'Pricing?' },
        expectedOutput: { answer: 'Our pricing starts at...' },
        name: 'Pricing Inquiry',
      },
      {
        input: { company: 'Acme', question: 'Refund policy?' },
        expectedOutput: { answer: 'We offer 30-day refunds...' },
        name: 'Refund Question',
      },
    ],
  },
});

console.log(`Added ${result.length} items`);

​
Get Item

const item = await client.promptDatasetItems.getPromptDatasetItem({
  id: 789,  // Item ID
});

​
Update Item

await client.promptDatasetItems.updatePromptDatasetItem({
  id: 789,  // Item ID
  body: {
    expectedOutput: { answer: 'Updated expected output...' },
    labels: ['updated'],
  },
});

​
Delete Item

await client.promptDatasetItems.deletePromptDatasetItem({
  id: 789,  // Item ID
});

​
Two-Step Upload Pattern

For uploading datasets from files (JSON, JSONL, CSV), use the two-step pattern: create dataset metadata first, then bulk insert items. 
import { readFileSync } from 'fs';

// Step 1: Create dataset metadata
const dataset = await client.promptDatasets.createPromptDataset({
  id: promptId,
  body: { name: 'Imported Dataset' },
});

// Step 2: Parse file and bulk insert items
const raw = JSON.parse(readFileSync('test-cases.json', 'utf-8'));
const items = raw.map((row: any) => ({
  input: row.input,
  expectedOutput: row.expectedOutput,
  name: row.name,
  labels: row.labels,
}));

await client.promptDatasetItems.bulkCreatePromptDatasetItems({
  id: dataset.id,
  body: { items },
});

​
Type Definitions

interface PromptDataset {
  id: number;
  name: string;
  description: string | null;
  promptGroupId: string;
  metadata: any;
  createdAt: Date | null;
  updatedAt: Date | null;
  createdBy: string | null;
  labels: string[] | null;
}

interface PromptDatasetItem {
  id: number;
  datasetId: number;
  name: string | null;
  input: any;
  expectedOutput: any;
  userFeedback: string | null;
  systemFeedback: string | null;
  labels: string[] | null;
  metadata: any;
  createdAt: Date | null;
  updatedAt: Date | null;
}

interface PromptDatasetExport {
  dataset: PromptDataset;
  promptGroup: {
    promptGroupId: string;
    latestPrompt: any;
    totalVersions: number;
  };
  items: PromptDatasetItem[];
  exportedAt: Date;
}

​
Method Reference

​
Dataset Methods (client.promptDatasets)

MethodDescriptionReturns
listPromptDatasets(request?)List all datasets (paginated)PageIterator<ListPromptDatasetsResponse>
listDatasetsForPrompt({ id })List datasets for a promptListDatasetsForPromptResponse[]
createPromptDataset({ id, body })Create dataset for promptPromptDataset
getPromptDataset({ id })Get dataset by IDPromptDataset
updatePromptDataset({ id, body })Update datasetPromptDataset
deletePromptDataset({ id, force? })Delete datasetPromptSuccessResponse
clonePromptDataset({ id, body })Clone dataset to another promptPromptDataset
exportPromptDataset({ id })Export dataset with all itemsPromptDatasetExport

​
Item Methods (client.promptDatasetItems)

MethodDescriptionReturns
listPromptDatasetItems({ id })List dataset itemsListPromptDatasetItemsResponse[]
createPromptDatasetItem({ id, body })Add single itemPromptDatasetItem
bulkCreatePromptDatasetItems({ id, body })Add multiple itemsBulkCreatePromptDatasetItemsResponse[]
getPromptDatasetItem({ id })Get item by IDPromptDatasetItem
updatePromptDatasetItem({ id, body })Update itemPromptDatasetItem
deletePromptDatasetItem({ id })Delete itemPromptSuccessResponse