Datasets & Experiments

Updated on Aug 14, 2026

Datasets and experiments allow you to systematically test and evaluate your AI agents. You create curated sets of input/output examples, run them against different models or prompt configurations, and compare results to find the best setup for your use case.

[!NOTE] Datasets and Experiments are available on paid plans.

Datasets

A dataset is a versioned collection of examples that you use as test inputs for experiments. Each example in a dataset contains:

Field	Required	Description
Input	Yes	The prompt or question to send to the model
Expected Output	No	The ideal response you expect from the model
Tags	No	Labels for categorizing and filtering examples
Metadata	No	Key-value pairs for additional context (accessible in prompt templates as `{{metadata.key}}`)

Creating a Dataset

Navigate to Improve > Datasets & Experiments in the left sidebar.
Click Dataset.
Provide a name and optional description.
Click Create Dataset.

Adding Examples to a Dataset

Once you have a dataset, add examples by using one of the following methods:

Manual entry—Click Add Examples > Add Manually, then fill in the input, expected output, tags, and metadata fields.

Import from file—Click Add Examples > Import from File and upload a CSV or JSON file. After upload, map your file columns to dataset fields (Input, Expected Output, Tags, Metadata, or Skip). A preview shows the first 10 rows before import.

Import from traces—Click Add Examples > Select from Traces to pull real inputs from your collected observations into the dataset.

Versioning

Every time you modify the dataset (add, edit, or delete examples), the platform creates a new version. You can:

Browse the version history from the version dropdown on the dataset detail page.
View examples as they existed in any historical version.
Select a specific version when running an experiment to ensure reproducible results.

Click View Full History in the version dropdown to see all versions with timestamps.

Filtering and Managing Columns

On the dataset detail page, use the Columns button to show or hide columns (Input, Expected Output, Tags, Metadata). Use the Filters button to narrow examples by tags or metadata keys.

Experiments

An experiment runs a dataset's examples through an LLM and optionally evaluates the outputs by using one or more evaluators. Use experiments to:

Test how different models respond to the same inputs.
Compare prompt template variations.
Measure quality with automated evaluators.
Track improvements over time.

Running an Experiment

Open a dataset and click Run Experiment (or navigate to the Experiments tab within the dataset and click Run Experiment).
Configure the experiment:

Setting	Required	Description
Experiment Name	Yes	A descriptive name for this run
LLM Integration	Yes	Which model/provider to use for generation
Dataset Version	Yes	Which version of the dataset to evaluate
Prompt Template	Yes	The template sent to the model (must include `{{input}}`)
Evaluators	No	Automated scoring methods to apply to each result

Click Run Experiment.

Prompt Templates

The prompt template defines what the model receives for each example. Use template variables to inject dataset values:

{{input}}—Required. The example's input field.
{{metadata.<key>}}—Any metadata key from the dataset items (for example, {{metadata.language}}).

Example template:

You are a helpful assistant. Answer the following question concisely.

Question: {{input}}

Evaluators

Evaluators automatically score each model output. Two types are available:

Exact Match—A deterministic check that compares the model output against the expected output. Returns a binary pass/fail.

LLM Judge—Uses a separate LLM to evaluate the output. You configure:

Custom evaluation name
LLM integration (can differ from the generation model)
Scoring prompt with criteria
Template variables ({{input}}, {{output}}, {{expected_output}})

You can add multiple evaluators to a single experiment. Each LLM Judge evaluator can use a different model and scoring criteria.

You can load evaluator configurations from saved evaluator templates, or configure them from scratch in the evaluator dialog.

Experiment Results

After an experiment starts, it progresses through the following statuses:

Status	Meaning
Pending	Queued, waiting to start
Generating	Sending inputs to the LLM and collecting outputs
Evaluating	Running evaluator scoring on generated outputs
Completed	All items processed successfully
Failed	Experiment stopped due to errors (partial results may be available)

The results page displays the following sections:

Configuration—Model, evaluators, and prompt template used
Progress—Live progress bar during execution
Summary cards—Total items, success rate, and average latency
Results grid—Row-by-row view with input, expected output, model output, evaluator scores, status, and latency
Score distribution charts—Histogram of scores for each evaluator

Click any result row to see the full detail including complete input/output text and individual evaluation reasoning.

Comparing Experiments

Select two or more experiments from the Experiments tab using the checkboxes and click Compare Selected. The comparison view displays the following sections:

Summary—Side-by-side cards with model, success rate, average latency, token usage, and evaluator configurations for each experiment
Metrics Overview—Bar charts comparing score distributions across experiments for each evaluator
Row-by-row comparison—Grid showing each input alongside every experiment's output and evaluator scores

Use experiment comparisons to identify which model, prompt, or configuration produces the best results for your specific use case.